Ground truth generalizability affects performance of the artificial intelligence model in automated vertebral fracture detection on plain lateral radiographs of the spine

Development and validation of deep learning models for identifying the brand of pedicle screws on plain spine radiographs

Background

In spinal revision surgery, previous pedicle screws (PS) may need to be replaced with new implants. Failure to accurately identify the brand of PS-based instrumentation preoperatively may increase the risk of perioperative complications. This study aimed to develop and validate an optimal deep learning (DL) model to identify the brand of PS-based instrumentation on plain radiographs of spine (PRS) using anteroposterior (AP) and lateral images.

Methods

A total of 529 patients who received PS-based instrumentation from seven manufacturers were enrolled in this retrospective study. The postoperative PRS were gathered as ground truths. The training, validation, and testing datasets contained 338, 85, and 106 patients, respectively. YOLOv5 was used to crop out the screws' trajectory, and the EfficientNet-b0 model was used to develop single models (AP, Lateral, Merge, and Concatenated) based on the different PRS images. The ensemble models were different combinations of the single models. Primary outcomes were the models' performance in accuracy, sensitivity, precision, F1-score, kappa value, and area under the curve (AUC). Secondary outcomes were the relative performance of models versus human readers and external validation of the DL models.

Results

The Lateral model had the most stable performance among single models. The discriminative performance was improved by the ensemble method. The AP + Lateral ensemble model had the most stable performance, with an accuracy of 0.9434, F1 score of 0.9388, and AUC of 0.9834. The performance of the ensemble models was comparable to that of experienced orthopedic surgeons and superior to that of inexperienced orthopedic surgeons. External validation revealed that the Lat + Concat ensemble model had the best accuracy (0.9412).

Conclusion

The DL models demonstrated stable performance in identifying the brand of PS-based instrumentation based on AP and/or lateral images of PRS, which may assist orthopedic spine surgeons in preoperative revision planning in clinical practice.

Machine learning value in the diagnosis of vertebral fractures: A systematic review and meta-analysis

PURPOSE

To evaluate the diagnostic accuracy of machine learning (ML) in detecting vertebral fractures, considering varying fracture classifications, patient populations, and imaging approaches.

METHOD

A systematic review and meta-analysis were conducted by searching PubMed, Embase, Cochrane Library, and Web of Science up to December 31, 2023, for studies using ML for vertebral fracture diagnosis. Bias risk was assessed using QUADAS-2. A bivariate mixed-effects model was used for the meta-analysis. Meta-analyses were performed according to five task types (vertebral fractures, osteoporotic vertebral fractures, differentiation of benign and malignant vertebral fractures, differentiation of acute and chronic vertebral fractures, and prediction of vertebral fractures). Subgroup analyses were conducted by different ML models (including ML and DL) and modeling methods (including CT, X-ray, MRI, and clinical features).

RESULTS

Eighty-one studies were included. ML demonstrated a diagnostic sensitivity of 0.91 and specificity of 0.95 for vertebral fractures. Subgroup analysis showed that DL (SROC 0.98) and CT (SROC 0.98) performed best overall. For osteoporotic fractures, ML showed a sensitivity of 0.93 and specificity of 0.96, with DL (SROC 0.99) and X-ray (SROC 0.99) performing better. For differentiating benign from malignant fractures, ML achieved a sensitivity of 0.92 and specificity of 0.93, with DL (SROC 0.96) and MRI (SROC 0.97) performing best. For differentiating acute from chronic vertebral fractures, ML showed a sensitivity of 0.92 and specificity of 0.93, with ML (SROC 0.96) and CT (SROC 0.97) performing best. For predicting vertebral fractures, ML had a sensitivity of 0.76 and specificity of 0.87, with ML (SROC 0.80) and clinical features (SROC 0.86) performing better.

CONCLUSIONS

ML, especially DL models applied to CT, MRI, and X-ray, shows high diagnostic accuracy for vertebral fractures. ML also effectively predicts osteoporotic vertebral fractures, aiding in tailored prevention strategies. Further research and validation are required to confirm ML's clinical efficacy.

Machine Learning and Deep Learning in Spinal Injury: A Narrative Review of Algorithms in Diagnosis and Prognosis

Spinal injuries, including cervical and thoracolumbar fractures, continue to be a major public health concern. Recent advancements in machine learning and deep learning technologies offer exciting prospects for improving both diagnostic and prognostic approaches in spinal injury care. This narrative review systematically explores the practical utility of these computational methods, with a focus on their application in imaging techniques such as computed tomography (CT) and magnetic resonance imaging (MRI), as well as in structured clinical data. Of the 39 studies included, 34 were focused on diagnostic applications, chiefly using deep learning to carry out tasks like vertebral fracture identification, differentiation between benign and malignant fractures, and AO fracture classification. The remaining five were prognostic, using machine learning to analyze parameters for predicting outcomes such as vertebral collapse and future fracture risk. This review highlights the potential benefit of machine learning and deep learning in spinal injury care, especially their roles in enhancing diagnostic capabilities, detailed fracture characterization, risk assessments, and individualized treatment planning.

Artificial intelligence in fracture detection with different image modalities and data types: A systematic review and meta-analysis

Artificial Intelligence (AI), encompassing Machine Learning and Deep Learning, has increasingly been applied to fracture detection using diverse imaging modalities and data types. This systematic review and meta-analysis aimed to assess the efficacy of AI in detecting fractures through various imaging modalities and data types (image, tabular, or both) and to synthesize the existing evidence related to AI-based fracture detection. Peer-reviewed studies developing and validating AI for fracture detection were identified through searches in multiple electronic databases without time limitations. A hierarchical meta-analysis model was used to calculate pooled sensitivity and specificity. A diagnostic accuracy quality assessment was performed to evaluate bias and applicability. Of the 66 eligible studies, 54 identified fractures using imaging-related data, nine using tabular data, and three using both. Vertebral fractures were the most common outcome (n = 20), followed by hip fractures (n = 18). Hip fractures exhibited the highest pooled sensitivity (92%; 95% CI: 87-96, p< 0.01) and specificity (90%; 95% CI: 85-93, p< 0.01). Pooled sensitivity and specificity using image data (92%; 95% CI: 90-94, p< 0.01; and 91%; 95% CI: 88-93, p < 0.01) were higher than those using tabular data (81%; 95% CI: 77-85, p< 0.01; and 83%; 95% CI: 76-88, p < 0.01), respectively. Radiographs demonstrated the highest pooled sensitivity (94%; 95% CI: 90-96, p < 0.01) and specificity (92%; 95% CI: 89-94, p< 0.01). Patient selection and reference standards were major concerns in assessing diagnostic accuracy for bias and applicability. AI displays high diagnostic accuracy for various fracture outcomes, indicating potential utility in healthcare systems for fracture diagnosis. However, enhanced transparency in reporting and adherence to standardized guidelines are necessary to improve the clinical applicability of AI. Review Registration: PROSPERO (CRD42021240359).

Generalizability of Deep Learning Classification of Spinal Osteoporotic Compression Fractures on Radiographs Using an Adaptation of the Modified-2 Algorithm-Based Qualitative Criteria

RATIONALE AND OBJECTIVES

Spinal osteoporotic compression fractures (OCFs) can be an early biomarker for osteoporosis but are often subtle, incidental, and underreported. To ensure early diagnosis and treatment of osteoporosis, we aimed to build a deep learning vertebral body classifier for OCFs as a critical component of our future automated opportunistic screening tool.

MATERIALS AND METHODS

We retrospectively assembled a local dataset, including 1790 subjects and 15,050 vertebral bodies (thoracic and lumbar). Each vertebral body was annotated using an adaption of the modified-2 algorithm-based qualitative criteria. The Osteoporotic Fractures in Men (MrOS) Study dataset provided thoracic and lumbar spine radiographs of 5994 men from six clinical centers. Using both datasets, five deep learning algorithms were trained to classify each individual vertebral body of the spine radiographs. Classification performance was compared for these models using multiple metrics, including the area under the receiver operating characteristic curve (AUC-ROC), sensitivity, specificity, and positive predictive value (PPV).

RESULTS

Our best model, built with ensemble averaging, achieved an AUC-ROC of 0.948 and 0.936 on the local dataset's test set and the MrOS dataset's test set, respectively. After setting the cutoff threshold to prioritize PPV, this model achieved a sensitivity of 54.5% and 47.8%, a specificity of 99.7% and 99.6%, and a PPV of 89.8% and 94.8%.

CONCLUSION

Our model achieved an AUC-ROC>0.90 on both datasets. This testing shows some generalizability to real-world clinical datasets and a suitable performance for a future opportunistic osteoporosis screening tool.

Using Artificial Intelligence to Diagnose Osteoporotic Vertebral Fractures on Plain Radiographs

Osteoporotic vertebral fracture (OVF) is a risk factor for morbidity and mortality in elderly population, and accurate diagnosis is important for improving treatment outcomes. OVF diagnosis suffers from high misdiagnosis and underdiagnosis rates, as well as high workload. Deep learning methods applied to plain radiographs, a simple, fast, and inexpensive examination, might solve this problem. We developed and validated a deep-learning-based vertebral fracture diagnostic system using area loss ratio, which assisted a multitasking network to perform skeletal position detection and segmentation and identify and grade vertebral fractures. As the training set and internal validation set, we used 11,397 plain radiographs from six community centers in Shanghai. For the external validation set, 1276 participants were recruited from the outpatient clinic of the Shanghai Sixth People's Hospital (1276 plain radiographs). Radiologists performed all X-ray images and used the Genant semiquantitative tool for fracture diagnosis and grading as the ground truth data. Accuracy, sensitivity, specificity, positive predictive value, and negative predictive value were used to evaluate diagnostic performance. The AI_OVF_SH system demonstrated high accuracy and computational speed in skeletal position detection and segmentation. In the internal validation set, the accuracy, sensitivity, and specificity with the AI_OVF_SH model were 97.41%, 84.08%, and 97.25%, respectively, for all fractures. The sensitivity and specificity for moderate fractures were 88.55% and 99.74%, respectively, and for severe fractures, they were 92.30% and 99.92%. In the external validation set, the accuracy, sensitivity, and specificity for all fractures were 96.85%, 83.35%, and 94.70%, respectively. For moderate fractures, the sensitivity and specificity were 85.61% and 99.85%, respectively, and 93.46% and 99.92% for severe fractures. Therefore, the AI_OVF_SH system is an efficient tool to assist radiologists and clinicians to improve the diagnosing of vertebral fractures. © 2023 The Authors. Journal of Bone and Mineral Research published by Wiley Periodicals LLC on behalf of American Society for Bone and Mineral Research (ASBMR).

Assessment of Fusion After Anterior Cervical Discectomy and Fusion Using Convolutional Neural Network Algorithm

BACKGROUND

A convolutional neural network (CNN) is a deep learning (DL) model specialized for image processing, analysis, and classification.

OBJECTIVE

In this study, we evaluated whether a CNN model using lateral cervical spine radiographs as input data can help assess fusion after anterior cervical discectomy and fusion (ACDF).

STUDY DESIGN

Diagnostic imaging study using DL.

PATIENT SAMPLE

We included 187 patients who underwent ACDF and fusion assessment with postoperative one-year computed tomography and neutral and dynamic lateral cervical spine radiographs.

OUTCOME MEASURES

The performance of the CNN-based DL algorithm was evaluated in terms of accuracy and area under the curve.

MATERIALS AND METHODS

Fusion or nonunion was confirmed by cervical spine computed tomography. Among the 187 patients, 69.5% (130 patients) were randomly selected as the training set, and the remaining 30.5% (57 patients) were assigned to the validation set to evaluate model performance. Radiographs of the cervical spine were used as input images to develop a CNN-based DL algorithm. The CNN algorithm used three radiographs (neutral, flexion, and extension) per patient and showed the diagnostic results as fusion (0) or nonunion (1) for each radiograph. By combining the results of the three radiographs, the final decision for a patient was determined to be fusion (fusion ≥2) or nonunion (fusion ≤1). By combining the results of the three radiographs, the final decision for a patient was determined as fusion (fusion ≥2) or nonunion (nonunion ≤1).

RESULTS

The CNN-based DL model demonstrated an accuracy of 89.5% and an area under the curve of 0.889 (95% confidence interval, 0.793-0.984).

CONCLUSION

The CNN algorithm for fusion assessment after ACDF trained using lateral cervical radiographs showed a relatively high diagnostic accuracy of 89.5% and is expected to be a useful aid in detecting pseudarthrosis.

Deep Learning Classification of Spinal Osteoporotic Compression Fractures on Radiographs using an Adaptation of the Genant Semiquantitative Criteria

RATIONALE AND OBJECTIVES

Osteoporosis affects 9% of individuals over 50 in the United States and 200 million women globally. Spinal osteoporotic compression fractures (OCFs), an osteoporosis biomarker, are often incidental and under-reported. Accurate automated opportunistic OCF screening can increase the diagnosis rate and ensure adequate treatment. We aimed to develop a deep learning classifier for OCFs, a critical component of our future automated opportunistic screening tool.

MATERIALS AND METHODS

The dataset from the Osteoporotic Fractures in Men Study comprised 4461 subjects and 15,524 spine radiographs. This dataset was split by subject: 76.5% training, 8.5% validation, and 15% testing. From the radiographs, 100,409 vertebral bodies were extracted, each assigned one of two labels adapted from the Genant semiquantitative system: moderate to severe fracture vs. normal/trace/mild fracture. GoogLeNet, a deep learning model, was trained to classify the vertebral bodies. The classification threshold on the predicted probability of OCF outputted by GoogLeNet was set to prioritize the positive predictive value (PPV) while balancing it with the sensitivity. Vertebral bodies with the top 0.75% predicted probabilities were classified as moderate to severe fracture.

RESULTS

Our model yielded a sensitivity of 59.8%, a PPV of 91.2%, and an F1 score of 0.72. The areas under the receiver operating characteristic curve (AUC-ROC) and the precision-recall curve were 0.99 and 0.82, respectively.

CONCLUSION

Our model classified vertebral bodies with an AUC-ROC of 0.99, providing a critical component for our future automated opportunistic screening tool. This could lead to earlier detection and treatment of OCFs.

A software program for automated compressive vertebral fracture detection on elderly women's lateral chest radiograph: Ofeye 1.0

Background

Because osteoporotic vertebral fracture (OVF) on chest radiographs is commonly missed in radiological reports, we aimed to develop a software program which offers automated detection of compressive vertebral fracture (CVF) on lateral chest radiographs, and which emphasizes CVF detection specificity with a low false positivity rate.

Methods

For model training, we retrieved 3,991 spine radiograph cases and 1,979 chest radiograph cases from 16 sources, with among them in total 1,404 cases had OVF. For model testing, we retrieved 542 chest radiograph cases and 162 spine radiograph cases from four independent clinics, with among them 215 cases had OVF. All cases were female subjects, and except for 31 training data cases which were spine trauma cases, all the remaining cases were post-menopausal women. Image data included DICOM (Digital Imaging and Communications in Medicine) format, hard film scanned PNG (Portable Network Graphics) format, DICOM exported PNG format, and PACS (Picture Archiving and Communication System) downloaded resolution reduced DICOM format. OVF classification included: minimal and mild grades with <20% or ≥20-25% vertebral height loss respectively, moderate grade with ≥25-40% vertebral height loss, severe grade with ≥40%-2/3 vertebral height loss, and collapsed grade with ≥2/3 vertebral height loss. The CVF detection base model was mainly composed of convolution layers that include convolution kernels of different sizes, pooling layers, up-sampling layers, feature merging layers, and residual modules. When the model loss function could not be further decreased with additional training, the model was considered to be optimal and termed 'base-model 1.0'. A user-friendly interface was also developed, with the synthesized software termed 'Ofeye 1.0'.

Results

Counting cases and with minimal and mild OVFs included, base-model 1.0 demonstrated a specificity of 97.1%, a sensitivity of 86%, and an accuracy of 93.9% for the 704 testing cases. In total, 33 OVFs in 30 cases had a false negative reading, which constituted a false negative rate of 14.0% (30/215) by counting all OVF cases. Eighteen OVFs in 15 cases had OVFs of ≥ moderate grades missed, which constituted a false negative rate of 7.0% (15/215, i.e., sensitivity 93%) if only counting cases with ≥ moderate grade OVFs missed. False positive reading was recorded in 13 vertebrae in 13 cases (one vertebra in each case), which constituted a false positivity rate of 2.7% (13/489). These vertebrae with false positivity labeling could be readily differentiated from a true OVF by a human reader. The software Ofeye 1.0 allows 'batch processing', for example, 100 radiographs can be processed in a single operation. This software can be integrated into hospital PACS, or installed in a standalone personal computer.

Conclusions

A user-friendly software program was developed for CVF detection on elderly women's lateral chest radiographs. It has an overall low false positivity rate, and for moderate and severe CVFs an acceptably low false negativity rate. The integration of this software into radiological practice is expected to improve osteoporosis management for elderly women.