Multireader Diagnostic Accuracy Imaging Studies: Fundamentals of Design and Analysis

Multicenter External Validation of the Accuracy of Computed Tomography Criteria for Detecting Thoracolumbar Posterior Ligamentous Complex Injury

BACKGROUND AND OBJECTIVE

Recent studies have proposed computed tomography (CT) criteria for posterior ligamentous complex (PLC) injury: disrupted if ≥2 CT findings, indeterminate if single finding, and intact if 0 CT findings. The study aims to validate the CT criteria for PLC injury externally.

METHODS

Three level 1 trauma centers enrolled 614 consecutive patients with acute thoracolumbar fractures (T1-L5) who received CT and MRI. Three reviewers from each center assessed CT for facet joint malalignment, horizontal laminar fracture, spinous process fracture, and interspinous widening and MRI for disrupted PLC, defined as black stripe discontinuity. The primary outcome is the diagnostic accuracy of CT criteria (0, 1, ≥2 findings) in detecting disrupted PLC on MRI using all CT readings. A subgroup analysis was performed for each participating center and reviewer. The inter-reader agreement on PLC status on MRI and CT criteria was assessed using Fleiss Kappa ( k ).

RESULTS

The positive predictive value for PLC injury was 0 findings 3%, single positive CT 43%, and ≥2 CT findings in 94%. The accuracy measures were consistent across various centers and reviewers. The area under the curve for ≥1 CT finding in detecting PLC injury ranged from 90% to 97%, indicating excellent discrimination for all centers. The inter-reader k on PLC status by MRI and overall CT findings was substantial ( k > 0.60).

CONCLUSION

This study externally validates the previously proposed CT criteria for PLC injury. A total of ≥2 positive CT findings or 0 CT findings can be used as criteria for a disrupted PLC (B-type injury) or intact PLC (A-type injuries), respectively, without added MRI. A single CT finding implies indeterminate PLC status and the need for further MRI assessment. The CT criteria will potentially guide MRI indications and treatment decisions for neurologically intact thoracolumbar burst fractures.

Diagnostic Performance of Fast Brain MRI Compared with Routine Clinical MRI in Patients with Glioma Grades 3 and 4: A Pilot Study

BACKGROUND AND PURPOSE

EPIMix is a fast brain MRI technique not previously investigated in patients with grade 3 and 4 gliomas. This pilot study aimed to investigate the diagnostic performance of EPIMix in the radiological treatment evaluation of adult patients with grade 3 and 4 gliomas compared with routine clinical MRI (rcMRI).

MATERIALS AND METHODS

Patients with grade 3 and 4 gliomas investigated with rcMRI and EPIMix were retrospectively included in the study. Three readers (R1-R3) participated in the radiological assessment applying the Response Assessment for Neuro-Oncology (RANO 2.0) criteria, of whom two (R1 and R2) independently evaluated EPIMix and later rcMRI by measuring contrast-enhancing and non-contrast-enhancing tumor regions at each follow-up. For cases with discrepant evaluations, an unblinded side-by-side (EPIMix and rcMRI) reading was performed together with a third reader (R3). Comparisons between methods (EPIMix versus rcMRI) were performed using the weighted Cohen κ. The sensitivity and specificity to progressive disease (PD) on a follow-up scan were calculated for EPIMix compared with rcMRI with receiver operating characteristic curves (ROC) to assess the area under the curve (AUC).

RESULTS

Of 35 patients (mean age, 53 years; 31% women), a total of 93 MRIs encompassing 58 follow-up investigations showed PD at a blinded reading in 33% of EPIMix (19/58, R1-2), while in 31% (18/58 exams, R1), and 34% (20/58 exams, R2) of rcMRI. An almost perfect agreement for tumor category assessment was found between EPIMix and rcMRI (EPIMixR1 versus rcMRIR1 κ = 0.96; EPIMixR2 versus rcMRIR2 κ = 0.89). The sensitivity for EPIMix to detect PD was 1.00 (0.81-1.00) for R1 and 0.90 (0.68-0.99) for R2, while the specificity was 0.97 (0.86-1.00) for R1 and R2. The AUC for PD was 0.99 for R1 (EPIMixR1 versus rcMRIR1) and 0.94 for R2 (EPIMixR2 versus rcMRIR2). The P value of the DeLong test AUCR1 versus AUCR2 was P = .20 (R1-R2).

CONCLUSIONS

In this pilot study, EPIMix was used as a fast MRI alternative for treatment evaluation of patients with glioma grades 3 and 4, with high but slightly lower diagnostic performance than rcMRI.

SMART: Development and Application of a Multimodal Multi-organ Trauma Screening Model for Abdominal Injuries in Emergency Settings

RATIONALE AND OBJECTIVES

Effective trauma care in emergency departments necessitates rapid diagnosis by interdisciplinary teams using various medical data. This study constructed a multimodal diagnostic model for abdominal trauma using deep learning on non-contrast computed tomography (CT) and unstructured text data, enhancing the speed and accuracy of solid organ assessments.

MATERIALS AND METHODS

Data were collected from patients undergoing abdominal CT scans. The SMART model (Screening for Multi-organ Assessment in Rapid Trauma) classifies trauma using text data (SMART_GPT), non-contrast CT scans (SMART_Image), or both. SMART_GPT uses the GPT-4 embedding API for text feature extraction, whereas SMART_Image incorporates nnU-Net and DenseNet121 for segmentation and classification. A composite model was developed by integrating multimodal data via logistic regression of SMART_GPT, SMART_Image, and patient demographics (age and gender).

RESULTS

This study included 2638 patients (459 positive, 2179 negative abdominal trauma cases). A trauma-based dataset included 1006 patients with 1632 real continuous data points for testing. SMART_GPT achieved a sensitivity of 81.3% and an area under the receiver operating characteristic curve (AUC) of 0.88 based on unstructured text data. SMART_Image exhibited a sensitivity of 87.5% and an AUC of 0.81 on non-contrast CT data, with the average sensitivity exceeding 90% at the organ level. The integrated SMART model achieved a sensitivity of 93.8% and an AUC of 0.88. In emergency department simulations, SMART reduced waiting times by over 64.24%.

CONCLUSION

SMART provides rapid, objective trauma diagnostics, improving emergency care efficiency, reducing patient wait times, and enabling multimodal screening in diverse emergency contexts.

Artificial intelligence driven 3D reconstruction for enhanced lung surgery planning

The increasing complexity of lung surgeries necessitates the need for enhanced imaging support to improve the precision and efficiency of preoperative planning. Despite the promise of 3D reconstruction, clinical adoption remains limited due to time constraints and insufficient validation. To address this, we evaluate an artificial intelligence-driven 3D reconstruction system for pulmonary vessels and bronchi in a retrospective, multi-center multi-reader multi-case study. Using a two-stage crossover design, ten thoracic surgeons assess 140 cases with and without the system's assistance. The system significantly improves the accuracy of anatomical variant identification by 8% (p < 0.01), reducing errors by 41%. Improvements in secondary endpoints are also observed. Operation procedure selection accuracy is improved by 8%, with a 35% decrease in errors. Preoperative planning time is decreased by 25%, and user satisfaction is high at 99%. These benefits are consistent across surgeons of varying experience. In conclusion, the artificial intelligence-driven 3D reconstruction system significantly improves the identification of anatomical variants, addressing a critical need in preoperative planning for thoracic surgery.

Development and Validation of a Sham-AI Model for Intracranial Aneurysm Detection at CT Angiography

Purpose To evaluate a sham-artificial intelligence (AI) model acting as a placebo control for a standard-AI model for diagnosis of intracranial aneurysm. Materials and Methods This retrospective crossover, blinded, multireader, multicase study was conducted from November 2022 to March 2023. A sham-AI model with near-zero sensitivity and similar specificity to a standard AI model was developed using 16 422 CT angiography examinations. Digital subtraction angiography-verified CT angiographic examinations from four hospitals were collected, half of which were processed by standard AI and the others by sham AI to generate sequence A; sequence B was generated in the reverse order. Twenty-eight radiologists from seven hospitals were randomly assigned to either sequence and then assigned to the other sequence after a washout period. The diagnostic performances of radiologists alone, radiologists with standard-AI assistance, and radiologists with sham-AI assistance were compared using sensitivity and specificity, and radiologists' susceptibility to sham AI suggestions was assessed. Results The testing dataset included 300 patients (median age, 61.0 years [IQR, 52.0-67.0]; 199 male), 50 of whom had aneurysms. Standard AI and sham AI performed as expected (sensitivity, 96.0% vs 0.0%; specificity, 82.0% vs 76.0%). The differences in sensitivity and specificity between standard AI-assisted and sham AI-assisted readings were 20.7% (95% CI: 15.8, 25.5 [superiority]) and 0.0% (95% CI: -2.0, 2.0 [noninferiority]), respectively. The difference between sham AI-assisted readings and radiologists alone was -2.6% (95% CI: -3.8, -1.4 [noninferiority]) for both sensitivity and specificity. After sham-AI suggestions, 5.3% (44 of 823) of true-positive and 1.2% (seven of 577) of false-negative results of radiologists alone were changed. Conclusion Radiologists' diagnostic performance was not compromised when aided by the proposed sham-AI model compared with their unassisted performance. Keywords: CT Angiography, Vascular, Intracranial Aneurysm, Sham AI Supplemental material is available for this article. Published under a CC BY 4.0 license. See also commentary by Mayfield and Romero in this issue.

Evaluating a clinically available artificial intelligence model for intracranial aneurysm detection: a multi-reader study and algorithmic audit

PURPOSE

We aimed to validate a clinically available artificial intelligence (AI) model to assist general radiologists in the detection of intracranial aneurysm (IA) in a multi-reader multi-case (MRMC) study, and to explore its performance in routine clinical settings.

METHODS

Two distinct cohorts of head CT angiography (CTA) data were assembled to validate an AI model. Cohort 1, comprising gold-standard consecutive CTA cases, was used in an MRMC study involving six board-certified general radiologists. Cohort 2, representing clinical CTA cases, was used to simulate a routine clinical setting. Following these evaluations, an algorithmic audit was conducted to identify any unusual or unexpected behaviors exhibited by the model.

RESULTS

Cohort 1 consisted of 131 CTA cases, while Cohort 2 included 515 CTA cases. In the MRMC study, the AI-assisted strategy demonstrated a significant improvement in aneurysm diagnostic performance, with the area under the receiver operating characteristic curve increasing from 0.815 (95%CI: 0.754-0.875) to 0.875 (95%CI: 0.831-0.921; p = 0.008). In the AI-based first-reader study, 60.4% of the CTA cases were identified as negative by the AI, with a high negative predictive value of 0.994 (95%CI: 0.977-0.999). The algorithmic audit highlighted two issues for improvement: the accurate detection of tiny aneurysms and the effective exclusion of false-positive lesions.

CONCLUSION

This study highlights the clinical utility of a high-performance AI model in detecting IAs, significantly improving general radiologists' diagnostic performance with the potential to reduce their workload in routine clinical practice. The algorithmic audit offers insights to guide the development and validation of future AI models.

Artificial intelligence in emergency and trauma radiology: ASER AI/ML expert panel Delphi consensus statement on research guidelines, practices, and priorities

BACKGROUND

Emergency/trauma radiology artificial intelligence (AI) is maturing along all stages of technology readiness, with research and development (R&D) ranging from data curation and algorithm development to post-market monitoring and retraining.

PURPOSE

To develop an expert consensus document on best research practices and methodological priorities for emergency/trauma radiology AI.

METHODS

A Delphi consensus exercise was conducted by the ASER AI/ML expert panel between 2022-2024. In phase 1, a steering committee (7 panelists) established key themes- curation; validity; human factors; workflow; barriers; future avenues; and ethics- and generated an edited, collated long-list of statements. In phase 2, two Delphi rounds using anonymous RAND/UCLA Likert grading were conducted with web-based data capture (round 1) and a bespoke excel document with literature hyperlinks (round 2). Between rounds, editing and knowledge synthesis helped maximize consensus. Statements reaching ≥80% agreement were included in the final document.

RESULTS

Delphi rounds 1 and 2 consisted of 81 and 78 items, respectively.18/21 expert panelists (86%) responded to round 1, and 15 to round 2 (17% drop-out). Consensus was reached for 65 statements. Observations were summarized and contextualized. Statements with unanimous consensus centered around transparent methodologic reporting; testing for generalizability and robustness with external data; and benchmarking performance with appropriate metrics and baselines. A manuscript draft was circulated to panelists for editing and final approval.

CONCLUSIONS

The document is meant as a framework to foster best-practices and further discussion among researchers working on various aspects of emergency and trauma radiology AI.

Effect of Artificial Intelligence as a Second Reader on the Lung Nodule Detection and Localization Accuracy of Radiologists and Non-radiology Physicians in Chest Radiographs: A Multicenter Reader Study

RATIONALE AND OBJECTIVES

Missed nodules in chest radiographs (CXRs) are common occurrences. We assessed the effect of artificial intelligence (AI) as a second reader on the accuracy of radiologists and non-radiology physicians in lung nodule detection and localization in CXRs.

MATERIALS AND METHODS

This retrospective study using the multi-reader multi-case design included 300 CXRs acquired from 40 hospitals across the US. All CXRs had a paired follow-up image (chest CT or CXR) to augment the ground truth establishment for the presence and location of nodules on CXRs by five independent thoracic radiologists. 15 readers (nine radiologists and six non-radiology physicians) read each CXR twice in a second-reader paradigm, once without AI and then immediately with AI assistance. The primary analysis assessed the difference in area-under-the-alternative-free-response-receiver-operating-characteristic-curve (AFROC) of readers with and without AI. Case-level area-under-the-receiver-operating-characteristic-curve (AUROC), sensitivity, and specificity were assessed in secondary analyses.

RESULTS

A total of 300 CXRs (147 with nodules, 153 without nodules) from 300 patients (mean age, 64 years ± 15 [standard deviation]; 174 women) were included. The mean AFROC of readers was 0.73 without AI and 0.81 with AI (95% CI of difference, 0.05-0.10). Case-level AUROC was 0.77 without AI and 0.84 with AI (95% CI of difference, 0.04-0.09). Case-level sensitivity was 72.8% and 83.5% (95% CI of difference, 6.8-14.6) and specificity was 71.1% and 72.0% (95% CI of difference, -0.8-2.6) without and with AI, respectively.

CONCLUSION

Using AI, readers detected and localized more nodules without any significant difference in false positive interpretations.

The use of predetermined change control plans to enable the release of new versions of software as a medical device

OBJECTIVES

This study investigates how Predetermined Change Control Plans (PCCPs) can support the Software Development Life Cycle (SDLC) of certain Software as a Medical Device (SaMD).

METHODS

Targeted surveys collected qualitative and quantitative data on the current regulatory change process for SaMD; the use of PCCPs; the potential parameters of PCCPs in terms of technical, clinical, usability, and administrative changes to SaMD; and whether PCCPs could be used more broadly for all SaMDs.

RESULTS

Results indicate that the current regulatory approach is not fit for purpose, specifically regarding fast-moving SaMD or continuous-learning AI SaMD. There was strong support for PCCPs to cover device technology, usability, and administrative changes, while clinical changes had limited support and required further investigation. The EU lags behind the US and now the UK in addressing these challenges and should look to legislate and implement PCCPs to ensure ongoing innovation and investment in digital health technologies.

CONCLUSION

This work is novel in the gathering of meaningful input from experts, practitioners, and regulatory professionals within the SaMD industry located in the EU, UK, and US on the value and need for PCCPs. This study has implications for practice and policy as it can inform SaMD guidance and legislation.

Added value of artificial intelligence solutions for arterial stenosis detection on head and neck CT angiography: A randomized crossover multi-reader multi-case study

PURPOSE

The purpose of this study was to investigate the added value of artificial intelligence (AI) solutions for the detection of arterial stenosis (AS) on head and neck CT angiography (CTA).

MATERIALS AND METHODS

Patients who underwent head and neck CTA examinations at two hospitals were retrospectively included. CTA examinations were randomized into group 1 (without AI-washout-with AI) and group 2 (with AI-washout-without AI), and six readers (two radiology residents, two non-neuroradiologists, and two neuroradiologists) independently interpreted each CTA examination without and with AI solutions. Additionally, reading time was recorded for each patient. Digital subtraction angiography was used as the standard of reference. The diagnostic performance for AS at lesion and patient levels with four AS thresholds (30 %, 50 %, 70 %, and 100 %) was assessed by calculating sensitivity, false-positive lesions index (FPLI), specificity, and accuracy.

RESULTS

A total of 268 patients (169 men, 63.1 %) with a median age of 65 years (first quartile, 57; third quartile, 72; age range: 28-88 years) were included. At the lesion level, AI improved the sensitivity of all readers by 5.2 % for detecting AS ≥ 30 % (P < 0.001). Concurrently, AI reduced the FPLI of all readers and specifically neuroradiologists for detecting non-occlusive AS (all P < 0.05). At the patient level, AI improved the accuracy of all readers by 4.1 % (73.9 % [1189/1608] without AI vs. 78.0 % [1254/1608] with AI) (P < 0.001). Sensitivity for AS ≥ 30 % and the specificity for AS ≥ 70 % increased for all readers with AI assistance (P = 0.01). The median reading time for all readers was reduced from 268 s without AI to 241 s with AI (P< 0.001).

CONCLUSION

AI-assisted diagnosis improves the performance of radiologists in detecting head and neck AS, and shortens reading time.

The translation of in-house imaging AI research into a medical device ensuring ethical and regulatory integrity

This manuscript delineates the pathway from in-house research on Artificial Intelligence (AI) to the development of a medical device, addressing critical phases including conceptualization, development, validation, and regulatory compliance. Key stages in the transformation process involve identifying clinical needs, data management, model training, and rigorous validation to ensure AI models are both robust and clinically relevant. Continuous post-deployment surveillance is essential to maintain performance and adapt to changes in clinical practice. The regulatory landscape is complex, encompassing stringent certification processes under the EU Medical Device Regulation (MDR) and the upcoming EU AI Act, which imposes additional compliance requirements aimed at mitigating AI-specific risks. Ethical considerations such as, emphasizing transparency, patient privacy, and equitable access to AI technologies, are paramount. The manuscript underscores the importance of interdisciplinary collaboration, between healthcare institutions and industry partners, and navigation of commercialization and market entry of AI devices. This overview provides a strategic framework for radiologists and healthcare leaders to effectively integrate AI into clinical practice, while adhering to regulatory and ethical standards, ultimately enhancing patient care and operational efficiency.

Neoadjuvant Chemotherapy VI-RADS Scores for Assessing Muscle-invasive Bladder Cancer Response to Neoadjuvant Immunotherapy with Multiparametric MRI

Background An accurate method of assessing the response of muscle-invasive bladder cancer (MIBC) to neoadjuvant treatment is needed for selecting candidates for bladder-sparing strategies. Purpose To evaluate the diagnostic accuracy and reproducibility of neoadjuvant chemotherapy Vesical Imaging Reporting and Data System (nacVI-RADS) scores and posttreatment Vesical Imaging Reporting and Data System (VI-RADS) scores when assessing MIBC response to neoadjuvant immunotherapy with multiparametric MRI (mpMRI). Materials and Methods A retrospective analysis of MRI scans was conducted in patients enrolled in the PURE-01 study (NCT02736266) from February 2017 to December 2019 who underwent pre- and postimmunotherapy mpMRI before radical cystectomy. Five readers independently reviewed the scans using VI-RADS and nacVI-RADS criteria. Diagnostic accuracy was evaluated for each reader, and the final histopathologic diagnosis served as the reference standard. Interreader agreement was assessed with the percentage of agreement, Conger κ, and Gwet agreement coefficient AC1. Results A total of 110 patients (median age, 67 years [IQR: 61-74]; 96 male) with 220 MRI scans were included; 80 (73%) patients had pure urothelial carcinoma. A total of 46 of 110 (42%) patients achieved a complete pathologic response. The sensitivity, specificity, and negative predictive value of nacVI-RADS 3 or higher for detecting residual disease (higher than stage ypT0) at radical cystectomy were 67%-84%, 63%-96%, and 63%-75%, respectively; for residual muscle-invasive disease (higher than stage ypT1), these values were 91%-98%, 55%-94%, and 93%-98%, respectively. The accuracy of nacVI-RADS was 72%-81% for stage ypT0 or higher disease and 71%-95% for stage ypT1 or higher disease. The accuracy of VI-RADS 3 or higher was 80%-95% for stage ypT1 or higher disease. The percentage of agreement for nacVI-RADS scores was 82% (κ = 0.62-0.65; AC1 = 0.65). Conclusion The nacVI-RADS scores showed good accuracy and reproducibility when assessing MIBC response to neoadjuvant immunotherapy. © RSNA, 2024 Supplemental material is available for this article.

Unveiling variability: A systematic review of reproducibility in visual EEG analysis, with focus on seizures

OBJECTIVE

Reproducibility is key for diagnostic tests involving subjective evaluation by experts. Our aim was to systematically review the reproducibility of visual analysis in clinical electroencephalogram (EEG). In this paper, we give data on the scope of EEG features found, and detailed reproducibility data for the most studied feature.

METHODS

We searched four databases for articles reporting reproducibility in clinical EEG, until June 2023. Two raters screened 24 553 citations, and then 2736 full texts. Quality was assessed according to the GRRAS guidelines.

RESULTS

We found 275 studies (268 interrater and 20 intrarater), addressing 606 different EEG features. Only 38 EEG features had been studied in >2 studies. Most studies had <50 patients and EEGs. The most often addressed feature was seizure detection (62 papers). Interrater reproducibility of seizure detection was substantial-to-almost-perfect with experienced raters and raw EEG (kappa .62-.88). With experienced raters and transformed EEG, reproducibility was substantial (kappa .63-.70). Inexperienced raters had lower reproducibility. Seizure lateralization reproducibility was moderate to substantial (kappa .58-.77) but lower than for seizure detection.

SIGNIFICANCE

Most EEG reproducibility studies are done only once. Intrarater studies are rare. The reproducibility of visual EEG analysis is variable. Interrater reproducibility for seizure detection is substantial-to-perfect with experienced raters and raw EEG, less with inexperienced raters or transformed EEG. The results of visual EEG analysis vary within the same rater, and between raters. There is a need for larger collaborative studies, using improved methodology, as well as more intrarater studies of EEG interpretation.

Assessing the impact of deep-learning assistance on the histopathological diagnosis of serous tubal intraepithelial carcinoma (STIC) in fallopian tubes

In recent years, it has become clear that artificial intelligence (AI) models can achieve high accuracy in specific pathology-related tasks. An example is our deep-learning model, designed to automatically detect serous tubal intraepithelial carcinoma (STIC), the precursor lesion to high-grade serous ovarian carcinoma, found in the fallopian tube. However, the standalone performance of a model is insufficient to determine its value in the diagnostic setting. To evaluate the impact of the use of this model on pathologists' performance, we set up a fully crossed multireader, multicase study, in which 26 participants, from 11 countries, reviewed 100 digitalized H&E-stained slides of fallopian tubes (30 cases/70 controls) with and without AI assistance, with a washout period between the sessions. We evaluated the effect of the deep-learning model on accuracy, slide review time and (subjectively perceived) diagnostic certainty, using mixed-models analysis. With AI assistance, we found a significant increase in accuracy (p < 0.01) whereby the average sensitivity increased from 82% to 93%. Further, there was a significant 44 s (32%) reduction in slide review time (p < 0.01). The level of certainty that the participants felt versus their own assessment also significantly increased, by 0.24 on a 10-point scale (p < 0.01). In conclusion, we found that, in a diverse group of pathologists and pathology residents, AI support resulted in a significant improvement in the accuracy of STIC diagnosis and was coupled with a substantial reduction in slide review time. This model has the potential to provide meaningful support to pathologists in the diagnosis of STIC, ultimately streamlining and optimizing the overall diagnostic process.

G4 & the balanced metric family - a novel approach to solving binary classification problems in medical device validation & verification studies

BACKGROUND

In medical device validation and verification studies, the area under the receiver operating characteristic curve (AUROC) is often used as a primary endpoint despite multiple reports showing its limitations. Hence, researchers are encouraged to consider alternative metrics as primary endpoints. A new metric called G4 is presented, which is the geometric mean of sensitivity, specificity, the positive predictive value, and the negative predictive value. G4 is part of a balanced metric family which includes the Unified Performance Measure (also known as P4) and the Matthews' Correlation Coefficient (MCC). The purpose of this manuscript is to unveil the benefits of using G4 together with the balanced metric family when analyzing the overall performance of binary classifiers.

RESULTS

Simulated datasets encompassing different prevalence rates of the minority class were analyzed under a multi-reader-multi-case study design. In addition, data from an independently published study that tested the performance of a unique ultrasound artificial intelligence algorithm in the context of breast cancer detection was also considered. Within each dataset, AUROC was reported alongside the balanced metric family for comparison. When the dataset prevalence and bias of the minority class approached 50%, all three balanced metrics provided equivalent interpretations of an AI's performance. As the prevalence rate increased / decreased and the data became more imbalanced, AUROC tended to overvalue / undervalue the true classifier performance, while the balanced metric family was resistant to such imbalance. Under certain circumstances where data imbalance was strong (minority-class prevalence < 10%), MCC was preferred for standalone assessments while P4 provided a stronger effect size when evaluating between-groups analyses. G4 acted as a middle ground for maximizing both standalone assessments and between-groups analyses.

CONCLUSIONS

Use of AUROC as the primary endpoint in binary classification problems provides misleading results as the dataset becomes more imbalanced. This is explicitly noticed when incorporating AUROC in medical device validation and verification studies. G4, P4, and MCC do not share this limitation and paint a more complete picture of a medical device's performance in a clinical setting. Therefore, researchers are encouraged to explore the balanced metric family when evaluating binary classification problems.

Impact of Patient-reported Symptom Information on the Interpretation of MRI of the Lumbar Spine

Background Distinguishing lumbar pain generators from incidental findings at MRI can be difficult. Dictated reports may become lists of findings that cannot be ranked in order of diagnostic importance. Purpose To determine whether patient-reported symptom information can improve the interpretation of lumbar spine MRI by using the spine specialist as the reference standard. Materials and Methods This prospective, single-center, multireader study analyzed 240 participants who completed pre-MRI symptom questionnaires between May 2022 and February 2023. At the time of clinical MRI reporting, radiologists recorded pain generators in consecutive participants, creating two study groups by alternating interpretations with versus without symptom questionnaire results (SQR). Diagnostic certainty was recorded using a numeric scale of 0 to 100. Types, levels, and sides of pain generators were compared with reference diagnoses by calculating Cohen κ values with 95% CIs. Participant characteristics and diagnostic certainties were compared using the Wilcoxon rank sum, Pearson χ2, or Kruskal-Wallis test. Interrater agreement was analyzed. Results There was no difference in age (P = .69) or sex (P = .60) between participants using SQR (n = 120; mean age, 61.0 years; 62 female) and not using SQR (n = 120; mean age, 62.5 years; 67 female). When radiologists were compared with specialists, agreements on pain generators were almost perfect for interpretations using SQR (type: κ = 0.82 [95% CI: 0.74,0.89]; level: κ = 0.88 [95% CI: 0.80, 0.95]; side: κ = 0.84 [95% CI: 0.75, 0.92]), but only fair to moderate for interpretations not using SQR (type: κ = 0.26 [95% CI: 0.15, 0.36]; level: κ = 0.51 [95% CI: 0.39, 0.63]; side: κ = 0.30 [95% CI: 0.18, 0.42]) (all P < .001). Diagnostic certainty was higher for MRI interpretations using SQR (mean, 80.4 ± 14.9 [SD]) than MRI interpretations not using SQR (60.5 ± 17.7) (P < .001). Interrater agreements were substantial (κ = 0.65-0.78) for MRI interpretations using SQR but only fair to moderate (κ = 0.24-0.49) for MRI interpretations not using SQR (all P < .001). Conclusion Patient-reported symptom information enabled radiologists to achieve nearly perfect diagnostic agreement with clinical experts. © RSNA, 2024 See also the editorial by Isikbay and Shah in this issue.

A web-based platform for studying the impact of artificial intelligence in video capsule endoscopy

Objective: Integrating artificial intelligence (AI) solutions into clinical practice, particularly in the field of video capsule endoscopy (VCE), necessitates the execution of rigorous clinical studies. Methods: This work introduces a novel software platform tailored to facilitate the conduct of multi-reader multi-case clinical studies in VCE. The platform, developed as a web application, prioritizes remote accessibility to accommodate multi-center studies. Notably, considerable attention was devoted to user interface and user experience design elements to ensure a seamless and engaging interface. To evaluate the usability of the platform, a pilot study is conducted. Results: The results indicate a high level of usability and acceptance among users, providing valuable insights into the expectations and preferences of gastroenterologists navigating AI-driven VCE solutions. Conclusion: This research lays a foundation for future advancements in AI integration within clinical VCE practice.

Implementing multiple imputations for addressing missing data in multireader multicase design studies

BACKGROUND

In computer-aided diagnosis (CAD) studies utilizing multireader multicase (MRMC) designs, missing data might occur when there are instances of misinterpretation or oversight by the reader or problems with measurement techniques. Improper handling of these missing data can lead to bias. However, little research has been conducted on addressing the missing data issue within the MRMC framework.

METHODS

We introduced a novel approach that integrates multiple imputation with MRMC analysis (MI-MRMC). An elaborate simulation study was conducted to compare the efficacy of our proposed approach with that of the traditional complete case analysis strategy within the MRMC design. Furthermore, we applied these approaches to a real MRMC design CAD study on aneurysm detection via head and neck CT angiograms to further validate their practicality.

RESULTS

Compared with traditional complete case analysis, the simulation study demonstrated the MI-MRMC approach provides an almost unbiased estimate of diagnostic capability, alongside satisfactory performance in terms of statistical power and the type I error rate within the MRMC framework, even in small sample scenarios. In the real CAD study, the proposed MI-MRMC method further demonstrated strong performance in terms of both point estimates and confidence intervals compared with traditional complete case analysis.

CONCLUSION

Within MRMC design settings, the adoption of an MI-MRMC approach in the face of missing data can facilitate the attainment of unbiased and robust estimates of diagnostic capability.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools. KEY POINTS.

Accuracy of Artificial Intelligence for Cervical Vertebral Maturation Assessment-A Systematic Review

Background/Objectives: To systematically review and summarize the existing scientific evidence on the diagnostic performance of artificial intelligence (AI) in assessing cervical vertebral maturation (CVM). This review aimed to evaluate the accuracy and reliability of AI algorithms in comparison to those of experienced clinicians. Methods: Comprehensive searches were conducted across multiple databases, including PubMed, Scopus, Web of Science, and Embase, using a combination of Boolean operators and MeSH terms. The inclusion criteria were cross-sectional studies with neural network research, reporting diagnostic accuracy, and involving human subjects. Data extraction and quality assessment were performed independently by two reviewers, with a third reviewer resolving any disagreements. The Quality Assessment of Diagnostic Accuracy Studies (QUADAS)-2 tool was used for bias assessment. Results: Eighteen studies met the inclusion criteria, predominantly employing supervised learning techniques, especially convolutional neural networks (CNNs). The diagnostic accuracy of AI models for CVM assessment varied widely, ranging from 57% to 95%. The factors influencing accuracy included the type of AI model, training data, and study methods. Geographic concentration and variability in the experience of radiograph readers also impacted the results. Conclusions: AI has considerable potential for enhancing the accuracy and reliability of CVM assessments in orthodontics. However, the variability in AI performance and the limited number of high-quality studies suggest the need for further research.

Interobserver Agreement and Performance of Concurrent AI Assistance for Radiographic Evaluation of Knee Osteoarthritis

Background Due to conflicting findings in the literature, there are concerns about a lack of objectivity in grading knee osteoarthritis (KOA) on radiographs. Purpose To examine how artificial intelligence (AI) assistance affects the performance and interobserver agreement of radiologists and orthopedists of various experience levels when evaluating KOA on radiographs according to the established Kellgren-Lawrence (KL) grading system. Materials and Methods In this retrospective observer performance study, consecutive standing knee radiographs from patients with suspected KOA were collected from three participating European centers between April 2019 and May 2022. Each center recruited four readers across radiology and orthopedic surgery at in-training and board-certified experience levels. KL grading (KL-0 = no KOA, KL-4 = severe KOA) on the frontal view was assessed by readers with and without assistance from a commercial AI tool. The majority vote of three musculoskeletal radiology consultants established the reference standard. The ordinal receiver operating characteristic method was used to estimate grading performance. Light kappa was used to estimate interrater agreement, and bootstrapped t statistics were used to compare groups. Results Seventy-five studies were included from each center, totaling 225 studies (mean patient age, 55 years ± 15 [SD]; 113 female patients). The KL grades were KL-0, 24.0% (n = 54); KL-1, 28.0% (n = 63); KL-2, 21.8% (n = 49); KL-3, 18.7% (n = 42); and KL-4, 7.6% (n = 17). Eleven readers completed their readings. Three of the six junior readers showed higher KL grading performance with versus without AI assistance (area under the receiver operating characteristic curve, 0.81 ± 0.017 [SEM] vs 0.88 ± 0.011 [P < .001]; 0.76 ± 0.018 vs 0.86 ± 0.013 [P < .001]; and 0.89 ± 0.011 vs 0.91 ± 0.009 [P = .008]). Interobserver agreement for KL grading among all readers was higher with versus without AI assistance (κ = 0.77 ± 0.018 [SEM] vs 0.85 ± 0.013; P < .001). Board-certified radiologists achieved almost perfect agreement for KL grading when assisted by AI (κ = 0.90 ± 0.01), which was higher than that achieved by the reference readers independently (κ = 0.84 ± 0.017; P = .01). Conclusion AI assistance increased junior readers' radiographic KOA grading performance and increased interobserver agreement for osteoarthritis grading across all readers and experience levels. Published under a CC BY 4.0 license. Supplemental material is available for this article.

Utility of abbreviated MRI in the post-treatment evaluation of rectal cancer

BACKGROUND

Post-treatment evaluation of patients with rectal cancer (RC) using magnetic resonance imaging (MRI) burdens medical resources, necessitating an exploration of abbreviated protocols.

PURPOSE

To evaluate the diagnostic performance of abbreviated MRI (A-MRI) for the post-treatment evaluation of RC patients.

MATERIAL AND METHODS

This retrospective study included RC patients who underwent non-contrast rectal MRI and standard liver MRI, as well as abdominal contrast-enhanced computed tomography (CECT) for post-treatment evaluation. A-MRI comprised diffusion-weighted imaging (DWI) and T2-weighted imaging of the upper abdomen and the pelvic cavity. Three radiologists independently reviewed A-MRI, CECT, and standard liver MRI in the detection of viable disease. The diagnostic performances were compared using a reference standard considering all available information, including pathology, FDG-PET, endoscopic results, and clinical follow-up.

RESULTS

We included 78 patients (50 men, 28 women; mean age=60.9 ± 10.2 years) and observed viable disease in 34 (43.6%). On a per-patient-basis analysis, A-MRI showed significantly higher sensitivity (95% vs. 81%, P = 0.04) and higher accuracy (93% vs. 82%, P < 0.01), compared to those of CECT, while A-MRI showed comparable sensitivity (91% vs. 91%, P = 0.42) and accuracy (97% vs. 98%, P = 0.06) to that of standard liver MRI. On a per-lesion-based analysis, A-MRI exhibited significantly superior lesion detectability than that of CECT (figure of merit 0.91 vs. 0.77, P < 0.01) and comparable to that of standard liver MRI (figure of merit 0.91 vs. 0.92, P = 0.75).

CONCLUSION

A-MRI exhibited higher sensitivity and diagnostic accuracy than those of CECT in the post-treatment evaluation of RC, while it showed comparable performances with standard liver MRI. A-MRI provides diagnostic added value in the follow-up of RC patients.

Best Practice for MRI Diagnostic Accuracy Research With Lessons and Examples from the LI-RADS Individual Participant Data Group

Medical imaging diagnostic test accuracy research is strengthened by adhering to best practices for study design, data collection, data documentation, and study reporting. In this review, key elements of such research are discussed, and specific recommendations provided for optimizing diagnostic accuracy study execution to improve uniformity, minimize common sources of bias and avoid potential pitfalls. Examples are provided regarding study methodology and data collection practices based on insights gained by the liver imaging reporting and data system (LI-RADS) individual participant data group, who have evaluated raw data from numerous MRI diagnostic accuracy studies for risk of bias and data integrity. The goal of this review is to outline strategies for investigators to improve research practices, and to help reviewers and readers better contextualize a study's findings while understanding its limitations. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY: Stage 3.

Nonaneurysmal perimesencephalic subarachnoid hemorrhage on noncontrast head CT: An accuracy, inter-rater, and intra-rater reliability study

BACKGROUND AND PURPOSE

To evaluate the reliability and accuracy of nonaneurysmal perimesencephalic subarachnoid hemorrhage (NAPSAH) on Noncontrast Head CT (NCCT) between numerous raters.

MATERIALS AND METHODS

45 NCCT of adult patients with SAH who also had a catheter angiography (CA) were independently evaluated by 48 diverse raters; 45 raters performed a second assessment one month later. For each case, raters were asked: 1) whether they judged the bleeding pattern to be perimesencephalic; 2) whether there was blood anterior to brainstem; 3) complete filling of the anterior interhemispheric fissure (AIF); 4) extension to the lateral part of the sylvian fissure (LSF); 5) frank intraventricular hemorrhage; 6) whether in the hypothetical presence of a negative CT angiogram they would still recommend CA. An automatic NAPSAH diagnosis was also generated by combining responses to questions 2-5. Reliability was estimated using Gwet's AC1 (κG), and the relationship between the NCCT diagnosis of NAPSAH and the recommendation to perform CA using Cramer's V test. Multi-rater accuracy of NCCT in predicting negative CA was explored.

RESULTS

Inter-rater reliability for the presence of NAPSAH was moderate (κG = 0.58; 95%CI: 0.47, 0.69), but improved to substantial when automatically generated (κG = 0.70; 95%CI: 0.59, 0.81). The most reliable criteria were the absence of AIF filling (κG = 0.79) and extension to LSF (κG = 0.79). Mean intra-rater reliability was substantial (κG = 0.65). NAPSAH weakly correlated with CA decision (V = 0.50). Mean sensitivity and specificity were 58% (95%CI: 44%, 71%) and 83 % (95%CI: 72 %, 94%), respectively.

CONCLUSION

NAPSAH remains a diagnosis of exclusion. The NCCT diagnosis was moderately reliable and its impact on clinical decisions modest.

Clinical Super-Resolution Computed Tomography of Bone Microstructure: Application in Musculoskeletal and Dental Imaging

PURPOSE

Clinical cone-beam computed tomography (CBCT) devices are limited to imaging features of half a millimeter in size and cannot quantify the tissue microstructure. We demonstrate a robust deep-learning method for enhancing clinical CT images, only requiring a limited set of easy-to-acquire training data.

METHODS

Knee tissue from five cadavers and six total knee replacement patients, and 14 teeth from eight patients were scanned using laboratory CT as training data for the developed super-resolution (SR) technique. The method was benchmarked against ex vivo test set, 52 osteochondral samples are imaged with clinical and laboratory CT. A quality assurance phantom was imaged with clinical CT to quantify the technical image quality. To visually assess the clinical image quality, musculoskeletal and maxillofacial CBCT studies were enhanced with SR and contrasted to interpolated images. A dental radiologist and surgeon reviewed the maxillofacial images.

RESULTS

The SR models predicted the bone morphological parameters on the ex vivo test set more accurately than conventional image processing. The phantom analysis confirmed higher spatial resolution on the SR images than interpolation, but image grayscales were modified. Musculoskeletal and maxillofacial CBCT images showed more details on SR than interpolation; however, artifacts were observed near the crown of the teeth. The readers assessed mediocre overall scores for both SR and interpolation. The source code and pretrained networks are publicly available.

CONCLUSION

Model training with laboratory modalities could push the resolution limit beyond state-of-the-art clinical musculoskeletal and dental CBCT. A larger maxillofacial training dataset is recommended for dental applications.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement From the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever‑growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi‑society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

A deep-learning model for intracranial aneurysm detection on CT angiography images in China: a stepwise, multicentre, early-stage clinical validation study

BACKGROUND

Artificial intelligence (AI) models in real-world implementation are scarce. Our study aimed to develop a CT angiography (CTA)-based AI model for intracranial aneurysm detection, assess how it helps clinicians improve diagnostic performance, and validate its application in real-world clinical implementation.

METHODS

We developed a deep-learning model using 16 546 head and neck CTA examination images from 14 517 patients at eight Chinese hospitals. Using an adapted, stepwise implementation and evaluation, 120 certified clinicians from 15 geographically different hospitals were recruited. Initially, the AI model was externally validated with images of 900 digital subtraction angiography-verified CTA cases (examinations) and compared with the performance of 24 clinicians who each viewed 300 of these cases (stage 1). Next, as a further external validation a multi-reader multi-case study enrolled 48 clinicians to individually review 298 digital subtraction angiography-verified CTA cases (stage 2). The clinicians reviewed each CTA examination twice (ie, with and without the AI model), separated by a 4-week washout period. Then, a randomised open-label comparison study enrolled 48 clinicians to assess the acceptance and performance of this AI model (stage 3). Finally, the model was prospectively deployed and validated in 1562 real-world clinical CTA cases.

FINDINGS

The AI model in the internal dataset achieved a patient-level diagnostic sensitivity of 0·957 (95% CI 0·939-0·971) and a higher patient-level diagnostic sensitivity than clinicians (0·943 [0·921-0·961] vs 0·658 [0·644-0·672]; p<0·0001) in the external dataset. In the multi-reader multi-case study, the AI-assisted strategy improved clinicians' diagnostic performance both on a per-patient basis (the area under the receiver operating characteristic curves [AUCs]; 0·795 [0·761-0·830] without AI vs 0·878 [0·850-0·906] with AI; p<0·0001) and a per-aneurysm basis (the area under the weighted alternative free-response receiver operating characteristic curves; 0·765 [0·732-0·799] vs 0·865 [0·839-0·891]; p<0·0001). Reading time decreased with the aid of the AI model (87·5 s vs 82·7 s, p<0·0001). In the randomised open-label comparison study, clinicians in the AI-assisted group had a high acceptance of the AI model (92·6% adoption rate), and a higher AUC when compared with the control group (0·858 [95% CI 0·850-0·866] vs 0·789 [0·780-0·799]; p<0·0001). In the prospective study, the AI model had a 0·51% (8/1570) error rate due to poor-quality CTA images and recognition failure. The model had a high negative predictive value of 0·998 (0·994-1·000) and significantly improved the diagnostic performance of clinicians; AUC improved from 0·787 (95% CI 0·766-0·808) to 0·909 (0·894-0·923; p<0·0001) and patient-level sensitivity improved from 0·590 (0·511-0·666) to 0·825 (0·759-0·880; p<0·0001).

INTERPRETATION

This AI model demonstrated strong clinical potential for intracranial aneurysm detection with improved clinician diagnostic performance, high acceptance, and practical implementation in real-world clinical cases.

FUNDING

National Natural Science Foundation of China.

TRANSLATION

For the Chinese translation of the abstract see Supplementary Materials section.

Deep learning for differentiation of osteolytic osteosarcoma and giant cell tumor around the knee joint on radiographs: a multicenter study

OBJECTIVES

To develop a deep learning (DL) model for differentiating between osteolytic osteosarcoma (OS) and giant cell tumor (GCT) on radiographs.

METHODS

Patients with osteolytic OS and GCT proven by postoperative pathology were retrospectively recruited from four centers (center A, training and internal testing; centers B, C, and D, external testing). Sixteen radiologists with different experiences in musculoskeletal imaging diagnosis were divided into three groups and participated with or without the DL model's assistance. DL model was generated using EfficientNet-B6 architecture, and the clinical model was trained using clinical variables. The performance of various models was compared using McNemar's test.

RESULTS

Three hundred thirty-three patients were included (mean age, 27 years ± 12 [SD]; 186 men). Compared to the clinical model, the DL model achieved a higher area under the curve (AUC) in both the internal (0.97 vs. 0.77, p = 0.008) and external test set (0.97 vs. 0.64, p < 0.001). In the total test set (including the internal and external test sets), the DL model achieved higher accuracy than the junior expert committee (93.1% vs. 72.4%; p < 0.001) and was comparable to the intermediate and senior expert committee (93.1% vs. 88.8%, p = 0.25; 87.1%, p = 0.35). With DL model assistance, the accuracy of the junior expert committee was improved from 72.4% to 91.4% (p = 0.051).

CONCLUSION

The DL model accurately distinguished osteolytic OS and GCT with better performance than the junior radiologists, whose own diagnostic performances were significantly improved with the aid of the model, indicating the potential for the differential diagnosis of the two bone tumors on radiographs.

CRITICAL RELEVANCE STATEMENT

The deep learning model can accurately distinguish osteolytic osteosarcoma and giant cell tumor on radiographs, which may help radiologists improve the diagnostic accuracy of two types of tumors.

KEY POINTS

• The DL model shows robust performance in distinguishing osteolytic osteosarcoma and giant cell tumor. • The diagnosis performance of the DL model is better than junior radiologists'. • The DL model shows potential for differentiating osteolytic osteosarcoma and giant cell tumor.

Developing, purchasing, implementing and monitoring AI tools in radiology: Practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.

Elective one-minute full brain multi-contrast MRI versus brain CT in pediatric patients: a prospective feasibility study

BACKGROUND

Brain CT can be used to evaluate pediatric patients with suspicion of cerebral pathology when anesthetic and MRI resources are scarce. This study aimed to assess if pediatric patients referred for an elective brain CT could endure a diagnostic fast brain MRI without general anesthesia using a one-minute multi-contrast EPI-based sequence (EPIMix) with comparable diagnostic performance.

METHODS

Pediatric patients referred for an elective brain CT between March 2019 and March 2020 were prospectively included and underwent EPIMix without general anesthesia in addition to CT. Three readers (R1-3) independently evaluated EPIMix and CT images on two separate occasions. The two main study outcomes were the tolerance to undergo an EPIMix scan without general anesthesia and its performance to classify a scan as normal or abnormal. Secondary outcomes were assessment of disease category, incidental findings, diagnostic image quality, diagnostic confidence, and image artifacts. Further, a side-by-side evaluation of EPIMix and CT was performed. The signal-to-noise ratio (SNR) was calculated for EPIMix on T1-weighted, T2-weighted, and ADC images. Descriptive statistics, Fisher's exact test, and Chi-squared test were used to compare the two imaging modalities.

RESULTS

EPIMix was well tolerated by all included patients (n = 15) aged 5-16 (mean 11, SD 3) years old. Thirteen cases on EPIMix and twelve cases on CT were classified as normal by all readers (R1-3), while two cases on EPIMix and three cases on CT were classified as abnormal by one reader (R1), (R1-3, p = 1.00). There was no evidence of a difference in diagnostic confidence, image quality, or the presence of motion artifacts between EPIMix and CT (R1-3, p ≥ 0.10). Side-by-side evaluation (R2 + R4 + R5) reviewed all scans as lacking significant pathological findings on EPIMix and CT images.

CONCLUSIONS

Full brain MRI-based EPIMix sequence was well tolerated without general anesthesia with a diagnostic performance comparable to CT in elective pediatric patients.

TRIAL REGISTRATION

This study was approved by the Swedish Ethical Review Authority (ethical approval number/ID Ethical approval 2017/2424-31/1). This study was a clinical trial study, with study protocol published at ClinicalTrials.gov with Trial registration number NCT03847051, date of registration 18/02/2019.

Developing, purchasing, implementing and monitoring AI tools in radiology: practical considerations. A multi-society statement from the ACR, CAR, ESR, RANZCR & RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones.This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools.Key points • The incorporation of artificial intelligence (AI) in radiological practice demands increased monitoring of its utility and safety.• Cooperation between developers, clinicians, and regulators will allow all involved to address ethical issues and monitor AI performance.• AI can fulfil its promise to advance patient well-being if all steps from development to integration in healthcare are rigorously evaluated.

Comparative effectiveness of standard vs. AI-assisted PET/CT reading workflow for pre-treatment lymphoma staging: a multi-institutional reader study evaluation

Background

Fluorine-18 fluorodeoxyglucose (FDG)-positron emission tomography/computed tomography (PET/CT) is widely used for staging high-grade lymphoma, with the time to evaluate such studies varying depending on the complexity of the case. Integrating artificial intelligence (AI) within the reporting workflow has the potential to improve quality and efficiency. The aims of the present study were to evaluate the influence of an integrated research prototype segmentation tool implemented within diagnostic PET/CT reading software on the speed and quality of reporting with variable levels of experience, and to assess the effect of the AI-assisted workflow on reader confidence and whether this tool influenced reporting behaviour.

Methods

Nine blinded reporters (three trainees, three junior consultants and three senior consultants) from three UK centres participated in a two-part reader study. A total of 15 lymphoma staging PET/CT scans were evaluated twice: first, using a standard PET/CT reporting workflow; then, after a 6-week gap, with AI assistance incorporating pre-segmentation of disease sites within the reading software. An even split of PET/CT segmentations with gold standard (GS), false-positive (FP) over-contour or false-negative (FN) under-contour were provided. The read duration was calculated using file logs, while the report quality was independently assessed by two radiologists with >15 years of experience. Confidence in AI assistance and identification of disease was assessed via online questionnaires for each case.

Results

There was a significant decrease in time between non-AI and AI-assisted reads (median 15.0 vs. 13.3 min, p < 0.001). Sub-analysis confirmed this was true for both junior (14.5 vs. 12.7 min, p = 0.03) and senior consultants (15.1 vs. 12.2 min, p = 0.03) but not for trainees (18.1 vs. 18.0 min, p = 0.2). There was no significant difference between report quality between reads. AI assistance provided a significant increase in confidence of disease identification (p < 0.001). This held true when splitting the data into FN, GS and FP. In 19/88 cases, participants did not identify either FP (31.8%) or FN (11.4%) segmentations. This was significantly greater for trainees (13/30, 43.3%) than for junior (3/28, 10.7%, p = 0.05) and senior consultants (3/30, 10.0%, p = 0.05).

Conclusions

The study findings indicate that an AI-assisted workflow achieves comparable performance to humans, demonstrating a marginal enhancement in reporting speed. Less experienced readers were more influenced by segmentation errors. An AI-assisted PET/CT reading workflow has the potential to increase reporting efficiency without adversely affecting quality, which could reduce costs and report turnaround times. These preliminary findings need to be confirmed in larger studies.

Developing, Purchasing, Implementing and Monitoring AI Tools in Radiology: Practical Considerations. A Multi-Society Statement from the ACR, CAR, ESR, RANZCR and RSNA

Artificial Intelligence (AI) carries the potential for unprecedented disruption in radiology, with possible positive and negative consequences. The integration of AI in radiology holds the potential to revolutionize healthcare practices by advancing diagnosis, quantification, and management of multiple medical conditions. Nevertheless, the ever-growing availability of AI tools in radiology highlights an increasing need to critically evaluate claims for its utility and to differentiate safe product offerings from potentially harmful, or fundamentally unhelpful ones. This multi-society paper, presenting the views of Radiology Societies in the USA, Canada, Europe, Australia, and New Zealand, defines the potential practical problems and ethical issues surrounding the incorporation of AI into radiological practice. In addition to delineating the main points of concern that developers, regulators, and purchasers of AI tools should consider prior to their introduction into clinical practice, this statement also suggests methods to monitor their stability and safety in clinical use, and their suitability for possible autonomous function. This statement is intended to serve as a useful summary of the practical issues which should be considered by all parties involved in the development of radiology AI resources, and their implementation as clinical tools. This article is simultaneously published in Insights into Imaging (DOI 10.1186/s13244-023-01541-3), Journal of Medical Imaging and Radiation Oncology (DOI 10.1111/1754-9485.13612), Canadian Association of Radiologists Journal (DOI 10.1177/08465371231222229), Journal of the American College of Radiology (DOI 10.1016/j.jacr.2023.12.005), and Radiology: Artificial Intelligence (DOI 10.1148/ryai.230513). Keywords: Artificial Intelligence, Radiology, Automation, Machine Learning Published under a CC BY 4.0 license. ©The Author(s) 2024. Editor's Note: The RSNA Board of Directors has endorsed this article. It has not undergone review or editing by this journal.

Improving traumatic fracture detection on radiographs with artificial intelligence support: a multi-reader study

Objectives

The aim of this study was to evaluate the diagnostic performance of nonspecialist readers with and without the use of an artificial intelligence (AI) support tool to detect traumatic fractures on radiographs of the appendicular skeleton.

Methods

The design was a retrospective, fully crossed multi-reader, multi-case study on a balanced dataset of patients (≥2 years of age) with an AI tool as a diagnostic intervention. Fifteen readers assessed 340 radiographic exams, with and without the AI tool in 2 different sessions and the time spent was automatically recorded. Reference standard was established by 3 consultant radiologists. Sensitivity, specificity, and false positives per patient were calculated.

Results

Patient-wise sensitivity increased from 72% to 80% (P < .05) and patient-wise specificity increased from 81% to 85% (P < .05) in exams aided by the AI tool compared to the unaided exams. The increase in sensitivity resulted in a relative reduction of missed fractures of 29%. The average rate of false positives per patient decreased from 0.16 to 0.14, corresponding to a relative reduction of 21%. There was no significant difference in average reading time spent per exam. The largest gain in fracture detection performance, with AI support, across all readers, was on nonobvious fractures with a significant increase in sensitivity of 11 percentage points (pp) (60%-71%).

Conclusions

The diagnostic performance for detection of traumatic fractures on radiographs of the appendicular skeleton improved among nonspecialist readers tested AI fracture detection support tool showed an overall reader improvement in sensitivity and specificity when supported by an AI tool. Improvement was seen in both sensitivity and specificity without negatively affecting the interpretation time.

Advances in knowledge

The division and analysis of obvious and nonobvious fractures are novel in AI reader comparison studies like this.

Evaluation of the Statistical Detection of Change Algorithm for Screening Patients with MS with New Lesion Activity on Longitudinal Brain MRI

BACKGROUND AND PURPOSE

Identification of new MS lesions on longitudinal MR imaging by human readers is time-consuming and prone to error. Our objective was to evaluate the improvement in the performance of subject-level detection by readers when assisted by the automated statistical detection of change algorithm.

MATERIALS AND METHODS

A total of 200 patients with MS with a mean interscan interval of 13.2 (SD, 2.4) months were included. Statistical detection of change was applied to the baseline and follow-up FLAIR images to detect potential new lesions for confirmation by readers (Reader + statistical detection of change method). This method was compared with readers operating in the clinical workflow (Reader method) for a subject-level detection of new lesions.

RESULTS

Reader + statistical detection of change found 30 subjects (15.0%) with at least 1 new lesion, while Reader detected 16 subjects (8.0%). As a subject-level screening tool, statistical detection of change achieved a perfect sensitivity of 1.00 (95% CI, 0.88-1.00) and a moderate specificity of 0.67 (95% CI, 0.59-0.74). The agreement on a subject level was 0.91 (95% CI, 0.87-0.95) between Reader + statistical detection of change and Reader, and 0.72 (95% CI, 0.66-0.78) between Reader + statistical detection of change and statistical detection of change.

CONCLUSIONS

The statistical detection of change algorithm can serve as a time-saving screening tool to assist human readers in verifying 3D FLAIR images of patients with MS with suspected new lesions. Our promising results warrant further evaluation of statistical detection of change in prospective multireader clinical studies.

Impact of Different Artificial Intelligence User Interfaces on Lung Nodule and Mass Detection on Chest Radiographs

Purpose

To explore the impact of different user interfaces (UIs) for artificial intelligence (AI) outputs on radiologist performance and user preference in detecting lung nodules and masses on chest radiographs.

Materials and Methods

A retrospective paired-reader study with a 4-week washout period was used to evaluate three different AI UIs compared with no AI output. Ten radiologists (eight radiology attending physicians and two trainees) evaluated 140 chest radiographs (81 with histologically confirmed nodules and 59 confirmed as normal with CT), with either no AI or one of three UI outputs: (a) text-only, (b) combined AI confidence score and text, or (c) combined text, AI confidence score, and image overlay. Areas under the receiver operating characteristic curve were calculated to compare radiologist diagnostic performance with each UI with their diagnostic performance without AI. Radiologists reported their UI preference.

Results

The area under the receiver operating characteristic curve improved when radiologists used the text-only output compared with no AI (0.87 vs 0.82; P < .001). There was no difference in performance for the combined text and AI confidence score output compared with no AI (0.77 vs 0.82; P = .46) and for the combined text, AI confidence score, and image overlay output compared with no AI (0.80 vs 0.82; P = .66). Eight of the 10 radiologists (80%) preferred the combined text, AI confidence score, and image overlay output over the other two interfaces.

Conclusion

Text-only UI output significantly improved radiologist performance compared with no AI in the detection of lung nodules and masses on chest radiographs, but user preference did not correspond with user performance.Keywords: Artificial Intelligence, Chest Radiograph, Conventional Radiography, Lung Nodule, Mass Detection© RSNA, 2023.

A concurrent, deep learning-based computer-aided detection system for prostate multiparametric MRI: a performance study involving experienced and less-experienced radiologists

OBJECTIVES

To evaluate the effect of a deep learning-based computer-aided diagnosis (DL-CAD) system on experienced and less-experienced radiologists in reading prostate mpMRI.

METHODS

In this retrospective, multi-reader multi-case study, a consecutive set of 184 patients examined between 01/2018 and 08/2019 were enrolled. Ground truth was combined targeted and 12-core systematic transrectal ultrasound-guided biopsy. Four radiologists, two experienced and two less-experienced, evaluated each case twice, once without (DL-CAD-) and once assisted by DL-CAD (DL-CAD+). ROC analysis, sensitivities, specificities, PPV and NPV were calculated to compare the diagnostic accuracy for the diagnosis of prostate cancer (PCa) between the two groups (DL-CAD- vs. DL-CAD+). Spearman's correlation coefficients were evaluated to assess the relationship between PI-RADS category and Gleason score (GS). Also, the median reading times were compared for the two reading groups.

RESULTS

In total, 172 patients were included in the final analysis. With DL-CAD assistance, the overall AUC of the less-experienced radiologists increased significantly from 0.66 to 0.80 (p = 0.001; cutoff ISUP GG ≥ 1) and from 0.68 to 0.80 (p = 0.002; cutoff ISUP GG ≥ 2). Experienced radiologists showed an AUC increase from 0.81 to 0.86 (p = 0.146; cutoff ISUP GG ≥ 1) and from 0.81 to 0.84 (p = 0.433; cutoff ISUP GG ≥ 2). Furthermore, the correlation between PI-RADS category and GS improved significantly in the DL-CAD + group (0.45 vs. 0.57; p = 0.03), while the median reading time was reduced from 157 to 150 s (p = 0.023).

CONCLUSIONS

DL-CAD assistance increased the mean detection performance, with the most significant benefit for the less-experienced radiologist; with the help of DL-CAD less-experienced radiologists reached performances comparable to that of experienced radiologists.

KEY POINTS

• DL-CAD used as a concurrent reading aid helps radiologists to distinguish between benign and cancerous lesions in prostate MRI. • With the help of DL-CAD, less-experienced radiologists may achieve detection performances comparable to that of experienced radiologists. • DL-CAD assistance increases the correlation between PI-RADS category and cancer grade.