AI in medical imaging grand challenges: translation from competition to research benefit and patient care

Overview of the Head and Neck Tumor Segmentation for Magnetic Resonance Guided Applications (HNTS-MRG) 2024 Challenge

Magnetic resonance (MR)-guided radiation therapy (RT) is enhancing head and neck cancer (HNC) treatment through superior soft tissue contrast and longitudinal imaging capabilities. However, manual tumor segmentation remains a significant challenge, spurring interest in artificial intelligence (AI)-driven automation. To accelerate innovation in this field, we present the Head and Neck Tumor Segmentation for MR-Guided Applications (HNTS-MRG) 2024 Challenge, a satellite event of the 27th International Conference on Medical Image Computing and Computer Assisted Intervention. This challenge addresses the scarcity of large, publicly available AI-ready adaptive RT datasets in HNC and explores the potential of incorporating multi-timepoint data to enhance RT auto-segmentation performance. Participants tackled two HNC segmentation tasks: automatic delineation of primary gross tumor volume (GTVp) and gross metastatic regional lymph nodes (GTVn) on pre-RT (Task 1) and mid-RT (Task 2) T2-weighted scans. The challenge provided 150 HNC cases for training and 50 for final testing hosted on grand-challenge.org using a Docker submission framework. In total, 19 independent teams from across the world qualified by submitting both their algorithms and corresponding papers, resulting in 18 submissions for Task 1 and 15 submissions for Task 2. Evaluation using the mean aggregated Dice Similarity Coefficient showed top-performing AI methods achieved scores of 0.825 in Task 1 and 0.733 in Task 2. These results surpassed clinician interobserver variability benchmarks, marking significant strides in automated tumor segmentation for MR-guided RT applications in HNC.

MIDRC mRALE Mastermind Grand Challenge: AI to predict COVID severity on chest radiographs

Purpose

The Medical Imaging and Data Resource Center (MIDRC) mRALE Mastermind Grand Challenge fostered the development of artificial intelligence (AI) techniques for the automated assignment of mRALE (modified radiographic assessment of lung edema) scores to portable chest radiographs from patients known to have COVID-19.

Approach

The challenge utilized 2079 training cases obtained from the publicly available MIDRC data commons, with validation and test cases sampled from not-yet-public MIDRC cases that were inaccessible to challenge participants. The reference standard mRALE scores for the challenge cases were established by a pool of 22 radiologist annotators. Using the MedICI challenge platform, participants submitted their trained algorithms encapsulated in Docker containers. Algorithms were evaluated by the challenge organizers on 814 test cases through two performance assessment metrics: quadratic-weighted kappa and prediction probability concordance.

Results

Nine AI algorithms were submitted to the challenge for assessment against the test set cases. The algorithm that demonstrated the highest agreement with the reference standard had a quadratic-weighted kappa of 0.885 and a prediction probability concordance of 0.875. Substantial variability in mRALE scores assigned by the annotators and output by the AI algorithms was observed.

Conclusions

The MIDRC mRALE Mastermind Grand Challenge revealed the potential of AI to assess COVID-19 severity from portable CXRs, demonstrating promising performance against the reference standard. The observed variability in mRALE scores highlights the challenges in standardizing severity assessment. These findings contribute to ongoing efforts to develop AI technologies for potential use in clinical practice and offer insights for the enhancement of COVID-19 severity assessment.

Multidisciplinary quantitative and qualitative assessment of IDH-mutant gliomas with full diagnostic deep learning image reconstruction

Rationale and Objectives: Diagnostic accuracy and therapeutic decision-making for IDH-mutant gliomas in tumor board reviews are based on MRI and multidisciplinary interactions.

Materials and Methods

This study explores the feasibility of deep learning-based reconstruction (DLR) in MRI for IDH-mutant gliomas. The research utilizes a multidisciplinary approach, engaging neuroradiologists, neurosurgeons, neuro-oncologists, and radiotherapists to evaluate qualitative aspects of DLR and conventional reconstructed (CR) sequences. Furthermore, quantitative image quality and tumor volumes according to Response Assessment in Neuro-Oncology (RANO) 2.0 standards were assessed.

Results

All DLR sequences consistently outperformed CR sequences (median of 4 for all) in qualitative image quality across all raters (p < 0.001 for all) and revealed higher SNR and CNR values (p < 0.001 for all). Preference for all DLR over CR was overwhelming, with ratings of 84 % from the neuroradiologist, 100 % from the neurosurgeon, 92 % from the neuro-oncologist, and 84 % from the radiation oncologist. The RANO 2.0 compliant measurements showed no significant difference between the CR and DRL sequences (p = 0.142).

Conclusion

This study demonstrates the clinical feasibility of DLR in MR imaging of IDH-mutant gliomas, with significant time savings of 29.6 % on average and non-inferior image quality to CR. DLR sequences received strong multidisciplinary preference, underscoring their potential for enhancing neuro-oncological decision-making and suitability for clinical implementation.

Overview of the Head and Neck Tumor Segmentation for Magnetic Resonance Guided Applications (HNTS-MRG) 2024 Challenge

Magnetic resonance (MR)-guided radiation therapy (RT) is enhancing head and neck cancer (HNC) treatment through superior soft tissue contrast and longitudinal imaging capabilities. However, manual tumor segmentation remains a significant challenge, spurring interest in artificial intelligence (AI)-driven automation. To accelerate innovation in this field, we present the Head and Neck Tumor Segmentation for MR-Guided Applications (HNTS-MRG) 2024 Challenge, a satellite event of the 27th International Conference on Medical Image Computing and Computer Assisted Intervention. This challenge addresses the scarcity of large, publicly available AI-ready adaptive RT datasets in HNC and explores the potential of incorporating multi-timepoint data to enhance RT auto-segmentation performance. Participants tackled two HNC segmentation tasks: automatic delineation of primary gross tumor volume (GTVp) and gross metastatic regional lymph nodes (GTVn) on pre-RT (Task 1) and mid-RT (Task 2) T2-weighted scans. The challenge provided 150 HNC cases for training and 50 for testing, hosted on grand-challenge.org using a Docker submission framework. In total, 19 independent teams from across the world qualified by submitting both their algorithms and corresponding papers, resulting in 18 submissions for Task 1 and 15 submissions for Task 2. Evaluation using the mean aggregated Dice Similarity Coefficient showed top-performing AI methods achieved scores of 0.825 in Task 1 and 0.733 in Task 2. These results surpassed clinician interobserver variability benchmarks, marking significant strides in automated tumor segmentation for MR-guided RT applications in HNC.

The Evolution of Artificial Intelligence in Medical Imaging: From Computer Science to Machine and Deep Learning

Artificial intelligence (AI), the wide spectrum of technologies aiming to give machines or computers the ability to perform human-like cognitive functions, began in the 1940s with the first abstract models of intelligent machines. Soon after, in the 1950s and 1960s, machine learning algorithms such as neural networks and decision trees ignited significant enthusiasm. More recent advancements include the refinement of learning algorithms, the development of convolutional neural networks to efficiently analyze images, and methods to synthesize new images. This renewed enthusiasm was also due to the increase in computational power with graphical processing units and the availability of large digital databases to be mined by neural networks. AI soon began to be applied in medicine, first through expert systems designed to support the clinician's decision and later with neural networks for the detection, classification, or segmentation of malignant lesions in medical images. A recent prospective clinical trial demonstrated the non-inferiority of AI alone compared with a double reading by two radiologists on screening mammography. Natural language processing, recurrent neural networks, transformers, and generative models have both improved the capabilities of making an automated reading of medical images and moved AI to new domains, including the text analysis of electronic health records, image self-labeling, and self-reporting. The availability of open-source and free libraries, as well as powerful computing resources, has greatly facilitated the adoption of deep learning by researchers and clinicians. Key concerns surrounding AI in healthcare include the need for clinical trials to demonstrate efficacy, the perception of AI tools as 'black boxes' that require greater interpretability and explainability, and ethical issues related to ensuring fairness and trustworthiness in AI systems. Thanks to its versatility and impressive results, AI is one of the most promising resources for frontier research and applications in medicine, in particular for oncological applications.

Assessing the Performance of Artificial Intelligence Models: Insights from the American Society of Functional Neuroradiology Artificial Intelligence Competition

BACKGROUND AND PURPOSE

Artificial intelligence models in radiology are frequently developed and validated using data sets from a single institution and are rarely tested on independent, external data sets, raising questions about their generalizability and applicability in clinical practice. The American Society of Functional Neuroradiology (ASFNR) organized a multicenter artificial intelligence competition to evaluate the proficiency of developed models in identifying various pathologies on NCCT, assessing age-based normality and estimating medical urgency.

MATERIALS AND METHODS

In total, 1201 anonymized, full-head NCCT clinical scans from 5 institutions were pooled to form the data set. The data set encompassed studies with normal findings as well as those with pathologies, including acute ischemic stroke, intracranial hemorrhage, traumatic brain injury, and mass effect (detection of these, task 1). NCCTs were also assessed to determine if findings were consistent with expected brain changes for the patient's age (task 2: age-based normality assessment) and to identify any abnormalities requiring immediate medical attention (task 3: evaluation of findings for urgent intervention). Five neuroradiologists labeled each NCCT, with consensus interpretations serving as the ground truth. The competition was announced online, inviting academic institutions and companies. Independent central analysis assessed the performance of each model. Accuracy, sensitivity, specificity, positive and negative predictive values, and receiver operating characteristic (ROC) curves were generated for each artificial intelligence model, along with the area under the ROC curve.

RESULTS

Four teams processed 1177 studies. The median age of patients was 62 years, with an interquartile range of 33 years. Nineteen teams from various academic institutions registered for the competition. Of these, 4 teams submitted their final results. No commercial entities participated in the competition. For task 1, areas under the ROC curve ranged from 0.49 to 0.59. For task 2, two teams completed the task with area under the ROC curve values of 0.57 and 0.52. For task 3, teams had little-to-no agreement with the ground truth.

CONCLUSIONS

To assess the performance of artificial intelligence models in real-world clinical scenarios, we analyzed their performance in the ASFNR Artificial Intelligence Competition. The first ASFNR Competition underscored the gap between expectation and reality; and the models largely fell short in their assessments. As the integration of artificial intelligence tools into clinical workflows increases, neuroradiologists must carefully recognize the capabilities, constraints, and consistency of these technologies. Before institutions adopt these algorithms, thorough validation is essential to ensure acceptable levels of performance in clinical settings.

Speeding Up and Improving Image Quality in Glioblastoma MRI Protocol by Deep Learning Image Reconstruction

A fully diagnostic MRI glioma protocol is key to monitoring therapy assessment but is time-consuming and especially challenging in critically ill and uncooperative patients. Artificial intelligence demonstrated promise in reducing scan time and improving image quality simultaneously. The purpose of this study was to investigate the diagnostic performance, the impact on acquisition acceleration, and the image quality of a deep learning optimized glioma protocol of the brain. Thirty-three patients with histologically confirmed glioblastoma underwent standardized brain tumor imaging according to the glioma consensus recommendations on a 3-Tesla MRI scanner. Conventional and deep learning-reconstructed (DLR) fluid-attenuated inversion recovery, and T2- and T1-weighted contrast-enhanced Turbo spin echo images with an improved in-plane resolution, i.e., super-resolution, were acquired. Two experienced neuroradiologists independently evaluated the image datasets for subjective image quality, diagnostic confidence, tumor conspicuity, noise levels, artifacts, and sharpness. In addition, the tumor volume was measured in the image datasets according to Response Assessment in Neuro-Oncology (RANO) 2.0, as well as compared between both imaging techniques, and various clinical-pathological parameters were determined. The average time saving of DLR sequences was 30% per MRI sequence. Simultaneously, DLR sequences showed superior overall image quality (all p < 0.001), improved tumor conspicuity and image sharpness (all p < 0.001, respectively), and less image noise (all p < 0.001), while maintaining diagnostic confidence (all p > 0.05), compared to conventional images. Regarding RANO 2.0, the volume of non-enhancing non-target lesions (p = 0.963), enhancing target lesions (p = 0.993), and enhancing non-target lesions (p = 0.951) did not differ between reconstruction types. The feasibility of the deep learning-optimized glioma protocol was demonstrated with a 30% reduction in acquisition time on average and an increased in-plane resolution. The evaluated DLR sequences improved subjective image quality and maintained diagnostic accuracy in tumor detection and tumor classification according to RANO 2.0.

Artificial Intelligence-Based Left Ventricular Ejection Fraction by Medical Students for Mortality and Readmission Prediction

INTRODUCTION

Point-of-care ultrasound has become a universal practice, employed by physicians across various disciplines, contributing to diagnostic processes and decision-making.

AIM

To assess the association of reduced (<50%) left-ventricular ejection fraction (LVEF) based on prospective point-of-care ultrasound operated by medical students using an artificial intelligence (AI) tool and 1-year primary composite outcome, including mortality and readmission for cardiovascular-related causes.

METHODS

Eight trained medical students used a hand-held ultrasound device (HUD) equipped with an AI-based tool for automatic evaluation of the LVEF of non-selected patients hospitalized in a cardiology department from March 2019 through March 2020.

RESULTS

The study included 82 patients (72 males aged 58.5 ± 16.8 years), of whom 34 (41.5%) were diagnosed with AI-based reduced LVEF. The rates of the composite outcome were higher among patients with reduced systolic function compared to those with preserved LVEF (41.2% vs. 16.7%, p = 0.014). Adjusting for pertinent variables, reduced LVEF independently predicted the composite outcome (HR 2.717, 95% CI 1.083-6.817, p = 0.033). As compared to those with LVEF ≥ 50%, patients with reduced LVEF had a longer length of stay and higher rates of the secondary composite outcome, including in-hospital death, advanced ventilatory support, shock, and acute decompensated heart failure.

CONCLUSION

AI-based assessment of reduced systolic function in the hands of medical students, independently predicted 1-year mortality and cardiovascular-related readmission and was associated with unfavorable in-hospital outcomes. AI utilization by novice users may be an important tool for risk stratification for hospitalized patients.