Lessons Learned in Building Expertly Annotated Multi-Institution Datasets and Hosting the RSNA AI Challenges

The Radiological Society of North America (RSNA) has held artificial intelligence competitions to tackle real-world medical imaging problems at least annually since 2017. This article examines the challenges and processes involved in organizing these competitions, with a specific emphasis on the creation and curation of high-quality datasets. The collection of diverse and representative medical imaging data involves dealing with issues of patient privacy and data security. Furthermore, ensuring quality and consistency in data, which includes expert labeling and accounting for various patient and imaging characteristics, necessitates substantial planning and resources. Overcoming these obstacles requires meticulous project management and adherence to strict timelines. The article also highlights the potential of crowdsourced annotation to progress medical imaging research. Through the RSNA competitions, an effective global engagement has been realized, resulting in innovative solutions to complex medical imaging problems, thus potentially transforming health care by enhancing diagnostic accuracy and patient outcomes. Keywords: Use of AI in Education, Artificial Intelligence © RSNA, 2024.

Patient-Specific, Mechanistic Models of Tumor Growth Incorporating Artificial Intelligence and Big Data

Despite the remarkable advances in cancer diagnosis, treatment, and management over the past decade, malignant tumors remain a major public health problem. Further progress in combating cancer may be enabled by personalizing the delivery of therapies according to the predicted response for each individual patient. The design of personalized therapies requires the integration of patient-specific information with an appropriate mathematical model of tumor response. A fundamental barrier to realizing this paradigm is the current lack of a rigorous yet practical mathematical theory of tumor initiation, development, invasion, and response to therapy. We begin this review with an overview of different approaches to modeling tumor growth and treatment, including mechanistic as well as data-driven models based on big data and artificial intelligence. We then present illustrative examples of mathematical models manifesting their utility and discuss the limitations of stand-alone mechanistic and data-driven models. We then discuss the potential of mechanistic models for not only predicting but also optimizing response to therapy on a patient-specific basis. We describe current efforts and future possibilities to integrate mechanistic and data-driven models. We conclude by proposing five fundamental challenges that must be addressed to fully realize personalized care for cancer patients driven by computational models.

Clinical and Genomic Predictors of Adverse Events in Newly Diagnosed Glioblastoma

PURPOSE

Adverse clinical events cause significant morbidity in patients with GBM (GBM). We examined whether genomic alterations were associated with AE (AE) in patients with GBM.

EXPERIMENTAL DESIGN

We identified adults with histologically confirmed IDH-wild-type GBM with targeted next-generation sequencing (OncoPanel) at Dana Farber Cancer Institute from 2013 to 2019. Seizure at presentation, lymphopenia, thromboembolic events, pseudoprogression, and early progression (within 6 months of diagnosis) were identified as AE. The biologic function of genetic variants was categorized as loss-of-function (LoF), no change in function, or gain-of-function (GoF) using a somatic tumor mutation knowledge base (OncoKB) and consensus protein function predictions. Associations between functional genomic alterations and AE were examined using univariate logistic regressions and multivariable regressions adjusted for additional clinical predictors.

RESULTS

Our study included 470 patients diagnosed with GBM who met the study criteria. We focused on 105 genes that had sequencing data available for ≥ 90% of the patients and were altered in ≥10% of the cohort. Following false-discovery rate (FDR) correction and multivariable adjustment, the TP53, RB1, IGF1R, and DIS3 LoF alterations were associated with lower odds of seizures, while EGFR, SMARCA4, GNA11, BRD4, and TCF3 GoF and SETD2 LoF alterations were associated with higher odds of seizures. For all other AE of interest, no significant associations were found with genomic alterations following FDR correction.

CONCLUSIONS

Genomic biomarkers based on functional variant analysis of a routine clinical panel may help identify AE in GBM, particularly seizures. Identifying these risk factors could improve the management of patients through better supportive care and consideration of prophylactic therapies.

Multinational External Validation of Autonomous Retinopathy of Prematurity Screening

Importance

Retinopathy of prematurity (ROP) is a leading cause of blindness in children, with significant disparities in outcomes between high-income and low-income countries, due in part to insufficient access to ROP screening.

Objective

To evaluate how well autonomous artificial intelligence (AI)-based ROP screening can detect more-than-mild ROP (mtmROP) and type 1 ROP.

Design, Setting, and Participants

This diagnostic study evaluated the performance of an AI algorithm, trained and calibrated using 2530 examinations from 843 infants in the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) study, on 2 external datasets (6245 examinations from 1545 infants in the Stanford University Network for Diagnosis of ROP [SUNDROP] and 5635 examinations from 2699 infants in the Aravind Eye Care Systems [AECS] telemedicine programs). Data were taken from 11 and 48 neonatal care units in the US and India, respectively. Data were collected from January 2012 to July 2021, and data were analyzed from July to December 2023.

Exposures

An imaging processing pipeline was created using deep learning to autonomously identify mtmROP and type 1 ROP in eye examinations performed via telemedicine.

Main Outcomes and Measures

The area under the receiver operating characteristics curve (AUROC) as well as sensitivity and specificity for detection of mtmROP and type 1 ROP at the eye examination and patient levels.

Results

The prevalence of mtmROP and type 1 ROP were 5.9% (91 of 1545) and 1.2% (18 of 1545), respectively, in the SUNDROP dataset and 6.2% (168 of 2699) and 2.5% (68 of 2699) in the AECS dataset. Examination-level AUROCs for mtmROP and type 1 ROP were 0.896 and 0.985, respectively, in the SUNDROP dataset and 0.920 and 0.982 in the AECS dataset. At the cross-sectional examination level, mtmROP detection had high sensitivity (SUNDROP: mtmROP, 83.5%; 95% CI, 76.6-87.7; type 1 ROP, 82.2%; 95% CI, 81.2-83.1; AECS: mtmROP, 80.8%; 95% CI, 76.2-84.9; type 1 ROP, 87.8%; 95% CI, 86.8-88.7). At the patient level, all infants who developed type 1 ROP screened positive (SUNDROP: 100%; 95% CI, 81.4-100; AECS: 100%; 95% CI, 94.7-100) prior to diagnosis.

Conclusions and Relevance

Where and when ROP telemedicine programs can be implemented, autonomous ROP screening may be an effective force multiplier for secondary prevention of ROP.

Not without Context-A Multiple Methods Study on Evaluation and Correction of Automated Brain Tumor Segmentations by Experts

RATIONALE AND OBJECTIVES

Brain tumor segmentations are integral to the clinical management of patients with glioblastoma, the deadliest primary brain tumor in adults. The manual delineation of tumors is time-consuming and highly provider-dependent. These two problems must be addressed by introducing automated, deep-learning-based segmentation tools. This study aimed to identify criteria experts use to evaluate the quality of automatically generated segmentations and their thought processes as they correct them.

MATERIALS AND METHODS

Multiple methods were used to develop a detailed understanding of the complex factors that shape experts' perception of segmentation quality and their thought processes in correcting proposed segmentations. Data from a questionnaire and semistructured interview with neuro-oncologists and neuroradiologists were collected between August and December 2021 and analyzed using a combined deductive and inductive approach.

RESULTS

Brain tumors are highly complex and ambiguous segmentation targets. Therefore, physicians rely heavily on the given context related to the patient and clinical context in evaluating the quality and need to correct brain tumor segmentation. Most importantly, the intended clinical application determines the segmentation quality criteria and editing decisions. Physicians' personal beliefs and preferences about the capabilities of AI algorithms and whether questionable areas should not be included are additional criteria influencing the perception of segmentation quality and appearance of an edited segmentation.

CONCLUSION

Our findings on experts' perceptions of segmentation quality will allow the design of improved frameworks for expert-centered evaluation of brain tumor segmentation models. In particular, the knowledge presented here can inspire the development of brain tumor-specific metrics for segmentation model training and evaluation.

Oxygenation Fluctuations Associated with Severe Retinopathy of Prematurity: Insights from a Multimodal Deep Learning Approach

Purpose

Retinopathy of prematurity (ROP) is one of the leading causes of blindness in children. Although the role of oxygen in the pathophysiology of ROP is well established, a precise understanding of the dynamic relationship between oxygen exposure ROP incidence and severity is lacking. The purpose of this study was to evaluate the correlation between time-dependent oxygen variables and the onset of ROP.

Design

Retrospective cohort study.

Participants

Two hundred thirty infants who were born at a single academic center and met the inclusion criteria were included. Infants are mainly born between January 2011 and October 2022.

Methods

Patient data were extracted from electronic health records (EHRs), with sufficient time-dependent oxygen data. Clinical outcomes for ROP were recorded as none/mild or moderate/severe (defined as type II or worse). Mixed-effects linear models were used to compare the 2 groups in terms of dynamic oxygen variables, such as daily average and the coefficient of variation (COV) fraction of inspired oxygen (FiO2). Support vector machine (SVM) and long-short-term memory (LSTM)-based multimodal models were trained with fivefold cross-validation to predict which infants would develop moderate/severe ROP. Gestational age (GA), birth weight, and time-dependent oxygen variables were used to develop predictive models.

Main Outcome Measures

Model cross-validation performance was evaluated by computing the mean area under the receiver operating characteristic (AUROC) curve, precision, recall, and F1 score.

Results

We found that both daily average and COV of FiO2 were associated with more severe ROP (adjusted P < 0.001). With fivefold cross-validation, the multimodal LSTM models had higher performance than the best static models (SVM using GA and 3 average FiO2 features) and SVM models trained on GA alone (mean AUROC = 0.89 ± 0.04 vs. 0.86 ± 0.05 vs. 0.83 ± 0.04).

Conclusions

The development of severe ROP might not only be influenced by oxygen exposure but also by its fluctuation, which provides direction for future study of pathophysiological factors associated with severe ROP development. Additionally, we demonstrated that multimodal neural networks can be a method to extract useful information from time-series data, which may be a valuable methodology for the investigation of other diseases using EHR data.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Design of the HPV-automated visual evaluation (PAVE) study: Validating a novel cervical screening strategy

Background

The HPV-automated visual evaluation (PAVE) Study is an extensive, multinational initiative designed to advance cervical cancer prevention in resource-constrained regions. Cervical cancer disproportionally affects regions with limited access to preventive measures. PAVE aims to assess a novel screening-triage-treatment strategy integrating self-sampled HPV testing, deep-learning-based automated visual evaluation (AVE), and targeted therapies.

Methods

Phase 1 efficacy involves screening up to 100,000 women aged 25-49 across nine countries, using self-collected vaginal samples for hierarchical HPV evaluation: HPV16, else HPV18/45, else HPV31/33/35/52/58, else HPV39/51/56/59/68 else negative. HPV-positive individuals undergo further evaluation, including pelvic exams, cervical imaging, and biopsies. AVE algorithms analyze images, assigning risk scores for precancer, validated against histologic high-grade precancer. Phase 1, however, does not integrate AVE results into patient management, contrasting them with local standard care.Phase 2 effectiveness focuses on deploying AVE software and HPV genotype data in real-time clinical decision-making, evaluating feasibility, acceptability, cost-effectiveness, and health communication of the PAVE strategy in practice.

Results

Currently, sites have commenced fieldwork, and conclusive results are pending.

Conclusions

The study aspires to validate a screen-triage-treat protocol utilizing innovative biomarkers to deliver an accurate, feasible, and cost-effective strategy for cervical cancer prevention in resource-limited areas. Should the study validate PAVE, its broader implementation could be recommended, potentially expanding cervical cancer prevention worldwide.

Funding

The consortial sites are responsible for their own study costs. Research equipment and supplies, and the NCI-affiliated staff are funded by the National Cancer Institute Intramural Research Program including supplemental funding from the Cancer Cures Moonshot Initiative. No commercial support was obtained. Brian Befano was supported by NCI/ NIH under Grant T32CA09168.

Artificial intelligence-based image analysis in clinical testing: lessons from cervical cancer screening

Novel screening and diagnostic tests based on artificial intelligence (AI) image recognition algorithms are proliferating. Some initial reports claim outstanding accuracy followed by disappointing lack of confirmation, including our own early work on cervical screening. This is a presentation of lessons learned, organized as a conceptual step-by-step approach to bridge the gap between the creation of an AI algorithm and clinical efficacy. The first fundamental principle is specifying rigorously what the algorithm is designed to identify and what the test is intended to measure (eg, screening, diagnostic, or prognostic). Second, designing the AI algorithm to minimize the most clinically important errors. For example, many equivocal cervical images cannot yet be labeled because the borderline between cases and controls is blurred. To avoid a misclassified case-control dichotomy, we have isolated the equivocal cases and formally included an intermediate, indeterminate class (severity order of classes: case>indeterminate>control). The third principle is evaluating AI algorithms like any other test, using clinical epidemiologic criteria. Repeatability of the algorithm at the borderline, for indeterminate images, has proven extremely informative. Distinguishing between internal and external validation is also essential. Linking the AI algorithm results to clinical risk estimation is the fourth principle. Absolute risk (not relative) is the critical metric for translating a test result into clinical use. Finally, generating risk-based guidelines for clinical use that match local resources and priorities is the last principle in our approach. We are particularly interested in applications to lower-resource settings to address health disparities. We note that similar principles apply to other domains of AI-based image analysis for medical diagnostic testing.

Discrepancies in Diagnosis of Treatment-Requiring Retinopathy of Prematurity

52% of treated eyes with retinopathy of prematurity in a multicenter cohort didn’t require intervention per evaluation by an independent reading center. An artificial intelligence system detected worse vascular severity in the group designed as treatment-requiring by reading center.

Breast Multiparametric MRI for Prediction of Neoadjuvant Chemotherapy Response in Breast Cancer: The BMMR2 Challenge

Purpose To describe the design, conduct, and results of the Breast Multiparametric MRI for prediction of neoadjuvant chemotherapy Response (BMMR2) challenge. Materials and Methods The BMMR2 computational challenge opened on May 28, 2021, and closed on December 21, 2021. The goal of the challenge was to identify image-based markers derived from multiparametric breast MRI, including diffusion-weighted imaging (DWI) and dynamic contrast-enhanced (DCE) MRI, along with clinical data for predicting pathologic complete response (pCR) following neoadjuvant treatment. Data included 573 breast MRI studies from 191 women (mean age [±SD], 48.9 years ± 10.56) in the I-SPY 2/American College of Radiology Imaging Network (ACRIN) 6698 trial (ClinicalTrials.gov: NCT01042379). The challenge cohort was split into training (60%) and test (40%) sets, with teams blinded to test set pCR outcomes. Prediction performance was evaluated by area under the receiver operating characteristic curve (AUC) and compared with the benchmark established from the ACRIN 6698 primary analysis. Results Eight teams submitted final predictions. Entries from three teams had point estimators of AUC that were higher than the benchmark performance (AUC, 0.782 [95% CI: 0.670, 0.893], with AUCs of 0.803 [95% CI: 0.702, 0.904], 0.838 [95% CI: 0.748, 0.928], and 0.840 [95% CI: 0.748, 0.932]). A variety of approaches were used, ranging from extraction of individual features to deep learning and artificial intelligence methods, incorporating DCE and DWI alone or in combination. Conclusion The BMMR2 challenge identified several models with high predictive performance, which may further expand the value of multiparametric breast MRI as an early marker of treatment response. Clinical trial registration no. NCT01042379 Keywords: MRI, Breast, Tumor Response Supplemental material is available for this article. © RSNA, 2024.

Agreement on Lesion Presence and Location at Colposcopy

OBJECTIVES/PURPOSE

The reproducibility and sensitivity of image-based colposcopy is low, but agreement on lesion presence and location remains to be explored. Here, we investigate the interobserver agreement on lesions on colposcopic images by evaluating and comparing marked lesions on digitized colposcopic images between colposcopists.

METHODS

Five colposcopists reviewed images from 268 colposcopic examinations. Cases were selected based on histologic diagnosis, i.e., normal/cervical intraepithelial neoplasia (CIN)1 ( n = 50), CIN2 ( n = 50), CIN3 ( n = 100), adenocarcinoma in situ ( n = 53), and cancer ( n = 15). We obtained digitized time-series images every 7-10 seconds from before acetic acid application to 2 minutes after application. Colposcopists were instructed to digitally annotate all areas with acetowhitening or suspect of lesions. To estimate the agreement on lesion presence and location, we assessed the proportion of images with annotations and the proportion of images with overlapping annotated area by at least 4 (4+) colposcopists, respectively.

RESULTS

We included images from 241 examinations (1 image from each) with adequate annotations. The proportion with a least 1 lesion annotated by 4+ colposcopists increased by severity of histologic diagnosis. Among the CIN3 cases, 84% had at least 1 lesion annotated by 4+ colposcopists, whereas 54% of normal/CIN1 cases had a lesion annotated. Notably, the proportion was 70% for adenocarcinoma in situ and 71% for cancer. Regarding lesion location, there was no linear association with severity of histologic diagnosis.

CONCLUSION

Despite that 80% of the CIN2 and CIN3 cases were annotated by 4+ colposcopists, we did not find increasing agreement on lesion location with histology severity. This underlines the subjective nature of colposcopy.

Expert-centered Evaluation of Deep Learning Algorithms for Brain Tumor Segmentation

Purpose To present results from a literature survey on practices in deep learning segmentation algorithm evaluation and perform a study on expert quality perception of brain tumor segmentation. Materials and Methods A total of 180 articles reporting on brain tumor segmentation algorithms were surveyed for the reported quality evaluation. Additionally, ratings of segmentation quality on a four-point scale were collected from medical professionals for 60 brain tumor segmentation cases. Results Of the surveyed articles, Dice score, sensitivity, and Hausdorff distance were the most popular metrics to report segmentation performance. Notably, only 2.8% of the articles included clinical experts' evaluation of segmentation quality. The experimental results revealed a low interrater agreement (Krippendorff α, 0.34) in experts' segmentation quality perception. Furthermore, the correlations between the ratings and commonly used quantitative quality metrics were low (Kendall tau between Dice score and mean rating, 0.23; Kendall tau between Hausdorff distance and mean rating, 0.51), with large variability among the experts. Conclusion The results demonstrate that quality ratings are prone to variability due to the ambiguity of tumor boundaries and individual perceptual differences, and existing metrics do not capture the clinical perception of segmentation quality. Keywords: Brain Tumor Segmentation, Deep Learning Algorithms, Glioblastoma, Cancer, Machine Learning Clinical trial registration nos. NCT00756106 and NCT00662506 Supplemental material is available for this article. © RSNA, 2023.

Reproducible and clinically translatable deep neural networks for cervical screening

Cervical cancer is a leading cause of cancer mortality, with approximately 90% of the 250,000 deaths per year occurring in low- and middle-income countries (LMIC). Secondary prevention with cervical screening involves detecting and treating precursor lesions; however, scaling screening efforts in LMIC has been hampered by infrastructure and cost constraints. Recent work has supported the development of an artificial intelligence (AI) pipeline on digital images of the cervix to achieve an accurate and reliable diagnosis of treatable precancerous lesions. In particular, WHO guidelines emphasize visual triage of women testing positive for human papillomavirus (HPV) as the primary screen, and AI could assist in this triage task. In this work, we implemented a comprehensive deep-learning model selection and optimization study on a large, collated, multi-geography, multi-institution, and multi-device dataset of 9462 women (17,013 images). We evaluated relative portability, repeatability, and classification performance. The top performing model, when combined with HPV type, achieved an area under the Receiver Operating Characteristics (ROC) curve (AUC) of 0.89 within our study population of interest, and a limited total extreme misclassification rate of 3.4%, on held-aside test sets. Our model also produced reliable and consistent predictions, achieving a strong quadratic weighted kappa (QWK) of 0.86 and a minimal %2-class disagreement (% 2-Cl. D.) of 0.69%, between image pairs across women. Our work is among the first efforts at designing a robust, repeatable, accurate and clinically translatable deep-learning model for cervical screening.

Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology: Clinical Applications and Challenges

The rapid progress of large language models (LLMs) driving generative artificial intelligence applications heralds the potential of opportunities in health care. We conducted a review up to April 2023 on Google Scholar, Embase, MEDLINE, and Scopus using the following terms: "large language models," "generative artificial intelligence," "ophthalmology," "ChatGPT," and "eye," based on relevance to this review. From a clinical viewpoint specific to ophthalmologists, we explore from the different stakeholders' perspectives-including patients, physicians, and policymakers-the potential LLM applications in education, research, and clinical domains specific to ophthalmology. We also highlight the foreseeable challenges of LLM implementation into clinical practice, including the concerns of accuracy, interpretability, perpetuating bias, and data security. As LLMs continue to mature, it is essential for stakeholders to jointly establish standards for best practices to safeguard patient safety.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Stroke-Related Visceral Alterations: A Voxel-Based Neuroanatomic Localization Study

OBJECTIVE

Functional and morphologic changes in extracranial organs can occur after acute brain injury. The neuroanatomic correlates of such changes are not fully known. Herein, we tested the hypothesis that brain infarcts are associated with cardiac and systemic abnormalities (CSAs) in a regionally specific manner.

METHODS

We generated voxelwise p value maps of brain infarcts for poststroke plasma cardiac troponin T (cTnT) elevation, QTc prolongation, in-hospital infection, and acute stress hyperglycemia (ASH) in 1,208 acute ischemic stroke patients prospectively recruited into the Heart-Brain Interactions Study. We examined the relationship between infarct location and CSAs using a permutation-based approach and identified clusters of contiguous voxels associated with p < 0.05.

RESULTS

cTnT elevation not attributable to a known cardiac reason was detected in 5.5%, QTc prolongation in the absence of a known provoker in 21.2%, ASH in 33.9%, and poststroke infection in 13.6%. We identified significant, spatially segregated voxel clusters for each CSA. The clusters for troponin elevation and QTc prolongation mapped to the right hemisphere. There were 3 clusters for ASH, the largest of which was in the left hemisphere. We found 2 clusters for poststroke infection, one associated with pneumonia in the left and one with urinary tract infection in the right hemisphere. The relationship between infarct location and CSAs persisted after adjusting for infarct volume.

INTERPRETATION

Our results show that there are discrete regions of brain infarcts associated with CSAs. This information could be used to bootstrap toward new markers for better differentiation between neurogenic and non-neurogenic mechanisms of poststroke CSAs. ANN NEUROL 2023;94:1155-1163.

Longitudinal assessment of demographic representativeness in the Medical Imaging and Data Resource Center open data commons

Purpose

The Medical Imaging and Data Resource Center (MIDRC) open data commons was launched to accelerate the development of artificial intelligence (AI) algorithms to help address the COVID-19 pandemic. The purpose of this study was to quantify longitudinal representativeness of the demographic characteristics of the primary MIDRC dataset compared to the United States general population (US Census) and COVID-19 positive case counts from the Centers for Disease Control and Prevention (CDC).

Approach

The Jensen-Shannon distance (JSD), a measure of similarity of two distributions, was used to longitudinally measure the representativeness of the distribution of (1) all unique patients in the MIDRC data to the 2020 US Census and (2) all unique COVID-19 positive patients in the MIDRC data to the case counts reported by the CDC. The distributions were evaluated in the demographic categories of age at index, sex, race, ethnicity, and the combination of race and ethnicity.

Results

Representativeness of the MIDRC data by ethnicity and the combination of race and ethnicity was impacted by the percentage of CDC case counts for which this was not reported. The distributions by sex and race have retained their level of representativeness over time.

Conclusion

The representativeness of the open medical imaging datasets in the curated public data commons at MIDRC has evolved over time as the number of contributing institutions and overall number of subjects have grown. The use of metrics, such as the JSD support measurement of representativeness, is one step needed for fair and generalizable AI algorithm development.

Toward fairness in artificial intelligence for medical image analysis: identification and mitigation of potential biases in the roadmap from data collection to model deployment

Purpose

To recognize and address various sources of bias essential for algorithmic fairness and trustworthiness and to contribute to a just and equitable deployment of AI in medical imaging, there is an increasing interest in developing medical imaging-based machine learning methods, also known as medical imaging artificial intelligence (AI), for the detection, diagnosis, prognosis, and risk assessment of disease with the goal of clinical implementation. These tools are intended to help improve traditional human decision-making in medical imaging. However, biases introduced in the steps toward clinical deployment may impede their intended function, potentially exacerbating inequities. Specifically, medical imaging AI can propagate or amplify biases introduced in the many steps from model inception to deployment, resulting in a systematic difference in the treatment of different groups.

Approach

Our multi-institutional team included medical physicists, medical imaging artificial intelligence/machine learning (AI/ML) researchers, experts in AI/ML bias, statisticians, physicians, and scientists from regulatory bodies. We identified sources of bias in AI/ML, mitigation strategies for these biases, and developed recommendations for best practices in medical imaging AI/ML development.

Results

Five main steps along the roadmap of medical imaging AI/ML were identified: (1) data collection, (2) data preparation and annotation, (3) model development, (4) model evaluation, and (5) model deployment. Within these steps, or bias categories, we identified 29 sources of potential bias, many of which can impact multiple steps, as well as mitigation strategies.

Conclusions

Our findings provide a valuable resource to researchers, clinicians, and the public at large.

Design of the HPV-Automated Visual Evaluation (PAVE) Study: Validating a Novel Cervical Screening Strategy

Objective

To describe the HPV-Automated Visual Evaluation (PAVE) Study, an international, multi-centric study designed to evaluate a novel cervical screen-triage-treat strategy for resource-limited settings as part of a global strategy to reduce cervical cancer burden. The PAVE strategy involves: 1) screening with self-sampled HPV testing; 2) triage of HPV-positive participants with a combination of extended genotyping and visual evaluation of the cervix assisted by deep-learning-based automated visual evaluation (AVE); and 3) treatment with thermal ablation or excision (Large Loop Excision of the Transformation Zone). The PAVE study has two phases: efficacy (2023-2024) and effectiveness (planned to begin in 2024-2025). The efficacy phase aims to refine and validate the screen-triage portion of the protocol. The effectiveness phase will examine acceptability and feasibility of the PAVE strategy into clinical practice, cost-effectiveness, and health communication within the PAVE sites.

Study design

Phase 1 Efficacy: Around 100,000 nonpregnant women, aged 25-49 years, without prior hysterectomy, and irrespective of HIV status, are being screened at nine study sites in resource-limited settings. Eligible and consenting participants perform self-collection of vaginal specimens for HPV testing using a FLOQSwab (Copan). Swabs are transported dry and undergo testing for HPV using a newly-redesigned isothermal DNA amplification HPV test (ScreenFire HPV RS), which has been designed to provide HPV genotyping by hierarchical risk groups: HPV16, else HPV18/45, else HPV31/33/35/52/58, else HPV39/51/56/59/68. HPV-negative individuals are considered negative for precancer/cancer and do not undergo further testing. HPV-positive individuals undergo pelvic examination with collection of cervical images and targeted biopsies of all acetowhite areas or endocervical sampling in the absence of visible lesions. Accuracy of histology diagnosis is evaluated across all sites. Cervical images are used to refine a deep learning AVE algorithm that classifies images as normal, indeterminate, or precancer+. AVE classifications are validated against the histologic endpoint of high-grade precancer determined by biopsy. The combination of HPV genotype and AVE classification is used to generate a risk score that corresponds to the risk of precancer (lower, medium, high, highest). During the efficacy phase, clinicians and patients within the PAVE sites will receive HPV testing results but not AVE results or risk scores. Treatment during the efficacy phase will be performed per local standard of care: positive Visual Inspection with Acetic Acid impression, high-grade colposcopic impression or CIN2+ on colposcopic biopsy, HPV positivity, or HPV 16,18/45 positivity. Follow up of triage negative patients and post treatment will follow standard of care protocols. The sensitivity of the PAVE strategy for detection of precancer will be compared to current SOC at a given level of specificity.Phase 2 Effectiveness: The AVE software will be downloaded to the new dedicated image analysis and thermal ablation devices (Liger Iris) into which the HPV genotype information can be entered to provide risk HPV-AVE risk scores for precancer to clinicians in real time. The effectiveness phase will examine clinician use of the PAVE strategy in practice, including feasibility and acceptability for clinicians and patients, cost-effectiveness, and health communication within the PAVE sites.

Conclusion

The goal of the PAVE study is to validate a screen-triage-treat protocol using novel biomarkers to provide an accurate, feasible, cost-effective strategy for cervical cancer prevention in resource-limited settings. If validated, implementation of PAVE at larger scale can be encouraged.

Funding

The consortial sites are responsible for their own study costs. Research equipment and supplies, and the NCI-affiliated staff are funded by the National Cancer Institute Intramural Research Program including supplemental funding from the Cancer Cures Moonshot Initiative. No commercial support was obtained. Brian Befano was supported by NCI/NIH under Grant T32CA09168. Date of protocol latest review: September 24 th 2023.

Transformer versus traditional natural language processing: how much data is enough for automated radiology report classification?

OBJECTIVES

Current state-of-the-art natural language processing (NLP) techniques use transformer deep-learning architectures, which depend on large training datasets. We hypothesized that traditional NLP techniques may outperform transformers for smaller radiology report datasets.

METHODS

We compared the performance of BioBERT, a deep-learning-based transformer model pre-trained on biomedical text, and three traditional machine-learning models (gradient boosted tree, random forest, and logistic regression) on seven classification tasks given free-text radiology reports. Tasks included detection of appendicitis, diverticulitis, bowel obstruction, and enteritis/colitis on abdomen/pelvis CT reports, ischemic infarct on brain CT/MRI reports, and medial and lateral meniscus tears on knee MRI reports (7,204 total annotated reports). The performance of NLP models on held-out test sets was compared after training using the full training set, and 2.5%, 10%, 25%, 50%, and 75% random subsets of the training data.

RESULTS

In all tested classification tasks, BioBERT performed poorly at smaller training sample sizes compared to non-deep-learning NLP models. Specifically, BioBERT required training on approximately 1,000 reports to perform similarly or better than non-deep-learning models. At around 1,250 to 1,500 training samples, the testing performance for all models began to plateau, where additional training data yielded minimal performance gain.

CONCLUSIONS

With larger sample sizes, transformer NLP models achieved superior performance in radiology report binary classification tasks. However, with smaller sizes (<1000) and more imbalanced training data, traditional NLP techniques performed better.

ADVANCES IN KNOWLEDGE

Our benchmarks can help guide clinical NLP researchers in selecting machine-learning models according to their dataset characteristics.

The RSNA Cervical Spine Fracture CT Dataset

This dataset is composed of cervical spine CT images with annotations related to fractures; it is available at https://www.kaggle.com/competitions/rsna-2022-cervical-spine-fracture-detection/.

Structural and functional vascular dysfunction within brain metastases is linked to pembrolizumab inefficacy

Structurally and functionally aberrant vasculature is a hallmark of tumor angiogenesis and treatment resistance. Given the synergistic link between aberrant tumor vasculature and immunosuppression, we analyzed perfusion MRI for 44 patients with brain metastases (BM) undergoing treatment with pembrolizumab. To date, vascular-immune communication, or the relationship between immune checkpoint inhibitor (ICI) efficacy and vascular architecture, has not been well-characterized in human imaging studies. We found that ICI-responsive BM possessed a structurally balanced vascular makeup, which was linked to improved vascular efficiency and an immune-stimulatory microenvironment. In contrast, ICI-resistant BM were characterized by a lack of immune cell infiltration and a highly aberrant vasculature dominated by large-caliber vessels. Peri-tumor region analysis revealed early functional changes predictive of ICI resistance before radiographic evidence on conventional MRI. This study was one of the largest functional imaging studies for BM and establishes a foundation for functional studies that illuminate the mechanisms linking patterns of vascular architecture with immunosuppression, as targeting these aspects of cancer biology may serve as the basis for future combination treatments.

Patient-specific, mechanistic models of tumor growth incorporating artificial intelligence and big data

Despite the remarkable advances in cancer diagnosis, treatment, and management that have occurred over the past decade, malignant tumors remain a major public health problem. Further progress in combating cancer may be enabled by personalizing the delivery of therapies according to the predicted response for each individual patient. The design of personalized therapies requires patient-specific information integrated into an appropriate mathematical model of tumor response. A fundamental barrier to realizing this paradigm is the current lack of a rigorous, yet practical, mathematical theory of tumor initiation, development, invasion, and response to therapy. In this review, we begin by providing an overview of different approaches to modeling tumor growth and treatment, including mechanistic as well as data-driven models based on ``big data" and artificial intelligence. Next, we present illustrative examples of mathematical models manifesting their utility and discussing the limitations of stand-alone mechanistic and data-driven models. We further discuss the potential of mechanistic models for not only predicting, but also optimizing response to therapy on a patient-specific basis. We then discuss current efforts and future possibilities to integrate mechanistic and data-driven models. We conclude by proposing five fundamental challenges that must be addressed to fully realize personalized care for cancer patients driven by computational models.

Automated detection of genetic relatedness from fundus photographs using Siamese Neural Networks

Heritability of common eye diseases and ocular traits are relatively high. Here, we develop an automated algorithm to detect genetic relatedness from color fundus photographs (FPs). We estimated the degree of shared ancestry amongst individuals in the UK Biobank using KING software. A convolutional Siamese neural network-based algorithm was trained to output a measure of genetic relatedness using 7224 pairs (3612 related and 3612 unrelated) of FPs. The model achieved high performance for prediction of genetic relatedness; when computed Euclidean distances were used to determine probability of relatedness, the area under the receiver operating characteristic curve (AUROC) for identifying related FPs reached 0.926. We performed external validation of our model using FPs from the LIFE-Adult study and achieved an AUROC of 0.69. An occlusion map indicates that the optic nerve and its surrounding area may be the most predictive of genetic relatedness. We demonstrate that genetic relatedness can be captured from FP features. This approach may be used to uncover novel biomarkers for common ocular diseases.

Bavituximab Decreases Immunosuppressive Myeloid-Derived Suppressor Cells in Newly Diagnosed Glioblastoma Patients

PURPOSE

We evaluated the efficacy of bavituximab-a mAb with anti-angiogenic and immunomodulatory properties-in newly diagnosed patients with glioblastoma (GBM) who also received radiotherapy and temozolomide. Perfusion MRI and myeloid-related gene transcription and inflammatory infiltrates in pre-and post-treatment tumor specimens were studied to evaluate on-target effects (NCT03139916).

PATIENTS AND METHODS

Thirty-three adults with IDH--wild-type GBM received 6 weeks of concurrent chemoradiotherapy, followed by 6 cycles of temozolomide (C1-C6). Bavituximab was given weekly, starting week 1 of chemoradiotherapy, for at least 18 weeks. The primary endpoint was proportion of patients alive at 12 months (OS-12). The null hypothesis would be rejected if OS-12 was ≥72%. Relative cerebral blood flow (rCBF) and vascular permeability (Ktrans) were calculated from perfusion MRIs. Peripheral blood mononuclear cells and tumor tissue were analyzed pre-treatment and at disease progression using RNA transcriptomics and multispectral immunofluorescence for myeloid-derived suppressor cells (MDSC) and macrophages.

RESULTS

The study met its primary endpoint with an OS-12 of 73% (95% confidence interval, 59%-90%). Decreased pre-C1 rCBF (HR, 4.63; P = 0.029) and increased pre-C1 Ktrans were associated with improved overall survival (HR, 0.09; P = 0.005). Pre-treatment overexpression of myeloid-related genes in tumor tissue was associated with longer survival. Post-treatment tumor specimens contained fewer immunosuppressive MDSCs (P = 0.01).

CONCLUSIONS

Bavituximab has activity in newly diagnosed GBM and resulted in on-target depletion of intratumoral immunosuppressive MDSCs. Elevated pre-treatment expression of myeloid-related transcripts in GBM may predict response to bavituximab.

FDU-Net: Deep Learning-Based Three-Dimensional Diffuse Optical Image Reconstruction

Near-infrared diffuse optical tomography (DOT) is a promising functional modality for breast cancer imaging; however, the clinical translation of DOT is hampered by technical limitations. Specifically, conventional finite element method (FEM)-based optical image reconstruction approaches are time-consuming and ineffective in recovering full lesion contrast. To address this, we developed a deep learning-based reconstruction model (FDU-Net) comprised of a Fully connected subnet, followed by a convolutional encoder-Decoder subnet, and a U-Net for fast, end-to-end 3D DOT image reconstruction. The FDU-Net was trained on digital phantoms that include randomly located singular spherical inclusions of various sizes and contrasts. Reconstruction performance was evaluated in 400 simulated cases with realistic noise profiles for the FDU-Net and conventional FEM approaches. Our results show that the overall quality of images reconstructed by FDU-Net is significantly improved compared to FEM-based methods and a previously proposed deep-learning network. Importantly, once trained, FDU-Net demonstrates substantially better capability to recover true inclusion contrast and location without using any inclusion information during reconstruction. The model was also generalizable to multi-focal and irregularly shaped inclusions unseen during training. Finally, FDU-Net, trained on simulated data, could successfully reconstruct a breast tumor from a real patient measurement. Overall, our deep learning-based approach demonstrates marked superiority over the conventional DOT image reconstruction methods while also offering over four orders of magnitude acceleration in computational time. Once adapted to the clinical breast imaging workflow, FDU-Net has the potential to provide real-time accurate lesion characterization by DOT to assist the clinical diagnosis and management of breast cancer.

Epidemiologic Evaluation of Retinopathy of Prematurity Severity in a Large Telemedicine Program in India Using Artificial Intelligence

PURPOSE

Epidemiological changes in retinopathy of prematurity (ROP) depend on neonatal care, neonatal mortality, and the ability to carefully titrate and monitor oxygen. We evaluate whether an artificial intelligence (AI) algorithm for assessing ROP severity in babies can be used to evaluate changes in disease epidemiology in babies from South India over a 5-year period.

DESIGN

Retrospective cohort study.

PARTICIPANTS

Babies (3093) screened for ROP at neonatal care units (NCUs) across the Aravind Eye Care System (AECS) in South India.

METHODS

Images and clinical data were collected as part of routine tele-ROP screening at the AECS in India over 2 time periods: August 2015 to October 2017 and March 2019 to December 2020. All babies in the original cohort were matched 1:3 by birthweight (BW) and gestational age (GA) with babies in the later cohort. We compared the proportion of eyes with moderate (type 2) or treatment-requiring (TR) ROP, and an AI-derived ROP vascular severity score (from retinal fundus images) at the initial tele-retinal screening exam for all babies in a district, VSS), in the 2 time periods.

MAIN OUTCOME MEASURES

Differences in the proportions of type 2 or worse and TR-ROP cases, and VSS between time periods.

RESULTS

Among BW and GA matched babies, the proportion [95% confidence interval {CI}] of babies with type 2 or worse and TR-ROP decreased from 60.9% [53.8%-67.7%] to 17.1% [14.0%-20.5%] (P < 0.001) and 16.8% [11.9%-22.7%] to 5.1% [3.4%-7.3%] (P < 0.001), over the 2 time periods. Similarly, the median [interquartile range] VSS in the population decreased from 2.9 [1.2] to 2.4 [1.8] (P < 0.001).

CONCLUSIONS

In South India, over a 5-year period, the proportion of babies developing moderate to severe ROP has dropped significantly for babies at similar demographic risk, strongly suggesting improvements in primary prevention of ROP. These results suggest that AI-based assessment of ROP severity may be a useful epidemiologic tool to evaluate temporal changes in ROP epidemiology.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found after the references.

Efficacy of Smartphone-Based Telescreening for Retinopathy of Prematurity With and Without Artificial Intelligence in India

Importance

Retinopathy of prematurity (ROP) telemedicine screening programs have been found to be effective, but they rely on widefield digital fundus imaging (WDFI) cameras, which are expensive, making them less accessible in low- to middle-income countries. Cheaper, smartphone-based fundus imaging (SBFI) systems have been described, but these have a narrower field of view (FOV) and have not been tested in a real-world, operational telemedicine setting.

Objective

To assess the efficacy of SBFI systems compared with WDFI when used by technicians for ROP screening with both artificial intelligence (AI) and human graders.

Design, Setting, and Participants

This prospective cross-sectional comparison study took place as a single-center ROP teleophthalmology program in India from January 2021 to April 2022. Premature infants who met normal ROP screening criteria and enrolled in the teleophthalmology screening program were included. Those who had already been treated for ROP were excluded.

Exposures

All participants had WDFI images and from 1 of 2 SBFI devices, the Make-In-India (MII) Retcam or Keeler Monocular Indirect Ophthalmoscope (MIO) devices. Two masked readers evaluated zone, stage, plus, and vascular severity scores (VSS, from 1-9) in all images. Smartphone images were then stratified by patient into training (70%), validation (10%), and test (20%) data sets and used to train a ResNet18 deep learning architecture for binary classification of normal vs preplus or plus disease, which was then used for patient-level predictions of referral warranted (RW)- and treatment requiring (TR)-ROP.

Main Outcome and Measures

Sensitivity and specificity of detection of RW-ROP, and TR-ROP by both human graders and an AI system and area under the receiver operating characteristic curve (AUC) of grader-assigned VSS. Sensitivity and specificity were compared between the 2 SBFI systems using Pearson χ2testing.

Results

A total of 156 infants (312 eyes; mean [SD] gestational age, 33.0 [3.0] weeks; 75 [48%] female) were included with paired examinations. Sensitivity and specificity were not found to be statistically different between the 2 SBFI systems. Human graders were effective with SBFI at detecting TR-ROP with a sensitivity of 100% and specificity of 83.49%. The AUCs with grader-assigned VSS only were 0.95 (95% CI, 0.91-0.99) and 0.96 (95% CI, 0.93-0.99) for RW-ROP and TR-ROP, respectively. For the AI system, the sensitivity of detecting TR-ROP sensitivity was 100% with specificity of 58.6%, and RW-ROP sensitivity was 80.0% with specificity of 59.3%.

Conclusions and Relevance

In this cross-sectional study, 2 different SBFI systems used by technicians in an ROP screening program were highly sensitive for TR-ROP. SBFI systems with AI may be a cost-effective method to improve the global capacity for ROP screening.

Clinical evaluation of k-space correlation informed motion artifact detection in segmented multi-slice MRI

Motion artifacts can negatively impact diagnosis, patient experience, and radiology workflow especially when a patient recall is required. Detecting motion artifacts while the patient is still in the scanner could potentially improve workflow and reduce costs by enabling immediate corrective action. We demonstrate in a clinical k-space dataset that using cross-correlation between adjacent phase-encoding lines can detect motion artifacts directly from raw k-space in multi-shot multi-slice scans. We train a split-attention residual network to examine the performance in predicting motion artifact severity. The network is trained on simulated data and tested on real clinical data.

Association of Biomarker-Based Artificial Intelligence With Risk of Racial Bias in Retinal Images

Importance

Although race is a social construct, it is associated with variations in skin and retinal pigmentation. Image-based medical artificial intelligence (AI) algorithms that use images of these organs have the potential to learn features associated with self-reported race (SRR), which increases the risk of racially biased performance in diagnostic tasks; understanding whether this information can be removed, without affecting the performance of AI algorithms, is critical in reducing the risk of racial bias in medical AI.

Objective

To evaluate whether converting color fundus photographs to retinal vessel maps (RVMs) of infants screened for retinopathy of prematurity (ROP) removes the risk for racial bias.

Design, Setting, and Participants

The retinal fundus images (RFIs) of neonates with parent-reported Black or White race were collected for this study. A u-net, a convolutional neural network (CNN) that provides precise segmentation for biomedical images, was used to segment the major arteries and veins in RFIs into grayscale RVMs, which were subsequently thresholded, binarized, and/or skeletonized. CNNs were trained with patients' SRR labels on color RFIs, raw RVMs, and thresholded, binarized, or skeletonized RVMs. Study data were analyzed from July 1 to September 28, 2021.

Main Outcomes and Measures

Area under the precision-recall curve (AUC-PR) and area under the receiver operating characteristic curve (AUROC) at both the image and eye level for classification of SRR.

Results

A total of 4095 RFIs were collected from 245 neonates with parent-reported Black (94 [38.4%]; mean [SD] age, 27.2 [2.3] weeks; 55 majority sex [58.5%]) or White (151 [61.6%]; mean [SD] age, 27.6 [2.3] weeks, 80 majority sex [53.0%]) race. CNNs inferred SRR from RFIs nearly perfectly (image-level AUC-PR, 0.999; 95% CI, 0.999-1.000; infant-level AUC-PR, 1.000; 95% CI, 0.999-1.000). Raw RVMs were nearly as informative as color RFIs (image-level AUC-PR, 0.938; 95% CI, 0.926-0.950; infant-level AUC-PR, 0.995; 95% CI, 0.992-0.998). Ultimately, CNNs were able to learn whether RFIs or RVMs were from Black or White infants regardless of whether images contained color, vessel segmentation brightness differences were nullified, or vessel segmentation widths were uniform.

Conclusions and Relevance

Results of this diagnostic study suggest that it can be very challenging to remove information relevant to SRR from fundus photographs. As a result, AI algorithms trained on fundus photographs have the potential for biased performance in practice, even if based on biomarkers rather than raw images. Regardless of the methodology used for training AI, evaluating performance in relevant subpopulations is critical.

Artificial Intelligence and Glaucoma: Going Back to Basics

There has been a recent surge in the number of publications centered on the use of artificial intelligence (AI) to diagnose various systemic diseases. The Food and Drug Administration has approved several algorithms for use in clinical practice. In ophthalmology, most advances in AI relate to diabetic retinopathy, which is a disease process with agreed upon diagnostic and classification criteria. However, this is not the case for glaucoma, which is a relatively complex disease without agreed-upon diagnostic criteria. Moreover, currently available public datasets that focus on glaucoma have inconstant label quality, further complicating attempts at training AI algorithms efficiently. In this perspective paper, we discuss specific details related to developing AI models for glaucoma and suggest potential steps to overcome current limitations.

An interpretable machine learning model for the prevention of contrast-induced nephropathy in patients undergoing lower extremity endovascular interventions for peripheral arterial disease

BACKGROUND

Contrast-induced nephropathy (CIN) is a postprocedural complication associated with increased morbidity and mortality. An important risk factor for development of CIN is renal impairment. Identification of patients at risk for acute renal failure will allow physicians to make appropriate decisions to minimize the incidence of CIN. We developed a machine learning model to stratify risk of acute renal failure that may assist in mitigating risk for CIN in patients with peripheral artery disease (PAD) undergoing endovascular interventions.

METHODS

We utilized the American College of Surgeons National Surgical Quality Improvement Program database to extract clinical and laboratory information associated with 14,444 patients who underwent lower extremity endovascular procedures between 2011 and 2018. Using 11,604 cases from 2011 to 2017 for training and 2840 cases from 2018 for testing, we developed a random forest model to predict risk of 30-day acute renal failure following infra-inguinal endovascular procedures.

RESULTS

Eight variables were identified as contributing optimally to model predictions, the most important being diabetes, preoperative BUN, and claudication. Using these variables, the model achieved an area under the receiver-operating characteristic (AU-ROC) curve of 0.81, accuracy of 0.83, sensitivity of 0.67, and specificity of 0.74. The model performed equally well on white and nonwhite patients (Delong p-value = 0.955) and patients age < 65 and patients age ≥ 65 (Delong p-value = 0.659).

CONCLUSIONS

We develop a model that fairly and accurately stratifies 30-day acute renal failure risk in patients undergoing lower extremity endovascular procedures for PAD. This model may assist in identifying patients who may benefit from strategies to prevent CIN.

Use of risk-based cervical screening programs in resource-limited settings

Cervical cancer screening and management in the U.S. has adopted a risk-based approach. However, the majority of cervical cancer cases and deaths occur in resource-limited settings, where screening and management are not widely available. We describe a conceptual model that optimizes cervical cancer screening and management in resource-limited settings by utilizing a risk-based approach. The principles of risk-based screening and management in resource limited settings include (1) ensure that the screening method effectively separates low-risk from high-risk patients; (2) directing resources to populations at the highest cancer risk; (3) screen using HPV testing via self-sampling; (4) utilize HPV genotyping to improve risk stratification and better determine who will benefit from treatment, and (5) automated visual evaluation with artificial intelligence may further improve risk stratification. Risk-based screening and management in resource limited settings can optimize prevention by focusing triage and treatment resources on the highest risk patients while minimizing interventions in lower risk patients.

Hematopoietic stem-cell gene therapy is associated with restored white matter microvascular function in cerebral adrenoleukodystrophy

Blood-brain barrier disruption marks the onset of cerebral adrenoleukodystrophy (CALD), a devastating cerebral demyelinating disease caused by loss of ABCD1 gene function. The underlying mechanism are not well understood, but evidence suggests that microvascular dysfunction is involved. We analyzed cerebral perfusion imaging in boys with CALD treated with autologous hematopoietic stem-cells transduced with the Lenti-D lentiviral vector that contains ABCD1 cDNA as part of a single group, open-label phase 2-3 safety and efficacy study (NCT01896102) and patients treated with allogeneic hematopoietic stem cell transplantation. We found widespread and sustained normalization of white matter permeability and microvascular flow. We demonstrate that ABCD1 functional bone marrow-derived cells can engraft in the cerebral vascular and perivascular space. Inverse correlation between gene dosage and lesion growth suggests that corrected cells contribute long-term to remodeling of brain microvascular function. Further studies are needed to explore the longevity of these effects.

Inconsistent Partitioning and Unproductive Feature Associations Yield Idealized Radiomic Models

Background Radiomics is the extraction of predefined mathematic features from medical images for the prediction of variables of clinical interest. While some studies report superlative accuracy of radiomic machine learning (ML) models, the published methodology is often incomplete, and the results are rarely validated in external testing data sets. Purpose To characterize the type, prevalence, and statistical impact of methodologic errors present in radiomic ML studies. Materials and Methods Radiomic ML publications were reviewed for the presence of performance-inflating methodologic flaws. Common flaws were subsequently reproduced with randomly generated features interpolated from publicly available radiomic data sets to demonstrate the precarious nature of reported findings. Results In an assessment of radiomic ML publications, the authors uncovered two general categories of data analysis errors: inconsistent partitioning and unproductive feature associations. In simulations, the authors demonstrated that inconsistent partitioning augments radiomic ML accuracy by 1.4 times from unbiased performance and that correcting for flawed methodologic results in areas under the receiver operating characteristic curve approaching a value of 0.5 (random chance). With use of randomly generated features, the authors illustrated that unproductive associations between radiomic features and gene sets can imply false causality for biologic phenomenon. Conclusion Radiomic machine learning studies may contain methodologic flaws that undermine their validity. This study provides a review template to avoid such flaws. © RSNA, 2022 Supplemental material is available for this article. See also the editorial by Jacobs in this issue.

Collaborative Privacy-preserving Approaches for Distributed Deep Learning Using Multi-Institutional Data

Deep learning (DL) algorithms have shown remarkable potential in automating various tasks in medical imaging and radiologic reporting. However, models trained on low quantities of data or only using data from a single institution often are not generalizable to other institutions, which may have different patient demographics or data acquisition characteristics. Therefore, training DL algorithms using data from multiple institutions is crucial to improving the robustness and generalizability of clinically useful DL models. In the context of medical data, simply pooling data from each institution to a central location to train a model poses several issues such as increased risk to patient privacy, increased costs for data storage and transfer, and regulatory challenges. These challenges of centrally hosting data have motivated the development of distributed machine learning techniques and frameworks for collaborative learning that facilitate the training of DL models without the need to explicitly share private medical data. The authors describe several popular methods for collaborative training and review the main considerations for deploying these models. They also highlight publicly available software frameworks for federated learning and showcase several real-world examples of collaborative learning. The authors conclude by discussing some key challenges and future research directions for distributed DL. They aim to introduce clinicians to the benefits, limitations, and risks of using distributed DL for the development of medical artificial intelligence algorithms. ©RSNA, 2023 Quiz questions for this article are available in the supplemental material.

REPRODUCIBLE AND CLINICALLY TRANSLATABLE DEEP NEURAL NETWORKS FOR CANCER SCREENING

Cervical cancer is a leading cause of cancer mortality, with approximately 90% of the 250,000 deaths per year occurring in low- and middle-income countries (LMIC). Secondary prevention with cervical screening involves detecting and treating precursor lesions; however, scaling screening efforts in LMIC has been hampered by infrastructure and cost constraints. Recent work has supported the development of an artificial intelligence (AI) pipeline on digital images of the cervix to achieve an accurate and reliable diagnosis of treatable precancerous lesions. In particular, WHO guidelines emphasize visual triage of women testing positive for human papillomavirus (HPV) as the primary screen, and AI could assist in this triage task. Published AI reports have exhibited overfitting, lack of portability, and unrealistic, near-perfect performance estimates. To surmount recognized issues, we implemented a comprehensive deep-learning model selection and optimization study on a large, collated, multi-institutional dataset of 9,462 women (17,013 images). We evaluated relative portability, repeatability, and classification performance. The top performing model, when combined with HPV type, achieved an area under the Receiver Operating Characteristics (ROC) curve (AUC) of 0.89 within our study population of interest, and a limited total extreme misclassification rate of 3.4%, on held-aside test sets. Our work is among the first efforts at designing a robust, repeatable, accurate and clinically translatable deep-learning model for cervical screening.

Machine Learning for Adrenal Gland Segmentation and Classification of Normal and Adrenal Masses at CT

Background Adrenal masses are common, but radiology reporting and recommendations for management can be variable. Purpose To create a machine learning algorithm to segment adrenal glands on contrast-enhanced CT images and classify glands as normal or mass-containing and to assess algorithm performance. Materials and Methods This retrospective study included two groups of contrast-enhanced abdominal CT examinations (development data set and secondary test set). Adrenal glands in the development data set were manually segmented by radiologists. Images in both the development data set and the secondary test set were manually classified as normal or mass-containing. Deep learning segmentation and classification models were trained on the development data set and evaluated on both data sets. Segmentation performance was evaluated with use of the Dice similarity coefficient (DSC), and classification performance with use of sensitivity and specificity. Results The development data set contained 274 CT examinations (251 patients; median age, 61 years; 133 women), and the secondary test set contained 991 CT examinations (991 patients; median age, 62 years; 578 women). The median model DSC on the development test set was 0.80 (IQR, 0.78-0.89) for normal glands and 0.84 (IQR, 0.79-0.90) for adrenal masses. On the development reader set, the median interreader DSC was 0.89 (IQR, 0.78-0.93) for normal glands and 0.89 (IQR, 0.85-0.97) for adrenal masses. Interreader DSC for radiologist manual segmentation did not differ from automated machine segmentation (P = .35). On the development test set, the model had a classification sensitivity of 83% (95% CI: 55, 95) and specificity of 89% (95% CI: 75, 96). On the secondary test set, the model had a classification sensitivity of 69% (95% CI: 58, 79) and specificity of 91% (95% CI: 90, 92). Conclusion A two-stage machine learning pipeline was able to segment the adrenal glands and differentiate normal adrenal glands from those containing masses. © RSNA, 2022 Online supplemental material is available for this article.

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis

IMPORTANCE

An accurate and robust artificial intelligence (AI) algorithm for detecting cancer in digital breast tomosynthesis (DBT) could significantly improve detection accuracy and reduce health care costs worldwide.

OBJECTIVES

To make training and evaluation data for the development of AI algorithms for DBT analysis available, to develop well-defined benchmarks, and to create publicly available code for existing methods.

DESIGN, SETTING, AND PARTICIPANTS

This diagnostic study is based on a multi-institutional international grand challenge in which research teams developed algorithms to detect lesions in DBT. A data set of 22 032 reconstructed DBT volumes was made available to research teams. Phase 1, in which teams were provided 700 scans from the training set, 120 from the validation set, and 180 from the test set, took place from December 2020 to January 2021, and phase 2, in which teams were given the full data set, took place from May to July 2021.

MAIN OUTCOMES AND MEASURES

The overall performance was evaluated by mean sensitivity for biopsied lesions using only DBT volumes with biopsied lesions; ties were broken by including all DBT volumes.

RESULTS

A total of 8 teams participated in the challenge. The team with the highest mean sensitivity for biopsied lesions was the NYU B-Team, with 0.957 (95% CI, 0.924-0.984), and the second-place team, ZeDuS, had a mean sensitivity of 0.926 (95% CI, 0.881-0.964). When the results were aggregated, the mean sensitivity for all submitted algorithms was 0.879; for only those who participated in phase 2, it was 0.926.

CONCLUSIONS AND RELEVANCE

In this diagnostic study, an international competition produced algorithms with high sensitivity for using AI to detect lesions on DBT images. A standardized performance benchmark for the detection task using publicly available clinical imaging data was released, with detailed descriptions and analyses of submitted algorithms accompanied by a public release of their predictions and code for selected methods. These resources will serve as a foundation for future research on computer-assisted diagnosis methods for DBT, significantly lowering the barrier of entry for new researchers.

Assessment of Retinopathy of Prematurity Regression and Reactivation Using an Artificial Intelligence-Based Vascular Severity Score

IMPORTANCE

One of the biggest challenges when using anti-vascular endothelial growth factor (VEGF) agents to treat retinopathy of prematurity (ROP) is the need to perform long-term follow-up examinations to identify eyes at risk of ROP reactivation requiring retreatment.

OBJECTIVE

To evaluate whether an artificial intelligence (AI)-based vascular severity score (VSS) can be used to analyze ROP regression and reactivation after anti-VEGF treatment and potentially identify eyes at risk of ROP reactivation requiring retreatment.

DESIGN, SETTING, AND PARTICIPANTS

This prognostic study was a secondary analysis of posterior pole fundus images collected during the multicenter, double-blind, investigator-initiated Comparing Alternative Ranibizumab Dosages for Safety and Efficacy in Retinopathy of Prematurity (CARE-ROP) randomized clinical trial, which compared 2 different doses of ranibizumab (0.12 mg vs 0.20 mg) for the treatment of ROP. The CARE-ROP trial screened and enrolled infants between September 5, 2014, and July 14, 2016. A total of 1046 wide-angle fundus images obtained from 19 infants at predefined study time points were analyzed. The analyses of VSS were performed between January 20, 2021, and November 18, 2022.

INTERVENTIONS

An AI-based algorithm assigned a VSS between 1 (normal) and 9 (most severe) to fundus images.

MAIN OUTCOMES AND MEASURES

Analysis of VSS in infants with ROP over time and VSS comparisons between the 2 treatment groups (0.12 mg vs 0.20 mg of ranibizumab) and between infants who did and did not receive retreatment for ROP reactivation.

RESULTS

Among 19 infants with ROP in the CARE-ROP randomized clinical trial, the median (range) postmenstrual age at first treatment was 36.4 (34.7-39.7) weeks; 10 infants (52.6%) were male, and 18 (94.7%) were White. The mean (SD) VSS was 6.7 (1.9) at baseline and significantly decreased to 2.7 (1.9) at week 1 (P < .001) and 2.9 (1.3) at week 4 (P < .001). The mean (SD) VSS of infants with ROP reactivation requiring retreatment was 6.5 (1.9) at the time of retreatment, which was significantly higher than the VSS at week 4 (P < .001). No significant difference was found in VSS between the 2 treatment groups, but the change in VSS between baseline and week 1 was higher for infants who later required retreatment (mean [SD], 7.8 [1.3] at baseline vs 1.7 [0.7] at week 1) vs infants who did not (mean [SD], 6.4 [1.9] at baseline vs 3.0 [2.0] at week 1). In eyes requiring retreatment, higher baseline VSS was correlated with earlier time of retreatment (Pearson r = -0.9997; P < .001).

CONCLUSIONS AND RELEVANCE

In this study, VSS decreased after ranibizumab treatment, consistent with clinical disease regression. In cases of ROP reactivation requiring retreatment, VSS increased again to values comparable with baseline values. In addition, a greater change in VSS during the first week after initial treatment was found to be associated with a higher risk of later ROP reactivation, and high baseline VSS was correlated with earlier retreatment. These findings may have implications for monitoring ROP regression and reactivation after anti-VEGF treatment.

Operationalization of Retinopathy of Prematurity Screening by the Application of the Essential Public Health Services Framework

Retinopathy of prematurity (ROP) is one of the leading causes of preventable pediatric blindness worldwide. ROP screening programs have been previously implemented in multiple low- and middle-income countries. On a global scale, it is crucial that evidence-based, standardized screening criteria are utilized in the early detection and treatment of ROP. In this review article, we utilize the National Public Health Performance Standards (NPHPS) Ten Essential Public Health Services Model organized by the core functions of assessment, policy development, and assurance to evaluate the barriers and successes of existing ROP screening programs. This framework can be applied to countries facing the third epidemic of ROP and can be used to establish a generalized model for eye care and screening worldwide.

Evaluation of an Artificial Intelligence System for Retinopathy of Prematurity Screening in Nepal and Mongolia

PURPOSE

To evaluate the performance of a deep learning (DL) algorithm for retinopathy of prematurity (ROP) screening in Nepal and Mongolia.

DESIGN

Retrospective analysis of prospectively collected clinical data.

PARTICIPANTS

Clinical information and fundus images were obtained from infants in 2 ROP screening programs in Nepal and Mongolia.

METHODS

Fundus images were obtained using the Forus 3nethra neo (Forus Health) in Nepal and the RetCam Portable (Natus Medical, Inc.) in Mongolia. The overall severity of ROP was determined from the medical record using the International Classification of ROP (ICROP). The presence of plus disease was determined independently in each image using a reference standard diagnosis. The Imaging and Informatics for ROP (i-ROP) DL algorithm was trained on images from the RetCam to classify plus disease and to assign a vascular severity score (VSS) from 1 through 9.

MAIN OUTCOME MEASURES

Area under the receiver operating characteristic curve and area under the precision-recall curve for the presence of plus disease or type 1 ROP and association between VSS and ICROP disease category.

RESULTS

The prevalence of type 1 ROP was found to be higher in Mongolia (14.0%) than in Nepal (2.2%; P < 0.001) in these data sets. In Mongolia (RetCam images), the area under the receiver operating characteristic curve for examination-level plus disease detection was 0.968, and the area under the precision-recall curve was 0.823. In Nepal (Forus images), these values were 0.999 and 0.993, respectively. The ROP VSS was associated with ICROP classification in both datasets (P < 0.001). At the population level, the median VSS was found to be higher in Mongolia (2.7; interquartile range [IQR], 1.3-5.4]) as compared with Nepal (1.9; IQR, 1.2-3.4; P < 0.001).

CONCLUSIONS

These data provide preliminary evidence of the effectiveness of the i-ROP DL algorithm for ROP screening in neonatal populations in Nepal and Mongolia using multiple camera systems and are useful for consideration in future clinical implementation of artificial intelligence-based ROP screening in low- and middle-income countries.

Variability in Plus Disease Diagnosis using Single and Serial Images

PURPOSE

To assess changes in retinopathy of prematurity (ROP) diagnosis in single and serial retinal images.

DESIGN

Cohort study.

PARTICIPANTS

Cases of ROP recruited from the Imaging and Informatics in Retinopathy of Prematurity (i-ROP) consortium evaluated by 7 graders.

METHODS

Seven ophthalmologists reviewed both single and 3 consecutive serial retinal images from 15 cases with ROP, and severity was assigned as plus, preplus, or none. Imaging data were acquired during routine ROP screening from 2011 to 2015, and a reference standard diagnosis was established for each image. A secondary analysis was performed using the i-ROP deep learning system to assign a vascular severity score (VSS) to each image, ranging from 1 to 9, with 9 being the most severe disease. This score has been previously demonstrated to correlate with the International Classification of ROP. Mean plus disease severity was calculated by averaging 14 labels per image in serial and single images to decrease noise.

MAIN OUTCOME MEASURES

Grading severity of ROP as defined by plus, preplus, or no ROP.

RESULTS

Assessment of serial retinal images changed the grading severity for > 50% of the graders, although there was wide variability. Cohen's kappa ranged from 0.29 to 1.0, which showed a wide range of agreement from slight to perfect by each grader. Changes in the grading of serial retinal images were noted more commonly in cases of preplus disease. The mean severity in cases with a diagnosis of plus disease and no disease did not change between single and serial images. The ROP VSS demonstrated good correlation with the range of expert classifications of plus disease and overall agreement with the mode class (P = 0.001). The VSS correlated with mean plus disease severity by expert diagnosis (correlation coefficient, 0.89). The more aggressive graders tended to be influenced by serial images to increase the severity of their grading. The VSS also demonstrated agreement with disease progression across serial images, which progressed to preplus and plus disease.

CONCLUSIONS

Clinicians demonstrated variability in ROP diagnosis when presented with both single and serial images. The use of deep learning as a quantitative assessment of plus disease has the potential to standardize ROP diagnosis and treatment.

Risk stratification with explainable machine learning for 30-day procedure-related mortality and 30-day unplanned readmission in patients with peripheral arterial disease

Predicting 30-day procedure-related mortality risk and 30-day unplanned readmission in patients undergoing lower extremity endovascular interventions for peripheral artery disease (PAD) may assist in improving patient outcomes. Risk prediction of 30-day mortality can help clinicians identify treatment plans to reduce the risk of death, and prediction of 30-day unplanned readmission may improve outcomes by identifying patients who may benefit from readmission prevention strategies. The goal of this study is to develop machine learning models to stratify risk of 30-day procedure-related mortality and 30-day unplanned readmission in patients undergoing lower extremity infra-inguinal endovascular interventions. We used a cohort of 14,444 cases from the American College of Surgeons National Surgical Quality Improvement Program database. For each outcome, we developed and evaluated multiple machine learning models, including Support Vector Machines, Multilayer Perceptrons, and Gradient Boosting Machines, and selected a random forest as the best-performing model for both outcomes. Our 30-day procedure-related mortality model achieved an AUC of 0.75 (95% CI: 0.71-0.79) and our 30-day unplanned readmission model achieved an AUC of 0.68 (95% CI: 0.67-0.71). Stratification of the test set by race (white and non-white), sex (male and female), and age (≥65 years and <65 years) and subsequent evaluation of demographic parity by AUC shows that both models perform equally well across race, sex, and age groups. We interpret the model globally and locally using Gini impurity and SHapley Additive exPlanations (SHAP). Using the top five predictors for death and mortality, we demonstrate differences in survival for subgroups stratified by these predictors, which underscores the utility of our model.

Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement

In response to the COVID-19 global pandemic, recent research has proposed creating deep learning based models that use chest radiographs (CXRs) in a variety of clinical tasks to help manage the crisis. However, the size of existing datasets of CXRs from COVID-19+ patients are relatively small, and researchers often pool CXR data from multiple sources, for example, using different x-ray machines in various patient populations under different clinical scenarios. Deep learning models trained on such datasets have been shown to overfit to erroneous features instead of learning pulmonary characteristics in a phenomenon known as shortcut learning. We propose adding feature disentanglement to the training process. This technique forces the models to identify pulmonary features from the images and penalizes them for learning features that can discriminate between the original datasets that the images come from. We find that models trained in this way indeed have better generalization performance on unseen data; in the best case we found that it improved AUC by 0.13 on held out data. We further find that this outperforms masking out non-lung parts of the CXRs and performing histogram equalization, both of which are recently proposed methods for removing biases in CXR datasets.

ACR’s Connect and AI-LAB technical framework

Objective

To develop a free, vendor-neutral software suite, the American College of Radiology (ACR) Connect, which serves as a platform for democratizing artificial intelligence (AI) for all individuals and institutions.

Materials and Methods

Among its core capabilities, ACR Connect provides educational resources; tools for dataset annotation; model building and evaluation; and an interface for collaboration and federated learning across institutions without the need to move data off hospital premises.

Results

The AI-LAB application within ACR Connect allows users to investigate AI models using their own local data while maintaining data security. The software enables non-technical users to participate in the evaluation and training of AI models as part of a larger, collaborative network.

Discussion

Advancements in AI have transformed automated quantitative analysis for medical imaging. Despite the significant progress in research, AI is currently underutilized in current clinical workflows. The success of AI model development depends critically on the synergy between physicians who can drive clinical direction, data scientists who can design effective algorithms, and the availability of high-quality datasets. ACR Connect and AI-LAB provide a way to perform external validation as well as collaborative, distributed training.

Conclusion

In order to create a collaborative AI ecosystem across clinical and technical domains, the ACR developed a platform that enables non-technical users to participate in education and model development.

Towards More Efficient Data Valuation in Healthcare Federated Learning using Ensembling

Federated Learning (FL) wherein multiple institutions collaboratively train a machine learning model without sharing data is becoming popular. Participating institutions might not contribute equally - some contribute more data, some better quality data or some more diverse data. To fairly rank the contribution of different institutions, Shapley value (SV) has emerged as the method of choice. Exact SV computation is impossibly expensive, especially when there are hundreds of contributors. Existing SV computation techniques use approximations. However, in healthcare where the number of contributing institutions are likely not of a colossal scale, computing exact SVs is still exorbitantly expensive, but not impossible. For such settings, we propose an efficient SV computation technique called SaFE (Shapley Value for Federated Learning using Ensembling). We empirically show that SaFE computes values that are close to exact SVs, and that it performs better than current SV approximations. This is particularly relevant in medical imaging setting where widespread heterogeneity across institutions is rampant and fast accurate data valuation is required to determine the contribution of each participant in multi-institutional collaborative learning.

SplitAVG: A Heterogeneity-Aware Federated Deep Learning Method for Medical Imaging

Federated learning is an emerging research paradigm for enabling collaboratively training deep learning models without sharing patient data. However, the data from different institutions are usually heterogeneous across institutions, which may reduce the performance of models trained using federated learning. In this study, we propose a novel heterogeneity-aware federated learning method, SplitAVG, to overcome the performance drops from data heterogeneity in federated learning. Unlike previous federated methods that require complex heuristic training or hyper parameter tuning, our SplitAVG leverages the simple network split and feature map concatenation strategies to encourage the federated model training an unbiased estimator of the target data distribution. We compare SplitAVG with seven state-of-the-art federated learning methods, using centrally hosted training data as the baseline on a suite of both synthetic and real-world federated datasets. We find that the performance of models trained using all the comparison federated learning methods degraded significantly with the increasing degrees of data heterogeneity. In contrast, SplitAVG method achieves comparable results to the baseline method under all heterogeneous settings, that it achieves 96.2% of the accuracy and 110.4% of the mean absolute error obtained by the baseline in a diabetic retinopathy binary classification dataset and a bone age prediction dataset, respectively, on highly heterogeneous data partitions. We conclude that SplitAVG method can effectively overcome the performance drops from variability in data distributions across institutions. Experimental results also show that SplitAVG can be adapted to different base convolutional neural networks (CNNs) and generalized to various types of medical imaging tasks. The code is publicly available at https://github.com/zm17943/SplitAVG.

QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results

Deep learning (DL) models have provided state-of-the-art performance in various medical imaging benchmarking challenges, including the Brain Tumor Segmentation (BraTS) challenges. However, the task of focal pathology multi-compartment segmentation (e.g., tumor and lesion sub-regions) is particularly challenging, and potential errors hinder translating DL models into clinical workflows. Quantifying the reliability of DL model predictions in the form of uncertainties could enable clinical review of the most uncertain regions, thereby building trust and paving the way toward clinical translation. Several uncertainty estimation methods have recently been introduced for DL medical image segmentation tasks. Developing scores to evaluate and compare the performance of uncertainty measures will assist the end-user in making more informed decisions. In this study, we explore and evaluate a score developed during the BraTS 2019 and BraTS 2020 task on uncertainty quantification (QU-BraTS) and designed to assess and rank uncertainty estimates for brain tumor multi-compartment segmentation. This score (1) rewards uncertainty estimates that produce high confidence in correct assertions and those that assign low confidence levels at incorrect assertions, and (2) penalizes uncertainty measures that lead to a higher percentage of under-confident correct assertions. We further benchmark the segmentation uncertainties generated by 14 independent participating teams of QU-BraTS 2020, all of which also participated in the main BraTS segmentation task. Overall, our findings confirm the importance and complementary value that uncertainty estimates provide to segmentation algorithms, highlighting the need for uncertainty quantification in medical image analyses. Finally, in favor of transparency and reproducibility, our evaluation code is made publicly available at https://github.com/RagMeh11/QU-BraTS.

Multi-population generalizability of a deep learning-based chest radiograph severity score for COVID-19

To tune and test the generalizability of a deep learning-based model for assessment of COVID-19 lung disease severity on chest radiographs (CXRs) from different patient populations. A published convolutional Siamese neural network-based model previously trained on hospitalized patients with COVID-19 was tuned using 250 outpatient CXRs. This model produces a quantitative measure of COVID-19 lung disease severity (pulmonary x-ray severity (PXS) score). The model was evaluated on CXRs from 4 test sets, including 3 from the United States (patients hospitalized at an academic medical center (N = 154), patients hospitalized at a community hospital (N = 113), and outpatients (N = 108)) and 1 from Brazil (patients at an academic medical center emergency department (N = 303)). Radiologists from both countries independently assigned reference standard CXR severity scores, which were correlated with the PXS scores as a measure of model performance (Pearson R). The Uniform Manifold Approximation and Projection (UMAP) technique was used to visualize the neural network results. Tuning the deep learning model with outpatient data showed high model performance in 2 United States hospitalized patient datasets (R = 0.88 and R = 0.90, compared to baseline R = 0.86). Model performance was similar, though slightly lower, when tested on the United States outpatient and Brazil emergency department datasets (R = 0.86 and R = 0.85, respectively). UMAP showed that the model learned disease severity information that generalized across test sets. A deep learning model that extracts a COVID-19 severity score on CXRs showed generalizable performance across multiple populations from 2 continents, including outpatients and hospitalized patients.

External Validation of a Retinopathy of Prematurity Screening Model Using Artificial Intelligence in 3 Low- and Middle-Income Populations

Importance

Retinopathy of prematurity (ROP) is a leading cause of preventable blindness that disproportionately affects children born in low- and middle-income countries (LMICs). In-person and telemedical screening examinations can reduce this risk but are challenging to implement in LMICs owing to the multitude of at-risk infants and lack of trained ophthalmologists.

Objective

To implement an ROP risk model using retinal images from a single baseline examination to identify infants who will develop treatment-requiring (TR)-ROP in LMIC telemedicine programs.

Design, Setting, and Participants

In this diagnostic study conducted from February 1, 2019, to June 30, 2021, retinal fundus images were collected from infants as part of an Indian ROP telemedicine screening program. An artificial intelligence (AI)-derived vascular severity score (VSS) was obtained from images from the first examination after 30 weeks' postmenstrual age. Using 5-fold cross-validation, logistic regression models were trained on 2 variables (gestational age and VSS) for prediction of TR-ROP. The model was externally validated on test data sets from India, Nepal, and Mongolia. Data were analyzed from October 20, 2021, to April 20, 2022.

Main Outcomes and Measures

Primary outcome measures included sensitivity, specificity, positive predictive value, and negative predictive value for predictions of future occurrences of TR-ROP; the number of weeks before clinical diagnosis when a prediction was made; and the potential reduction in number of examinations required.

Results

A total of 3760 infants (median [IQR] postmenstrual age, 37 [5] weeks; 1950 male infants [51.9%]) were included in the study. The diagnostic model had a sensitivity and specificity, respectively, for each of the data sets as follows: India, 100.0% (95% CI, 87.2%-100.0%) and 63.3% (95% CI, 59.7%-66.8%); Nepal, 100.0% (95% CI, 54.1%-100.0%) and 77.8% (95% CI, 72.9%-82.2%); and Mongolia, 100.0% (95% CI, 93.3%-100.0%) and 45.8% (95% CI, 39.7%-52.1%). With the AI model, infants with TR-ROP were identified a median (IQR) of 2.0 (0-11) weeks before TR-ROP diagnosis in India, 0.5 (0-2.0) weeks before TR-ROP diagnosis in Nepal, and 0 (0-5.0) weeks before TR-ROP diagnosis in Mongolia. If low-risk infants were never screened again, the population could be effectively screened with 45.0% (India, 664/1476), 38.4% (Nepal, 151/393), and 51.3% (Mongolia, 266/519) fewer examinations required.

Conclusions and Relevance

Results of this diagnostic study suggest that there were 2 advantages to implementation of this risk model: (1) the number of examinations for low-risk infants could be reduced without missing cases of TR-ROP, and (2) high-risk infants could be identified and closely monitored before development of TR-ROP.