Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients

Comparison of 21 artificial intelligence algorithms in automated diabetic retinopathy screening using handheld fundus camera

BACKGROUND

Diabetic retinopathy (DR) is a common complication of diabetes and may lead to irreversible visual loss. Efficient screening and improved treatment of both diabetes and DR have amended visual prognosis for DR. The number of patients with diabetes is increasing and telemedicine, mobile handheld devices and automated solutions may alleviate the burden for healthcare. We compared the performance of 21 artificial intelligence (AI) algorithms for referable DR screening in datasets taken by handheld Optomed Aurora fundus camera in a real-world setting.

PATIENTS AND METHODS

Prospective study of 156 patients (312 eyes) attending DR screening and follow-up. Both papilla- and macula-centred 50° fundus images were taken from each eye. DR was graded by experienced ophthalmologists and 21 AI algorithms.

RESULTS

Most eyes, 183 out of 312 (58.7%), had no DR and mild NPDR was noted in 21 (6.7%) of the eyes. Moderate NPDR was detected in 66 (21.2%) of the eyes, severe NPDR in 1 (0.3%), and PDR in 41 (13.1%) composing a group of 34.6% of eyes with referable DR. The AI algorithms achieved a mean agreement of 79.4% for referable DR, but the results varied from 49.4% to 92.3%. The mean sensitivity for referable DR was 77.5% (95% CI 69.1-85.8) and specificity 80.6% (95% CI 72.1-89.2). The rate for images ungradable by AI varied from 0% to 28.2% (mean 1.9%). Nineteen out of 21 (90.5%) AI algorithms resulted in grading for DR at least in 98% of the images.

CONCLUSIONS

Fundus images captured with Optomed Aurora were suitable for DR screening. The performance of the AI algorithms varied considerably emphasizing the need for external validation of screening algorithms in real-world settings before their clinical application.

Diagnostic Accuracy of Artificial Intelligence-Based Automated Diabetic Retinopathy Screening in Real-World Settings: A Systematic Review and Meta-Analysis

PURPOSE

To evaluate the diagnostic accuracy of artificial intelligence (AI)-based automated diabetic retinopathy (DR) screening in real-world settings.

DESIGN

Systematic review and meta-analysis METHODS: We conducted a systematic review of relevant literature from January 2012 to August 2022 using databases including PubMed, Scopus and Web of Science. The quality of studies was evaluated using Quality Assessment for Diagnostic Accuracy Studies 2 (QUADAS-2) checklist. We calculated pooled accuracy, sensitivity, specificity, and diagnostic odds ratio (DOR) as summary measures. The study protocol was registered with the International Prospective Register of Systematic Reviews (PROSPERO - CRD42022367034).

RESULTS

We included 34 studies which utilized AI algorithms for diagnosing DR based on real-world fundus images. Quality assessment of these studies indicated a low risk of bias and low applicability concern. Among gradable images, the overall pooled accuracy, sensitivity, specificity, and DOR were 81%, 94% (95% CI: 92.0-96.0), 89% (95% CI: 85.0-92.0) and 128 (95% CI: 80-204) respectively. Sub-group analysis showed that, when acceptable quality imaging could be obtained, non-mydriatic fundus images had a better DOR of 143 (95% CI: 82-251) and studies using 2 field images had a better DOR of 161 (95% CI 74-347). Our meta-regression analysis revealed a statistically significant association between DOR and variables such as the income status, and the type of fundus camera.

CONCLUSION

Our findings indicate that AI algorithms have acceptable performance in screening for DR using fundus images compared to human graders. Implementing a fundus camera with AI-based software has the potential to assist ophthalmologists in reducing their workload and improving the accuracy of DR diagnosis.

Transparency in Artificial Intelligence Reporting in Ophthalmology-A Scoping Review

Topic

This scoping review summarizes artificial intelligence (AI) reporting in ophthalmology literature in respect to model development and validation. We characterize the state of transparency in reporting of studies prospectively validating models for disease classification.

Clinical Relevance

Understanding what elements authors currently describe regarding their AI models may aid in the future standardization of reporting. This review highlights the need for transparency to facilitate the critical appraisal of models prior to clinical implementation, to minimize bias and inappropriate use. Transparent reporting can improve effective and equitable use in clinical settings.

Methods

Eligible articles (as of January 2022) from PubMed, Embase, Web of Science, and CINAHL were independently screened by 2 reviewers. All observational and clinical trial studies evaluating the performance of an AI model for disease classification of ophthalmic conditions were included. Studies were evaluated for reporting of parameters derived from reporting guidelines (CONSORT-AI, MI-CLAIM) and our previously published editorial on model cards. The reporting of these factors, which included basic model and dataset details (source, demographics), and prospective validation outcomes, were summarized.

Results

Thirty-seven prospective validation studies were included in the scoping review. Eleven additional associated training and/or retrospective validation studies were included if this information could not be determined from the primary articles. These 37 studies validated 27 unique AI models; multiple studies evaluated the same algorithms (EyeArt, IDx-DR, and Medios AI). Details of model development were variably reported; 18 of 27 models described training dataset annotation and 10 of 27 studies reported training data distribution. Demographic information of training data was rarely reported; 7 of the 27 unique models reported age and gender and only 2 reported race and/or ethnicity. At the level of prospective clinical validation, age and gender of populations was more consistently reported (29 and 28 of 37 studies, respectively), but only 9 studies reported race and/or ethnicity data. Scope of use was difficult to discern for the majority of models. Fifteen studies did not state or imply primary users.

Conclusion

Our scoping review demonstrates variable reporting of information related to both model development and validation. The intention of our study was not to assess the quality of the factors we examined, but to characterize what information is, and is not, regularly reported. Our results suggest the need for greater transparency in the reporting of information necessary to determine the appropriateness and fairness of these tools prior to clinical use.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Rate and Predictors of Misclassification of Active Diabetic Macular Edema as Detected by an Automated Retinal Image Analysis System

INTRODUCTION

The aim of this work is to estimate the sensitivity, specificity, and misclassification rate of an automated retinal image analysis system (ARIAS) in diagnosing active diabetic macular edema (DME) and to identify factors associated with true and false positives.

METHODS

We conducted a cross-sectional study of prospectively enrolled patients with diabetes mellitus (DM) referred to a tertiary medical retina center for screening or management of DME. All patients underwent two-field fundus photography (macula- and disc-centered) with a true-color confocal camera; images were processed by EyeArt V.2.1.0 (Woodland Hills, CA, USA). Active DME was defined as the presence of intraretinal or subretinal fluid on spectral-domain optical coherence tomography (SD-OCT). Sensitivity and specificity and their 95% confidence intervals (CIs) were calculated. Variables associated with true (i.e., DME labeled as present by ARIAS + fluid on SD-OCT) and false positives (i.e., DME labeled as present by ARIAS + no fluid on SD-OCT) of active DME were explored.

RESULTS

A total of 298 eyes were included; 92 eyes (31%) had active DME. ARIAS sensitivity and specificity were 82.61% (95% CI 72.37-89.60) and 84.47% (95% CI 78.34-89.10). The misclassification rate was 16%. Factors associated with true positives included younger age (p = 0.01), shorter DM duration (p = 0.006), presence of hard exudates (p = 0.005), and microaneurysms (p = 0.002). Factors associated with false positives included longer DM duration (p = 0.01), worse diabetic retinopathy severity (p = 0.008), history of inactivated DME (p < 0.001), and presence of hard exudates (p < 0.001), microaneurysms (p < 0.001), or epiretinal membrane (p = 0.06).

CONCLUSIONS

The sensitivity of ARIAS was diminished in older patients and those without DME-related fundus lesions, while the specificity was reduced in cases with a history of inactivated DME. ARIAS performed well in screening for naïve DME but is not effective in surveillance inactivated DME.

A pilot cost-analysis study comparing AI-based EyeArt® and ophthalmologist assessment of diabetic retinopathy in minority women in Oslo, Norway

BACKGROUND

Diabetic retinopathy (DR) is the leading cause of adult blindness in the working age population worldwide, which can be prevented by early detection. Regular eye examinations are recommended and crucial for detecting sight-threatening DR. Use of artificial intelligence (AI) to lessen the burden on the healthcare system is needed.

PURPOSE

To perform a pilot cost-analysis study for detecting DR in a cohort of minority women with DM in Oslo, Norway, that have the highest prevalence of diabetes mellitus (DM) in the country, using both manual (ophthalmologist) and autonomous (AI) grading. This is the first study in Norway, as far as we know, that uses AI in DR- grading of retinal images.

METHODS

On Minority Women's Day, November 1, 2017, in Oslo, Norway, 33 patients (66 eyes) over 18 years of age diagnosed with DM (T1D and T2D) were screened. The Eidon - True Color Confocal Scanner (CenterVue, United States) was used for retinal imaging and graded for DR after screening had been completed, by an ophthalmologist and automatically, using EyeArt Automated DR Detection System, version 2.1.0 (EyeArt, EyeNuk, CA, USA). The gradings were based on the International Clinical Diabetic Retinopathy (ICDR) severity scale [1] detecting the presence or absence of referable DR. Cost-minimization analyses were performed for both grading methods.

RESULTS

33 women (64 eyes) were eligible for the analysis. A very good inter-rater agreement was found: 0.98 (P < 0.01), between the human and AI-based EyeArt grading system for detecting DR. The prevalence of DR was 18.6% (95% CI: 11.4-25.8%), and the sensitivity and specificity were 100% (95% CI: 100-100% and 95% CI: 100-100%), respectively. The cost difference for AI screening compared to human screening was $143 lower per patient (cost-saving) in favour of AI.

CONCLUSION

Our results indicate that The EyeArt AI system is both a reliable, cost-saving, and useful tool for DR grading in clinical practice.

Ocular biomarkers: useful incidental findings by deep learning algorithms in fundus photographs

BACKGROUND/OBJECTIVES

Artificial intelligence can assist with ocular image analysis for screening and diagnosis, but it is not yet capable of autonomous full-spectrum screening. Hypothetically, false-positive results may have unrealized screening potential arising from signals persisting despite training and/or ambiguous signals such as from biomarker overlap or high comorbidity. The study aimed to explore the potential to detect clinically useful incidental ocular biomarkers by screening fundus photographs of hypertensive adults using diabetic deep learning algorithms.

SUBJECTS/METHODS

Patients referred for treatment-resistant hypertension were imaged at a hospital unit in Perth, Australia, between 2016 and 2022. The same 45° colour fundus photograph selected for each of the 433 participants imaged was processed by three deep learning algorithms. Two expert retinal specialists graded all false-positive results for diabetic retinopathy in non-diabetic participants.

RESULTS

Of the 29 non-diabetic participants misclassified as positive for diabetic retinopathy, 28 (97%) had clinically useful retinal biomarkers. The models designed to screen for fewer diseases captured more incidental disease. All three algorithms showed a positive correlation between severity of hypertensive retinopathy and misclassified diabetic retinopathy.

CONCLUSIONS

The results suggest that diabetic deep learning models may be responsive to hypertensive and other clinically useful retinal biomarkers within an at-risk, hypertensive cohort. Observing that models trained for fewer diseases captured more incidental pathology increases confidence in signalling hypotheses aligned with using self-supervised learning to develop autonomous comprehensive screening. Meanwhile, non-referable and false-positive outputs of other deep learning screening models could be explored for immediate clinical use in other populations.

Machine learning algorithms for the prognostication of abdominal aortic aneurysm progression: a systematic review

INTRODUCTION

Abdominal aortic aneurysm (AAA), often characterized by an abdominal aortic diameter over 3.0 cm, is managed through screening, surveillance, and surgical intervention. AAA growth can be heterogeneous and rupture carries a high mortality rate, with size and certain risk factors influencing rupture risk. Research is ongoing to accurately predict individual AAA growth rates for personalized management. Machine learning, a subset of artificial intelligence, has shown promise in various medical fields, including endoleak detection post-EVAR. However, its application for predicting AAA growth remains insufficiently explored, thus necessitating further investigation. Subsequently, this paper aims to summarize the current status of machine learning in predicting AAA growth.

EVIDENCE ACQUISITION

A systematic database search of Embase, MEDLINE, Cochrane, PubMed and Google Scholar from inception till December 2022 was conducted of original articles that discussed the use of machine learning in predicting AAA growth using the aforementioned databases.

EVIDENCE SYNTHESIS

Overall, 2742 articles were extracted, of which seven retrospective studies involving 410 patients were included using a predetermined criteria. Six out of seven studies applied a supervised learning approach for their machine learning (ML) models, with considerable diversity observed within specific ML models. The majority of the studies concluded that machine learning models perform better in predicting AAA growth in comparison to reference models. All studies focused on predicting AAA growth over specified durations. Maximal luminal diameter was the most frequently used indicator, with alternative predictors being AAA volume, ILT (intraluminal thrombus) and flow-medicated diameter (FMD).

CONCLUSIONS

The nascent field of applying machine learning (ML) for Abdominal Aortic Aneurysm (AAA) expansion prediction exhibits potential to enhance predictive accuracy across diverse parameters. Future studies must emphasize evidencing clinical utility in a healthcare system context, thereby ensuring patient outcome improvement. This will necessitate addressing key ethical implications in establishing prospective studies related to this topic and collaboration among pivotal stakeholders within the AI field.

Target Product Profile for a Machine Learning-Automated Retinal Imaging Analysis Software for Use in English Diabetic Eye Screening: Protocol for a Mixed Methods Study

BACKGROUND

Diabetic eye screening (DES) represents a significant opportunity for the application of machine learning (ML) technologies, which may improve clinical and service outcomes. However, successful integration of ML into DES requires careful product development, evaluation, and implementation. Target product profiles (TPPs) summarize the requirements necessary for successful implementation so these can guide product development and evaluation.

OBJECTIVE

This study aims to produce a TPP for an ML-automated retinal imaging analysis software (ML-ARIAS) system for use in DES in England.

METHODS

This work will consist of 3 phases. Phase 1 will establish the characteristics to be addressed in the TPP. A list of candidate characteristics will be generated from the following sources: an overview of systematic reviews of diagnostic test TPPs; a systematic review of digital health TPPs; and the National Institute for Health and Care Excellence's Evidence Standards Framework for Digital Health Technologies. The list of characteristics will be refined and validated by a study advisory group (SAG) made up of representatives from key stakeholders in DES. This includes people with diabetes; health care professionals; health care managers and leaders; and regulators and policy makers. In phase 2, specifications for these characteristics will be drafted following a series of semistructured interviews with participants from these stakeholder groups. Data collected from these interviews will be analyzed using the shortlist of characteristics as a framework, after which specifications will be drafted to create a draft TPP. Following approval by the SAG, in phase 3, the draft will enter an internet-based Delphi consensus study with participants sought from the groups previously identified, as well as ML-ARIAS developers, to ensure feasibility. Participants will be invited to score characteristic and specification pairs on a scale from "definitely exclude" to "definitely include," and suggest edits. The document will be iterated between rounds based on participants' feedback. Feedback on the draft document will be sought from a group of ML-ARIAS developers before its final contents are agreed upon in an in-person consensus meeting. At this meeting, representatives from the stakeholder groups previously identified (minus ML-ARIAS developers, to avoid bias) will be presented with the Delphi results and feedback of the user group and asked to agree on the final contents by vote.

RESULTS

Phase 1 was completed in November 2023. Phase 2 is underway and expected to finish in March 2024. Phase 3 is expected to be complete in July 2024.

CONCLUSIONS

The multistakeholder development of a TPP for an ML-ARIAS for use in DES in England will help developers produce tools that serve the needs of patients, health care providers, and their staff. The TPP development process will also provide methods and a template to produce similar documents in other disease areas.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/50568.

Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review

BACKGROUND

Generative artificial intelligence tools and applications (GenAI) are being increasingly used in health care. Physicians, specialists, and other providers have started primarily using GenAI as an aid or tool to gather knowledge, provide information, train, or generate suggestive dialogue between physicians and patients or between physicians and patients' families or friends. However, unless the use of GenAI is oriented to be helpful in clinical service encounters that can improve the accuracy of diagnosis, treatment, and patient outcomes, the expected potential will not be achieved. As adoption continues, it is essential to validate the effectiveness of the infusion of GenAI as an intelligent technology in service encounters to understand the gap in actual clinical service use of GenAI.

OBJECTIVE

This study synthesizes preliminary evidence on how GenAI assists, guides, and automates clinical service rendering and encounters in health care The review scope was limited to articles published in peer-reviewed medical journals.

METHODS

We screened and selected 0.38% (161/42,459) of articles published between January 1, 2020, and May 31, 2023, identified from PubMed. We followed the protocols outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to select highly relevant studies with at least 1 element on clinical use, evaluation, and validation to provide evidence of GenAI use in clinical services. The articles were classified based on their relevance to clinical service functions or activities using the descriptive and analytical information presented in the articles.

RESULTS

Of 161 articles, 141 (87.6%) reported using GenAI to assist services through knowledge access, collation, and filtering. GenAI was used for disease detection (19/161, 11.8%), diagnosis (14/161, 8.7%), and screening processes (12/161, 7.5%) in the areas of radiology (17/161, 10.6%), cardiology (12/161, 7.5%), gastrointestinal medicine (4/161, 2.5%), and diabetes (6/161, 3.7%). The literature synthesis in this study suggests that GenAI is mainly used for diagnostic processes, improvement of diagnosis accuracy, and screening and diagnostic purposes using knowledge access. Although this solves the problem of knowledge access and may improve diagnostic accuracy, it is oriented toward higher value creation in health care.

CONCLUSIONS

GenAI informs rather than assisting or automating clinical service functions in health care. There is potential in clinical service, but it has yet to be actualized for GenAI. More clinical service-level evidence that GenAI is used to streamline some functions or provides more automated help than only information retrieval is needed. To transform health care as purported, more studies related to GenAI applications must automate and guide human-performed services and keep up with the optimism that forward-thinking health care organizations will take advantage of GenAI.

Benefit-Risk Assessment of ChatGPT Applications in the Field of Diabetes and Metabolic Illnesses: A Qualitative Study

Background

The use of ChatGPT and artificial intelligence (AI) in the management of metabolic and endocrine disorders presents both significant opportunities and notable risks.

Objectives

To investigate the benefits and risks associated with the application of ChatGPT in managing diabetes and metabolic illnesses by exploring the perspectives of endocrinologists and diabetologists.

Methods and materials

The study employed a qualitative research approach. A semi-structured in-depth interview guide was developed. A convenience sample of 25 endocrinologists and diabetologists was enrolled and interviewed. All interviews were audiotaped and verbatim transcribed; then, thematic analysis was used to determine the themes in the data.

Results

The findings of the thematic analysis resulted in 19 codes and 9 major themes regarding the benefits of implementing AI and ChatGPT in managing diabetes and metabolic illnesses. Moreover, the extracted risks of implementing AI and ChatGPT in managing diabetes and metabolic illnesses were categorized into 7 themes and 14 codes. The benefits of heightened diagnostic precision, tailored treatment, and efficient resource utilization have potential to improve patient results. Concurrently, the identification of potential challenges, such as data security concerns and the need for AI that can be explained, enables stakeholders to proactively tackle these issues.

Conclusions

Regulatory frameworks must evolve to keep pace with the rapid adoption of AI in healthcare. Sustained attention to ethical considerations, including obtaining patient consent, safeguarding data privacy, ensuring accountability, and promoting fairness, remains critical. Despite its potential impact on the human aspect of healthcare, AI will remain an integral component of patient-centered care. Striking a balance between AI-assisted decision-making and human expertise is essential to uphold trust and provide comprehensive patient care.

An Economic Analysis for the Use of Artificial Intelligence in Screening for Diabetic Retinopathy in Trinidad and Tobago

This is a systematic review of 25 publications on the topics of the prevalence and cost of diabetic retinopathy (DR) in Trinidad and Tobago, the cost of traditional methods of screening for DR, and the use and cost of artificial intelligence (AI) in screening for DR. Analysis of these publications was used to identify and make estimates for how resources allocated to ophthalmology in public health systems in Trinidad and Tobago can be more efficiently utilized by employing AI in diagnosing treatable DR. DR screening was found to be an effective method of detecting the disease. Screening was found to be a universally cost-effective method of disease prevention and for altering the natural history of the disease in the spectrum of low-middle to high-income economies, such as Rwanda, Thailand, China, South Korea, and Singapore. AI and deep learning systems were found to be clinically superior to, or as effective as, human graders in areas where they were deployed, indicating that the systems are clinically safe. They have been shown to improve access to diabetic retinal screening, improve compliance with screening appointments, and prove to be cost-effective, especially in rural areas. Trinidad and Tobago, which is estimated to be disproportionately more affected by the burden of DR when projected out to the mid-21st century, stands to save as much as US$60 million annually from the implementation of an AI-based system to screen for DR versus conventional manual grading.

Enhancing Vault Prediction and ICL Sizing Through Advanced Machine Learning Models

PURPOSE

To use artificial intelligence (AI) technology to accurately predict vault and Implantable Collamer Lens (ICL) size.

METHODS

The methodology focused on enhancing predictive capabilities through the fusion of machine-learning algorithms. Specifically, AdaBoost, Random Forest, Decision Tree, Support Vector Regression, LightGBM, and XGBoost were integrated into a majority-vote model. The performance of each model was evaluated using appropriate metrics such as accuracy, precision, F1-score, and area under the curve (AUC).

RESULTS

The majority-vote model exhibited the highest performance among the classification models, with an accuracy of 81.9% area under the curve (AUC) of 0.807. Notably, LightGBM (accuracy = 0.788, AUC = 0.803) and XGBoost (ACC = 0.790, AUC = 0.801) demonstrated competitive results. For the ICL size prediction, the Random Forest model achieved an impressive accuracy of 85.3% (AUC = 0.973), whereas XG-Boost (accuracy = 0.834, AUC = 0.961) and LightGBM (accuracy = 0.816, AUC = 0.961) maintained their compatibility.

CONCLUSIONS

This study highlights the potential of diverse machine learning algorithms to enhance postoperative vault and ICL size prediction, ultimately contributing to the safety of ICL implantation procedures. Furthermore, the introduction of the novel majority-vote model demonstrates its capability to combine the advantages of multiple models, yielding superior accuracy. Importantly, this study will empower ophthalmologists to use a precise tool for vault prediction, facilitating informed ICL size selection in clinical practice. [J Refract Surg. 2024;40(3):e126-e132.].

Assessing Diabetic Retinopathy Staging With AI: A Comparative Analysis Between Pseudocolor and LED Imaging

Purpose

To compare the diagnostic performance of artificial intelligence (AI)-based diabetic retinopathy (DR) staging system across pseudocolor, simulated white light (SWL), and light-emitting diode (LED) camera imaging modalities.

Methods

A cross-sectional investigation involved patients with diabetes undergoing imaging with an iCare DRSplus confocal LED camera and an Optos confocal, ultra-widefield pseudocolor camera, with and without SWL. Macula-centered and optic nerve-centered 45 × 45-degree photographs were processed using EyeArt v2.1. Human graders established the ground truth (GT) for DR severity on dilated fundus exams. Sensitivity and weighted Cohen's weighted kappa (wκ) were calculated. An ordinal generalized linear mixed model identified factors influencing accurate DR staging.

Results

The study included 362 eyes from 189 patients. The LED camera excelled in identifying sight-threatening DR stages (sensitivity = 0.83, specificity = 0.95 for proliferative DR) and had the highest agreement with the GT (wκ = 0.71). The addition of SWL to pseudocolor imaging resulted in decreased performance (sensitivity = 0.33, specificity = 0.98 for proliferative DR; wκ = 0.55). Peripheral lesions reduced the likelihood of being staged in the same or higher DR category by 80% (P < 0.001).

Conclusions

Pseudocolor and LED cameras, although proficient, demonstrated non-interchangeable performance, with the LED camera exhibiting superior accuracy in identifying advanced DR stages. These findings underscore the importance of implementing AI systems trained for ultra-widefield imaging, considering the impact of peripheral lesions on correct DR staging.

Translational Relevance

This study underscores the need for artificial intelligence-based systems specifically trained for ultra-widefield imaging in diabetic retinopathy assessment.

Economic evaluation for medical artificial intelligence: accuracy vs. cost-effectiveness in a diabetic retinopathy screening case

Artificial intelligence (AI) models have shown great accuracy in health screening. However, for real-world implementation, high accuracy may not guarantee cost-effectiveness. Improving AI's sensitivity finds more high-risk patients but may raise medical costs while increasing specificity reduces unnecessary referrals but may weaken detection capability. To evaluate the trade-off between AI model performance and the long-running cost-effectiveness, we conducted a cost-effectiveness analysis in a nationwide diabetic retinopathy (DR) screening program in China, comprising 251,535 participants with diabetes over 30 years. We tested a validated AI model in 1100 different diagnostic performances (presented as sensitivity/specificity pairs) and modeled annual screening scenarios. The status quo was defined as the scenario with the most accurate AI performance. The incremental cost-effectiveness ratio (ICER) was calculated for other scenarios against the status quo as cost-effectiveness metrics. Compared to the status quo (sensitivity/specificity: 93.3%/87.7%), six scenarios were cost-saving and seven were cost-effective. To achieve cost-saving or cost-effective, the AI model should reach a minimum sensitivity of 88.2% and specificity of 80.4%. The most cost-effective AI model exhibited higher sensitivity (96.3%) and lower specificity (80.4%) than the status quo. In settings with higher DR prevalence and willingness-to-pay levels, the AI needed higher sensitivity for optimal cost-effectiveness. Urban regions and younger patient groups also required higher sensitivity in AI-based screening. In real-world DR screening, the most accurate AI model may not be the most cost-effective. Cost-effectiveness should be independently evaluated, which is most likely to be affected by the AI's sensitivity.

Image quality assessment of retinal fundus photographs for diabetic retinopathy in the machine learning era: a review

This study aimed to evaluate the image quality assessment (IQA) and quality criteria employed in publicly available datasets for diabetic retinopathy (DR). A literature search strategy was used to identify relevant datasets, and 20 datasets were included in the analysis. Out of these, 12 datasets mentioned performing IQA, but only eight specified the quality criteria used. The reported quality criteria varied widely across datasets, and accessing the information was often challenging. The findings highlight the importance of IQA for AI model development while emphasizing the need for clear and accessible reporting of IQA information. The study suggests that automated quality assessments can be a valid alternative to manual labeling and emphasizes the importance of establishing quality standards based on population characteristics, clinical use, and research purposes. In conclusion, image quality assessment is important for AI model development; however, strict data quality standards must not limit data sharing. Given the importance of IQA for developing, validating, and implementing deep learning (DL) algorithms, it's recommended that this information be reported in a clear, specific, and accessible way whenever possible. Automated quality assessments are a valid alternative to the traditional manual labeling process, and quality standards should be determined according to population characteristics, clinical use, and research purpose.

Implementing and evaluating a fully functional AI-enabled model for chronic eye disease screening in a real clinical environment

BACKGROUND

Artificial intelligence (AI) has the potential to increase the affordability and accessibility of eye disease screening, especially with the recent approval of AI-based diabetic retinopathy (DR) screening programs in several countries.

METHODS

This study investigated the performance, feasibility, and user experience of a seamless hardware and software solution for screening chronic eye diseases in a real-world clinical environment in Germany. The solution integrated AI grading for DR, age-related macular degeneration (AMD), and glaucoma, along with specialist auditing and patient referral decision. The study comprised several components: (1) evaluating the entire system solution from recruitment to eye image capture and AI grading for DR, AMD, and glaucoma; (2) comparing specialist's grading results with AI grading results; (3) gathering user feedback on the solution.

RESULTS

A total of 231 patients were recruited, and their consent forms were obtained. The sensitivity, specificity, and area under the curve for DR grading were 100.00%, 80.10%, and 90.00%, respectively. For AMD grading, the values were 90.91%, 78.79%, and 85.00%, and for glaucoma grading, the values were 93.26%, 76.76%, and 85.00%. The analysis of all false positive cases across the three diseases and their comparison with the final referral decisions revealed that only 17 patients were falsely referred among the 231 patients. The efficacy analysis of the system demonstrated the effectiveness of the AI grading process in the study's testing environment. Clinical staff involved in using the system provided positive feedback on the disease screening process, particularly praising the seamless workflow from patient registration to image transmission and obtaining the final result. Results from a questionnaire completed by 12 participants indicated that most found the system easy, quick, and highly satisfactory. The study also revealed room for improvement in the AMD model, suggesting the need to enhance its training data. Furthermore, the performance of the glaucoma model grading could be improved by incorporating additional measures such as intraocular pressure.

CONCLUSIONS

The implementation of the AI-based approach for screening three chronic eye diseases proved effective in real-world settings, earning positive feedback on the usability of the integrated platform from both the screening staff and auditors. The auditing function has proven valuable for obtaining efficient second opinions from experts, pointing to its potential for enhancing remote screening capabilities.

TRIAL REGISTRATION

Institutional Review Board of the Hamburg Medical Chamber (Ethik-Kommission der Ärztekammer Hamburg): 2021-10574-BO-ff.

How Can Artificial Intelligence Be Implemented Effectively in Diabetic Retinopathy Screening in Japan?

Diabetic retinopathy (DR) is a major microvascular complication of diabetes, affecting a substantial portion of diabetic patients worldwide. Timely intervention is pivotal in mitigating the risk of blindness associated with DR, yet early detection remains a challenge due to the absence of early symptoms. Screening programs have emerged as a strategy to address this burden, and this paper delves into the role of artificial intelligence (AI) in advancing DR screening in Japan. There are two pathways for DR screening in Japan: a health screening pathway and a clinical referral path from physicians to ophthalmologists. AI technologies that realize automated image classification by applying deep learning are emerging. These technologies have exhibited substantial promise, achieving sensitivity and specificity levels exceeding 90% in prospective studies. Moreover, we introduce the potential of Generative AI and large language models (LLMs) to transform healthcare delivery, particularly in patient engagement, medical records, and decision support. Considering the use of AI in DR screening in Japan, we propose to follow a seven-step framework for systematic screening and emphasize the importance of integrating AI into a well-designed screening program. Automated scoring systems with AI enhance screening quality, but their effectiveness depends on their integration into the broader screening ecosystem. LLMs emerge as an important tool to fill gaps in the screening process, from personalized invitations to reporting results, facilitating a seamless and efficient system. However, it is essential to address concerns surrounding technical accuracy and governance before full-scale integration into the healthcare system. In conclusion, this review highlights the challenges in the current screening pathway and the potential for AI, particularly LLM, to revolutionize DR screening in Japan. The future direction will depend on leadership from ophthalmologists and stakeholders to address long-standing challenges in DR screening so that all people have access to accessible and effective screening.

Real-world evaluation of smartphone-based artificial intelligence to screen for diabetic retinopathy in Dominica: a clinical validation study

OBJECTIVE

Several artificial intelligence (AI) systems for diabetic retinopathy screening have been validated but there is limited evidence on their performance in real-world settings. This study aimed to assess the performance of an AI software deployed within the diabetic retinopathy screening programme in Dominica.

METHODS AND ANALYSIS

We conducted a prospective, cross-sectional clinical validation study. Patients with diabetes aged 18 years and above attending the diabetic retinopathy screening in primary care facilities in Dominica from 5 June to 3 July 2021 were enrolled.Grading was done at the point of care by the field grader, followed by counselling and referral to the eye clinic. Images were then graded by an AI system. Sensitivity, specificity with 95% CIs and area under the curve (AUC) were calculated for comparing the AI to field grader as gold standard.

RESULTS

A total of 587 participants were screened. The AI had a sensitivity and specificity for detecting referable diabetic retinopathy of 77.5% and 91.5% compared with the grader, for all participants, including ungradable images. The AUC was 0.8455. Excluding 52 participants deemed ungradable by the grader, the AI had a sensitivity and specificity of 81.4% and 91.5%, with an AUC of 0.9648.

CONCLUSION

This study provides evidence that AI has the potential to be deployed to assist a diabetic screening programme in a middle-income real-world setting and perform with reasonable accuracy compared with a specialist grader.

Barriers to Implementation of Teleretinal Diabetic Retinopathy Screening Programs Across the University of California

Aim: To describe barriers to implementation of diabetic retinopathy (DR) teleretinal screening programs and artificial intelligence (AI) integration at the University of California (UC). Methods: Institutional representatives from UC Los Angeles, San Diego, San Francisco, Irvine, and Davis were surveyed for the year of their program's initiation, active status at the time of survey (December 2021), number of primary care clinics involved, screening image quality, types of eye providers, image interpretation turnaround time, and billing codes used. Representatives were asked to rate perceptions toward barriers to teleretinal DR screening and AI implementation using a 5-point Likert scale. Results: Four UC campuses had active DR teleretinal screening programs at the time of survey and screened between 246 and 2,123 patients at 1-6 clinics per campus. Sites reported variation between poor-quality photos (<5% to 15%) and average image interpretation time (1-5 days). Patient education, resource availability, and infrastructural support were identified as barriers to DR teleretinal screening. Cost and integration into existing technology infrastructures were identified as barriers to AI integration in DR screening. Conclusions: Despite the potential to increase access to care, there remain several barriers to widespread implementation of DR teleretinal screening. More research is needed to develop best practices to overcome these barriers.

Two-year recall for people with no diabetic retinopathy: a multi-ethnic population-based retrospective cohort study using real-world data to quantify the effect

BACKGROUND/AIMS

The English Diabetic Eye Screening Programme (DESP) offers people living with diabetes (PLD) annual screening. Less frequent screening has been advocated among PLD without diabetic retinopathy (DR), but evidence for each ethnic group is limited. We examined the potential effect of biennial versus annual screening on the detection of sight-threatening diabetic retinopathy (STDR) and proliferative diabetic retinopathy (PDR) among PLD without DR from a large urban multi-ethnic English DESP.

METHODS

PLD in North-East London DESP (January 2012 to December 2021) with no DR on two prior consecutive screening visits with up to 8 years of follow-up were examined. Annual STDR and PDR incidence rates, overall and by ethnicity, were quantified. Delays in identification of STDR and PDR events had 2-year screening intervals been used were determined.

FINDINGS

Among 82 782 PLD (37% white, 36% South Asian, and 16% black people), there were 1788 incident STDR cases over mean (SD) 4.3 (2.4) years (STDR rate 0.51, 95% CI 0.47 to 0.55 per 100-person-years). STDR incidence rates per 100-person-years by ethnicity were 0.55 (95% CI 0.48 to 0.62) for South Asian, 0.34 (95% CI 0.29 to 0.40) for white, and 0.77 (95% CI 0.65 to 0.90) for black people. Biennial screening would have delayed diagnosis by 1 year for 56.3% (1007/1788) with STDR and 43.6% (45/103) with PDR. Standardised cumulative rates of delayed STDR per 100 000 persons for each ethnic group were 1904 (95% CI 1683 to 2154) for black people, 1276 (95% CI 1153 to 1412) for South Asian people, and 844 (95% CI 745 to 955) for white people.

INTERPRETATION

Biennial screening would have delayed detection of some STDR and PDR by 1 year, especially among those of black ethnic origin, leading to healthcare inequalities.

What are the perceptions and concerns of people living with diabetes and National Health Service staff around the potential implementation of AI-assisted screening for diabetic eye disease? Development and validation of a survey for use in a secondary care screening setting

INTRODUCTION

The English National Health Service (NHS) Diabetic Eye Screening Programme (DESP) performs around 2.3 million eye screening appointments annually, generating approximately 13 million retinal images that are graded by humans for the presence or severity of diabetic retinopathy. Previous research has shown that automated retinal image analysis systems, including artificial intelligence (AI), can identify images with no disease from those with diabetic retinopathy as safely and effectively as human graders, and could significantly reduce the workload for human graders. Some algorithms can also determine the level of severity of the retinopathy with similar performance to humans. There is a need to examine perceptions and concerns surrounding AI-assisted eye-screening among people living with diabetes and NHS staff, if AI was to be introduced into the DESP, to identify factors that may influence acceptance of this technology.

METHODS AND ANALYSIS

People living with diabetes and staff from the North East London (NEL) NHS DESP were invited to participate in two respective focus groups to codesign two online surveys exploring their perceptions and concerns around the potential introduction of AI-assisted screening.Focus group participants were representative of the local population in terms of ages and ethnicity. Participants' feedback was taken into consideration to update surveys which were circulated for further feedback. Surveys will be piloted at the NEL DESP and followed by semistructured interviews to assess accessibility, usability and to validate the surveys.Validated surveys will be distributed by other NHS DESP sites, and also via patient groups on social media, relevant charities and the British Association of Retinal Screeners. Post-survey evaluative interviews will be undertaken among those who consent to participate in further research.

ETHICS AND DISSEMINATION

Ethical approval has been obtained by the NHS Research Ethics Committee (IRAS ID: 316631). Survey results will be shared and discussed with focus groups to facilitate preparation of findings for publication and to inform codesign of outreach activities to address concerns and perceptions identified.

Clinician-Driven AI: Code-Free Self-Training on Public Data for Diabetic Retinopathy Referral

Importance

Democratizing artificial intelligence (AI) enables model development by clinicians with a lack of coding expertise, powerful computing resources, and large, well-labeled data sets.

Objective

To determine whether resource-constrained clinicians can use self-training via automated machine learning (ML) and public data sets to design high-performing diabetic retinopathy classification models.

Design, Setting, and Participants

This diagnostic quality improvement study was conducted from January 1, 2021, to December 31, 2021. A self-training method without coding was used on 2 public data sets with retinal images from patients in France (Messidor-2 [n = 1748]) and the UK and US (EyePACS [n = 58 689]) and externally validated on 1 data set with retinal images from patients of a private Egyptian medical retina clinic (Egypt [n = 210]). An AI model was trained to classify referable diabetic retinopathy as an exemplar use case. Messidor-2 images were assigned adjudicated labels available on Kaggle; 4 images were deemed ungradable and excluded, leaving 1744 images. A total of 300 images randomly selected from the EyePACS data set were independently relabeled by 3 blinded retina specialists using the International Classification of Diabetic Retinopathy protocol for diabetic retinopathy grade and diabetic macular edema presence; 19 images were deemed ungradable, leaving 281 images. Data analysis was performed from February 1 to February 28, 2021.

Exposures

Using public data sets, a teacher model was trained with labeled images using supervised learning. Next, the resulting predictions, termed pseudolabels, were used on an unlabeled public data set. Finally, a student model was trained with the existing labeled images and the additional pseudolabeled images.

Main Outcomes and Measures

The analyzed metrics for the models included the area under the receiver operating characteristic curve (AUROC), accuracy, sensitivity, specificity, and F1 score. The Fisher exact test was performed, and 2-tailed P values were calculated for failure case analysis.

Results

For the internal validation data sets, AUROC values for performance ranged from 0.886 to 0.939 for the teacher model and from 0.916 to 0.951 for the student model. For external validation of automated ML model performance, AUROC values and accuracy were 0.964 and 93.3% for the teacher model, 0.950 and 96.7% for the student model, and 0.890 and 94.3% for the manually coded bespoke model, respectively.

Conclusions and Relevance

These findings suggest that self-training using automated ML is an effective method to increase both model performance and generalizability while decreasing the need for costly expert labeling. This approach advances the democratization of AI by enabling clinicians without coding expertise or access to large, well-labeled private data sets to develop their own AI models.

Artificial Intelligence and Diabetic Retinopathy: AI Framework, Prospective Studies, Head-to-head Validation, and Cost-effectiveness

Current guidelines recommend that individuals with diabetes receive yearly eye exams for detection of referable diabetic retinopathy (DR), one of the leading causes of new-onset blindness. For addressing the immense screening burden, artificial intelligence (AI) algorithms have been developed to autonomously screen for DR from fundus photography without human input. Over the last 10 years, many AI algorithms have achieved good sensitivity and specificity (>85%) for detection of referable DR compared with human graders; however, many questions still remain. In this narrative review on AI in DR screening, we discuss key concepts in AI algorithm development as a background for understanding the algorithms. We present the AI algorithms that have been prospectively validated against human graders and demonstrate the variability of reference standards and cohort demographics. We review the limited head-to-head validation studies where investigators attempt to directly compare the available algorithms. Next, we discuss the literature regarding cost-effectiveness, equity and bias, and medicolegal considerations, all of which play a role in the implementation of these AI algorithms in clinical practice. Lastly, we highlight ongoing efforts to bridge gaps in AI model data sets to pursue equitable development and delivery.

Deep learning detection of diabetic retinopathy in Scotland's diabetic eye screening programme

BACKGROUND/AIMS

Support vector machine-based automated grading (known as iGradingM) has been shown to be safe, cost-effective and robust in the diabetic retinopathy (DR) screening (DES) programme in Scotland. It triages screening episodes as gradable with no DR versus manual grading required. The study aim was to develop a deep learning-based autograder using images and gradings from DES and to compare its performance with that of iGradingM.

METHODS

Retinal images, quality assurance (QA) data and routine DR grades were obtained from national datasets in 179 944 patients for years 2006-2016. QA grades were available for 744 images. We developed a deep learning-based algorithm to detect whether either eye contained ungradable images or any DR. The sensitivity and specificity were evaluated against consensus QA grades and routine grades.

RESULTS

Images used in QA which were ungradable or with DR were detected by deep learning with better specificity compared with manual graders (p<0.001) and with iGradingM (p<0.001) at the same sensitivities. Any DR according to the DES final grade was detected with 89.19% (270 392/303 154) sensitivity and 77.41% (500 945/647 158) specificity. Observable disease and referable disease were detected with sensitivities of 96.58% (16 613/17 201) and 98.48% (22 600/22 948), respectively. Overall, 43.84% of screening episodes would require manual grading.

CONCLUSION

A deep learning-based system for DR grading was evaluated in QA data and images from 11 years in 50% of people attending a national DR screening programme. The system could reduce the manual grading workload at the same sensitivity compared with the current automated grading system.

Test accuracy of artificial intelligence-based grading of fundus images in diabetic retinopathy screening: A systematic review

OBJECTIVES

To systematically review the accuracy of artificial intelligence (AI)-based systems for grading of fundus images in diabetic retinopathy (DR) screening.

METHODS

We searched MEDLINE, EMBASE, the Cochrane Library and the ClinicalTrials.gov from 1st January 2000 to 27th August 2021. Accuracy studies published in English were included if they met the pre-specified inclusion criteria. Selection of studies for inclusion, data extraction and quality assessment were conducted by one author with a second reviewer independently screening and checking 20% of titles. Results were analysed narratively.

RESULTS

Forty-three studies evaluating 15 deep learning (DL) and 4 machine learning (ML) systems were included. Nine systems were evaluated in a single study each. Most studies were judged to be at high or unclear risk of bias in at least one QUADAS-2 domain. Sensitivity for referable DR and higher grades was ≥85% while specificity varied and was <80% for all ML systems and in 6/31 studies evaluating DL systems. Studies reported high accuracy for detection of ungradable images, but the latter were analysed and reported inconsistently. Seven studies reported that AI was more sensitive but less specific than human graders.

CONCLUSIONS

AI-based systems are more sensitive than human graders and could be safe to use in clinical practice but have variable specificity. However, for many systems evidence is limited, at high risk of bias and may not generalise across settings. Therefore, pre-implementation assessment in the target clinical pathway is essential to obtain reliable and applicable accuracy estimates.

Diabetic Retinopathy Screening Using Smartphone-Based Fundus Photography and Deep-Learning Artificial Intelligence in the Yucatan Peninsula: A Field Study

BACKGROUND

To compare the performance of Medios (offline) and EyeArt (online) artificial intelligence (AI) algorithms for detecting diabetic retinopathy (DR) on images captured using fundus-on-smartphone photography in a remote outreach field setting.

METHODS

In June, 2019 in the Yucatan Peninsula, 248 patients, many of whom had chronic visual impairment, were screened for DR using two portable Remidio fundus-on-phone cameras, and 2130 images obtained were analyzed, retrospectively, by Medios and EyeArt. Screening performance metrics also were determined retrospectively using masked image analysis combined with clinical examination results as the reference standard.

RESULTS

A total of 129 patients were determined to have some level of DR; 119 patients had no DR. Medios was capable of evaluating every patient with a sensitivity (95% confidence intervals [CIs]) of 94% (88%-97%) and specificity of 94% (88%-98%). Owing primarily to photographer error, EyeArt evaluated 156 patients with a sensitivity of 94% (86%-98%) and specificity of 86% (77%-93%). In a head-to-head comparison of 110 patients, the sensitivities of Medios and EyeArt were 99% (93%-100%) and 95% (87%-99%). The specificities for both were 88% (73%-97%).

CONCLUSIONS

Medios and EyeArt AI algorithms demonstrated high levels of sensitivity and specificity for detecting DR when applied in this real-world field setting. Both programs should be considered in remote, large-scale DR screening campaigns where immediate results are desirable, and in the case of EyeArt, online access is possible.

A snapshot of artificial intelligence research 2019-2021: is it replacing or assisting physicians?

Artificial intelligence (AI) has the potential to be a disruptive technology in healthcare. Recently, there is increased speculation that AI may be used to replace healthcare providers in the future. To answer this question, we reviewed over 21 000 articles published in medical specialty journals between 2019 and 2021 to evaluate whether these AI models were intended to assist or replace healthcare providers. We also evaluated whether all Food and Drug Administration (FDA)-approved AI models were used to assist or replace healthcare providers. We find that most AI models published in this time period were intended to assist rather than replace healthcare providers, and that most of the published AI models performed tasks that could not be done by healthcare providers.

Towards population-independent, multi-disease detection in fundus photographs

Independent validation studies of automatic diabetic retinopathy screening systems have recently shown a drop of screening performance on external data. Beyond diabetic retinopathy, this study investigates the generalizability of deep learning (DL) algorithms for screening various ocular anomalies in fundus photographs, across heterogeneous populations and imaging protocols. The following datasets are considered: OPHDIAT (France, diabetic population), OphtaMaine (France, general population), RIADD (India, general population) and ODIR (China, general population). Two multi-disease DL algorithms were developed: a Single-Dataset (SD) network, trained on the largest dataset (OPHDIAT), and a Multiple-Dataset (MD) network, trained on multiple datasets simultaneously. To assess their generalizability, both algorithms were evaluated whenever training and test data originate from overlapping datasets or from disjoint datasets. The SD network achieved a mean per-disease area under the receiver operating characteristic curve (mAUC) of 0.9571 on OPHDIAT. However, it generalized poorly to the other three datasets (mAUC < 0.9). When all four datasets were involved in training, the MD network significantly outperformed the SD network (p = 0.0058), indicating improved generality. However, in leave-one-dataset-out experiments, performance of the MD network was significantly lower on populations unseen during training than on populations involved in training (p < 0.0001), indicating imperfect generalizability.

Artificial intelligence in ophthalmology: The path to the real-world clinic

Artificial intelligence (AI) has great potential to transform healthcare by enhancing the workflow and productivity of clinicians, enabling existing staff to serve more patients, improving patient outcomes, and reducing health disparities. In the field of ophthalmology, AI systems have shown performance comparable with or even better than experienced ophthalmologists in tasks such as diabetic retinopathy detection and grading. However, despite these quite good results, very few AI systems have been deployed in real-world clinical settings, challenging the true value of these systems. This review provides an overview of the current main AI applications in ophthalmology, describes the challenges that need to be overcome prior to clinical implementation of the AI systems, and discusses the strategies that may pave the way to the clinical translation of these systems.

Performance of an artificial intelligence automated system for diabetic eye screening in a large English population

AIMS

A diabetic eye screening programme has huge value in reducing avoidable sight loss by identifying diabetic retinopathy at a stage when it can be treated. Artificial intelligence automated systems can be used for diabetic eye screening but are not employed in the national English Diabetic Eye Screening Programme. The aim was to report the performance of a commercially available deep-learning artificial intelligence software in a large English population.

METHODS

9817 anonymised image sets from 10,000 consecutive diabetic eye screening episodes were presented to an artificial intelligence software. The sensitivity and specificity of the artificial intelligence system for detecting diabetic retinopathy were determined using the diabetic eye screening programme manual grade according to national protocols as the reference standard.

RESULTS

For no diabetic retinopathy versus any diabetic retinopathy, the sensitivity of the artificial intelligence grading system was 69.7% and specificity 92.2%. The performance of the artificial intelligence system was superior for no or mild diabetic retinopathy versus significant or referrable diabetic retinopathy with a sensitivity of 95.4% and specificity of 92.0%. No cases were identified in which the artificial intelligence grade had missed significant diabetic retinopathy.

CONCLUSION

The performance of a commercially available deep-learning artificial intelligence system for identifying diabetic retinopathy in an English national Diabetic Eye Screening Programme is presented. Using the pre-defined settings artificial intelligence performance was highest when identifying diabetic retinopathy which requires an action by the screening programme.

Validation of diagnostic accuracy of retinal image grading by trained non-ophthalmologist grader for detecting diabetic retinopathy and diabetic macular edema

PURPOSE

To validate the fundus image grading results by a trained grader (Non-ophthalmologist) and an ophthalmologist grader for detecting diabetic retinopathy (DR) and diabetic macular oedema (DMO) against fundus examination by a retina specialist (gold standard).

METHODS

A prospective diagnostic accuracy study was conducted using 2002 non-mydriatic colour fundus images from 1001 patients aged ≥40 years. Using the Aravind Diabetic Retinopathy Evaluation Software (ADRES) images were graded by both a trained non-ophthalmologist grader (grader-1) and an ophthalmologist (grader-2). Sensitivity, specificity, positive predictive value and negative predictive value were calculated for grader-1 and grader-2 against the grading results by an independent retina specialist who performed dilated fundus examination for every study participant.

RESULTS

Out of 1001 patients included, 42% were women and the mean ± (SD) age was 55.8 (8.39) years. For moderate or worse DR, the sensitivity and specificity for grading by grader-1 with respect to the gold standard was 66.9% and 91.0% respectively and the same for the ophthalmologist was 83.6% and 80.3% respectively. For referable DMO, grader-1 and grader-2 had a sensitivity of 74.6% and 85.6% respectively and a specificity of 83.7% and 79.8% respectively.

CONCLUSIONS

Our results demonstrate good level of accuracy for the fundus image grading performed by a trained non-ophthalmologist which was comparable with the grading by an ophthalmologist. Engaging trained non-ophthalmologists potentially can enhance the efficiency of DR diagnosis using fundus images. Further study with multiple non-ophthalmologist graders is needed to verify the results and strategies to improve agreement for DMO diagnosis are needed.

A multi-centre prospective evaluation of THEIA™ to detect diabetic retinopathy (DR) and diabetic macular oedema (DMO) in the New Zealand screening program

PURPOSE

To validate the potential application of THEIA™ as clinical decision making assistant in a national screening program.

METHODS

A total of 900 patients were recruited from either an urban large eye hospital, or a semi-rural optometrist led screening provider, as they were attending their appointment as part of New Zealand Diabetic Eye Screening Programme. The de-identified images were independently graded by three senior specialists, and final results were aggregated using New Zealand grading scheme, which was then converted to referable/non-referable and Healthy/mild/more than mild/sight threatening categories.

RESULTS

THEIA™ managed to grade all images obtained during the study. Comparing the adjudicated images from the specialist grading team, "ground truth", with the grading by the AI platform in detecting "sight threatening" disease, at the patient level THEIA™ achieved 100% imageability, 100% [98.49-100.00%] sensitivity and [97.02-99.16%] specificity, and negative predictive value of 100%. In other words, THEIA™ did not miss any patients with "more than mild" or "sight threatening" disease. The level of agreement between the clinicians and the aggregated results was (k value: 0.9881, 0.9557, and 0.9175), and the level of agreement between THEIA™ and the aggregated labels was (k value: 0.9515).

CONCLUSION

This multi-centre prospective trial showed that THEIA™ did not miss referable disease when screening for diabetic retinopathy and maculopathy. It also had a very high level of granularity in reporting the disease level. As THEIA™ has been tested on a variety of cameras, operating in a range of clinics (rural/urban, ophthalmologist-led\optometrist-led), we believe that it will be a suitable addition to a public diabetic screening program.

Cost-Utility Analysis of Deep Learning and Trained Human Graders for Diabetic Retinopathy Screening in a Nationwide Program

INTRODUCTION

Deep learning (DL) for screening diabetic retinopathy (DR) has the potential to address limited healthcare resources by enabling expanded access to healthcare. However, there is still limited health economic evaluation, particularly in low- and middle-income countries, on this subject to aid decision-making for DL adoption.

METHODS

In the context of a middle-income country (MIC), using Thailand as a model, we constructed a decision tree-Markov hybrid model to estimate lifetime costs and outcomes of Thailand's national DR screening program via DL and trained human graders (HG). We calculated the incremental cost-effectiveness ratio (ICER) between the two strategies. Sensitivity analyses were performed to probe the influence of modeling parameters.

RESULTS

From a societal perspective, screening with DL was associated with a reduction in costs of ~ US$ 2.70, similar quality-adjusted life-years (QALY) of + 0.0043, and an incremental net monetary benefit of ~ US$ 24.10 in the base case. In sensitivity analysis, DL remained cost-effective even with a price increase from US$ 1.00 to US$ 4.00 per patient at a Thai willingness-to-pay threshold of ~ US$ 4.997 per QALY gained. When further incorporating recent findings suggesting improved compliance to treatment referral with DL, our analysis models effectiveness benefits of ~ US$ 20 to US$ 50 depending on compliance.

CONCLUSION

DR screening using DL in an MIC using Thailand as a model may result in societal cost-savings and similar health outcomes compared with HG. This study may provide an economic rationale to expand DL-based DR screening in MICs as an alternative solution for limited availability of skilled human resources for primary screening, particularly in MICs with similar prevalence of diabetes and low compliance to referrals for treatment.

Artificial Intelligence for Diabetic Retinopathy Screening Using Color Retinal Photographs: From Development to Deployment

Diabetic retinopathy (DR), a leading cause of preventable blindness, is expected to remain a growing health burden worldwide. Screening to detect early sight-threatening lesions of DR can reduce the burden of vision loss; nevertheless, the process requires intensive manual labor and extensive resources to accommodate the increasing number of patients with diabetes. Artificial intelligence (AI) has been shown to be an effective tool which can potentially lower the burden of screening DR and vision loss. In this article, we review the use of AI for DR screening on color retinal photographs in different phases of application, ranging from development to deployment. Early studies of machine learning (ML)-based algorithms using feature extraction to detect DR achieved a high sensitivity but relatively lower specificity. Robust sensitivity and specificity were achieved with the application of deep learning (DL), although ML is still used in some tasks. Public datasets were utilized in retrospective validations of the developmental phases in most algorithms, which require a large number of photographs. Large prospective clinical validation studies led to the approval of DL for autonomous screening of DR although the semi-autonomous approach may be preferable in some real-world settings. There have been few reports on real-world implementations of DL for DR screening. It is possible that AI may improve some real-world indicators for eye care in DR, such as increased screening uptake and referral adherence, but this has not been proven. The challenges in deployment may include workflow issues, such as mydriasis to lower ungradable cases; technical issues, such as integration into electronic health record systems and integration into existing camera systems; ethical issues, such as data privacy and security; acceptance of personnel and patients; and health-economic issues, such as the need to conduct health economic evaluations of using AI in the context of the country. The deployment of AI for DR screening should follow the governance model for AI in healthcare which outlines four main components: fairness, transparency, trustworthiness, and accountability.

Artificial Intelligence Based Analysis of Corneal Confocal Microscopy Images for Diagnosing Peripheral Neuropathy: A Binary Classification Model

Diabetic peripheral neuropathy (DPN) is the leading cause of neuropathy worldwide resulting in excess morbidity and mortality. We aimed to develop an artificial intelligence deep learning algorithm to classify the presence or absence of peripheral neuropathy (PN) in participants with diabetes or pre-diabetes using corneal confocal microscopy (CCM) images of the sub-basal nerve plexus. A modified ResNet-50 model was trained to perform the binary classification of PN (PN+) versus no PN (PN-) based on the Toronto consensus criteria. A dataset of 279 participants (149 PN-, 130 PN+) was used to train (n = 200), validate (n = 18), and test (n = 61) the algorithm, utilizing one image per participant. The dataset consisted of participants with type 1 diabetes (n = 88), type 2 diabetes (n = 141), and pre-diabetes (n = 50). The algorithm was evaluated using diagnostic performance metrics and attribution-based methods (gradient-weighted class activation mapping (Grad-CAM) and Guided Grad-CAM). In detecting PN+, the AI-based DLA achieved a sensitivity of 0.91 (95%CI: 0.79-1.0), a specificity of 0.93 (95%CI: 0.83-1.0), and an area under the curve (AUC) of 0.95 (95%CI: 0.83-0.99). Our deep learning algorithm demonstrates excellent results for the diagnosis of PN using CCM. A large-scale prospective real-world study is required to validate its diagnostic efficacy prior to implementation in screening and diagnostic programmes.

Classification of diabetic retinopathy: Past, present and future

Diabetic retinopathy (DR) is a leading cause of visual impairment and blindness worldwide. Since DR was first recognized as an important complication of diabetes, there have been many attempts to accurately classify the severity and stages of disease. These historical classification systems evolved as understanding of disease pathophysiology improved, methods of imaging and assessing DR changed, and effective treatments were developed. Current DR classification systems are effective, and have been the basis of major research trials and clinical management guidelines for decades. However, with further new developments such as recognition of diabetic retinal neurodegeneration, new imaging platforms such as optical coherence tomography and ultra wide-field retinal imaging, artificial intelligence and new treatments, our current classification systems have significant limitations that need to be addressed. In this paper, we provide a historical review of different classification systems for DR, and discuss the limitations of our current classification systems in the context of new developments. We also review the implications of new developments in the field, to see how they might feature in a future, updated classification.

Digital innovations for retinal care in diabetic retinopathy

AIM

The purpose of this review is to examine the applications of novel digital technology domains for the screening and management of patients with diabetic retinopathy (DR).

METHODS

A PubMed engine search was performed, using the terms "Telemedicine", "Digital health", "Telehealth", "Telescreening", "Artificial intelligence", "Deep learning", "Smartphone", "Triage", "Screening", "Home-based", "Monitoring", "Ophthalmology", "Diabetes", "Diabetic Retinopathy", "Retinal imaging". Full-text English language studies from January 1, 2010, to February 1, 2022, and reference lists were considered for the conceptual framework of this review.

RESULTS

Diabetes mellitus and its eye complications, including DR, are particularly well suited to digital technologies, providing an ideal model for telehealth initiatives and real-world applications. The current development in the adoption of telemedicine, artificial intelligence and remote monitoring as an alternative to or in addition to traditional forms of care will be discussed.

CONCLUSIONS

Advances in digital health have created an ecosystem ripe for telemedicine in the field of DR to thrive. Stakeholders and policymakers should adopt a participatory approach to ensure sustained implementation of these technologies after the COVID-19 pandemic. This article belongs to the Topical Collection "Diabetic Eye Disease", managed by Giuseppe Querques.

Recommendations for the development and use of imaging test sets to investigate the test performance of artificial intelligence in health screening

Rigorous evaluation of artificial intelligence (AI) systems for image classification is essential before deployment into health-care settings, such as screening programmes, so that adoption is effective and safe. A key step in the evaluation process is the external validation of diagnostic performance using a test set of images. We conducted a rapid literature review on methods to develop test sets, published from 2012 to 2020, in English. Using thematic analysis, we mapped themes and coded the principles using the Population, Intervention, and Comparator or Reference standard, Outcome, and Study design framework. A group of screening and AI experts assessed the evidence-based principles for completeness and provided further considerations. From the final 15 principles recommended here, five affect population, one intervention, two comparator, one reference standard, and one both reference standard and comparator. Finally, four are appliable to outcome and one to study design. Principles from the literature were useful to address biases from AI; however, they did not account for screening specific biases, which we now incorporate. The principles set out here should be used to support the development and use of test sets for studies that assess the accuracy of AI within screening programmes, to ensure they are fit for purpose and minimise bias.

The effectiveness of artificial intelligence-based automated grading and training system in education of manual detection of diabetic retinopathy

Background

The purpose of this study is to develop an artificial intelligence (AI)-based automated diabetic retinopathy (DR) grading and training system from a real-world diabetic dataset of China, and in particular, to investigate its effectiveness as a learning tool of DR manual grading for medical students.

Methods

We developed an automated DR grading and training system equipped with an AI-driven diagnosis algorithm to highlight highly prognostic related regions in the input image. Less experienced prospective physicians received pre- and post-training tests by the AI diagnosis platform. Then, changes in the diagnostic accuracy of the participants were evaluated.

Results

We randomly selected 8,063 cases diagnosed with DR and 7,925 with non-DR fundus images from type 2 diabetes patients. The automated DR grading system we developed achieved accuracy, sensitivity/specificity, and AUC values of 0.965, 0.965/0.966, and 0.980 for moderate or worse DR (95 percent CI: 0.976-0.984). When the graders received assistance from the output of the AI system, the metrics were enhanced in varying degrees. The automated DR grading system helped to improve the accuracy of human graders, i.e., junior residents and medical students, from 0.947 and 0.915 to 0.978 and 0.954, respectively.

Conclusion

The AI-based systemdemonstrated high diagnostic accuracy for the detection of DR on fundus images from real-world diabetics, and could be utilized as a training aid system for trainees lacking formal instruction on DR management.

Evaluation of a New Neural Network Classifier for Diabetic Retinopathy

BACKGROUND

Medical image segmentation is a well-studied subject within the field of image processing. The goal of this research is to create an AI retinal screening grading system that is both accurate and fast. We introduce a new segmentation network which achieves state-of-the-art results on semantic segmentation of color fundus photographs. By applying the net-work to identify anatomical markers of diabetic retinopathy (DR) and diabetic macular edema (DME), we collect sufficient information to classify patients by grades R0 and R1 or above, M0 and M1.

METHODS

The AI grading system was trained on screening data to evaluate the presence of DR and DME. The core algorithm of the system is a deep learning network that segments relevant anatomical features in a retinal image. Patients were graded according to the standard NHS Diabetic Eye Screening Program feature-based grading protocol.

RESULTS

The algorithm performance was evaluated with a series of 6,981 patient retinal images from routine diabetic eye screenings. It correctly predicted 98.9% of retinopathy events and 95.5% of maculopathy events. Non-disease events prediction rate was 68.6% for retinopathy and 81.2% for maculopathy.

CONCLUSION

This novel deep learning model was trained and tested on patient data from annual diabetic retinopathy screenings can classify with high accuracy the DR and DME status of a person with diabetes. The system can be easily reconfigured according to any grading protocol, without running a long AI training procedure. The incorporation of the AI grading system can increase the graders' productivity and improve the final outcome accuracy of the screening process.

Artificial Intelligence Detection of Diabetic Retinopathy: Subgroup Comparison of the EyeArt System with Ophthalmologists' Dilated Exams

Objective

To compare general ophthalmologists, retina specialists, and the EyeArt Artificial Intelligence (AI) system to the clinical reference standard for detecting more than mild diabetic retinopathy (mtmDR).

Design

Prospective, pivotal, multicenter trial conducted from April 2017 to May 2018.

Participants

Participants were aged ≥ 18 years who had diabetes mellitus and underwent dilated ophthalmoscopy. A total of 521 of 893 participants met these criteria and completed the study protocol.

Testing

Participants underwent 2-field fundus photography (macula centered, disc centered) for the EyeArt system, dilated ophthalmoscopy, and 4-widefield stereoscopic dilated fundus photography for reference standard grading.

Main Outcome Measures

For mtmDR detection, sensitivity and specificity of EyeArt gradings of 2-field, fundus photographs and ophthalmoscopy grading versus a rigorous clinical reference standard comprising Reading Center grading of 4-widefield stereoscopic dilated fundus photographs using the ETDRS severity scale. The AI system provided automatic eye-level results regarding mtmDR.

Results

Overall, 521 participants (999 eyes) at 10 centers underwent dilated ophthalmoscopy: 406 by nonretina and 115 by retina specialists. Reading Center graded 207 positive and 792 eyes negative for mtmDR. Of these 999 eyes, 26 eyes were ungradable by the EyeArt system, leaving 973 eyes with both EyeArt and Reading Center gradings. Retina specialists correctly identified 22 of 37 eyes as positive (sensitivity 59.5%) and 182 of 184 eyes as negative (specificity 98.9%) for mtmDR versus the EyeArt AI system that identified 36 of 37 as positive (sensitivity 97%) and 162 of 184 eyes as negative (specificity of 88%) for mtmDR. General ophthalmologists correctly identified 35 of 170 eyes as positive (sensitivity 20.6%) and 607 of 608 eyes as negative (specificity 99.8%) for mtmDR compared with the EyeArt AI system that identified 164 of 170 as positive (sensitivity 96.5%) and 525 of 608 eyes as negative (specificity 86%) for mtmDR.

Conclusions

The AI system had a higher sensitivity for detecting mtmDR than either general ophthalmologists or retina specialists compared with the clinical reference standard. It can potentially serve as a low-cost point-of-care diabetic retinopathy detection tool and help address the diabetic eye screening burden.

A New Deep Learning Algorithm with Activation Mapping for Diabetic Retinopathy: Backtesting after 10 Years of Tele-Ophthalmology

We report the development of a deep learning algorithm (AI) to detect signs of diabetic retinopathy (DR) from fundus images. For this, we use a ResNet-50 neural network with a double resolution, the addition of Squeeze–Excitation blocks, pre-trained in ImageNet, and trained for 50 epochs using the Adam optimizer. The AI-based algorithm not only classifies an image as pathological or not but also detects and highlights those signs that allow DR to be identified. For development, we have used a database of about half a million images classified in a real clinical environment by family doctors (FDs), ophthalmologists, or both. The AI was able to detect more than 95% of cases worse than mild DR and had 70% fewer misclassifications of healthy cases than FDs. In addition, the AI was able to detect DR signs in 1258 patients before they were detected by FDs, representing 7.9% of the total number of DR patients detected by the FDs. These results suggest that AI is at least comparable to the evaluation of FDs. We suggest that it may be useful to use signaling tools such as an aid to diagnosis rather than an AI as a stand-alone tool.

Artificial Intelligence Algorithms in Diabetic Retinopathy Screening

PURPOSE OF REVIEW

In this review, we focus on artificial intelligence (AI) algorithms for diabetic retinopathy (DR) screening and risk stratification and factors to consider when implementing AI algorithms in the clinic.

RECENT FINDINGS

AI algorithms have been adopted, and have received regulatory approval, for automated detection of referable DR with clinically acceptable diagnostic performance. While these metrics are an important first step, performance metrics that go beyond measures of technical accuracy are needed to fully evaluate the impact of AI algorithm on patient outcomes. Recent advances in AI present an exciting opportunity to improve patient care. Using DR as an example, we have reviewed factors to consider in the implementation of AI algorithms in real-world clinical practice. These include real-world evaluation of safety, efficacy, and equity (bias); impact on patient outcomes; ethical, logistical, and regulatory factors.

Multicenter Evaluation of Diagnostic Circulating Biomarkers to Detect Sight-Threatening Diabetic Retinopathy

Question

Can circulating serum biomarkers distinguish people with sight-threatening diabetic retinopathy (STDR) from those with no DR?

Findings

This multicenter cross-sectional study of 538 participants found an incremental benefit of circulating cystatin C beyond the standard clinical variables in discriminating STDR from no DR. Cystatin C outperformed 12 other biomarkers found to be distinguished in STDR in previous research.

Meaning

Results of this study suggest the consideration of circulating cystatin C levels as a triage test in prioritizing people with type 2 diabetes from the community for retinal screening in resource-restricted settings.

Importance

It is a global challenge to provide regular retinal screening for all people with diabetes to detect sight-threatening diabetic retinopathy (STDR).

Objective

To determine if circulating biomarkers could be used to prioritize people with type 2 diabetes for retinal screening to detect STDR.

Design, Setting, and Participants

This cross-sectional study collected data from October 22, 2018, to December 31, 2021. All laboratory staff were masked to the clinical diagnosis, assigned a study cohort, and provided with the database containing the clinical data. This was a multicenter study conducted in parallel in 3 outpatient ophthalmology clinics in the UK and 2 centers in India. Adults 40 years and older were categorized into 4 groups: (1) no history of diabetes, (2) type 2 diabetes of at least 5 years’ duration with no evidence of DR, (3) nonproliferative DR with diabetic macular edema (DME), or (4) proliferative DR. STDR comprised groups 3 and 4.

Exposures

Thirteen previously verified biomarkers were measured using enzyme-linked immunosorbent assay.

Main Outcomes and Measures

Severity of DR and presence of DME were diagnosed using fundus photographs and optical coherence tomography. Weighted logistic regression and receiver operating characteristic curve analysis (ROC) were performed to identify biomarkers that discriminate STDR from no DR beyond the standard clinical parameters of age, disease duration, ethnicity (in the UK) and hemoglobin A_1c.

Results

A total of 538 participants (mean [SD] age, 60.8 [9.8] years; 319 men [59.3%]) were recruited into the study. A total of 264 participants (49.1%) were from India (group 1, 54 [20.5%]; group 2, 53 [20.1%]; group 3, 52 [19.7%]; group 4, 105 [39.8%]), and 274 participants (50.9%) were from the UK (group 1, 50 [18.2%]; group 2, 70 [25.5%]; group 3, 55 [20.1%]; group 4, 99 [36.1%]). ROC analysis (no DR vs STDR) showed that in addition to age, disease duration, ethnicity (in the UK) and hemoglobin A_1c, inclusion of cystatin C had near-acceptable discrimination power in both countries (area under the receiver operating characteristic curve [AUC], 0.779; 95% CI, 0.700-0.857 in 215 patients in the UK with complete data; AUC, 0.696; 95% CI, 0.602-0.791 in 208 patients in India with complete data).

Conclusions and Relevance

Results of this cross-sectional study suggest that serum cystatin C had good discrimination power in the UK and India. Circulating cystatin-C levels may be considered as a test to identify those who require prioritization for retinal screening for STDR.