An international comparison of retinopathy of prematurity grading performance within the Benefits of Oxygen Saturation Targeting II trials

P Score: A Reference Image-Based Clinical Grading Scale for Vascular Change in Retinopathy of Prematurity

PURPOSE

The International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), acknowledged that plus-like retinopathy of prematurity (ROP) vascular changes occurs along a spectrum. Historically, clinician-experts demonstrate variable agreement for plus diagnosis. We developed a 9-photograph reference image set for grading plus-like changes and compared intergrader agreement of the set with standard grading with no plus, preplus, and plus disease.

DESIGN

Retinal photographic grading and expert consensus opinion.

PARTICIPANTS

The development set included 34 international ICROP3 committee members. The validation set included 30 ophthalmologists with ROP expertise (15 ICROP3 committee members and 15 non-ICROP3 members) METHODS: Nine ROP fundus images (P1 through P9) representing increasing degrees of zone I vascular tortuosity and dilation, based on the 34 ICROP3 committee members' gradings and consensus image reviews, were used to establish standard photographs for the plus (P) score. Study participants graded 150 fundus photographs 2 ways, separated by a 1-week washout period: (1) no plus, preplus, or plus disease and (2) choosing the closest P score image.

MAIN OUTCOME MEASURES

Intergrader agreement measured by intraclass correlation coefficient.

RESULTS

Intergrader agreement was higher using the P score (intraclass correlation coefficient, 0.75; 95% confidence interval, 0.71-0.79) than no plus, preplus, or plus disease (intraclass correlation coefficient, 0.67; 95% confidence interval, 0.62-0.72). Mean ± standard deviation P scores for images with mode gradings of no plus, preplus, and plus disease were 2.5 ± 0.7, 4.8 ± 0.8, and 7.4 ± 0.8, respectively.

CONCLUSIONS

Intergrader agreement of plus-like vascular change in ROP using the P score is high. We now incorporate this 9-image reference set into ICROP3 for use in clinician daily practice alongside zone, stage, and plus assessment. P score is not yet meant to replace plus diagnosis for treatment decisions, but its use at our institutions has permitted better comparison between examinations for progression and regression, communication between examiners, and documentation of vascular change without fundus imaging. P score also could provide more detailed ROP classification for clinical trials, consistent with the spectrum of plus-like change that is now formally part of the International Classification of Retinopathy of Prematurity.

FINANCIAL DISCLOSURE(S)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Reduction of pathological retinal neovascularization, vessel obliteration, and artery tortuosity by PEDF protein in an oxygen-induced ischemic retinopathy rat model

Retinopathy of prematurity (ROP) is a severe retinal disease in premature infants characterized by pathological neovascularization, obliteration of retinal vessels and increased vessel tortuosity. Currently, there are no completely satisfactory treatments for ROP. Pigment epithelium-derived factor (PEDF), a potent inhibitor of angiogenesis, appears late in gestation and its deficiency may be linked to development of ROP. This study investigates the preclinical efficacy of PEDF protein alone or in combination with VEGF antagonists for treating ROP. The safety of PEDF protein in the rat eye was assessed using functional in vivo measurements and histology. The efficacy of intravitreal injections (IVI) of various treatments was evaluated in a rat oxygen-induced retinopathy (OIR) model using in vivo imaging and flatmount analyses. No functional or histological side-effects were found in rat eyes after intravitreal PEDF protein injection. PEDF protein alone or combined with anti-VEGF drugs significantly reduced pathological neovascularization and vessel obliteration, comparable to the effects of anti-VEGF drugs alone. Regarding arterial tortuosity, treatment with a combination of PEDF, and VEGF antagonist was more effective than treatment with anti-VEGF alone. IVI of PEDF protein is safe. PEDF protein alone or combined with VEGF antagonists shows similar efficacy in reducing pathological neovascularization and vessel obliteration as anti-VEGF agents. Furthermore, only treatments involving PEDF protein, alone or with VEGF antagonists, significantly improved the quality of retinal vasculature. Thus, PEDF protein alone or combined with anti-VEGF agents presents a promising alternative to current anti-VEGF treatments for ROP.

Implementation of optical coherence tomography in retinopathy of prematurity screening

PURPOSE OF REVIEW

In this review, we explore the investigational applications of optical coherence tomography (OCT) in retinopathy of prematurity (ROP), the insights they have delivered thus far, and key milestones for its integration into the standard of care.

RECENT FINDINGS

While OCT has been widely integrated into clinical management of common retinal diseases, its use in pediatric contexts has been undermined by limitations in ergonomics, image acquisition time, and field of view. Recently, investigational handheld OCT devices have been reported with advancements including ultra-widefield view, noncontact use, and high-speed image capture permitting real-time en face visualization. These developments are compelling for OCT as a more objective alternative with reduced neonatal stress compared to indirect ophthalmoscopy and/or fundus photography as a means of classifying and monitoring ROP.

SUMMARY

OCT may become a viable modality in management of ROP. Ongoing innovation surrounding handheld devices should aim to optimize patient comfort and image resolution in the retinal periphery. Future clinical investigations may seek to objectively characterize features of peripheral stage and explore novel biomarkers of disease activity.

Retinal blood flow velocities in infants with retinopathy of prematurity after intravitreal administration of bevacizumab

BACKGROUND/OBJECTIVES

Progression of retinopathy of prematurity (ROP) is associated with increased retinal blood flow velocities. We investigated changes of central retinal arterial and venous blood flow after intravitreal administration of bevacizumab.

SUBJECTS/METHODS

Prospective observational study using serial ultrasound Doppler imaging in preterm infants with bevacizumab-treated ROP. Eyes were examined 1 [0-2] days before injection (median [interquartile range]), and at three time points after injection (1 [1-2] days, 6 [3-8] days, and 17 [9-28] days). Preterm infants with ROP stage 2 displaying spontaneous regression served as controls.

RESULTS

In 21 eyes of 12 infants with bevacizumab-treated ROP, peak arterial systolic velocity declined from 13.6 [11.0-16.3] cm/s prior to intravitreal bevacizumab to 11.2 [9.4-13.9] cm/s, 10.6 [9.2-13.3] cm/s and 9.3 [8.2-11.0] cm/s at discharge (p  =  .002). There was also a decline of the arterial velocity time integral (from 3.1 [2.3-3.9] cm to 2.9 [2.4-3.5], 2.7 [2.3-3.2] cm and 2.2 [2.0-2.7], p  =  .021) and mean velocity in the central retinal vein (from 4.5 [3.6-5.8] cm/s to 3.7 [2.6-4.1] cm/s, 3.5 [3.0-4.3] cm/s, and 3.2 [2.8-4.6] cm/s, p  =  .012). Arterial end-diastolic velocity and resistance index remained unchanged. Blood flow velocities in bevacizumab-treated eyes examined before injection were significantly higher than those measured in untreated eyes that ultimately showed spontaneous regression of ROP. Sequential examinations in these controls did not reveal any declines of retinal blood flow velocities.

CONCLUSION

Increased retinal arterial and venous blood flow velocities in infants with threshold ROP decline following intravitreal bevacizumab injection.

Retinal detachment following retinopathy of prematurity in France: Screening and treatment pathways

AIM

Preterm children are highly vulnerable to sensorial impairments through Retinopathy Of Prematurity (ROP). The objective was to determine whether some cases of ROP requiring surgery could be secondary to deficiencies in care pathways.

METHODS

Descriptive study of neonatal characteristics and the screening/treatment pathways of children treated for stage ≥4A ROP from 2009 to 2020 in a referral unit in France.

RESULTS

Twenty-five preterm children (44 eyes) were included: median gestational age was 25 weeks, and median birthweight was 700 grams. Eighty-four per cent had received at least one fundus examination, 50% of which were completed on time. At the time of retinal detachment diagnosis, only 36% of the children had received laser or anti-vascular endothelial growth factor (VEGF) intra-vitreal injection. ROP stage was only reported in 8%, and the zone or type was reported in 16% of the files.

CONCLUSION

The risk of blindness and the effectiveness of laser or anti-VEGF treatment highlight the need to enhance screening and treatment practices in France.

Development and validation of an artificial intelligence based screening tool for detection of retinopathy of prematurity in a South Indian population

Purpose

The primary objective of this study was to develop and validate an AI algorithm as a screening tool for the detection of retinopathy of prematurity (ROP).

Participants

Images were collected from infants enrolled in the KIDROP tele-ROP screening program.

Methods

We developed a deep learning (DL) algorithm with 227,326 wide-field images from multiple camera systems obtained from the KIDROP tele-ROP screening program in India over an 11-year period. 37,477 temporal retina images were utilized with the dataset split into train (n = 25,982, 69.33%), validation (n = 4,006, 10.69%), and an independent test set (n = 7,489, 19.98%). The algorithm consists of a binary classifier that distinguishes between the presence of ROP (Stages 1-3) and the absence of ROP. The image labels were retrieved from the daily registers of the tele-ROP program. They consist of per-eye diagnoses provided by trained ROP graders based on all images captured during the screening session. Infants requiring treatment and a proportion of those not requiring urgent referral had an additional confirmatory diagnosis from an ROP specialist.

Results

Of the 7,489 temporal images analyzed in the test set, 2,249 (30.0%) images showed the presence of ROP. The sensitivity and specificity to detect ROP was 91.46% (95% CI: 90.23%-92.59%) and 91.22% (95% CI: 90.42%-91.97%), respectively, while the positive predictive value (PPV) was 81.72% (95% CI: 80.37%-83.00%), negative predictive value (NPV) was 96.14% (95% CI: 95.60%-96.61%) and the AUROC was 0.970.

Conclusion

The novel ROP screening algorithm demonstrated high sensitivity and specificity in detecting the presence of ROP. A prospective clinical validation in a real-world tele-ROP platform is under consideration. It has the potential to lower the number of screening sessions required to be conducted by a specialist for a high-risk preterm infant thus significantly improving workflow efficiency.

Increased risk of retinopathy of prematurity since increased O2 saturation targets: A multi-centre study

BACKGROUND/AIMS

Retinopathy of prematurity (ROP) is a leading cause of visual impairment in premature neonates. The BOOST II, SUPPORT and COT trials recommended increasing O2 saturation targets for pre-term neonates to reduce mortality; however, this is a risk factor for ROP. We aimed to determine whether these targets increased prevalence of ROP among pre-term neonates and higher risk groups.

METHODS

Retrospective cohort study conducted using data from the Australian and New Zealand Neonatal Network. 17 298 neonate cohort born 2012-2018 at <32 weeks' GA and/or <1500 g BW was analysed. Adjusted odds ratios (aORs) were calculated for post-2015 risk of: any ROP; ROP ≥ Stage 2; and treated ROP. Sub-analysis stratified at <28 GA, < 26 weeks' GA, <1500 g BW and <1000 g BW was performed.

RESULTS

Risk of any ROP increased in the post-2015 group (aOR = 1.23, 95% confidence interval (CI) = 1.14-1.32), <28 weeks' GA (aOR = 1.31, 95% CI = 1.17-1.46), <26 weeks (aOR = 1.57, 95% CI = 1.28-1.91), <1500 g (aOR = 1.24, 95% CI = 1.14-1.34) and <1000 g (aOR = 1.34, 95% CI = 1.20-1.50). ROP ≥ Stage 2 increased at <28 weeks (aOR = 1.30, 95% CI = 1.16-1.46), <26 weeks (aOR = 1.57, 95% CI = 1.28-1.91), <1500 g (aOR = 1.18, 95% CI = 1.08-1.30), and <1000 g (aOR = 1.26, 95% CI = 1.13-1.42).

CONCLUSION

O2 therapy guidelines since 2015 have resulted in decreased mortality but increased risk of ROP. Individualised NICU adjustments of ROP screening/follow-up methods are necessary to address the clinical burden.

Multicenter Validation of Deep Learning Algorithm ROP.AI for the Automated Diagnosis of Plus Disease in ROP

Purpose

Retinopathy of prematurity (ROP) is a sight-threatening vasoproliferative retinal disease affecting premature infants. The detection of plus disease, a severe form of ROP requiring treatment, remains challenging owing to subjectivity, frequency, and time intensity of retinal examinations. Recent artificial intelligence (AI) algorithms developed to detect plus disease aims to alleviate these challenges; however, they have not been tested against a diverse neonatal population. Our study aims to validate ROP.AI, an AI algorithm developed from a single cohort, against a multicenter Australian cohort to determine its performance in detecting plus disease.

Methods

Retinal images captured during routine ROP screening from May 2021 to February 2022 across five major tertiary centers throughout Australia were collected and uploaded to ROP.AI. AI diagnostic output was compared with one of five ROP experts. Sensitivity, specificity, negative predictive value, and area under the receiver operator curve were determined.

Results

We collected 8052 images. The area under the receiver operator curve for the diagnosis of plus disease was 0.75. ROP.AI achieved 84% sensitivity, 43% specificity, and 96% negative predictive value for the detection of plus disease after operating point optimization.

Conclusions

ROP.AI was able to detect plus disease in an external, multicenter cohort despite being trained from a single center. Algorithm performance was demonstrated without preprocessing or augmentation, simulating real-world clinical applicability. Further training may improve generalizability for clinical implementation.

Translational Relevance

These results demonstrate ROP.AI's potential as a screening tool for the detection of plus disease in future clinical practice and provides a solution to overcome current diagnostic challenges.

Evaluation of an Artificial Intelligence System for Retinopathy of Prematurity Screening in Nepal and Mongolia

PURPOSE

To evaluate the performance of a deep learning (DL) algorithm for retinopathy of prematurity (ROP) screening in Nepal and Mongolia.

DESIGN

Retrospective analysis of prospectively collected clinical data.

PARTICIPANTS

Clinical information and fundus images were obtained from infants in 2 ROP screening programs in Nepal and Mongolia.

METHODS

Fundus images were obtained using the Forus 3nethra neo (Forus Health) in Nepal and the RetCam Portable (Natus Medical, Inc.) in Mongolia. The overall severity of ROP was determined from the medical record using the International Classification of ROP (ICROP). The presence of plus disease was determined independently in each image using a reference standard diagnosis. The Imaging and Informatics for ROP (i-ROP) DL algorithm was trained on images from the RetCam to classify plus disease and to assign a vascular severity score (VSS) from 1 through 9.

MAIN OUTCOME MEASURES

Area under the receiver operating characteristic curve and area under the precision-recall curve for the presence of plus disease or type 1 ROP and association between VSS and ICROP disease category.

RESULTS

The prevalence of type 1 ROP was found to be higher in Mongolia (14.0%) than in Nepal (2.2%; P < 0.001) in these data sets. In Mongolia (RetCam images), the area under the receiver operating characteristic curve for examination-level plus disease detection was 0.968, and the area under the precision-recall curve was 0.823. In Nepal (Forus images), these values were 0.999 and 0.993, respectively. The ROP VSS was associated with ICROP classification in both datasets (P < 0.001). At the population level, the median VSS was found to be higher in Mongolia (2.7; interquartile range [IQR], 1.3-5.4]) as compared with Nepal (1.9; IQR, 1.2-3.4; P < 0.001).

CONCLUSIONS

These data provide preliminary evidence of the effectiveness of the i-ROP DL algorithm for ROP screening in neonatal populations in Nepal and Mongolia using multiple camera systems and are useful for consideration in future clinical implementation of artificial intelligence-based ROP screening in low- and middle-income countries.

Federated Learning in Ocular Imaging: Current Progress and Future Direction

Advances in artificial intelligence deep learning (DL) have made tremendous impacts on the field of ocular imaging over the last few years. Specifically, DL has been utilised to detect and classify various ocular diseases on retinal photographs, optical coherence tomography (OCT) images, and OCT-angiography images. In order to achieve good robustness and generalisability of model performance, DL training strategies traditionally require extensive and diverse training datasets from various sites to be transferred and pooled into a "centralised location". However, such a data transferring process could raise practical concerns related to data security and patient privacy. Federated learning (FL) is a distributed collaborative learning paradigm which enables the coordination of multiple collaborators without the need for sharing confidential data. This distributed training approach has great potential to ensure data privacy among different institutions and reduce the potential risk of data leakage from data pooling or centralisation. This review article aims to introduce the concept of FL, provide current evidence of FL in ocular imaging, and discuss potential challenges as well as future applications.

Retinopathy of prematurity screening: A narrative review of current programs, teleophthalmology, and diagnostic support systems

PURPOSE

Neonatal care in middle-income countries has improved over the last decade, leading to a "third epidemic" of retinopathy of prematurity (ROP). Without concomitant improvements in ROP screening infrastructure, reduction of ROP-associated visual loss remains a challenge worldwide. The emergence of teleophthalmology screening programs and artificial intelligence (AI) technologies represents promising methods to address this growing unmet demand in ROP screening. An improved understanding of current ROP screening programs may inform the adoption of these novel technologies in ROP care.

METHODS

A critical narrative review of the literature was carried out. Publications that were representative of established or emerging ROP screening programs in high-, middle-, and low-income countries were selected for review. Screening programs were reviewed for inclusion criteria, screening frequency and duration, modality, and published sensitivity and specificity.

RESULTS

Screening inclusion criteria, including age and birth weight cutoffs, showed significant heterogeneity globally. Countries of similar income tend to have similar criteria. Three primary screening modalities including binocular indirect ophthalmoscopy (BIO), wide-field digital retinal imaging (WFDRI), and teleophthalmology were identified and reviewed. BIO has documented limitations in reduced interoperator agreement, scalability, and geographical access barriers, which are mitigated in part by WFDRI. Teleophthalmology screening may address limitations in ROP screening workforce distribution and training. Opportunities for AI technologies were identified in the context of these limitations, including interoperator reliability and possibilities for point-of-care diagnosis.

CONCLUSION

Limitations in the current ROP screening include scalability, geographical access, and high screening burden with low treatment yield. These may be addressable through increased adoption of teleophthalmology and AI technologies. As the global incidence of ROP continues to increase, implementation of these novel modalities requires greater consideration.

Retinopathy of Prematurity: A Global Perspective and Recent Developments

Retinopathy of prematurity (ROP) is a significant cause of potentially preventable blindness in preterm infants worldwide. It is a disease caused by abnormal retinal vascularization that, if not detected and treated in a timely manner, can lead to retinal detachment and severe long term vision impairment. Neonatologists and pediatricians have an important role in the prevention, detection, and management of ROP. Geographic differences in the epidemiology of ROP have been seen globally over the last several decades because of regional differences in neonatal care. Our understanding of the pathophysiology, risk factors, prevention, screening, diagnosis, and treatment of ROP have also evolved over the years. New technological advances are now allowing for the incorporation of telemedicine and artificial intelligence in the management of ROP. In this comprehensive update, we provide a comprehensive review of pathophysiology, classification, diagnosis, global screening, and treatment of ROP. Key historical milestones as well as touching upon the very recent updates to the ROP classification system and technological advances in the field of artificial intelligence and ROP will also be discussed.

ADS-Net: attention-awareness and deep supervision based network for automatic detection of retinopathy of prematurity

Retinopathy of prematurity (ROP) is a proliferative vascular disease, which is one of the most dangerous and severe ocular complications in premature infants. Automatic ROP detection system can assist ophthalmologists in the diagnosis of ROP, which is safe, objective, and cost-effective. Unfortunately, due to the large local redundancy and the complex global dependencies in medical image processing, it is challenging to learn the discriminative representation from ROP-related fundus images. To bridge this gap, a novel attention-awareness and deep supervision based network (ADS-Net) is proposed to detect the existence of ROP (Normal or ROP) and 3-level ROP grading (Mild, Moderate, or Severe). First, to balance the problems of large local redundancy and complex global dependencies in images, we design a multi-semantic feature aggregation (MsFA) module based on self-attention mechanism to take full advantage of convolution and self-attention, generating attention-aware expressive features. Then, to solve the challenge of difficult training of deep model and further improve ROP detection performance, we propose an optimization strategy with deeply supervised loss. Finally, the proposed ADS-Net is evaluated on ROP screening and grading tasks with per-image and per-examination strategies, respectively. In terms of per-image classification pattern, the proposed ADS-Net achieves 0.9552 and 0.9037 for Kappa index in ROP screening and grading, respectively. Experimental results demonstrate that the proposed ADS-Net generally outperforms other state-of-the-art classification networks, showing the effectiveness of the proposed method.

Identification of treatment-warranted retinopathy of prematurity by neonatal nurse specialist

OBJECTIVE

To determine the accuracy in the identification of infants with treatment-warranted retinopathy of prematurity (ROP) by a trained and experienced ROP neonatal nurse specialist compared with skilled ophthalmologists.

METHODS

A single-centre, prospective, blinded, agreement study was performed on a cohort of infants undergoing ROP screening. An experienced ROP neonatal nurse specialist obtained retinal images using a wide field digital retinal imaging system (WFDRI) on 127 infants and identified those with treatment-warranted ROP. This interpretation was compared with the interpretation of the same images by skilled ophthalmologists. The accuracy of the ROP nurse specialist's interpretation was assessed for sensitivity and specificity compared with the gold standard interpretation by the ophthalmologists.

RESULTS

The ROP nurse specialist performed 345 ROP screens on both eyes of 127 infants. The mean (SD) gestation age (weeks) and birth weight (g) of the infants screened was 26.8 (2.8) and 929 (327), respectively. The nurse specialist correctly identified all 8 infants with treatment-warranted ROP and 118/119 infants without. The sensitivity and specificity (95% CI) of ROP screening episodes were 100% (63% to 100%) and 99.7% (98.4% to 100.0%), respectively.

CONCLUSION

A trained and experienced ROP neonatal nurse specialist can correctly identify infants with treatment-warranted ROP using WFDRI. Further work is required to examine the generalisability of this finding and its impact on ROP screening services.

International Classification of Retinopathy of Prematurity, Third Edition

PURPOSE

The International Classification of Retinopathy of Prematurity is a consensus statement that creates a standard nomenclature for classification of retinopathy of prematurity (ROP). It was initially published in 1984, expanded in 1987, and revisited in 2005. This article presents a third revision, the International Classification of Retinopathy of Prematurity, Third Edition (ICROP3), which is now required because of challenges such as: (1) concerns about subjectivity in critical elements of disease classification; (2) innovations in ophthalmic imaging; (3) novel pharmacologic therapies (e.g., anti-vascular endothelial growth factor agents) with unique regression and reactivation features after treatment compared with ablative therapies; and (4) recognition that patterns of ROP in some regions of the world do not fit neatly into the current classification system.

DESIGN

Review of evidence-based literature, along with expert consensus opinion.

PARTICIPANTS

International ROP expert committee assembled in March 2019 representing 17 countries and comprising 14 pediatric ophthalmologists and 20 retinal specialists, as well as 12 women and 22 men.

METHODS

The committee was initially divided into 3 subcommittees-acute phase, regression or reactivation, and imaging-each of which used iterative videoconferences and an online message board to identify key challenges and approaches. Subsequently, the entire committee used iterative videoconferences, 2 in-person multiday meetings, and an online message board to develop consensus on classification.

MAIN OUTCOME MEASURES

Consensus statement.

RESULTS

The ICROP3 retains current definitions such as zone (location of disease), stage (appearance of disease at the avascular-vascular junction), and circumferential extent of disease. Major updates in the ICROP3 include refined classification metrics (e.g., posterior zone II, notch, subcategorization of stage 5, and recognition that a continuous spectrum of vascular abnormality exists from normal to plus disease). Updates also include the definition of aggressive ROP to replace aggressive-posterior ROP because of increasing recognition that aggressive disease may occur in larger preterm infants and beyond the posterior retina, particularly in regions of the world with limited resources. ROP regression and reactivation are described in detail, with additional description of long-term sequelae.

CONCLUSIONS

These principles may improve the quality and standardization of ROP care worldwide and may provide a foundation to improve research and clinical care.

Evaluation of a Deep Learning-Derived Quantitative Retinopathy of Prematurity Severity Scale

PURPOSE

To evaluate the clinical usefulness of a quantitative deep learning-derived vascular severity score for retinopathy of prematurity (ROP) by assessing its correlation with clinical ROP diagnosis and by measuring clinician agreement in applying a novel scale.

DESIGN

Analysis of existing database of posterior pole fundus images and corresponding ophthalmoscopic examinations using 2 methods of assigning a quantitative scale to vascular severity.

PARTICIPANTS

Images were from clinical examinations of patients in the Imaging and Informatics in ROP Consortium. Four ophthalmologists and 1 study coordinator evaluated vascular severity on a scale from 1 to 9.

METHODS

A quantitative vascular severity score (1-9) was applied to each image using a deep learning algorithm. A database of 499 images was developed for assessment of interobserver agreement.

MAIN OUTCOME MEASURES

Distribution of deep learning-derived vascular severity scores with the clinical assessment of zone (I, II, or III), stage (0, 1, 2, or 3), and extent (<3 clock hours, 3-6 clock hours, and >6 clock hours) of stage 3 evaluated using multivariate linear regression and weighted κ values and Pearson correlation coefficients for interobserver agreement on a 1-to-9 vascular severity scale.

RESULTS

For deep learning analysis, a total of 6344 clinical examinations were analyzed. A higher deep learning-derived vascular severity score was associated with more posterior disease, higher disease stage, and higher extent of stage 3 disease (P < 0.001 for all). For a given ROP stage, the vascular severity score was higher in zone I than zones II or III (P < 0.001). Multivariate regression found zone, stage, and extent all were associated independently with the severity score (P < 0.001 for all). For interobserver agreement, the mean ± standard deviation weighted κ value was 0.67 ± 0.06, and the Pearson correlation coefficient ± standard deviation was 0.88 ± 0.04 on the use of a 1-to-9 vascular severity scale.

CONCLUSIONS

A vascular severity scale for ROP seems feasible for clinical adoption; corresponds with zone, stage, extent of stage 3, and plus disease; and facilitates the use of objective technology such as deep learning to improve the consistency of ROP diagnosis.

Variability in Plus Disease Identified Using a Deep Learning-Based Retinopathy of Prematurity Severity Scale

PURPOSE

Retinopathy of prematurity is a leading cause of childhood blindness worldwide, but clinical diagnosis is subjective, which leads to treatment differences. Our goal was to determine objective differences in the diagnosis of plus disease between clinicians using an automated retinopathy of prematurity (ROP) vascular severity score.

DESIGN

This retrospective cohort study used data from the Imaging and Informatics in ROP Consortium, which comprises 8 tertiary care centers in North America. Fundus photographs of all infants undergoing ROP screening examinations between July 1, 2011, and December 31, 2016, were obtained.

PARTICIPANTS

Infants meeting ROP screening criteria who were diagnosed with plus disease and treatment initiated by an examining physician based on ophthalmoscopic examination results.

METHODS

An ROP severity score (1-9) was generated for each image using a deep learning (DL) algorithm.

MAIN OUTCOME MEASURES

The mean, median, and range of ROP vascular severity scores overall and for each examiner when the diagnosis of plus disease was made.

RESULTS

A total of 5255 clinical examinations in 871 babies were analyzed. Of these, 168 eyes were diagnosed with plus disease by 11 different examiners and were included in the study. The mean ± standard deviation vascular severity score for patients diagnosed with plus disease was 7.4 ± 1.9, median was 8.5 (interquartile range, 5.8-8.9), and range was 1.1 to 9.0. Within some examiners, variability in the level of vascular severity diagnosed as plus disease was present, and 1 examiner routinely diagnosed plus disease in patients with less severe disease than the other examiners (P < 0.01).

CONCLUSIONS

We observed variability both between and within examiners in the diagnosis of plus disease using DL. Prospective evaluation of clinical trial data using an objective measurement of vascular severity may help to define better the minimum necessary level of vascular severity for the diagnosis of plus disease or how other clinical features such as zone, stage, and extent of peripheral disease ought to be incorporated in treatment decisions.

Application of Artificial Intelligence in Targeting Retinal Diseases

Retinal diseases affect an increasing number of patients worldwide because of the aging population. Request for diagnostic imaging in ophthalmology is ramping up, while the number of specialists keeps shrinking. Cutting-edge technology embedding artificial intelligence (AI) algorithms are thus advocated to help ophthalmologists perform their clinical tasks as well as to provide a source for the advancement of novel biomarkers. In particular, optical coherence tomography (OCT) evaluation of the retina can be augmented by algorithms based on machine learning and deep learning to early detect, qualitatively localize and quantitatively measure epi/intra/subretinal abnormalities or pathological features of macular or neural diseases. In this paper, we discuss the use of AI to facilitate efficacy and accuracy of retinal imaging in those diseases increasingly treated by intravitreal vascular endothelial growth factor (VEGF) inhibitors (i.e. anti-VEGF drugs), also including integration and interpretation features in the process. We review recent advances by AI in diabetic retinopathy, age-related macular degeneration, and retinopathy of prematurity that envision a potentially key role of highly automated systems in screening, early diagnosis, grading and individualized therapy. We discuss benefits and critical aspects of automating the evaluation of disease activity, recurrences, the timing of retreatment and therapeutically potential novel targets in ophthalmology. The impact of massive employment of AI to optimize clinical assistance and encourage tailored therapies for distinct patterns of retinal diseases is also discussed.

Practice Guidelines for Ocular Telehealth-Diabetic Retinopathy, Third Edition

Contributors The following document and appendices represent the third edition of the Practice Guidelines for Ocular Telehealth-Diabetic Retinopathy. These guidelines were developed by the Diabetic Retinopathy Telehealth Practice Guidelines Working Group. This working group consisted of a large number of subject matter experts in clinical applications for telehealth in ophthalmology. The editorial committee consisted of Mark B. Horton, OD, MD, who served as working group chair and Christopher J. Brady, MD, MHS, and Jerry Cavallerano, OD, PhD, who served as cochairs. The writing committees were separated into seven different categories. They are as follows: 1.Clinical/operational: Jerry Cavallerano, OD, PhD (Chair), Gail Barker, PhD, MBA, Christopher J. Brady, MD, MHS, Yao Liu, MD, MS, Siddarth Rathi, MD, MBA, Veeral Sheth, MD, MBA, Paolo Silva, MD, and Ingrid Zimmer-Galler, MD. 2.Equipment: Veeral Sheth, MD (Chair), Mark B. Horton, OD, MD, Siddarth Rathi, MD, MBA, Paolo Silva, MD, and Kristen Stebbins, MSPH. 3.Quality assurance: Mark B. Horton, OD, MD (Chair), Seema Garg, MD, PhD, Yao Liu, MD, MS, and Ingrid Zimmer-Galler, MD. 4.Glaucoma: Yao Liu, MD, MS (Chair) and Siddarth Rathi, MD, MBA. 5.Retinopathy of prematurity: Christopher J. Brady, MD, MHS (Chair) and Ingrid Zimmer-Galler, MD. 6.Age-related macular degeneration: Christopher J. Brady, MD, MHS (Chair) and Ingrid Zimmer-Galler, MD. 7.Autonomous and computer assisted detection, classification and diagnosis of diabetic retinopathy: Michael Abramoff, MD, PhD (Chair), Michael F. Chiang, MD, and Paolo Silva, MD.

Telemedicine for Retinopathy of Prematurity

Background: Retinopathy of prematurity (ROP) is a disease of the retinal vasculature that remains a leading cause of childhood blindness worldwide despite improvements in the systemic care of premature newborns. Screening for ROP is effective and cost-effective, but in many areas, access to skilled examiners to conduct dilated examinations is poor. Remote screening with retinal photography is an alternative strategy that may allow for improved ROP care. Methods: The current literature was reviewed to find clinical trials and expert consensus documents on the state-of-the-art of telemedicine for ROP. Results: Several studies have confirmed the utility of telemedicine for ROP. In addition, several clinical studies have reported favorable long-term results. Many investigators have reinforced the need for detailed protocols on image acquisition and image interpretation. Conclusions: Telemedicine for ROP appears to be a viable alternative to live ophthalmoscopic examinations in many circumstances. Standardization and documentation afforded by telemedicine may provide additional benefits to providers and their patients. With continued improvements in image quality and affordability of imaging systems as well as improved automated image interpretation tools anticipated in the near future, telemedicine for ROP is expected to play an expanding role for a uniquely vulnerable patient population.

Artificial Intelligence in Retinopathy of Prematurity Diagnosis

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The diagnosis of ROP is subclassified by zone, stage, and plus disease, with each area demonstrating significant intra- and interexpert subjectivity and disagreement. In addition to improved efficiencies for ROP screening, artificial intelligence may lead to automated, quantifiable, and objective diagnosis in ROP. This review focuses on the development of artificial intelligence for automated diagnosis of plus disease in ROP and highlights the clinical and technical challenges of both the development and implementation of artificial intelligence in the real world.

Aggressive Posterior Retinopathy of Prematurity: Clinical and Quantitative Imaging Features in a Large North American Cohort

PURPOSE

Aggressive posterior retinopathy of prematurity (AP-ROP) is a vision-threatening disease with a significant rate of progression to retinal detachment. The purpose of this study was to characterize AP-ROP quantitatively by demographics, rate of disease progression, and a deep learning-based vascular severity score.

DESIGN

Retrospective analysis.

PARTICIPANTS

The Imaging and Informatics in ROP cohort from 8 North American centers, consisting of 947 patients and 5945 clinical eye examinations with fundus images, was used. Pretreatment eyes were categorized by disease severity: none, mild, type 2 or pre-plus, treatment-requiring (TR) without AP-ROP, TR with AP-ROP. Analyses compared TR with AP-ROP and TR without AP-ROP to investigate differences between AP-ROP and other TR disease.

METHODS

A reference standard diagnosis was generated for each eye examination using previously published methods combining 3 independent image-based gradings and 1 ophthalmoscopic grading. All fundus images were analyzed using a previously published deep learning system and were assigned a score from 1 through 9.

MAIN OUTCOME MEASURES

Birth weight, gestational age, postmenstrual age, and vascular severity score.

RESULTS

Infants who demonstrated AP-ROP were more premature by birth weight (617 g vs. 679 g; P = 0.01) and gestational age (24.3 weeks vs. 25.0 weeks; P < 0.01) and reached peak severity at an earlier postmenstrual age (34.7 weeks vs. 36.9 weeks; P < 0.001) compared with infants with TR without AP-ROP. The mean vascular severity score was greatest in TR with AP-ROP infants compared with TR without AP-ROP infants (8.79 vs. 7.19; P < 0.001). Analyzing the severity score over time, the rate of progression was fastest in infants with AP-ROP (P < 0.002 at 30-32 weeks).

CONCLUSIONS

Premature infants in North America with AP-ROP are born younger and demonstrate disease earlier than infants with less severe ROP. Disease severity is quantifiable with a deep learning-based score, which correlates with clinically identified categories of disease, including AP-ROP. The rate of progression to peak disease is greatest in eyes that demonstrate AP-ROP compared with other treatment-requiring eyes. Analysis of quantitative characteristics of AP-ROP may help improve diagnosis and treatment of an aggressive, vision-threatening form of ROP.

Deep Learning Algorithm for Automated Diagnosis of Retinopathy of Prematurity Plus Disease

PURPOSE

This study describes the initial development of a deep learning algorithm, ROP.AI, to automatically diagnose retinopathy of prematurity (ROP) plus disease in fundal images.

METHODS

ROP.AI was trained using 6974 fundal images from Australasian image databases. Each image was given a diagnosis as part of real-world routine ROP screening and classified as normal or plus disease. The algorithm was trained using 80% of the images and validated against the remaining 20% within a hold-out test set. Performance in diagnosing plus disease was evaluated against an external set of 90 images. Performance in detecting pre-plus disease was also tested. As a screening tool, the algorithm's operating point was optimized for sensitivity and negative predictive value, and its performance reevaluated.

RESULTS

For plus disease diagnosis within the 20% hold-out test set, the algorithm achieved a 96.6% sensitivity, 98.0% specificity, and 97.3% ± 0.7% accuracy. Area under the receiver operating characteristic curve was 0.993. Within the independent test set, the algorithm achieved a 93.9% sensitivity, 80.7% specificity, and 95.8% negative predictive value. For detection of pre-plus and plus disease, the algorithm achieved 81.4% sensitivity, 80.7% specificity, and 80.7% negative predictive value. Following the identification of an optimized operating point, the algorithm diagnosed plus disease with a 97.0% sensitivity and 97.8% negative predictive value.

CONCLUSIONS

ROP.AI is a deep learning algorithm able to automatically diagnose ROP plus disease with high sensitivity and negative predictive value.

TRANSLATIONAL RELEVANCE

In the context of increasing global disease burden, future development may improve access to ROP diagnosis and care.

Accuracy and Reliability of Eye-Based vs Quadrant-Based Diagnosis of Plus Disease in Retinopathy of Prematurity

Importance

Presence of plus disease in retinopathy of prematurity is the most critical element in identifying treatment-requiring disease. However, there is significant variability in plus disease diagnosis. In particular, plus disease has been defined as 2 or more quadrants of vascular abnormality, and it is not clear whether it is more reliably and accurately diagnosed by eye-based assessment of overall retinal appearance or by quadrant-based assessment combining grades of 4 individual quadrants.

Objective

To compare eye-based vs quadrant-based diagnosis of plus disease and to provide insight for ophthalmologists about the diagnostic process.

Design, Setting, and Participants

In this multicenter cohort study, we developed a database of 197 wide-angle retinal images from 141 preterm infants from neonatal intensive care units at 9 academic institutions (enrolled from July 2011 to December 2016). Each image was assigned a reference standard diagnosis based on consensus image-based and clinical diagnosis. Data analysis was performed from February 2017 to September 2017.

Interventions

Six graders independently diagnosed each of the 4 quadrants (cropped images) of the 197 eyes (quadrant-based diagnosis) as well as the entire image (eye-based diagnosis). Images were displayed individually, in random order. Quadrant-based diagnosis of plus disease was made when 2 or more quadrants were diagnosed as indicating plus disease by combining grades of individual quadrants post hoc.

Main Outcomes and Measures

Intragrader and intergrader reliability (absolute agreement and κ statistic) and accuracy compared with the reference standard diagnosis.

Results

Of the 141 included preterm infants, 65 (46.1%) were female and 116 (82.3%) white, and the mean (SD) gestational age was 27.0 (2.6) weeks. There was variable agreement between eye-based and quadrant-based diagnosis among the 6 graders (Cohen κ range, 0.32-0.75). Four graders showed underdiagnosis of plus disease with quadrant-based diagnosis compared with eye-based diagnosis (by McNemar test). Intergrader agreement of quadrant-based diagnosis was lower than that of eye-based diagnosis (Fleiss κ, 0.75 [95% CI, 0.71-0.78] vs 0.55 [95% CI, 0.51-0.59]). The accuracy of eye-based diagnosis compared with the reference standard diagnosis was substantial to near-perfect, whereas that of quadrant-based plus disease diagnosis was only moderate to substantial for each grader.

Conclusions and Relevance

Graders had lower reliability and accuracy using quadrant-based diagnosis combining grades of individual quadrants than with eye-based diagnosis, suggesting that eye-based diagnosis has advantages over quadrant-based diagnosis. This has implications for more precise definitions of plus disease regarding the criterion of 2 or more quadrants, clinical care, computer-based image analysis, and education for all ophthalmologists who manage retinopathy of prematurity.

A Quantitative Severity Scale for Retinopathy of Prematurity Using Deep Learning to Monitor Disease Regression After Treatment

Importance

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide, but treatment failure and disease recurrence are important causes of adverse outcomes in patients with treatment-requiring ROP (TR-ROP).

Objectives

To apply an automated ROP vascular severity score obtained using a deep learning algorithm and to assess its utility for objectively monitoring ROP regression after treatment.

Design, Setting, and Participants

This retrospective cohort study used data from the Imaging and Informatics in ROP consortium, which comprises 9 tertiary referral centers in North America that screen high volumes of at-risk infants for ROP. Images of 5255 clinical eye examinations from 871 infants performed between July 2011 and December 2016 were assessed for eligibility in the present study. The disease course was assessed with time across the numerous examinations for patients with TR-ROP. Infants born prematurely meeting screening criteria for ROP who developed TR-ROP and who had images captured within 4 weeks before and after treatment as well as at the time of treatment were included.

Main Outcomes and Measures

The primary outcome was mean (SD) ROP vascular severity score before, at time of, and after treatment. A deep learning classifier was used to assign a continuous ROP vascular severity score, which ranged from 1 (normal) to 9 (most severe), at each examination. A secondary outcome was the difference in ROP vascular severity score among eyes treated with laser or the vascular endothelial growth factor antagonist bevacizumab. Differences between groups for both outcomes were assessed using unpaired 2-tailed t tests with Bonferroni correction.

Results

Of 5255 examined eyes, 91 developed TR-ROP, of which 46 eyes met the inclusion criteria based on the available images. The mean (SD) birth weight of those patients was 653 (185) g, with a mean (SD) gestational age of 24.9 (1.3) weeks. The mean (SD) ROP vascular severity scores significantly increased 2 weeks prior to treatment (4.19 [1.75]), peaked at treatment (7.43 [1.89]), and decreased for at least 2 weeks after treatment (4.00 [1.88]) (all P < .001). Eyes requiring retreatment with laser had higher ROP vascular severity scores at the time of initial treatment compared with eyes receiving a single treatment (P < .001).

Conclusions and Relevance

This quantitative ROP vascular severity score appears to consistently reflect clinical disease progression and posttreatment regression in eyes with TR-ROP. These study results may have implications for the monitoring of patients with ROP for treatment failure and disease recurrence and for determining the appropriate level of disease severity for primary treatment in eyes with aggressive disease.

Monitoring Disease Progression With a Quantitative Severity Scale for Retinopathy of Prematurity Using Deep Learning

Importance

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide, but clinical diagnosis is subjective and qualitative.

Objective

To describe a quantitative ROP severity score derived using a deep learning algorithm designed to evaluate plus disease and to assess its utility for objectively monitoring ROP progression.

Design, Setting, and Participants

This retrospective cohort study included images from 5255 clinical examinations of 871 premature infants who met the ROP screening criteria of the Imaging and Informatics in ROP (i-ROP) Consortium, which comprises 9 tertiary care centers in North America, from July 1, 2011, to December 31, 2016. Data analysis was performed from July 2017 to May 2018.

Exposure

A deep learning algorithm was used to assign a continuous ROP vascular severity score from 1 (most normal) to 9 (most severe) at each examination based on a single posterior photograph compared with a reference standard diagnosis (RSD) simplified into 4 categories: no ROP, mild ROP, type 2 ROP or pre-plus disease, or type 1 ROP. Disease course was assessed longitudinally across multiple examinations for all patients.

Main Outcomes and Measures

Mean ROP vascular severity score progression over time compared with the RSD.

Results

A total of 5255 clinical examinations from 871 infants (mean [SD] gestational age, 27.0 [2.0] weeks; 493 [56.6%] male; mean [SD] birth weight, 949 [271] g) were analyzed. The median severity scores for each category were as follows: 1.1 (interquartile range [IQR], 1.0-1.5) (no ROP), 1.5 (IQR, 1.1-3.4) (mild ROP), 4.6 (IQR, 2.4-5.3) (type 2 and pre-plus), and 7.5 (IQR, 5.0-8.7) (treatment-requiring ROP) (P < .001). When the long-term differences in the median severity scores across time between the eyes progressing to treatment and those who did not eventually require treatment were compared, the median score was higher in the treatment group by 0.06 at 30 to 32 weeks, 0.75 at 32 to 34 weeks, 3.56 at 34 to 36 weeks, 3.71 at 36 to 38 weeks, and 3.24 at 38 to 40 weeks postmenstrual age (P < .001 for all comparisons).

Conclusions and Relevance

The findings suggest that the proposed ROP vascular severity score is associated with category of disease at a given point in time and clinical progression of ROP in premature infants. Automated image analysis may be used to quantify clinical disease progression and identify infants at high risk for eventually developing treatment-requiring ROP. This finding has implications for quality and delivery of ROP care and for future approaches to disease classification.

Artificial intelligence and deep learning in ophthalmology

Artificial intelligence (AI) based on deep learning (DL) has sparked tremendous global interest in recent years. DL has been widely adopted in image recognition, speech recognition and natural language processing, but is only beginning to impact on healthcare. In ophthalmology, DL has been applied to fundus photographs, optical coherence tomography and visual fields, achieving robust classification performance in the detection of diabetic retinopathy and retinopathy of prematurity, the glaucoma-like disc, macular oedema and age-related macular degeneration. DL in ocular imaging may be used in conjunction with telemedicine as a possible solution to screen, diagnose and monitor major eye diseases for patients in primary care and community settings. Nonetheless, there are also potential challenges with DL application in ophthalmology, including clinical and technical challenges, explainability of the algorithm results, medicolegal issues, and physician and patient acceptance of the AI 'black-box' algorithms. DL could potentially revolutionise how ophthalmology is practised in the future. This review provides a summary of the state-of-the-art DL systems described for ophthalmic applications, potential challenges in clinical deployment and the path forward.

Estimate of incidence of ROP requiring treatment in extreme preterms and impact on service-7 year review in tertiary unit

BACKGROUND/OBJECTIVES

Retinopathy of prematurity (ROP) is a potentially blinding disorder affecting premature infants. Our Eye Unit supports two neonatal intensive care units (NICUs), one provides neonatal surgical and medical facilities and the other is exclusively medical. Our objectives were to (1) to identify the annual rate of ROP treatments during the period 2009-2015 and (2) to estimate the incidence of ROP treatment in babies born very prematurely (<27 weeks).

SUBJECTS/METHODS

Records for all infants treated for ROP by our unit during the period 2009-2015 were reviewed. We calculated numbers treated in each year. Records of babies born under 27 weeks of gestation and cared for in the non-surgical NICU were also reviewed. Their requirement for laser treatments for ROP was calculated by the week of gestation at birth.

RESULTS

In the two NICUs combined, 95 infants were treated for ROP between 2009 and 2015. The numbers treated increased from 9/158 (5.7%) of babies screened in 2009 to 22/159 (13.8%) in 2015 (p_trend = 0.004). The rate of laser treatment for ROP increased as gestation at birth decreased: from 12/100 (12%) of babies born at 26 weeks to 17/29 (59%) of babies born at 23 weeks (p_trend = 0.001).

CONCLUSION

The number of laser treatments for ROP carried out by this unit has increased steadily between 2009 and 2015 and this may in part be due to the increased need for ROP treatment in extremely preterm babies, whose survival has increased in the same period. These data may aid planning for ROP services.

Evaluation of a deep learning image assessment system for detecting severe retinopathy of prematurity

BACKGROUND

Prior work has demonstrated the near-perfect accuracy of a deep learning retinal image analysis system for diagnosing plus disease in retinopathy of prematurity (ROP). Here we assess the screening potential of this scoring system by determining its ability to detect all components of ROP diagnosis.

METHODS

Clinical examination and fundus photography were performed at seven participating centres. A deep learning system was trained to detect plus disease, generating a quantitative assessment of retinal vascular abnormality (the i-ROP plus score) on a 1-9 scale. Overall ROP disease category was established using a consensus reference standard diagnosis combining clinical and image-based diagnosis. Experts then ranked ordered a second data set of 100 posterior images according to overall ROP severity.

RESULTS

4861 examinations from 870 infants were analysed. 155 examinations (3%) had a reference standard diagnosis of type 1 ROP. The i-ROP deep learning (DL) vascular severity score had an area under the receiver operating curve of 0.960 for detecting type 1 ROP. Establishing a threshold i-ROP DL score of 3 conferred 94% sensitivity, 79% specificity, 13% positive predictive value and 99.7% negative predictive value for type 1 ROP. There was strong correlation between expert rank ordering of overall ROP severity and the i-ROP DL vascular severity score (Spearman correlation coefficient=0.93; p<0.0001).

CONCLUSION

The i-ROP DL system accurately identifies diagnostic categories and overall disease severity in an automated fashion, after being trained only on posterior pole vascular morphology. These data provide proof of concept that a deep learning screening platform could improve objectivity of ROP diagnosis and accessibility of screening.

Automated Diagnosis of Plus Disease in Retinopathy of Prematurity Using Deep Convolutional Neural Networks

Importance

Retinopathy of prematurity (ROP) is a leading cause of childhood blindness worldwide. The decision to treat is primarily based on the presence of plus disease, defined as dilation and tortuosity of retinal vessels. However, clinical diagnosis of plus disease is highly subjective and variable.

Objective

To implement and validate an algorithm based on deep learning to automatically diagnose plus disease from retinal photographs.

Design, Setting, and Participants

A deep convolutional neural network was trained using a data set of 5511 retinal photographs. Each image was previously assigned a reference standard diagnosis (RSD) based on consensus of image grading by 3 experts and clinical diagnosis by 1 expert (ie, normal, pre-plus disease, or plus disease). The algorithm was evaluated by 5-fold cross-validation and tested on an independent set of 100 images. Images were collected from 8 academic institutions participating in the Imaging and Informatics in ROP (i-ROP) cohort study. The deep learning algorithm was tested against 8 ROP experts, each of whom had more than 10 years of clinical experience and more than 5 peer-reviewed publications about ROP. Data were collected from July 2011 to December 2016. Data were analyzed from December 2016 to September 2017.

Exposures

A deep learning algorithm trained on retinal photographs.

Main Outcomes and Measures

Receiver operating characteristic analysis was performed to evaluate performance of the algorithm against the RSD. Quadratic-weighted κ coefficients were calculated for ternary classification (ie, normal, pre-plus disease, and plus disease) to measure agreement with the RSD and 8 independent experts.

Results

Of the 5511 included retinal photographs, 4535 (82.3%) were graded as normal, 805 (14.6%) as pre-plus disease, and 172 (3.1%) as plus disease, based on the RSD. Mean (SD) area under the receiver operating characteristic curve statistics were 0.94 (0.01) for the diagnosis of normal (vs pre-plus disease or plus disease) and 0.98 (0.01) for the diagnosis of plus disease (vs normal or pre-plus disease). For diagnosis of plus disease in an independent test set of 100 retinal images, the algorithm achieved a sensitivity of 93% with 94% specificity. For detection of pre-plus disease or worse, the sensitivity and specificity were 100% and 94%, respectively. On the same test set, the algorithm achieved a quadratic-weighted κ coefficient of 0.92 compared with the RSD, outperforming 6 of 8 ROP experts.

Conclusions and Relevance

This fully automated algorithm diagnosed plus disease in ROP with comparable or better accuracy than human experts. This has potential applications in disease detection, monitoring, and prognosis in infants at risk of ROP.

Plus Disease in Retinopathy of Prematurity: More Than Meets the ICROP?

Retinopathy of prematurity (ROP), a vasoproliferative retinal disease affecting premature infants, is a leading cause of childhood blindness throughout the world. Plus disease, defined as venous dilatation and arteriolar tortuosity within the posterior retinal vessels greater than or equal to that of a standard published photograph, is the most critical finding in identifying treatment-requiring ROP. Despite an internationally accepted definition of plus disease, there is significant variability in diagnostic process and outcome, producing variable levels of reported intra- and interexpert agreement. Several potential explanations for poor agreement have been proposed, including attention to undefined vascular features such as venous tortuosity, focus on narrower or wider field of view, unfamiliarity with digital images, the magnification and apparent severity of the standard photograph, and cut-off point differences among experts as to the level of tortuosity and dilation sufficient for "plus disease" along a continuum. Moreover, differences in diagnostic consistency among groups of experts separated both geographically and chronologically have been reported. These findings have implications for clinical care, research, and education, and highlight the need for a more precise definition of plus disease and objective diagnostic methods for ROP.