External quality assurance for image grading in the Scottish Diabetic Retinopathy Screening Programme

Deep learning detection of diabetic retinopathy in Scotland's diabetic eye screening programme

BACKGROUND/AIMS

Support vector machine-based automated grading (known as iGradingM) has been shown to be safe, cost-effective and robust in the diabetic retinopathy (DR) screening (DES) programme in Scotland. It triages screening episodes as gradable with no DR versus manual grading required. The study aim was to develop a deep learning-based autograder using images and gradings from DES and to compare its performance with that of iGradingM.

METHODS

Retinal images, quality assurance (QA) data and routine DR grades were obtained from national datasets in 179 944 patients for years 2006-2016. QA grades were available for 744 images. We developed a deep learning-based algorithm to detect whether either eye contained ungradable images or any DR. The sensitivity and specificity were evaluated against consensus QA grades and routine grades.

RESULTS

Images used in QA which were ungradable or with DR were detected by deep learning with better specificity compared with manual graders (p<0.001) and with iGradingM (p<0.001) at the same sensitivities. Any DR according to the DES final grade was detected with 89.19% (270 392/303 154) sensitivity and 77.41% (500 945/647 158) specificity. Observable disease and referable disease were detected with sensitivities of 96.58% (16 613/17 201) and 98.48% (22 600/22 948), respectively. Overall, 43.84% of screening episodes would require manual grading.

CONCLUSION

A deep learning-based system for DR grading was evaluated in QA data and images from 11 years in 50% of people attending a national DR screening programme. The system could reduce the manual grading workload at the same sensitivity compared with the current automated grading system.

Handheld fundus camera performance, image quality and outcomes of diabetic retinopathy grading in a pilot screening study

PURPOSE

To compare the performance and image quality of the handheld fundus camera to standard table-top fundus cameras in diabetic retinopathy (DR) screening. The reliability and diagnostic accuracy of DR grading performed by an ophthalmologist and a photographer reader were evaluated.

MATERIALS AND METHODS

157 patients with diabetes, attending screening or follow-up of DR, were evaluated by fundus photographs taken in mydriasis by Optomed Aurora and Canon or Zeiss Visucam fundus cameras. The image quality and the severity of DR were evaluated independently by an ophthalmologist and experienced photographer. The sensitivity, specificity and reliability of the assessments were determined.

RESULTS

1884 fundus images from 314 eyes were analysed. In 53% of all eyes, DR was not present. 10% had mild non-proliferative diabetic retinopathy (NPDR), 16% moderate NPDR, 6% severe NPDR and 16% proliferative diabetic retinopathy (PDR). The DR grading outcomes by Aurora highly equalled to those of Canon or Zeiss (κ = 0.93, 95% CI 0.91 to 0.94), and there was almost perfect agreement in grading between the ophthalmologist and photographer (κ = 0.96, 95% CI 0.95 to 0.97). The image quality of Aurora was sufficient for reliable assessment according to both graders in 84-88% of the cases.

CONCLUSION

The Optomed Aurora fundus camera seems appropriate for DR screening. The sufficient image quality and high diagnostic accuracy for DR grading are supportive for a less expensive and easily transportable screening system for DR. Immediate image grading carried out by a photographer would further improve and speed up the screening process in all settings.

Competency level assessment of healthcare practitioners in managing diabetes and diabetic eye disease in the district health system of Limpopo province, South Africa

Background: There are many gaps in the management of diabetes mellitus (DM) and diabetic eye disease in the district health system (DHS) of South Africa (SA). National guidelines recommend annual eye examinations for patients with DM.Aim: The purpose of this study was to describe the self-reported skill levels of healthcare practitioners (HCPs) to conduct eye examination procedures required for a proposed diabetic retinopathy (DR) screening model.Setting: The study was conducted in public health institutions of Waterberg district and Mankweng Hospital complex (Capricorn district) in Limpopo province, SA.Methods: A cross-sectional design using purposive sampling was conducted, and questionnaires were distributed to a total of 74 HCPs. The questionnaires distributed included questions relating to the competency levels of primary healthcare nurses (PHC nurses), optometrists, ophthalmic nurses and medical officers (MOs) regarding examination procedures in the management of patients with DM and whether they agreed with the developed DR screening model.Results: All the PHC nurses had knowledge about all the examination procedures required in the proposed DR screening model, whilst 94.1% of MOs exhibited knowledge regarding the procedures required from them. Optometrists lacked knowledge of grading DR, and ophthalmic nurses were least knowledgeable about conducting internal and external eye examinations and in detecting and grading DR.Conclusion: The proposed DR screening model did not need modification. The involvement of dieticians and more ophthalmic nurses could be beneficial to the DR screening model.

Diagnostic accuracy of diabetic retinopathy grading by an artificial intelligence-enabled algorithm compared with a human standard for wide-field true-colour confocal scanning and standard digital retinal images

BACKGROUND

Photographic diabetic retinopathy screening requires labour-intensive grading of retinal images by humans. Automated retinal image analysis software (ARIAS) could provide an alternative to human grading. We compare the performance of an ARIAS using true-colour, wide-field confocal scanning images and standard fundus images in the English National Diabetic Eye Screening Programme (NDESP) against human grading.

METHODS

Cross-sectional study with consecutive recruitment of patients attending annual diabetic eye screening. Imaging with mydriasis was performed (two-field protocol) with the EIDON platform (CenterVue, Padua, Italy) and standard NDESP cameras. Human grading was carried out according to NDESP protocol. Images were processed by EyeArt V.2.1.0 (Eyenuk Inc, Woodland Hills, California). The reference standard for analysis was the human grade of standard NDESP images.

RESULTS

We included 1257 patients. Sensitivity estimates for retinopathy grades were: EIDON images; 92.27% (95% CI: 88.43% to 94.69%) for any retinopathy, 99% (95% CI: 95.35% to 100%) for vision-threatening retinopathy and 100% (95% CI: 61% to 100%) for proliferative retinopathy. For NDESP images: 92.26% (95% CI: 88.37% to 94.69%) for any retinopathy, 100% (95% CI: 99.53% to 100%) for vision-threatening retinopathy and 100% (95% CI: 61% to 100%) for proliferative retinopathy. One case of vision-threatening retinopathy (R1M1) was missed by the EyeArt when analysing the EIDON images, but identified by the human graders. The EyeArt identified all cases of vision-threatening retinopathy in the standard images.

CONCLUSION

EyeArt identified diabetic retinopathy in EIDON images with similar sensitivity to standard images in a large-scale screening programme, exceeding the sensitivity threshold recommended for a screening test. Further work to optimise the identification of 'no retinopathy' and to understand the differential lesion detection in the two imaging systems would enhance the use of these two innovative technologies in a diabetic retinopathy screening setting.

Assessment of Training Outcomes of Nurse Readers for Diabetic Retinopathy Telescreening: Validation Study

BACKGROUND

With the high prevalence of diabetic retinopathy and its significant visual consequences if untreated, timely identification and management of diabetic retinopathy is essential. Teleophthalmology programs have assisted in screening a large number of individuals at risk for vision loss from diabetic retinopathy. Training nonophthalmological readers to assess remote fundus images for diabetic retinopathy may further improve the efficiency of such programs.

OBJECTIVE

This study aimed to evaluate the performance, safety implications, and progress of 2 ophthalmology nurses trained to read and assess diabetic retinopathy fundus images within a hospital diabetic retinopathy telescreening program.

METHODS

In this retrospective interobserver study, 2 ophthalmology nurses followed a specific training program within a hospital diabetic retinopathy telescreening program and were trained to assess diabetic retinopathy images at 2 levels of intervention: detection of diabetic retinopathy (level 1) and identification of referable disease (level 2). The reliability of the assessment by level 1-trained readers in 266 patients and of the identification of patients at risk of vision loss from diabetic retinopathy by level 2-trained readers in 559 more patients were measured. The learning curve, sensitivity, and specificity of the readings were evaluated using a group consensus gold standard.

RESULTS

An almost perfect agreement was measured in identifying the presence of diabetic retinopathy in both level 1 readers (κ=0.86 and 0.80) and in identifying referable diabetic retinopathy by level 2 readers (κ=0.80 and 0.83). At least substantial agreement was measured in the level 2 readers for macular edema (κ=0.79 and 0.88) for all eyes. Good screening threshold sensitivities and specificities were obtained for all level readers, with sensitivities of 90.6% and 96.9% and specificities of 95.1% and 85.1% for level 1 readers (readers A and B) and with sensitivities of 86.8% and 91.2% and specificities of 91.7% and 97.0% for level 2 readers (readers A and B). This performance was achieved immediately after training and remained stable throughout the study.

CONCLUSIONS

Notwithstanding the small number of trained readers, this study validates the screening performance of level 1 and level 2 diabetic retinopathy readers within this training program, emphasizing practical experience, and allows the establishment of an ongoing assessment clinic. This highlights the importance of supervised, hands-on experience and may help set parameters to further calibrate the training of diabetic retinopathy readers for safe screening programs.

Incidence of sight-threatening diabetic retinopathy in people with Type 2 diabetes mellitus and numbers needed to screen: a systematic review

AIM

To investigate the incidence of sight-threatening diabetic retinopathy in Type 2 diabetes mellitus.

BACKGROUND

In most countries, yearly or biennial screening intervals for diabetic retinopathy in people with Type 2 diabetes are recommended. Fewer screening sessions reduce the effort required of people with Type 2 diabetes and reduce healthcare costs.

METHODS

We conducted a search of PubMed, Embase, Web of Science and the COCHRANE Library for studies published betweeen 1 January 2000 and 1 January 2017. Eligible studies were those that included general populations of >100 people with Type 2 diabetes mellitus. Additional study population criteria were absence of moderate diabetic retinopathy or more severe diabetic retinopathy at last screening session and at least two gradable retinal screening sessions. Outcomes of interest in the included studies were moderate and severe non-proliferative diabetic retinopathy (R2), proliferative diabetic retinopathy (R3) or maculopathy (M1), collectively known as sight-threatening or referable diabetic retinopathy.

RESULTS

A total of 17 studies were included. In people with Type 2 diabetes without or with only mild diabetic retinopathy at baseline, the average incidence rates of sight-threatening diabetic retinopathy were ~1 per 100 person-years and ~8 per 100 person-years, respectively. The average numbers needed to screen to detect one case of sight-threatening diabetic retinopathy were 175 and 19 in people without and with mild retinopathy at last screening, respectively.

CONCLUSION

In people with Type 2 diabetes without retinopathy at last screening, the incidence of severe sight-threatening retinopathy at the subsequent screening session was low. In people with mild retinopathy, progression to sight-threatening diabetic retinopathy was nearly 10-fold higher. This review supports lengthening of the screening interval of patients with Type 2 diabetes without retinopathy at last screening session.

Multiple Effects of Intravitreal Aflibercept on Microvascular Regression in Eyes with Diabetic Macular Edema

PURPOSE

To evaluate the effects of intravitreal aflibercept (IVA) on the number of microaneurysms and sizes of nonperfused areas (NPAs) in eyes with diabetic macular edema (DME).

DESIGN

Interventional, prospective study.

PARTICIPANTS

Twenty-five eyes of 25 DME patients (average age, 64.0±8.8 years) were treated with 3 consecutive monthly IVA injections.

METHODS

Fluorescein angiography (FA) and OCT were performed before the IVA injections (baseline) and at 1 week after the IVA treatment. The number of microaneurysms and the ischemic index (ISI), a measure of NPA, were determined. The correlations between central retinal thickness (CRT) and number of microaneurysms and the ISI were also determined.

MAIN OUTCOME MEASURES

The mean number of microaneurysms and NPA evaluated as the ISI.

RESULTS

At baseline, the mean CRT was 485.7±90.6 μm. After treatment, the mean CRT was reduced significantly to 376.9±81.6 μm (P = 0.1 × 10^-5, repeated analysis of variance). The mean number of microaneurysms was decreased significantly from 49.6±33.2 at baseline to 24.8±18.1 at 3 months after the initial treatment. This was a 50.4±21.2% reduction (P = 0.3 × 10^-5, paired t test). The mean ISI was also decreased significantly from 55.5±20.4% at baseline to 28.8±16.8% after treatment (P = 0.3 × 10^-5, paired t test). This was a reduction of 43.3±28.5%. A significant correlation was found between the CRT and number of microaneurysms at both baseline (r = 0.56; P = 0.004) and after treatment (r = 0.53; P = 0.006). A significant correlation was found between CRT and ISI at baseline (r = -0.39; P = 0.03) but not after treatment (r = -0.06; P = 0.79).

CONCLUSIONS

The reduction in the number of microaneurysms was correlated with reduction in CRT.

Automated Diabetic Retinopathy Image Assessment Software: Diagnostic Accuracy and Cost-Effectiveness Compared with Human Graders

OBJECTIVE

With the increasing prevalence of diabetes, annual screening for diabetic retinopathy (DR) by expert human grading of retinal images is challenging. Automated DR image assessment systems (ARIAS) may provide clinically effective and cost-effective detection of retinopathy. We aimed to determine whether ARIAS can be safely introduced into DR screening pathways to replace human graders.

DESIGN

Observational measurement comparison study of human graders following a national screening program for DR versus ARIAS.

PARTICIPANTS

Retinal images from 20 258 consecutive patients attending routine annual diabetic eye screening between June 1, 2012, and November 4, 2013.

METHODS

Retinal images were manually graded following a standard national protocol for DR screening and were processed by 3 ARIAS: iGradingM, Retmarker, and EyeArt. Discrepancies between manual grades and ARIAS results were sent to a reading center for arbitration.

MAIN OUTCOME MEASURES

Screening performance (sensitivity, false-positive rate) and diagnostic accuracy (95% confidence intervals of screening-performance measures) were determined. Economic analysis estimated the cost per appropriate screening outcome.

RESULTS

Sensitivity point estimates (95% confidence intervals) of the ARIAS were as follows: EyeArt 94.7% (94.2%-95.2%) for any retinopathy, 93.8% (92.9%-94.6%) for referable retinopathy (human graded as either ungradable, maculopathy, preproliferative, or proliferative), 99.6% (97.0%-99.9%) for proliferative retinopathy; Retmarker 73.0% (72.0 %-74.0%) for any retinopathy, 85.0% (83.6%-86.2%) for referable retinopathy, 97.9% (94.9%-99.1%) for proliferative retinopathy. iGradingM classified all images as either having disease or being ungradable. EyeArt and Retmarker saved costs compared with manual grading both as a replacement for initial human grading and as a filter prior to primary human grading, although the latter approach was less cost-effective.

CONCLUSIONS

Retmarker and EyeArt systems achieved acceptable sensitivity for referable retinopathy when compared with that of human graders and had sufficient specificity to make them cost-effective alternatives to manual grading alone. ARIAS have the potential to reduce costs in developed-world health care economies and to aid delivery of DR screening in developing or remote health care settings.

Modelling the cost-effectiveness of adopting risk-stratified approaches to extended screening intervals in the national diabetic retinopathy screening programme in Scotland

AIMS

To assess the cost-effectiveness of adopting risk-stratified approaches to extended screening intervals in the national diabetic retinopathy screening programme in Scotland.

METHODS

A continuous-time hidden Markov model was fitted to national longitudinal screening data to derive transition probabilities between observed non-referable and referable retinopathy states. These were incorporated in a decision model simulating progression, costs and visual acuity outcomes for a synthetic cohort with a covariate distribution matching that of the Scottish diabetic screening population. The cost-effectiveness of adopting extended (2-year) screening for groups with no observed retinopathy was then assessed over a 30-year time horizon.

RESULTS

Individuals with a current grade of no retinopathy on two consecutive screening episodes face the lowest risk of progressing to referable disease. For the cohort as a whole, the incremental cost per quality-adjusted life year gained for annual vs. biennial screening ranged from approximately £74 000 (for those with no retinopathy and a prior observed grade of mild or observable background retinopathy) to approximately £232 000 per quality-adjusted life year gained (for those with no retinopathy on two consecutive screening episodes). The corresponding incremental cost-effectiveness ratios in the subgroup with Type 1 diabetes were substantially lower; approximately £22 000 to £85 000 per quality-adjusted life year gained, respectively.

CONCLUSIONS

Biennial screening for individuals with diabetes who have no retinopathy is likely to deliver significant savings for a very small increase in the risk of adverse visual acuity and quality of life outcomes. There is greater uncertainty regarding the long-term cost-effectiveness of adopting biennial screening in younger people with Type 1 diabetes.

Centralized Reading Center Improves Quality in a Real-World Setting

BACKGROUND AND OBJECTIVE

Diabetic retinopathy (DR) screening is standard of care, but its utility diminishes unless high quality is maintained. This project describes the improvement in diagnostic accuracy achieved at a large health maintenance organization by implementing a centralized reading center.

PATIENTS AND METHODS

Data were extracted from the electronic health records of patients with diabetes who underwent DR screening between January 1, 2009, and December 31, 2013, at 13 medical centers. Certified ophthalmic assistants and technicians were trained to read DR images under the supervision of a retinal specialist. This program was pilot-tested prior to full implementation.

RESULTS

DR prevalence among the 13 centers in 2009 was 10.1%, far below the national average. After implementation, prevalence increased to 22.1%. The program also resulted in improved access to follow-up.

CONCLUSION

The results demonstrate the value of a reading center in decreasing diagnostic error and achieving other improvements in a real-world setting.

Grader agreement, and sensitivity and specificity of digital photography in a community optometry-based diabetic eye screening program

Background

Digital retinal photography with mydriasis is the preferred modality for diabetes eye screening. The purpose of this study was to evaluate agreement in grading levels between primary and secondary graders and to calculate their sensitivity and specificity for identifying sight-threatening disease in an optometry-based retinopathy screening program.

Methods

This was a retrospective study using data from 8,977 patients registered in the North Nottinghamshire retinal screening program. In all cases, the ophthalmology diagnosis was used as the arbitrator and considered to be the gold standard. Kappa statistics were used to evaluate the level of agreement between graders.

Results

Agreement between primary and secondary graders was 51.4% and 79.7% for detecting no retinopathy (R0) and background retinopathy (R1), respectively. For preproliferative (R2) and proliferative retinopathy (R3) at primary grading, agreement between the primary and secondary grader was 100%. Where there was disagreement between the primary and secondary grader for R1, only 2.6% (n=41) were upgraded by an ophthalmologist. The sensitivity and specificity for detecting R3 was 78.2% and 98.1%, respectively. None of the patients upgraded from any level of retinopathy to R3 required photocoagulation therapy. The observed kappa between the primary and secondary grader was 0.3223 (95% confidence interval 0.2937–0.3509), ie, fair agreement, and between the primary grader and ophthalmology for R3 was 0.5667 (95% confidence interval 0.4557–0.6123), ie, moderate agreement.

Conclusion

These data provide information on the safety of a community optometry-based retinal screening program for screening as a primary and as a secondary grader. The level of agreement between the primary and secondary grader at a higher level of retinopathy (R2 and R3) was 100%. Sensitivity and specificity for R3 were 78.2% and 98.1%, respectively. None of the false-negative results required photocoagulation therapy.

Predicted impact of extending the screening interval for diabetic retinopathy: the Scottish Diabetic Retinopathy Screening programme

AIMS/HYPOTHESIS

The aim of our study was to identify subgroups of patients attending the Scottish Diabetic Retinopathy Screening (DRS) programme who might safely move from annual to two yearly retinopathy screening.

METHODS

This was a retrospective cohort study of screening data from the DRS programme collected between 2005 and 2011 for people aged ≥12 years with type 1 or type 2 diabetes in Scotland. We used hidden Markov models to calculate the probabilities of transitions to referable diabetic retinopathy (referable background or proliferative retinopathy) or referable maculopathy.

RESULTS

The study included 155,114 individuals with no referable diabetic retinopathy or maculopathy at their first DRS examination and with one or more further DRS examinations. There were 11,275 incident cases of referable diabetic eye disease (9,204 referable maculopathy, 2,071 referable background or proliferative retinopathy). The observed transitions to referable background or proliferative retinopathy were lower for people with no visible retinopathy vs mild background retinopathy at their prior examination (respectively, 1.2% vs 8.1% for type 1 diabetes and 0.6% vs 5.1% for type 2 diabetes). The lowest probability for transitioning to referable background or proliferative retinopathy was among people with two consecutive screens showing no visible retinopathy, where the probability was <0.3% for type 1 and <0.2% for type 2 diabetes at 2 years.

CONCLUSIONS/INTERPRETATION

Transition rates to referable diabetic eye disease were lowest among people with type 2 diabetes and two consecutive screens showing no visible retinopathy. If such people had been offered two yearly screening the DRS service would have needed to screen 40% fewer people in 2009.

Assessment of automated disease detection in diabetic retinopathy screening using two-field photography

Aim

To assess the performance of automated disease detection in diabetic retinopathy screening using two field mydriatic photography.

Methods

Images from 8,271 sequential patient screening episodes from a South London diabetic retinopathy screening service were processed by the Medalytix iGrading™ automated grading system. For each screening episode macular-centred and disc-centred images of both eyes were acquired and independently graded according to the English national grading scheme. Where discrepancies were found between the automated result and original manual grade, internal and external arbitration was used to determine the final study grades. Two versions of the software were used: one that detected microaneurysms alone, and one that detected blot haemorrhages and exudates in addition to microaneurysms. Results for each version were calculated once using both fields and once using the macula-centred field alone.

Results

Of the 8,271 episodes, 346 (4.2%) were considered unassessable. Referable disease was detected in 587 episodes (7.1%). The sensitivity of the automated system for detecting unassessable images ranged from 97.4% to 99.1% depending on configuration. The sensitivity of the automated system for referable episodes ranged from 98.3% to 99.3%. All the episodes that included proliferative or pre-proliferative retinopathy were detected by the automated system regardless of configuration (192/192, 95% confidence interval 98.0% to 100%). If implemented as the first step in grading, the automated system would have reduced the manual grading effort by between 2,183 and 3,147 patient episodes (26.4% to 38.1%).

Conclusion

Automated grading can safely reduce the workload of manual grading using two field, mydriatic photography in a routine screening service.