Automatic Detection of Diabetic Retinopathy in Retinal Fundus Photographs Based on Deep Learning Algorithm

MSTNet: Multi-scale spatial-aware transformer with multi-instance learning for diabetic retinopathy classification

Diabetic retinopathy (DR), the leading cause of vision loss among diabetic adults worldwide, underscores the importance of early detection and timely treatment using fundus images to prevent vision loss. However, existing deep learning methods struggle to capture the correlation and contextual information of subtle lesion features with the current scale of dataset. To this end, we propose a novel Multi-scale Spatial-aware Transformer Network (MSTNet) for DR classification. MSTNet encodes information from image patches at varying scales as input features, constructing a dual-pathway backbone network comprised of two Transformer encoders of different sizes to extract both local details and global context from images. To fully leverage structural prior knowledge, we introduce a Spatial-aware Module (SAM) to capture spatial local information within the images. Furthermore, considering the differences between medical and natural images, specifically that regions of interest in medical images often lack distinct subjectivity and continuity, we employ a Multiple Instance Learning (MIL) strategy to aggregate features from diverse regions, thereby enhancing correlation to subtle lesion areas. Ultimately, a cross-fusion classifier integrates dual-pathway features to produce the final classification result. We evaluate MSTNet on four public DR datasets, including APTOS2019, RFMiD2020, Messidor, and IDRiD. Extensive experiments demonstrate that MSTNet exhibits superior diagnostic and grading accuracy, achieving improvements of up to 2.0% in terms of ACC and 1.2% in terms of F1 score, highlighting its effectiveness in accurately assessing fundus images.

The outcome of diabetic retinopathy health education program in patients with type 2 diabetes mellitus: a quasi-experimental study

Diabetic retinopathy (DR) may develop into sight-threatening DR and vision loss if early intervention is not carried out. This study was aimed to assess the effectiveness of DR health education program for patients with type 2 diabetes mellitus (T2DM). The quasi-experimental research design was applied. The intervention group underwent a web-based DR health education program while the control group was followed up the usual way at an ophthalmology clinic for 1 year. Data were analysed using descriptive statistics, repeated measures ANOVA and general linear model to evaluate the mean difference between groups. A total of 180 patients with T2DM were enrolled in the study, with equal number in the control and intervention groups, respectively, with 28% of dropout rate. There was a significant mean difference in knowledge score [F (1178)  = 116.57, P = 0.001], diabetes self-care [F (1178)  = 116.57, P = 0.001] and visual-related quality of life [F (1178)  = 12.70), P = 0.001] between the control and the intervention groups. The intervention group scored the highest in all three categories. Educational interventions positively affected adherence to self-care and visual-related quality of life in type 2 diabetic patients as shown in this study. DRHEP should be considered an added benefit in T2DM management, starting with comprehensive care enrollment.

Enhancing diabetic retinopathy diagnosis: automatic segmentation of hyperreflective foci in OCT via deep learning

OBJECTIVE

Hyperreflective foci (HRF) are small, punctate lesions ranging from 20 to 50 μ m and exhibiting high reflective intensity in optical coherence tomography (OCT) images of patients with diabetic retinopathy (DR). The purpose of the model proposed in this paper is to precisely identify and segment the HRF in OCT images of patients with DR. This method is essential for assisting ophthalmologists in the early diagnosis and assessing the effectiveness of treatment and prognosis. In this study, we introduce an HRF segmentation algorithm based on KiU-Net, the algorithm that comprises the Kite-Net branch using up-sampling coding to collect more detailed information and a three-layer U-Net branch to extract high-level semantic information. To enhance the capacity of a single-branch network, we also design a cross-attention block (CAB) which combines the information extracted from two branches. The experimental results demonstrate that the number of parameters of our model is significantly reduced, and the sensitivity (SE) and the dice similarity coefficient (DSC) are respectively improved to 72.90 % and 66.84 % . Considering the SE and precision(P) of the segmentation, as well as the recall ratio and recall P of HRF, we believe that this model outperforms most advanced medical image segmentation algorithms and significantly relieves the strain on ophthalmologists.

PURPOSE

Hyperreflective foci (HRF) are small, punctate lesions ranging from 20 to 50 μm with high reflective intensity in optical coherence tomography (OCT) images of patients with diabetic retinopathy (DR). This study aims to develop a model that precisely identifies and segments HRF in OCT images of DR patients. Accurate segmentation of HRF is essential for assisting ophthalmologists in early diagnosis and in assessing the effectiveness of treatment and prognosis.

METHODS

We introduce an HRF segmentation algorithm based on the KiU-Net architecture. The model comprises two branches: a Kite-Net branch that uses up-sampling coding to capture detailed information, and a three-layer U-Net branch that extracts high-level semantic information. To enhance the capacity of the network, we designed a cross-attention block (CAB) that combines the information extracted from both branches, effectively integrating detail and semantic features.

RESULTS

Experimental results demonstrate that our model significantly reduces the number of parameters while improving performance. The sensitivity (SE) and Dice Similarity Coefficient (DSC) of our model are improved to 72.90% and 66.84%, respectively. Considering the SE and precision (P) of the segmentation, as well as the recall ratio and precision of HRF detection, our model outperforms most advanced medical image segmentation algorithms CONCLUSION: The proposed HRF segmentation algorithm effectively identifies and segments HRF in OCT images of DR patients, outperforming existing methods. This advancement can significantly alleviate the burden on ophthalmologists by aiding in early diagnosis and treatment evaluation, ultimately improving patient outcomes.

Multiple instance learning based classification of diabetic retinopathy in weakly-labeled widefield OCTA en face images

Diabetic retinopathy (DR), a pathologic change of the human retinal vasculature, is the leading cause of blindness in working-age adults with diabetes mellitus. Optical coherence tomography angiography (OCTA), a functional extension of optical coherence tomography, has shown potential as a tool for early diagnosis of DR through its ability to visualize the retinal vasculature in all spatial dimensions. Previously introduced deep learning-based classifiers were able to support the detection of DR in OCTA images, but require expert labeling at the pixel level, a labor-intensive and expensive process. We present a multiple instance learning-based network, MIL-ResNet,14 that is capable of detecting biomarkers in an OCTA dataset with high accuracy, without the need for annotations other than the information whether a scan is from a diabetic patient or not. The dataset we used for this study was acquired with a diagnostic ultra-widefield swept-source OCT device with a MHz A-scan rate. We were able to show that our proposed method outperforms previous state-of-the-art networks for this classification task, ResNet14 and VGG16. In addition, our network pays special attention to clinically relevant biomarkers and is robust against adversarial attacks. Therefore, we believe that it could serve as a powerful diagnostic decision support tool for clinical ophthalmic screening.

Detecting dry eye from ocular surface videos based on deep learning

OBJECTIVE

To assess the performance of convolutional neural networks (CNNs) for automated diagnosis of dry eye (DE) in patients undergoing video keratoscopy based on single ocular surface video frames.

DESIGN

This retrospective cohort study included 244 ocular surface videos from 244 eyes of 244 subjects based on corneal topography. A total of 116 eyes were normal while 128 eyes had DE based on clinical evaluations.

METHODS

We developed a deep transfer learning model to directly identify DE from ocular surface videos. We evaluated the performance of the CNN model based on objective accuracy metrics. We assessed the clinical relevance of the findings by evaluating class activations maps.

MAIN OUTCOME MEASURE

Area under the receiver operating characteristics curve (AUC), accuracy, specificity, and sensitivity.

RESULTS

The AUC of the model for discriminating normal eyes from eyes with DE was 0.98. Network activation maps suggested that the lower paracentral cornea was the most important region for detection of DE by the CNN model.

CONCLUSIONS

Deep transfer learning achieved a high diagnostic accuracy in detecting DE based on non-invasive ocular surface videos at levels that may prove useful in clinical practice.

Deep Learning and Medical Image Processing Techniques for Diabetic Retinopathy: A Survey of Applications, Challenges, and Future Trends

Diabetic retinopathy (DR) is a common eye retinal disease that is widely spread all over the world. It leads to the complete loss of vision based on the level of severity. It damages both retinal blood vessels and the eye's microscopic interior layers. To avoid such issues, early detection of DR is essential in association with routine screening methods to discover mild causes in manual initiation. But these diagnostic procedures are extremely difficult and expensive. The unique contributions of the study include the following: first, providing detailed background of the DR disease and the traditional detection techniques. Second, the various imaging techniques and deep learning applications in DR are presented. Third, the different use cases and real-life scenarios are explored relevant to DR detection wherein deep learning techniques have been implemented. The study finally highlights the potential research opportunities for researchers to explore and deliver effective performance results in diabetic retinopathy detection.

Detection of Diabetic Retinopathy Using Extracted 3D Features from OCT Images

Diabetic retinopathy (DR) is a major health problem that can lead to vision loss if not treated early. In this study, a three-step system for DR detection utilizing optical coherence tomography (OCT) is presented. First, the proposed system segments the retinal layers from the input OCT images. Second, 3D features are extracted from each retinal layer that include the first-order reflectivity and the 3D thickness of the individual OCT layers. Finally, backpropagation neural networks are used to classify OCT images. Experimental studies on 188 cases confirm the advantages of the proposed system over related methods, achieving an accuracy of 96.81%, using the leave-one-subject-out (LOSO) cross-validation. These outcomes show the potential of the suggested method for DR detection using OCT images.

Deep Learning for Diabetic Retinopathy Analysis: A Review, Research Challenges, and Future Directions

Deep learning (DL) enables the creation of computational models comprising multiple processing layers that learn data representations at multiple levels of abstraction. In the recent past, the use of deep learning has been proliferating, yielding promising results in applications across a growing number of fields, most notably in image processing, medical image analysis, data analysis, and bioinformatics. DL algorithms have also had a significant positive impact through yielding improvements in screening, recognition, segmentation, prediction, and classification applications across different domains of healthcare, such as those concerning the abdomen, cardiac, pathology, and retina. Given the extensive body of recent scientific contributions in this discipline, a comprehensive review of deep learning developments in the domain of diabetic retinopathy (DR) analysis, viz., screening, segmentation, prediction, classification, and validation, is presented here. A critical analysis of the relevant reported techniques is carried out, and the associated advantages and limitations highlighted, culminating in the identification of research gaps and future challenges that help to inform the research community to develop more efficient, robust, and accurate DL models for the various challenges in the monitoring and diagnosis of DR.

The Role of Medical Image Modalities and AI in the Early Detection, Diagnosis and Grading of Retinal Diseases: A Survey

Traditional dilated ophthalmoscopy can reveal diseases, such as age-related macular degeneration (AMD), diabetic retinopathy (DR), diabetic macular edema (DME), retinal tear, epiretinal membrane, macular hole, retinal detachment, retinitis pigmentosa, retinal vein occlusion (RVO), and retinal artery occlusion (RAO). Among these diseases, AMD and DR are the major causes of progressive vision loss, while the latter is recognized as a world-wide epidemic. Advances in retinal imaging have improved the diagnosis and management of DR and AMD. In this review article, we focus on the variable imaging modalities for accurate diagnosis, early detection, and staging of both AMD and DR. In addition, the role of artificial intelligence (AI) in providing automated detection, diagnosis, and staging of these diseases will be surveyed. Furthermore, current works are summarized and discussed. Finally, projected future trends are outlined. The work done on this survey indicates the effective role of AI in the early detection, diagnosis, and staging of DR and/or AMD. In the future, more AI solutions will be presented that hold promise for clinical applications.

Deep Learning Models for Segmenting Non-perfusion Area of Color Fundus Photographs in Patients With Branch Retinal Vein Occlusion

Purpose

To develop artificial intelligence (AI)-based deep learning (DL) models for automatically detecting the ischemia type and the non-perfusion area (NPA) from color fundus photographs (CFPs) of patients with branch retinal vein occlusion (BRVO).

Methods

This was a retrospective analysis of 274 CFPs from patients diagnosed with BRVO. All DL models were trained using a deep convolutional neural network (CNN) based on 45 degree CFPs covering the fovea and the optic disk. We first trained a DL algorithm to identify BRVO patients with or without the necessity of retinal photocoagulation from 219 CFPs and validated the algorithm on 55 CFPs. Next, we trained another DL algorithm to segment NPA from 104 CFPs and validated it on 29 CFPs, in which the NPA was manually delineated by 3 experienced ophthalmologists according to fundus fluorescein angiography. Both DL models have been cross-validated 5-fold. The recall, precision, accuracy, and area under the curve (AUC) were used to evaluate the DL models in comparison with three types of independent ophthalmologists of different seniority.

Results

In the first DL model, the recall, precision, accuracy, and area under the curve (AUC) were 0.75 ± 0.08, 0.80 ± 0.07, 0.79 ± 0.02, and 0.82 ± 0.03, respectively, for predicting the necessity of laser photocoagulation for BRVO CFPs. The second DL model was able to segment NPA in CFPs of BRVO with an AUC of 0.96 ± 0.02. The recall, precision, and accuracy for segmenting NPA was 0.74 ± 0.05, 0.87 ± 0.02, and 0.89 ± 0.02, respectively. The performance of the second DL model was nearly comparable with the senior doctors and significantly better than the residents.

Conclusion

These results indicate that the DL models can directly identify and segment retinal NPA from the CFPs of patients with BRVO, which can further guide laser photocoagulation. Further research is needed to identify NPA of the peripheral retina in BRVO, or other diseases, such as diabetic retinopathy.

Which Color Channel Is Better for Diagnosing Retinal Diseases Automatically in Color Fundus Photographs?

Color fundus photographs are the most common type of image used for automatic diagnosis of retinal diseases and abnormalities. As all color photographs, these images contain information about three primary colors, i.e., red, green, and blue, in three separate color channels. This work aims to understand the impact of each channel in the automatic diagnosis of retinal diseases and abnormalities. To this end, the existing works are surveyed extensively to explore which color channel is used most commonly for automatically detecting four leading causes of blindness and one retinal abnormality along with segmenting three retinal landmarks. From this survey, it is clear that all channels together are typically used for neural network-based systems, whereas for non-neural network-based systems, the green channel is most commonly used. However, from the previous works, no conclusion can be drawn regarding the importance of the different channels. Therefore, systematic experiments are conducted to analyse this. A well-known U-shaped deep neural network (U-Net) is used to investigate which color channel is best for segmenting one retinal abnormality and three retinal landmarks.

Assessment of image quality on color fundus retinal images using the automatic retinal image analysis

Image quality assessment is essential for retinopathy detection on color fundus retinal image. However, most studies focused on the classification of good and poor quality without considering the different types of poor quality. This study developed an automatic retinal image analysis (ARIA) method, incorporating transfer net ResNet50 deep network with the automatic features generation approach to automatically assess image quality, and distinguish eye-abnormality-associated-poor-quality from artefact-associated-poor-quality on color fundus retinal images. A total of 2434 retinal images, including 1439 good quality and 995 poor quality (483 eye-abnormality-associated-poor-quality and 512 artefact-associated-poor-quality), were used for training, testing, and 10-ford cross-validation. We also analyzed the external validation with the clinical diagnosis of eye abnormality as the reference standard to evaluate the performance of the method. The sensitivity, specificity, and accuracy for testing good quality against poor quality were 98.0%, 99.1%, and 98.6%, and for differentiating between eye-abnormality-associated-poor-quality and artefact-associated-poor-quality were 92.2%, 93.8%, and 93.0%, respectively. In external validation, our method achieved an area under the ROC curve of 0.997 for the overall quality classification and 0.915 for the classification of two types of poor quality. The proposed approach, ARIA, showed good performance in testing, 10-fold cross validation and external validation. This study provides a novel angle for image quality screening based on the different poor quality types and corresponding dealing methods. It suggested that the ARIA can be used as a screening tool in the preliminary stage of retinopathy grading by telemedicine or artificial intelligence analysis.

A systematic review of transfer learning-based approaches for diabetic retinopathy detection

Cases of diabetes and related diabetic retinopathy (DR) have been increasing at an alarming rate in modern times. Early detection of DR is an important problem since it may cause permanent blindness in the late stages. In the last two decades, many different approaches have been applied in DR detection. Reviewing academic literature shows that deep neural networks (DNNs) have become the most preferred approach for DR detection. Among these DNN approaches, Convolutional Neural Network (CNN) models are the most used ones in the field of medical image classification. Designing a new CNN architecture is a tedious and time-consuming approach. Additionally, training an enormous number of parameters is also a difficult task. Due to this reason, instead of training CNNs from scratch, using pre-trained models has been suggested in recent years as transfer learning approach. Accordingly, the present study as a review focuses on DNN and Transfer Learning based applications of DR detection considering 43 publications between 2015 and 2021. The published papers are summarized using 3 figures and 10 tables, giving information about 29 pre-trained CNN models, 13 DR data sets and standard performance metrics.

A Deep Learning Framework for Earlier Prediction of Diabetic Retinopathy from Fundus Photographs

Diabetic patients can also be identified immediately utilizing retinopathy photos, but it is a challenging task. The blood veins visible in fundus photographs are used in several disease diagnosis approaches. We sought to replicate the findings published in implementation and verification of a deep learning approach for diabetic retinopathy identification in retinal fundus pictures. To address this issue, the suggested investigative study uses recurrent neural networks (RNN) to retrieve characteristics from deep networks. As a result, using computational approaches to identify certain disorders automatically might be a fantastic solution. We developed and tested several iterations of a deep learning framework to forecast the progression of diabetic retinopathy in diabetic individuals who have undergone teleretinal diabetic retinopathy assessment in a basic healthcare environment. A collection of one-field or three-field colour fundus pictures served as the input for both iterations. Utilizing the proposed DRNN methodology, advanced identification of the diabetic state was performed utilizing HE detected in an eye's blood vessel. This research demonstrates the difficulties in duplicating deep learning approach findings, as well as the necessity for more reproduction and replication research to verify deep learning techniques, particularly in the field of healthcare picture processing. This development investigates the utilization of several other Deep Neural Network Frameworks on photographs from the dataset after they have been treated to suitable image computation methods such as local average colour subtraction to assist in highlighting the germane characteristics from a fundoscopy, thus, also enhancing the identification and assessment procedure of diabetic retinopathy and serving as a skilled guidelines framework for practitioners all over the globe.

A Hybrid Deep Learning-Metaheuristic Model for Diagnosis of Diabetic Retinopathy

International Diabetes Federation (IDF) reports that diabetes is one of the rapidly growing illnesses. About 463 million adults between 20-79 years have diabetes. There are also millions of undiagnosed patients. It is estimated that there will be about 578 million diabetics by 2030 [1]. Diabetes reasons different eye diseases. Diabetic retinopathy (DR) is one of them and is also one of the most common vision loss or blindness worldwide. DR progresses slowly and has few indicators in the early stages. It makes the diagnosis of DR a problematic task. Automated systems promise to support the diagnosis of DR. Many deep learning-based models have been developed for DR classification. This study aims to support ophthalmologists in the diagnosis process and increase the diagnosis performance of DR through a hybrid model. A publicly available Messidor-2 dataset was used in this study, comprised of retinal images. In the proposed model, first, images were pre-processed and a deep learning model, namely, InceptionV3 was used in feature extraction where a transfer learning approach is applied. Next, the number of features in obtained feature vectors was decreased with feature selection by Simulated Annealing (SA). Lastly, the best representation features were used in XGBoost model. The XGBoost algorithm gives an accuracy of 92.26% in a binary classification task. This study shows that a pre-trained ConvNet with a metaheuristic algorithm for feature selection gives a satisfactory result in the diagnosis of DR.

The Role of Different Retinal Imaging Modalities in Predicting Progression of Diabetic Retinopathy: A Survey

Diabetic retinopathy (DR) is a devastating condition caused by progressive changes in the retinal microvasculature. It is a leading cause of retinal blindness in people with diabetes. Long periods of uncontrolled blood sugar levels result in endothelial damage, leading to macular edema, altered retinal permeability, retinal ischemia, and neovascularization. In order to facilitate rapid screening and diagnosing, as well as grading of DR, different retinal modalities are utilized. Typically, a computer-aided diagnostic system (CAD) uses retinal images to aid the ophthalmologists in the diagnosis process. These CAD systems use a combination of machine learning (ML) models (e.g., deep learning (DL) approaches) to speed up the diagnosis and grading of DR. In this way, this survey provides a comprehensive overview of different imaging modalities used with ML/DL approaches in the DR diagnosis process. The four imaging modalities that we focused on are fluorescein angiography, fundus photographs, optical coherence tomography (OCT), and OCT angiography (OCTA). In addition, we discuss limitations of the literature that utilizes such modalities for DR diagnosis. In addition, we introduce research gaps and provide suggested solutions for the researchers to resolve. Lastly, we provide a thorough discussion about the challenges and future directions of the current state-of-the-art DL/ML approaches. We also elaborate on how integrating different imaging modalities with the clinical information and demographic data will lead to promising results for the scientists when diagnosing and grading DR. As a result of this article's comparative analysis and discussion, it remains necessary to use DL methods over existing ML models to detect DR in multiple modalities.

Automatic Screening for Ocular Anomalies Using Fundus Photographs

SIGNIFICANCE

Screening for ocular anomalies using fundus photography is key to prevent vision impairment and blindness. With the growing and aging population, automated algorithms that can triage fundus photographs and provide instant referral decisions are relevant to scale-up screening and face the shortage of ophthalmic expertise.

PURPOSE

This study aimed to develop a deep learning algorithm that detects any ocular anomaly in fundus photographs and to evaluate this algorithm for "normal versus anomalous" eye examination classification in the diabetic and general populations.

METHODS

The deep learning algorithm was developed and evaluated in two populations: the diabetic and general populations. Our patient cohorts consist of 37,129 diabetic patients from the OPHDIAT diabetic retinopathy screening network in Paris, France, and 7356 general patients from the OphtaMaine private screening network, in Le Mans, France. Each data set was divided into a development subset and a test subset of more than 4000 examinations each. For ophthalmologist/algorithm comparison, a subset of 2014 examinations from the OphtaMaine test subset was labeled by a second ophthalmologist. First, the algorithm was trained on the OPHDIAT development subset. Then, it was fine-tuned on the OphtaMaine development subset.

RESULTS

On the OPHDIAT test subset, the area under the receiver operating characteristic curve for normal versus anomalous classification was 0.9592. On the OphtaMaine test subset, the area under the receiver operating characteristic curve was 0.8347 before fine-tuning and 0.9108 after fine-tuning. On the ophthalmologist/algorithm comparison subset, the second ophthalmologist achieved a specificity of 0.8648 and a sensitivity of 0.6682. For the same specificity, the fine-tuned algorithm achieved a sensitivity of 0.8248.

CONCLUSIONS

The proposed algorithm compares favorably with human performance for normal versus anomalous eye examination classification using fundus photography. Artificial intelligence, which previously targeted a few retinal pathologies, can be used to screen for ocular anomalies comprehensively.

Downregulation of circ-UBAP2 ameliorates oxidative stress and dysfunctions of human retinal microvascular endothelial cells (hRMECs) via miR-589-5p/EGR1 axis

Hsa_circ_0001850_circ_0001850 (circ-UBAP2) is reported to be upregulated in diabetic retinopathy (DR). However, its role in high glucose (HG)-triggered oxidative stress and vascular dysfunction in DR is unclear. This study aimed to investigate the potential of circUBAP2 in DR. The content of malondialdehyde (MDA), and the activities of superoxide dismutase (SOD) and glutathione peroxidase (GSH-PX) were analyzed using the corresponding kits. Western blotting was performed to detect the protein expression of Nrf2, HO-1, and SOD-1. MTT assay was conducted to assess cell viability. A transwell migration assay was used to determine the migration ability of human retinal microvascular endothelial cells (hRMECs). A Matrigel tube formation assay was performed to analyze tube formation. The targeting relationships were verified using a luciferase reporter assay. We found that the circ-UBAP2 expression increased in DR patients and HG-treated hRMECs. Downregulation of circ-UBAP2 ameliorated HG-induced oxidative stress and dysfunction of hRMECs. Mechanistically, circ-UBAP2 sponges miR-589-5p, which is downregulated under hyperglycemic conditions. In addition, EGR1 was confirmed to be a target gene of miR-589-5p and was overexpressed in HG-treated hRMECs. In addition, EGR1 reversed the effects of miR-589-5p and induced oxidative stress and dysfunction in hRMECs. Taken together, knockdown of circ-UBAP2 relieved HG-induced oxidative stress and dysfunctions of the hRMECs through the miR-589-5p/EGR1 axis, which may offer a promising therapeutic target for DR.

Deep-Learning-Based Pre-Diagnosis Assessment Module for Retinal Photographs: A Multicenter Study

Purpose

Artificial intelligence (AI) deep learning (DL) has been shown to have significant potential for eye disease detection and screening on retinal photographs in different clinical settings, particular in primary care. However, an automated pre-diagnosis image assessment is essential to streamline the application of the developed AI-DL algorithms. In this study, we developed and validated a DL-based pre-diagnosis assessment module for retinal photographs, targeting image quality (gradable vs. ungradable), field of view (macula-centered vs. optic-disc-centered), and laterality of the eye (right vs. left).

Methods

A total of 21,348 retinal photographs from 1914 subjects from various clinical settings in Hong Kong, Singapore, and the United Kingdom were used for training, internal validation, and external testing for the DL module, developed by two DL-based algorithms (EfficientNet-B0 and MobileNet-V2).

Results

For image-quality assessment, the pre-diagnosis module achieved area under the receiver operating characteristic curve (AUROC) values of 0.975, 0.999, and 0.987 in the internal validation dataset and the two external testing datasets, respectively. For field-of-view assessment, the module had an AUROC value of 1.000 in all of the datasets. For laterality-of-the-eye assessment, the module had AUROC values of 1.000, 0.999, and 0.985 in the internal validation dataset and the two external testing datasets, respectively.

Conclusions

Our study showed that this three-in-one DL module for assessing image quality, field of view, and laterality of the eye of retinal photographs achieved excellent performance and generalizability across different centers and ethnicities.

Translational Relevance

The proposed DL-based pre-diagnosis module realized accurate and automated assessments of image quality, field of view, and laterality of the eye of retinal photographs, which could be further integrated into AI-based models to improve operational flow for enhancing disease screening and diagnosis.

Automated Detection and Diagnosis of Diabetic Retinopathy: A Comprehensive Survey

Diabetic Retinopathy (DR) is a leading cause of vision loss in the world. In the past few years, artificial intelligence (AI) based approaches have been used to detect and grade DR. Early detection enables appropriate treatment and thus prevents vision loss. For this purpose, both fundus and optical coherence tomography (OCT) images are used to image the retina. Next, Deep-learning (DL)-/machine-learning (ML)-based approaches make it possible to extract features from the images and to detect the presence of DR, grade its severity and segment associated lesions. This review covers the literature dealing with AI approaches to DR such as ML and DL in classification and segmentation that have been published in the open literature within six years (2016-2021). In addition, a comprehensive list of available DR datasets is reported. This list was constructed using both the PICO (P-Patient, I-Intervention, C-Control, O-Outcome) and Preferred Reporting Items for Systematic Review and Meta-analysis (PRISMA) 2009 search strategies. We summarize a total of 114 published articles which conformed to the scope of the review. In addition, a list of 43 major datasets is presented.

Performance and Limitation of Machine Learning Algorithms for Diabetic Retinopathy Screening: Meta-analysis

Background

Diabetic retinopathy (DR), whose standard diagnosis is performed by human experts, has high prevalence and requires a more efficient screening method. Although machine learning (ML)–based automated DR diagnosis has gained attention due to recent approval of IDx-DR, performance of this tool has not been examined systematically, and the best ML technique for use in a real-world setting has not been discussed.

Objective

The aim of this study was to systematically examine the overall diagnostic accuracy of ML in diagnosing DR of different categories based on color fundus photographs and to determine the state-of-the-art ML approach.

Methods

Published studies in PubMed and EMBASE were searched from inception to June 2020. Studies were screened for relevant outcomes, publication types, and data sufficiency, and a total of 60 out of 2128 (2.82%) studies were retrieved after study selection. Extraction of data was performed by 2 authors according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and the quality assessment was performed according to the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2). Meta-analysis of diagnostic accuracy was pooled using a bivariate random effects model. The main outcomes included diagnostic accuracy, sensitivity, and specificity of ML in diagnosing DR based on color fundus photographs, as well as the performances of different major types of ML algorithms.

Results

The primary meta-analysis included 60 color fundus photograph studies (445,175 interpretations). Overall, ML demonstrated high accuracy in diagnosing DR of various categories, with a pooled area under the receiver operating characteristic (AUROC) ranging from 0.97 (95% CI 0.96-0.99) to 0.99 (95% CI 0.98-1.00). The performance of ML in detecting more-than-mild DR was robust (sensitivity 0.95; AUROC 0.97), and by subgroup analyses, we observed that robust performance of ML was not limited to benchmark data sets (sensitivity 0.92; AUROC 0.96) but could be generalized to images collected in clinical practice (sensitivity 0.97; AUROC 0.97). Neural network was the most widely used method, and the subgroup analysis revealed a pooled AUROC of 0.98 (95% CI 0.96-0.99) for studies that used neural networks to diagnose more-than-mild DR.

Conclusions

This meta-analysis demonstrated high diagnostic accuracy of ML algorithms in detecting DR on color fundus photographs, suggesting that state-of-the-art, ML-based DR screening algorithms are likely ready for clinical applications. However, a significant portion of the earlier published studies had methodology flaws, such as the lack of external validation and presence of spectrum bias. The results of these studies should be interpreted with caution.

Deep learning-based detection and stage grading for optimising diagnosis of diabetic retinopathy

AIMS

To establish an automated method for identifying referable diabetic retinopathy (DR), defined as moderate nonproliferative DR and above, using deep learning-based lesion detection and stage grading.

MATERIALS AND METHODS

A set of 12,252 eligible fundus images of diabetic patients were manually annotated by 45 licenced ophthalmologists and were randomly split into training, validation, and internal test sets (ratio of 7:1:2). Another set of 565 eligible consecutive clinical fundus images was established as an external test set. For automated referable DR identification, four deep learning models were programmed based on whether two factors were included: DR-related lesions and DR stages. Sensitivity, specificity and the area under the receiver operating characteristic curve (AUC) were reported for referable DR identification, while precision and recall were reported for lesion detection.

RESULTS

Adding lesion information to the five-stage grading model improved the AUC (0.943 vs. 0.938), sensitivity (90.6% vs. 90.5%) and specificity (80.7% vs. 78.5%) of the model for identifying referable DR in the internal test set. Adding stage information to the lesion-based model increased the AUC (0.943 vs. 0.936) and sensitivity (90.6% vs. 76.7%) of the model for identifying referable DR in the internal test set. Similar trends were also seen in the external test set. DR lesion types with high precision results were preretinal haemorrhage, hard exudate, vitreous haemorrhage, neovascularisation, cotton wool spots and fibrous proliferation.

CONCLUSIONS

The herein described automated model employed DR lesions and stage information to identify referable DR and displayed better diagnostic value than models built without this information.

Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis

Deep learning (DL) has the potential to transform medical diagnostics. However, the diagnostic accuracy of DL is uncertain. Our aim was to evaluate the diagnostic accuracy of DL algorithms to identify pathology in medical imaging. Searches were conducted in Medline and EMBASE up to January 2020. We identified 11,921 studies, of which 503 were included in the systematic review. Eighty-two studies in ophthalmology, 82 in breast disease and 115 in respiratory disease were included for meta-analysis. Two hundred twenty-four studies in other specialities were included for qualitative review. Peer-reviewed studies that reported on the diagnostic accuracy of DL algorithms to identify pathology using medical imaging were included. Primary outcomes were measures of diagnostic accuracy, study design and reporting standards in the literature. Estimates were pooled using random-effects meta-analysis. In ophthalmology, AUC's ranged between 0.933 and 1 for diagnosing diabetic retinopathy, age-related macular degeneration and glaucoma on retinal fundus photographs and optical coherence tomography. In respiratory imaging, AUC's ranged between 0.864 and 0.937 for diagnosing lung nodules or lung cancer on chest X-ray or CT scan. For breast imaging, AUC's ranged between 0.868 and 0.909 for diagnosing breast cancer on mammogram, ultrasound, MRI and digital breast tomosynthesis. Heterogeneity was high between studies and extensive variation in methodology, terminology and outcome measures was noted. This can lead to an overestimation of the diagnostic accuracy of DL algorithms on medical imaging. There is an immediate need for the development of artificial intelligence-specific EQUATOR guidelines, particularly STARD, in order to provide guidance around key issues in this field.

Segmentation of Laser Marks of Diabetic Retinopathy in the Fundus Photographs Using Lightweight U-Net

Diabetic retinopathy (DR) is a prevalent vision-threatening disease worldwide. Laser marks are the scars left after panretinal photocoagulation, a treatment to prevent patients with severe DR from losing vision. In this study, we develop a deep learning algorithm based on the lightweight U-Net to segment laser marks from the color fundus photos, which could help indicate a stage or providing valuable auxiliary information for the care of DR patients. We prepared our training and testing data, manually annotated by trained and experienced graders from Image Reading Center, Zhongshan Ophthalmic Center, publicly available to fill the vacancy of public image datasets dedicated to the segmentation of laser marks. The lightweight U-Net, along with two postprocessing procedures, achieved an AUC of 0.9824, an optimal sensitivity of 94.16%, and an optimal specificity of 92.82% on the segmentation of laser marks in fundus photographs. With accurate segmentation and high numeric metrics, the lightweight U-Net method showed its reliable performance in automatically segmenting laser marks in fundus photographs, which could help the AI assist the diagnosis of DR in the severe stage.

A scoping review of transfer learning research on medical image analysis using ImageNet

OBJECTIVE

Employing transfer learning (TL) with convolutional neural networks (CNNs), well-trained on non-medical ImageNet dataset, has shown promising results for medical image analysis in recent years. We aimed to conduct a scoping review to identify these studies and summarize their characteristics in terms of the problem description, input, methodology, and outcome.

MATERIALS AND METHODS

To identify relevant studies, MEDLINE, IEEE, and ACM digital library were searched for studies published between June 1st, 2012 and January 2nd, 2020. Two investigators independently reviewed articles to determine eligibility and to extract data according to a study protocol defined a priori.

RESULTS

After screening of 8421 articles, 102 met the inclusion criteria. Of 22 anatomical areas, eye (18%), breast (14%), and brain (12%) were the most commonly studied. Data augmentation was performed in 72% of fine-tuning TL studies versus 15% of the feature-extracting TL studies. Inception models were the most commonly used in breast related studies (50%), while VGGNet was the common in eye (44%), skin (50%) and tooth (57%) studies. AlexNet for brain (42%) and DenseNet for lung studies (38%) were the most frequently used models. Inception models were the most frequently used for studies that analyzed ultrasound (55%), endoscopy (57%), and skeletal system X-rays (57%). VGGNet was the most common for fundus (42%) and optical coherence tomography images (50%). AlexNet was the most frequent model for brain MRIs (36%) and breast X-Rays (50%). 35% of the studies compared their model with other well-trained CNN models and 33% of them provided visualization for interpretation.

DISCUSSION

This study identified the most prevalent tracks of implementation in the literature for data preparation, methodology selection and output evaluation for various medical image analysis tasks. Also, we identified several critical research gaps existing in the TL studies on medical image analysis. The findings of this scoping review can be used in future TL studies to guide the selection of appropriate research approaches, as well as identify research gaps and opportunities for innovation.

Convolutional Neural Networks with Transfer Learning for Recognition of COVID-19: A Comparative Study of Different Approaches

To judge the ability of convolutional neural networks (CNNs) to effectively and efficiently transfer image representations learned on the ImageNet dataset to the task of recognizing COVID-19 in this work, we propose and analyze four approaches. For this purpose, we use VGG16, ResNetV2, InceptionResNetV2, DenseNet121, and MobileNetV2 CNN models pre-trained on ImageNet dataset to extract features from X-ray images of COVID and Non-COVID patients. Simulations study performed by us reveal that these pre-trained models have a different level of ability to transfer image representation. We find that in the approaches that we have proposed, if we use either ResNetV2 or DenseNet121 to extract features, then the performance of these approaches to detect COVID-19 is better. One of the important findings of our study is that the use of principal component analysis for feature selection improves efficiency. The approach using the fusion of features outperforms all the other approaches, and with this approach, we could achieve an accuracy of 0.94 for a three-class classification problem. This work will not only be useful for COVID-19 detection but also for any domain with small datasets.

Lesion-aware convolutional neural network for chest radiograph classification

AIM

To investigate the performance of a deep-learning approach termed lesion-aware convolutional neural network (LACNN) to identify 14 different thoracic diseases on chest X-rays (CXRs).

MATERIALS AND METHODS

In total, 10,738 CXRs of 3,526 patients were collected retrospectively. Of these, 1,937 CXRs of 598 patients were selected for training and optimising the lesion-detection network (LDN) of LACNN. The remaining 8,801 CXRs from 2,928 patients were used to train and test the classification network of LACNN. The discriminative performance of the deep-learning approach was compared with that obtained by the radiologists. In addition, its generalisation was validated on the independent public dataset, ChestX-ray14. The decision-making process of the model was visualised by occlusion testing, and the effect of the integration of CXRs and non-image data on model performance was also investigated. In a systematic evaluation, F1 score, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) metrics were calculated.

RESULTS

The model generated statistically significantly higher AUC performance compared with radiologists on atelectasis, mass, and nodule, with AUC values of 0.831 (95% confidence interval [CI]: 0.807-0.855), 0.959 (95% CI: 0.944-0.974), and 0.928 (95% CI: 0.906-0.950), respectively. For the other 11 pathologies, there were no statistically significant differences. The average time to complete each CXR classification in the testing dataset was substantially longer for the radiologists (∼35 seconds) than for the LACNN (∼0.197 seconds). In the ChestX-ray14 dataset, the present model also showed competitive performance in comparison with other state-of-the-art deep-learning approaches. Model performance was slightly improved when introducing non-image data.

CONCLUSION

The proposed LACNN achieved radiologist-level performance in identifying thoracic diseases on CXRs, and could potentially expand patient access to CXR diagnostics.

Application of non-mydriatic fundus examination and artificial intelligence to promote the screening of diabetic retinopathy in the endocrine clinic: an observational study of T2DM patients in Tianjin, China

Background

We aimed to determine the role of non-mydriatic fundus examination and artificial intelligence (AI) in screening diabetic retinopathy (DR) in patients with diabetes in the Metabolic Disease Management Center (MMC) in Tianjin, China.

Methods

Adult patients with type 2 diabetes mellitus who were first treated by MMC in Tianjin First Central Hospital and Tianjin 4th Center Hospital were divided into two groups according to the time that MMC was equipped with the non-mydriatic ophthalmoscope and AI system and could complete fundus examination independently (the former was the control group, the latter was the observation group). The observation indices were as follows: the incidence of DR, the fundus screening rate of the two groups, and fundus screening of diabetic patients with different course of disease.

Results

A total of 5039 patients were enrolled in this study. The incidence rate of DR was 18.6%, 29.8%, and 49.6% in patients with diabetes duration of ⩽1 year, 1-5 years, and >5 years, respectively. The screening rate of fundus in the observation group was significantly higher compared with the control group (81.3% versus 28.4%, χ ² = 1430.918, p < 0.001). The DR screening rate of the observation group was also significantly higher compared with the control group in patients with diabetes duration of ⩽1 year (77.3% versus 20.6%; χ ² = 797.534, p < 0.001), 1-5 years (82.5% versus 31.0%; χ ² = 197.124, p < 0.001) and ⩾5 years (86.9% versus 37.1%; χ² = 475.609, p < 0.001).

Conclusions

In the case of limited medical resources, MMC can carry out one-stop examination, treatment, and management of DR through non-mydratic fundus examination and AI assistance, thus incorporating the DR screening process into the endocrine clinic, so as to facilitate early diagnosis.

Automated detection of diabetic retinopathy in fundus images using fused features

Diabetic retinopathy (DR) is one of the severe eye conditions due to diabetes complication which can lead to vision loss if left untreated. In this paper, a computationally simple, yet very effective, DR detection method is proposed. First, a segmentation independent two-stage preprocessing based technique is proposed which can effectively extract DR pathognomonic signs; both bright and red lesions, and blood vessels from the eye fundus image. Then, the performance of Local Binary Patterns (LBP), Local Ternary Patterns (LTP), Dense Scale-Invariant Feature Transform (DSIFT) and Histogram of Oriented Gradients (HOG) as a feature descriptor for fundus images, is thoroughly analyzed. SVM kernel-based classifiers are trained and tested, using a 5-fold cross-validation scheme, on both newly acquired fundus image database from the local hospital and combined database created from the open-sourced available databases. The classification accuracy of 96.6% with 0.964 sensitivity and 0.969 specificity is achieved using a Cubic SVM classifier with LBP and LTP fused features for the local database. More importantly, in out-of-sample testing on the combined database, the model gives an accuracy of 95.21% with a sensitivity of 0.970 and specificity of 0.932. This indicates the proposed model is very well-fitted and generalized which is further corroborated by the presented train-test curves.

Leveraging Multimodal Deep Learning Architecture with Retina Lesion Information to Detect Diabetic Retinopathy

Purpose

To improve disease severity classification from fundus images using a hybrid architecture with symptom awareness for diabetic retinopathy (DR).

Methods

We used 26,699 fundus images of 17,834 diabetic patients from three Taiwanese hospitals collected in 2007 to 2018 for DR severity classification. Thirty-seven ophthalmologists verified the images using lesion annotation and severity classification as the ground truth. Two deep learning fusion architectures were proposed: late fusion, which combines lesion and severity classification models in parallel using a postprocessing procedure, and two-stage early fusion, which combines lesion detection and classification models sequentially and mimics the decision-making process of ophthalmologists. Messidor-2 was used with 1748 images to evaluate and benchmark the performance of the architecture. The primary evaluation metrics were classification accuracy, weighted κ statistic, and area under the receiver operating characteristic curve (AUC).

Results

For hospital data, a hybrid architecture achieved a good detection rate, with accuracy and weighted κ of 84.29% and 84.01%, respectively, for five-class DR grading. It also classified the images of early stage DR more accurately than conventional algorithms. The Messidor-2 model achieved an AUC of 97.09% in referral DR detection compared to AUC of 85% to 99% for state-of-the-art algorithms that learned from a larger database.

Conclusions

Our hybrid architectures strengthened and extracted characteristics from DR images, while improving the performance of DR grading, thereby increasing the robustness and confidence of the architectures for general use.

Translational Relevance

The proposed fusion architectures can enable faster and more accurate diagnosis of various DR pathologies than that obtained in current manual clinical practice.

Classification of advanced and early stages of diabetic retinopathy from non-diabetic subjects by an ordinary least squares modeling method applied to OCTA images

As the prevalence of diabetic retinopathy (DR) continues to rise, there is a need to develop computer-aided screening methods. The current study reports and validates an ordinary least squares (OLS) method to model optical coherence tomography angiography (OCTA) images and derive OLS parameters for classifying proliferative DR (PDR) and no/mild non-proliferative DR (NPDR) from non-diabetic subjects. OLS parameters were correlated with vessel metrics quantified from OCTA images and were used to determine predicted probabilities of PDR, no/mild NPDR, and non-diabetics. The classification rates of PDR and no/mild NPDR from non-diabetic subjects were 94% and 91%, respectively. The method had excellent predictive ability and was validated. With further development, the method may have potential clinical utility and contribute to image-based computer-aided screening and classification of stages of DR and other ocular and systemic diseases.

Transfer Learning for Automated OCTA Detection of Diabetic Retinopathy

Purpose

To test the feasibility of using deep learning for optical coherence tomography angiography (OCTA) detection of diabetic retinopathy.

Methods

A deep-learning convolutional neural network (CNN) architecture, VGG16, was employed for this study. A transfer learning process was implemented to retrain the CNN for robust OCTA classification. One dataset, consisting of images of 32 healthy eyes, 75 eyes with diabetic retinopathy (DR), and 24 eyes with diabetes but no DR (NoDR), was used for training and cross-validation. A second dataset consisting of 20 NoDR and 26 DR eyes was used for external validation. To demonstrate the feasibility of using artificial intelligence (AI) screening of DR in clinical environments, the CNN was incorporated into a graphical user interface (GUI) platform.

Results

With the last nine layers retrained, the CNN architecture achieved the best performance for automated OCTA classification. The cross-validation accuracy of the retrained classifier for differentiating among healthy, NoDR, and DR eyes was 87.27%, with 83.76% sensitivity and 90.82% specificity. The AUC metrics for binary classification of healthy, NoDR, and DR eyes were 0.97, 0.98, and 0.97, respectively. The GUI platform enabled easy validation of the method for AI screening of DR in a clinical environment.

Conclusions

With a transfer learning process for retraining, a CNN can be used for robust OCTA classification of healthy, NoDR, and DR eyes. The AI-based OCTA classification platform may provide a practical solution to reducing the burden of experienced ophthalmologists with regard to mass screening of DR patients.

Translational Relevance

Deep-learning-based OCTA classification can alleviate the need for manual graders and improve DR screening efficiency.

Deep Learning Frameworks for Diabetic Retinopathy Detection with Smartphone-based Retinal Imaging Systems

Diabetic Retinopathy (DR) may result in various degrees of vision loss and even blindness if not diagnosed in a timely manner. Therefore, having an annual eye exam helps early detection to prevent vision loss in earlier stages, especially for diabetic patients. Recent technological advances made smartphone-based retinal imaging systems available on the market to perform small-sized, low-powered, and affordable DR screening in diverse environments. However, the accuracy of DR detection depends on the field of view and image quality. Since smartphone-based retinal imaging systems have much more compact designs than a traditional fundus camera, captured images are likely to be the low quality with a smaller field of view. Our motivation in this paper is to develop an automatic DR detection model for smartphone-based retinal images using the deep learning approach with the ResNet50 network. This study first utilized the well-known AlexNet, GoogLeNet, and ResNet50 architectures, using the transfer learning approach. Second, these frameworks were retrained with retina images from several datasets including EyePACS, Messidor, IDRiD, and Messidor-2 to investigate the effect of using images from the single, cross, and multiple datasets. Third, the proposed ResNet50 model is applied to smartphone-based synthetic images to explore the DR detection accuracy of smartphone-based retinal imaging systems. Based on the vision-threatening diabetic retinopathy detection results, the proposed approach achieved a high classification accuracy of 98.6%, with a 98.2% sensitivity and a 99.1% specificity while its AUC was 0.9978 on the independent test dataset. As the main contributions, DR detection accuracy was improved using the deep transfer learning approach for the ResNet50 network with publicly available datasets and the effect of the field of view in smartphone-based retinal imaging was studied. Although a smaller number of images were used in the training set compared with the existing studies, considerably acceptable high accuracies for validation and testing data were obtained.