The possibility of the combination of OCT and fundus images for improving the diagnostic accuracy of deep learning for age-related macular degeneration: a preliminary experiment

Enhancing MRI Brain Images Using Contourlet Transform and Adaptive Histogram Equalization

Human brain can be viewed using MRI images. These images will be useful for physicians, only if their quality is good. We propose a new method called, Contourlet Based Two Stage Adaptive Histogram Equalization (CBTSA), that uses Nonsubsampled Contourlet Transform (NSCT) for smoothing images and adaptive histogram equalization (AHE), under two occasions, called stages, for enhancement of the low contrast MRI images. The given MRI image is fragmented into equal sized sub-images and NSCT is applied to each of the sub-images. AHE is imposed on each resultant sub-image. All processed images are merged and AHE is applied again to the merged image. The clarity of the output image obtained by our method has outperformed the output image produced by traditional methods. The quality was measured and compared using criteria like, Entropy, Absolute Mean Brightness Error (AMBE) and Peak Signal to Noise Ratio (PSNR).

Synthetic OCT data in challenging conditions: three-dimensional OCT and presence of abnormalities

Nowadays, retinal optical coherence tomography (OCT) plays an important role in ophthalmology and automatic analysis of the OCT is of real importance: image denoising facilitates a better diagnosis and image segmentation and classification are undeniably critical in treatment evaluation. Synthetic OCT was recently considered to provide a benchmark for quantitative comparison of automatic algorithms and to be utilized in the training stage of novel solutions based on deep learning. Due to complicated data structure in retinal OCTs, a limited number of delineated OCT datasets are already available in presence of abnormalities; furthermore, the intrinsic three-dimensional (3D) structure of OCT is ignored in many public 2D datasets. We propose a new synthetic method, applicable to 3D data and feasible in presence of abnormalities like diabetic macular edema (DME). In this method, a limited number of OCT data is used during the training step and the Active Shape Model is used to produce synthetic OCTs plus delineation of retinal boundaries and location of abnormalities. Statistical comparison of thickness maps showed that synthetic dataset can be used as a statistically acceptable representative of the original dataset (p > 0.05). Visual inspection of the synthesized vessels was also promising. Regarding the texture features of the synthesized datasets, Q-Q plots were used, and even in cases that the points have slightly digressed from the straight line, the p-values of the Kolmogorov–Smirnov test rejected the null hypothesis and showed the same distribution in texture features of the real and the synthetic data. The proposed algorithm provides a unique benchmark for comparison of OCT enhancement methods and a tailored augmentation method to overcome the limited number of OCTs in deep learning algorithms.

Reconstruction of visible light optical coherence tomography images retrieved from discontinuous spectral data using a conditional generative adversarial network

Achieving high resolution in optical coherence tomography typically requires the continuous extension of the spectral bandwidth of the light source. This work demonstrates an alternative approach: combining two discrete spectral windows located in the visible spectrum with a trained conditional generative adversarial network (cGAN) to reconstruct a high-resolution image equivalent to that generated using a continuous spectral band. The cGAN was trained using OCT image pairs acquired with the continuous and discontinuous visible range spectra to learn the relation between low- and high-resolution data. The reconstruction performance was tested using 6000 B-scans of a layered phantom, micro-beads and ex-vivo mouse ear tissue. The resultant cGAN-generated images demonstrate an image quality and axial resolution which approaches that of the high-resolution system.

Development and Validation of an Explainable Artificial Intelligence Framework for Macular Disease Diagnosis Based on OCT Images

PURPOSE

To develop and validate an artificial intelligence framework for identifying multiple retinal lesions at image-level and performing an explainable macular disease diagnosis at eye-level in optical coherence tomography (OCT) images.

METHODS

26,815 OCT images were collected from 865 eyes, and 9 retinal lesions and 3 macular diseases were labelled by ophthalmologists, including diabetic macular edema (DME) and dry/wet age-related macular degeneration (dry/wet AMD). We applied deep learning to classify retinal lesion at image-level and random forests to achieve an explainable disease diagnosis at eye-level. The performance of the integrated two-stage framework was evaluated and compared with human experts.

RESULTS

On testing dataset of 2,480 OCT images from 80 eyes, deep learning model achieved average Area Under Curve (AUC) of 0.978 (95% CI, 0.971-0.983) for lesion classification. And random forests performed accurate disease diagnosis with 0% error rate, which achieved the same accuracy as one of human experts and was better than the other 3 experts. It also revealed that the detection of specific lesions in the center of macular region had more contribution to macular disease diagnosis.

CONCLUSIONS

The integrated method achieved high accuracy and interpretability in retinal lesion classification and macular disease diagnosis in OCT images, and could have the potential to facilitate the clinical diagnosis.

AN AUTOMATED METHOD TO DETECT AGE-RELATED MACULAR DEGENERATION FROM OPTICAL COHERENCE TOMOGRAPHIC IMAGES

Age-related macular degeneration (AMD) is an eye disease that affects the elderly. AMD’s prevalence is increasing as society’s population ages; thus, early detection is critical to prevent vision loss in the elderly. Arrangement of a comprehensive examination of the eye for AMD detection is a challenging task. This paper suggests a new poly scale and dual path (PSDP) convolutional neural network (CNN) architecture for early-stage AMD diagnosis automatically. The proposed PSDP architecture has nine convolutional layers to classify the input image as AMD or normal. A PSDP architecture is used to enhance classification efficiency due to the high variation in size and shape of perforation present in OCT images. The poly scale approach employs filters of various sizes to extract features from local regions more effectively. Simultaneously, the dual path architecture incorporates features extracted from different CNN layers to boost features in the global regions. The sigmoid function is used to classify images into binary categories. The Mendeley data set is used to train the proposed network and tested on Mendeley, Duke, SD-OCT Noor, and OCTID data sets. The testing accuracy of the network in Mendeley, Duke, SD-OCT Noor, and OCT-ID is 99.73%,96.66%,94.89%,99.61%, respectively. The comparison with alternative approaches showed that the proposed algorithm is efficient in detecting AMD. Despite having been trained on the Mendeley data set, the proposed model exhibited good detection accuracy when tested on other data sets. This shows that the suggested model can distinguish AMD/Normal images from various data sets. As compared to other methods, the findings show that the proposed algorithm is efficient at detecting AMD. Rapid eye scanning for early detection of AMD could be possible with the proposed architecture. The proposed CNN can be applied in real-time due to its lower complexity and less learnable parameters.

A novel multiscale and multipath convolutional neural network based age-related macular degeneration detection using OCT images

BACKGROUND AND OBJECTIVE

One of the significant retinal diseases that affected older people is called Age-related Macular Degeneration (AMD). The first stage creates a blur effect on vision and later leads to central vision loss. Most people overlooked the primary stage blurring and converted it into an advanced stage. There is no proper treatment to cure the disease. So the early detection of AMD is essential to prevent its extension into the advanced stage. This paper proposes a novel deep Convolutional Neural Network (CNN) architecture to automate AMD diagnosis early from Optical Coherence Tomographic (OCT) images.

METHODS

The proposed architecture is a multiscale and multipath CNN with six convolutional layers. The multiscale convolution layer permits the network to produce many local structures with various filter dimensions. The multipath feature extraction permits CNN to merge more features regarding the sparse local and fine global structures. The performance of the proposed architecture is evaluated through ten-fold cross-validation methods using different classifiers like support vector machine, multi-layer perceptron, and random forest.

RESULTS

The proposed CNN with the random forest classifier gives the best classification accuracy results. The proposed method is tested on data set 1, data set 2, data set 3, data set 4, and achieved an accuracy of 0.9666, 0.9897, 0.9974, and 0.9978 respectively, with random forest classifier. Also, we tested the combination of first three data sets and achieved an accuracy of 0.9902.

CONCLUSIONS

An efficient algorithm for detecting AMD from OCT images is proposed based on a multiscale and multipath CNN architecture. Comparison with other approaches produced results that exhibit the efficiency of the proposed algorithm in the detection of AMD. The proposed architecture can be applied in rapid screening of the eye for the early detection of AMD. Due to less complexity and fewer learnable parameters.

DETECTION OF AGE-RELATED MACULAR DEGENERATION FROM OCT IMAGES USING DOUBLE SCALE CNN ARCHITECTURE

An ocular disease that affects the elderly is Age-related Macular Degeneration (AMD). Because of the aging population in society, AMD incidence is increasing; early diagnosis is vital to avoid vision loss in the elderly. It is a challenging process to organize a comprehensive eye screening system for detecting AMD. This paper proposes a novel Double Scale Convolutional Neural Network (DSCNN) architecture for an accurate AMD diagnosis. The architecture proposed is a DSCNN with six convolutional layers for classifying AMD or normal images. The double-scale convolution layer enables many local structures to be generated with two different filter sizes. In this proposed network, the sigmoid function is used as the classifier. The proposed CNN network is trained on the Mendeley data set and tested on four data sets, namely Mendeley, OCTID, Duke, SD-OCT Noor data set, and achieved an accuracy of 99.46%, 98.08%, 96.66%, and 94.89% respectively. The comparison with alternative methods provided results showing the efficacy of the proposed algorithm in detecting AMD. Although the proposed model is trained only on the Mendeley data set, it achieved good detection accuracy when evaluated with other data sets. This indicates the proposed model’s ability to classify AMD/Normal images from different data sets. Comparison with other approaches produced results that exhibit the efficiency of the proposed algorithm in detecting AMD. The proposed architecture can be applied in the rapid screening of the eye for the early detection of AMD. Due to less complexity and fewer learnable parameters, the proposed CNN can be implemented in real-time.

Assessing the Clinical Utility of Expanded Macular OCTs Using Machine Learning

Purpose

Optical coherence tomography (OCT) is widely used in the management of retinal pathologies, including age-related macular degeneration (AMD), diabetic macular edema (DME), and primary open-angle glaucoma (POAG). We used machine learning techniques to understand diagnostic performance gains from expanding macular OCT B-scans compared with foveal-only OCT B-scans for these conditions.

Methods

Electronic medical records were extracted to obtain 61 B-scans per eye from patients with AMD, diabetic retinopathy, or POAG. We constructed deep neural networks and random forest ensembles and generated area under the receiver operating characteristic (AUROC) and area under the precision recall (AUPR) curves.

Results

After extracting 630,000 OCT images, we achieved improved AUROC and AUPR curves when comparing the central image (one B-scan) to all images (61 B-scans). The AUROC and AUPR points of diminishing return for diagnostic accuracy for macular OCT coverage were found to be within 2.75 to 4.00 mm (14–19 B-scans), 4.25 to 4.50 mm (20–21 B-scans), and 4.50 to 6.25 mm (21–28 B-scans) for AMD, DME, and POAG, respectively. All models with >0.25 mm of coverage had statistically significantly improved AUROC/AUPR curves for all diseases (P < 0.05).

Conclusions

Systematically expanded macular coverage models demonstrated significant differences in total macular coverage required for improved diagnostic accuracy, with the largest macular area being relevant in POAG followed by DME and then AMD. These findings support our hypothesis that the extent of macular coverage by OCT imaging in the clinical setting, for any of the three major disorders, has a measurable impact on the functionality of artificial intelligence decision support.

Translational Relevance

We used machine learning techniques to improve OCT imaging standards for common retinal disease diagnoses.

Multimodal, multitask, multiattention (M3) deep learning detection of reticular pseudodrusen: Toward automated and accessible classification of age-related macular degeneration

OBJECTIVE

Reticular pseudodrusen (RPD), a key feature of age-related macular degeneration (AMD), are poorly detected by human experts on standard color fundus photography (CFP) and typically require advanced imaging modalities such as fundus autofluorescence (FAF). The objective was to develop and evaluate the performance of a novel multimodal, multitask, multiattention (M3) deep learning framework on RPD detection.

MATERIALS AND METHODS

A deep learning framework (M3) was developed to detect RPD presence accurately using CFP alone, FAF alone, or both, employing >8000 CFP-FAF image pairs obtained prospectively (Age-Related Eye Disease Study 2). The M3 framework includes multimodal (detection from single or multiple image modalities), multitask (training different tasks simultaneously to improve generalizability), and multiattention (improving ensembled feature representation) operation. Performance on RPD detection was compared with state-of-the-art deep learning models and 13 ophthalmologists; performance on detection of 2 other AMD features (geographic atrophy and pigmentary abnormalities) was also evaluated.

RESULTS

For RPD detection, M3 achieved an area under the receiver-operating characteristic curve (AUROC) of 0.832, 0.931, and 0.933 for CFP alone, FAF alone, and both, respectively. M3 performance on CFP was very substantially superior to human retinal specialists (median F1 score = 0.644 vs 0.350). External validation (the Rotterdam Study) demonstrated high accuracy on CFP alone (AUROC, 0.965). The M3 framework also accurately detected geographic atrophy and pigmentary abnormalities (AUROC, 0.909 and 0.912, respectively), demonstrating its generalizability.

CONCLUSIONS

This study demonstrates the successful development, robust evaluation, and external validation of a novel deep learning framework that enables accessible, accurate, and automated AMD diagnosis and prognosis.

Multi-Modal Retinal Image Classification With Modality-Specific Attention Network

Recently, automatic diagnostic approaches have been widely used to classify ocular diseases. Most of these approaches are based on a single imaging modality (e.g., fundus photography or optical coherence tomography (OCT)), which usually only reflect the oculopathy to a certain extent, and neglect the modality-specific information among different imaging modalities. This paper proposes a novel modality-specific attention network (MSAN) for multi-modal retinal image classification, which can effectively utilize the modality-specific diagnostic features from fundus and OCT images. The MSAN comprises two attention modules to extract the modality-specific features from fundus and OCT images, respectively. Specifically, for the fundus image, ophthalmologists need to observe local and global pathologies at multiple scales (e.g., from microaneurysms at the micrometer level, optic disc at millimeter level to blood vessels through the whole eye). Therefore, we propose a multi-scale attention module to extract both the local and global features from fundus images. Moreover, large background regions exist in the OCT image, which is meaningless for diagnosis. Thus, a region-guided attention module is proposed to encode the retinal layer-related features and ignore the background in OCT images. Finally, we fuse the modality-specific features to form a multi-modal feature and train the multi-modal retinal image classification network. The fusion of modality-specific features allows the model to combine the advantages of fundus and OCT modality for a more accurate diagnosis. Experimental results on a clinically acquired multi-modal retinal image (fundus and OCT) dataset demonstrate that our MSAN outperforms other well-known single-modal and multi-modal retinal image classification methods.

A Multimodal Imaging-Based Deep Learning Model for Detecting Treatment-Requiring Retinal Vascular Diseases: Model Development and Validation Study

BACKGROUND

Retinal vascular diseases, including diabetic macular edema (DME), neovascular age-related macular degeneration (nAMD), myopic choroidal neovascularization (mCNV), and branch and central retinal vein occlusion (BRVO/CRVO), are considered vision-threatening eye diseases. However, accurate diagnosis depends on multimodal imaging and the expertise of retinal ophthalmologists.

OBJECTIVE

The aim of this study was to develop a deep learning model to detect treatment-requiring retinal vascular diseases using multimodal imaging.

METHODS

This retrospective study enrolled participants with multimodal ophthalmic imaging data from 3 hospitals in Taiwan from 2013 to 2019. Eye-related images were used, including those obtained through retinal fundus photography, optical coherence tomography (OCT), and fluorescein angiography with or without indocyanine green angiography (FA/ICGA). A deep learning model was constructed for detecting DME, nAMD, mCNV, BRVO, and CRVO and identifying treatment-requiring diseases. Model performance was evaluated and is presented as the area under the curve (AUC) for each receiver operating characteristic curve.

RESULTS

A total of 2992 eyes of 2185 patients were studied, with 239, 1209, 1008, 211, 189, and 136 eyes in the control, DME, nAMD, mCNV, BRVO, and CRVO groups, respectively. Among them, 1898 eyes required treatment. The eyes were divided into training, validation, and testing groups in a 5:1:1 ratio. In total, 5117 retinal fundus photos, 9316 OCT images, and 20,922 FA/ICGA images were used. The AUCs for detecting mCNV, DME, nAMD, BRVO, and CRVO were 0.996, 0.995, 0.990, 0.959, and 0.988, respectively. The AUC for detecting treatment-requiring diseases was 0.969. From the heat maps, we observed that the model could identify retinal vascular diseases.

CONCLUSIONS

Our study developed a deep learning model to detect retinal diseases using multimodal ophthalmic imaging. Furthermore, the model demonstrated good performance in detecting treatment-requiring retinal diseases.

Automated Identification of Referable Retinal Pathology in Teleophthalmology Setting

Purpose

This study aims to meet a growing need for a fully automated, learning-based interpretation tool for retinal images obtained remotely (e.g. teleophthalmology) through different imaging modalities that may include imperfect (uninterpretable) images.

Methods

A retrospective study of 1148 optical coherence tomography (OCT) and color fundus photography (CFP) retinal images obtained using Topcon's Maestro care unit on 647 patients with diabetes. To identify retinal pathology, a Convolutional Neural Network (CNN) with dual-modal inputs (i.e. CFP and OCT images) was developed. We developed a novel alternate gradient descent algorithm to train the CNN, which allows for the use of uninterpretable CFP/OCT images (i.e. ungradable images that do not contain sufficient image biomarkers for the reviewer to conclude absence or presence of retinal pathology). Specifically, a 9:1 ratio to split the training and testing dataset was used for training and validating the CNN. Paired CFP/OCT inputs (obtained from a single eye of a patient) were grouped as retinal pathology negative (RPN; 924 images) in the absence of retinal pathology in both imaging modalities, or if one of the imaging modalities was uninterpretable and the other without retinal pathology. If any imaging modality exhibited referable retinal pathology, the corresponding CFP/OCT inputs were deemed retinal pathology positive (RPP; 224 images) if any imaging modality exhibited referable retinal pathology.

Results

Our approach achieved 88.60% (95% confidence interval [CI] = 82.76% to 94.43%) accuracy in identifying pathology, along with the false negative rate (FNR) of 12.28% (95% CI = 6.26% to 18.31%), recall (sensitivity) of 87.72% (95% CI = 81.69% to 93.74%), specificity of 89.47% (95% CI = 83.84% to 95.11%), and area under the curve of receiver operating characteristic (AUC-ROC) was 92.74% (95% CI = 87.71% to 97.76%).

Conclusions

Our model can be successfully deployed in clinical practice to facilitate automated remote retinal pathology identification.

Translational Relevance

A fully automated tool for early diagnosis of retinal pathology might allow for earlier treatment and improved visual outcomes.

Artificial intelligence for the detection of age-related macular degeneration in color fundus photographs: A systematic review and meta-analysis

BACKGROUND

Age-related macular degeneration (AMD) is one of the leading causes of vision loss in the elderly population. The application of artificial intelligence (AI) provides convenience for the diagnosis of AMD. This systematic review and meta-analysis aimed to quantify the performance of AI in detecting AMD in fundus photographs.

METHODS

We searched PubMed, Embase, Web of Science and the Cochrane Library before December 31st, 2020 for studies reporting the application of AI in detecting AMD in color fundus photographs. Then, we pooled the data for analysis. PROSPERO registration number: CRD42020197532.

FINDINGS

19 studies were finally selected for systematic review and 13 of them were included in the quantitative synthesis. All studies adopted human graders as reference standard. The pooled area under the receiver operating characteristic curve (AUROC) was 0.983 (95% confidence interval (CI):0.979-0.987). The pooled sensitivity, specificity, and diagnostic odds ratio (DOR) were 0.88 (95% CI:0.88-0.88), 0.90 (95% CI:0.90-0.91), and 275.27 (95% CI:158.43-478.27), respectively. Threshold analysis was performed and a potential threshold effect was detected among the studies (Spearman correlation coefficient: -0.600, P = 0.030), which was the main cause for the heterogeneity. For studies applying convolutional neural networks in the Age-Related Eye Disease Study database, the pooled AUROC, sensitivity, specificity, and DOR were 0.983 (95% CI:0.978-0.988), 0.88 (95% CI:0.88-0.88), 0.91 (95% CI:0.91-0.91), and 273.14 (95% CI:130.79-570.43), respectively.

INTERPRETATION

Our data indicated that AI was able to detect AMD in color fundus photographs. The application of AI-based automatic tools is beneficial for the diagnosis of AMD.

FUNDING

Capital Health Research and Development of Special (2020-1-2052).

Application of Deep Learning for Diagnosing, Classifying, and Treating Age-Related Macular Degeneration

Age-related macular degeneration (AMD) affects nearly 200 million people and is the third leading cause of irreversible vision loss worldwide. Deep learning, a branch of artificial intelligence that can learn image recognition based on pre-existing datasets, creates an opportunity for more accurate and efficient diagnosis, classification, and treatment of AMD on both individual and population levels. Current algorithms based on fundus photography and optical coherence tomography imaging have already achieved diagnostic accuracy levels comparable to human graders. This accuracy can be further increased when deep learning algorithms are simultaneously applied to multiple diagnostic imaging modalities. Combined with advances in telemedicine and imaging technology, deep learning can enable large populations of patients to be screened than would otherwise be possible and allow ophthalmologists to focus on seeing those patients who are in need of treatment, thus reducing the number of patients with significant visual impairment from AMD.

Feasibility study to improve deep learning in OCT diagnosis of rare retinal diseases with few-shot classification

Deep learning (DL) has been successfully applied to the diagnosis of ophthalmic diseases. However, rare diseases are commonly neglected due to insufficient data. Here, we demonstrate that few-shot learning (FSL) using a generative adversarial network (GAN) can improve the applicability of DL in the optical coherence tomography (OCT) diagnosis of rare diseases. Four major classes with a large number of datasets and five rare disease classes with a few-shot dataset are included in this study. Before training the classifier, we constructed GAN models to generate pathological OCT images of each rare disease from normal OCT images. The Inception-v3 architecture was trained using an augmented training dataset, and the final model was validated using an independent test dataset. The synthetic images helped in the extraction of the characteristic features of each rare disease. The proposed DL model demonstrated a significant improvement in the accuracy of the OCT diagnosis of rare retinal diseases and outperformed the traditional DL models, Siamese network, and prototypical network. By increasing the accuracy of diagnosing rare retinal diseases through FSL, clinicians can avoid neglecting rare diseases with DL assistance, thereby reducing diagnosis delay and patient burden.

Deep learning can generate traditional retinal fundus photographs using ultra-widefield images via generative adversarial networks

BACKGROUND AND OBJECTIVE

Retinal imaging has two major modalities, traditional fundus photography (TFP) and ultra-widefield fundus photography (UWFP). This study demonstrates the feasibility of a state-of-the-art deep learning-based domain transfer from UWFP to TFP.

METHODS

A cycle-consistent generative adversarial network (CycleGAN) was used to automatically translate the UWFP to the TFP domain. The model was based on an unpaired dataset including anonymized 451 UWFP and 745 TFP images. To apply CycleGAN to an independent dataset, we randomly divided the data into training (90%) and test (10%) datasets. After automated image registration and masking dark frames, the generator and discriminator networks were trained. Additional twelve publicly available paired TFP and UWFP images were used to calculate the intensity histograms and structural similarity (SSIM) indices.

RESULTS

We observed that all UWFP images were successfully translated into TFP-style images by CycleGAN, and the main structural information of the retina and optic nerve was retained. The model did not generate fake features in the output images. Average histograms demonstrated that the intensity distribution of the generated output images provided a good match to the ground truth images, with an average SSIM level of 0.802.

CONCLUSIONS

Our approach enables automated synthesis of TFP images directly from UWFP without a manual pre-conditioning process. The generated TFP images might be useful for clinicians in investigating posterior pole and for researchers in integrating TFP and UWFP databases. This is also likely to save scan time and will be more cost-effective for patients by avoiding additional examinations for an accurate diagnosis.

Artificial Intelligence for Automated Overlay of Fundus Camera and Scanning Laser Ophthalmoscope Images

Purpose

The purpose of this study was to evaluate the ability to align two types of retinal images taken on different platforms; color fundus (CF) photographs and infrared scanning laser ophthalmoscope (IR SLO) images using mathematical warping and artificial intelligence (AI).

Methods

We collected 109 matched pairs of CF and IR SLO images. An AI algorithm utilizing two separate networks was developed. A style transfer network (STN) was used to segment vessel structures. A registration network was used to align the segmented images to each. Neither network used a ground truth dataset. A conventional image warping algorithm was used as a control. Software displayed image pairs as a 5 × 5 checkerboard grid composed of alternating subimages. This technique permitted vessel alignment determination by human observers and 5 masked graders evaluated alignment by the AI and conventional warping in 25 fields for each image.

Results

Our new AI method was superior to conventional warping at generating vessel alignment as judged by masked human graders (P < 0.0001). The average number of good/excellent matches increased from 90.5% to 94.4% with AI method.

Conclusions

AI permitted a more accurate overlay of CF and IR SLO images than conventional mathematical warping. This is a first step toward developing an AI that could allow overlay of all types of fundus images by utilizing vascular landmarks.

Translational Relevance

The ability to align and overlay imaging data from multiple instruments and manufacturers will permit better analysis of this complex data helping understand disease and predict treatment.

Toward automated severe pharyngitis detection with smartphone camera using deep learning networks

PURPOSE

Severe pharyngitis is frequently associated with inflammations caused by streptococcal pharyngitis, which can cause immune-mediated and post-infectious complications. The recent global pandemic of coronavirus disease (COVID-19) encourages the use of telemedicine for patients with respiratory symptoms. This study, therefore, purposes automated detection of severe pharyngitis using a deep learning framework with self-taken throat images.

METHODS

A dataset composed of two classes of 131 throat images with pharyngitis and 208 normal throat images was collected. Before the training classifier, we constructed a cycle consistency generative adversarial network (CycleGAN) to augment the training dataset. The ResNet50, Inception-v3, and MobileNet-v2 architectures were trained with transfer learning and validated using a randomly selected test dataset. The performance of the models was evaluated based on the accuracy and area under the receiver operating characteristic curve (ROC-AUC).

RESULTS

The CycleGAN-based synthetic images reflected the pragmatic characteristic features of pharyngitis. Using the synthetic throat images, the deep learning model demonstrated a significant improvement in the accuracy of the pharyngitis diagnosis. ResNet50 with GAN-based augmentation showed the best ROC-AUC of 0.988 for pharyngitis detection in the test dataset. In the 4-fold cross-validation using the ResNet50, the highest detection accuracy and ROC-AUC achieved were 95.3% and 0.992, respectively.

CONCLUSION

The deep learning model for smartphone-based pharyngitis screening allows fast identification of severe pharyngitis with a potential of the timely diagnosis of pharyngitis. In the recent pandemic of COVID-19, this framework will help patients with upper respiratory symptoms to improve convenience in diagnosis and reduce transmission.

[Multimodal imaging and evaluation in the age of artificial intelligence]

Multimodal imaging is able to image the retina in unprecedented detail, and the joint analysis (integration) of these data not only enables the securing of diagnoses, but also a more precise definition; however, humans encounter temporal and cognitive limitations in the analysis of this amount of information, so that the potential of a joint examination of the findings is largely unused to date. Automatic image processing and methods, which are summarized under the collective term of artificial intelligence (AI), are able to overcome the bottleneck in the evaluation and to exploit the full potential of the available data. A basic understanding of AI methods and the ability to implement them will become increasingly more important for ophthalmologists in the future. In this article we give an insight into the functionality of AI methods and the current state of research in the field of automatic image analysis.

The American Society of Retina Specialists Artificial Intelligence Task Force Report

Artificial intelligence (AI) is a growing area that relies on the heavy use of diagnostic imaging within the field of retina to offer exciting advancements in diagnostic capability to better understand and manage retinal conditions such as diabetic retinopathy, diabetic macular edema, age-related macular degeneration, and retinopathy of prematurity. However, there are discrepancies between the findings of these AI programs and their referral recommendations compared with evidence-based referral patterns, such as Preferred Practice Patterns by the American Academy of Ophthalmology. The overall focus of this task force report is to first describe the work in AI being completed in the management of retinal conditions. This report also discusses the guidelines of the Preferred Practice Pattern and how they can be used in the emerging field of AI.

Automated diagnoses of age-related macular degeneration and polypoidal choroidal vasculopathy using bi-modal deep convolutional neural networks

AIMS

To investigate the efficacy of a bi-modality deep convolutional neural network (DCNN) framework to categorise age-related macular degeneration (AMD) and polypoidal choroidal vasculopathy (PCV) from colour fundus images and optical coherence tomography (OCT) images.

METHODS

A retrospective cross-sectional study was proposed of patients with AMD or PCV who came to Peking Union Medical College Hospital. Diagnoses of all patients were confirmed by two retinal experts based on diagnostic gold standard for AMD and PCV. Patients with concurrent retinal vascular diseases were excluded. Colour fundus images and spectral domain OCT images were taken from dilated eyes of patients and healthy controls, and anonymised. All images were pre-labelled into normal, dry or wet AMD or PCV. ResNet-50 models were used as the backbone and alternate machine learning models including random forest classifiers were constructed for further comparison. For human-machine comparison, the same testing data set was diagnosed by three retinal experts independently. All images from the same participant were presented only within a single partition subset.

RESULTS

On a test set of 143 fundus and OCT image pairs from 80 eyes (20 eyes per-group), the bi-modal DCNN demonstrated the best performance, with accuracy 87.4%, sensitivity 88.8% and specificity 95.6%, and a perfect agreement with diagnostic gold standard (Cohen's κ 0.828), exceeds slightly over the best expert (Human1, Cohen's κ 0.810). For recognising PCV, the model outperformed the best expert as well.

CONCLUSION

A bi-modal DCNN for automated classification of AMD and PCV is accurate and promising in the realm of public health.

CycleGAN-based deep learning technique for artifact reduction in fundus photography

PURPOSE

A low quality of fundus photograph with artifacts may lead to false diagnosis. Recently, a cycle-consistent generative adversarial network (CycleGAN) has been introduced as a tool to generate images without matching paired images. Therefore, herein, we present a deep learning technique that removes the artifacts automatically in a fundus photograph using a CycleGAN model.

METHODS

This study included a total of 2206 anonymized retinal images including 1146 with artifacts and 1060 without artifacts. In this experiment, we applied the CycleGAN model to color fundus photographs with a pixel resolution of 256 × 256 × 3. To apply the CycleGAN to an independent dataset, we randomly divided the data into training (90%) and test (10%) datasets. Additionally, we adopted the automated quality evaluation (AQE) to assess the retinal image quality.

RESULTS

From the results, we observed that the artifacts such as overall haze, edge haze, lashes, arcs, and uneven illumination were successfully reduced by the CycleGAN in the generated images, and the main information of the retina was essentially retained. Further, we observed that most of the generated images exhibited improved AQE grade values when compared with the original images with artifacts.

CONCLUSION

Thus, we could conclude that the CycleGAN technique can effectively reduce the artifacts and improve the quality of fundus photographs, and it may be beneficial for clinicians in analyzing the low-quality fundus photographs. Future studies should improve the quality and resolution of the generated image to provide a more detailed fundus photography.

Multi-view deep learning for rigid gas permeable lens base curve fitting based on Pentacam images

Many studies in the rigid gas permeable (RGP) lens fitting field have focused on providing the best fit for patients with irregular astigmatism, a challenging issue. Despite the ease and accuracy of fitting in the current fitting methods, no studies have provided a high-pace solution with the final best fit to assist experts. This work presents a deep learning solution for identifying features in Pentacam four refractive maps and RGP base curve identification. An authentic dataset of 247 samples of Pentacam four refractive maps was gathered, providing a multi-view image of the corneal structure. Scratch-based convolutional neural network (CNN) architectures and well-known CNN architectures such as AlexNet, GoogLeNet, and ResNet have been used to extract features and transfer learning. Features are aggregated through a fusion technique. Based on a comparison of means square error (MSE) of normalized labels, the multi-view scratch-based CNN provided R-squared of 0.849, 0.846, 0.835, and 0.834 followed by GoogLeNet, comparable with current methods. Transfer learning outperforms various scratch-based CNN models, through which proper specifications some scratch-based models were able to increase coefficient of determinations. CNNs on multi-view Pentacam images have enabled fast detection of the RGP lens base curve, higher patient satisfaction, and reduced chair time. Graphical abstract The Pentacam four refractive maps is learned by the proposed scratch-based and transfer learning-based CNN methodology. The deep network-based solutions enable identification of rigid gas permeable lens for patients with irregular astigmatism.

Accuracy of a deep convolutional neural network in the detection of myopic macular diseases using swept-source optical coherence tomography

This study examined and compared outcomes of deep learning (DL) in identifying swept-source optical coherence tomography (OCT) images without myopic macular lesions [i.e., no high myopia (nHM) vs. high myopia (HM)], and OCT images with myopic macular lesions [e.g., myopic choroidal neovascularization (mCNV) and retinoschisis (RS)]. A total of 910 SS-OCT images were included in the study as follows and analyzed by k-fold cross-validation (k = 5) using DL's renowned model, Visual Geometry Group-16: nHM, 146 images; HM, 531 images; mCNV, 122 images; and RS, 111 images (n = 910). The binary classification of OCT images with or without myopic macular lesions; the binary classification of HM images and images with myopic macular lesions (i.e., mCNV and RS images); and the ternary classification of HM, mCNV, and RS images were examined. Additionally, sensitivity, specificity, and the area under the curve (AUC) for the binary classifications as well as the correct answer rate for ternary classification were examined. The classification results of OCT images with or without myopic macular lesions were as follows: AUC, 0.970; sensitivity, 90.6%; specificity, 94.2%. The classification results of HM images and images with myopic macular lesions were as follows: AUC, 1.000; sensitivity, 100.0%; specificity, 100.0%. The correct answer rate in the ternary classification of HM images, mCNV images, and RS images were as follows: HM images, 96.5%; mCNV images, 77.9%; and RS, 67.6% with mean, 88.9%.Using noninvasive, easy-to-obtain swept-source OCT images, the DL model was able to classify OCT images without myopic macular lesions and OCT images with myopic macular lesions such as mCNV and RS with high accuracy. The study results suggest the possibility of conducting highly accurate screening of ocular diseases using artificial intelligence, which may improve the prevention of blindness and reduce workloads for ophthalmologists.

Deep learning-based smart speaker to confirm surgical sites for cataract surgeries: A pilot study

Wrong-site surgeries can occur due to the absence of an appropriate surgical time-out. However, during a time-out, surgical participants are unable to review the patient's charts due to their aseptic hands. To improve the conditions in surgical time-outs, we introduce a deep learning-based smart speaker to confirm the surgical information prior to cataract surgeries. This pilot study utilized the publicly available audio vocabulary dataset and recorded audio data published by the authors. The audio clips of the target words, such as left, right, cataract, phacoemulsification, and intraocular lens, were selected to determine and confirm surgical information in the time-out speech. A deep convolutional neural network model was trained and implemented in the smart speaker that was developed using a mini development board and commercial speakerphone. To validate our model in the consecutive speeches during time-outs, we generated 200 time-out speeches for cataract surgeries by randomly selecting the surgical statuses of the surgical participants. After the training process, the deep learning model achieved an accuracy of 96.3% for the validation dataset of short-word audio clips. Our deep learning-based smart speaker achieved an accuracy of 93.5% for the 200 time-out speeches. The surgical and procedural accuracy was 100%. Additionally, on validating the deep learning model by using web-generated time-out speeches and video clips for general surgery, the model exhibited a robust and good performance. In this pilot study, the proposed deep learning-based smart speaker was able to successfully confirm the surgical information during the time-out speech. Future studies should focus on collecting real-world time-out data and automatically connecting the device to electronic health records. Adopting smart speaker-assisted time-out phases will improve the patients' safety during cataract surgeries, particularly in relation to wrong-site surgeries.

Health Economic and Safety Considerations for Artificial Intelligence Applications in Diabetic Retinopathy Screening

Systematic screening for diabetic retinopathy (DR) has been widely recommended for early detection in patients with diabetes to address preventable vision loss. However, substantial manpower and financial resources are required to deploy opportunistic screening and transition to systematic DR screening programs. The advent of artificial intelligence (AI) technologies may improve access and reduce the financial burden for DR screening while maintaining comparable or enhanced clinical effectiveness. To deploy an AI-based DR screening program in a real-world setting, it is imperative that health economic assessment (HEA) and patient safety analyses are conducted to guide appropriate allocation of resources and design safe, reliable systems. Few studies published to date include these considerations when integrating AI-based solutions into DR screening programs. In this article, we provide an overview of the current state-of-the-art of AI technology (focusing on deep learning systems), followed by an appraisal of existing literature on the applications of AI in ophthalmology. We also discuss practical considerations that drive the development of a successful DR screening program, such as the implications of false-positive or false-negative results and image gradeability. Finally, we examine different plausible methods for HEA and safety analyses that can be used to assess concerns regarding AI-based screening.

Prognostic analysis of histopathological images using pre-trained convolutional neural networks: application to hepatocellular carcinoma

Histopathological images contain rich phenotypic descriptions of the molecular processes underlying disease progression. Convolutional neural networks, state-of-the-art image analysis techniques in computer vision, automatically learn representative features from such images which can be useful for disease diagnosis, prognosis, and subtyping. Hepatocellular carcinoma (HCC) is the sixth most common type of primary liver malignancy. Despite the high mortality rate of HCC, little previous work has made use of CNN models to explore the use of histopathological images for prognosis and clinical survival prediction of HCC. We applied three pre-trained CNN models-VGG 16, Inception V3 and ResNet 50-to extract features from HCC histopathological images. Sample visualization and classification analyses based on these features showed a very clear separation between cancer and normal samples. In a univariate Cox regression analysis, 21.4% and 16% of image features on average were significantly associated with overall survival (OS) and disease-free survival (DFS), respectively. We also observed significant correlations between these features and integrated biological pathways derived from gene expression and copy number variation. Using an elastic net regularized Cox Proportional Hazards model of OS constructed from Inception image features, we obtained a concordance index (C-index) of 0.789 and a significant log-rank test (p = 7.6E-18). We also performed unsupervised classification to identify HCC subgroups from image features. The optimal two subgroups discovered using Inception model image features showed significant differences in both overall (C-index = 0.628 and p = 7.39E-07) and DFS (C-index = 0.558 and p = 0.012). Our work demonstrates the utility of extracting image features using pre-trained models by using them to build accurate prognostic models of HCC as well as highlight significant correlations between these features, clinical survival, and relevant biological pathways. Image features extracted from HCC histopathological images using the pre-trained CNN models VGG 16, Inception V3 and ResNet 50 can accurately distinguish normal and cancer samples. Furthermore, these image features are significantly correlated with survival and relevant biological pathways.

Explainable Machine Learning Approach as a Tool to Understand Factors Used to Select the Refractive Surgery Technique on the Expert Level

Purpose

Recently, laser refractive surgery options, including laser epithelial keratomileusis, laser in situ keratomileusis, and small incision lenticule extraction, successfully improved patients' quality of life. Evidence-based recommendation for an optimal surgery technique is valuable in increasing patient satisfaction. We developed an interpretable multiclass machine learning model that selects the laser surgery option on the expert level.

Methods

A multiclass XGBoost model was constructed to classify patients into four categories including laser epithelial keratomileusis, laser in situ keratomileusis, small incision lenticule extraction, and contraindication groups. The analysis included 18,480 subjects who intended to undergo refractive surgery at the B&VIIT Eye center. Training (n = 10,561) and internal validation (n = 2640) were performed using subjects who visited between 2016 and 2017. The model was trained based on clinical decisions of highly experienced experts and ophthalmic measurements. External validation (n = 5279) was conducted using subjects who visited in 2018. The SHapley Additive ex-Planations technique was adopted to explain the output of the XGBoost model.

Results

The multiclass XGBoost model exhibited an accuracy of 81.0% and 78.9% when tested on the internal and external validation datasets, respectively. The SHapley Additive ex-Planations explanations for the results were consistent with prior knowledge from ophthalmologists. The explanation from one-versus-one and one-versus-rest XGBoost classifiers was effective for easily understanding users in the multicategorical classification problem.

Conclusions

This study suggests an expert-level multiclass machine learning model for selecting the refractive surgery for patients. It also provided a clinical understanding in a multiclass problem based on an explainable artificial intelligence technique.

Translational Relevance

Explainable machine learning exhibits a promising future for increasing the practical use of artificial intelligence in ophthalmic clinics.

A generative adversarial network approach to predicting postoperative appearance after orbital decompression surgery for thyroid eye disease

PURPOSE

Orbital decompression for thyroid-associated ophthalmopathy (TAO) is an ophthalmic plastic surgery technique to prevent optic neuropathy and reduce exophthalmos. Because the postoperative appearance can significantly change, sometimes it is difficult to make decisions regarding decompression surgery. Herein, we present a deep learning technique to synthesize the realistic postoperative appearance for orbital decompression surgery.

METHODS

This data-driven approach is based on a conditional generative adversarial network (GAN) to transform preoperative facial input images into predicted postoperative images. The conditional GAN model was trained on 109 pairs of matched pre- and postoperative facial images through data augmentation.

RESULTS

When the conditional variable was changed, the synthesized facial image was transferred from a preoperative image to a postoperative image. The predicted postoperative images were similar to the ground truth postoperative images. We also found that GAN-based synthesized images can improve the deep learning classification performance between the pre- and postoperative status using a small training dataset. However, a relatively low quality of synthesized images was noted after a readout by clinicians.

CONCLUSIONS

Using this framework, we synthesized TAO facial images that can be queried using conditioning on the orbital decompression status. The synthesized postoperative images may be helpful for patients in determining the impact of decompression surgery. However, the quality of the generated image should be further improved. The proposed deep learning technique based on a GAN can rapidly synthesize such realistic images of the postoperative appearance, suggesting that a GAN can function as a decision support tool for plastic and cosmetic surgery techniques.

Multimodal Retinal Image Analysis via Deep Learning for the Diagnosis of Intermediate Dry Age-Related Macular Degeneration: A Feasibility Study

Results

The CNN trained using OCT alone showed a diagnostic accuracy of 94%, whilst the OCT-A trained CNN resulted in an accuracy of 91%. When multiple modalities were combined, the CNN accuracy increased to 96% in the AMD cohort.

Conclusions

Here we demonstrate that superior diagnostic accuracy can be achieved when deep learning is combined with multimodal image analysis.