Accurate tissue interface segmentation via adversarial pre-segmentation of anterior segment OCT images

Generative artificial intelligence in ophthalmology

Generative AI has revolutionized medicine over the past several years. A generative adversarial network (GAN) is a deep learning framework that has become a powerful technique in medicine, particularly in ophthalmology and image analysis. In this paper we review the current ophthalmic literature involving GANs, and highlight key contributions in the field. We briefly touch on ChatGPT, another application of generative AI, and its potential in ophthalmology. We also explore the potential uses for GANs in ocular imaging, with a specific emphasis on 3 primary domains: image enhancement, disease identification, and generating of synthetic data. PubMed, Ovid MEDLINE, Google Scholar were searched from inception to October 30, 2022 to identify applications of GAN in ophthalmology. A total of 40 papers were included in this review. We cover various applications of GANs in ophthalmic-related imaging including optical coherence tomography, orbital magnetic resonance imaging, fundus photography, and ultrasound; however, we also highlight several challenges, that resulted in the generation of inaccurate and atypical results during certain iterations. Finally, we examine future directions and considerations for generative AI in ophthalmology.

Inpainting Saturation Artifact in Anterior Segment Optical Coherence Tomography

The cornea is an important refractive structure in the human eye. The corneal segmentation technique provides valuable information for clinical diagnoses, such as corneal thickness. Non-contact anterior segment optical coherence tomography (AS-OCT) is a prevalent ophthalmic imaging technique that can visualize the anterior and posterior surfaces of the cornea. Nonetheless, during the imaging process, saturation artifacts are commonly generated due to the tangent of the corneal surface at that point, which is normal to the incident light source. This stripe-shaped saturation artifact covers the corneal surface, causing blurring of the corneal edge, reducing the accuracy of corneal segmentation. To settle this matter, an inpainting method that introduces structural similarity and frequency loss is proposed to remove the saturation artifact in AS-OCT images. Specifically, the structural similarity loss reconstructs the corneal structure and restores corneal textural details. The frequency loss combines the spatial domain with the frequency domain to ensure the overall consistency of the image in both domains. Furthermore, the performance of the proposed method in corneal segmentation tasks is evaluated, and the results indicate a significant benefit for subsequent clinical analysis.

Patch-based CNN for corneal segmentation of AS-OCT images: Effect of the number of classes and image quality upon performance

Anterior segment optical coherence tomography (AS-OCT) is a fundamental ophthalmic imaging technique. AS-OCT images can be examined by experts and segmented to provide quantitative metrics that inform clinical decision making. Manual segmentation of these images is time-consuming and subjective, encouraging software developers in the field to automate segmentation procedures. Traditional programing segmentation approaches are being replaced by deep learning methods, which have shown state-of-the-art performance in AS-OCT image analysis. In this study, a method based on patch-based convolutional neural networks (CNN) was used to segment the three main boundaries of the cornea: the epithelium, Bowman's layer, and the endothelium. To assess the effect of the number of classes on performance, the model was designed as a patch-based boundary classifier using 4 and 8 classes. The effect of image quality was also assessed using different data distributions during the training process. While the Dice coefficient and probability revealed greater precision for the 8 class models, the boundary error metric indicated comparable performance. Additionally, for 8 class models, the image quality test had only a small negative effect on performance, which may be an indication of the robustness of the model and could also suggest that the data augmentation methods did not show significant improvement. These findings contribute to the development of automatic segmentation techniques for AS-OCT images, since patch-based methods have been largely unexplored in favor of other deep learning techniques. The overall performance of the proposed method is comparable to other well-established segmentation methods.

Deep Learning Model for the Detection of Corneal Edema Before Descemet Membrane Endothelial Keratoplasty on Optical Coherence Tomography Images

Purpose

Descemet membrane endothelial keratoplasty (DMEK) is the preferred method for treating corneal endothelial dysfunction, such as Fuchs endothelial corneal dystrophy (FECD). The surgical indication is based on the patients' symptoms and the presence of corneal edema. We developed an automated tool based on deep learning to detect edema in corneal optical coherence tomography images. This study aimed to evaluate this approach in edema detection before Descemet membrane endothelial keratoplasty surgery, for patients with or without FECD.

Methods

We used our previously described model allowing to classify each pixel in the corneal optical coherence tomography images as "normal" or "edema." We included 1992 images of normal and preoperative edematous corneas. We calculated the edema fraction (EF), defined as the ratio between the number of pixels labeled as "edema," and those representing the cornea for each patient. Differential central corneal thickness (DCCT), defined as the difference in central corneal thickness before and 6 months after surgery, was used to quantify preoperative edema. AUC of EF for the edema detection was calculated for Several DCCT thresholds and a value of 20 µm was selected to define significant edema as it provided the highest area under the curve value.

Results

The area under the curve of the receiver operating characteristic curve for EF for the detection of 20 µm of DCCT was 0.97 for all patients, 0.96 for Fuchs and normal only and 0.99 for non-FECD and normal patients. The optimal EF threshold was 0.143 for all patients and patients with FECD.

Conclusions

Our model is capable of objectively detecting minimal corneal edema before Descemet membrane endothelial keratoplasty surgery.

Translational Relevance

Deep learning can help to interpret optical coherence tomography scans and aid the surgeon in decision-making.

A review of generative adversarial network applications in optical coherence tomography image analysis

Optical coherence tomography (OCT) has revolutionized ophthalmic clinical practice and research, as a result of the high-resolution images that the method is able to capture in a fast, non-invasive manner. Although clinicians can interpret OCT images qualitatively, the ability to quantitatively and automatically analyse these images represents a key goal for eye care by providing clinicians with immediate and relevant metrics to inform best clinical practice. The range of applications and methods to analyse OCT images is rich and rapidly expanding. With the advent of deep learning methods, the field has experienced significant progress with state-of-the-art-performance for several OCT image analysis tasks. Generative adversarial networks (GANs) represent a subfield of deep learning that allows for a range of novel applications not possible in most other deep learning methods, with the potential to provide more accurate and robust analyses. In this review, the progress in this field and clinical impact are reviewed and the potential future development of applications of GANs to OCT image processing are discussed.

Automatic and quantitative measurement of alveolar bone level in OCT images using deep learning

We propose a method to automatically segment the periodontal structures of the tooth enamel and the alveolar bone using convolutional neural network (CNN) and to measure quantitatively and automatically the alveolar bone level (ABL) by detecting the cemento-enamel junction and the alveolar bone crest in optical coherence tomography (OCT) images. The tooth enamel and the alveolar bone regions were automatically segmented using U-Net, Dense-UNet, and U²-Net, and the ABL was quantitatively measured as the distance between the cemento-enamel junction and the alveolar bone crest using image processing. The mean distance difference (MDD) measured by our suggested method ranged from 0.19 to 0.22 mm for the alveolar bone crest (ABC) and from 0.18 to 0.32 mm for the cemento-enamel junction (CEJ). All CNN models showed the mean absolute error (MAE) of less than 0.25 mm in the x and y coordinates and greater than 90% successful detection rate (SDR) at 0.5 mm for both the ABC and the CEJ. The CNN models showed high segmentation accuracies in the tooth enamel and the alveolar bone regions, and the ABL measurements at the incisors by detected results from CNN predictions demonstrated high correlation and reliability with the ground truth in OCT images.

Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey

BACKGROUND

Recent advances in deep learning techniques have led to improved diagnostic abilities in ophthalmology. A generative adversarial network (GAN), which consists of two competing types of deep neural networks, including a generator and a discriminator, has demonstrated remarkable performance in image synthesis and image-to-image translation. The adoption of GAN for medical imaging is increasing for image generation and translation, but it is not familiar to researchers in the field of ophthalmology. In this work, we present a literature review on the application of GAN in ophthalmology image domains to discuss important contributions and to identify potential future research directions.

METHODS

We performed a survey on studies using GAN published before June 2021 only, and we introduced various applications of GAN in ophthalmology image domains. The search identified 48 peer-reviewed papers in the final review. The type of GAN used in the analysis, task, imaging domain, and the outcome were collected to verify the usefulness of the GAN.

RESULTS

In ophthalmology image domains, GAN can perform segmentation, data augmentation, denoising, domain transfer, super-resolution, post-intervention prediction, and feature extraction. GAN techniques have established an extension of datasets and modalities in ophthalmology. GAN has several limitations, such as mode collapse, spatial deformities, unintended changes, and the generation of high-frequency noises and artifacts of checkerboard patterns.

CONCLUSIONS

The use of GAN has benefited the various tasks in ophthalmology image domains. Based on our observations, the adoption of GAN in ophthalmology is still in a very early stage of clinical validation compared with deep learning classification techniques because several problems need to be overcome for practical use. However, the proper selection of the GAN technique and statistical modeling of ocular imaging will greatly improve the performance of each image analysis. Finally, this survey would enable researchers to access the appropriate GAN technique to maximize the potential of ophthalmology datasets for deep learning research.

Dissecting the Profile of Corneal Thickness With Keratoconus Progression Based on Anterior Segment Optical Coherence Tomography

Purpose

To characterize the corneal and epithelial thickness at different stages of keratoconus (KC), using a deep learning based corneal segmentation algorithm for anterior segment optical coherence tomography (AS-OCT).

Methods

An AS-OCT dataset was constructed in this study with 1,430 images from 715 eyes, which included 118 normal eyes, 134 mild KC, 239 moderate KC, 153 severe KC, and 71 scarring KC. A deep learning based corneal segmentation algorithm was applied to isolate the epithelial and corneal tissues from the background. Based on the segmentation results, the thickness of epithelial and corneal tissues was automatically measured in the center 6 mm area. One-way ANOVA and linear regression were performed in 20 equally divided zones to explore the trend of the thickness changes at different locations with the KC progression. The 95% confidence intervals (CI) of epithelial thickness and corneal thickness in a specific zone were calculated to reveal the difference of thickness distribution among different groups.

Results

Our data showed that the deep learning based corneal segmentation algorithm can achieve accurate tissue segmentation and the error range of measured thickness was less than 4 μm between our method and the results from clinical experts, which is approximately one image pixel. Statistical analyses revealed significant corneal thickness differences in all the divided zones (P < 0.05). The entire corneal thickness grew gradually thinner with the progression of the KC, and their trends were more pronounced around the pupil center with a slight shift toward the temporal and inferior side. Especially the epithelial thicknesses were thinner gradually from a normal eye to severe KC. Due to the formation of the corneal scarring, epithelial thickness had irregular fluctuations in the scarring KC.

Conclusion

Our study demonstrates that our deep learning method based on AS-OCT images could accurately delineate the corneal tissues and further successfully characterize the epithelial and corneal thickness changes at different stages of the KC progression.

Generative adversarial networks in medical image segmentation: A review

PURPOSE

Since Generative Adversarial Network (GAN) was introduced into the field of deep learning in 2014, it has received extensive attention from academia and industry, and a lot of high-quality papers have been published. GAN effectively improves the accuracy of medical image segmentation because of its good generating ability and capability to capture data distribution. This paper introduces the origin, working principle, and extended variant of GAN, and it reviews the latest development of GAN-based medical image segmentation methods.

METHOD

To find the papers, we searched on Google Scholar and PubMed with the keywords like "segmentation", "medical image", and "GAN (or generative adversarial network)". Also, additional searches were performed on Semantic Scholar, Springer, arXiv, and the top conferences in computer science with the above keywords related to GAN.

RESULTS

We reviewed more than 120 GAN-based architectures for medical image segmentation that were published before September 2021. We categorized and summarized these papers according to the segmentation regions, imaging modality, and classification methods. Besides, we discussed the advantages, challenges, and future research directions of GAN in medical image segmentation.

CONCLUSIONS

We discussed in detail the recent papers on medical image segmentation using GAN. The application of GAN and its extended variants has effectively improved the accuracy of medical image segmentation. Obtaining the recognition of clinicians and patients and overcoming the instability, low repeatability, and uninterpretability of GAN will be an important research direction in the future.

Updates in deep learning research in ophthalmology

Ophthalmology has been one of the early adopters of artificial intelligence (AI) within the medical field. Deep learning (DL), in particular, has garnered significant attention due to the availability of large amounts of data and digitized ocular images. Currently, AI in Ophthalmology is mainly focused on improving disease classification and supporting decision-making when treating ophthalmic diseases such as diabetic retinopathy, age-related macular degeneration (AMD), glaucoma and retinopathy of prematurity (ROP). However, most of the DL systems (DLSs) developed thus far remain in the research stage and only a handful are able to achieve clinical translation. This phenomenon is due to a combination of factors including concerns over security and privacy, poor generalizability, trust and explainability issues, unfavorable end-user perceptions and uncertain economic value. Overcoming this challenge would require a combination approach. Firstly, emerging techniques such as federated learning (FL), generative adversarial networks (GANs), autonomous AI and blockchain will be playing an increasingly critical role to enhance privacy, collaboration and DLS performance. Next, compliance to reporting and regulatory guidelines, such as CONSORT-AI and STARD-AI, will be required to in order to improve transparency, minimize abuse and ensure reproducibility. Thirdly, frameworks will be required to obtain patient consent, perform ethical assessment and evaluate end-user perception. Lastly, proper health economic assessment (HEA) must be performed to provide financial visibility during the early phases of DLS development. This is necessary to manage resources prudently and guide the development of DLS.

Corneal Edema Visualization With Optical Coherence Tomography Using Deep Learning: Proof of Concept

PURPOSE

Optical coherence tomography (OCT) is essential for the diagnosis and follow-up of corneal edema, but assessment can be challenging in minimal or localized edema. The objective was to develop and validate a novel automated tool to detect and visualize corneal edema with OCT.

METHODS

We trained a convolutional neural network to classify each pixel in the corneal OCT images as "normal" or "edema" and to generate colored heat maps of the result. The development set included 199 OCT images of normal and edematous corneas. We validated the model's performance on 607 images of normal and edematous corneas of various conditions. The main outcome measure was the edema fraction (EF), defined as the ratio between the number of pixels labeled as edema and those representing the cornea for each scan. Overall accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve were determined to evaluate the model's performance.

RESULTS

Mean EF was 0.0087 ± 0.01 in the normal scans and 0.805 ± 0.26 in the edema scans (P < 0.0001). Area under the receiver operating characteristic curve for EF in the diagnosis of corneal edema in individual scans was 0.994. The optimal threshold for distinguishing normal from edematous corneas was 6.8%, with an accuracy of 98.7%, sensitivity of 96.4%, and specificity of 100%.

CONCLUSIONS

The model accurately detected corneal edema and distinguished between normal and edematous cornea OCT scans while providing colored heat maps of edema presence.

Inpainting for Saturation Artifacts in Optical Coherence Tomography Using Dictionary-Based Sparse Representation

Saturation artifacts in optical coherence tomography (OCT) occur when received signal exceeds the dynamic range of spectrometer. Saturation artifact shows a streaking pattern and could impact the quality of OCT images, leading to inaccurate medical diagnosis. In this paper, we automatically localize saturation artifacts and propose an artifact correction method via inpainting. We adopt a dictionary-based sparse representation scheme for inpainting. Experimental results demonstrate that, in both case of synthetic artifacts and real artifacts, our method outperforms interpolation method and Euler's elastica method in both qualitative and quantitative results. The generic dictionary offers similar image quality when applied to tissue samples which are excluded from dictionary training. This method may have the potential to be widely used in a variety of OCT images for the localization and inpainting of the saturation artifacts.

Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning

Hematoxylin and Eosin (H&E) staining is the 'gold-standard' method in histopathology. However, standard H&E staining of high-quality tissue sections requires long sample preparation times including sample embedding, which restricts its application for 'real-time' disease diagnosis. Due to this reason, a label-free alternative technique like non-linear multimodal (NLM) imaging, which is the combination of three non-linear optical modalities including coherent anti-Stokes Raman scattering, two-photon excitation fluorescence and second-harmonic generation, is proposed in this work. To correlate the information of the NLM images with H&E images, this work proposes computational staining of NLM images using deep learning models in a supervised and an unsupervised approach. In the supervised and the unsupervised approach, conditional generative adversarial networks (CGANs) and cycle conditional generative adversarial networks (cycle CGANs) are used, respectively. Both CGAN and cycle CGAN models generate pseudo H&E images, which are quantitatively analyzed based on mean squared error, structure similarity index and color shading similarity index. The mean of the three metrics calculated for the computationally generated H&E images indicate significant performance. Thus, utilizing CGAN and cycle CGAN models for computational staining is beneficial for diagnostic applications without performing a laboratory-based staining procedure. To the author's best knowledge, it is the first time that NLM images are computationally stained to H&E images using GANs in an unsupervised manner.

Dissecting the Profile of Corneal Thickness With Keratoconus Progression Based on Anterior Segment Optical Coherence Tomography

PURPOSE

To characterize the corneal and epithelial thickness at different stages of keratoconus (KC), using a deep learning based corneal segmentation algorithm for anterior segment optical coherence tomography (AS-OCT).

METHODS

An AS-OCT dataset was constructed in this study with 1,430 images from 715 eyes, which included 118 normal eyes, 134 mild KC, 239 moderate KC, 153 severe KC, and 71 scarring KC. A deep learning based corneal segmentation algorithm was applied to isolate the epithelial and corneal tissues from the background. Based on the segmentation results, the thickness of epithelial and corneal tissues was automatically measured in the center 6 mm area. One-way ANOVA and linear regression were performed in 20 equally divided zones to explore the trend of the thickness changes at different locations with the KC progression. The 95% confidence intervals (CI) of epithelial thickness and corneal thickness in a specific zone were calculated to reveal the difference of thickness distribution among different groups.

RESULTS

Our data showed that the deep learning based corneal segmentation algorithm can achieve accurate tissue segmentation and the error range of measured thickness was less than 4 μm between our method and the results from clinical experts, which is approximately one image pixel. Statistical analyses revealed significant corneal thickness differences in all the divided zones (P < 0.05). The entire corneal thickness grew gradually thinner with the progression of the KC, and their trends were more pronounced around the pupil center with a slight shift toward the temporal and inferior side. Especially the epithelial thicknesses were thinner gradually from a normal eye to severe KC. Due to the formation of the corneal scarring, epithelial thickness had irregular fluctuations in the scarring KC.

CONCLUSION

Our study demonstrates that our deep learning method based on AS-OCT images could accurately delineate the corneal tissues and further successfully characterize the epithelial and corneal thickness changes at different stages of the KC progression.