Multiple surface segmentation using convolution neural nets: application to retinal layer segmentation in OCT images

Validation of Deep Learning-Based Automatic Retinal Layer Segmentation Algorithms for Age-Related Macular Degeneration with 2 Spectral-Domain OCT Devices

Purpose

Segmentations of retinal layers in spectral-domain OCT (SD-OCT) images serve as a crucial tool for identifying and analyzing the progression of various retinal diseases, encompassing a broad spectrum of abnormalities associated with age-related macular degeneration (AMD). The training of deep learning algorithms necessitates well-defined ground truth labels, validated by experts, to delineate boundaries accurately. However, this resource-intensive process has constrained the widespread application of such algorithms across diverse OCT devices. This work validates deep learning image segmentation models across multiple OCT devices by testing robustness in generating clinically relevant metrics.

Design

Prospective comparative study.

Participants

Adults >50 years of age with no AMD to advanced AMD, as defined in the Age-Related Eye Disease Study, in ≥1 eye, were enrolled. Four hundred two SD-OCT scans were used in this study.

Methods

We evaluate 2 separate state-of-the-art segmentation algorithms through a training process using images obtained from 1 OCT device (Heidelberg-Spectralis) and subsequent testing using images acquired from 2 OCT devices (Heidelberg-Spectralis and Zeiss-Cirrus). This assessment is performed on a dataset that encompasses a range of retinal pathologies, spanning from disease-free conditions to severe forms of AMD, with a focus on evaluating the device independence of the algorithms.

Main Outcome Measures

Performance metrics (including mean squared error, mean absolute error [MAE], and Dice coefficients) for the segmentations of the internal limiting membrane (ILM), retinal pigment epithelium (RPE), and RPE to Bruch's membrane region, along with en face thickness maps, volumetric estimations (in mm3). Violin plots and Bland-Altman plots comparing predictions against ground truth are also presented.

Results

The UNet and DeepLabv3, trained on Spectralis B-scans, demonstrate clinically useful outcomes when applied to Cirrus test B-scans. Review of the Cirrus test data by 2 independent annotators revealed that the aggregated MAE in pixels for ILM was 1.82 ± 0.24 (equivalent to 7.0 ± 0.9 μm) and for RPE was 2.46 ± 0.66 (9.5 ± 2.6 μm). Additionally, the Dice similarity coefficient for the RPE drusen complex region, comparing predictions to ground truth, reached 0.87 ± 0.01.

Conclusions

In the pursuit of task-specific goals such as retinal layer segmentation, a segmentation network has the capacity to acquire domain-independent features from a large training dataset. This enables the utilization of the network to execute tasks in domains where ground truth is hard to generate.

Financial Disclosures

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.

Boundary-Repairing Dual-Path Network for Retinal Layer Segmentation in OCT Image with Pigment Epithelial Detachment

Automatic retinal layer segmentation in optical coherence tomography (OCT) images is crucial for the diagnosis of ocular diseases. Currently, automatic retinal layer segmentation works well with normal OCT images. However, pigment epithelial detachment (PED) dramatically alters the retinal structure, causing blurred boundaries and partial disappearance of the Bruch's Membrane (BM), thus posing challenges to the segmentation. To tackle these problems, we propose a novel dual-path U-shaped network for simultaneous layer segmentation and boundary regression. This network first designs a feature interaction fusion (FIF) module to strengthen the boundary shape constraints in the layer path. To address the challenge posed by partial BM disappearance and boundary-blurring, we propose a layer boundary repair (LBR) module. This module aims to use contrastive loss to enhance the confidence of blurred boundary regions and refine the segmentation of layer boundaries through the re-prediction head. In addition, we introduce a novel bilateral threshold distance map (BTDM) designed for the boundary path. The BTDM serves to emphasize information within boundary regions. This map, combined with the updated probability map, culminates in topology-guaranteed segmentation results achieved through a topology correction (TC) module. We investigated the proposed network on two severely deformed datasets (i.e., OCTA-500 and Aier-PED) and one slightly deformed dataset (i.e., DUKE). The proposed method achieves an average Dice score of 94.26% on the OCTA-500 dataset, which was 1.5% higher than BAU-Net and outperformed other methods. In the DUKE and Aier-PED datasets, the proposed method achieved average Dice scores of 91.65% and 95.75%, respectively.

Diverse Data Generation for Retinal Layer Segmentation With Potential Structure Modeling

Accurate retinal layer segmentation on optical coherence tomography (OCT) images is hampered by the challenges of collecting OCT images with diverse pathological characterization and balanced distribution. Current generative models can produce high-realistic images and corresponding labels without quantitative limitations by fitting distributions of real collected data. Nevertheless, the diversity of their generated data is still limited due to the inherent imbalance of training data. To address these issues, we propose an image-label pair generation framework that generates diverse and balanced potential data from imbalanced real samples. Specifically, the framework first generates diverse layer masks, and then generates plausible OCT images corresponding to these layer masks using two customized diffusion probabilistic models respectively. To learn from imbalanced data and facilitate balanced generation, we introduce pathological-related conditions to guide the generation processes. To enhance the diversity of the generated image-label pairs, we propose a potential structure modeling technique that transfers the knowledge of diverse sub-structures from lowly- or non-pathological samples to highly pathological samples. We conducted extensive experiments on two public datasets for retinal layer segmentation. Firstly, our method generates OCT images with higher image quality and diversity compared to other generative methods. Furthermore, based on the extensive training with the generated OCT images, downstream retinal layer segmentation tasks demonstrate improved results. The code is publicly available at: https://github.com/nicetomeetu21/GenPSM.

Deep-learning-based automated measurement of outer retinal layer thickness for use in the assessment of age-related macular degeneration, applicable to both swept-source and spectral-domain OCT imaging

Effective biomarkers are required for assessing the progression of age-related macular degeneration (AMD), a prevalent and progressive eye disease. This paper presents a deep learning-based automated algorithm, applicable to both swept-source OCT (SS-OCT) and spectral-domain OCT (SD-OCT) scans, for measuring outer retinal layer (ORL) thickness as a surrogate biomarker for outer retinal degeneration, e.g., photoreceptor disruption, to assess AMD progression. The algorithm was developed based on a modified TransUNet model with clinically annotated retinal features manifested in the progression of AMD. The algorithm demonstrates a high accuracy with an intersection of union (IoU) of 0.9698 in the testing dataset for segmenting ORL using both SS-OCT and SD-OCT datasets. The robustness and applicability of the algorithm are indicated by strong correlation (r = 0.9551, P < 0.0001 in the central-fovea 3 mm-circle, and r = 0.9442, P < 0.0001 in the 5 mm-circle) and agreement (the mean bias = 0.5440 um in the 3-mm circle, and 1.392 um in the 5-mm circle) of the ORL thickness measurements between SS-OCT and SD-OCT scans. Comparative analysis reveals significant differences (P < 0.0001) in ORL thickness among 80 normal eyes, 30 intermediate AMD eyes with reticular pseudodrusen, 49 intermediate AMD eyes with drusen, and 40 late AMD eyes with geographic atrophy, highlighting its potential as an independent biomarker for predicting AMD progression. The findings provide valuable insights into the ORL alterations associated with different stages of AMD and emphasize the potential of ORL thickness as a sensitive indicator of AMD severity and progression.

Simultaneous alignment and surface regression using hybrid 2D-3D networks for 3D coherent layer segmentation of retinal OCT images with full and sparse annotations

Layer segmentation is important to quantitative analysis of retinal optical coherence tomography (OCT). Recently, deep learning based methods have been developed to automate this task and yield remarkable performance. However, due to the large spatial gap and potential mismatch between the B-scans of an OCT volume, all of them were based on 2D segmentation of individual B-scans, which may lose the continuity and diagnostic information of the retinal layers in 3D space. Besides, most of these methods required dense annotation of the OCT volumes, which is labor-intensive and expertise-demanding. This work presents a novel framework based on hybrid 2D-3D convolutional neural networks (CNNs) to obtain continuous 3D retinal layer surfaces from OCT volumes, which works well with both full and sparse annotations. The 2D features of individual B-scans are extracted by an encoder consisting of 2D convolutions. These 2D features are then used to produce the alignment displacement vectors and layer segmentation by two 3D decoders coupled via a spatial transformer module. Two losses are proposed to utilize the retinal layers' natural property of being smooth for B-scan alignment and layer segmentation, respectively, and are the key to the semi-supervised learning with sparse annotation. The entire framework is trained end-to-end. To the best of our knowledge, this is the first work that attempts 3D retinal layer segmentation in volumetric OCT images based on CNNs. Experiments on a synthetic dataset and three public clinical datasets show that our framework can effectively align the B-scans for potential motion correction, and achieves superior performance to state-of-the-art 2D deep learning methods in terms of both layer segmentation accuracy and cross-B-scan 3D continuity in both fully and semi-supervised settings, thus offering more clinical values than previous works.

Arterial hypertension and retinal layer thickness: the Beijing Eye Study

PURPOSE

To investigate relationships between blood pressure and the thickness of single retinal layers in the macula.

METHODS

Participants of the population-based Beijing Eye Study, free of retinal or optic nerve disease, underwent medical and ophthalmological examinations including optical coherence tomographic examination of the macula. Applying a multiple-surface segmentation solution, we automatically segmented the retina into its various layers.

RESULTS

The study included 2237 participants (mean age 61.8±8.4 years, range 50-93 years). Mean thicknesses of the retinal nerve fibre layer (RNFL), ganglion cell layer (GCL), inner plexiform layer, inner nuclear layer (INL), outer plexiform layer, outer nuclear layer/external limiting membrane, ellipsoid zone, photoreceptor outer segments (POS) and retinal pigment epithelium-Bruch membrane were 31.1±2.3 µm, 39.7±3.5 µm, 38.4±3.3 µm, 34.8±2.0 µm, 28.1±3.0 µm, 79.2±7.3 µm, 22.9±0.6 µm, 19.2±3.3 µm and 20.7±1.4 µm, respectively. In multivariable analysis, higher systolic blood pressure (SBP) and diastolic blood pressure (DBP) were associated with thinner GCL and thicker INL, after adjusting for age, sex and axial length (all p<0.0056). Higher SBP was additionally associated with thinner POS and higher DBP with thinner RNFL. For an elevation of SBP/DBP by 10 mm Hg, the RNFL, GCL, INL and POS changed by 2.0, 3.0, 1.5 and 2.0 µm, respectively.

CONCLUSIONS

Thickness of RNFL, GCL and POS was inversely and INL thickness was positively associated with higher blood pressure, while the thickness of the other retinal layers was not significantly correlated with blood pressure. The findings may be helpful for refinement of the morphometric detection of retinal diseases.

Deep Learning-Assisted Measurements of Photoreceptor Ellipsoid Zone Area and Outer Segment Volume as Biomarkers for Retinitis Pigmentosa

The manual segmentation of retinal layers from OCT scan images is time-consuming and costly. The deep learning approach has potential for the automatic delineation of retinal layers to significantly reduce the burden of human graders. In this study, we compared deep learning model (DLM) segmentation with manual correction (DLM-MC) to conventional manual grading (MG) for the measurements of the photoreceptor ellipsoid zone (EZ) area and outer segment (OS) volume in retinitis pigmentosa (RP) to assess whether DLM-MC can be a new gold standard for retinal layer segmentation and for the measurement of retinal layer metrics. Ninety-six high-speed 9 mm 31-line volume scans obtained from 48 patients with RPGR-associated XLRP were selected based on the following criteria: the presence of an EZ band within the scan limit and a detectable EZ in at least three B-scans in a volume scan. All the B-scan images in each volume scan were manually segmented for the EZ and proximal retinal pigment epithelium (pRPE) by two experienced human graders to serve as the ground truth for comparison. The test volume scans were also segmented by a DLM and then manually corrected for EZ and pRPE by the same two graders to obtain DLM-MC segmentation. The EZ area and OS volume were determined by interpolating the discrete two-dimensional B-scan EZ-pRPE layer over the scan area. Dice similarity, Bland-Altman analysis, correlation, and linear regression analyses were conducted to assess the agreement between DLM-MC and MG for the EZ area and OS volume measurements. For the EZ area, the overall mean dice score (SD) between DLM-MC and MG was 0.8524 (0.0821), which was comparable to 0.8417 (0.1111) between two MGs. For the EZ area > 1 mm2, the average dice score increased to 0.8799 (0.0614). When comparing DLM-MC to MG, the Bland-Altman plots revealed a mean difference (SE) of 0.0132 (0.0953) mm2 and a coefficient of repeatability (CoR) of 1.8303 mm2 for the EZ area and a mean difference (SE) of 0.0080 (0.0020) mm3 and a CoR of 0.0381 mm3 for the OS volume. The correlation coefficients (95% CI) were 0.9928 (0.9892-0.9952) and 0.9938 (0.9906-0.9958) for the EZ area and OS volume, respectively. The linear regression slopes (95% CI) were 0.9598 (0.9399-0.9797) and 1.0104 (0.9909-1.0298), respectively. The results from this study suggest that the manual correction of deep learning model segmentation can generate EZ area and OS volume measurements in excellent agreement with those of conventional manual grading in RP. Because DLM-MC is more efficient for retinal layer segmentation from OCT scan images, it has the potential to reduce the burden of human graders in obtaining quantitative measurements of biomarkers for assessing disease progression and treatment outcomes in RP.

Graph Attention U-Net for Retinal Layer Surface Detection and Choroid Neovascularization Segmentation in OCT Images

Choroidal neovascularization (CNV) is a typical symptom of age-related macular degeneration (AMD) and is one of the leading causes for blindness. Accurate segmentation of CNV and detection of retinal layers are critical for eye disease diagnosis and monitoring. In this paper, we propose a novel graph attention U-Net (GA-UNet) for retinal layer surface detection and CNV segmentation in optical coherence tomography (OCT) images. Due to retinal layer deformation caused by CNV, it is challenging for existing models to segment CNV and detect retinal layer surfaces with the correct topological order. We propose two novel modules to address the challenge. The first module is a graph attention encoder (GAE) in a U-Net model that automatically integrates topological and pathological knowledge of retinal layers into the U-Net structure to achieve effective feature embedding. The second module is a graph decorrelation module (GDM) that takes reconstructed features by the decoder of the U-Net as inputs, it then decorrelates and removes information unrelated to retinal layer for improved retinal layer surface detection. In addition, we propose a new loss function to maintain the correct topological order of retinal layers and the continuity of their boundaries. The proposed model learns graph attention maps automatically during training and performs retinal layer surface detection and CNV segmentation simultaneously with the attention maps during inference. We evaluated the proposed model on our private AMD dataset and another public dataset. Experiment results show that the proposed model outperformed the competing methods for retinal layer surface detection and CNV segmentation and achieved new state of the arts on the datasets.

Automatic Segmentation of Retinal Layers in Multiple Neurodegenerative Disorder Scenarios

Retinal Optical Coherence Tomography (OCT) allows the non-invasive direct observation of the central nervous system, enabling the measurement and extraction of biomarkers from neural tissue that can be helpful in the assessment of ocular, systemic and Neurological Disorders (ND). Deep learning models can be trained to segment the retinal layers for biomarker extraction. However, the onset of ND can have an impact on the neural tissue, which can lead to the degraded performance of models not exposed to images displaying signs of disease during training. We present a fully automatic approach for the retinal layer segmentation in multiple neurodegenerative disorder scenarios, using an annotated dataset of patients of the most prevalent NDs: Alzheimer's disease, Parkinson's disease, multiple sclerosis and essential tremor, along with healthy control patients. Furthermore, we present a two-part, comprehensive study on the effects of ND on the performance of these models. The results show that images of healthy patients may not be sufficient for the robust training of automated segmentation models intended for the analysis of ND patients, and that using images representative of different NDs can increase the model performance. These results indicate that the presence or absence of patients of ND in datasets should be taken into account when training deep learning models for retinal layer segmentation, and that the proposed approach can provide a valuable tool for the robust and reliable diagnosis in multiple scenarios of ND.

Deep learning network with differentiable dynamic programming for retina OCT surface segmentation

Multiple-surface segmentation in optical coherence tomography (OCT) images is a challenging problem, further complicated by the frequent presence of weak image boundaries. Recently, many deep learning-based methods have been developed for this task and yield remarkable performance. Unfortunately, due to the scarcity of training data in medical imaging, it is challenging for deep learning networks to learn the global structure of the target surfaces, including surface smoothness. To bridge this gap, this study proposes to seamlessly unify a U-Net for feature learning with a constrained differentiable dynamic programming module to achieve end-to-end learning for retina OCT surface segmentation to explicitly enforce surface smoothness. It effectively utilizes the feedback from the downstream model optimization module to guide feature learning, yielding better enforcement of global structures of the target surfaces. Experiments on Duke AMD (age-related macular degeneration) and JHU MS (multiple sclerosis) OCT data sets for retinal layer segmentation demonstrated that the proposed method was able to achieve subvoxel accuracy on both datasets, with the mean absolute surface distance (MASD) errors of 1.88 ± 1.96μm and 2.75 ± 0.94μm, respectively, over all the segmented surfaces.

Accurate drusen segmentation in optical coherence tomography via order-constrained regression of retinal layer heights

Drusen are an important biomarker for age-related macular degeneration (AMD). Their accurate segmentation based on optical coherence tomography (OCT) is therefore relevant to the detection, staging, and treatment of disease. Since manual OCT segmentation is resource-consuming and has low reproducibility, automatic techniques are required. In this work, we introduce a novel deep learning based architecture that directly predicts the position of layers in OCT and guarantees their correct order, achieving state-of-the-art results for retinal layer segmentation. In particular, the average absolute distance between our model's prediction and the ground truth layer segmentation in an AMD dataset is 0.63, 0.85, and 0.44 pixel for Bruch's membrane (BM), retinal pigment epithelium (RPE) and ellipsoid zone (EZ), respectively. Based on layer positions, we further quantify drusen load with excellent accuracy, achieving 0.994 and 0.988 Pearson correlation between drusen volumes estimated by our method and two human readers, and increasing the Dice score to 0.71 ± 0.16 (from 0.60 ± 0.23) and 0.62 ± 0.23 (from 0.53 ± 0.25), respectively, compared to a previous state-of-the-art method. Given its reproducible, accurate, and scalable results, our method can be used for the large-scale analysis of OCT data.

Toward reliable calcification detection: calibration of uncertainty in object detection from coronary optical coherence tomography images

SIGNIFICANCE

Optical coherence tomography (OCT) has become increasingly essential in assisting the treatment of coronary artery disease (CAD). However, unidentified calcified regions within a narrowed artery could impair the outcome of the treatment. Fast and objective identification is paramount to automatically procuring accurate readings on calcifications within the artery.

AIM

We aim to rapidly identify calcification in coronary OCT images using a bounding box and reduce the prediction bias in automated prediction models.

APPROACH

We first adopt a deep learning-based object detection model to rapidly draw the calcified region from coronary OCT images using a bounding box. We measure the uncertainty of predictions based on the expected calibration errors, thus assessing the certainty level of detection results. To calibrate confidence scores of predictions, we implement dependent logistic calibration using each detection result's confidence and center coordinates.

RESULTS

We implemented an object detection module to draw the boundary of the calcified region at a rate of 140 frames per second. With the calibrated confidence score of each prediction, we lower the uncertainty of predictions in calcification detection and eliminate the estimation bias from various object detection methods. The calibrated confidence of prediction results in a confidence error of ∼ 0.13 , suggesting that the confidence calibration on calcification detection could provide a more trustworthy result.

CONCLUSIONS

Given the rapid detection and effective calibration of the proposed work, we expect that it can assist in clinical evaluation of treating the CAD during the imaging-guided procedure.

Exploring healthy retinal aging with deep learning

Purpose

To study the individual course of retinal changes caused by healthy aging using deep learning.

Design

Retrospective analysis of a large data set of retinal OCT images.

Participants

A total of 85 709 adults between the age of 40 and 75 years of whom OCT images were acquired in the scope of the UK Biobank population study.

Methods

We created a counterfactual generative adversarial network (GAN), a type of neural network that learns from cross-sectional, retrospective data. It then synthesizes high-resolution counterfactual OCT images and longitudinal time series. These counterfactuals allow visualization and analysis of hypothetical scenarios in which certain characteristics of the imaged subject, such as age or sex, are altered, whereas other attributes, crucially the subject's identity and image acquisition settings, remain fixed.

Main Outcome Measures

Using our counterfactual GAN, we investigated subject-specific changes in the retinal layer structure as a function of age and sex. In particular, we measured changes in the retinal nerve fiber layer (RNFL), combined ganglion cell layer plus inner plexiform layer (GCIPL), inner nuclear layer to the inner boundary of the retinal pigment epithelium (INL-RPE), and retinal pigment epithelium (RPE).

Results

Our counterfactual GAN is able to smoothly visualize the individual course of retinal aging. Across all counterfactual images, the RNFL, GCIPL, INL-RPE, and RPE changed by -0.1 μm ± 0.1 μm, -0.5 μm ± 0.2 μm, -0.2 μm ± 0.1 μm, and 0.1 μm ± 0.1 μm, respectively, per decade of age. These results agree well with previous studies based on the same cohort from the UK Biobank population study. Beyond population-wide average measures, our counterfactual GAN allows us to explore whether the retinal layers of a given eye will increase in thickness, decrease in thickness, or stagnate as a subject ages.

Conclusion

This study demonstrates how counterfactual GANs can aid research into retinal aging by generating high-resolution, high-fidelity OCT images, and longitudinal time series. Ultimately, we envision that they will enable clinical experts to derive and explore hypotheses for potential imaging biomarkers for healthy and pathologic aging that can be refined and tested in prospective clinical trials.

Financial Disclosures

Proprietary or commercial disclosure may be found after the references.

Multiscale joint segmentation method for retinal optical coherence tomography images using a bidirectional wave algorithm and improved graph theory

Morphology and functional metrics of retinal layers are important biomarkers for many human ophthalmic diseases. Automatic and accurate segmentation of retinal layers is crucial for disease diagnosis and research. To improve the performance of retinal layer segmentation, a multiscale joint segmentation framework for retinal optical coherence tomography (OCT) images based on bidirectional wave algorithm and improved graph theory is proposed. In this framework, the bidirectional wave algorithm was used to segment edge information in multiscale images, and the improved graph theory was used to modify edge information globally, to realize automatic and accurate segmentation of eight retinal layer boundaries. This framework was tested on two public datasets and two OCT imaging systems. The test results show that, compared with other state-of-the-art methods, this framework does not need data pre-training and parameter pre-adjustment on different datasets, and can achieve sub-pixel retinal layer segmentation on a low-configuration computer.

Automated detection and growth tracking of 3D bio-printed organoid clusters using optical coherence tomography with deep convolutional neural networks

Organoids are advancing the development of accurate prediction of drug efficacy and toxicity in vitro. These advancements are attributed to the ability of organoids to recapitulate key structural and functional features of organs and parent tumor. Specifically, organoids are self-organized assembly with a multi-scale structure of 30-800 μm, which exacerbates the difficulty of non-destructive three-dimensional (3D) imaging, tracking and classification analysis for organoid clusters by traditional microscopy techniques. Here, we devise a 3D imaging, segmentation and analysis method based on Optical coherence tomography (OCT) technology and deep convolutional neural networks (CNNs) for printed organoid clusters (Organoid Printing and optical coherence tomography-based analysis, OPO). The results demonstrate that the organoid scale influences the segmentation effect of the neural network. The multi-scale information-guided optimized EGO-Net we designed achieves the best results, especially showing better recognition workout for the biologically significant organoid with diameter ≥50 μm than other neural networks. Moreover, OPO achieves to reconstruct the multiscale structure of organoid clusters within printed microbeads and calibrate the printing errors by segmenting the printed microbeads edges. Overall, the classification, tracking and quantitative analysis based on image reveal that the growth process of organoid undergoes morphological changes such as volume growth, cavity creation and fusion, and quantitative calculation of the volume demonstrates that the growth rate of organoid is associated with the initial scale. The new method we proposed enable the study of growth, structural evolution and heterogeneity for the organoid cluster, which is valuable for drug screening and tumor drug sensitivity detection based on organoids.

Applications of Artificial Intelligence in Optical Coherence Tomography Angiography Imaging

Optical coherence tomography angiography (OCTA) and artificial intelligence (AI) are two emerging fields that complement each other. OCTA enables the noninvasive, in vivo, 3D visualization of retinal blood flow with a micrometer resolution, which has been impossible with other imaging modalities. As it does not need dye-based injections, it is also a safer procedure for patients. AI has excited great interest in many fields of daily life, by enabling automatic processing of huge amounts of data with a performance that greatly surpasses previous algorithms. It has been used in many breakthrough studies in recent years, such as the finding that AlphaGo can beat humans in the strategic board game of Go. This paper will give a short introduction into both fields and will then explore the manifold applications of AI in OCTA imaging that have been presented in the recent years. These range from signal generation over signal enhancement to interpretation tasks like segmentation and classification. In all these areas, AI-based algorithms have achieved state-of-the-art performance that has the potential to improve standard care in ophthalmology when integrated into the daily clinical routine.

Retinal layer segmentation in optical coherence tomography (OCT) using a 3D deep-convolutional regression network for patients with age-related macular degeneration

Introduction - Retinal layer segmentation in optical coherence tomography (OCT) images is an important approach for detecting and prognosing disease. Automating segmentation using robust machine learning techniques lead to computationally efficient solutions and significantly reduces the cost of labor-intensive labeling, which is traditionally performed by trained graders at a reading center, sometimes aided by semi-automated algorithms. Although several algorithms have been proposed since the revival of deep learning, eyes with severe pathological conditions continue to challenge fully automated segmentation approaches. There remains an opportunity to leverage the underlying spatial correlations between the retinal surfaces in the segmentation approach. Methods - Some of these proposed traditional methods can be expanded to utilize the three-dimensional spatial context governing the retinal image volumes by replacing the use of 2D filters with 3D filters. Towards this purpose, we propose a spatial-context, continuity and anatomical relationship preserving semantic segmentation algorithm, which utilizes the 3D spatial context from the image volumes with the use of 3D filters. We propose a 3D deep neural network capable of learning the surface positions of the layers in the retinal volumes. Results - We utilize a dataset of OCT images from patients with Age-related Macular Degeneration (AMD) to assess performance of our model and provide both qualitative (including segmentation maps and thickness maps) and quantitative (including error metric comparisons and volumetric comparisons) results, which demonstrate that our proposed method performs favorably even for eyes with pathological changes caused by severe retinal diseases. The Mean Absolute Error (MAE) and Root Mean Squared Error (RMSE) for patients with a wide range of AMD severity scores (0-11) were within 0.84±0.41 and 1.33±0.73 pixels, respectively, which are significantly better than some of the other state-of-the-art algorithms. Conclusion - The results demonstrate the utility of extracting features from the entire OCT volume by treating the volume as a correlated entity and show the benefit of utilizing 3D autoencoder based regression networks for smoothing the approximated retinal layers by inducing shape based regularization constraints.

Intraretinal Layer Segmentation Using Cascaded Compressed U-Nets

Reliable biomarkers quantifying neurodegeneration and neuroinflammation in central nervous system disorders such as Multiple Sclerosis, Alzheimer's dementia or Parkinson's disease are an unmet clinical need. Intraretinal layer thicknesses on macular optical coherence tomography (OCT) images are promising noninvasive biomarkers querying neuroretinal structures with near cellular resolution. However, changes are typically subtle, while tissue gradients can be weak, making intraretinal segmentation a challenging task. A robust and efficient method that requires no or minimal manual correction is an unmet need to foster reliable and reproducible research as well as clinical application. Here, we propose and validate a cascaded two-stage network for intraretinal layer segmentation, with both networks being compressed versions of U-Net (CCU-INSEG). The first network is responsible for retinal tissue segmentation from OCT B-scans. The second network segments eight intraretinal layers with high fidelity. At the post-processing stage, we introduce Laplacian-based outlier detection with layer surface hole filling by adaptive non-linear interpolation. Additionally, we propose a weighted version of focal loss to minimize the foreground-background pixel imbalance in the training data. We train our method using 17,458 B-scans from patients with autoimmune optic neuropathies, i.e., multiple sclerosis, and healthy controls. Voxel-wise comparison against manual segmentation produces a mean absolute error of 2.3 μm, outperforming current state-of-the-art methods on the same data set. Voxel-wise comparison against external glaucoma data leads to a mean absolute error of 2.6 μm when using the same gold standard segmentation approach, and 3.7 μm mean absolute error in an externally segmented data set. In scans from patients with severe optic atrophy, 3.5% of B-scan segmentation results were rejected by an experienced grader, whereas this was the case in 41.4% of B-scans segmented with a graph-based reference method. The validation results suggest that the proposed method can robustly segment macular scans from eyes with even severe neuroretinal changes.

Self-Supervised Sequence Recovery for Semi-Supervised Retinal Layer Segmentation

Automated layer segmentation plays an important role for retinal disease diagnosis in optical coherence tomography (OCT) images. However, the severe retinal diseases result in the performance degeneration of automated layer segmentation approaches. In this paper, we present a robust semi-supervised retinal layer segmentation network to relieve the model failures on abnormal retinas, in which we obtain the lesion features from the labeled images with disease-balanced distribution, and utilize the unlabeled images to supplement the layer structure information. Specifically, in our proposed method, the cross-consistency training is utilized over the predictions of the different decoders, and we enforce a consistency between different decoder predictions to improve the encoders representation. Then, we proposed a sequence prediction branch based on self-supervised manner, which is designed to predict the position of each jigsaw puzzle to obtain sensory perception of the retinal layer structure. To this task, a layer spatial pyramid pooling (LSPP) module is designed to extract multi-scale layer spatial features. Furthermore, we use the optical coherence tomography angiography (OCTA) to supplement the information damaged by diseases. The experimental results validate that our method achieves more robust results compared with current supervised segmentation methods. Meanwhile, advanced segmentation performance can be obtained compared with state-of-the-art semi-supervised segmentation methods.

Surgical scene generation and adversarial networks for physics-based iOCT synthesis

The development and integration of intraoperative optical coherence tomography (iOCT) into modern operating rooms has motivated novel procedures directed at improving the outcome of ophthalmic surgeries. Although computer-assisted algorithms could further advance such interventions, the limited availability and accessibility of iOCT systems constrains the generation of dedicated data sets. This paper introduces a novel framework combining a virtual setup and deep learning algorithms to generate synthetic iOCT data in a simulated environment. The virtual setup reproduces the geometry of retinal layers extracted from real data and allows the integration of virtual microsurgical instrument models. Our scene rendering approach extracts information from the environment and considers iOCT typical imaging artifacts to generate cross-sectional label maps, which in turn are used to synthesize iOCT B-scans via a generative adversarial network. In our experiments we investigate the similarity between real and synthetic images, show the relevance of using the generated data for image-guided interventions and demonstrate the potential of 3D iOCT data synthesis.

Globally optimal OCT surface segmentation using a constrained IPM optimization

Segmentation of multiple surfaces in optical coherence tomography (OCT) images is a challenging problem, further complicated by the frequent presence of weak boundaries, varying layer thicknesses, and mutual influence between adjacent surfaces. The traditional graph-based optimal surface segmentation method has proven its effectiveness with its ability to capture various surface priors in a uniform graph model. However, its efficacy heavily relies on handcrafted features that are used to define the surface cost for the "goodness" of a surface. Recently, deep learning (DL) is emerging as a powerful tool for medical image segmentation thanks to its superior feature learning capability. Unfortunately, due to the scarcity of training data in medical imaging, it is nontrivial for DL networks to implicitly learn the global structure of the target surfaces, including surface interactions. This study proposes to parameterize the surface cost functions in the graph model and leverage DL to learn those parameters. The multiple optimal surfaces are then simultaneously detected by minimizing the total surface cost while explicitly enforcing the mutual surface interaction constraints. The optimization problem is solved by the primal-dual interior-point method (IPM), which can be implemented by a layer of neural networks, enabling efficient end-to-end training of the whole network. Experiments on spectral-domain optical coherence tomography (SD-OCT) retinal layer segmentation demonstrated promising segmentation results with sub-pixel accuracy.

CT image crack segmentation method based on linear feature enhancement

Industrial computed tomography (CT) crack segmentation is a key technology in industrial CT image processing. Unfortunately, the interference of artifact and noise in CT image often bring great trouble to the crack segmentation. In order to improve the segmentation accuracy of cracks in CT images, we propose to develop and test a new crack segmentation algorithm based on linear feature enhancement by analyzing the features of cracks in CT images. Firstly, the total variational model is used to denoise the input image. Next, a Frangi multiscale filter is used to extract linear structures in the image, and then the extracted linear structures are used to enhance the contrast of the image. Finally, the cracks in the image are detected and segmented by Otsu algorithm. By comparing with the manual segmentation results, the average intersection-over-union (IOU) reaches 86.10% and the average F1 score reaches 92.44%, which verifies the effectiveness and correctness of the algorithm developed in this study. Overall, experiments demonstrate that the new algorithm improves the accuracy of crack segmentation and it is effective applying to industry CT images.

Rapid, Spontaneous Resolution of Prominent Subretinal Infiltrate in Vitreoretinal Lymphoma

Purpose:

This case report describes a patient with vitreoretinal lymphoma who subacutely developed a large, peripapillary subretinal infiltrate that rapidly and spontaneously resolved.

Methods:

A case report is presented.

Results:

A 65-year-old Asian-American woman was referred for evaluation of a dense, peripapillary subretinal infiltrate in the left eye. A diagnostic vitrectomy revealed large, atypical lymphocytes with irregularly shaped nuclei, and mutational testing was positive for myeloid differentiation primary response 88 ( MYD88). Prior to surgery, the patient’s subretinal infiltrate had begun to resolve spontaneously, a process that continued after surgery without initiation of systemic or local ocular therapy.

Conclusions:

Patients with vitreoretinal lymphoma may present with transient, subretinal infiltrates that can resolve without treatment.

Segmentation of Preretinal Space in Optical Coherence Tomography Images Using Deep Neural Networks

This paper proposes an efficient segmentation of the preretinal area between the inner limiting membrane (ILM) and posterior cortical vitreous (PCV) of the human eye in an image obtained with the use of optical coherence tomography (OCT). The research was carried out using a database of three-dimensional OCT imaging scans obtained with the Optovue RTVue XR Avanti device. Various types of neural networks (UNet, Attention UNet, ReLayNet, LFUNet) were tested for semantic segmentation, their effectiveness was assessed using the Dice coefficient and compared to the graph theory techniques. Improvement in segmentation efficiency was achieved through the use of relative distance maps. We also show that selecting a larger kernel size for convolutional layers can improve segmentation quality depending on the neural network model. In the case of PVC, we obtain the effectiveness reaching up to 96.35%. The proposed solution can be widely used to diagnose vitreomacular traction changes, which is not yet available in scientific or commercial OCT imaging solutions.

A Hybrid Model Composed of Two Convolutional Neural Networks (CNNs) for Automatic Retinal Layer Segmentation of OCT Images in Retinitis Pigmentosa (RP)

Purpose

We propose and evaluate a hybrid model composed of two convolutional neural networks (CNNs) with different architectures for automatic segmentation of retina layers in spectral domain optical coherence tomography (SD-OCT) B-scans of retinitis pigmentosa (RP).

Methods

The hybrid model consisted of a U-Net for initial semantic segmentation and a sliding-window (SW) CNN for refinement by correcting the segmentation errors of U-Net. The U-Net construction followed Ronneberger et al. (2015) with an input image size of 256 × 32. The SW model was similar to our previously reported approach. Training image patches were generated from 480 horizontal midline B-scans obtained from 220 patients with RP and 20 normal participants. Testing images were 160 midline B-scans from a separate group of 80 patients with RP. The Spectralis segmentation of B-scans was manually corrected for the boundaries of the inner limiting membrane, inner nuclear layer, ellipsoid zone (EZ), retinal pigment epithelium, and Bruch's membrane by one grader for the training set and two for the testing set. The trained U-Net and SW, as well as the hybrid model, were used to classify all pixels in the testing B-scans. Bland–Altman and correlation analyses were conducted to compare layer boundary lines, EZ width, and photoreceptor outer segment (OS) length and area determined by the models to those by human graders.

Results

The mean times to classify a B-scan image were 0.3, 65.7, and 2.4 seconds for U-Net, SW, and the hybrid model, respectively. The mean ± SD accuracies to segment retinal layers were 90.8% ± 4.8% and 90.7% ± 4.0% for U-Net and SW, respectively. The hybrid model improved mean ± SD accuracy to 91.5% ± 4.8% (P < 0.039 vs. U-Net), resulting in an improvement in layer boundary segmentation as revealed by Bland–Altman analyses. EZ width, OS length, and OS area measured by the models were highly correlated with those measured by the human graders (r > 0.95 for EZ width; r > 0.83 for OS length; r > 0.97 for OS area; P < 0.05). The hybrid model further improved the performance of measuring retinal layer thickness by correcting misclassification of retinal layers from U-Net.

Conclusions

While the performances of U-Net and the SW model were comparable in delineating various retinal layers, U-Net was much faster than the SW model to segment B-scan images. The hybrid model that combines the two improves automatic retinal layer segmentation from OCT images in RP.

Translational Relevance

A hybrid deep machine learning model composed of CNNs with different architectures can be more effective than either model separately for automatic analysis of SD-OCT scan images, which is becoming increasingly necessary with current high-resolution, high-density volume scans.

Robust region encoding and layer attribute protection for the segmentation of retina with multifarious abnormalities

PURPOSE

To robustly segment retinal layers that are affected by complex variety of retinal diseases for optical coherence tomography angiography (OCTA) en face projection generation.

METHODS

In this paper, we propose a robust retinal layer segmentation model to reduce the impact of multifarious abnormalities on model performance. OCTA vascular distribution that is regarded as the supplements of spectral domain optical coherence tomography (SD-OCT) structural information is introduced to improve the robustness of layer region encoding. To further reduce the sensitivity of region encoding to retinal abnormalities, we propose a multitask layer-wise refinement (MLR) module that can refine the initial layer region segmentation results layer-by-layer. Finally, we design a region-to-surface transformation (RtST) module without additional training parameters to convert the encoding layer regions to their corresponding layer surfaces. This transformation from layer regions to layer surfaces can remove the inaccurate segmentation regions, and the layer surfaces are easier to be used to protect the retinal layer natures than layer regions.

RESULTS

Experimental data includes 273 eyes, where 95 eyes are normal and 178 eyes contain complex retinal diseases, including age-related macular degeneration (AMD), diabetic retinopathy (DR), central serous chorioretinopathy (CSC), choroidal neovascularization (CNV), and so forth. The dice similarity coefficient (DSC: %) of superficial, deep and outer retina achieves 98.92, 97.48, and 98.87 on normal eyes and 98.35, 95.33, and 98.17 on abnormal eyes. Compared with other commonly used layer segmentation models, our model achieves the state-of-the-art layer segmentation performance.

CONCLUSIONS

The final results prove that our proposed model obtains outstanding performance and has enough ability to resist retinal abnormalities. Besides, OCTA modality is helpful for retinal layer segmentation.

Deep Learning-Based Retinal Nerve Fiber Layer Thickness Measurement of Murine Eyes

Purpose

To design a robust and automated estimation method for measuring the retinal nerve fiber layer (RNFL) thickness using spectral domain optical coherence tomography (SD-OCT).

Methods

We developed a deep learning-based image segmentation network for automated segmentation of the RNFL in SD-OCT B-scans of mouse eyes. In total, 5500 SD-OCT B-scans (5200 B-scans were used as training data with the remaining 300 B-scans used as testing data) were used to develop this segmentation network. Postprocessing operations were then applied on the segmentation results to fill any discontinuities or remove any speckles in the RNFL. Subsequently, a three-dimensional retina thickness map was generated by z-stacking 100 segmentation processed thickness B-scan images together. Finally, the average absolute difference between algorithm predicted RNFL thickness compared to the ground truth manual human segmentation was calculated.

Results

The proposed method achieves an average dice similarity coefficient of 0.929 in the SD-OCT segmentation task and an average absolute difference of 0.0009 mm in thickness estimation task on the basis of the testing dataset. We also evaluated our segmentation algorithm on another biological dataset with SD-OCT volumes for RNFL thickness after the optic nerve crush injury. Results were shown to be comparable between the predicted and manually measured retina thickness values.

Conclusions

Experimental results demonstrate that our automated segmentation algorithm reliably predicts the RNFL thickness in SD-OCT volumes of mouse eyes compared to laborious and more subjective manual SD-OCT RNFL segmentation.

Translational Relevance

Automated segmentation using a deep learning-based algorithm for murine eye OCT effectively and rapidly produced nerve fiber layer thicknesses comparable to manual segmentation.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

A column-based deep learning method for the detection and quantification of atrophy associated with AMD in OCT scans

The objective quantification of retinal atrophy associated with age-related macular degeneration (AMD) is required for clinical diagnosis, follow-up, treatment efficacy evaluation, and clinical research. Spectral Domain Optical Coherence Tomography (OCT) has become an essential imaging technology to evaluate the macula. This paper describes a novel automatic method for the identification and quantification of atrophy associated with AMD in OCT scans and its visualization in the corresponding infrared imaging (IR) image. The method is based on the classification of light scattering patterns in vertical pixel-wide columns (A-scans) in OCT slices (B-scans) in which atrophy appears with a custom column-based convolutional neural network (CNN). The network classifies individual columns with 3D column patches formed by adjacent neighboring columns from the volumetric OCT scan. Subsequent atrophy columns form atrophy segments which are then projected onto the IR image and are used to identify and segment atrophy lesions in the IR image and to measure their areas and distances from the fovea. Experimental results on 106 clinical OCT scans (5,207 slices) in which cRORA atrophy (the end point of advanced dry AMD) was identified in 2,952 atrophy segments and 1,046 atrophy lesions yield a mean F₁ score of 0.78 (std 0.06) and an AUC of 0.937, both close to the observer variability. Automated computer-based detection and quantification of atrophy associated with AMD using a column-based CNN classification in OCT scans can be performed at expert level and may be a useful clinical decision support and research tool for the diagnosis, follow-up and treatment of retinal degenerations and dystrophies.

CNN-based CP-OCT sensor integrated with a subretinal injector for retinal boundary tracking and injection guidance

SIGNIFICANCE

Subretinal injection is an effective way of delivering transplant genes and cells to treat many degenerative retinal diseases. However, the technique requires high-dexterity and microscale precision of experienced surgeons, who have to overcome the physiological hand tremor and limited visualization of the subretinal space.

AIM

To automatically guide the axial motion of microsurgical tools (i.e., a subretinal injector) with microscale precision in real time using a fiber-optic common-path swept-source optical coherence tomography distal sensor.

APPROACH

We propose, implement, and study real-time retinal boundary tracking of A-scan optical coherence tomography (OCT) images using a convolutional neural network (CNN) for automatic depth targeting of a selected retinal boundary for accurate subretinal injection guidance. A simplified 1D U-net is used for the retinal layer segmentation on A-scan OCT images. A Kalman filter, combining retinal boundary position measurement by CNN and velocity measurement by cross correlation between consecutive A-scan images, is applied to optimally estimate the retinal boundary position. Unwanted axial motions of the surgical tools are compensated by a piezoelectric linear motor based on the retinal boundary tracking.

RESULTS

CNN-based segmentation on A-scan OCT images achieves the mean unsigned error (MUE) of ∼3  pixels (8.1  μm) using an ex vivo bovine retina model. GPU parallel computing allows real-time inference (∼2  ms) and thus real-time retinal boundary tracking. Involuntary tremors, which include low-frequency draft in hundreds of micrometers and physiological tremors in tens of micrometers, are compensated effectively. The standard deviations of photoreceptor (PR) and choroid (CH) boundary positions get as low as 10.8  μm when the depth targeting is activated.

CONCLUSIONS

A CNN-based common-path OCT distal sensor successfully tracks retinal boundaries, especially the PR/CH boundary for subretinal injection, and automatically guides the tooltip's axial position in real time. The microscale depth targeting accuracy of our system shows its promising possibility for clinical application.

Deep Learning Image Feature Recognition Algorithm for Judgment on the Rationality of Landscape Planning and Design

This paper uses an improved deep learning algorithm to judge the rationality of the design of landscape image feature recognition. The preprocessing of the image is proposed to enhance the data. The deficiencies in landscape feature extraction are further addressed based on the new model. Then, the two-stage training method of the model is used to solve the problems of long training time and convergence difficulties in deep learning. Innovative methods for zoning and segmentation training of landscape pattern features are proposed, which makes model training faster and generates more creative landscape patterns. Because of the impact of too many types of landscape elements in landscape images, traditional convolutional neural networks can no longer effectively solve this problem. On this basis, a fully convolutional neural network model is designed to perform semantic segmentation of landscape elements in landscape images. Through the method of deconvolution, the pixel-level semantic segmentation is realized. Compared with the 65% accuracy rate of the convolutional neural network, the fully convolutional neural network has an accuracy rate of 90.3% for the recognition of landscape elements. The method is effective, accurate, and intelligent for the classification of landscape element design, which better improves the accuracy of classification, greatly reduces the cost of landscape element design classification, and ensures that the technical method is feasible. This paper classifies landscape behavior based on this model for full convolutional neural network landscape images and demonstrates the effectiveness of using the model. In terms of landscape image processing, the image evaluation provides a certain basis.

Real-time noise reduction based on ground truth free deep learning for optical coherence tomography

Optical coherence tomography (OCT) is a high-resolution non-invasive 3D imaging modality, which has been widely used for biomedical research and clinical studies. The presence of noise on OCT images is inevitable which will cause problems for post-image processing and diagnosis. The frame-averaging technique that acquires multiple OCT images at the same or adjacent locations can enhance the image quality significantly. Both conventional frame averaging methods and deep learning-based methods using averaged frames as ground truth have been reported. However, conventional averaging methods suffer from the limitation of long image acquisition time, while deep learning-based methods require complicated and tedious ground truth label preparation. In this work, we report a deep learning-based noise reduction method that does not require clean images as ground truth for model training. Three network structures, including Unet, super-resolution residual network (SRResNet), and our modified asymmetric convolution-SRResNet (AC-SRResNet), were trained and evaluated using signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), edge preservation index (EPI) and computation time (CT). The effectiveness of these three trained models on OCT images of different samples and different systems was also investigated and confirmed. The SNR improvement for different sample images for L₂-loss-trained Unet, SRResNet, and AC-SRResNet are 20.83 dB, 24.88 dB, and 22.19 dB, respectively. The SNR improvement for public images from different system for L₁-loss-trained Unet, SRResNet, and AC-SRResNet are 19.36 dB, 20.11 dB, and 22.15 dB, respectively. AC-SRResNet and SRResNet demonstrate better denoising effect than Unet with longer computation time. AC-SRResNet demonstrates better edge preservation capability than SRResNet while Unet is close to AC-SRResNet. Eventually, we incorporated Unet, SRResNet, and AC-SRResNet into our graphic processing unit accelerated OCT imaging system for online noise reduction evaluation. Real-time noise reduction for OCT images with size of 512×512 pixels for Unet, SRResNet, and AC-SRResNet at 64 fps, 19 fps, and 17 fps were achieved respectively.

Application of Deep Learning for Diagnosing, Classifying, and Treating Age-Related Macular Degeneration

Age-related macular degeneration (AMD) affects nearly 200 million people and is the third leading cause of irreversible vision loss worldwide. Deep learning, a branch of artificial intelligence that can learn image recognition based on pre-existing datasets, creates an opportunity for more accurate and efficient diagnosis, classification, and treatment of AMD on both individual and population levels. Current algorithms based on fundus photography and optical coherence tomography imaging have already achieved diagnostic accuracy levels comparable to human graders. This accuracy can be further increased when deep learning algorithms are simultaneously applied to multiple diagnostic imaging modalities. Combined with advances in telemedicine and imaging technology, deep learning can enable large populations of patients to be screened than would otherwise be possible and allow ophthalmologists to focus on seeing those patients who are in need of treatment, thus reducing the number of patients with significant visual impairment from AMD.

Real-time tool to layer distance estimation for robotic subretinal injection using intraoperative 4D OCT

The emergence of robotics could enable ophthalmic microsurgical procedures that were previously not feasible due to the precision limits of manual delivery, for example, targeted subretinal injection. Determining the distance between the needle tip, the internal limiting membrane (ILM), and the retinal pigment epithelium (RPE) both precisely and reproducibly is required for safe and successful robotic retinal interventions. Recent advances in intraoperative optical coherence tomography (iOCT) have opened the path for 4D image-guided surgery by providing near video-rate imaging with micron-level resolution to visualize retinal structures, surgical instruments, and tool-tissue interactions. In this work, we present a novel pipeline to precisely estimate the distance between the injection needle and the surface boundaries of two retinal layers, the ILM and the RPE, from iOCT volumes. To achieve high computational efficiency, we reduce the analysis to the relevant area around the needle tip. We employ a convolutional neural network (CNN) to segment the tool surface, as well as the retinal layer boundaries from selected iOCT B-scans within this tip area. This results in the generation and processing of 3D surface point clouds for the tool, ILM and RPE from the B-scan segmentation maps, which in turn allows the estimation of the minimum distance between the resulting tool and layer point clouds. The proposed method is evaluated on iOCT volumes from ex-vivo porcine eyes and achieves an average error of 9.24 µm and 8.61 µm measuring the distance from the needle tip to the ILM and the RPE, respectively. The results demonstrate that this approach is robust to the high levels of noise present in iOCT B-scans and is suitable for the interventional use case by providing distance feedback at an average update rate of 15.66 Hz.

DeepRetina: Layer Segmentation of Retina in OCT Images Using Deep Learning

Purpose

To automate the segmentation of retinal layers, we propose DeepRetina, a method based on deep neural networks.

Methods

DeepRetina uses the improved Xception65 to extract and learn the characteristics of retinal layers. The Xception65-extracted feature maps are inputted to an atrous spatial pyramid pooling module to obtain multiscale feature information. This information is then recovered to capture clearer retinal layer boundaries in the encoder-decoder module, thus completing retinal layer auto-segmentation of the retinal optical coherence tomography (OCT) images.

Results

We validated this method using a retinal OCT image database containing 280 volumes (40 B-scans per volume) to demonstrate its effectiveness. The results showed that the method exhibits excellent performance in terms of the mean intersection over union and sensitivity (Se), which are as high as 90.41 and 92.15%, respectively. The intersection over union and Se values of the nerve fiber layer, ganglion cell layer, inner plexiform layer, inner nuclear layer, outer plexiform layer, outer nuclear layer, outer limiting membrane, photoreceptor inner segment, photoreceptor outer segment, and pigment epithelium layer were found to be above 88%.

Conclusions

DeepRetina can automate the segmentation of retinal layers and has great potential for the early diagnosis of fundus retinal diseases. In addition, our approach will provide a segmentation model framework for other types of tissues and cells in clinical practice.

Translational Relevance

Automating the segmentation of retinal layers can help effectively diagnose and monitor clinical retinal diseases. In addition, it requires only a small amount of manual segmentation, significantly improving work efficiency.

Extraction of Retinal Layers Through Convolution Neural Network (CNN) in an OCT Image for Glaucoma Diagnosis

Glaucoma is a progressive and deteriorating optic neuropathy that leads to visual field defects. The damage occurs as glaucoma is irreversible, so early and timely diagnosis is of significant importance. The proposed system employs the convolution neural network (CNN) for automatic segmentation of the retinal layers. The inner limiting membrane (ILM) and retinal pigmented epithelium (RPE) are used to calculate cup-to-disc ratio (CDR) for glaucoma diagnosis. The proposed system uses structure tensors to extract candidate layer pixels, and a patch across each candidate layer pixel is extracted, which is classified using CNN. The proposed framework is based upon VGG-16 architecture for feature extraction and classification of retinal layer pixels. The output feature map is merged into SoftMax layer for classification and produces probability map for central pixel of each patch and decides whether it is ILM, RPE, or background pixels. Graph search theory refines the extracted layers by interpolating the missing points, and these extracted ILM and RPE are finally used to compute CDR value and diagnose glaucoma. The proposed system is validated using a local dataset of optical coherence tomography images from 196 patients, including normal and glaucoma subjects. The dataset contains manually annotated ILM and RPE layers; manually extracted patches for ILM, RPE, and background pixels; CDR values; and eventually final finding related to glaucoma. The proposed system is able to extract ILM and RPE with a small absolute mean error of 6.03 and 5.56, respectively, and it finds CDR value within average range of ± 0.09 as compared with glaucoma expert. The proposed system achieves average sensitivity, specificity, and accuracies of 94.6, 94.07, and 94.68, respectively.

Approaches to quantify optical coherence tomography angiography metrics

Optical coherence tomography (OCT) has revolutionized the field of ophthalmology in the last three decades. As an OCT extension, OCT angiography (OCTA) utilizes a fast OCT system to detect motion contrast in ocular tissue and provides a three-dimensional representation of the ocular vasculature in a non-invasive, dye-free manner. The first OCT machine equipped with OCTA function was approved by U.S. Food and Drug Administration in 2016 and now it is widely applied in clinics. To date, numerous methods have been developed to aid OCTA interpretation and quantification. In this review, we focused on the workflow of OCTA-based interpretation, beginning from the generation of the OCTA images using signal decorrelation, which we divided into intensity-based, phase-based and phasor-based methods. We further discussed methods used to address image artifacts that are commonly observed in clinical settings, to the algorithms for image enhancement, binarization, and OCTA metrics extraction. We believe a better grasp of these technical aspects of OCTA will enhance the understanding of the technology and its potential application in disease diagnosis and management. Moreover, future studies will also explore the use of ocular OCTA as a window to link ocular vasculature to the function of other organs such as the kidney and brain.

Automated diagnoses of age-related macular degeneration and polypoidal choroidal vasculopathy using bi-modal deep convolutional neural networks

AIMS

To investigate the efficacy of a bi-modality deep convolutional neural network (DCNN) framework to categorise age-related macular degeneration (AMD) and polypoidal choroidal vasculopathy (PCV) from colour fundus images and optical coherence tomography (OCT) images.

METHODS

A retrospective cross-sectional study was proposed of patients with AMD or PCV who came to Peking Union Medical College Hospital. Diagnoses of all patients were confirmed by two retinal experts based on diagnostic gold standard for AMD and PCV. Patients with concurrent retinal vascular diseases were excluded. Colour fundus images and spectral domain OCT images were taken from dilated eyes of patients and healthy controls, and anonymised. All images were pre-labelled into normal, dry or wet AMD or PCV. ResNet-50 models were used as the backbone and alternate machine learning models including random forest classifiers were constructed for further comparison. For human-machine comparison, the same testing data set was diagnosed by three retinal experts independently. All images from the same participant were presented only within a single partition subset.

RESULTS

On a test set of 143 fundus and OCT image pairs from 80 eyes (20 eyes per-group), the bi-modal DCNN demonstrated the best performance, with accuracy 87.4%, sensitivity 88.8% and specificity 95.6%, and a perfect agreement with diagnostic gold standard (Cohen's κ 0.828), exceeds slightly over the best expert (Human1, Cohen's κ 0.810). For recognising PCV, the model outperformed the best expert as well.

CONCLUSION

A bi-modal DCNN for automated classification of AMD and PCV is accurate and promising in the realm of public health.

Correction propagation for user-assisted optical coherence tomography segmentation: general framework and application to Bruch's membrane segmentation

Optical coherence tomography (OCT) is a commonly used ophthalmic imaging modality. While OCT has traditionally been viewed cross-sectionally (i.e., as a sequence of B-scans), higher A-scan rates have increased interest in en face OCT visualization and analysis. The recent clinical introduction of OCT angiography (OCTA) has further spurred this interest, with chorioretinal OCTA being predominantly displayed via en face projections. Although en face visualization and quantitation are natural for many retinal features (e.g., drusen and vasculature), it requires segmentation. Because manual segmentation of volumetric OCT data is prohibitively laborious in many settings, there has been significant research and commercial interest in developing automatic segmentation algorithms. While these algorithms have achieved impressive results, the variability of image qualities and the variety of ocular pathologies cause even the most robust automatic segmentation algorithms to err. In this study, we develop a user-assisted segmentation approach, complementary to fully-automatic methods, wherein correction propagation is used to reduce the burden of manually correcting automatic segmentations. The approach is evaluated for Bruch's membrane segmentation in eyes with advanced age-related macular degeneration.

Multi-Task Joint Learning Model for Segmenting and Classifying Tongue Images Using a Deep Neural Network

Automatic tongue image segmentation and tongue image classification are two crucial tongue characterization tasks in traditional Chinese medicine (TCM). Due to the complexity of tongue segmentation and fine-grained traits of tongue image classification, both tasks are challenging. Fortunately, from the perspective of computer vision, these two tasks are highly interrelated, making them compatible with the idea of Multi-Task Joint learning (MTL). By sharing the underlying parameters and adding two different task loss functions, an MTL method for segmenting and classifying tongue images is proposed in this paper. Moreover, two state-of-the-art deep neural network variants (UNET and Discriminative Filter Learning (DFL)) are fused into the MTL to perform these two tasks. To the best of our knowledge, our method is the first attempt to manage both tasks simultaneously with MTL. We conducted extensive experiments with the proposed method. The experimental results show that our joint method outperforms the existing tongue characterization methods. Besides, visualizations and ablation studies are provided to aid in understanding our approach, which suggest that our method is highly consistent with human perception.

Application of machine learning in ophthalmic imaging modalities

In clinical ophthalmology, a variety of image-related diagnostic techniques have begun to offer unprecedented insights into eye diseases based on morphological datasets with millions of data points. Artificial intelligence (AI), inspired by the human multilayered neuronal system, has shown astonishing success within some visual and auditory recognition tasks. In these tasks, AI can analyze digital data in a comprehensive, rapid and non-invasive manner. Bioinformatics has become a focus particularly in the field of medical imaging, where it is driven by enhanced computing power and cloud storage, as well as utilization of novel algorithms and generation of data in massive quantities. Machine learning (ML) is an important branch in the field of AI. The overall potential of ML to automatically pinpoint, identify and grade pathological features in ocular diseases will empower ophthalmologists to provide high-quality diagnosis and facilitate personalized health care in the near future. This review offers perspectives on the origin, development, and applications of ML technology, particularly regarding its applications in ophthalmic imaging modalities.

Phase unwrapping based on a residual en-decoder network for phase images in Fourier domain Doppler optical coherence tomography

To solve the phase unwrapping problem for phase images in Fourier domain Doppler optical coherence tomography (DOCT), we propose a deep learning-based residual en-decoder network (REDN) method. In our approach, we reformulate the definition for obtaining the true phase as obtaining an integer multiple of 2π at each pixel by semantic segmentation. The proposed REDN architecture can provide recognition performance with pixel-level accuracy. To address the lack of phase images that are noise and wrapping free from DOCT systems for training, we used simulated images synthesized with DOCT phase image background noise features. An evaluation study on simulated images, DOCT phase images of phantom milk flowing in a plastic tube and a mouse artery, was performed. Meanwhile, a comparison study with recently proposed deep learning-based DeepLabV3+ and PhaseNet methods for signal phase unwrapping and traditional modified networking programming (MNP) method was also performed. Both visual inspection and quantitative metrical evaluation based on accuracy, specificity, sensitivity, root-mean-square-error, total-variation, and processing time demonstrate the robustness, effectiveness and superiority of our method. The proposed REDN method will benefit accurate and fast DOCT phase image-based diagnosis and evaluation when the detected phase is wrapped and will enrich the deep learning-based image processing platform for DOCT images.

Deep learning segmentation for optical coherence tomography measurements of the lower tear meniscus

The tear meniscus contains most of the tear fluid and therefore is a good indicator for the state of the tear film. Previously, we used a custom-built optical coherence tomography (OCT) system to study the lower tear meniscus by automatically segmenting the image data with a thresholding-based segmentation algorithm (TBSA). In this report, we investigate whether the results of this image segmentation algorithm are suitable to train a neural network in order to obtain similar or better segmentation results with shorter processing times. Considering the class imbalance problem, we compare two approaches, one directly segmenting the tear meniscus (DSA), the other first localizing the region of interest and then segmenting within the higher resolution image section (LSA). A total of 6658 images labeled by the TBSA were used to train deep convolutional neural networks with supervised learning. Five-fold cross-validation reveals a sensitivity of 96.36% and 96.43%, a specificity of 99.98% and 99.86% and a Jaccard index of 93.24% and 93.16% for the DSA and LSA, respectively. Average segmentation times are up to 228 times faster than the TBSA. Additionally, we report the behavior of the DSA and LSA in cases challenging for the TBSA and further test the applicability to measurements acquired with a commercially available OCT system. The application of deep learning for the segmentation of the tear meniscus provides a powerful tool for the assessment of the tear film, supporting studies for the investigation of the pathophysiology of dry eye-related diseases.

RedCap: residual encoder-decoder capsule network for holographic image reconstruction

A capsule network, as an advanced technique in deep learning, is designed to overcome information loss in the pooling operation and internal data representation of a convolutional neural network (CNN). It has shown promising results in several applications, such as digit recognition and image segmentation. In this work, we investigate for the first time the use of capsule network in digital holographic reconstruction. The proposed residual encoder-decoder capsule network, which we call RedCap, uses a novel windowed spatial dynamic routing algorithm and residual capsule block, which extends the idea of a residual block. Compared with the CNN-based neural network, RedCap exhibits much better experimental results in digital holographic reconstruction, while having a dramatic 75% reduction in the number of parameters. It indicates that RedCap is more efficient in the way it processes data and requires a much less memory storage for the learned model, which therefore makes it possible to be applied to some challenging situations with limited computational resources, such as portable devices.