End-to-End Multi-Task Learning Approaches for the Joint Epiretinal Membrane Segmentation and Screening in OCT Images

The Role of Artificial Intelligence in Epiretinal Membrane Care: A Scoping Review

Topic

In ophthalmology, artificial intelligence (AI) demonstrates potential in using ophthalmic imaging across diverse diseases, often matching ophthalmologists' performance. However, the range of machine learning models for epiretinal membrane (ERM) management, which differ in methodology, application, and performance, remains largely unsynthesized.

Clinical Relevance

Epiretinal membrane management relies on clinical evaluation and imaging, with surgical intervention considered in cases of significant impairment. AI analysis of ophthalmic images and clinical features could enhance ERM detection, characterization, and prognostication, potentially improving clinical decision-making. This scoping review aims to evaluate the methodologies, applications, and reported performance of AI models in ERM diagnosis, characterization, and prognostication.

Methods

A comprehensive literature search was conducted across 5 electronic databases including Ovid MEDLINE, EMBASE, Cochrane Central Register of Controlled Trials, Cochrane Database of Systematic Reviews, and Web of Science Core Collection from inception to November 14, 2024. Studies pertaining to AI algorithms in the context of ERM were included. The primary outcomes measured will be the reported design, application in ERM management, and performance of each AI model.

Results

Three hundred ninety articles were retrieved, with 33 studies meeting inclusion criteria. There were 30 studies (91%) reporting their training and validation methods. Altogether, 61 distinct AI models were included. OCT scans and fundus photographs were used in 26 (79%) and 7 (21%) papers, respectively. Supervised learning and both supervised and unsupervised learning were used in 32 (97%) and 1 (3%) studies, respectively. Twenty-seven studies (82%) developed or adapted AI models using images, whereas 5 (15%) had models using both images and clinical features, and 1 (3%) used preoperative and postoperative clinical features without ophthalmic images. Study objectives were categorized into 3 stages of ERM care. Twenty-three studies (70%) implemented AI for diagnosis (stage 1), 1 (3%) identified ERM characteristics (stage 2), and 6 (18%) predicted vision impairment after diagnosis or postoperative vision outcomes (stage 3). No articles studied treatment planning. Three studies (9%) used AI in stages 1 and 2. Of the 16 studies comparing AI performance to human graders (i.e., retinal specialists, general ophthalmologists, and trainees), 10 (63%) reported equivalent or higher performance.

Conclusion

Artificial intelligence-driven assessments of ophthalmic images and clinical features demonstrated high performance in detecting ERM, identifying its morphological properties, and predicting visual outcomes following ERM surgery. Future research might consider the validation of algorithms for clinical applications in personal treatment plan development, ideally to identify patients who might benefit most from surgery.

Financial Disclosures

The author(s) have no proprietary or commercial interest in any materials discussed in this article.

Multi-task global optimization-based method for vascular landmark detection

Vascular landmark detection plays an important role in medical analysis and clinical treatment. However, due to the complex topology and similar local appearance around landmarks, the popular heatmap regression based methods always suffer from the landmark confusion problem. Vascular landmarks are connected by vascular segments and have special spatial correlations, which can be utilized for performance improvement. In this paper, we propose a multi-task global optimization-based framework for accurate and automatic vascular landmark detection. A multi-task deep learning network is exploited to accomplish landmark heatmap regression, vascular semantic segmentation, and orientation field regression simultaneously. The two auxiliary objectives are highly correlated with the heatmap regression task and help the network incorporate the structural prior knowledge. During inference, instead of performing a max-voting strategy, we propose a global optimization-based post-processing method for final landmark decision. The spatial relationships between neighboring landmarks are utilized explicitly to tackle the landmark confusion problem. We evaluated our method on a cerebral MRA dataset with 564 volumes, a cerebral CTA dataset with 510 volumes, and an aorta CTA dataset with 50 volumes. The experiments demonstrate that the proposed method is effective for vascular landmark localization and achieves state-of-the-art performance.

Interpretable detection of epiretinal membrane from optical coherence tomography with deep neural networks

This study aimed to automatically detect epiretinal membranes (ERM) in various OCT-scans of the central and paracentral macula region and classify them by size using deep-neural-networks (DNNs). To this end, 11,061 OCT-images were included and graded according to the presence of an ERM and its size (small 100-1000 µm, large > 1000 µm). The data set was divided into training, validation and test sets (75%, 10%, 15% of the data, respectively). An ensemble of DNNs was trained and saliency maps were generated using Guided-Backprob. OCT-scans were also transformed into a one-dimensional-value using t-SNE analysis. The DNNs' receiver-operating-characteristics on the test set showed a high performance for no-ERM, small-ERM and large-ERM cases (AUC: 0.99, 0.92, 0.99, respectively; 3-way accuracy: 89%), with small-ERMs being the most difficult ones to detect. t-SNE analysis sorted cases by size and, in particular, revealed increased classification uncertainty at the transitions between groups. Saliency maps reliably highlighted ERM, regardless of the presence of other OCT features (i.e. retinal-thickening, intraretinal pseudo-cysts, epiretinal-proliferation) and entities such as ERM-retinoschisis, macular-pseudohole and lamellar-macular-hole. This study showed therefore that DNNs can reliably detect and grade ERMs according to their size not only in the fovea but also in the paracentral region. This is also achieved in cases of hard-to-detect, small-ERMs. In addition, the generated saliency maps can be used to highlight small-ERMs that might otherwise be missed. The proposed model could be used for screening-programs or decision-support-systems in the future.

Methods for real-time feature-guided image fusion of intrasurgical volumetric optical coherence tomography with digital microscopy

4D-microscope-integrated optical coherence tomography (4D-MIOCT) is an emergent multimodal imaging technology in which live volumetric OCT (4D-OCT) is implemented in tandem with standard stereo color microscopy. 4D-OCT provides ophthalmic surgeons with many useful visual cues not available in standard microscopy; however it is challenging for the surgeon to effectively integrate cues from simultaneous-but-separate imaging in real-time. In this work, we demonstrate progress towards solving this challenge via the fusion of data from each modality guided by segmented 3D features. In this way, a more readily interpretable visualization that combines and registers important cues from both modalities is presented to the surgeon.

Semi-supervised aortic dissections segmentation: A time-dependent weighted feedback fusion framework

The segmentation of true lumen (TL) and false lumen (FL) plays an important role in the diagnosis and treatment of aortic dissection (AD). Although the deep learning methods have achieved remarkable performance for this task, a large number of labeled data are required for training. In order to alleviate the burden of doctors' labeling, in this paper, a novel time-dependent weighted feedback fusion based semi-supervised aortic dissections segmentation framework is proposed by effectively leveraging the unlabeled data. A feedback network is additionally extended to encode the predicted output from the backbone network into high-level feature space, which is then fused with the original feature information of the image to fix previous potential mistakes, thereby segmentation accuracy is improved iteratively. To utilize both labeled data and unlabeled data, the fused feature space flows into the network again to generate the second feedback and make sure consistency with the previous one. The utilization of image feature space provides better robustness and accuracy for the proposed structure. Experiments show that our method outperforms five existing state-of-the-art semi-supervised segmentation methods on both a type-B AD dataset and a public dataset.

LOCTseg: A lightweight fully convolutional network for end-to-end optical coherence tomography segmentation

This article presents a novel end-to-end automatic solution for semantic segmentation of optical coherence tomography (OCT) images. OCT is a non-invasive imaging technology widely used in clinical practice due to its ability to acquire high-resolution cross-sectional images of the ocular fundus. Due to the large variability of the retinal structures, OCT segmentation is usually carried out manually and requires expert knowledge. This study introduces a novel fully convolutional network (FCN) architecture designated by LOCTSeg, for end-to-end automatic segmentation of diagnostic markers in OCT b-scans. LOCTSeg is a lightweight deep FCN optimized for balancing performance and efficiency. Unlike state-of-the-art FCNs used in image segmentation, LOCTSeg achieves competitive inference speed without sacrificing segmentation accuracy. The proposed LOCTSeg is evaluated on two publicly available benchmarking datasets: (1) annotated retinal OCT image database (AROI) comprising 1136 images, and (2) healthy controls and multiple sclerosis lesions (HCMS) consisting of 1715 images. Moreover, we evaluated the proposed LOCTSeg with a private dataset of 250 OCT b-scans acquired from epiretinal membrane (ERM) and healthy patients. Results of the evaluation demonstrate empirically the effectiveness of the proposed algorithm, which improves the state-of-the-art Dice score from 69% to 73% and from 91% to 92% on AROI and HCMS datasets, respectively. Furthermore, LOCTSeg outperforms comparable lightweight FCNs' Dice score by margins between 4% and 15% on ERM segmentation.