Image Projection Network: 3D to 2D Image Segmentation in OCTA Images

Encoding 3D information in 2D feature maps for brain CT-Angiography

We propose learnable 3D pooling (L3P), a CNN module designed to compress 3D information into 2D feature maps using anisotropic convolutions and unidirectional max pooling. Specifically, we used L3P followed by a 2D network to generate predictions from 3D brain CT-Angiography (CTA) in the context of large vessel occlusion (LVO). To further demonstrate its versatility, we extended its application to 3D brain MRI analysis for brain age prediction. First, we designed an experiment to classify the LVO-affected hemisphere (left or right), projecting the input CTA into the sagittal plane, which allowed to assess the ability of L3P to encode the 3D location where the location information was in the 3D-to-2D compression axis. Second, we evaluated the use of L3P on LVO detection as a binary classification (presence or absence). We compared the L3P models performance to that of 2D and stroke-specific 3D models. L3P models achieved results equivalent to stroke-specific 3D models while requiring fewer parameters and resources and provided better results than 2D models using maximum intensity projection images as input. The generalizability of L3P approach was evaluated on the LVO-affected hemisphere detection using data from a single site for training/validation and data from 36 other sites for testing, achieving an AUC of 0.83 on the test set. L3P also performed comparably or better than a fully 3D network on a brain age prediction task with a separate T1 MRI dataset, demonstrating its versatility across different tasks and imaging modalities. Additionally, L3P models generated more interpretable feature maps.

Retinal OCT image segmentation with deep learning: A review of advances, datasets, and evaluation metrics

Optical coherence tomography (OCT) is a widely used imaging technology in ophthalmic clinical practice, providing non-invasive access to high-resolution retinal images. Segmentation of anatomical structures and pathological lesions in retinal OCT images, directly impacts clinical decisions. While commercial OCT devices segment multiple retinal layers in healthy eyes, their performance degrades severely under pathological conditions. In recent years, the rapid advancements in deep learning have significantly driven research in OCT image segmentation. This review provides a comprehensive overview of the latest developments in deep learning-based segmentation methods for retinal OCT images. Additionally, it summarizes the medical significance, publicly available datasets, and commonly used evaluation metrics in this field. The review also discusses the current challenges faced by the research community and highlights potential future directions.

CMFNet: a cross-dimensional modal fusion network for accurate vessel segmentation based on OCTA data

Optical coherence tomography angiography (OCTA) is a novel non-invasive retinal vessel imaging technique that can display high-resolution 3D vessel structures. The quantitative analysis of retinal vessel morphology plays an important role in the automatic screening and diagnosis of fundus diseases. The existing segmentation methods struggle to effectively use the 3D volume data and 2D projection maps of OCTA images simultaneously, which leads to problems such as discontinuous microvessel segmentation results and deviation of morphological estimation. To enhance diagnostic support for fundus diseases, we propose a cross-dimensional modal fusion network (CMFNet) using both 3D volume data and 2D projection maps for accurate OCTA vessel segmentation. Firstly, we use different encoders to generate 2D projection features and 3D volume data features from projection maps and volume data, respectively. Secondly, we design an attentional cross-feature projection learning module to purify 3D volume data features and learn its projection features along the depth direction. Then, we develop a cross-dimensional hierarchical fusion module to effectively fuse coded features learned from the volume data and projection maps. In addition, we extract high-level semantic weight information and map it to the cross-dimensional hierarchical fusion process to enhance fusion performance. To validate the efficacy of our proposed method, we conducted experimental evaluations using the publicly available dataset: OCTA-500. The experimental results show that our method achieves state-of-the-art performance.

Physics-Based Optical Coherence Tomography Angiography (OCTA) Image Correction for Shadow Compensation

Optical coherence tomography (OCT) is being widely applied in clinical studies to investigate insight into the retina under the retinal pigment epithelium. Optical coherence tomography angiography (OCTA) is one of the functional extensions of OCT, for visualizing retinal circulation. Due to obstruction of light propagation, such as vitreous floaters or pupil boundaries, OCTA remains challenged by shadow artifacts that can disrupt volumetric data. Detecting and removing these shadow artifacts are crucial when quantifying indicators of retinal disease progression. We simplified an optical attenuation model of shadow formation in OCTA to a linear illumination transformation. And learn its parameters using an adversarial neural network. Our framework also consists of a sub-network for shadows automatic detection. We experimented our method on 28 OCTA images of normal eyes and compared the non-perfusion area (NPA), an index to measure retinal vascularity. The results showed that the NPA adjusted to a reasonable range after image processing using our method. Furthermore, we tested 150 OCTA images of synthesis artifacts, and the mean absolute error(MAE) values reached 0.83 after shadow removal.

Randomness-Restricted Diffusion Model for Ocular Surface Structure Segmentation

Ocular surface diseases affect a significant portion of the population worldwide. Accurate segmentation and quantification of different ocular surface structures are crucial for the understanding of these diseases and clinical decision-making. However, the automated segmentation of the ocular surface structure is relatively unexplored and faces several challenges. Ocular surface structure boundaries are often inconspicuous and obscured by glare from reflections. In addition, the segmentation of different ocular structures always requires training of multiple individual models. Thus, developing a one-model-fits-all segmentation approach is desirable. In this paper, we introduce a randomness-restricted diffusion model for multiple ocular surface structure segmentation. First, a time-controlled fusion-attention module (TFM) is proposed to dynamically adjust the information flow within the diffusion model, based on the temporal relationships between the network's input and time. TFM enables the network to effectively utilize image features to constrain the randomness of the generation process. We further propose a low-frequency consistency filter and a new loss to alleviate model uncertainty and error accumulation caused by the multi-step denoising process. Extensive experiments have shown that our approach can segment seven different ocular surface structures. Our method performs better than both dedicated ocular surface segmentation methods and general medical image segmentation methods. We further validated the proposed method over two clinical datasets, and the results demonstrated that it is beneficial to clinical applications, such as the meibomian gland dysfunction grading and aqueous deficient dry eye diagnosis.

Dual-branch image projection network for geographic atrophy segmentation in retinal OCT images

Existing geographic atrophy (GA) segmentation tasks can only use 3D data, ignoring the fact that a large number of B-scan images contain lesion information. In this work, we proposed a multistage dual-branch image projection network (DIPN) to learn feature information in B-scan images to assist GA segmentation. Considering that segmenting 3D data slices using a 2D network architecture ignores the neighboring information between volume data slices, we introduced ConvLSTM. In addition, to make the network focus on the attention in the projection direction to capture the contextual relationships, we proposed the projection attention module. Meanwhile, considering that the current projection network uses a unidirectional pooling operation to achieve feature projection, multi-scale features and channel information are ignored in the projection process. Therefore, we proposed an adaptive pooling module that aims to adaptively reduce feature dimensions when grasping multi-scale features and channel information. Finally, to mitigate the effect of image contrast on the network segmentation performance, we proposed a contrastive learning enhancement module (CLE). To validate the effectiveness of our proposed method, we conducted experiments on two different datasets. The segmentation results show that our method is more effective than other methods in the GA segmentation task and the foveal avascular zone (FAZ) segmentation task.

VSR-Net: Vessel-Like Structure Rehabilitation Network With Graph Clustering

The morphologies of vessel-like structures, such as blood vessels and nerve fibres, play significant roles in disease diagnosis, e.g., Parkinson's disease. Although deep network-based refinement segmentation and topology-preserving segmentation methods recently have achieved promising results in segmenting vessel-like structures, they still face two challenges: 1) existing methods often have limitations in rehabilitating subsection ruptures in segmented vessel-like structures; 2) they are typically overconfident in predicted segmentation results. To tackle these two challenges, this paper attempts to leverage the potential of spatial interconnection relationships among subsection ruptures from the structure rehabilitation perspective. Based on this perspective, we propose a novel Vessel-like Structure Rehabilitation Network (VSR-Net) to both rehabilitate subsection ruptures and improve the model calibration based on coarse vessel-like structure segmentation results. VSR-Net first constructs subsection rupture clusters via a Curvilinear Clustering Module (CCM). Then, the well-designed Curvilinear Merging Module (CMM) is applied to rehabilitate the subsection ruptures to obtain the refined vessel-like structures. Extensive experiments on six 2D/3D medical image datasets show that VSR-Net significantly outperforms state-of-the-art (SOTA) refinement segmentation methods with lower calibration errors. Additionally, we provide quantitative analysis to explain the morphological difference between the VSR-Net's rehabilitation results and ground truth (GT), which are smaller compared to those between SOTA methods and GT, demonstrating that our method more effectively rehabilitates vessel-like structures.

Boundary-Repairing Dual-Path Network for Retinal Layer Segmentation in OCT Image with Pigment Epithelial Detachment

Automatic retinal layer segmentation in optical coherence tomography (OCT) images is crucial for the diagnosis of ocular diseases. Currently, automatic retinal layer segmentation works well with normal OCT images. However, pigment epithelial detachment (PED) dramatically alters the retinal structure, causing blurred boundaries and partial disappearance of the Bruch's Membrane (BM), thus posing challenges to the segmentation. To tackle these problems, we propose a novel dual-path U-shaped network for simultaneous layer segmentation and boundary regression. This network first designs a feature interaction fusion (FIF) module to strengthen the boundary shape constraints in the layer path. To address the challenge posed by partial BM disappearance and boundary-blurring, we propose a layer boundary repair (LBR) module. This module aims to use contrastive loss to enhance the confidence of blurred boundary regions and refine the segmentation of layer boundaries through the re-prediction head. In addition, we introduce a novel bilateral threshold distance map (BTDM) designed for the boundary path. The BTDM serves to emphasize information within boundary regions. This map, combined with the updated probability map, culminates in topology-guaranteed segmentation results achieved through a topology correction (TC) module. We investigated the proposed network on two severely deformed datasets (i.e., OCTA-500 and Aier-PED) and one slightly deformed dataset (i.e., DUKE). The proposed method achieves an average Dice score of 94.26% on the OCTA-500 dataset, which was 1.5% higher than BAU-Net and outperformed other methods. In the DUKE and Aier-PED datasets, the proposed method achieved average Dice scores of 91.65% and 95.75%, respectively.

VDMNet: A Deep Learning Framework with Vessel Dynamic Convolution and Multi-Scale Fusion for Retinal Vessel Segmentation

Retinal vessel segmentation is crucial for diagnosing and monitoring ophthalmic and systemic diseases. Optical Coherence Tomography Angiography (OCTA) enables detailed imaging of the retinal microvasculature, but existing methods for OCTA segmentation face significant limitations, such as susceptibility to noise, difficulty in handling class imbalance, and challenges in accurately segmenting complex vascular morphologies. In this study, we propose VDMNet, a novel segmentation network designed to overcome these challenges by integrating several advanced components. Firstly, we introduce the Fast Multi-Head Self-Attention (FastMHSA) module to effectively capture both global and local features, enhancing the network's robustness against complex backgrounds and pathological interference. Secondly, the Vessel Dynamic Convolution (VDConv) module is designed to dynamically adapt to curved and crossing vessels, thereby improving the segmentation of complex morphologies. Furthermore, we employ the Multi-Scale Fusion (MSF) mechanism to aggregate features across multiple scales, enhancing the detection of fine vessels while maintaining vascular continuity. Finally, we propose Weighted Asymmetric Focal Tversky Loss (WAFT Loss) to address class imbalance issues, focusing on the accurate segmentation of small and difficult-to-detect vessels. The proposed framework was evaluated on the publicly available ROSE-1 and OCTA-3M datasets. Experimental results demonstrated that our model effectively preserved the edge information of tiny vessels and achieved state-of-the-art performance in retinal vessel segmentation across several evaluation metrics. These improvements highlight VDMNet's superior ability to capture both fine vascular details and overall vessel connectivity, making it a robust solution for retinal vessel segmentation.

AI-based 3D analysis of retinal vasculature associated with retinal diseases using OCT angiography

Retinal vasculature is the only vascular system in the human body that can be observed in a non-invasive manner, with a phenotype associated with a wide range of ocular, cerebral, and cardiovascular diseases. OCT and OCT angiography (OCTA) provide powerful imaging methods to visualize the three-dimensional morphological and functional information of the retina. In this study, based on OCT and OCTA multimodal inputs, a multitask convolutional neural network model was built to realize 3D segmentation of retinal blood vessels and disease classification for different retinal diseases, overcoming the limitations of existing methods that can only perform 2D analysis of OCTA. Two hundred thirty sets of OCT and OCTA data from 109 patients, including 138,000 cross-sectional images in normal and diseased eyes (age-related macular degeneration, retinal vein occlusion, and central serous chorioretinopathy), were collected from four commercial OCT systems for model training, validation, and testing. Experimental results verified that the proposed method was able to achieve a DICE coefficient of 0.956 for 3D segmentation of blood vessels and an accuracy of 91.49% for disease classification, and further enabled us to evaluate the 3D reconstruction of retinal vessels, explore the interlayer connections of superficial and deep vasculatures, and reveal the 3D quantitative vessel characteristics in different retinal diseases.

DIAS: A dataset and benchmark for intracranial artery segmentation in DSA sequences

The automated segmentation of Intracranial Arteries (IA) in Digital Subtraction Angiography (DSA) plays a crucial role in the quantification of vascular morphology, significantly contributing to computer-assisted stroke research and clinical practice. Current research primarily focuses on the segmentation of single-frame DSA using proprietary datasets. However, these methods face challenges due to the inherent limitation of single-frame DSA, which only partially displays vascular contrast, thereby hindering accurate vascular structure representation. In this work, we introduce DIAS, a dataset specifically developed for IA segmentation in DSA sequences. We establish a comprehensive benchmark for evaluating DIAS, covering full, weak, and semi-supervised segmentation methods. Specifically, we propose the vessel sequence segmentation network, in which the sequence feature extraction module effectively captures spatiotemporal representations of intravascular contrast, achieving intracranial artery segmentation in 2D+Time DSA sequences. For weakly-supervised IA segmentation, we propose a novel scribble learning-based image segmentation framework, which, under the guidance of scribble labels, employs cross pseudo-supervision and consistency regularization to improve the performance of the segmentation network. Furthermore, we introduce the random patch-based self-training framework, aimed at alleviating the performance constraints encountered in IA segmentation due to the limited availability of annotated DSA data. Our extensive experiments on the DIAS dataset demonstrate the effectiveness of these methods as potential baselines for future research and clinical applications. The dataset and code are publicly available at https://doi.org/10.5281/zenodo.11401368 and https://github.com/lseventeen/DIAS.

Diverse Data Generation for Retinal Layer Segmentation With Potential Structure Modeling

Accurate retinal layer segmentation on optical coherence tomography (OCT) images is hampered by the challenges of collecting OCT images with diverse pathological characterization and balanced distribution. Current generative models can produce high-realistic images and corresponding labels without quantitative limitations by fitting distributions of real collected data. Nevertheless, the diversity of their generated data is still limited due to the inherent imbalance of training data. To address these issues, we propose an image-label pair generation framework that generates diverse and balanced potential data from imbalanced real samples. Specifically, the framework first generates diverse layer masks, and then generates plausible OCT images corresponding to these layer masks using two customized diffusion probabilistic models respectively. To learn from imbalanced data and facilitate balanced generation, we introduce pathological-related conditions to guide the generation processes. To enhance the diversity of the generated image-label pairs, we propose a potential structure modeling technique that transfers the knowledge of diverse sub-structures from lowly- or non-pathological samples to highly pathological samples. We conducted extensive experiments on two public datasets for retinal layer segmentation. Firstly, our method generates OCT images with higher image quality and diversity compared to other generative methods. Furthermore, based on the extensive training with the generated OCT images, downstream retinal layer segmentation tasks demonstrate improved results. The code is publicly available at: https://github.com/nicetomeetu21/GenPSM.

Medical image registration and its application in retinal images: a review

Medical image registration is vital for disease diagnosis and treatment with its ability to merge diverse information of images, which may be captured under different times, angles, or modalities. Although several surveys have reviewed the development of medical image registration, they have not systematically summarized the existing medical image registration methods. To this end, a comprehensive review of these methods is provided from traditional and deep-learning-based perspectives, aiming to help audiences quickly understand the development of medical image registration. In particular, we review recent advances in retinal image registration, which has not attracted much attention. In addition, current challenges in retinal image registration are discussed and insights and prospects for future research provided.

The Framework of Quantifying Biomarkers of OCT and OCTA Images in Retinal Diseases

Despite the significant advancements facilitated by previous research in introducing a plethora of retinal biomarkers, there is a lack of research addressing the clinical need for quantifying different biomarkers and prioritizing their importance for guiding clinical decision making in the context of retinal diseases. To address this issue, our study introduces a novel framework for quantifying biomarkers derived from optical coherence tomography (OCT) and optical coherence tomography angiography (OCTA) images in retinal diseases. We extract 452 feature parameters from five feature types, including local binary patterns (LBP) features of OCT and OCTA, capillary and large vessel features, and the foveal avascular zone (FAZ) feature. Leveraging this extensive feature set, we construct a classification model using a statistically relevant p value for feature selection to predict retinal diseases. We obtain a high accuracy of 0.912 and F1-score of 0.906 in the task of disease classification using this framework. We find that OCT and OCTA's LBP features provide a significant contribution of 77.12% to the significance of biomarkers in predicting retinal diseases, suggesting their potential as latent indicators for clinical diagnosis. This study employs a quantitative analysis framework to identify potential biomarkers for retinal diseases in OCT and OCTA images. Our findings suggest that LBP parameters, skewness and kurtosis values of capillary, the maximum, mean, median, and standard deviation of large vessel, as well as the eccentricity, compactness, flatness, and anisotropy index of FAZ, may serve as significant indicators of retinal conditions.

Soul: An OCTA dataset based on Human Machine Collaborative Annotation Framework

Branch retinal vein occlusion (BRVO) is the most prevalent retinal vascular disease that constitutes a threat to vision due to increased venous pressure caused by venous effluent in the space, leading to impaired visual function. Optical Coherence Tomography Angiography (OCTA) is an innovative non-invasive technique that offers high-resolution three-dimensional structures of retinal blood vessels. Most publicly available datasets are collected from single visits with different patients, encompassing various eye diseases for distinct tasks and areas. Moreover, due to the intricate nature of eye structure, professional labeling not only relies on the expertise of doctors but also demands considerable time and effort. Therefore, we have developed a BRVO-focused dataset named Soul (Source of ocular vascular) and propose a human machine collaborative annotation framework (HMCAF) using scrambled retinal blood vessels data. Soul is categorized into 6 subsets based on injection frequency and follow-up duration. The dataset comprises original images, corresponding blood vessel labels, and clinical text information sheets which can be effectively utilized when combined with machine learning.

Exploring autism via the retina: Comparative insights in children with autism spectrum disorder and typical development

Autism spectrum disorder (ASD) is a widely recognized neurodevelopmental disorder, yet the identification of reliable imaging biomarkers for its early diagnosis remains a challenge. Considering the specific manifestations of ASD in the eyes and the interconnectivity between the brain and the eyes, this study investigates ASD through the lens of retinal analysis. We specifically examined differences in the macular region of the retina using optical coherence tomography (OCT)/optical coherence tomography angiography (OCTA) images between children diagnosed with ASD and those with typical development (TD). Our findings present potential novel characteristics of ASD: the thickness of the ellipsoid zone (EZ) with cone photoreceptors was significantly increased in ASD; the large-caliber arteriovenous of the inner retina was significantly reduced in ASD; these changes in the EZ and arteriovenous were more significant in the left eye than in the right eye. These observations of photoreceptor alterations, vascular function changes, and lateralization phenomena in ASD warrant further investigation, and we hope that this work can advance interdisciplinary understanding of ASD.

Robust Ultrafast Projection Pipeline for Structural and Angiography Imaging of Fourier-Domain Optical Coherence Tomography

The current methods to generate projections for structural and angiography imaging of Fourier-Domain optical coherence tomography (FD-OCT) are significantly slow for prediagnosis improvement, prognosis, real-time surgery guidance, treatments, and lesion boundary definition. This study introduced a robust ultrafast projection pipeline (RUPP) and aimed to develop and evaluate the efficacy of RUPP. RUPP processes raw interference signals to generate structural projections without the need for Fourier Transform. Various angiography reconstruction algorithms were utilized for efficient projections. Traditional methods were compared to RUPP using PSNR, SSIM, and processing time as evaluation metrics. The study used 22 datasets (hand skin: 9; labial mucosa: 13) from 8 volunteers, acquired with a swept-source optical coherence tomography system. RUPP significantly outperformed traditional methods in processing time, requiring only 0.040 s for structural projections, which is 27 times faster than traditional summation projections. For angiography projections, the best RUPP variation took 0.15 s, making it 7518 times faster than the windowed eigen decomposition method. However, PSNR decreased by 41-45% and SSIM saw reductions of 25-74%. RUPP demonstrated remarkable speed improvements over traditional methods, indicating its potential for real-time structural and angiography projections in FD-OCT, thereby enhancing clinical prediagnosis, prognosis, surgery guidance, and treatment efficacy.

OCTA-based AMD Stage Grading Enhancement via Class-Conditioned Style Transfer

Optical Coherence Tomography Angiography (OCTA) is a promising diagnostic tool for age-related macular degeneration (AMD), providing non-invasive visualization of sub-retinal vascular networks. This research explores the effectiveness of deep neural network (DNN) classifiers trained exclusively on OCTA images for AMD diagnosis. To address the challenge of limited data, we combine OCTA data from two instruments-Heidelberg and Optovue-and leverage style transfer technique, CycleGAN, to convert samples between these domains. This strategy introduces additional content into each domain, enriching the training dataset and improving classification accuracy. To enhance the CycleGAN for downstream classification tasks, we propose integrating class-related constraints during training, which can be implemented in either supervised or unsupervised manner with a pretrained classifier. The experimental results demonstrate that the proposed class-conditioned CycleGAN is effective and elevates DNN classification accuracy in both OCTA domains.

Deep local-to-global feature learning for medical image super-resolution

Medical images play a vital role in medical analysis by providing crucial information about patients' pathological conditions. However, the quality of these images can be compromised by many factors, such as limited resolution of the instruments, artifacts caused by movements, and the complexity of the scanned areas. As a result, low-resolution (LR) images cannot provide sufficient information for diagnosis. To address this issue, researchers have attempted to apply image super-resolution (SR) techniques to restore the high-resolution (HR) images from their LR counterparts. However, these techniques are designed for generic images, and thus suffer from many challenges unique to medical images. An obvious one is the diversity of the scanned objects; for example, the organs, tissues, and vessels typically appear in different sizes and shapes, and are thus hard to restore with standard convolution neural networks (CNNs). In this paper, we develop a dynamic-local learning framework to capture the details of these diverse areas, consisting of deformable convolutions with adjustable kernel shapes. Moreover, the global information between the tissues and organs is vital for medical diagnosis. To preserve global information, we propose pixel-pixel and patch-patch global learning using a non-local mechanism and a vision transformer (ViT), respectively. The result is a novel CNN-ViT neural network with Local-to-Global feature learning for medical image SR, referred to as LGSR, which can accurately restore both local details and global information. We evaluate our method on six public datasets and one large-scale private dataset, which include five different types of medical images (i.e., Ultrasound, OCT, Endoscope, CT, and MRI images). Experiments show that the proposed method achieves superior PSNR/SSIM and visual performance than the state of the arts with competitive computational costs, measured in network parameters, runtime, and FLOPs. What is more, the experiment conducted on OCT image segmentation for the downstream task demonstrates a significantly positive performance effect of LGSR.

Involving logical clinical knowledge into deep neural networks to improve bladder tumor segmentation

Segmentation of bladder tumors from medical radiographic images is of great significance for early detection, diagnosis and prognosis evaluation of bladder cancer. Deep Convolution Neural Networks (DCNNs) have been successfully used for bladder tumor segmentation, but the segmentation based on DCNN is data-hungry for model training and ignores clinical knowledge. From the clinical view, bladder tumors originate from the mucosal surface of bladder and must rely on the bladder wall to survive and grow. This clinical knowledge of tumor location is helpful to improve the bladder tumor segmentation. To achieve this, we propose a novel bladder tumor segmentation method, which incorporates the clinical logic rules of bladder tumor and bladder wall into DCNNs to harness the tumor segmentation. Clinical logical rules provide a semantic and human-readable knowledge representation and are easy for knowledge acquisition from clinicians. In addition, incorporating logical rules of clinical knowledge helps to reduce the data dependency of the segmentation network, and enables precise segmentation results even with limited number of annotated images. Experiments on bladder MR images collected from the collaborating hospital validate the effectiveness of the proposed bladder tumor segmentation method.

Domain generalization for retinal vessel segmentation via Hessian-based vector field

Blessed by vast amounts of data, learning-based methods have achieved remarkable performance in countless tasks in computer vision and medical image analysis. Although these deep models can simulate highly nonlinear mapping functions, they are not robust with regard to the domain shift of input data. This is a significant concern that impedes the large-scale deployment of deep models in medical images since they have inherent variation in data distribution due to the lack of imaging standardization. Therefore, researchers have explored many domain generalization (DG) methods to alleviate this problem. In this work, we introduce a Hessian-based vector field that can effectively model the tubular shape of vessels, which is an invariant feature for data across various distributions. The vector field serves as a good embedding feature to take advantage of the self-attention mechanism in a vision transformer. We design paralleled transformer blocks that stress the local features with different scales. Furthermore, we present a novel data augmentation method that introduces perturbations in image style while the vessel structure remains unchanged. In experiments conducted on public datasets of different modalities, we show that our model achieves superior generalizability compared with the existing algorithms. Our code and trained model are publicly available at https://github.com/MedICL-VU/Vector-Field-Transformer.

Toward Ground-Truth Optical Coherence Tomography via Three-Dimensional Unsupervised Deep Learning Processing and Data

Optical coherence tomography (OCT) can perform non-invasive high-resolution three-dimensional (3D) imaging and has been widely used in biomedical fields, while it is inevitably affected by coherence speckle noise which degrades OCT imaging performance and restricts its applications. Here we present a novel speckle-free OCT imaging strategy, named toward-ground-truth OCT ( t GT-OCT), that utilizes unsupervised 3D deep-learning processing and leverages OCT 3D imaging features to achieve speckle-free OCT imaging. Specifically, our proposed t GT-OCT utilizes an unsupervised 3D-convolution deep-learning network trained using random 3D volumetric data to distinguish and separate speckle from real structures in 3D imaging volumetric space; moreover, t GT-OCT effectively further reduces speckle noise and reveals structures that would otherwise be obscured by speckle noise while preserving spatial resolution. Results derived from different samples demonstrated the high-quality speckle-free 3D imaging performance of t GT-OCT and its advancement beyond the previous state-of-the-art. The code is available online: https://github.com/Voluntino/tGT-OCT.

MEMO: dataset and methods for robust multimodal retinal image registration with large or small vessel density differences

The measurement of retinal blood flow (RBF) in capillaries can provide a powerful biomarker for the early diagnosis and treatment of ocular diseases. However, no single modality can determine capillary flowrates with high precision. Combining erythrocyte-mediated angiography (EMA) with optical coherence tomography angiography (OCTA) has the potential to achieve this goal, as EMA can measure the absolute RBF of retinal microvasculature and OCTA can provide the structural images of capillaries. However, multimodal retinal image registration between these two modalities remains largely unexplored. To fill this gap, we establish MEMO, the first public multimodal EMA and OCTA retinal image dataset. A unique challenge in multimodal retinal image registration between these modalities is the relatively large difference in vessel density (VD). To address this challenge, we propose a segmentation-based deep-learning framework (VDD-Reg), which provides robust results despite differences in vessel density. VDD-Reg consists of a vessel segmentation module and a registration module. To train the vessel segmentation module, we further designed a two-stage semi-supervised learning framework (LVD-Seg) combining supervised and unsupervised losses. We demonstrate that VDD-Reg outperforms existing methods quantitatively and qualitatively for cases of both small VD differences (using the CF-FA dataset) and large VD differences (using our MEMO dataset). Moreover, VDD-Reg requires as few as three annotated vessel segmentation masks to maintain its accuracy, demonstrating its feasibility.

Deep Multimodal Fusion of Data With Heterogeneous Dimensionality via Projective Networks

The use of multimodal imaging has led to significant improvements in the diagnosis and treatment of many diseases. Similar to clinical practice, some works have demonstrated the benefits of multimodal fusion for automatic segmentation and classification using deep learning-based methods. However, current segmentation methods are limited to fusion of modalities with the same dimensionality (e.g., 3D + 3D, 2D + 2D), which is not always possible, and the fusion strategies implemented by classification methods are incompatible with localization tasks. In this work, we propose a novel deep learning-based framework for the fusion of multimodal data with heterogeneous dimensionality (e.g., 3D + 2D) that is compatible with localization tasks. The proposed framework extracts the features of the different modalities and projects them into the common feature subspace. The projected features are then fused and further processed to obtain the final prediction. The framework was validated on the following tasks: segmentation of geographic atrophy (GA), a late-stage manifestation of age-related macular degeneration, and segmentation of retinal blood vessels (RBV) in multimodal retinal imaging. Our results show that the proposed method outperforms the state-of-the-art monomodal methods on GA and RBV segmentation by up to 3.10% and 4.64% Dice, respectively.

OCTA-500: A retinal dataset for optical coherence tomography angiography study

Optical coherence tomography angiography (OCTA) is a novel imaging modality that has been widely utilized in ophthalmology and neuroscience studies to observe retinal vessels and microvascular systems. However, publicly available OCTA datasets remain scarce. In this paper, we introduce the largest and most comprehensive OCTA dataset dubbed OCTA-500, which contains OCTA imaging under two fields of view (FOVs) from 500 subjects. The dataset provides rich images and annotations including two modalities (OCT/OCTA volumes), six types of projections, four types of text labels (age/gender/eye/disease) and seven types of segmentation labels (large vessel/capillary/artery/vein/2D FAZ/3D FAZ/retinal layers). Then, we propose a multi-object segmentation task called CAVF, which integrates capillary segmentation, artery segmentation, vein segmentation, and FAZ segmentation under a unified framework. In addition, we optimize the 3D-to-2D image projection network (IPN) to IPN-V2 to serve as one of the segmentation baselines. Experimental results demonstrate that IPN-V2 achieves an about 10% mIoU improvement over IPN on CAVF task. Finally, we further study the impact of several dataset characteristics: the training set size, the model input (OCT/OCTA, 3D volume/2D projection), the baseline networks, and the diseases. The dataset and code are publicly available at: https://ieee-dataport.org/open-access/octa-500.

DISCOVER: 2-D multiview summarization of Optical Coherence Tomography Angiography for automatic diabetic retinopathy diagnosis

Diabetic Retinopathy (DR), an ocular complication of diabetes, is a leading cause of blindness worldwide. Traditionally, DR is monitored using Color Fundus Photography (CFP), a widespread 2-D imaging modality. However, DR classifications based on CFP have poor predictive power, resulting in suboptimal DR management. Optical Coherence Tomography Angiography (OCTA) is a recent 3-D imaging modality offering enhanced structural and functional information (blood flow) with a wider field of view. This paper investigates automatic DR severity assessment using 3-D OCTA. A straightforward solution to this task is a 3-D neural network classifier. However, 3-D architectures have numerous parameters and typically require many training samples. A lighter solution consists in using 2-D neural network classifiers processing 2-D en-face (or frontal) projections and/or 2-D cross-sectional slices. Such an approach mimics the way ophthalmologists analyze OCTA acquisitions: (1) en-face flow maps are often used to detect avascular zones and neovascularization, and (2) cross-sectional slices are commonly analyzed to detect macular edemas, for instance. However, arbitrary data reduction or selection might result in information loss. Two complementary strategies are thus proposed to optimally summarize OCTA volumes with 2-D images: (1) a parametric en-face projection optimized through deep learning and (2) a cross-sectional slice selection process controlled through gradient-based attribution. The full summarization and DR classification pipeline is trained from end to end. The automatic 2-D summary can be displayed in a viewer or printed in a report to support the decision. We show that the proposed 2-D summarization and classification pipeline outperforms direct 3-D classification with the advantage of improved interpretability.

LA-Net: layer attention network for 3D-to-2D retinal vessel segmentation in OCTA images

Objective.Retinal vessel segmentation from optical coherence tomography angiography (OCTA) volumes is significant in analyzing blood supply structures and the diagnosing ophthalmic diseases. However, accurate retinal vessel segmentation in 3D OCTA remains challenging due to the interference of choroidal blood flow signals and the variations in retinal vessel structure.Approach.This paper proposes a layer attention network (LA-Net) for 3D-to-2D retinal vessel segmentation. The network comprises a 3D projection path and a 2D segmentation path. The key component in the 3D path is the proposed multi-scale layer attention module, which effectively learns the layer features of OCT and OCTA to attend to the retinal vessel layer while suppressing the choroidal vessel layer. This module also efficiently captures 3D multi-scale information for improved semantic understanding during projection. In the 2D path, a reverse boundary attention module is introduced to explore and preserve boundary and shape features of retinal vessels by focusing on non-salient regions in deep features.Main results.Experimental results in two subsets of the OCTA-500 dataset showed that our method achieves advanced segmentation performance with Dice similarity coefficients of 93.04% and 89.74%, respectively.Significance.The proposed network provides reliable 3D-to-2D segmentation of retinal vessels, with potential for application in various segmentation tasks that involve projecting the input image. Implementation code:https://github.com/y8421036/LA-Net.

Applications of machine learning in time-domain fluorescence lifetime imaging: a review

Many medical imaging modalities have benefited from recent advances in Machine Learning (ML), specifically in deep learning, such as neural networks. Computers can be trained to investigate and enhance medical imaging methods without using valuable human resources. In recent years, Fluorescence Lifetime Imaging (FLIm) has received increasing attention from the ML community. FLIm goes beyond conventional spectral imaging, providing additional lifetime information, and could lead to optical histopathology supporting real-time diagnostics. However, most current studies do not use the full potential of machine/deep learning models. As a developing image modality, FLIm data are not easily obtainable, which, coupled with an absence of standardisation, is pushing back the research to develop models which could advance automated diagnosis and help promote FLIm. In this paper, we describe recent developments that improve FLIm image quality, specifically time-domain systems, and we summarise sensing, signal-to-noise analysis and the advances in registration and low-level tracking. We review the two main applications of ML for FLIm: lifetime estimation and image analysis through classification and segmentation. We suggest a course of action to improve the quality of ML studies applied to FLIm. Our final goal is to promote FLIm and attract more ML practitioners to explore the potential of lifetime imaging.

Segmentation of Low-Light Optical Coherence Tomography Angiography Images under the Constraints of Vascular Network Topology

Optical coherence tomography angiography (OCTA) offers critical insights into the retinal vascular system, yet its full potential is hindered by challenges in precise image segmentation. Current methodologies struggle with imaging artifacts and clarity issues, particularly under low-light conditions and when using various high-speed CMOS sensors. These challenges are particularly pronounced when diagnosing and classifying diseases such as branch vein occlusion (BVO). To address these issues, we have developed a novel network based on topological structure generation, which transitions from superficial to deep retinal layers to enhance OCTA segmentation accuracy. Our approach not only demonstrates improved performance through qualitative visual comparisons and quantitative metric analyses but also effectively mitigates artifacts caused by low-light OCTA, resulting in reduced noise and enhanced clarity of the images. Furthermore, our system introduces a structured methodology for classifying BVO diseases, bridging a critical gap in this field. The primary aim of these advancements is to elevate the quality of OCTA images and bolster the reliability of their segmentation. Initial evaluations suggest that our method holds promise for establishing robust, fine-grained standards in OCTA vascular segmentation and analysis.

Artificial intelligence and hemodynamic studies in optical coherence tomography angiography for diabetic retinopathy evaluation: A review

Diabetic retinopathy (DR) is a rapidly emerging retinal abnormality worldwide, which can cause significant vision loss by disrupting the vascular structure in the retina. Recently, optical coherence tomography angiography (OCTA) has emerged as an effective imaging tool for diagnosing and monitoring DR. OCTA produces high-quality 3-dimensional images and provides deeper visualization of retinal vessel capillaries and plexuses. The clinical relevance of OCTA in detecting, classifying, and planning therapeutic procedures for DR patients has been highlighted in various studies. Quantitative indicators obtained from OCTA, such as blood vessel segmentation of the retina, foveal avascular zone (FAZ) extraction, retinal blood vessel density, blood velocity, flow rate, capillary vessel pressure, and retinal oxygen extraction, have been identified as crucial hemodynamic features for screening DR using computer-aided systems in artificial intelligence (AI). AI has the potential to assist physicians and ophthalmologists in developing new treatment options. In this review, we explore how OCTA has impacted the future of DR screening and early diagnosis. It also focuses on how analysis methods have evolved over time in clinical trials. The future of OCTA imaging and its continued use in AI-assisted analysis is promising and will undoubtedly enhance the clinical management of DR.

Visualization and quantization of 3D retinal vessels in OCTA images

Optical coherence tomography angiography (OCTA) has been increasingly used in the analysis of ophthalmic diseases in recent years. Automatic vessel segmentation in 2D OCTA projection images is commonly used in clinical practice. However, OCTA provides a 3D volume of the retinal blood vessels with rich spatial distribution information, and it is incomplete to segment retinal vessels only in 2D projection images. Here, considering that it is difficult to manually label 3D vessels, we introduce a 3D vessel segmentation and reconstruction method for OCTA images with only 2D vessel labels. We implemented 3D vessel segmentation in the OCTA volume using a specially trained 2D vessel segmentation model. The 3D vessel segmentation results are further used to calculate 3D vessel parameters and perform 3D reconstruction. The experimental results on the public dataset OCTA-500 demonstrate that 3D vessel parameters have higher sensitivity to vascular alteration than 2D vessel parameters, which makes it meaningful for clinical analysis. The 3D vessel reconstruction provides vascular visualization in different retinal layers that can be used to monitor the development of retinal diseases. Finally, we also illustrate the use of 3D reconstruction results to determine the relationship between the location of arteries and veins.

A Multi-Scale Fusion and Transformer Based Registration Guided Speckle Noise Reduction for OCT Images

Optical coherence tomography (OCT) images are inevitably affected by speckle noise because OCT is based on low-coherence interference. Multi-frame averaging is one of the effective methods to reduce speckle noise. Before averaging, the misalignment between images must be calibrated. In this paper, in order to reduce misalignment between images caused during the acquisition, a novel multi-scale fusion and Transformer based (MsFTMorph) method is proposed for deformable retinal OCT image registration. The proposed method captures global connectivity and locality with convolutional vision transformer and also incorporates a multi-resolution fusion strategy for learning the global affine transformation. Comparative experiments with other state-of-the-art registration methods demonstrate that the proposed method achieves higher registration accuracy. Guided by the registration, subsequent multi-frame averaging shows better results in speckle noise reduction. The noise is suppressed while the edges can be preserved. In addition, our proposed method has strong cross-domain generalization, which can be directly applied to images acquired by different scanners with different modes.

Innovative Macula Capillaries Plexuses Visualization with OCTA B-Scan Graph Representation: Transforming OCTA B-Scan into OCTA Graph Representation

Purpose

The aim of this study is to transform optical coherence tomography angiography (OCTA) scans into innovative OCTA graphs, serving as novel biomarkers representing the macular vasculature.

Patients and Methods

The study included 90 healthy subjects and 39 subjects with various abnormalities (29 with diabetic retinopathy, 5 with age-related macular degeneration, and 5 with choroid neovascularization). OCTA 5µm macular coronal views (MCVs) were generated for each subject, followed by blood vessel segmentation and skeleton processing. Subsequently, the blood vessel density index, blood vessel skeleton index, and blood vessel tortuosity index were computed. The graphs of each metric were plotted against the axial axes of the OCTA B-scan, representing the integrity of vasculature at successive 5µm macular depths.

Results

The results revealed two significant findings. First, the B-scans from OCTA can be transformed into OCTA graphs, yielding three specific OCTA graphs in this study. These graphs provide new biomarkers for assessing the integrity of deep vascular complex (DVC) and superficial vascular complex (SVC) within the macula. Second, a statistically significant difference was observed between normal (n=90) and abnormal (n=39) subjects, with a t-test p-value significantly lower than 0.001. The Mann-Whitney u-test also yielded significant difference but only between the 90 normal and 29 DR subjects.

Conclusion

The novel OCTA graphs offer a unique representation of the macula's SVC and DVC, suggesting their potential in aiding physicians in the diagnosis of eye health within OCTA clinics. Further research is warranted to finalize the shape of these newly derived OCTA graphs and establish their clinical relevance and utility.

Image-based insilico investigation of hemodynamics and biomechanics in healthy and diabetic human retinas

Retinal hemodynamics and biomechanics play a significant role in understanding the pathophysiology of several ocular diseases. However, these parameters are significantly affected due to changed blood vessel morphology ascribed to pathological conditions, particularly diabetes. In this study, an image-based computational fluid dynamics (CFD) model is applied to examine the effects of changed vascular morphology due to diabetes on blood flow velocity, vorticity, wall shear stress (WSS), and oxygen distribution and compare it with healthy. The 3D patient-specific vascular architecture of diabetic and healthy retina is extracted from Optical Coherence Tomography Angiography (OCTA) images and fundus to extract the capillary level information. Further, Fluid-structure interaction (FSI) simulations have been performed to compare the induced tissue stresses in diabetic and healthy conditions. Results illustrate that most arterioles possess higher velocity, vorticity, WSS, and lesser oxygen concentration than arteries for healthy and diabetic cases. However, an opposite trend is observed for venules and veins. Comparisons show that, on average, the blood flow velocity in the healthy case decreases by 42 % in arteries and 21 % in veins, respectively, compared to diabetic. In addition, the WSS and von Mises stress (VMS) in healthy case decrease by 49 % and 72 % in arteries and by 6 % and 28 % in veins, respectively, when compared with diabetic, making diabetic blood vessels more susceptible to wall rupture and tissue damage. The in-silico results may help predict the possible abnormalities region early, helping the ophthalmologists use these estimates as prognostic tools and tailor patient-specific treatment plans.

Microsurgery Robots: Applications, Design, and Development

Microsurgical techniques have been widely utilized in various surgical specialties, such as ophthalmology, neurosurgery, and otolaryngology, which require intricate and precise surgical tool manipulation on a small scale. In microsurgery, operations on delicate vessels or tissues require high standards in surgeons' skills. This exceptionally high requirement in skills leads to a steep learning curve and lengthy training before the surgeons can perform microsurgical procedures with quality outcomes. The microsurgery robot (MSR), which can improve surgeons' operation skills through various functions, has received extensive research attention in the past three decades. There have been many review papers summarizing the research on MSR for specific surgical specialties. However, an in-depth review of the relevant technologies used in MSR systems is limited in the literature. This review details the technical challenges in microsurgery, and systematically summarizes the key technologies in MSR with a developmental perspective from the basic structural mechanism design, to the perception and human-machine interaction methods, and further to the ability in achieving a certain level of autonomy. By presenting and comparing the methods and technologies in this cutting-edge research, this paper aims to provide readers with a comprehensive understanding of the current state of MSR research and identify potential directions for future development in MSR.

Federated learning for diagnosis of age-related macular degeneration

This paper presents a federated learning (FL) approach to train deep learning models for classifying age-related macular degeneration (AMD) using optical coherence tomography image data. We employ the use of residual network and vision transformer encoders for the normal vs. AMD binary classification, integrating four unique domain adaptation techniques to address domain shift issues caused by heterogeneous data distribution in different institutions. Experimental results indicate that FL strategies can achieve competitive performance similar to centralized models even though each local model has access to a portion of the training data. Notably, the Adaptive Personalization FL strategy stood out in our FL evaluations, consistently delivering high performance across all tests due to its additional local model. Furthermore, the study provides valuable insights into the efficacy of simpler architectures in image classification tasks, particularly in scenarios where data privacy and decentralization are critical using both encoders. It suggests future exploration into deeper models and other FL strategies for a more nuanced understanding of these models' performance. Data and code are available at https://github.com/QIAIUNCC/FL_UNCC_QIAI.

Robust AMD Stage Grading with Exclusively OCTA Modality Leveraging 3D Volume

Age-related Macular Degeneration (AMD) is a degenerative eye disease that causes central vision loss. Optical Coherence Tomography Angiography (OCTA) is an emerging imaging modality that aids in the diagnosis of AMD by displaying the pathogenic vessels in the subretinal space. In this paper, we investigate the effectiveness of OCTA from the view of deep classifiers. To the best of our knowledge, this is the first study that solely uses OCTA for AMD stage grading. By developing a 2D classifier based on OCTA projections, we identify that segmentation errors in retinal layers significantly affect the accuracy of classification. To address this issue, we propose analyzing 3D OCTA volumes directly using a 2D convolutional neural network trained with additional projection supervision. Our experimental results show that we achieve over 80% accuracy on a four-stage grading task on both error-free and error-prone test sets, which is significantly higher than 60%, the accuracy of human experts. This demonstrates that OCTA provides sufficient information for AMD stage grading and the proposed 3D volume analyzer is more robust when dealing with OCTA data with segmentation errors.

A transfer learning-based multimodal neural network combining metadata and multiple medical images for glaucoma type diagnosis

Glaucoma is an acquired optic neuropathy, which can lead to irreversible vision loss. Deep learning(DL), especially convolutional neural networks(CNN), has achieved considerable success in the field of medical image recognition due to the availability of large-scale annotated datasets and CNNs. However, obtaining fully annotated datasets like ImageNet in the medical field is still a challenge. Meanwhile, single-modal approaches remain both unreliable and inaccurate due to the diversity of glaucoma disease types and the complexity of symptoms. In this paper, a new multimodal dataset for glaucoma is constructed and a new multimodal neural network for glaucoma diagnosis and classification (GMNNnet) is proposed aiming to address both of these issues. Specifically, the dataset includes the five most important types of glaucoma labels, electronic medical records and four kinds of high-resolution medical images. The structure of GMNNnet consists of three branches. Branch 1 consisting of convolutional, cyclic and transposition layers processes patient metadata, branch 2 uses Unet to extract features from glaucoma segmentation based on domain knowledge, and branch 3 uses ResFormer to directly process glaucoma medical images.Branch one and branch two are mixed together and then processed by the Catboost classifier. We introduce a gradient-weighted class activation mapping (Grad-GAM) method to increase the interpretability of the model and a transfer learning method for the case of insufficient training data,i.e.,fine-tuning CNN models pre-trained from natural image dataset to medical image tasks. The results show that GMNNnet can better present the high-dimensional information of glaucoma and achieves excellent performance under multimodal data.

Using Ensemble OCT-Derived Features beyond Intensity Features for Enhanced Stargardt Atrophy Prediction with Deep Learning

Stargardt disease is the most common form of juvenile-onset macular dystrophy. Spectral-domain optical coherence tomography (SD-OCT) imaging provides an opportunity to directly measure changes to retinal layers due to Stargardt atrophy. Generally, atrophy segmentation and prediction can be conducted using mean intensity feature maps generated from the relevant retinal layers. In this paper, we report an approach using advanced OCT-derived features to augment and enhance data beyond the commonly used mean intensity features for enhanced prediction of Stargardt atrophy with an ensemble deep learning neural network. With all the relevant retinal layers, this neural network architecture achieves a median Dice coefficient of 0.830 for six-month predictions and 0.828 for twelve-month predictions, showing a significant improvement over a neural network using only mean intensity, which achieved Dice coefficients of 0.744 and 0.762 for six-month and twelve-month predictions, respectively. When using feature maps generated from different layers of the retina, significant differences in performance were observed. This study shows promising results for using multiple OCT-derived features beyond intensity for assessing the prognosis of Stargardt disease and quantifying the rate of progression.

Optical coherence tomography image despeckling based on tensor singular value decomposition and fractional edge detection

Optical coherence tomography (OCT) imaging is a technique that is frequently used to diagnose medical conditions. However, coherent noise, sometimes referred to as speckle noise, can dramatically reduce the quality of OCT images, which has an adverse effect on how OCT images are used. In order to enhance the quality of OCT images, a speckle noise reduction technique is developed, and this method is modelled as a low-rank tensor approximation issue. The grouped 3D tensors are first transformed into the transform domain using tensor singular value decomposition (t-SVD). Then, to cut down on speckle noise, transform coefficients are thresholded. Finally, the inverse transform can be used to produce images with speckle suppression. To further enhance the despeckling results, a feature-guided thresholding approach based on fractional edge detection and an adaptive backward projection technique are also presented. Experimental results indicate that the presented algorithm outperforms several comparison methods in relation to speckle suppression, objective metrics, and edge preservation.

OCT2Former: A retinal OCT-angiography vessel segmentation transformer

BACKGROUND AND OBJECTIVE

Retinal vessel segmentation plays an important role in the automatic retinal disease screening and diagnosis. How to segment thin vessels and maintain the connectivity of vessels are the key challenges of the retinal vessel segmentation task. Optical coherence tomography angiography (OCTA) is a noninvasive imaging technique that can reveal high-resolution retinal vessels. Aiming at make full use of its characteristic of high resolution, a new end-to-end transformer based network named as OCT²Former (OCT-a Transformer) is proposed to segment retinal vessel accurately in OCTA images.

METHODS

The proposed OCT²Former is based on encoder-decoder structure, which mainly includes dynamic transformer encoder and lightweight decoder. Dynamic transformer encoder consists of dynamic token aggregation transformer and auxiliary convolution branch, in which the multi-head dynamic token aggregation attention based dynamic token aggregation transformer is designed to capture the global retinal vessel context information from the first layer throughout the network and the auxiliary convolution branch is proposed to compensate for the lack of inductive bias of the transformer and assist in the efficient feature extraction. A convolution based lightweight decoder is proposed to decode features efficiently and reduce the complexity of the proposed OCT²Former.

RESULTS

The proposed OCT²Former is validated on three publicly available datasets i.e. OCTA-SS, ROSE-1, OCTA-500 (subset OCTA-6M and OCTA-3M). The Jaccard indexes of the proposed OCT²Former on these datasets are 0.8344, 0.7855, 0.8099 and 0.8513, respectively, outperforming the best convolution based network 1.43, 1.32, 0.75 and 1.46%, respectively.

CONCLUSION

The experimental results have demonstrated that the proposed OCT²Former can achieve competitive performance on retinal OCTA vessel segmentation tasks.

Transforming medical imaging with Transformers? A comparative review of key properties, current progresses, and future perspectives

Transformer, one of the latest technological advances of deep learning, has gained prevalence in natural language processing or computer vision. Since medical imaging bear some resemblance to computer vision, it is natural to inquire about the status quo of Transformers in medical imaging and ask the question: can the Transformer models transform medical imaging? In this paper, we attempt to make a response to the inquiry. After a brief introduction of the fundamentals of Transformers, especially in comparison with convolutional neural networks (CNNs), and highlighting key defining properties that characterize the Transformers, we offer a comprehensive review of the state-of-the-art Transformer-based approaches for medical imaging and exhibit current research progresses made in the areas of medical image segmentation, recognition, detection, registration, reconstruction, enhancement, etc. In particular, what distinguishes our review lies in its organization based on the Transformer's key defining properties, which are mostly derived from comparing the Transformer and CNN, and its type of architecture, which specifies the manner in which the Transformer and CNN are combined, all helping the readers to best understand the rationale behind the reviewed approaches. We conclude with discussions of future perspectives.

Speckle attenuation for optical coherence tomography images using the generalized low rank approximations of matrices

A frequently used technology in medical diagnosis is optical coherence tomography (OCT). However, coherent noise, also known as speckle noise, has the potential to severely reduce the quality of OCT images, which would be detrimental to the use of OCT images for disease diagnosis. In this paper, a despeckling method is proposed to effectively reduce the speckle noise in OCT images using the generalized low rank approximations of matrices (GLRAM). Specifically, the Manhattan distance (MD)-based block matching method is first used to find nonlocal similar blocks for the reference one. The left and right projection matrices shared by these image blocks are then found using the GLRAM approach, and an adaptive method based on asymptotic matrix reconstruction is proposed to determine how many eigenvectors are present in the left and right projection matrices. Finally, all the reconstructed image blocks are aggregated to create the despeckled OCT image. In addition, an edge-guided adaptive back-projection strategy is used to improve the despeckling performance of the proposed method. Experiments with synthetic and real OCT images show that the presented method performs well in both objective measurements and visual evaluation.

SDA-UNet: a hepatic vein segmentation network based on the spatial distribution and density awareness of blood vessels

Objective.Hepatic vein segmentation is a fundamental task for liver diagnosis and surgical navigation planning. Unlike other organs, the liver is the only organ with two sets of venous systems. Meanwhile, the segmentation target distribution in the hepatic vein scene is extremely unbalanced. The hepatic veins occupy a small area in abdominal CT slices. The morphology of each person's hepatic vein is different, which also makes segmentation difficult. The purpose of this study is to develop an automated hepatic vein segmentation model that guides clinical diagnosis.Approach.We introduce the 3D spatial distribution and density awareness (SDA) of hepatic veins and propose an automatic segmentation network based on 3D U-Net which includes a multi-axial squeeze and excitation module (MASE) and a distribution correction module (DCM). The MASE restrict the activation area to the area with hepatic veins. The DCM improves the awareness of the sparse spatial distribution of the hepatic veins. To obtain global axial information and spatial information at the same time, we study the effect of different training strategies on hepatic vein segmentation. Our method was evaluated by a public dataset and a private dataset. The Dice coefficient achieves 71.37% and 69.58%, improving 3.60% and 3.30% compared to the other SOTA models, respectively. Furthermore, metrics based on distance and volume also show the superiority of our method.Significance.The proposed method greatly reduced false positive areas and improved the segmentation performance of the hepatic vein in CT images. It will assist doctors in making accurate diagnoses and surgical navigation planning.

Corrections to "Image Projection Network: 3D to 2D Image Segmentation in OCTA Images"

In the above article [1], there is an error in (3). Instead of [Formula: see text] It should be [Formula: see text].

Retinal OCTA Image Segmentation Based on Global Contrastive Learning

The automatic segmentation of retinal vessels is of great significance for the analysis and diagnosis of retinal related diseases. However, the imbalanced data in retinal vascular images remain a great challenge. Current image segmentation methods based on deep learning almost always focus on local information in a single image while ignoring the global information of the entire dataset. To solve the problem of data imbalance in optical coherence tomography angiography (OCTA) datasets, this paper proposes a medical image segmentation method (contrastive OCTA segmentation net, COSNet) based on global contrastive learning. First, the feature extraction module extracts the features of OCTA image input and maps them to the segment head and the multilayer perceptron (MLP) head, respectively. Second, a contrastive learning module saves the pixel queue and pixel embedding of each category in the feature map into the memory bank, generates sample pairs through a mixed sampling strategy to construct a new contrastive loss function, and forces the network to learn local information and global information simultaneously. Finally, the segmented image is fine tuned to restore positional information of deep vessels. The experimental results show the proposed method can improve the accuracy (ACC), the area under the curve (AUC), and other evaluation indexes of image segmentation compared with the existing methods. This method could accomplish segmentation tasks in imbalanced data and extend to other segmentation tasks.

Applications of Artificial Intelligence in Optical Coherence Tomography Angiography Imaging

Optical coherence tomography angiography (OCTA) and artificial intelligence (AI) are two emerging fields that complement each other. OCTA enables the noninvasive, in vivo, 3D visualization of retinal blood flow with a micrometer resolution, which has been impossible with other imaging modalities. As it does not need dye-based injections, it is also a safer procedure for patients. AI has excited great interest in many fields of daily life, by enabling automatic processing of huge amounts of data with a performance that greatly surpasses previous algorithms. It has been used in many breakthrough studies in recent years, such as the finding that AlphaGo can beat humans in the strategic board game of Go. This paper will give a short introduction into both fields and will then explore the manifold applications of AI in OCTA imaging that have been presented in the recent years. These range from signal generation over signal enhancement to interpretation tasks like segmentation and classification. In all these areas, AI-based algorithms have achieved state-of-the-art performance that has the potential to improve standard care in ophthalmology when integrated into the daily clinical routine.

Disentangled Representation Learning for OCTA Vessel Segmentation With Limited Training Data

Optical coherence tomography angiography (OCTA) is an imaging modality that can be used for analyzing retinal vasculature. Quantitative assessment of en face OCTA images requires accurate segmentation of the capillaries. Using deep learning approaches for this task faces two major challenges. First, acquiring sufficient manual delineations for training can take hundreds of hours. Second, OCTA images suffer from numerous contrast-related artifacts that are currently inherent to the modality and vary dramatically across scanners. We propose to solve both problems by learning a disentanglement of an anatomy component and a local contrast component from paired OCTA scans. With the contrast removed from the anatomy component, a deep learning model that takes the anatomy component as input can learn to segment vessels with a limited portion of the training images being manually labeled. Our method demonstrates state-of-the-art performance for OCTA vessel segmentation.

Retinal Structure Detection in OCTA Image via Voting-Based Multitask Learning

Automated detection of retinal structures, such as retinal vessels (RV), the foveal avascular zone (FAZ), and retinal vascular junctions (RVJ), are of great importance for understanding diseases of the eye and clinical decision-making. In this paper, we propose a novel Voting-based Adaptive Feature Fusion multi-task network (VAFF-Net) for joint segmentation, detection, and classification of RV, FAZ, and RVJ in optical coherence tomography angiography (OCTA). A task-specific voting gate module is proposed to adaptively extract and fuse different features for specific tasks at two levels: features at different spatial positions from a single encoder, and features from multiple encoders. In particular, since the complexity of the microvasculature in OCTA images makes simultaneous precise localization and classification of retinal vascular junctions into bifurcation/crossing a challenging task, we specifically design a task head by combining the heatmap regression and grid classification. We take advantage of three different en face angiograms from various retinal layers, rather than following existing methods that use only a single en face. We carry out extensive experiments on three OCTA datasets acquired using different imaging devices, and the results demonstrate that the proposed method performs on the whole better than either the state-of-the-art single-purpose methods or existing multi-task learning solutions. We also demonstrate that our multi-task learning method generalizes across other imaging modalities, such as color fundus photography, and may potentially be used as a general multi-task learning tool. We also construct three datasets for multiple structure detection, and part of these datasets with the source code and evaluation benchmark have been released for public access.