Joint Segment-Level and Pixel-Wise Losses for Deep Learning Based Retinal Vessel Segmentation

Accurate Retinal Vessel Segmentation in Color Fundus Images via Fully Attention-Based Networks

Automatic retinal vessel segmentation is important for the diagnosis and prevention of ophthalmic diseases. The existing deep learning retinal vessel segmentation models always treat each pixel equally. However, the multi-scale vessel structure is a vital factor affecting the segmentation results, especially in thin vessels. To address this crucial gap, we propose a novel Fully Attention-based Network (FANet) based on attention mechanisms to adaptively learn rich feature representation and aggregate the multi-scale information. Specifically, the framework consists of the image pre-processing procedure and the semantic segmentation networks. Green channel extraction (GE) and contrast limited adaptive histogram equalization (CLAHE) are employed as pre-processing to enhance the texture and contrast of retinal blood images. Besides, the network combines two types of attention modules with the U-Net. We propose a lightweight dual-direction attention block to model global dependencies and reduce intra-class inconsistencies, in which the weights of feature maps are updated based on the semantic correlation between pixels. The dual-direction attention block utilizes horizontal and vertical pooling operations to produce the attention map. In this way, the network aggregates global contextual information from semantic-closer regions or a series of pixels belonging to the same object category. Meanwhile, we adopt the selective kernel (SK) unit to replace the standard convolution for obtaining multi-scale features of different receptive field sizes generated by soft attention. Furthermore, we demonstrate that the proposed model can effectively identify irregular, noisy, and multi-scale retinal vessels. The abundant experiments on DRIVE, STARE, and CHASE_DB1 datasets show that our method achieves state-of-the-art performance.

Introducing frequency representation into convolution neural networks for medical image segmentation via twin-Kernel Fourier convolution

BACKGROUND AND OBJECTIVE

For medical image segmentation, deep learning-based methods have achieved state-of-the-art performance. However, the powerful spectral representation in the field of image processing is rarely considered in these models.

METHODS

In this work, we propose to introduce frequency representation into convolution neural networks (CNNs) and design a novel model, tKFC-Net, to combine powerful feature representation in both frequency and spatial domains. Through the Fast Fourier Transform (FFT) operation, frequency representation is employed on pooling, upsampling, and convolution without any adjustments to the network architecture. Furthermore, we replace original convolution with twin-Kernel Fourier Convolution (t-KFC), a new designed convolution layer, to specify the convolution kernels for particular functions and extract features from different frequency components.

RESULTS

We experimentally show that our method has an edge over other models in the task of medical image segmentation. Evaluated on four datasets-skin lesion segmentation (ISIC 2018), retinal blood vessel segmentation (DRIVE), lung segmentation (COVID-19-CT-Seg), and brain tumor segmentation (BraTS 2019), the proposed model achieves outstanding results: the metric F1-Score is 0.878 for ISIC 2018, 0.8185 for DRIVE, 0.9830 for COVID-19-CT-Seg, and 0.8457 for BraTS 2019.

CONCLUSION

The introduction of spectral representation retains spectral features which result in more accurate segmentation. The proposed method is orthogonal to other topology improvement methods and very convenient to be combined.

BSEResU-Net: An attention-based before-activation residual U-Net for retinal vessel segmentation

BACKGROUND AND OBJECTIVES

Retinal vessels are a major feature used for the physician to diagnose many retinal diseases, such as cardiovascular disease and Glaucoma. Therefore, the designing of an auto-segmentation algorithm for retinal vessel draw great attention in medical field. Recently, deep learning methods, especially convolutional neural networks (CNNs) show extraordinary potential for the task of vessel segmentation. However, most of the deep learning methods only take advantage of the shallow networks with a traditional cross-entropy objective, which becomes the main obstacle to further improve the performance on a task that is imbalanced. We therefore propose a new type of residual U-Net called Before-activation Squeeze-and-Excitation ResU-Net (BSEResu-Net) to tackle the aforementioned issues.

METHODS

Our BSEResU-Net can be viewed as an encoder/decoder framework that constructed by Before-activation Squeeze-and-Excitation blocks (BSE Blocks). In comparison to the current existing CNN structures, we utilize a new type of residual block structure, namely BSE block, in which the attention mechanism is combined with skip connection to boost the performance. What's more, the network could consistently gain accuracy from the increasing depth as we incorporate more residual blocks, attributing to the dropblock mechanism used in BSE blocks. A joint loss function which is based on the dice and cross-entropy loss functions is also introduced to achieve more balanced segmentation between the vessel and non-vessel pixels.

RESULTS

The proposed BSEResU-Net is evaluated on the publicly available DRIVE, STARE and HRF datasets. It achieves the F1-score of 0.8324, 0.8368 and 0.8237 on DRIVE, STARE and HRF dataset, respectively. Experimental results show that the proposed BSEResU-Net outperforms current state-of-the-art algorithms.

CONCLUSIONS

The proposed algorithm utilizes a new type of residual blocks called BSE residual blocks for vessel segmentation. Together with a joint loss function, it shows outstanding performance both on low and high-resolution fundus images.

Multichannel Retinal Blood Vessel Segmentation Based on the Combination of Matched Filter and U-Net Network

Aiming at the current problem of insufficient extraction of small retinal blood vessels, we propose a retinal blood vessel segmentation algorithm that combines supervised learning and unsupervised learning algorithms. In this study, we use a multiscale matched filter with vessel enhancement capability and a U-Net model with a coding and decoding network structure. Three channels are used to extract vessel features separately, and finally, the segmentation results of the three channels are merged. The algorithm proposed in this paper has been verified and evaluated on the DRIVE, STARE, and CHASE_DB1 datasets. The experimental results show that the proposed algorithm can segment small blood vessels better than most other methods. We conclude that our algorithm has reached 0.8745, 0.8903, and 0.8916 on the three datasets in the sensitivity metric, respectively, which is nearly 0.1 higher than other existing methods.

CMM-Net: Contextual multi-scale multi-level network for efficient biomedical image segmentation

Medical image segmentation of tissue abnormalities, key organs, or blood vascular system is of great significance for any computerized diagnostic system. However, automatic segmentation in medical image analysis is a challenging task since it requires sophisticated knowledge of the target organ anatomy. This paper develops an end-to-end deep learning segmentation method called Contextual Multi-Scale Multi-Level Network (CMM-Net). The main idea is to fuse the global contextual features of multiple spatial scales at every contracting convolutional network level in the U-Net. Also, we re-exploit the dilated convolution module that enables an expansion of the receptive field with different rates depending on the size of feature maps throughout the networks. In addition, an augmented testing scheme referred to as Inversion Recovery (IR) which uses logical "OR" and "AND" operators is developed. The proposed segmentation network is evaluated on three medical imaging datasets, namely ISIC 2017 for skin lesions segmentation from dermoscopy images, DRIVE for retinal blood vessels segmentation from fundus images, and BraTS 2018 for brain gliomas segmentation from MR scans. The experimental results showed superior state-of-the-art performance with overall dice similarity coefficients of 85.78%, 80.27%, and 88.96% on the segmentation of skin lesions, retinal blood vessels, and brain tumors, respectively. The proposed CMM-Net is inherently general and could be efficiently applied as a robust tool for various medical image segmentations.

Quantification of Blood Flow Velocity in the Human Conjunctival Microvessels Using Deep Learning-Based Stabilization Algorithm

The quantification of blood flow velocity in the human conjunctiva is clinically essential for assessing microvascular hemodynamics. Since the conjunctival microvessel is imaged in several seconds, eye motion during image acquisition causes motion artifacts limiting the accuracy of image segmentation performance and measurement of the blood flow velocity. In this paper, we introduce a novel customized optical imaging system for human conjunctiva with deep learning-based segmentation and motion correction. The image segmentation process is performed by the Attention-UNet structure to achieve high-performance segmentation results in conjunctiva images with motion blur. Motion correction processes with two steps—registration and template matching—are used to correct for large displacements and fine movements. The image displacement values decrease to 4–7 μm during registration (first step) and less than 1 μm during template matching (second step). With the corrected images, the blood flow velocity is calculated for selected vessels considering temporal signal variances and vessel lengths. These methods for resolving motion artifacts contribute insights into studies quantifying the hemodynamics of the conjunctiva, as well as other tissues.

Assessing fairness in performance evaluation of publicly available retinal blood vessel segmentation algorithms

In the literature, various algorithms have been proposed for automatically extracting blood vessels from retinal images. In general, they are developed and evaluated using several publicly available datasets such as the DRIVE and STARE datasets. For performance evaluation, several metrics such as Sensitivity, Specificity, and Accuracy have been widely used. However, not all methods in the literature have been fairly evaluated and compared among their counterparts. In particular, for some publicly available algorithms, the performance is measured only for the area inside the field of view (FOV) of each retinal image while the rest use the complete image for the performance evaluation. Therefore, performing a comparison of the performance of methods in the latter group with those in the former group may lead to a misleading justification. This study aims to assess fairness in the performance evaluation of various publicly available retinal blood vessel segmentation algorithms. The conducted study allows getting several meaningful results: (i) a guideline to assess fairness in performance evaluation of retinal vessel segmentation algorithms, (ii) a more proper performance comparison of retinal vessel segmentation algorithms in the literature, and (iii) a suggestion regarding the use of performance evaluation metrics that will not lead to misleading comparison and justification.

CSU-Net: A Context Spatial U-Net for Accurate Blood Vessel Segmentation in Fundus Images

Blood vessel segmentation in fundus images is a critical procedure in the diagnosis of ophthalmic diseases. Recent deep learning methods achieve high accuracy in vessel segmentation but still face the challenge to segment the microvascular and detect the vessel boundary. This is due to the fact that common Convolutional Neural Networks (CNN) are unable to preserve rich spatial information and a large receptive field simultaneously. Besides, CNN models for vessel segmentation usually are trained by equal pixel level cross-entropy loss, which tend to miss fine vessel structures. In this paper, we propose a novel Context Spatial U-Net (CSU-Net) for blood vessel segmentation. Compared with the other U-Net based models, we design a two-channel encoder: a context channel with multi-scale convolution to capture more receptive field and a spatial channel with large kernel to retain spatial information. Also, to combine and strengthen the features extracted from two paths, we introduce a feature fusion module (FFM) and an attention skip module (ASM). Furthermore, we propose a structure loss, which adds a spatial weight to cross-entropy loss and guide the network to focus more on the thin vessels and boundaries. We evaluated this model on three public datasets: DRIVE, CHASE-DB1 and STARE. The results show that the CSU-Net achieves higher segmentation accuracy than the current state-of-the-art methods.

On the quantitative effects of compression of retinal fundus images on morphometric vascular measurements in VAMPIRE

BACKGROUND AND OBJECTIVES

This paper reports a quantitative analysis of the effects of joint photographic experts group (JPEG) image compression of retinal fundus camera images on automatic vessel segmentation and on morphometric vascular measurements derived from it, including vessel width, tortuosity and fractal dimension.

METHODS

Measurements are computed with vascular assessment and measurement platform for images of the retina (VAMPIRE), a specialized software application adopted in many international studies on retinal biomarkers. For reproducibility, we use three public archives of fundus images (digital retinal images for vessel extraction (DRIVE), automated retinal image analyzer (ARIA), high-resolution fundus (HRF)). We generate compressed versions of original images in a range of representative levels.

RESULTS

We compare the resulting vessel segmentations with ground truth maps and morphological measurements of the vascular network with those obtained from the original (uncompressed) images. We assess the segmentation quality with sensitivity, specificity, accuracy, area under the curve and Dice coefficient. We assess the agreement between VAMPIRE measurements from compressed and uncompressed images with correlation, intra-class correlation and Bland-Altman analysis.

CONCLUSIONS

Results suggest that VAMPIRE width-related measurements (central retinal artery equivalent (CRAE), central retinal vein equivalent (CRVE), arteriolar-venular width ratio (AVR)), the fractal dimension (FD) and arteriolar tortuosity have excellent agreement with those from the original images, remaining substantially stable even for strong loss of quality (20% of the original), suggesting the suitability of VAMPIRE in association studies with compressed images.

An efficient retinal blood vessel segmentation in eye fundus images by using optimized top-hat and homomorphic filtering

BACKGROUND AND OBJECTIVE

Automatic segmentation of retinal blood vessels makes a major contribution in CADx of various ophthalmic and cardiovascular diseases. A procedure to segment thin and thick retinal vessels is essential for medical analysis and diagnosis of related diseases. In this article, a novel methodology for robust vessel segmentation is proposed, handling the existing challenges presented in the literature.

METHODS

The proposed methodology consists of three stages, pre-processing, main processing, and post-processing. The first stage consists of applying filters for image smoothing. The main processing stage is divided into two configurations, the first to segment thick vessels through the new optimized top-hat, homomorphic filtering, and median filter. Then, the second configuration is used to segment thin vessels using the proposed optimized top-hat, homomorphic filtering, matched filter, and segmentation using the MCET-HHO multilevel algorithm. Finally, morphological image operations are carried out in the post-processing stage.

RESULTS

The proposed approach was assessed by using two publicly available databases (DRIVE and STARE) through three performance metrics: specificity, sensitivity, and accuracy. Analyzing the obtained results, an average of 0.9860, 0.7578 and 0.9667 were respectively achieved for DRIVE dataset and 0.9836, 0.7474 and 0.9580 for STARE dataset.

CONCLUSIONS

The numerical results obtained by the proposed technique, achieve competitive average values with the up-to-date techniques. The proposed approach outperform all leading unsupervised methods discussed in terms of specificity and accuracy. In addition, it outperforms most of the state-of-the-art supervised methods without the computational cost associated with these algorithms. Detailed visual analysis has shown that a more precise segmentation of thin vessels was possible with the proposed approach when compared with other procedures.

Lightweight pyramid network with spatial attention mechanism for accurate retinal vessel segmentation

PURPOSE

The morphological characteristics of retinal vessels are vital for the early diagnosis of pathological diseases such as diabetes and hypertension. However, the low contrast and complex morphology pose a challenge to automatic retinal vessel segmentation. To extract precise semantic features, more convolution and pooling operations are adopted, but some structural information is potentially ignored.

METHODS

In the paper, we propose a novel lightweight pyramid network (LPN) fusing multi-scale features with spatial attention mechanism to preserve the structure information of retinal vessels. The pyramid hierarchy model is constructed to generate multi-scale representations, and its semantic features are strengthened with the introduction of the attention mechanism. The combination of multi-scale features contributes to its accurate prediction.

RESULTS

The LPN is evaluated on benchmark datasets DRIVE, STARE and CHASE, and the results indicate its state-of-the-art performance (e.g., ACC of 97.09[Formula: see text]/97.49[Formula: see text]/97.48[Formula: see text], AUC of 98.79[Formula: see text]/99.01[Formula: see text]/98.91[Formula: see text] on the DRIVE, STARE and CHASE datasets, respectively). The robustness and generalization ability of the LPN are further proved in cross-training experiment.

CONCLUSION

The visualization experiment reveals the semantic gap between various scales of the pyramid and verifies the effectiveness of the attention mechanism, which provide a potential basis for the pyramid hierarchy model in multi-scale vessel segmentation task. Furthermore, the number of model parameters is greatly reduced.

Fast and efficient retinal blood vessel segmentation method based on deep learning network

The segmentation of the retinal vascular tree presents a major step for detecting ocular pathologies. The clinical context expects higher segmentation performance with a reduced processing time. For higher accurate segmentation, several automated methods have been based on Deep Learning (DL) networks. However, the used convolutional layers bring to a higher computational complexity and so for execution times. For such need, this work presents a new DL based method for retinal vessel tree segmentation. Our main contribution consists in suggesting a new U-form DL architecture using lightweight convolution blocks in order to preserve a higher segmentation performance while reducing the computational complexity. As a second main contribution, preprocessing and data augmentation steps are proposed with respect to the retinal image and blood vessel characteristics. The proposed method is tested on DRIVE and STARE databases, which can achieve a better trade-off between the retinal blood vessel detection rate and the detection time with average accuracy of 0.978 and 0.98 in 0.59 s and 0.48 s per fundus image on GPU NVIDIA GTX 980 platforms, respectively for DRIVE and STARE database fundus images.

Learning-based algorithms for vessel tracking: A review

Developing efficient vessel-tracking algorithms is crucial for imaging-based diagnosis and treatment of vascular diseases. Vessel tracking aims to solve recognition problems such as key (seed) point detection, centerline extraction, and vascular segmentation. Extensive image-processing techniques have been developed to overcome the problems of vessel tracking that are mainly attributed to the complex morphologies of vessels and image characteristics of angiography. This paper presents a literature review on vessel-tracking methods, focusing on machine-learning-based methods. First, the conventional machine-learning-based algorithms are reviewed, and then, a general survey of deep-learning-based frameworks is provided. On the basis of the reviewed methods, the evaluation issues are introduced. The paper is concluded with discussions about the remaining exigencies and future research.

Applications of deep learning in fundus images: A review

The use of fundus images for the early screening of eye diseases is of great clinical importance. Due to its powerful performance, deep learning is becoming more and more popular in related applications, such as lesion segmentation, biomarkers segmentation, disease diagnosis and image synthesis. Therefore, it is very necessary to summarize the recent developments in deep learning for fundus images with a review paper. In this review, we introduce 143 application papers with a carefully designed hierarchy. Moreover, 33 publicly available datasets are presented. Summaries and analyses are provided for each task. Finally, limitations common to all tasks are revealed and possible solutions are given. We will also release and regularly update the state-of-the-art results and newly-released datasets at https://github.com/nkicsl/Fundus_Review to adapt to the rapid development of this field.

Towards Automated Eye Diagnosis: An Improved Retinal Vessel Segmentation Framework Using Ensemble Block Matching 3D Filter

Automated detection of vision threatening eye disease based on high resolution retinal fundus images requires accurate segmentation of the blood vessels. In this regard, detection and segmentation of finer vessels, which are obscured by a considerable degree of noise and poor illumination, is particularly challenging. These noises include (systematic) additive noise and multiplicative (speckle) noise, which arise due to various practical limitations of the fundus imaging systems. To address this inherent issue, we present an efficient unsupervised vessel segmentation strategy as a step towards accurate classification of eye diseases from the noisy fundus images. To that end, an ensemble block matching 3D (BM3D) speckle filter is proposed for removal of unwanted noise leading to improved detection. The BM3D-speckle filter, despite its ability to recover finer details (i.e., vessels in fundus images), yields a pattern of checkerboard artifacts in the aftermath of multiplicative (speckle) noise removal. These artifacts are generally ignored in the case of satellite images; however, in the case of fundus images, these artifacts have a degenerating effect on the segmentation or detection of fine vessels. To counter that, an ensemble of BM3D-speckle filter is proposed to suppress these artifacts while further sharpening the recovered vessels. This is subsequently used to devise an improved unsupervised segmentation strategy that can detect fine vessels even in the presence of dominant noise and yields an overall much improved accuracy. Testing was carried out on three publicly available databases namely Structured Analysis of the Retina (STARE), Digital Retinal Images for Vessel Extraction (DRIVE) and CHASE_DB1. We have achieved a sensitivity of 82.88, 81.41 and 82.03 on DRIVE, SATARE, and CHASE_DB1, respectively. The accuracy is also boosted to 95.41, 95.70 and 95.61 on DRIVE, SATARE, and CHASE_DB1, respectively. The performance of the proposed methods on images with pathologies was observed to be more convincing than the performance of similar state-of-the-art methods.

A Hybrid Unsupervised Approach for Retinal Vessel Segmentation

Retinal vessel segmentation (RVS) is a significant source of useful information for monitoring, identification, initial medication, and surgical development of ophthalmic disorders. Most common disorders, i.e., stroke, diabetic retinopathy (DR), and cardiac diseases, often change the normal structure of the retinal vascular network. A lot of research has been committed to building an automatic RVS system. But, it is still an open issue. In this article, a framework is recommended for RVS with fast execution and competing outcomes. An initial binary image is obtained by the application of the MISODATA on the preprocessed image. For vessel structure enhancement, B-COSFIRE filters are utilized along with thresholding to obtain another binary image. These two binary images are combined by logical AND-type operation. Then, it is fused with the enhanced image of B-COSFIRE filters followed by thresholding to obtain the vessel location map (VLM). The methodology is verified on four different datasets: DRIVE, STARE, HRF, and CHASE_DB1, which are publicly accessible for benchmarking and validation. The obtained results are compared with the existing competing methods.

ELEMENT: Multi-Modal Retinal Vessel Segmentation Based on a Coupled Region Growing and Machine Learning Approach

Vascular structures in the retina contain important information for the detection and analysis of ocular diseases, including age-related macular degeneration, diabetic retinopathy and glaucoma. Commonly used modalities in diagnosis of these diseases are fundus photography, scanning laser ophthalmoscope (SLO) and fluorescein angiography (FA). Typically, retinal vessel segmentation is carried out either manually or interactively, which makes it time consuming and prone to human errors. In this research, we propose a new multi-modal framework for vessel segmentation called ELEMENT (vEsseL sEgmentation using Machine lEarning and coNnecTivity). This framework consists of feature extraction and pixel-based classification using region growing and machine learning. The proposed features capture complementary evidence based on grey level and vessel connectivity properties. The latter information is seamlessly propagated through the pixels at the classification phase. ELEMENT reduces inconsistencies and speeds up the segmentation throughput. We analyze and compare the performance of the proposed approach against state-of-the-art vessel segmentation algorithms in three major groups of experiments, for each of the ocular modalities. Our method produced higher overall performance, with an overall accuracy of 97.40%, compared to 25 of the 26 state-of-the-art approaches, including six works based on deep learning, evaluated on the widely known DRIVE fundus image dataset. In the case of the STARE, CHASE-DB, VAMPIRE FA, IOSTAR SLO and RC-SLO datasets, the proposed framework outperformed all of the state-of-the-art methods with accuracies of 98.27%, 97.78%, 98.34%, 98.04% and 98.35%, respectively.

DoFE: Domain-Oriented Feature Embedding for Generalizable Fundus Image Segmentation on Unseen Datasets

Deep convolutional neural networks have significantly boosted the performance of fundus image segmentation when test datasets have the same distribution as the training datasets. However, in clinical practice, medical images often exhibit variations in appearance for various reasons, e.g., different scanner vendors and image quality. These distribution discrepancies could lead the deep networks to over-fit on the training datasets and lack generalization ability on the unseen test datasets. To alleviate this issue, we present a novel Domain-oriented Feature Embedding (DoFE) framework to improve the generalization ability of CNNs on unseen target domains by exploring the knowledge from multiple source domains. Our DoFE framework dynamically enriches the image features with additional domain prior knowledge learned from multi-source domains to make the semantic features more discriminative. Specifically, we introduce a Domain Knowledge Pool to learn and memorize the prior information extracted from multi-source domains. Then the original image features are augmented with domain-oriented aggregated features, which are induced from the knowledge pool based on the similarity between the input image and multi-source domain images. We further design a novel domain code prediction branch to infer this similarity and employ an attention-guided mechanism to dynamically combine the aggregated features with the semantic features. We comprehensively evaluate our DoFE framework on two fundus image segmentation tasks, including the optic cup and disc segmentation and vessel segmentation. Our DoFE framework generates satisfying segmentation results on unseen datasets and surpasses other domain generalization and network regularization methods.

A review of machine learning methods for retinal blood vessel segmentation and artery/vein classification

The eye affords a unique opportunity to inspect a rich part of the human microvasculature non-invasively via retinal imaging. Retinal blood vessel segmentation and classification are prime steps for the diagnosis and risk assessment of microvascular and systemic diseases. A high volume of techniques based on deep learning have been published in recent years. In this context, we review 158 papers published between 2012 and 2020, focussing on methods based on machine and deep learning (DL) for automatic vessel segmentation and classification for fundus camera images. We divide the methods into various classes by task (segmentation or artery-vein classification), technique (supervised or unsupervised, deep and non-deep learning, hand-crafted methods) and more specific algorithms (e.g. multiscale, morphology). We discuss advantages and limitations, and include tables summarising results at-a-glance. Finally, we attempt to assess the quantitative merit of DL methods in terms of accuracy improvement compared to other methods. The results allow us to offer our views on the outlook for vessel segmentation and classification for fundus camera images.

Retinal Vessel Segmentation by Deep Residual Learning with Wide Activation

Purpose

Retinal blood vessel image segmentation is an important step in ophthalmological analysis. However, it is difficult to segment small vessels accurately because of low contrast and complex feature information of blood vessels. The objective of this study is to develop an improved retinal blood vessel segmentation structure (WA-Net) to overcome these challenges.

Methods

This paper mainly focuses on the width of deep learning. The channels of the ResNet block were broadened to propagate more low-level features, and the identity mapping pathway was slimmed to maintain parameter complexity. A residual atrous spatial pyramid module was used to capture the retinal vessels at various scales. We applied weight normalization to eliminate the impacts of the mini-batch and improve segmentation accuracy. The experiments were performed on the DRIVE and STARE datasets. To show the generalizability of WA-Net, we performed cross-training between datasets.

Results

The global accuracy and specificity within datasets were 95.66% and 96.45% and 98.13% and 98.71%, respectively. The accuracy and area under the curve of the interdataset diverged only by 1%∼2% compared with the performance of the corresponding intradataset.

Conclusion

All the results show that WA-Net extracts more detailed blood vessels and shows superior performance on retinal blood vessel segmentation tasks.

Ψ-Net: Focusing on the border areas of intracerebral hemorrhage on CT images

BACKGROUND AND OBJECTIVE

The volume of the intracerebral hemorrhage (ICH) obtained from CT scans is essential for quantification and treatment planning. However,a fast and accurate volume acquisition brings great challenges. On the one hand, it is both time consuming and operator dependent for manual segmentation, which is the gold standard for volume estimation. On the other hand, low contrast to normal tissues, irregular shapes and distributions of the hemorrhage make the existing automatic segmentation methods hard to achieve satisfactory performance.

METHOD

To solve above problems, a CNN-based architecture is proposed in this work, consisting of a novel model, which is named as Ψ-Net and a multi-level training strategy. In the structure of Ψ-Net, a self-attention block and a contextual-attention block is designed to suppresses the irrelevant information and segment border areas of the hemorrhage more finely. Further, an multi-level training strategy is put forward to facilitate the training process. By adding the slice-level learning and a weighted loss, the multi-level training strategy effectively alleviates the problems of vanishing gradient and the class imbalance. The proposed training strategy could be applied to most of the segmentation networks, especially for complex models and on small datasets.

RESULTS

The proposed architecture is evaluated on a spontaneous ICH dataset and a traumatic ICH dataset. Compared to the previous works on the ICH sementation, the proposed architecture obtains the state-of-the-art performance(Dice of 0.950) on the spontaneous ICH, and comparable results(Dice of 0.895) with the best method on the traumatic ICH. On the other hand, the time consumption of the proposed architecture is much less than the previous methods on both training and inference. Morever, experiment results on various of models prove the universality of the multi-level training strategy.

CONCLUSIONS

This study proposed a novel CNN-based architecture, Ψ-Net with multi-level training strategy. It takes less time for training and achives superior performance than previous ICH segmentaion methods.

Evolutionary Compression of Deep Neural Networks for Biomedical Image Segmentation

Biomedical image segmentation is lately dominated by deep neural networks (DNNs) due to their surpassing expert-level performance. However, the existing DNN models for biomedical image segmentation are generally highly parameterized, which severely impede their deployment on real-time platforms and portable devices. To tackle this difficulty, we propose an evolutionary compression method (ECDNN) to automatically discover efficient DNN architectures for biomedical image segmentation. Different from the existing studies, ECDNN can optimize network loss and number of parameters simultaneously during the evolution, and search for a set of Pareto-optimal solutions in a single run, which is useful for quantifying the tradeoff in satisfying different objectives, and flexible for compressing DNN when preference information is uncertain. In particular, a set of novel genetic operators is proposed for automatically identifying less important filters over the whole network. Moreover, a pruning operator is designed for eliminating convolutional filters from layers involved in feature map concatenation, which is commonly adopted in DNN architectures for capturing multi-level features from biomedical images. Experiments carried out on compressing DNN for retinal vessel and neuronal membrane segmentation tasks show that ECDNN can not only improve the performance without any retraining but also discover efficient network architectures that well maintain the performance. The superiority of the proposed method is further validated by comparison with the state-of-the-art methods.

Temporally coherent cardiac motion tracking from cine MRI: Traditional registration method and modern CNN method

Purpose

Cardiac motion tracking enables quantitative evaluation of myocardial strain, which is clinically interesting in cardiovascular disease research. However, motion tracking is difficult to perform manually. In this paper, we aim to develop and compare two fully automated motion tracking methods for the steady state free precession (SSFP) cine magnetic resonance imaging (MRI), and explore their use in real clinical scenario with different patient groups.

Methods

We proposed two automated cardiac motion tracking method: (a) a traditional registration‐based method, named full cardiac cycle registration, which simultaneously tracks all cine frames within a full cardiac cycle by joint registration of all frames; and (b) a modern convolutional neural network (CNN)‐based method, named Groupwise MotionNet, which enhances the temporal coherence by fusing motion along a continuous time scale. Both methods were evaluated on the healthy volunteer data from the MICCAI 2011 STACOM Challenge, as well as on patient data including hypertrophic cardiomyopathy (HCM) and myocardial infarction (MI).

Results

The full cardiac cycle registration method achieved an average end‐point error (EPE) 2.89 ± 1.57 mm for cardiac motion tracking, with computation time of around 9 min per short‐axis cine MRI (size 128 × 128, 30 cardiac phases). In comparison, the Groupwise MotionNet achieved an average EPE of 0.94 ± 1.59 mm, taking < 1 s for a full cardiac phases. Further experiments showed that registration method had stable performance, independent of patient cohort and MRI machine, while the CNN‐based method relied on the training data to deliver consistently accurate results.

Conclusion

Both registration‐based and CNN‐based method can track the cardiac motion from SSFP cine MRI in a fully automated manner, while taking temporal coherence into account. The registration method is generic, robust, but relatively slow; the CNN‐based method trained with heterogeneous data was able to achieve high tracking accuracy with real‐time performance.

Sequential vessel segmentation via deep channel attention network

Accurately segmenting contrast-filled vessels from X-ray coronary angiography (XCA) image sequence is an essential step for the diagnosis and therapy of coronary artery disease. However, developing automatic vessel segmentation is particularly challenging due to the overlapping structures, low contrast and the presence of complex and dynamic background artifacts in XCA images. This paper develops a novel encoder-decoder deep network architecture which exploits the several contextual frames of 2D+t sequential images in a sliding window centered at current frame to segment 2D vessel masks from the current frame. The architecture is equipped with temporal-spatial feature extraction in encoder stage, feature fusion in skip connection layers and channel attention mechanism in decoder stage. In the encoder stage, a series of 3D convolutional layers are employed to hierarchically extract temporal-spatial features. Skip connection layers subsequently fuse the temporal-spatial feature maps and deliver them to the corresponding decoder stages. To efficiently discriminate vessel features from the complex and noisy backgrounds in the XCA images, the decoder stage effectively utilizes channel attention blocks to refine the intermediate feature maps from skip connection layers for subsequently decoding the refined features in 2D ways to produce the segmented vessel masks. Furthermore, Dice loss function is implemented to train the proposed deep network in order to tackle the class imbalance problem in the XCA data due to the wide distribution of complex background artifacts. Extensive experiments by comparing our method with other state-of-the-art algorithms demonstrate the proposed method's superior performance over other methods in terms of the quantitative metrics and visual validation. To facilitate the reproductive research in XCA community, we publicly release our dataset and source codes at https://github.com/Binjie-Qin/SVS-net.

A Novel Deep Learning Pipeline for Retinal Vessel Detection In Fluorescein Angiography

While recent advances in deep learning have significantly advanced the state of the art for vessel detection in color fundus (CF) images, the success for detecting vessels in fluorescein angiography (FA) has been stymied due to the lack of labeled ground truth datasets. We propose a novel pipeline to detect retinal vessels in FA images using deep neural networks (DNNs) that reduces the effort required for generating labeled ground truth data by combining two key components: cross-modality transfer and human-in-the-loop learning. The cross-modality transfer exploits concurrently captured CF and fundus FA images. Binary vessels maps are first detected from CF images with a pre-trained neural network and then are geometrically registered with and transferred to FA images via robust parametric chamfer alignment to a preliminary FA vessel detection obtained with an unsupervised technique. Using the transferred vessels as initial ground truth labels for deep learning, the human-in-the-loop approach progressively improves the quality of the ground truth labeling by iterating between deep-learning and labeling. The approach significantly reduces manual labeling effort while increasing engagement. We highlight several important considerations for the proposed methodology and validate the performance on three datasets. Experimental results demonstrate that the proposed pipeline significantly reduces the annotation effort and the resulting deep learning methods outperform prior existing FA vessel detection methods by a significant margin. A new public dataset, RECOVERY-FA19, is introduced that includes high-resolution ultra-widefield images and accurately labeled ground truth binary vessel maps.

Tissue-specific and interpretable sub-segmentation of whole tumour burden on CT images by unsupervised fuzzy clustering

BACKGROUND

Cancer typically exhibits genotypic and phenotypic heterogeneity, which can have prognostic significance and influence therapy response. Computed Tomography (CT)-based radiomic approaches calculate quantitative features of tumour heterogeneity at a mesoscopic level, regardless of macroscopic areas of hypo-dense (i.e., cystic/necrotic), hyper-dense (i.e., calcified), or intermediately dense (i.e., soft tissue) portions.

METHOD

With the goal of achieving the automated sub-segmentation of these three tissue types, we present here a two-stage computational framework based on unsupervised Fuzzy C-Means Clustering (FCM) techniques. No existing approach has specifically addressed this task so far. Our tissue-specific image sub-segmentation was tested on ovarian cancer (pelvic/ovarian and omental disease) and renal cell carcinoma CT datasets using both overlap-based and distance-based metrics for evaluation.

RESULTS

On all tested sub-segmentation tasks, our two-stage segmentation approach outperformed conventional segmentation techniques: fixed multi-thresholding, the Otsu method, and automatic cluster number selection heuristics for the K-means clustering algorithm. In addition, experiments showed that the integration of the spatial information into the FCM algorithm generally achieves more accurate segmentation results, whilst the kernelised FCM versions are not beneficial. The best spatial FCM configuration achieved average Dice similarity coefficient values starting from 81.94±4.76 and 83.43±3.81 for hyper-dense and hypo-dense components, respectively, for the investigated sub-segmentation tasks.

CONCLUSIONS

The proposed intelligent framework could be readily integrated into clinical research environments and provides robust tools for future radiomic biomarker validation.

A new robust method for blood vessel segmentation in retinal fundus images based on weighted line detector and hidden Markov model

BACKGROUND AND OBJECTIVE

Automatic vessel segmentation is a crucial preliminary processing step to facilitate ophthalmologist diagnosis in some diseases. But, due to the complexity of retinal fundus image, there are some problems on accurate segmentation of retinal vessel. In this paper, a new method for retinal vessel segmentation is proposed to handle two main problems: thin vessel missing and false detection in difficult regions.

METHODS

First, an improved line detector is proposed and used to fast extract the major structures of vessels. Then, Hidden Markov model (HMM) is applied to effectively detect vessel centerlines that include thin vessels. Finally, a denoising approach is presented to remove noises and two types of vessels are unified to obtain the complete segmentation results.

RESULTS

Our method is tested on two public databases (DRIVE and STARE databases), and five measures namely accuracy (Acc), sensitivity (Se), specificity (Sp), Dice coefficient (Dc), structural similarity index (SSIM) and feature similarity index (FSIM) are used to evaluate our segmentation performance. The respective values of the performance measures are 0.9475, 0.7262, 0.9803, 0.7781, 0.9992 and 0.9793 for DRIVE dataset and 0.9535, 0.7865, 0.9730, 0.7764, 0.9987 and 0.9742 for STARE dataset.

CONCLUSIONS

The experiment results show that our method outperforms most published state-of-the-art methods and is better the result of a human observer. Moreover, in term of specificity, our proposed algorithm can obtain the best score among the unsupervised methods. Meanwhile, there are excellent structure and feature similarities between our result and the ground truth according to achieved SSIM and FSIM. Visual inspection on the segmentation results shows that the proposed method produces more accurate segmentations on some difficult regions such as optic disc and central light reflex while detecting thin vessels effectively compared with the other methods.

DeepBranch: Deep Neural Networks for Branch Point Detection in Biomedical Images

Morphology reconstruction of tree-like structures in volumetric images, such as neurons, retinal blood vessels, and bronchi, is of fundamental interest for biomedical research. 3D branch points play an important role in many reconstruction applications, especially for graph-based or seed-based reconstruction methods and can help to visualize the morphology structures. There are a few hand-crafted models proposed to detect the branch points. However, they are highly dependent on the empirical setting of the parameters for different images. In this paper, we propose a DeepBranch model for branch point detection with two-level designed convolutional networks, a candidate region segmenter and a false positive reducer. On the first level, an improved 3D U-Net model with anisotropic convolution kernels is employed to detect initial candidates. Compared with the traditional sliding window strategy, the improved 3D U-Net can avoid massive redundant computations and dramatically speed up the detection process by employing dense-inference with fully convolutional neural networks (FCN). On the second level, a method based on multi-scale multi-view convolutional neural networks (MSMV-Net) is proposed for false positive reduction by feeding multi-scale views of 3D volumes into multiple streams of 2D convolution neural networks (CNNs), which can take full advantage of spatial contextual information as well as fit different sizes. Experiments on multiple 3D biomedical images of neurons, retinal blood vessels and bronchi confirm that the proposed 3D branch point detection method outperforms other state-of-the-art detection methods, and is helpful for graph-based or seed-based reconstruction methods.

A neural network approach to segment brain blood vessels in digital subtraction angiography

BACKGROUND AND OBJECTIVE

Cerebrovascular diseases (CVDs) affect a large number of patients and often have devastating outcomes. The hallmarks of CVDs are the abnormalities formed on brain blood vessels, including protrusions, narrows, widening, and bifurcation of the blood vessels. CVDs are often diagnosed by digital subtraction angiography (DSA) yet the interpretation of DSA is challenging as one must carefully examine each brain blood vessel. The objective of this work is to develop a computerized analysis approach for automated segmentation of brain blood vessels.

METHODS

In this work, we present a U-net based deep learning approach, combined with pre-processing, to track and segment brain blood vessels in DSA images. We compared the results given by the deep learning approach with manually marked ground truth using accuracy, sensitivity, specificity, and Dice coefficient.

RESULTS

Our results showed that the proposed approach achieved an accuracy of 0.978, with a standard deviation of 0.00796, a sensitivity of 0.76 with a standard deviation of 0.096, a specificity of 0.994 with a standard deviation of 0.0036, and an average Dice coefficient was 0.8268 with a standard deviation of 0.052.

CONCLUSIONS

Our findings show that the deep learning approach can achieve satisfactory performance as a computer-aided analysis tool to assist clinicians in diagnosing CVDs.

Enhancement of blurry retinal image based on non-uniform contrast stretching and intensity transfer

Proper contrast and sufficient illuminance are important in clearly identifying the retinal structures, while the required quality cannot always be guaranteed due to major reasons like acquisition process and diseases. To ensure the effectiveness of enhancement, two solutions are developed for blurry retinal images with sufficient illuminance and insufficient illuminance, respectively. The proposed contrast stretching and intensity transfer are main steps in both of the two solutions. The contrast stretching is based on base-intensity removal and non-uniform addition. We assume that a base-intensity exists in an image, which mainly supports the basic illuminance but has less contribution to texture information. The base-intensity is estimated by the constrained Gaussian function and then removed. The non-uniform addition using compressed Gamma map is further developed to improve the contrast. Additionally, an effective intensity transfer strategy is introduced, which can provide required illuminance for a single channel after contrast stretching. The color correction can be achieved if the intensity transfer is performed on three channels. Results show that the proposed solutions can effectively improve the contrast and illuminance, and good visual perception for quality degraded retinal images is obtained. Illustration of contrast stretching based on a signal colour channel.

Multiloss Function Based Deep Convolutional Neural Network for Segmentation of Retinal Vasculature into Arterioles and Venules

The arterioles and venules (AV) classification of retinal vasculature is considered as the first step in the development of an automated system for analysing the vasculature biomarker association with disease prognosis. Most of the existing AV classification methods depend on the accurate segmentation of retinal blood vessels. Moreover, the unavailability of large-scale annotated data is a major hindrance in the application of deep learning techniques for AV classification. This paper presents an encoder-decoder based fully convolutional neural network for classification of retinal vasculature into arterioles and venules, without requiring the preliminary step of vessel segmentation. An optimized multiloss function is used to learn the pixel-wise and segment-wise retinal vessel labels. The proposed method is trained and evaluated on DRIVE, AVRDB, and a newly created AV classification dataset; and it attains 96%, 98%, and 97% accuracy, respectively. The new AV classification dataset is comprised of 700 annotated retinal images, which will offer the researchers a benchmark to compare their AV classification results.

Deep Retinal Image Segmentation with Regularization Under Geometric Priors

Vessel segmentation of retinal images is a key diagnostic capability in ophthalmology. This problem faces several challenges including low contrast, variable vessel size and thickness, and presence of interfering pathology such as micro-aneurysms and hemorrhages. Early approaches addressing this problem employed hand-crafted filters to capture vessel structures, accompanied by morphological post-processing. More recently, deep learning techniques have been employed with significantly enhanced segmentation accuracy. We propose a novel domain enriched deep network that consists of two components: 1) a representation network that learns geometric features specific to retinal images, and 2) a custom designed computationally efficient residual task network that utilizes the features obtained from the representation layer to perform pixel-level segmentation. The representation and task networks are jointly learned for any given training set. To obtain physically meaningful and practically effective representation filters, we propose two new constraints that are inspired by expected prior structure on these filters: 1) orientation constraint that promotes geometric diversity of curvilinear features, and 2) a data adaptive noise regularizer that penalizes false positives. Multi-scale extensions are developed to enable accurate detection of thin vessels. Experiments performed on three challenging benchmark databases under a variety of training scenarios show that the proposed prior guided deep network outperforms state of the art alternatives as measured by common evaluation metrics, while being more economical in network size and inference time.

Retinal vessel segmentation using dense U-net with multiscale inputs

A color fundus image is an image of the inner wall of the eyeball taken with a fundus camera. Doctors can observe retinal vessel changes in the image, and these changes can be used to diagnose many serious diseases such as atherosclerosis, glaucoma, and age-related macular degeneration. Automated segmentation of retinal vessels can facilitate more efficient diagnosis of these diseases. We propose an improved U-net architecture to segment retinal vessels. Multiscale input layer and dense block are introduced into the conventional U-net, so that the network can make use of richer spatial context information. The proposed method is evaluated on the public dataset DRIVE, achieving 0.8199 in sensitivity and 0.9561 in accuracy. Especially for thin blood vessels, which are difficult to detect because of their low contrast with the background pixels, the segmentation results have been improved.

Deep vessel segmentation by learning graphical connectivity

We propose a novel deep learning based system for vessel segmentation. Existing methods using CNNs have mostly relied on local appearances learned on the regular image grid, without consideration of the graphical structure of vessel shape. Effective use of the strong relationship that exists between vessel neighborhoods can help improve the vessel segmentation accuracy. To this end, we incorporate a graph neural network into a unified CNN architecture to jointly exploit both local appearances and global vessel structures. We extensively perform comparative evaluations on four retinal image datasets and a coronary artery X-ray angiography dataset, showing that the proposed method outperforms or is on par with current state-of-the-art methods in terms of the average precision and the area under the receiver operating characteristic curve. Statistical significance on the performance difference between the proposed method and each comparable method is suggested by conducting a paired t-test. In addition, ablation studies support the particular choices of algorithmic detail and hyperparameter values of the proposed method. The proposed architecture is widely applicable since it can be applied to expand any type of CNN-based vessel segmentation method to enhance the performance.