HyperDense-Net: A Hyper-Densely Connected CNN for Multi-Modal Image Segmentation

A Review of Publicly Available Automatic Brain Segmentation Methodologies, Machine Learning Models, Recent Advancements, and Their Comparison

BACKGROUND

The noninvasive study of the structure and functions of the brain using neuroimaging techniques is increasingly being used for its clinical and research perspective. The morphological and volumetric changes in several regions and structures of brains are associated with the prognosis of neurological disorders such as Alzheimer's disease, epilepsy, schizophrenia, etc. and the early identification of such changes can have huge clinical significance. The accurate segmentation of three-dimensional brain magnetic resonance images into tissue types (i.e., grey matter, white matter, cerebrospinal fluid) and brain structures, thus, has huge importance as they can act as early biomarkers. The manual segmentation though considered the "gold standard" is time-consuming, subjective, and not suitable for bigger neuroimaging studies. Several automatic segmentation tools and algorithms have been developed over the years; the machine learning models particularly those using deep convolutional neural network (CNN) architecture are increasingly being applied to improve the accuracy of automatic methods.

PURPOSE

The purpose of the study is to understand the current and emerging state of automatic segmentation tools, their comparison, machine learning models, their reliability, and shortcomings with an intent to focus on the development of improved methods and algorithms.

METHODS

The study focuses on the review of publicly available neuroimaging tools, their comparison, and emerging machine learning models particularly those based on CNN architecture developed and published during the last five years.

CONCLUSION

Several software tools developed by various research groups and made publicly available for automatic segmentation of the brain show variability in their results in several comparison studies and have not attained the level of reliability required for clinical studies. The machine learning models particularly three dimensional fully convolutional network models can provide a robust and efficient alternative with relation to publicly available tools but perform poorly on unseen datasets. The challenges related to training, computation cost, reproducibility, and validation across distinct scanning modalities for machine learning models need to be addressed.

Uncertainty Aware Temporal-Ensembling Model for Semi-Supervised ABUS Mass Segmentation

Accurate breast mass segmentation of automated breast ultrasound (ABUS) images plays a crucial role in 3D breast reconstruction which can assist radiologists in surgery planning. Although the convolutional neural network has great potential for breast mass segmentation due to the remarkable progress of deep learning, the lack of annotated data limits the performance of deep CNNs. In this article, we present an uncertainty aware temporal ensembling (UATE) model for semi-supervised ABUS mass segmentation. Specifically, a temporal ensembling segmentation (TEs) model is designed to segment breast mass using a few labeled images and a large number of unlabeled images. Considering the network output contains correct predictions and unreliable predictions, equally treating each prediction in pseudo label update and loss calculation may degrade the network performance. To alleviate this problem, the uncertainty map is estimated for each image. Then an adaptive ensembling momentum map and an uncertainty aware unsupervised loss are designed and integrated with TEs model. The effectiveness of the proposed UATE model is mainly verified on an ABUS dataset of 107 patients with 170 volumes, including 13382 2D labeled slices. The Jaccard index (JI), Dice similarity coefficient (DSC), pixel-wise accuracy (AC) and Hausdorff distance (HD) of the proposed method on testing set are 63.65%, 74.25%, 99.21% and 3.81mm respectively. Experimental results demonstrate that our semi-supervised method outperforms the fully supervised method, and get a promising result compared with existing semi-supervised methods.

DIKA-Nets: Domain-invariant knowledge-guided attention networks for brain skull stripping of early developing macaques

As non-human primates, macaques have a close phylogenetic relationship to human beings and have been proven to be a valuable and widely used animal model in human neuroscience research. Accurate skull stripping (aka. brain extraction) of brain magnetic resonance imaging (MRI) is a crucial prerequisite in neuroimaging analysis of macaques. Most of the current skull stripping methods can achieve satisfactory results for human brains, but when applied to macaque brains, especially during early brain development, the results are often unsatisfactory. In fact, the early dynamic, regionally-heterogeneous development of macaque brains, accompanied by poor and age-related contrast between different anatomical structures, poses significant challenges for accurate skull stripping. To overcome these challenges, we propose a fully-automated framework to effectively fuse the age-specific intensity information and domain-invariant prior knowledge as important guiding information for robust skull stripping of developing macaques from 0 to 36 months of age. Specifically, we generate Signed Distance Map (SDM) and Center of Gravity Distance Map (CGDM) based on the intermediate segmentation results as guidance. Instead of using local convolution, we fuse all information using the Dual Self-Attention Module (DSAM), which can capture global spatial and channel-dependent information of feature maps. To extensively evaluate the performance, we adopt two relatively-large challenging MRI datasets from rhesus macaques and cynomolgus macaques, respectively, with a total of 361 scans from two different scanners with different imaging protocols. We perform cross-validation by using one dataset for training and the other one for testing. Our method outperforms five popular brain extraction tools and three deep-learning-based methods on cross-source MRI datasets without any transfer learning.

Ultra-Dense Denoising Network: Application to Cardiac Catheter-Based X-Ray Procedures

Reducing radiation dose in cardiac catheter-based X-ray procedures increases safety but also image noise and artifacts. Excessive noise and artifacts can compromise vital image information, which can affect clinical decision-making. Developing more effective X-ray denoising methodologies will be beneficial to both patients and healthcare professionals by allowing imaging at lower radiation dose without compromising image information. This paper proposes a framework based on a convolutional neural network (CNN), namely Ultra-Dense Denoising Network (UDDN), for low-dose X-ray image denoising. To promote feature extraction, we designed a novel residual block which establishes a solid correlation among multiple-path neural units via abundant cross connections in its representation enhancement section. Experiments on synthetic additive noise X-ray data show that the UDDN achieves statistically significant higher peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) than other comparative methods. We enhanced the clinical adaptability of our framework by training using normally-distributed noise and tested on clinical data taken from procedures at St. Thomas' hospital in London. The performance was assessed by using local SNR and by clinical voting using ten cardiologists. The results show that the UDDN outperforms the other comparative methods and is a promising solution to this challenging but clinically impactful task.

AMD-GAN: Attention encoder and multi-branch structure based generative adversarial networks for fundus disease detection from scanning laser ophthalmoscopy images

The scanning laser ophthalmoscopy (SLO) has become an important tool for the determination of peripheral retinal pathology, in recent years. However, the collected SLO images are easily interfered by the eyelash and frame of the devices, which heavily affect the key feature extraction of the images. To address this, we propose a generative adversarial network called AMD-GAN based on the attention encoder (AE) and multi-branch (MB) structure for fundus disease detection from SLO images. Specifically, the designed generator consists of two parts: the AE and generation flow network, where the real SLO images are encoded by the AE module to extract features and the generation flow network to handle the random Gaussian noise by a series of residual block with up-sampling (RU) operations to generate fake images with the same size as the real ones, where the AE is also used to mine features for generator. For discriminator, a ResNet network using MB is devised by copying the stage 3 and stage 4 structures of the ResNet-34 model to extract deep features. Furthermore, the depth-wise asymmetric dilated convolution is leveraged to extract local high-level contextual features and accelerate the training process. Besides, the last layer of discriminator is modified to build the classifier to detect the diseased and normal SLO images. In addition, the prior knowledge of experts is utilized to improve the detection results. Experimental results on the two local SLO datasets demonstrate that our proposed method is promising in detecting the diseased and normal SLO images with the experts labeling.

Block Level Skip Connections Across Cascaded V-Net for Multi-Organ Segmentation

Multi-organ segmentation is a challenging task due to the label imbalance and structural differences between different organs. In this work, we propose an efficient cascaded V-Net model to improve the performance of multi-organ segmentation by establishing dense Block Level Skip Connections (BLSC) across cascaded V-Net. Our model can take full advantage of features from the first stage network and make the cascaded structure more efficient. We also combine stacked small and large kernels with an inception-like structure to help our model to learn more patterns, which produces superior results for multi-organ segmentation. In addition, some small organs are commonly occluded by large organs and have unclear boundaries with other surrounding tissues, which makes them hard to be segmented. We therefore first locate the small organs through a multi-class network and crop them randomly with the surrounding region, then segment them with a single-class network. We evaluated our model on SegTHOR 2019 challenge unseen testing set and Multi-Atlas Labeling Beyond the Cranial Vault challenge validation set. Our model has achieved an average dice score gain of 1.62 percents and 3.90 percents compared to traditional cascaded networks on these two datasets, respectively. For hard-to-segment small organs, such as the esophagus in SegTHOR 2019 challenge, our technique has achieved a gain of 5.63 percents on dice score, and four organs in Multi-Atlas Labeling Beyond the Cranial Vault challenge have achieved a gain of 5.27 percents on average dice score.

ψ-Net: Stacking Densely Convolutional LSTMs for Sub-Cortical Brain Structure Segmentation

Sub-cortical brain structure segmentation is of great importance for diagnosing neuropsychiatric disorders. However, developing an automatic approach to segmenting sub-cortical brain structures remains very challenging due to the ambiguous boundaries, complex anatomical structures, and large variance of shapes. This paper presents a novel deep network architecture, namely Ψ -Net, for sub-cortical brain structure segmentation, aiming at selectively aggregating features and boosting the information propagation in a deep convolutional neural network (CNN). To achieve this, we first formulate a densely convolutional LSTM module (DC-LSTM) to selectively aggregate the convolutional features with the same spatial resolution at the same stage of a CNN. This helps to promote the discriminativeness of features at each CNN stage. Second, we stack multiple DC-LSTMs from the deepest stage to the shallowest stage to progressively enrich low-level feature maps with high-level context. We employ two benchmark datasets on sub-cortical brain structure segmentation, and perform various experiments to evaluate the proposed Ψ -Net. The experimental results show that our network performs favorably against the state-of-the-art methods on both benchmark datasets.

A Method of DC Arc Detection in All-Electric Aircraft

Arc faults in an aircraft’s power distribution system (PDS) often leads to cable and equipment damage, which seriously threatens the personal safety of the passengers and pilots. An accurate and real-time arc fault detection method is needed for the Solid-State Power Controller (SSPC), which is a key protection equipment in a PDS. In this paper, a new arc detection method is proposed based on the improved LeNet5 Convolutional Neural Network (CNN) model after a Time–Frequency Analysis (TFA) of the DC currents was obtained, which makes the arc detection more real-time. The CNN is proposed to detect the DC arc fault for its advantage in recognizing more time–frequency joint details in the signals; the new structure also combines the adaptive and multidimensional advantages of the TFA and image intelligent recognition. It is confirmed by experimental data that the combined TFA–CNN can distinguish arc faults accurately when the whole training database has been repeatedly trained 3 to 5 times. For the TFA, two kinds of methods were compared, the Short-Time Fourier Transform (STFT) and Discrete Wavelet Transform (DWT). The results show that DWT is more suitable for DC arc fault detection. The experimental results demonstrated the effectiveness of the proposed method.

Multi-modality self-attention aware deep network for 3D biomedical segmentation

BACKGROUND

Deep learning based on segmentation models have been gradually applied in biomedical images and achieved state-of-the-art performance for 3D biomedical segmentation. However, most of existing biomedical segmentation researches take account of the application cases with adapting a single type of medical images from the corresponding examining method. Considering of practical clinic application of the radiology examination for diseases, the multiple image examination methods are normally required for final diagnosis especially in some severe diseases like cancers. Therefore, by considering the cases of employing multi-modal images and exploring the effective multi-modality fusion based on deep networks, we do the research to make full use of complementary information of multi-modal images referring to the clinic experiences of radiologists in image analysis.

METHODS

Referring to the human radiologist diagnosis experience, we discuss and propose a new self-attention aware mechanism to improve the segmentation performance by paying the different attention on different modal images and different symptoms. Firstly, we propose a multi-path encoder and decoder deep network for 3D biomedical segmentation. Secondly, to leverage the complementary information among different modalities, we introduce a structure of attention mechanism called the Multi-Modality Self-Attention Aware (MMSA) convolution. Multi-modal images we used in the paper are different modalities of MR scanning images, which are input into the network separately. Then self-attention weight fusion of multi-modal features is performed with our proposed MMSA, which can adaptively adjust the fusion weights according to the learned contribution degree of different modalities and different features revealing the different symptoms from the labeled data.

RESULTS

Experiments have been done on the public competition dataset BRATS-2015. The results show that our proposed method achieves dice scores of 0.8726, 0.6563, 0.8313 for the whole tumor, the tumor core and the enhancing tumor core, respectively. Comparing with the U-Net with SE block, the scores are increased by 0.0212,0.031,0.0304.

CONCLUSIONS

We present a multi-modality self-attention aware convolution, which have better segmentation results based on the adaptive weighting fusion mechanism with exploiting the multiple medical image modalities. Experimental results demonstrate the effectiveness of our method and prominent application in the multi-modality fusion based medical image analysis.

Unpaired Multi-Modal Segmentation via Knowledge Distillation

Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model, we introduce a novel loss term inspired by knowledge distillation, by explicitly constraining the KL-divergence of our derived prediction distributions between modalities. We have extensively validated our approach on two multi-class segmentation problems: i) cardiac structure segmentation, and ii) abdominal organ segmentation. Different network settings, i.e., 2D dilated network and 3D U-net, are utilized to investigate our method's general efficacy. Experimental results on both tasks demonstrate that our novel multi-modal learning scheme consistently outperforms single-modal training and previous multi-modal approaches.

A 3D Spatially Weighted Network for Segmentation of Brain Tissue From MRI

The segmentation of brain tissue in MRI is valuable for extracting brain structure to aid diagnosis, treatment and tracking the progression of different neurologic diseases. Medical image data are volumetric and some neural network models for medical image segmentation have addressed this using a 3D convolutional architecture. However, this volumetric spatial information has not been fully exploited to enhance the representative ability of deep networks, and these networks have not fully addressed the practical issues facing the analysis of multimodal MRI data. In this paper, we propose a spatially-weighted 3D network (SW-3D-UNet) for brain tissue segmentation of single-modality MRI, and extend it using multimodality MRI data. We validate our model on the MRBrainS13 and MALC12 datasets. This unpublished model ranked first on the leaderboard of the MRBrainS13 Challenge.

Using Deep Convolutional Neural Networks for Neonatal Brain Image Segmentation

INTRODUCTION

Deep learning neural networks are especially potent at dealing with structured data, such as images and volumes. Both modified LiviaNET and HyperDense-Net performed well at a prior competition segmenting 6-month-old infant magnetic resonance images, but neonatal cerebral tissue type identification is challenging given its uniquely inverted tissue contrasts. The current study aims to evaluate the two architectures to segment neonatal brain tissue types at term equivalent age.

METHODS

Both networks were retrained over 24 pairs of neonatal T1 and T2 data from the Developing Human Connectome Project public data set and validated on another eight pairs against ground truth. We then reported the best-performing model from training and its performance by computing the Dice similarity coefficient (DSC) for each tissue type against eight test subjects.

RESULTS

During the testing phase, among the segmentation approaches tested, the dual-modality HyperDense-Net achieved the best statistically significantly test mean DSC values, obtaining 0.94/0.95/0.92 for the tissue types and took 80 h to train and 10 min to segment, including preprocessing. The single-modality LiviaNET was better at processing T2-weighted images than processing T1-weighted images across all tissue types, achieving mean DSC values of 0.90/0.90/0.88 for gray matter, white matter, and cerebrospinal fluid, respectively, while requiring 30 h to train and 8 min to segment each brain, including preprocessing.

DISCUSSION

Our evaluation demonstrates that both neural networks can segment neonatal brains, achieving previously reported performance. Both networks will be continuously retrained over an increasingly larger repertoire of neonatal brain data and be made available through the Canadian Neonatal Brain Platform to better serve the neonatal brain imaging research community.

Multiple Axial Spine Indices Estimation via Dense Enhancing Network With Cross-Space Distance-Preserving Regularization

Automatic estimation of axial spine indices is clinically desired for various spine computer aided procedures, such as disease diagnosis, therapeutic evaluation, pathophysiological understanding, risk assessment, and biomechanical modeling. Currently, the spine indices are manually measured by physicians, which is time-consuming and laborious. Even worse, the tedious manual procedure might result in inaccurate measurement. To deal with this problem, in this paper, we aim at developing an automatic method to estimate multiple indices from axial spine images. Inspired by the success of deep learning for regression problems and the densely connected network for image classification, we propose a dense enhancing network (DE-Net) which uses the dense enhancing blocks (DEBs) as its main body, where a feature enhancing layer is added to each of the bypass in a dense block. The DEB is designed to enhance discriminative feature embedding from the intervertebral disc and the dural sac areas. In addition, the cross-space distance-preserving regularization (CSDPR), which enforces consistent inter-sample distances between the output and the label spaces, is proposed to regularize the loss function of the DE-Net. To train and validate the proposed method, we collected 895 axial spine MRI images from 143 subjects and manually measured the indices as the ground truth. The results show that all deep learning models obtain very small prediction errors, and the proposed DE-Net with CSDPR acquires the smallest error among all methods, indicating that our method has great potential for spine computer aided procedures.

Recalibrating 3D ConvNets with Project & Excite

Fully Convolutional Neural Networks (F-CNNs) achieve state-of-the-art performance for segmentation tasks in computer vision and medical imaging. Recently, computational blocks termed squeeze and excitation (SE) have been introduced to recalibrate F-CNN feature maps both channel- and spatial-wise, boosting segmentation performance while only minimally increasing the model complexity. So far, the development of SE blocks has focused on 2D architectures. For volumetric medical images, however, 3D F-CNNs are a natural choice. In this article, we extend existing 2D recalibration methods to 3D and propose a generic compress-process-recalibrate pipeline for easy comparison of such blocks. We further introduce Project & Excite (PE) modules, customized for 3D networks. In contrast to existing modules, Project & Excite does not perform global average pooling but compresses feature maps along different spatial dimensions of the tensor separately to retain more spatial information that is subsequently used in the excitation step. We evaluate the modules on two challenging tasks, whole-brain segmentation of MRI scans and whole-body segmentation of CT scans. We demonstrate that PE modules can be easily integrated into 3D F-CNNs, boosting performance up to 0.3 in Dice Score and outperforming 3D extensions of other recalibration blocks, while only marginally increasing the model complexity. Our code is publicly available on https://github.com/ai-med/squeezeandexcitation.

Multilevel and Multiscale Feature Aggregation in Deep Networks for Facial Constitution Classification

Constitution classification is the basis and core content of TCM constitution research. In order to improve the accuracy of constitution classification, this paper proposes a multilevel and multiscale features aggregation method within the convolutional neural network, which consists of four steps. First, it uses the pretrained VGG16 as the basic network and then refines the network structure through supervised feature learning so as to capture local image features. Second, it extracts the image features of different layers from the fine-tuned VGG16 model, which are then dimensionally reduced by principal component analysis (PCA). Third, it uses another pretrained NASNetMobile network for supervised feature learning, where the previous layer features of the global average pooling layer are outputted. Similarly, these features are dimensionally reduced by PCA and then are fused with the features of different layers in VGG16 after the PCA. Finally, all features are aggregated with the fully connected layers of the fine-tuned VGG16, and then the constitution classification is performed. The conducted experiments show that using the multilevel and multiscale feature aggregation is very effective in the constitution classification, and the accuracy on the test dataset reaches 69.61%.

Deep neural network for automatic characterization of lesions on 68Ga-PSMA-11 PET/CT

PURPOSE

This study proposes an automated prostate cancer (PC) lesion characterization method based on the deep neural network to determine tumor burden on ⁶⁸Ga-PSMA-11 PET/CT to potentially facilitate the optimization of PSMA-directed radionuclide therapy.

METHODS

We collected ⁶⁸Ga-PSMA-11 PET/CT images from 193 patients with metastatic PC at three medical centers. For proof-of-concept, we focused on the detection of pelvis bone and lymph node lesions. A deep neural network (triple-combining 2.5D U-Net) was developed for the automated characterization of these lesions. The proposed method simultaneously extracts features from axial, coronal, and sagittal planes, which mimics the workflow of physicians and reduces computational and memory requirements.

RESULTS

Among all the labeled lesions, the network achieved 99% precision, 99% recall, and an F1 score of 99% on bone lesion detection and 94%, precision 89% recall, and an F1 score of 92% on lymph node lesion detection. The segmentation accuracy is lower than the detection. The performance of the network was correlated with the amount of training data.

CONCLUSION

We developed a deep neural network to characterize automatically the PC lesions on ⁶⁸Ga-PSMA-11 PET/CT. The preliminary test within the pelvic area confirms the potential of deep learning methods. Increasing the amount of training data should further enhance the performance of the proposed method and may ultimately allow whole-body assessments.

Deep CNN ensembles and suggestive annotations for infant brain MRI segmentation

Precise 3D segmentation of infant brain tissues is an essential step towards comprehensive volumetric studies and quantitative analysis of early brain development. However, computing such segmentations is very challenging, especially for 6-month infant brain, due to the poor image quality, among other difficulties inherent to infant brain MRI, e.g., the isointense contrast between white and gray matter and the severe partial volume effect due to small brain sizes. This study investigates the problem with an ensemble of semi-dense fully convolutional neural networks (CNNs), which employs T1-weighted and T2-weighted MR images as input. We demonstrate that the ensemble agreement is highly correlated with the segmentation errors. Therefore, our method provides measures that can guide local user corrections. To the best of our knowledge, this work is the first ensemble of 3D CNNs for suggesting annotations within images. Our quasi-dense architecture allows the efficient propagation of gradients during training, while limiting the number of parameters, requiring one order of magnitude less parameters than popular medical image segmentation networks such as 3D U-Net (Çiçek, et al.). We also investigated the impact that early or late fusions of multiple image modalities might have on the performances of deep architectures. We report evaluations of our method on the public data of the MICCAI iSEG-2017 Challenge on 6-month infant brain MRI segmentation, and show very competitive results among 21 teams, ranking first or second in most metrics.

Accurate and robust segmentation of neuroanatomy in T1-weighted MRI by combining spatial priors with deep convolutional neural networks

Neuroanatomical segmentation in magnetic resonance imaging (MRI) of the brain is a prerequisite for quantitative volume, thickness, and shape measurements, as well as an important intermediate step in many preprocessing pipelines. This work introduces a new highly accurate and versatile method based on 3D convolutional neural networks for the automatic segmentation of neuroanatomy in T1‐weighted MRI. In combination with a deep 3D fully convolutional architecture, efficient linear registration‐derived spatial priors are used to incorporate additional spatial context into the network. An aggressive data augmentation scheme using random elastic deformations is also used to regularize the networks, allowing for excellent performance even in cases where only limited labeled training data are available. Applied to hippocampus segmentation in an elderly population (mean Dice coefficient = 92.1%) and subcortical segmentation in a healthy adult population (mean Dice coefficient = 89.5%), we demonstrate new state‐of‐the‐art accuracies and a high robustness to outliers. Further validation on a multistructure segmentation task in a scan–rescan dataset demonstrates accuracy (mean Dice coefficient = 86.6%) similar to the scan–rescan reliability of expert manual segmentations (mean Dice coefficient = 86.9%), and improved reliability compared to both expert manual segmentations and automated segmentations using FIRST. Furthermore, our method maintains a highly competitive runtime performance (e.g., requiring only 10 s for left/right hippocampal segmentation in 1 × 1 × 1 mm³ MNI stereotaxic space), orders of magnitude faster than conventional multiatlas segmentation methods.