HALSR-Net: Improving CNN Segmentation of Cardiac Left Ventricle MRI with Hybrid Attention and Latent Space Reconstruction

Accurate cardiac MRI segmentation is vital for detailed cardiac analysis, yet the manual process is labor-intensive and prone to variability. Despite advancements in MRI technology, there remains a significant need for automated methods that can reliably and efficiently segment cardiac structures. This paper introduces HALSR-Net, a novel multi-level segmentation architecture designed to improve the accuracy and reproducibility of cardiac segmentation from Cine-MRI acquisitions, focusing on the left ventricle (LV). The methodology consists of two main phases: first, the extraction of the region of interest (ROI) using a regression model that accurately predicts the location of a bounding box around the LV; second, the semantic segmentation step based on HALSR-Net architecture. This architecture incorporates a Hybrid Attention Pooling Module (HAPM) that merges attention and pooling mechanisms to enhance feature extraction and capture contextual information. Additionally, a reconstruction module leverages latent space features to further improve segmentation accuracy. Experiments conducted on an in-house clinical dataset and two public datasets (ACDC and LVQuan19) demonstrate that HALSR-Net outperforms state-of-the-art architectures, achieving up to 98% accuracy and F1-score for the segmentation of the LV cavity and myocardium. The proposed approach effectively addresses the limitations of existing methods, offering a more accurate and robust solution for cardiac MRI segmentation, thereby likely to improve cardiac function analysis and patient care.

Adaptive dynamic inference for few-shot left atrium segmentation

Accurate segmentation of the left atrium (LA) from late gadolinium-enhanced cardiac magnetic resonance (LGE CMR) images is crucial for aiding the treatment of patients with atrial fibrillation. Few-shot learning holds significant potential for achieving accurate LA segmentation with low demand on high-cost labeled LGE CMR data and fast generalization across different centers. However, accurate LA segmentation with few-shot learning is a challenging task due to the low-intensity contrast between the LA and other neighboring organs in LGE CMR images. To address this issue, we propose an Adaptive Dynamic Inference Network (ADINet) that explicitly models the differences between the foreground and background. Specifically, ADINet leverages dynamic collaborative inference (DCI) and dynamic reverse inference (DRI) to adaptively allocate semantic-aware and spatial-specific convolution weights and indication information. These allocations are conditioned on the support foreground and background knowledge, utilizing pixel-wise correlations, for different spatial positions of query images. The convolution weights adapt to different visual patterns based on spatial positions, enabling effective encoding of differences between foreground and background regions. Meanwhile, the indication information adapts to the background visual pattern to reversely decode foreground LA regions, leveraging their spatial complementarity. To promote the learning of ADINet, we propose hierarchical supervision, which enforces spatial consistency and differences between the background and foreground regions through pixel-wise semantic supervision and pixel-pixel correlation supervision. We demonstrated the performance of ADINet on three LGE CMR datasets from different centers. Compared to state-of-the-art methods with ten available samples, ADINet yielded better segmentation performance in terms of four metrics.

Self-adaptive deep learning-based segmentation for universal and functional clinical and preclinical CT image analysis

BACKGROUND

Methods to monitor cardiac functioning non-invasively can accelerate preclinical and clinical research into novel treatment options for heart failure. However, manual image analysis of cardiac substructures is resource-intensive and error-prone. While automated methods exist for clinical CT images, translating these to preclinical μCT data is challenging. We employed deep learning to automate the extraction of quantitative data from both CT and μCT images.

METHODS

We collected a public dataset of cardiac CT images of human patients, as well as acquired μCT images of wild-type and accelerated aging mice. The left ventricle, myocardium, and right ventricle were manually segmented in the μCT training set. After template-based heart detection, two separate segmentation neural networks were trained using the nnU-Net framework.

RESULTS

The mean Dice score of the CT segmentation results (0.925 ± 0.019, n = 40) was superior to those achieved by state-of-the-art algorithms. Automated and manual segmentations of the μCT training set were nearly identical. The estimated median Dice score (0.940) of the test set results was comparable to existing methods. The automated volume metrics were similar to manual expert observations. In aging mice, ejection fractions had significantly decreased, and myocardial volume increased by age 24 weeks.

CONCLUSIONS

With further optimization, automated data extraction expands the application of (μ)CT imaging, while reducing subjectivity and workload. The proposed method efficiently measures the left and right ventricular ejection fraction and myocardial mass. With uniform translation between image types, cardiac functioning in diastolic and systolic phases can be monitored in both animals and humans.

Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation

Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches. Recent success of deep learning-based segmentation methods usually relies on a large amount of labeled data, which is particularly difficult and costly to obtain, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. Semi-supervised learning has emerged as an appealing strategy and been widely applied to medical image segmentation tasks to train deep models with limited annotations. In this paper, we present a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarize both the technical novelties and empirical results. Furthermore, we analyze and discuss the limitations and several unsolved problems of existing approaches. We hope this review can inspire the research community to explore solutions to this challenge and further advance the field of medical image segmentation.

Computing thickness of irregularly-shaped thin walls using a locally semi-implicit scheme with extrapolation to solve the Laplace equation: Application to the right ventricle

Cardiac Magnetic Resonance (CMR) Imaging is currently considered the gold standard imaging modality in cardiology. However, it is accompanied by a tradeoff between spatial resolution and acquisition time. Providing accurate measures of thin walls relative to the image resolution may prove challenging. One such anatomical structure is the cardiac right ventricle. Methods for measuring thickness of wall-like anatomical structures often rely on the Laplace equation to provide point-to-point correspondences between both boundaries. This work presents limex, a novel method to solve the Laplace equation using ghost nodes and providing extrapolated values, which is tested on three different datasets: a mathematical phantom, a set of biventricular segmentations from CMR images of ten pigs and the database used at the RV Segmentation Challenge held at MICCAI'12. Thickness measurements using the proposed methodology are more accurate than state-of-the-art methods, especially with the coarsest image resolutions, yielding mean L1 norms of the error between 43.28% and 86.52% lower than the second-best methods on the different test datasets. It is also computationally affordable. Limex has outperformed other state-of-the-art methods in classifying RV myocardial segments by their thickness.

Anatomical Prior-Based Automatic Segmentation for Cardiac Substructures from Computed Tomography Images

Cardiac substructure segmentation is a prerequisite for cardiac diagnosis and treatment, providing a basis for accurate calculation, modeling, and analysis of the entire cardiac structure. CT (computed tomography) imaging can be used for a noninvasive qualitative and quantitative evaluation of the cardiac anatomy and function. Cardiac substructures have diverse grayscales, fuzzy boundaries, irregular shapes, and variable locations. We designed a deep learning-based framework to improve the accuracy of the automatic segmentation of cardiac substructures. This framework integrates cardiac anatomical knowledge; it uses prior knowledge of the location, shape, and scale of cardiac substructures and separately processes the structures of different scales. Through two successive segmentation steps with a coarse-to-fine cascaded network, the more easily segmented substructures were coarsely segmented first; then, the more difficult substructures were finely segmented. The coarse segmentation result was used as prior information and combined with the original image as the input for the model. Anatomical knowledge of the large-scale substructures was embedded into the fine segmentation network to guide and train the small-scale substructures, achieving efficient and accurate segmentation of ten cardiac substructures. Sixty cardiac CT images and ten substructures manually delineated by experienced radiologists were retrospectively collected; the model was evaluated using the DSC (Dice similarity coefficient), Recall, Precision, and the Hausdorff distance. Compared with current mainstream segmentation models, our approach demonstrated significantly higher segmentation accuracy, with accurate segmentation of ten substructures of different shapes and sizes, indicating that the segmentation framework fused with prior anatomical knowledge has superior segmentation performance and can better segment small targets in multi-target segmentation tasks.

Min-Max Similarity: A Contrastive Semi-Supervised Deep Learning Network for Surgical Tools Segmentation

A common problem with segmentation of medical images using neural networks is the difficulty to obtain a significant number of pixel-level annotated data for training. To address this issue, we proposed a semi-supervised segmentation network based on contrastive learning. In contrast to the previous state-of-the-art, we introduce Min-Max Similarity (MMS), a contrastive learning form of dual-view training by employing classifiers and projectors to build all-negative, and positive and negative feature pairs, respectively, to formulate the learning as solving a MMS problem. The all-negative pairs are used to supervise the networks learning from different views and to capture general features, and the consistency of unlabeled predictions is measured by pixel-wise contrastive loss between positive and negative pairs. To quantitatively and qualitatively evaluate our proposed method, we test it on four public endoscopy surgical tool segmentation datasets and one cochlear implant surgery dataset, which we manually annotated. Results indicate that our proposed method consistently outperforms state-of-the-art semi-supervised and fully supervised segmentation algorithms. And our semi-supervised segmentation algorithm can successfully recognize unknown surgical tools and provide good predictions. Also, our MMS approach could achieve inference speeds of about 40 frames per second (fps) and is suitable to deal with the real-time video segmentation.

BayeSeg: Bayesian modeling for medical image segmentation with interpretable generalizability

Due to the cross-domain distribution shift aroused from diverse medical imaging systems, many deep learning segmentation methods fail to perform well on unseen data, which limits their real-world applicability. Recent works have shown the benefits of extracting domain-invariant representations on domain generalization. However, the interpretability of domain-invariant features remains a great challenge. To address this problem, we propose an interpretable Bayesian framework (BayeSeg) through Bayesian modeling of image and label statistics to enhance model generalizability for medical image segmentation. Specifically, we first decompose an image into a spatial-correlated variable and a spatial-variant variable, assigning hierarchical Bayesian priors to explicitly force them to model the domain-stable shape and domain-specific appearance information respectively. Then, we model the segmentation as a locally smooth variable only related to the shape. Finally, we develop a variational Bayesian framework to infer the posterior distributions of these explainable variables. The framework is implemented with neural networks, and thus is referred to as deep Bayesian segmentation. Quantitative and qualitative experimental results on prostate segmentation and cardiac segmentation tasks have shown the effectiveness of our proposed method. Moreover, we investigated the interpretability of BayeSeg by explaining the posteriors and analyzed certain factors that affect the generalization ability through further ablation studies. Our code is released via https://zmiclab.github.io/projects.html.

Fully Automatic initialization and segmentation of left and right ventricles for large-scale cardiac MRI using a deeply supervised network and 3D-ASM

BACKGROUND AND OBJECTIVE

The sheer volume of data generated by population imaging studies is unparalleled by current capabilities to extract objective and quantitative cardiac phenotypes; subjective and time-consuming manual image analysis remains the gold standard. Automated image analytics to compute quantitative imaging biomarkers of cardiac function are desperately needed. Data volumes and their variability pose a challenge to most state-of-the-art methods for endo- and epicardial contours, which lack robustness when applied to very large datasets. Our aim is to develop an analysis pipeline for the automatic quantification of cardiac function from cine magnetic resonance imaging data.

METHOD

This work adopt 4,638 cardiac MRI cases coming from UK Biobank with ground truth available for left and RV contours. A hybrid and robust algorithm is proposed to improve the accuracy of automatic left and right ventricle segmentation by harnessing the localization accuracy of deep learning and the morphological accuracy of 3D-ASM (three-dimensional active shape models). The contributions of this paper are three-fold. First, a fully automatic method is proposed for left and right ventricle initialization and cardiac MRI segmentation by taking full advantage of spatiotemporal constraint. Second, a deeply supervised network is introduced to train and segment the heart. Third, the 3D-ASM image search procedure is improved by combining image intensity models with convolutional neural network (CNN) derived distance maps improving endo- and epicardial edge localization.

RESULTS

The proposed architecture outperformed the state of the art for cardiac MRI segmentation from UK Biobank. The statistics of RV landmarks detection errors for Triscuspid valve and RV apex are 4.17 mm and 5.58 mm separately. The overlap metric, mean contour distance, Hausdorff distance and cardiac functional parameters are calculated for the LV (Left Ventricle) and RV (Right Ventricle) contour segmentation. Bland-Altman analysis for clinical parameters shows that the results from our automated image analysis pipelines are in good agreement with results from expert manual analysis.

CONCLUSIONS

Our hybrid scheme combines deep learning and statistical shape modeling for automatic segmentation of the LV/RV from cardiac MRI datasets is effective and robust and can compute cardiac functional indexes from population imaging.

Multi-modality cardiac image computing: A survey

Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optimal methods to integrate information from different modalities. This paper aims to provide a comprehensive review of multi-modality imaging in cardiology, the computing methods, the validation strategies, the related clinical workflows and future perspectives. For the computing methodologies, we have a favored focus on the three tasks, i.e., registration, fusion and segmentation, which generally involve multi-modality imaging data, either combining information from different modalities or transferring information across modalities. The review highlights that multi-modality cardiac imaging data has the potential of wide applicability in the clinic, such as trans-aortic valve implantation guidance, myocardial viability assessment, and catheter ablation therapy and its patient selection. Nevertheless, many challenges remain unsolved, such as missing modality, modality selection, combination of imaging and non-imaging data, and uniform analysis and representation of different modalities. There is also work to do in defining how the well-developed techniques fit in clinical workflows and how much additional and relevant information they introduce. These problems are likely to continue to be an active field of research and the questions to be answered in the future.

Deep learning-based left ventricular segmentation demonstrates improved performance on respiratory motion-resolved whole-heart reconstructions

Introduction

Deep learning (DL)-based segmentation has gained popularity for routine cardiac magnetic resonance (CMR) image analysis and in particular, delineation of left ventricular (LV) borders for LV volume determination. Free-breathing, self-navigated, whole-heart CMR exams provide high-resolution, isotropic coverage of the heart for assessment of cardiac anatomy including LV volume. The combination of whole-heart free-breathing CMR and DL-based LV segmentation has the potential to streamline the acquisition and analysis of clinical CMR exams. The purpose of this study was to compare the performance of a DL-based automatic LV segmentation network trained primarily on computed tomography (CT) images in two whole-heart CMR reconstruction methods: (1) an in-line respiratory motion-corrected (Mcorr) reconstruction and (2) an off-line, compressed sensing-based, multi-volume respiratory motion-resolved (Mres) reconstruction. Given that Mres images were shown to have greater image quality in previous studies than Mcorr images, we hypothesized that the LV volumes segmented from Mres images are closer to the manual expert-traced left ventricular endocardial border than the Mcorr images.

Method

This retrospective study used 15 patients who underwent clinically indicated 1.5 T CMR exams with a prototype ECG-gated 3D radial phyllotaxis balanced steady state free precession (bSSFP) sequence. For each reconstruction method, the absolute volume difference (AVD) of the automatically and manually segmented LV volumes was used as the primary quantity to investigate whether 3D DL-based LV segmentation generalized better on Mcorr or Mres 3D whole-heart images. Additionally, we assessed the 3D Dice similarity coefficient between the manual and automatic LV masks of each reconstructed 3D whole-heart image and the sharpness of the LV myocardium-blood pool interface. A two-tail paired Student's t-test (alpha = 0.05) was used to test the significance in this study.

Results & Discussion

The AVD in the respiratory Mres reconstruction was lower than the AVD in the respiratory Mcorr reconstruction: 7.73 ± 6.54 ml vs. 20.0 ± 22.4 ml, respectively (n = 15, p-value = 0.03). The 3D Dice coefficient between the DL-segmented masks and the manually segmented masks was higher for Mres images than for Mcorr images: 0.90 ± 0.02 vs. 0.87 ± 0.03 respectively, with a p-value = 0.02. Sharpness on Mres images was higher than on Mcorr images: 0.15 ± 0.05 vs. 0.12 ± 0.04, respectively, with a p-value of 0.014 (n = 15).

Conclusion

We conclude that the DL-based 3D automatic LV segmentation network trained on CT images and fine-tuned on MR images generalized better on Mres images than on Mcorr images for quantifying LV volumes.

MRXCAT2.0: Synthesis of realistic numerical phantoms by combining left-ventricular shape learning, biophysical simulations and tissue texture generation

BACKGROUND

Standardised performance assessment of image acquisition, reconstruction and processing methods is limited by the absence of images paired with ground truth reference values. To this end, we propose MRXCAT2.0 to generate synthetic data, covering healthy and pathological function, using a biophysical model. We exemplify the approach by generating cardiovascular magnetic resonance (CMR) images of healthy, infarcted, dilated and hypertrophic left-ventricular (LV) function.

METHOD

In MRXCAT2.0, the XCAT torso phantom is coupled with a statistical shape model, describing population (patho)physiological variability, and a biophysical model, providing known and detailed functional ground truth of LV morphology and function. CMR balanced steady-state free precession images are generated using MRXCAT2.0 while realistic image appearance is ensured by assigning texturized tissue properties to the phantom labels.

FINDING

Paired CMR image and ground truth data of LV function were generated with a range of LV masses (85-140 g), ejection fractions (34-51%) and peak radial and circumferential strains (0.45 to 0.95 and - 0.18 to - 0.13, respectively). These ranges cover healthy and pathological cases, including infarction, dilated and hypertrophic cardiomyopathy. The generation of the anatomy takes a few seconds and it improves on current state-of-the-art models where the pathological representation is not explicitly addressed. For the full simulation framework, the biophysical models require approximately two hours, while image generation requires a few minutes per slice.

CONCLUSION

MRXCAT2.0 offers synthesis of realistic images embedding population-based anatomical and functional variability and associated ground truth parameters to facilitate a standardized assessment of CMR acquisition, reconstruction and processing methods.

Open-source, fully-automated hybrid cardiac substructure segmentation: development and optimisation

Radiotherapy for thoracic and breast tumours is associated with a range of cardiotoxicities. Emerging evidence suggests cardiac substructure doses may be more predictive of specific outcomes, however, quantitative data necessary to develop clinical planning constraints is lacking. Retrospective analysis of patient data is required, which relies on accurate segmentation of cardiac substructures. In this study, a novel model was designed to deliver reliable, accurate, and anatomically consistent segmentation of 18 cardiac substructures on computed tomography (CT) scans. Thirty manually contoured CT scans were included. The proposed multi-stage method leverages deep learning (DL), multi-atlas mapping, and geometric modelling to automatically segment the whole heart, cardiac chambers, great vessels, heart valves, coronary arteries, and conduction nodes. Segmentation performance was evaluated using the Dice similarity coefficient (DSC), mean distance to agreement (MDA), Hausdorff distance (HD), and volume ratio. Performance was reliable, with no errors observed and acceptable variation in accuracy between cases, including in challenging cases with imaging artefacts and atypical patient anatomy. The median DSC range was 0.81-0.93 for whole heart and cardiac chambers, 0.43-0.76 for great vessels and conduction nodes, and 0.22-0.53 for heart valves. For all structures the median MDA was below 6 mm, median HD ranged 7.7-19.7 mm, and median volume ratio was close to one (0.95-1.49) for all structures except the left main coronary artery (2.07). The fully automatic algorithm takes between 9 and 23 min per case. The proposed fully-automatic method accurately delineates cardiac substructures on radiotherapy planning CT scans. Robust and anatomically consistent segmentations, particularly for smaller structures, represents a major advantage of the proposed segmentation approach. The open-source software will facilitate more precise evaluation of cardiac doses and risks from available clinical datasets.

Learned iterative segmentation of highly variable anatomy from limited data: Applications to whole heart segmentation for congenital heart disease

Training deep learning models that segment an image in one step typically requires a large collection of manually annotated images that captures the anatomical variability in a cohort. This poses challenges when anatomical variability is extreme but training data is limited, as when segmenting cardiac structures in patients with congenital heart disease (CHD). In this paper, we propose an iterative segmentation model and show that it can be accurately learned from a small dataset. Implemented as a recurrent neural network, the model evolves a segmentation over multiple steps, from a single user click until reaching an automatically determined stopping point. We develop a novel loss function that evaluates the entire sequence of output segmentations, and use it to learn model parameters. Segmentations evolve predictably according to growth dynamics encapsulated by training data, which consists of images, partially completed segmentations, and the recommended next step. The user can easily refine the final segmentation by examining those that are earlier or later in the output sequence. Using a dataset of 3D cardiac MR scans from patients with a wide range of CHD types, we show that our iterative model offers better generalization to patients with the most severe heart malformations.

Cardiac MRI segmentation with sparse annotations: Ensembling deep learning uncertainty and shape priors

The performance of deep learning for cardiac magnetic resonance imaging (MRI) segmentation is oftentimes degraded when using small datasets and sparse annotations for training or adapting a pre-trained model to previously unseen datasets. Here, we developed and evaluated an approach to addressing some of these issues to facilitate broader use of deep learning for short-axis cardiac MRI segmentation. We developed a globally optimal label fusion (GOLF) algorithm that enforced spatial smoothness to generate consensus segmentation from segmentation predictions provided by a deep learning ensemble algorithm. The GOLF consensus was entered into an uncertainty-guided coupled continuous kernel cut (ugCCKC) algorithm that employed normalized cut, image-grid continuous regularization, and "nesting" and circular shape priors of the left ventricular myocardium (LVM) and cavity (LVC). In addition, the uncertainty measurements derived from the segmentation predictions were used to constrain the similarity of GOLF and final segmentation. We optimized ugCCKC through upper bound relaxation, for which we developed an efficient coupled continuous max-flow algorithm implemented in an iterative manner. We showed that GOLF yielded average symmetric surface distance (ASSD) 0.2-0.8 mm lower than an averaging method with higher or similar Dice similarity coefficient (DSC). We also demonstrated that ugCCKC incorporating the shape priors improved DSC by 0.01-0.05 and reduced ASSD by 0.1-0.9 mm. In addition, we integrated GOLF and ugCCKC into a deep learning ensemble algorithm by refining the segmentation of an unannotated dataset and using the refined segmentation to update the trained models. With the proposed framework, we demonstrated the capability of using relatively small datasets (5-10 subjects) with sparse (5-25% slices labeled) annotations to train a deep learning algorithm, while achieving DSC of 0.871-0.893 for LVM and 0.933-0.959 for LVC on the LVQuan dataset, and these were 0.844-0.871 for LVM and 0.923-0.931 for LVC on the ACDC dataset. Furthermore, we showed that the proposed approach can be adapted to substantially alleviate the domain shift issue. Moreover, we calculated a number of commonly used LV function measurements using the derived segmentation and observed strong correlations (Pearson r=0.77-1.00, p<0.001) between algorithm and manual LV function analyses. These results suggest that the developed approaches can be used to facilitate broader application of deep learning in research and clinical cardiac MR imaging workflow.

Cardiac segmentation on late gadolinium enhancement MRI: A benchmark study from multi-sequence cardiac MR segmentation challenge

Accurate computing, analysis and modeling of the ventricles and myocardium from medical images are important, especially in the diagnosis and treatment management for patients suffering from myocardial infarction (MI). Late gadolinium enhancement (LGE) cardiac magnetic resonance (CMR) provides an important protocol to visualize MI. However, compared with the other sequences LGE CMR images with gold standard labels are particularly limited. This paper presents the selective results from the Multi-Sequence Cardiac MR (MS-CMR) Segmentation challenge, in conjunction with MICCAI 2019. The challenge offered a data set of paired MS-CMR images, including auxiliary CMR sequences as well as LGE CMR, from 45 patients who underwent cardiomyopathy. It was aimed to develop new algorithms, as well as benchmark existing ones for LGE CMR segmentation focusing on myocardial wall of the left ventricle and blood cavity of the two ventricles. In addition, the paired MS-CMR images could enable algorithms to combine the complementary information from the other sequences for the ventricle segmentation of LGE CMR. Nine representative works were selected for evaluation and comparisons, among which three methods are unsupervised domain adaptation (UDA) methods and the other six are supervised. The results showed that the average performance of the nine methods was comparable to the inter-observer variations. Particularly, the top-ranking algorithms from both the supervised and UDA methods could generate reliable and robust segmentation results. The success of these methods was mainly attributed to the inclusion of the auxiliary sequences from the MS-CMR images, which provide important label information for the training of deep neural networks. The challenge continues as an ongoing resource, and the gold standard segmentation as well as the MS-CMR images of both the training and test data are available upon registration via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mscmrseg/).

Learning disentangled representations in the imaging domain

Disentangled representation learning has been proposed as an approach to learning general representations even in the absence of, or with limited, supervision. A good general representation can be fine-tuned for new target tasks using modest amounts of data, or used directly in unseen domains achieving remarkable performance in the corresponding task. This alleviation of the data and annotation requirements offers tantalising prospects for applications in computer vision and healthcare. In this tutorial paper, we motivate the need for disentangled representations, revisit key concepts, and describe practical building blocks and criteria for learning such representations. We survey applications in medical imaging emphasising choices made in exemplar key works, and then discuss links to computer vision applications. We conclude by presenting limitations, challenges, and opportunities.

A Novel U-Net Based Deep Learning Method for 3D Cardiovascular MRI Segmentation

Medical multiobjective image segmentation aims to group pixels to form multiple regions based on the different properties of the medical images. Segmenting the 3D cardiovascular magnetic resonance (CMR) images is still a challenging task owing to several reasons, including individual differences in heart shapes, varying signal intensities, and differences in data signal-to-noise ratios. This paper proposes a novel and efficient U-Net-based 3D sparse convolutional network named SparseVoxNet. In this network, there are direct connections between any two layers with the same feature-map size, and the number of connections is reduced. Therefore, the SparseVoxNet can effectively cope with the optimization problem of gradients vanishing when training a 3D deep neural network model on small sample data by significantly decreasing the network depth, and achieveing better feature representation using a spatial self-attention mechanism finally. The proposed method in this paper has been thoroughly evaluated on the HVSMR 2016 dataset. Compared with other methods, the method achieves better performance.

Mutual enhancing learning-based automatic segmentation of CT cardiac substructure

Objective. Current segmentation practice for thoracic cancer RT considers the whole heart as a single organ despite increased risks of cardiac toxicities from irradiation of specific cardiac substructures. Segmenting up to 15 different cardiac substructures can be a very time-intensive process, especially due to their different volume sizes and anatomical variations amongst different patients. In this work, a new deep learning (DL)-based mutual enhancing strategy is introduced for accurate and automatic segmentation, especially of smaller substructures such as coronary arteries. Approach. Our proposed method consists of three subnetworks: retina U-net, classification module, and segmentation module. Retina U-net is used as a backbone network architecture that aims to learn deep features from the whole heart. Whole heart feature maps from retina U-net are then transferred to four different sets of classification modules to generate classification localization maps of coronary arteries, great vessels, chambers of the heart, and valves of the heart. Each classification module is in sync with its corresponding subsequent segmentation module in a bootstrapping manner, allowing them to share their encoding paths to generate a mutual enhancing strategy. We evaluated our method on three different datasets: institutional CT datasets (55 subjects) 2) publicly available Multi-Modality Whole Heart Segmentation (MM-WHS) challenge datasets (120 subjects), and Automated Cardiac Diagnosis Challenge (ACDC) datasets (100 subjects). For institutional datasets, we performed five-fold cross-validation on training data (45 subjects) and performed inference on separate hold-out data (10 subjects). For each subject, 15 cardiac substructures were manually contoured by a resident physician and evaluated by an attending radiation oncologist. For the MM-WHS dataset, we trained the network on 100 datasets and performed an inference on a separate hold-out dataset with 20 subjects, each with 7 cardiac substructures. For ACDC datasets, we performed five-fold cross-validation on 100 datasets, each with 3 cardiac substructures. We compared the proposed method against four different network architectures: 3D U-net, mask R-CNN, mask scoring R-CNN, and proposed network without classification module. Segmentation accuracies were statistically compared through dice similarity coefficient, Jaccard, 95% Hausdorff distance, mean surface distance, root mean square distance, center of mass distance, and volume difference. Main results. The proposed method generated cardiac substructure segmentations with significantly higher accuracy (P < 0.05) for small substructures, especially for coronary arteries such as left anterior descending artery (CA-LADA) and right coronary artery (CA-RCA) in comparison to four competing methods. For large substructures (i.e. chambers of the heart), our method yielded comparable results to mask scoring R-CNN method, resulting in significantly (P < 0.05) improved segmentation accuracy in comparison to 3D U-net and mask R-CNN. Significance. A new DL-based mutual enhancing strategy was introduced for automatic segmentation of cardiac substructures. Overall results of this work demonstrate the ability of the proposed method to improve segmentation accuracies of smaller substructures such as coronary arteries without largely compromising the segmentation accuracies of larger substructures. Fast and accurate segmentations of up to 15 substructures can possibly be used as a tool to rapidly generate substructure segmentations followed by physicians’ reviews to improve clinical workflow.

From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review

Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.

AB-ResUNet+: Improving Multiple Cardiovascular Structure Segmentation from Computed Tomography Angiography Images

Accurate segmentation of cardiovascular structures plays an important role in many clinical applications. Recently, fully convolutional networks (FCNs), led by the UNet architecture, have significantly improved the accuracy and speed of semantic segmentation tasks, greatly improving medical segmentation and analysis tasks. The UNet architecture makes heavy use of contextual information. However, useful channel features are not fully exploited. In this work, we present an improved UNet architecture that exploits residual learning, squeeze and excitation operations, Atrous Spatial Pyramid Pooling (ASPP), and the attention mechanism for accurate and effective segmentation of complex cardiovascular structures and name it AB-ResUNet+. The channel attention block is inserted into the skip connection to optimize the coding ability of each layer. The ASPP block is located at the bottom of the network and acts as a bridge between the encoder and decoder. This increases the field of view of the filters and allows them to include a wider context. The proposed AB-ResUNet+ is evaluated on eleven datasets of different cardiovascular structures, including coronary sinus (CS), descending aorta (DA), inferior vena cava (IVC), left atrial appendage (LAA), left atrial wall (LAW), papillary muscle (PM), posterior mitral leaflet (PML), proximal ascending aorta (PAA), pulmonary aorta (PA), right ventricular wall (RVW), and superior vena cava (SVC). Our experimental evaluations show that the proposed AB-ResUNet+ significantly outperforms the UNet, ResUNet, and ResUNet++ architecture by achieving higher values in terms of Dice coefficient and mIoU.

Adaptive Hierarchical Dual Consistency for Semi-Supervised Left Atrium Segmentation on Cross-Domain Data

Semi-supervised learning provides great significance in left atrium (LA) segmentation model learning with insufficient labelled data. Generalising semi-supervised learning to cross-domain data is of high importance to further improve model robustness. However, the widely existing distribution difference and sample mismatch between different data domains hinder the generalisation of semi-supervised learning. In this study, we alleviate these problems by proposing an Adaptive Hierarchical Dual Consistency (AHDC) for the semi-supervised LA segmentation on cross-domain data. The AHDC mainly consists of a Bidirectional Adversarial Inference module (BAI) and a Hierarchical Dual Consistency learning module (HDC). The BAI overcomes the difference of distributions and the sample mismatch between two different domains. It mainly learns two mapping networks adversarially to obtain two matched domains through mutual adaptation. The HDC investigates a hierarchical dual learning paradigm for cross-domain semi-supervised segmentation based on the obtained matched domains. It mainly builds two dual-modelling networks for mining the complementary information in both intra-domain and inter-domain. For the intra-domain learning, a consistency constraint is applied to the dual-modelling targets to exploit the complementary modelling information. For the inter-domain learning, a consistency constraint is applied to the LAs modelled by two dual-modelling networks to exploit the complementary knowledge among different data domains. We demonstrated the performance of our proposed AHDC on four 3D late gadolinium enhancement cardiac MR (LGE-CMR) datasets from different centres and a 3D CT dataset. Compared to other state-of-the-art methods, our proposed AHDC achieved higher segmentation accuracy, which indicated its capability in the cross-domain semi-supervised LA segmentation.

AtrialJSQnet: A New framework for joint segmentation and quantification of left atrium and scars incorporating spatial and shape information

Left atrial (LA) and atrial scar segmentation from late gadolinium enhanced magnetic resonance imaging (LGE MRI) is an important task in clinical practice. The automatic segmentation is however still challenging due to the poor image quality, the various LA shapes, the thin wall, and the surrounding enhanced regions. Previous methods normally solved the two tasks independently and ignored the intrinsic spatial relationship between LA and scars. In this work, we develop a new framework, namely AtrialJSQnet, where LA segmentation, scar projection onto the LA surface, and scar quantification are performed simultaneously in an end-to-end style. We propose a mechanism of shape attention (SA) via an implicit surface projection to utilize the inherent correlation between LA cavity and scars. In specific, the SA scheme is embedded into a multi-task architecture to perform joint LA segmentation and scar quantification. Besides, a spatial encoding (SE) loss is introduced to incorporate continuous spatial information of the target in order to reduce noisy patches in the predicted segmentation. We evaluated the proposed framework on 60 post-ablation LGE MRIs from the MICCAI2018 Atrial Segmentation Challenge. Moreover, we explored the domain generalization ability of the proposed AtrialJSQnet on 40 pre-ablation LGE MRIs from this challenge and 30 post-ablation multi-center LGE MRIs from another challenge (ISBI2012 Left Atrium Fibrosis and Scar Segmentation Challenge). Extensive experiments on public datasets demonstrated the effect of the proposed AtrialJSQnet, which achieved competitive performance over the state-of-the-art. The relatedness between LA segmentation and scar quantification was explicitly explored and has shown significant performance improvements for both tasks. The code has been released via https://zmiclab.github.io/projects.html.

TransDIR: Deformable imaging registration network based on transformer to improve the feature extraction ability

PURPOSE

Imaging registration have a significant contribution to guide and support physicians in the process of decision making for diagnosis, prognosis and treatment. However, existing registration methods based on convolutional neural network can not extract global feature effectively, which significantly influences the registration performance. Moreover, the smoothness of displacement vector field (DVF) fails to be ensured due to miss folding penalty.

METHODS

In order to capture abundant global information as well as local information, we have proposed a novel 3D deformable image registration network based on Transformer (TransDIR). In the encoding phase, Transformer with the atrous reduction attention block is designed to capture the long-distance dependencies which are crucial for extracting global information. Zero padding position encoder is embedded into Transformer to capture the local information. In the decoding phase, up-sampling module based on attention mechanism is designed to increase the significance of ROIs. Because of adding folding penalty term into loss function, the smoothness of DVF is improved.

RESULTS

Finally, we carried out experiments on OASIS, LPBA40, MGH10 and MM-WHS open datasets to validate the effectiveness of TransDIR. Compared with LapIRN, the DSC score is improved by 1.1% and 0.9% on OASIS and LPBA40, separately. In addition, compared with VoxelMorph, the DSC score is improved by 2.8% on the basis of folding index decreased by hundreds of times on MM-WHS.

CONCLUSIONS

The results show that the TransDIR achieves robust registration and promising generalizability compared with LapIRN and VoxelMorph. This article is protected by copyright. All rights reserved.

A deep-learning approach for direct whole-heart mesh reconstruction

Automated construction of surface geometries of cardiac structures from volumetric medical images is important for a number of clinical applications. While deep-learning-based approaches have demonstrated promising reconstruction precision, these approaches have mostly focused on voxel-wise segmentation followed by surface reconstruction and post-processing techniques. However, such approaches suffer from a number of limitations including disconnected regions or incorrect surface topology due to erroneous segmentation and stair-case artifacts due to limited segmentation resolution. We propose a novel deep-learning-based approach that directly predicts whole heart surface meshes from volumetric CT and MR image data. Our approach leverages a graph convolutional neural network to predict deformation on mesh vertices from a pre-defined mesh template to reconstruct multiple anatomical structures in a 3D image volume. Our method demonstrated promising performance of generating whole heart reconstructions with as good or better accuracy than prior deep-learning-based methods on both CT and MR data. Furthermore, by deforming a template mesh, our method can generate whole heart geometries with better anatomical consistency and produce high-resolution geometries from lower resolution input image data. Our method was also able to produce temporally-consistent surface mesh predictions for heart motion from CT or MR cine sequences, and therefore can potentially be applied for efficiently constructing 4D whole heart dynamics. Our code and pre-trained networks are available at https://github.com/fkong7/MeshDeformNet.

Heart Region Segmentation using Dense VNet from Multimodality Images

Cardiovascular diseases (CVD) have been identified as one of the most common causes of death in the world. Advanced development of imaging techniques is allowing timely detection of CVD and helping physicians in providing correct treatment plans in saving lives. Segmentation and Identification of various substructures of the heart are very important in modeling a digital twin of the patient-specific heart. Manual delineation of various substructures of the heart is tedious and time-consuming. Here we have implemented Dense VNet for detecting substructures of the heart from both CT and MRI multimodality data. Due to the limited availability of data we have implemented an on-the-fly elastic deformation data augmentation technique. The result of the proposed has been shown to outperform other methods reported in the literature on both CT and MRI datasets.

Self-Attentive Spatial Adaptive Normalization for Cross-Modality Domain Adaptation

Despite the successes of deep neural networks on many challenging vision tasks, they often fail to generalize to new test domains that are not distributed identically to the training data. The domain adaptation becomes more challenging for cross-modality medical data with a notable domain shift. Given that specific annotated imaging modalities may not be accessible nor complete. Our proposed solution is based on the cross-modality synthesis of medical images to reduce the costly annotation burden by radiologists and bridge the domain gap in radiological images. We present a novel approach for image-to-image translation in medical images, capable of supervised or unsupervised (unpaired image data) setups. Built upon adversarial training, we propose a learnable self-attentive spatial normalization of the deep convolutional generator network's intermediate activations. Unlike previous attention-based image-to-image translation approaches, which are either domain-specific or require distortion of the source domain's structures, we unearth the importance of the auxiliary semantic information to handle the geometric changes and preserve anatomical structures during image translation. We achieve superior results for cross-modality segmentation between unpaired MRI and CT data for multi-modality whole heart and multi-modal brain tumor MRI (T1/T2) datasets compared to the state-of-the-art methods. We also observe encouraging results in cross-modality conversion for paired MRI and CT images on a brain dataset. Furthermore, a detailed analysis of the cross-modality image translation, thorough ablation studies confirm our proposed method's efficacy.

Few-Shot Learning by a Cascaded Framework With Shape-Constrained Pseudo Label Assessment for Whole Heart Segmentation

Automatic and accurate 3D cardiac image segmentation plays a crucial role in cardiac disease diagnosis and treatment. Even though CNN based techniques have achieved great success in medical image segmentation, the expensive annotation, large memory consumption, and insufficient generalization ability still pose challenges to their application in clinical practice, especially in the case of 3D segmentation from high-resolution and large-dimension volumetric imaging. In this paper, we propose a few-shot learning framework by combining ideas of semi-supervised learning and self-training for whole heart segmentation and achieve promising accuracy with a Dice score of 0.890 and a Hausdorff distance of 18.539 mm with only four labeled data for training. When more labeled data provided, the model can generalize better across institutions. The key to success lies in the selection and evolution of high-quality pseudo labels in cascaded learning. A shape-constrained network is built to assess the quality of pseudo labels, and the self-training stages with alternative global-local perspectives are employed to improve the pseudo labels. We evaluate our method on the CTA dataset of the MM-WHS 2017 Challenge and a larger multi-center dataset. In the experiments, our method outperforms the state-of-the-art methods significantly and has great generalization ability on the unseen data. We also demonstrate, by a study of two 4D (3D+T) CTA data, the potential of our method to be applied in clinical practice.

Operative Workflow from CT to 3D Printing of the Heart: Opportunities and Challenges

Medical images do not provide a natural visualization of 3D anatomical structures, while 3D digital models are able to solve this problem. Interesting applications based on these models can be found in the cardiovascular field. The generation of a good-quality anatomical model of the heart is one of the most complex tasks in this context. Its 3D representation has the potential to provide detailed spatial information concerning the heart’s structure, also offering the opportunity for further investigations if combined with additive manufacturing. When investigated, the adaption of printed models turned out to be beneficial in complex surgical procedure planning, for training, education and medical communication. In this paper, we will illustrate the difficulties that may be encountered in the workflow from a stack of Computed Tomography (CT) to the hand-held printed heart model. An important goal will consist in the realization of a heart model that can take into account real wall thickness variability. Stereolithography printing technology will be exploited with a commercial rigid resin. A flexible material will be tested too, but results will not be so satisfactory. As a preliminary validation of this kind of approach, print accuracy will be evaluated by directly comparing 3D scanner acquisitions to the original Standard Tessellation Language (STL) files.

Automated cardiac segmentation of cross-modal medical images using unsupervised multi-domain adaptation and spatial neural attention structure

Accurate cardiac segmentation of multimodal images, e.g., magnetic resonance (MR), computed tomography (CT) images, plays a pivot role in auxiliary diagnoses, treatments and postoperative assessments of cardiovascular diseases. However, training a well-behaved segmentation model for the cross-modal cardiac image analysis is challenging, due to their diverse appearances/distributions from different devices and acquisition conditions. For instance, a well-trained segmentation model based on the source domain of MR images is often failed in the segmentation of CT images. In this work, a cross-modal images-oriented cardiac segmentation scheme is proposed using a symmetric full convolutional neural network (SFCNN) with the unsupervised multi-domain adaptation (UMDA) and a spatial neural attention (SNA) structure, termed UMDA-SNA-SFCNN, having the merits of without the requirement of any annotation on the test domain. Specifically, UMDA-SNA-SFCNN incorporates SNA to the classic adversarial domain adaptation network to highlight the relevant regions, while restraining the irrelevant areas in the cross-modal images, so as to suppress the negative transfer in the process of unsupervised domain adaptation. In addition, the multi-layer feature discriminators and a predictive segmentation-mask discriminator are established to connect the multi-layer features and segmentation mask of the backbone network, SFCNN, to realize the fine-grained alignment of unsupervised cross-modal feature domains. Extensive confirmative and comparative experiments on the benchmark Multi-Modality Whole Heart Challenge dataset show that the proposed model is superior to the state-of-the-art cross-modal segmentation methods.

Personalising left-ventricular biophysical models of the heart using parametric physics-informed neural networks

We present a parametric physics-informed neural network for the simulation of personalised left-ventricular biomechanics. The neural network is constrained to the biophysical problem in two ways: (i) the network output is restricted to a subspace built from radial basis functions capturing characteristic deformations of left ventricles and (ii) the cost function used for training is the energy potential functional specifically tailored for hyperelastic, anisotropic, nearly-incompressible active materials. The radial bases are generated from the results of a nonlinear Finite Element model coupled with an anatomical shape model derived from high-resolution cardiac images. We show that, by coupling the neural network with a simplified circulation model, we can efficiently generate computationally inexpensive estimations of cardiac mechanics. Our model is 30 times faster than the reference Finite Element model used, including training time, while yielding satisfactory average errors in the predictions of ejection fraction (-3%), peak systolic pressure (7%), stroke work (4%) and myocardial strains (14%). This physics-informed neural network is well suited to efficiently augment cardiac images with functional data and to generate large sets of synthetic cases for training deep network classifiers while it provides efficient personalization to the specific patient of interest with a high level of detail.

An 8-layer residual U-Net with deep supervision for segmentation of the left ventricle in cardiac CT angiography

BACKGROUND AND OBJECTIVES

Accurate segmentation of left ventricle (LV) is a fundamental step in evaluation of cardiac function. Cardiac CT angiography (CCTA) has become an important clinical diagnostic method for cardio-vascular disease (CVD) due to its non-invasive, short exam time, and low cost. To obtain the segmentation of the LV in CCTA scans, we present a deep learning method based on an 8-layer residual U-Net with deep supervision.

METHODS

Based on the original 4-layer U-Net, our method deepened the network to eight layers, which increased the fitting capacity of the network, thus greatly improved its LV recognition capability. Residual blocks were incorporated to optimize the network from the increased depth. Auxiliary paths as deep supervision were introduced to supervise the intermediate information to improve the segmentation quality. In this study, we collected CCTA scans of 100 patients. Eighty patients with 1600 discrete slices were used to train the LV segmentation and the remaining 20 patients with 400 discrete slices were used for testing our method. An interactive graph cut algorithm was utilized reliably to annotate the LV reference standard that was further confirmed by cardiologists. Online data augmentation was performed in the training process to improve the generalization and robustness of our method.

RESULTS

Compared with the segmentation results from the original U-Net and FC-DenseNet56 with Dice similarity coefficient (DSC) of 0.878±0.230 and 0.897±0.189, respectively, our method demonstrated higher segmentation accuracy and robustness for varying LV shape, size, and contrast, achieving DSC of 0.927±0.139. Without online data augmentation, our method resulted in inferior performance with DSC of 0.911±0.170. In addition, compared with the provided results from other existing studies in the LV segmentation of cardiac CT images, our method achieved a competitive performance for the LV segmentation.

CONCLUSIONS

The proposed 8-layer residual U-Net with deep supervision accurately and efficiently segments the LV in CCTA scans. This method has potential advantages to be a reliable segmentation method and useful for the evaluation of cardiac function in the future study.

Automatic cardiac cine MRI segmentation and heart disease classification

Cardiac cine magnetic resonance imaging (MRI) continues to be recognized as an established modality for non-invasive assessment of the function and structure of the cardiovascular system. Making full use of fully convolutional neural networks CNNs ability to operate pixel-wise classification, cine MRI sequences can be segmented and volumetric features of three key heart structures are computed for disease prediction. The three key heart structures are the left ventricle cavity, right ventricle cavity and the left ventricle myocardium. In this paper, we suggest an automated pipeline for both cardiac segmentation and diagnosis. The study was conducted on a dataset of 150 patients from Dijon Hospital in the context of the post-2017 Medical Image Computing and Computer Assisted Intervention MICCAI, Automated Cardiac Diagnosis Challenge (ACDC). The challenge consists in two phases: (i) a segmentation contest, where performance is evaluated on dice overlap coefficient and Hausdorff distance metrics, and a (ii) diagnosis contest for heart disease classification. For this aim, we propose the use of a deep learning based network for segmentation of the three key cardiac structures within short-axis cine MRI sequences and a classifier ensemble for heart disease classification. The deep learning segmentation network is a UNet fully convolutional neural network variant with fewer trainable parameters. The classifier ensemble consists in combining three classifiers, namely a multilayer perceptron, a random forest and a support vector machine. Before feeding the segmentation network, a preliminary step consists in localizing heart region and cropping input images to a restricted region of interest (ROI). This is achieved by a signal processing based approach and aims at reducing multi-class imbalance and computational load. We achieved nearly state of the art accuracy performances for both the segmentation and disease classification challenges. Reporting a mean dice overlap coefficient of 0.92 for the three cardiac structures segmentation, along with good limits of agreement for the various derived clinical indices, leading to an accuracy of 0.92 for the disease classification on unseen data.

CAB U-Net: An end-to-end category attention boosting algorithm for segmentation

With the development of machine learning and artificial intelligence, many convolutional neural networks (CNNs) based segmentation methods have been proposed for 3D cardiac segmentation. In this paper, we propose the category attention boosting (CAB) module, which combines the deep network calculation graph with the boosting method. On the one hand, we add the attention mechanism into the gradient boosting process, which enhances the information of coarse segmentation without high computation cost. On the other hand, we introduce the CAB module into the 3D U-Net segmentation network and construct a new multi-scale boosting model CAB U-Net which strengthens the gradient flow in the network and makes full use of the low resolution feature information. Thanks to the advantage that end-to-end networks can adaptively adjust the internal parameters, CAB U-Net can make full use of the complementary effects among different base learners. Extensive experiments on public datasets show that our approach can achieve superior performance over the state-of-the-art methods.

Fully automated 3D segmentation and separation of multiple cervical vertebrae in CT images using a 2D convolutional neural network

BACKGROUND AND OBJECTIVE

We investigated a novel method using a 2D convolutional neural network (CNN) to identify superior and inferior vertebrae in a single slice of CT images, and a post-processing for 3D segmentation and separation of cervical vertebrae.

METHODS

The cervical spines of patients (N == 17, 1684 slices) from Severance and Gangnam Severance Hospitals (S/GSH) and healthy controls (N == 24, 3490 slices) from Seoul National University Bundang Hospital (SNUBH) were scanned by using various volumetric CT protocols. To prepare gold standard masks of cervical spine in CT images, each spine was segmented by using conventional image-processing methods and manually corrected by an expert. The gold standard masks were preprocessed and labeled into superior and inferior cervical vertebrae separately in the axial slices. The 2D U-Net model was trained by using the disease dataset (S/GSH) and additional validation was performed by using the healthy control dataset (SNUBH), and then the training and validation were repeated by switching the two datasets.

RESULTS

In case of the model was trained with the disease dataset (S/GSH) and validated with the healthy control (SNUBH), the mean and standard deviation (SD) of the Dice similarity coefficient (DSC), Jaccard similarity coefficient (JSC), mean surface distance (MSD), and Hausdorff surface distance (HSD) were 94.37%% ± 1.45%, 89.47%% ± 2.55%, 0.33 ± 0.12 mm and 20.89 ± 3.98 mm, and 88.67%% ± 5.82%, 80.83%% ± 8.09%, 1.05 ± 0.63 mm and 29.17 ± 19.74 mm, respectively. In case of the model was trained with the healthy control (SNUBH) and validated with the disease dataset (S/GSH), the mean and SD of DSC, JSC, MSD, and HSD were 96.23%% ± 1.55%, 92.95%% ± 2.58%, 0.39 ± 0.20 mm and 16.23 ± 6.72 mm, and 93.15%% ± 3.09%, 87.54%% ± 5.11%, 0.38 ± 0.17 mm and 20.85 ± 7.11 mm, respectively.

CONCLUSIONS

The results demonstrated that our fully automated method achieved comparable accuracies with inter- and intra-observer variabilities of manual segmentation by human experts, which is time consuming.

Simultaneous Multi-Structure Segmentation of the Heart and Peripheral Tissues in Contrast Enhanced Cardiac Computed Tomography Angiography

Contrast enhanced cardiac computed tomography angiography (CTA) is a prominent imaging modality for diagnosing cardiovascular diseases non-invasively. It assists the evaluation of the coronary artery patency and provides a comprehensive assessment of structural features of the heart and great vessels. However, physicians are often required to evaluate different cardiac structures and measure their size manually. Such task is very time-consuming and tedious due to the large number of image slices in 3D data. We present a fully automatic method based on a combined multi-atlas and corrective segmentation approach to label the heart and its associated cardiovascular structures. This method also automatically separates other surrounding intrathoracic structures from CTA images. Quantitative assessment of the proposed method is performed on 36 studies with a reference standard obtained from expert manual segmentation of various cardiac structures. Qualitative evaluation is also performed by expert readers to score 120 studies of the automatic segmentation. The quantitative results showed an overall Dice of 0.93, Hausdorff distance of 7.94 mm, and mean surface distance of 1.03 mm between automatically and manually segmented cardiac structures. The visual assessment also attained an excellent score for the automatic segmentation. The average processing time was 2.79 minutes. Our results indicate the proposed automatic framework significantly improves accuracy and computational speed in conventional multi-atlas based approach, and it provides comprehensive and reliable multi-structural segmentation of CTA images that is valuable for clinical application.

Evaluation of algorithms for Multi-Modality Whole Heart Segmentation: An open-access grand challenge

Knowledge of whole heart anatomy is a prerequisite for many clinical applications. Whole heart segmentation (WHS), which delineates substructures of the heart, can be very valuable for modeling and analysis of the anatomy and functions of the heart. However, automating this segmentation can be challenging due to the large variation of the heart shape, and different image qualities of the clinical data. To achieve this goal, an initial set of training data is generally needed for constructing priors or for training. Furthermore, it is difficult to perform comparisons between different methods, largely due to differences in the datasets and evaluation metrics used. This manuscript presents the methodologies and evaluation results for the WHS algorithms selected from the submissions to the Multi-Modality Whole Heart Segmentation (MM-WHS) challenge, in conjunction with MICCAI 2017. The challenge provided 120 three-dimensional cardiac images covering the whole heart, including 60 CT and 60 MRI volumes, all acquired in clinical environments with manual delineation. Ten algorithms for CT data and eleven algorithms for MRI data, submitted from twelve groups, have been evaluated. The results showed that the performance of CT WHS was generally better than that of MRI WHS. The segmentation of the substructures for different categories of patients could present different levels of challenge due to the difference in imaging and variations of heart shapes. The deep learning (DL)-based methods demonstrated great potential, though several of them reported poor results in the blinded evaluation. Their performance could vary greatly across different network structures and training strategies. The conventional algorithms, mainly based on multi-atlas segmentation, demonstrated good performance, though the accuracy and computational efficiency could be limited. The challenge, including provision of the annotated training data and the blinded evaluation for submitted algorithms on the test data, continues as an ongoing benchmarking resource via its homepage (www.sdspeople.fudan.edu.cn/zhuangxiahai/0/mmwhs/).

Atrial scar quantification via multi-scale CNN in the graph-cuts framework

•

Propose a fully automatic method for left atrial scar quantification, with promising performance.

•

Formulate a new framework of scar quantification based on surface projection and graph-cuts framework.

•

Propose the multi-scale learning CNN, combined with the random shift training strategy, to learn and predict the graph potentials, which significantly improves the performance of the proposed method, and enables the full automation of the framework.

•

Provide thorough validation and parameter studies for the proposed techniques using fifty-eight clinical images.

Late gadolinium enhancement magnetic resonance imaging (LGE MRI) appears to be a promising alternative for scar assessment in patients with atrial fibrillation (AF). Automating the quantification and analysis of atrial scars can be challenging due to the low image quality. In this work, we propose a fully automated method based on the graph-cuts framework, where the potentials of the graph are learned on a surface mesh of the left atrium (LA) using a multi-scale convolutional neural network (MS-CNN). For validation, we have included fifty-eight images with manual delineations. MS-CNN, which can efficiently incorporate both the local and global texture information of the images, has been shown to evidently improve the segmentation accuracy of the proposed graph-cuts based method. The segmentation could be further improved when the contribution between the t-link and n-link weights of the graph is balanced. The proposed method achieves a mean accuracy of 0.856 ± 0.033 and mean Dice score of 0.702 ± 0.071 for LA scar quantification. Compared to the conventional methods, which are based on the manual delineation of LA for initialization, our method is fully automatic and has demonstrated significantly better Dice score and accuracy (p < 0.01). The method is promising and can be potentially useful in diagnosis and prognosis of AF.

Disentangled representation learning in cardiac image analysis

Typically, a medical image offers spatial information on the anatomy (and pathology) modulated by imaging specific characteristics. Many imaging modalities including Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) can be interpreted in this way. We can venture further and consider that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. Here, we explicitly learn this decomposed (disentangled) representation of imaging data, focusing in particular on cardiac images. We propose Spatial Decomposition Network (SDNet), which factorises 2D medical images into spatial anatomical factors and non-spatial modality factors. We demonstrate that this high-level representation is ideally suited for several medical image analysis tasks, such as semi-supervised segmentation, multi-task segmentation and regression, and image-to-image synthesis. Specifically, we show that our model can match the performance of fully supervised segmentation models, using only a fraction of the labelled images. Critically, we show that our factorised representation also benefits from supervision obtained either when we use auxiliary tasks to train the model in a multi-task setting (e.g. regressing to known cardiac indices), or when aggregating multimodal data from different sources (e.g. pooling together MRI and CT data). To explore the properties of the learned factorisation, we perform latent-space arithmetic and show that we can synthesise CT from MR and vice versa, by swapping the modality factors. We also demonstrate that the factor holding image specific information can be used to predict the input modality with high accuracy. Code will be made available at https://github.com/agis85/anatomy_modality_decomposition.

Convolutional neural network-based approach for segmentation of left ventricle myocardial scar from 3D late gadolinium enhancement MR images

PURPOSE

Accurate three-dimensional (3D) segmentation of myocardial replacement fibrosis (i.e., scar) is emerging as a potentially valuable tool for risk stratification and procedural planning in patients with ischemic cardiomyopathy. The main purpose of this study was to develop a semiautomated method using a 3D convolutional neural network (CNN)-based for the segmentation of left ventricle (LV) myocardial scar from 3D late gadolinium enhancement magnetic resonance (LGE-MR) images.

METHODS

Our proposed CNN is built upon several convolutional and pooling layers aimed at choosing appropriate features from LGE-MR images to distinguish between myocardial scar and healthy tissues of the left ventricle. In contrast to previous methods that consider image intensity as the sole feature, CNN-based algorithms have the potential to improve the accuracy of scar segmentation through the creation of unconventional features that separate scar from normal myocardium in the feature space. The first step of our pipeline was to manually delineate the left ventricular myocardium, which was used as the region of interest for scar segmentation. Our developed algorithm was trained using 265,220 volume patches extracted from ten 3D LGE-MR images, then was validated on 450,454 patches from a testing dataset of 24 3D LGE-MR images, all obtained from patients with chronic myocardial infarction. We evaluated our method in the context of several alternative methods by comparing algorithm-generated segmentations to manual delineations performed by experts.

RESULTS

Our CNN-based method reported an average Dice similarity coefficient (DSC) and Jaccard Index (JI) of 93.63% ± 2.6% and 88.13% ± 4.70%. In comparison to several previous methods, including K-nearest neighbor (KNN), hierarchical max flow (HMF), full width at half maximum (FWHM), and signal threshold to reference mean (STRM), the developed algorithm reported significantly higher accuracy for DSC with a P-value less than 0.0001.

CONCLUSIONS

Our experimental results demonstrated that our CNN-based proposed method yielded the highest accuracy of all contemporary LV myocardial scar segmentation methodologies, inclusive of the most widely used signal intensity-based methods, such as FWHM and STRM. To our knowledge, this is the first description of LV myocardial scar tissue segmentation from 3D LGE-MR images using a CNN-based method.

Iterative Segmentation from Limited Training Data: Applications to Congenital Heart Disease

We propose a new iterative segmentation model which can be accurately learned from a small dataset. A common approach is to train a model to directly segment an image, requiring a large collection of manually annotated images to capture the anatomical variability in a cohort. In contrast, we develop a segmentation model that recursively evolves a segmentation in several steps, and implement it as a recurrent neural network. We learn model parameters by optimizing the intermediate steps of the evolution in addition to the final segmentation. To this end, we train our segmentation propagation model by presenting incomplete and/or inaccurate input segmentations paired with a recommended next step. Our work aims to alleviate challenges in segmenting heart structures from cardiac MRI for patients with congenital heart disease (CHD), which encompasses a range of morphological deformations and topological changes. We demonstrate the advantages of this approach on a dataset of 20 images from CHD patients, learning a model that accurately segments individual heart chambers and great vessels. Compared to direct segmentation, the iterative method yields more accurate segmentation for patients with the most severe CHD malformations.

Correlated Regression Feature Learning for Automated Right Ventricle Segmentation

Accurate segmentation of right ventricle (RV) from cardiac magnetic resonance (MR) images can help a doctor to robustly quantify the clinical indices including ejection fraction. In this paper, we develop one regression convolutional neural network (RegressionCNN) which combines a holistic regression model and a convolutional neural network (CNN) together to determine boundary points' coordinates of RV directly and simultaneously. In our approach, we take the fully connected layers of CNN as the holistic regression model to perform RV segmentation, and the feature maps extracted by convolutional layers of CNN are converted into 1-D vector to connect holistic regression model. Such connection allows us to make full use of the optimization algorithm to constantly optimize the convolutional layers to directly learn the holistic regression model in the training process rather than separate feature extraction and regression model learning. Therefore, RegressionCNN can achieve optimally convolutional feature learning for accurately catching the regression features that are more correlated to RV regression segmentation task in training process, and this can reduce the latent mismatch influence between the feature extraction and the following regression model learning. We evaluate the performance of RegressionCNN on cardiac MR images acquired of 145 human subjects from two clinical centers. The results have shown that RegressionCNN's results are highly correlated (average boundary correlation coefficient equals 0.9827) and consistent with the manual delineation (average dice metric equals 0.8351). Hence, RegressionCNN could be an effective way to segment RV from cardiac MR images accurately and automatically.