Convolutional Neural Network With Shape Prior Applied to Cardiac MRI Segmentation

Hybrid deep learning for computational precision in cardiac MRI segmentation: Integrating Autoencoders, CNNs, and RNNs for enhanced structural analysis

Recent advancements in cardiac imaging have been significantly enhanced by integrating deep learning models, offering transformative potential in early diagnosis and patient care. The research paper explores the application of hybrid deep learning methodologies, focusing on the roles of Autoencoders, Convolutional Neural Networks (CNNs), and Recurrent Neural Networks (RNNs) in enhancing cardiac image analysis. The study implements a comprehensive approach, combining traditional algorithms such as Sobel, Watershed, and Otsu's Thresholding with advanced deep learning models to achieve precise and accurate imaging outcomes. The Autoencoder model, developed for image enhancement and feature extraction, achieved a notable accuracy of 99.66% on the test data. Optimized for image recognition tasks, the CNN model demonstrated a high precision rate of 98.9%. The RNN model, utilized for sequential data analysis, showed a prediction accuracy of 98%, further underscoring the robustness of the hybrid framework. The research drew upon a diverse range of academic databases and pertinent publications within cardiac imaging and deep learning, focusing on peer-reviewed articles and studies published in the past five years. Models were implemented using the TensorFlow and Keras frameworks. The proposed methodology was evaluated in the clinical validation phase using advanced imaging protocols, including the QuickScan technique and balanced steady-state free precession (bSSFP) imaging. The validation metrics were promising: the Signal-to-Noise Ratio (SNR) was improved by 15%, the Contrast-to-Noise Ratio (CNR) saw an enhancement of 12%, and the ejection fraction (EF) analysis provided a 95% correlation with manually segmented data. These metrics confirm the efficacy of the models, showing significant improvements in image quality and diagnostic accuracy. The integration of adversarial defense strategies, such as adversarial training and model ensembling, have been analyzed to enhance model robustness against malicious inputs. The reliability and comparison of the model's ability have been investigated to maintain clinical integrity, even in adversarial attacks that could otherwise compromise segmentation outcomes. These findings indicate that integrating Autoencoders, CNNs, and RNNs within a hybrid deep-learning framework is promising for enhancing cardiac MRI segmentation and early diagnosis. The study contributes to the field by demonstrating the applicability of these advanced techniques in clinical settings, paving the way for improved patient outcomes through more accurate and timely diagnoses.

CapNet: An Automatic Attention-Based with Mixer Model for Cardiovascular Magnetic Resonance Image Segmentation

Deep neural networks have shown excellent performance in medical image segmentation, especially for cardiac images. Transformer-based models, though having advantages over convolutional neural networks due to the ability of long-range dependence learning, still have shortcomings such as having a large number of parameters and and high computational cost. Additionally, for better results, they are often pretrained on a larger data, thus requiring large memory size and increasing resource expenses. In this study, we propose a new lightweight but efficient model, namely CapNet, based on convolutions and mixing modules for cardiac segmentation from magnetic resonance images (MRI) that can be trained from scratch with a small amount of parameters. To handle varying sizes and shapes which often occur in cardiac systolic and diastolic phases, we propose attention modules for pooling, spatial, and channel information. We also propose a novel loss called the Tversky Shape Power Distance function based on the shape dissimilarity between labels and predictions that shows promising performances compared to other losses. Experiments on three public datasets including ACDC benchmark, Sunnybrook data, and MS-CMR challenge are conducted and compared with other state of the arts (SOTA). For binary segmentation, the proposed CapNet obtained the Dice similarity coefficient (DSC) of 94% and 95.93% for respectively the Endocardium and Epicardium regions with Sunnybrook dataset, 94.49% for Endocardium, and 96.82% for Epicardium with the ACDC data. Regarding the multiclass case, the average DSC by CapNet is 93.05% for the ACDC data; and the DSC scores for the MS-CMR are 94.59%, 92.22%, and 93.99% for respectively the bSSFP, T2-SPAIR, and LGE sequences of the MS-CMR. Moreover, the statistical significance analysis tests with p-value < 0.05 compared with transformer-based methods and some CNN-based approaches demonstrated that the CapNet, though having fewer training parameters, is statistically significant. The promising evaluation metrics show comparative results in both Dice and IoU indices compared to SOTA CNN-based and Transformer-based architectures.

A multi-task deep learning model based on comprehensive feature integration and self-attention mechanism for predicting response to anti-PD1/PD-L1

BACKGROUND

Immune checkpoint inhibitor (ICI) has been widely used in the treatment of advanced cancers, but predicting their efficacy remains challenging. Traditional biomarkers are numerous but exhibit heterogeneity within populations. For comprehensively utilizing the ICI-related biomarkers, we aim to conduct multidimensional feature selection and deep learning model construction.

METHODS

We used statistical and machine learning methods to map features of different levels to next-generation sequencing gene expression. We integrated genes from different sources into the feature input of a deep learning model, by means of self-attention mechanism.

RESULTS

We performed feature selection at the single-cell sequencing level, PD-L1 (CD274) analysis level, tumor mutational burden (TMB)/mismatch repair (MMR) level, and somatic copy number alteration (SCNA) level, obtaining 96 feature genes. Based on the pan-cancer dataset, we trained a multi-task deep learning model. We tested the model in the bladder urothelial carcinoma testing set 1 (AUC = 0.62, n = 298), bladder urothelial carcinoma testing set 2 (AUC = 0.66, n = 89), non-small cell lung cancer testing set (AUC = 0.85, n = 27), and skin cutaneous melanoma testing set (AUC = 0.71, n = 27).

CONCLUSION

Our study demonstrates the potential of the deep learning model for integrating multidimensional features in predicting the outcome of ICI. Our study also provides a potential methodological case for medical scenarios requiring the integration of multiple levels of features.

FedIOD: Federated Multi-Organ Segmentation From Partial Labels by Exploring Inter-Organ Dependency

Multi-organ segmentation is a fundamental task and existing approaches usually rely on large-scale fully-labeled images for training. However, data privacy and incomplete/partial labels make those approaches struggle in practice. Federated learning is an emerging tool to address data privacy but federated learning with partial labels is under-explored. In this work, we explore generating full supervision by building and aggregating inter-organ dependency based on partial labels and propose a single-encoder-multi-decoder framework named FedIOD. To simulate the annotation process where each organ is labeled by referring to other closely-related organs, a transformer module is introduced and the learned self-attention matrices modeling pairwise inter-organ dependency are used to build pseudo full labels. By using those pseudo-full labels for regularization in each client, the shared encoder is trained to extract rich and complete organ-related features rather than being biased toward certain organs. Then, each decoder in FedIOD projects the shared organ-related features into a specific space trained by the corresponding partial labels. Experimental results based on five widely-used datasets, including LiTS, KiTS, MSD, BCTV, and ACDC, demonstrate the effectiveness of FedIOD, outperforming the state-of-the-art approaches under in-federation evaluation and achieving the second-best performance under out-of-federation evaluation for multi-organ segmentation from partial labels.

ConvNextUNet: A small-region attentioned model for cardiac MRI segmentation

Cardiac MRI segmentation is a significant research area in medical image processing, holding immense clinical and scientific importance in assisting the diagnosis and treatment of heart diseases. Currently, existing cardiac MRI segmentation algorithms are often constrained by specific datasets and conditions, leading to a notable decrease in segmentation performance when applied to diverse datasets. These limitations affect the algorithm's overall performance and generalization capabilities. Inspired by ConvNext, we introduce a two-dimensional cardiac MRI segmentation U-shaped network called ConvNextUNet. It is the first application of a combination of ConvNext and the U-shaped architecture in the field of cardiac MRI segmentation. Firstly, we incorporate up-sampling modules into the original ConvNext architecture and combine it with the U-shaped framework to achieve accurate reconstruction. Secondly, we integrate Input Stem into ConvNext, and introduce attention mechanisms along the bridging path. By merging features extracted from both the encoder and decoder, a probability distribution is obtained through linear and nonlinear transformations, serving as attention weights, thereby enhancing the signal of the same region of interest. The resulting attention weights are applied to the decoder features, highlighting the region of interest. This allows the model to simultaneously consider local context and global details during the learning phase, fully leveraging the advantages of both global and local perception for a more comprehensive understanding of cardiac anatomical structures. Consequently, the model demonstrates a clear advantage and robust generalization capability, especially in small-region segmentation. Experimental results on the ACDC, LVQuan19, and RVSC datasets confirm that the ConvNextUNet model outperforms the current state-of-the-art models, particularly in small-region segmentation tasks. Furthermore, we conducted cross-dataset training and testing experiments, which revealed that the pre-trained model can accurately segment diverse cardiac datasets, showcasing its powerful generalization capabilities. The source code of this project is available at https://github.com/Zemin-Cai/ConvNextUNet.

The beating heart: artificial intelligence for cardiovascular application in the clinic

Artificial intelligence (AI) integration in cardiac magnetic resonance imaging presents new and exciting avenues for advancing patient care, automating post-processing tasks, and enhancing diagnostic precision and outcomes. The use of AI significantly streamlines the examination workflow through the reduction of acquisition and postprocessing durations, coupled with the automation of scan planning and acquisition parameters selection. This has led to a notable improvement in examination workflow efficiency, a reduction in operator variability, and an enhancement in overall image quality. Importantly, AI unlocks new possibilities to achieve spatial resolutions that were previously unattainable in patients. Furthermore, the potential for low-dose and contrast-agent-free imaging represents a stride toward safer and more patient-friendly diagnostic procedures. Beyond these benefits, AI facilitates precise risk stratification and prognosis evaluation by adeptly analysing extensive datasets. This comprehensive review article explores recent applications of AI in the realm of cardiac magnetic resonance imaging, offering insights into its transformative potential in the field.

CBAM_SAUNet: A novel attention U-Net for effective segmentation of corner cases

U-Net has been demonstrated to be effective for the task of medical image segmentation. Additionally, integrating attention mechanism into U-Net has been shown to yield significant benefits. The Shape Attentive U-Net (SAUNet) is one such recently proposed attention U-Net that also focuses on interpretability. Furthermore, recent research has focused on identification and reporting of corner cases in segmentation to accelerate the utilisation of deep learning models in clinical practise. However, achieving good model performance on such corner cases is a less-explored research area. In this paper, we propose CBAM_SAUNet which enhances the dual attention decoder block of SAUNet to improve its performance on corner cases. We achieve this by utilising a novel variant of the Convolutional Block Attention Module (CBAM)'s channel attention in the decoder block of SAUNet. We demonstrate the effectiveness of CBAM_SAUNet in the Automated Cardiac Diagnosis Challenge (ACDC) cardiac MRI segmentation challenge. Our proposed novel approach results in improvement in the Dice scores of 12% for Left Ventricle (LV) as well as Right Ventricle (RV) segmentation and 8% for Myocardium (MYO) for the identified corner-case dataset.

A new dataset of oral panoramic x-ray images and parallel network using transformers for medical image segmentation

INTRODUCTION

Accurate segmentation of the key mandibular region in the oral panoramic X-ray image is crucial for the diagnosis of the mandibular region and the planning of implant surgery. Because the oral panoramic X-ray image contains many important anatomical information for implant treatment evaluation. However, the fuzzy boundary between each region in the image makes the segmentation task very challenging. In data-driven segmentation methods, corresponding datasets are often required. Due to the limited oral data set at present, there is a bottleneck in clinical application.

MATERIALS AND METHODS

In this paper, we build a panoramic X-ray image dataset for the mandibular region. The dataset has a total of 711 images. The dataset is divided into 8 categories based on the number of teeth and treatment conditions. The annotations include mandible, normal teeth, treated teeth and implants. In terms of network segmentation. According to the local and global characteristics of the dataset, we designed a CBTrans partition network by paralleling the convolution block and the Swin-transform block of the bottleneck structure.

RESULTS

The experimental results show that our proposed network achieves excellent performance on the mandibular region segmentation dataset and the common retina dataset DRIVE.

CONCLUSION

CBTrans can better extract features locally and globally by combining CNN of the bottleneck structure and Swin Transformer in parallel. CBTrans demonstrates performance advantages over other similar hybrid architecture models.

Cardiac Magnetic Resonance Imaging as a Risk Stratification Tool in COVID-19 Myocarditis

The aim of this retrospective study was to identify myocardial injury after COVID-19 inflammation and explore whether myocardial damage could be a possible cause of the persistent symptoms following COVID-19 infection in previously healthy individuals. This study included 139 patients who were enrolled between January and June 2021, with a mean age of 46.7 ± 15.2 years, of whom 68 were men and 71 were women without known cardiac or pulmonary diseases. All patients underwent clinical work-up, laboratory analysis, cardiac ultrasound, and CMR on a 1.5 T scanner using a recommended protocol for morphological and functional assessment before and after contrast media application with multi-parametric sequences. In 39% of patients, late gadolinium enhancement (LGE) was found as a sign of myocarditis. Fibrinogen was statistically significantly higher in patients with LGE than in those without LGE (4.3 ± 0.23 vs. 3.2 ± 0.14 g/L, p < 0.05, respectively), as well as D-dimer (1.8 ± 0.3 vs. 0.8 ± 0.1 mg/L FEU). Also, troponin was statistically significantly higher in patients with myocardial LGE (13.1 ± 0.4 ng/L) compared to those with normal myocardium (4.9 ± 0.3 ng/L, p < 0.001). We demonstrated chest pain, fatigue, and elevated troponin to be independent predictors for LGE. Septal LGE was shown to be a predictor for arrhythmias. The use of CMR is a potential risk stratification tool in evaluating outcomes following COVID-19 myocarditis.

DECTNet: Dual Encoder Network combined convolution and Transformer architecture for medical image segmentation

Automatic and accurate segmentation of medical images plays an essential role in disease diagnosis and treatment planning. Convolution neural networks have achieved remarkable results in medical image segmentation in the past decade. Meanwhile, deep learning models based on Transformer architecture also succeeded tremendously in this domain. However, due to the ambiguity of the medical image boundary and the high complexity of physical organization structures, implementing effective structure extraction and accurate segmentation remains a problem requiring a solution. In this paper, we propose a novel Dual Encoder Network named DECTNet to alleviate this problem. Specifically, the DECTNet embraces four components, which are a convolution-based encoder, a Transformer-based encoder, a feature fusion decoder, and a deep supervision module. The convolutional structure encoder can extract fine spatial contextual details in images. Meanwhile, the Transformer structure encoder is designed using a hierarchical Swin Transformer architecture to model global contextual information. The novel feature fusion decoder integrates the multi-scale representation from two encoders and selects features that focus on segmentation tasks by channel attention mechanism. Further, a deep supervision module is used to accelerate the convergence of the proposed method. Extensive experiments demonstrate that, compared to the other seven models, the proposed method achieves state-of-the-art results on four segmentation tasks: skin lesion segmentation, polyp segmentation, Covid-19 lesion segmentation, and MRI cardiac segmentation.

Study of multistep Dense U-Net-based automatic segmentation for head MRI scans

BACKGROUND

Despite extensive efforts to obtain accurate segmentation of magnetic resonance imaging (MRI) scans of a head, it remains challenging primarily due to variations in intensity distribution, which depend on the equipment and parameters used.

PURPOSE

The goal of this study is to evaluate the effectiveness of an automatic segmentation method for head MRI scans using a multistep Dense U-Net (MDU-Net) architecture.

METHODS

The MDU-Net-based method comprises two steps. The first step is to segment the scalp, skull, and whole brain from head MRI scans using a convolutional neural network (CNN). In the first step, a hybrid network is used to combine 2.5D Dense U-Net and 3D Dense U-Net structure. This hybrid network acquires logits in three orthogonal planes (axial, coronal, and sagittal) using 2.5D Dense U-Nets and fuses them by averaging. The resultant fused probability map with head MRI scans then serves as the input to a 3D Dense U-Net. In this process, different ratios of active contour loss and focal loss are applied. The second step is to segment the cerebrospinal fluid (CSF), white matter, and gray matter from extracted brain MRI scans using CNNs. In the second step, the histogram of the extracted brain MRI scans is standardized and then a 2.5D Dense U-Net is used to further segment the brain's specific tissues using the focal loss. A dataset of 100 head MRI scans from an OASIS-3 dataset was used for training, internal validation, and testing, with ratios of 80%, 10%, and 10%, respectively. Using the proposed approach, we segmented the head MRI scans into five areas (scalp, skull, CSF, white matter, and gray matter) and evaluated the segmentation results using the Dice similarity coefficient (DSC) score, Hausdorff distance (HD), and the average symmetric surface distance (ASSD) as evaluation metrics. We compared these results with those obtained using the Res-U-Net, Dense U-Net, U-Net++, Swin-Unet, and H-Dense U-Net models.

RESULTS

The MDU-Net model showed DSC values of 0.933, 0.830, 0.833, 0.953, and 0.917 in the scalp, skull, CSF, white matter, and gray matter, respectively. The corresponding HD values were 2.37, 2.89, 2.13, 1.52, and 1.53 mm, respectively. The ASSD values were 0.50, 1.63, 1.28, 0.26, and 0.27 mm, respectively. Comparing these results with other models revealed that the MDU-Net model demonstrated the best performance in terms of the DSC values for the scalp, CSF, white matter, and gray matter. When compared with the H-Dense U-Net model, which showed the highest performance among the other models, the MDU-Net model showed substantial improvements in the HD view, particularly in the gray matter region, with a difference of approximately 9%. In addition, in terms of the ASSD, the MDU-Net model outperformed the H-Dense U-Net model, showing an approximately 7% improvements in the white matter and approximately 9% improvements in the gray matter.

CONCLUSION

Compared with existing models in terms of DSC, HD, and ASSD, the proposed MDU-Net model demonstrated the best performance on average and showed its potential to enhance the accuracy of automatic segmentation for head MRI scans.

Automatic view plane prescription for cardiac magnetic resonance imaging via supervision by spatial relationship between views

BACKGROUND

View planning for the acquisition of cardiac magnetic resonance (CMR) imaging remains a demanding task in clinical practice.

PURPOSE

Existing approaches to its automation relied either on an additional volumetric image not typically acquired in clinic routine, or on laborious manual annotations of cardiac structural landmarks. This work presents a clinic-compatible, annotation-free system for automatic CMR view planning.

METHODS

The system mines the spatial relationship-more specifically, locates the intersecting lines-between the target planes and source views, and trains U-Net-based deep networks to regress heatmaps defined by distances from the intersecting lines. On the one hand, the intersection lines are the prescription lines prescribed by the technologists at the time of image acquisition using cardiac landmarks, and retrospectively identified from the spatial relationship. On the other hand, as the spatial relationship is self-contained in properly stored data, for example, in the DICOM format, the need for additional manual annotation is eliminated. In addition, the interplay of the multiple target planes predicted in a source view is utilized in a stacked hourglass architecture consisting of repeated U-Net-style building blocks to gradually improve the regression. Then, a multiview planning strategy is proposed to aggregate information from the predicted heatmaps for all the source views of a target plane, for a globally optimal prescription, mimicking the similar strategy practiced by skilled human prescribers. For performance evaluation, the retrospectively identified planes prescribed by the technologists are used as the ground truth, and the plane angle differences and localization distances between the planes prescribed by our system and the ground truth are compared.

RESULTS

The retrospective experiments include 181 clinical CMR exams, which are randomly split into training, validation, and test sets in the ratio of 64:16:20. Our system yields the mean angular difference and point-to-plane distance of 5.68∘ $^\circ$ and 3.12 mm, respectively, on the held-out test set. It not only achieves superior accuracy to existing approaches including conventional atlas-based and newer deep-learning-based in prescribing the four standard CMR planes but also demonstrates prescription of the first cardiac-anatomy-oriented plane(s) from the body-oriented scout.

CONCLUSIONS

The proposed system demonstrates accurate automatic CMR view plane prescription based on deep learning on properly archived data, without the need for further manual annotation. This work opens a new direction for automatic view planning of anatomy-oriented medical imaging beyond CMR.

Attention-UNet architectures with pretrained backbones for multi-class cardiac MR image segmentation

Segmentation architectures based on deep learning proficient extraordinary results in medical imaging technologies. Computed tomography (CT) images and Magnetic Resonance Imaging (MRI) in diagnosis and treatment are increasing and significantly support the diagnostic process by removing the bottlenecks of manual segmentation. Cardiac Magnetic Resonance Imaging (CMRI) is a state-of-the-art imaging technique used to acquire vital heart measurements and has received extensive attention from researchers for automatic segmentation. Deep learning methods offer high-precision segmentation but still pose several difficulties, such as pixel homogeneity in nearby organs. The motivated study using the attention mechanism approach was introduced for medical images for automated algorithms. The experiment focuses on observing the impact of the attention mechanism with and without pretrained backbone networks on the UNet model. For the same, three networks are considered: Attention-UNet, Attention-UNet with resnet50 pretrained backbone and Attention-UNet with densenet121 pretrained backbone. The experiments are performed on the ACDC Challenge 2017 dataset. The performance is evaluated by conducting a comparative analysis based on the Dice Coefficient, IoU Coefficient, and cross-entropy loss calculations. The Attention-UNet, Attention-UNet with resnet50 pretrained backbone, and Attention-UNet with densenet121 pretrained backbone networks obtained Dice Coefficients of 0.9889, 0.9720, and 0.9801, respectively, along with corresponding IoU scores of 0.9781, 0.9457, and 0.9612. Results compared with the state-of-the-art methods indicate that the methods are on par with, or even superior in terms of both the Dice coefficient and Intersection-over-union.

Automatic right ventricular segmentation for cine cardiac magnetic resonance images based on a new deep atlas network

BACKGROUND

The high morbidity and mortality of heart disease present a significant threat to human health. The development of methods for the quick and accurate diagnosis of heart diseases, enabling their effective treatment, has become a key issue of concern. Right ventricular (RV) segmentation from cine cardiac magnetic resonance (CMR) images plays a significant role in evaluating cardiac function for clinical diagnosis and prognosis. However, due to the complex structure of the RV, traditional segmentation methods are ineffective for RV segmentation.

PURPOSE

In this paper, we propose a new deep atlas network to improve the learning efficiency and segmentation accuracy of a deep learning network by integrating multi-atlas.

METHODS

First, a dense multi-scale U-net (DMU-net) is presented to acquire transformation parameters from atlas images to target images. The transformation parameters map the atlas image labels to the target image labels. Second, using a spatial transformation layer, the atlas images are deformed based on these parameters. Finally, the network is optimized by backpropagation with two loss functions where the mean squared error function (MSE) is used to measure the similarity of the input images and transformed images. Further, the Dice metric (DM) is used to quantify the overlap between the predicted contours and the ground truth. In our experiments, 15 datasets are used in testing, and 20 cine CMR images are selected as atlas.

RESULTS

The mean values and standard deviations for the DM and Hausdorff distance are 0.871 and 4.67 mm, 0.104 and 2.528 mm, respectively. The correlation coefficients of endo-diastolic volume, endo-systolic volume, ejection fraction, and stroke volume are 0.984, 0.926, 0.980, and 0.991, respectively, and the mean differences between all of the mentioned parameters are 3.2, -1.7, 0.02, and 4.9, respectively. Most of these differences are within the allowable range of 95%, indicating that the results are acceptable and show good consistency. The segmentation results obtained in this method are compared with those obtained by other methods that provide satisfactory performance. The other methods provide better segmentation effects at the base, but either no segmentation or the wrong segmentation at the top, which demonstrate that the deep atlas network can improve top-area segmentation accuracy.

CONCLUSION

Our results indicate that the proposed method can achieve better segmentation results than the previous methods, with both high relevance and consistency, and has the potential for clinical application.

Anatomical Prior-Based Automatic Segmentation for Cardiac Substructures from Computed Tomography Images

Cardiac substructure segmentation is a prerequisite for cardiac diagnosis and treatment, providing a basis for accurate calculation, modeling, and analysis of the entire cardiac structure. CT (computed tomography) imaging can be used for a noninvasive qualitative and quantitative evaluation of the cardiac anatomy and function. Cardiac substructures have diverse grayscales, fuzzy boundaries, irregular shapes, and variable locations. We designed a deep learning-based framework to improve the accuracy of the automatic segmentation of cardiac substructures. This framework integrates cardiac anatomical knowledge; it uses prior knowledge of the location, shape, and scale of cardiac substructures and separately processes the structures of different scales. Through two successive segmentation steps with a coarse-to-fine cascaded network, the more easily segmented substructures were coarsely segmented first; then, the more difficult substructures were finely segmented. The coarse segmentation result was used as prior information and combined with the original image as the input for the model. Anatomical knowledge of the large-scale substructures was embedded into the fine segmentation network to guide and train the small-scale substructures, achieving efficient and accurate segmentation of ten cardiac substructures. Sixty cardiac CT images and ten substructures manually delineated by experienced radiologists were retrospectively collected; the model was evaluated using the DSC (Dice similarity coefficient), Recall, Precision, and the Hausdorff distance. Compared with current mainstream segmentation models, our approach demonstrated significantly higher segmentation accuracy, with accurate segmentation of ten substructures of different shapes and sizes, indicating that the segmentation framework fused with prior anatomical knowledge has superior segmentation performance and can better segment small targets in multi-target segmentation tasks.

A task-unified network with transformer and spatial-temporal convolution for left ventricular quantification

Quantification of the cardiac function is vital for diagnosing and curing the cardiovascular diseases. Left ventricular function measurement is the most commonly used measure to evaluate the function of cardiac in clinical practice, how to improve the accuracy of left ventricular quantitative assessment results has always been the subject of research by medical researchers. Although considerable efforts have been put forward to measure the left ventricle (LV) automatically using deep learning methods, the accurate quantification is yet a challenge work as a result of the changeable anatomy structure of heart in the systolic diastolic cycle. Besides, most methods used direct regression method which lacks of visual based analysis. In this work, a deep learning segmentation and regression task-unified network with transformer and spatial-temporal convolution is proposed to segment and quantify the LV simultaneously. The segmentation module leverages a U-Net like 3D Transformer model to predict the contour of three anatomy structures, while the regression module learns spatial-temporal representations from the original images and the reconstruct feature map from segmentation path to estimate the finally desired quantification metrics. Furthermore, we employ a joint task loss function to train the two module networks. Our framework is evaluated on the MICCAI 2017 Left Ventricle Full Quantification Challenge dataset. The results of experiments demonstrate the effectiveness of our framework, which achieves competitive cardiac quantification metric results and at the same time produces visualized segmentation results that are conducive to later analysis.

CAT: Constrained Adversarial Training for Anatomically-Plausible Semi-Supervised Segmentation

Deep learning models for semi-supervised medical image segmentation have achieved unprecedented performance for a wide range of tasks. Despite their high accuracy, these models may however yield predictions that are considered anatomically impossible by clinicians. Moreover, incorporating complex anatomical constraints into standard deep learning frameworks remains challenging due to their non-differentiable nature. To address these limitations, we propose a Constrained Adversarial Training (CAT) method that learns how to produce anatomically plausible segmentations. Unlike approaches focusing solely on accuracy measures like Dice, our method considers complex anatomical constraints like connectivity, convexity, and symmetry which cannot be easily modeled in a loss function. The problem of non-differentiable constraints is solved using a Reinforce algorithm which enables to obtain a gradient for violated constraints. To generate constraint-violating examples on the fly, and thereby obtain useful gradients, our method adopts an adversarial training strategy which modifies training images to maximize the constraint loss, and then updates the network to be robust to these adversarial examples. The proposed method offers a generic and efficient way to add complex segmentation constraints on top of any segmentation network. Experiments on synthetic data and four clinically-relevant datasets demonstrate the effectiveness of our method in terms of segmentation accuracy and anatomical plausibility.

Radiomics-Based Quality Control System for Automatic Cardiac Segmentation: A Feasibility Study

PURPOSE

In the past decade, there has been a rapid increase in the development of automatic cardiac segmentation methods. However, the automatic quality control (QC) of these segmentation methods has received less attention. This study aims to address this gap by developing an automatic pipeline that incorporates DL-based cardiac segmentation and radiomics-based quality control.

METHODS

In the DL-based localization and segmentation part, the entire heart was first located and cropped. Then, the cropped images were further utilized for the segmentation of the right ventricle cavity (RVC), myocardium (MYO), and left ventricle cavity (LVC). As for the radiomics-based QC part, a training radiomics dataset was created with segmentation tasks of various quality. This dataset was used for feature extraction, selection, and QC model development. The model performance was then evaluated using both internal and external testing datasets.

RESULTS

In the internal testing dataset, the segmentation model demonstrated a great performance with a dice similarity coefficient (DSC) of 0.954 for whole heart segmentations. Images were then appropriately cropped to 160 × 160 pixels. The models also performed well for cardiac substructure segmentations. The DSC values were 0.863, 0.872, and 0.940 for RVC, MYO, and LVC for 2D masks and 0.928, 0.886, and 0.962 for RVC, MYO, and LVC for 3D masks with an attention-UNet. After feature selection with the radiomics dataset, we developed a series of models to predict the automatic segmentation quality and its DSC value for the RVC, MYO, and LVC structures. The mean absolute values for our best prediction models were 0.060, 0.032, and 0.021 for 2D segmentations and 0.027, 0.017, and 0.011 for 3D segmentations, respectively. Additionally, the radiomics-based classification models demonstrated a high negative detection rate of >0.85 in all 2D groups. In the external dataset, models showed similar results.

CONCLUSIONS

We developed a pipeline including cardiac substructure segmentation and QC at both the slice (2D) and subject (3D) levels. Our results demonstrate that the radiomics method possesses great potential for the automatic QC of cardiac segmentation.

BATFormer: Towards Boundary-Aware Lightweight Transformer for Efficient Medical Image Segmentation

OBJECTIVE

Transformers, born to remedy the inadequate receptive fields of CNNs, have drawn explosive attention recently. However, the daunting computational complexity of global representation learning, together with rigid window partitioning, hinders their deployment in medical image segmentation. This work aims to address the above two issues in transformers for better medical image segmentation.

METHODS

We propose a boundary-aware lightweight transformer (BATFormer) that can build cross-scale global interaction with lower computational complexity and generate windows flexibly under the guidance of entropy. Specifically, to fully explore the benefits of transformers in long-range dependency establishment, a cross-scale global transformer (CGT) module is introduced to jointly utilize multiple small-scale feature maps for richer global features with lower computational complexity. Given the importance of shape modeling in medical image segmentation, a boundary-aware local transformer (BLT) module is constructed. Different from rigid window partitioning in vanilla transformers which would produce boundary distortion, BLT adopts an adaptive window partitioning scheme under the guidance of entropy for both computational complexity reduction and shape preservation.

RESULTS

BATFormer achieves the best performance in Dice of 92.84 %, 91.97 %, 90.26 %, and 96.30 % for the average, right ventricle, myocardium, and left ventricle respectively on the ACDC dataset and the best performance in Dice, IoU, and ACC of 90.76 %, 84.64 %, and 96.76 % respectively on the ISIC 2018 dataset. More importantly, BATFormer requires the least amount of model parameters and the lowest computational complexity compared to the state-of-the-art approaches.

CONCLUSION AND SIGNIFICANCE

Our results demonstrate the necessity of developing customized transformers for efficient and better medical image segmentation. We believe the design of BATFormer is inspiring and extendable to other applications/frameworks.

Deep Learning Synthetic Strain: Quantitative Assessment of Regional Myocardial Wall Motion at MRI

Purpose

To assess the feasibility of a newly developed algorithm, called deep learning synthetic strain (DLSS), to infer myocardial velocity from cine steady-state free precession (SSFP) images and detect wall motion abnormalities in patients with ischemic heart disease.

Materials and Methods

In this retrospective study, DLSS was developed by using a data set of 223 cardiac MRI examinations including cine SSFP images and four-dimensional flow velocity data (November 2017 to May 2021). To establish normal ranges, segmental strain was measured in 40 individuals (mean age, 41 years ± 17 [SD]; 30 men) without cardiac disease. Then, DLSS performance in the detection of wall motion abnormalities was assessed in a separate group of patients with coronary artery disease, and these findings were compared with consensus results of four independent cardiothoracic radiologists (ground truth). Algorithm performance was evaluated by using receiver operating characteristic curve analysis.

Results

Median peak segmental radial strain in individuals with normal cardiac MRI findings was 38% (IQR: 30%-48%). Among patients with ischemic heart disease (846 segments in 53 patients; mean age, 61 years ± 12; 41 men), the Cohen κ among four cardiothoracic readers for detecting wall motion abnormalities was 0.60-0.78. DLSS achieved an area under the receiver operating characteristic curve of 0.90. Using a fixed 30% threshold for abnormal peak radial strain, the algorithm achieved a sensitivity, specificity, and accuracy of 86%, 85%, and 86%, respectively.

Conclusion

The deep learning algorithm had comparable performance with subspecialty radiologists in inferring myocardial velocity from cine SSFP images and identifying myocardial wall motion abnormalities at rest in patients with ischemic heart disease.Keywords: Neural Networks, Cardiac, MR Imaging, Ischemia/Infarction Supplemental material is available for this article. © RSNA, 2023.

Automated diagnosis of cardiovascular diseases from cardiac magnetic resonance imaging using deep learning models: A review

In recent years, cardiovascular diseases (CVDs) have become one of the leading causes of mortality globally. At early stages, CVDs appear with minor symptoms and progressively get worse. The majority of people experience symptoms such as exhaustion, shortness of breath, ankle swelling, fluid retention, and other symptoms when starting CVD. Coronary artery disease (CAD), arrhythmia, cardiomyopathy, congenital heart defect (CHD), mitral regurgitation, and angina are the most common CVDs. Clinical methods such as blood tests, electrocardiography (ECG) signals, and medical imaging are the most effective methods used for the detection of CVDs. Among the diagnostic methods, cardiac magnetic resonance imaging (CMRI) is increasingly used to diagnose, monitor the disease, plan treatment and predict CVDs. Coupled with all the advantages of CMR data, CVDs diagnosis is challenging for physicians as each scan has many slices of data, and the contrast of it might be low. To address these issues, deep learning (DL) techniques have been employed in the diagnosis of CVDs using CMR data, and much research is currently being conducted in this field. This review provides an overview of the studies performed in CVDs detection using CMR images and DL techniques. The introduction section examined CVDs types, diagnostic methods, and the most important medical imaging techniques. The following presents research to detect CVDs using CMR images and the most significant DL methods. Another section discussed the challenges in diagnosing CVDs from CMRI data. Next, the discussion section discusses the results of this review, and future work in CVDs diagnosis from CMR images and DL techniques are outlined. Finally, the most important findings of this study are presented in the conclusion section.

White matter injury detection based on preterm infant cranial ultrasound images

Introduction

White matter injury (WMI) is now the major disease that seriously affects the quality of life of preterm infants and causes cerebral palsy of children, which also causes periventricular leuko-malacia (PVL) in severe cases. The study aimed to develop a method based on cranial ultrasound images to evaluate the risk of WMI.

Methods

This study proposed an ultrasound radiomics diagnostic system to predict the WMI risk. A multi-task deep learning model was used to segment white matter and predict the WMI risk simultaneously. In total, 158 preterm infants with 807 cranial ultrasound images were enrolled. WMI occurred in 32preterm infants (20.3%, 32/158).

Results

Ultrasound radiomics diagnostic system implemented a great result with AUC of 0.845 in the testing set. Meanwhile, multi-task deep learning model preformed a promising result both in segmentation of white matter with a Dice coefficient of 0.78 and prediction of WMI risk with AUC of 0.863 in the testing cohort.

Discussion

In this study, we presented a data-driven diagnostic system for white matter injury in preterm infants. The system combined multi-task deep learning and traditional radiomics features to achieve automatic detection of white matter regions on the one hand, and design a fusion strategy of deep learning features and manual radiomics features on the other hand to obtain stable and efficient diagnostic performance.

Left ventricle segmentation combining deep learning and deformable models with anatomical constraints

Segmentation of the left ventricle is a key approach in Cardiac Magnetic Resonance Imaging for calculating biomarkers in diagnosis. Since there is substantial effort required from the expert, many automatic segmentation methods have been proposed, in which deep learning networks have obtained remarkable performance. However, one of the main limitations of these approaches is the production of segmentations that contain anatomical errors. To avoid this limitation, we propose a new fully-automatic left ventricle segmentation method combining deep learning and deformable models. We propose a new level set energy formulation that includes exam-specific information estimated from the deep learning segmentation and shape constraints. The method is part of a pipeline containing pre-processing steps and a failure correction post-processing step. Experiments were conducted with the Sunnybrook and ACDC public datasets, and a private dataset. Results suggest that the method is competitive, that it can produce anatomically consistent segmentations, has good generalization ability, and is often able to estimate biomarkers close to the expert.

Multilevel comparison of deep learning models for function quantification in cardiovascular magnetic resonance: On the redundancy of architectural variations

Background

Cardiac function quantification in cardiovascular magnetic resonance requires precise contouring of the heart chambers. This time-consuming task is increasingly being addressed by a plethora of ever more complex deep learning methods. However, only a small fraction of these have made their way from academia into clinical practice. In the quality assessment and control of medical artificial intelligence, the opaque reasoning and associated distinctive errors of neural networks meet an extraordinarily low tolerance for failure.

Aim

The aim of this study is a multilevel analysis and comparison of the performance of three popular convolutional neural network (CNN) models for cardiac function quantification.

Methods

U-Net, FCN, and MultiResUNet were trained for the segmentation of the left and right ventricles on short-axis cine images of 119 patients from clinical routine. The training pipeline and hyperparameters were kept constant to isolate the influence of network architecture. CNN performance was evaluated against expert segmentations for 29 test cases on contour level and in terms of quantitative clinical parameters. Multilevel analysis included breakdown of results by slice position, as well as visualization of segmentation deviations and linkage of volume differences to segmentation metrics via correlation plots for qualitative analysis.

Results

All models showed strong correlation to the expert with respect to quantitative clinical parameters (rz ' = 0.978, 0.977, 0.978 for U-Net, FCN, MultiResUNet respectively). The MultiResUNet significantly underestimated ventricular volumes and left ventricular myocardial mass. Segmentation difficulties and failures clustered in basal and apical slices for all CNNs, with the largest volume differences in the basal slices (mean absolute error per slice: 4.2 ± 4.5 ml for basal, 0.9 ± 1.3 ml for midventricular, 0.9 ± 0.9 ml for apical slices). Results for the right ventricle had higher variance and more outliers compared to the left ventricle. Intraclass correlation for clinical parameters was excellent (≥0.91) among the CNNs.

Conclusion

Modifications to CNN architecture were not critical to the quality of error for our dataset. Despite good overall agreement with the expert, errors accumulated in basal and apical slices for all models.

Deep learning-based image segmentation model using an MRI-based convolutional neural network for physiological evaluation of the heart

Background and Objective: Cardiovascular disease is a high-fatality health issue. Accurate measurement of cardiovascular function depends on precise segmentation of physiological structure and accurate evaluation of functional parameters. Structural segmentation of heart images and calculation of the volume of different ventricular activity cycles form the basis for quantitative analysis of physiological function and can provide the necessary support for clinical physiological diagnosis, as well as the analysis of various cardiac diseases. Therefore, it is important to develop an efficient heart segmentation algorithm. Methods: A total of 275 nuclear magnetic resonance imaging (MRI) heart scans were collected, analyzed, and preprocessed from Huaqiao University Affiliated Strait Hospital, and the data were used in our improved deep learning model, which was designed based on the U-net network. The training set included 80% of the images, and the remaining 20% was the test set. Based on five time phases from end-diastole (ED) to end-systole (ES), the segmentation findings showed that it is possible to achieve improved segmentation accuracy and computational complexity by segmenting the left ventricle (LV), right ventricle (RV), and myocardium (myo). Results: We improved the Dice index of the LV to 0.965 and 0.921, and the Hausdorff index decreased to 5.4 and 6.9 in the ED and ES phases, respectively; RV Dice increased to 0.938 and 0.860, and the Hausdorff index decreased to 11.7 and 12.6 in the ED and ES, respectively; myo Dice increased to 0.889 and 0.901, and the Hausdorff index decreased to 8.3 and 9.2 in the ED and ES, respectively. Conclusion: The model obtained in the final experiment provided more accurate segmentation of the left and right ventricles, as well as the myocardium, from cardiac MRI. The data from this model facilitate the prediction of cardiovascular disease in real-time, thereby providing potential clinical utility.

Interpretable cardiac anatomy modeling using variational mesh autoencoders

Cardiac anatomy and function vary considerably across the human population with important implications for clinical diagnosis and treatment planning. Consequently, many computer-based approaches have been developed to capture this variability for a wide range of applications, including explainable cardiac disease detection and prediction, dimensionality reduction, cardiac shape analysis, and the generation of virtual heart populations. In this work, we propose a variational mesh autoencoder (mesh VAE) as a novel geometric deep learning approach to model such population-wide variations in cardiac shapes. It embeds multi-scale graph convolutions and mesh pooling layers in a hierarchical VAE framework to enable direct processing of surface mesh representations of the cardiac anatomy in an efficient manner. The proposed mesh VAE achieves low reconstruction errors on a dataset of 3D cardiac meshes from over 1,000 patients with acute myocardial infarction, with mean surface distances between input and reconstructed meshes below the underlying image resolution. We also find that it outperforms a voxelgrid-based deep learning benchmark in terms of both mean surface distance and Hausdorff distance while requiring considerably less memory. Furthermore, we explore the quality and interpretability of the mesh VAE's latent space and showcase its ability to improve the prediction of major adverse cardiac events over a clinical benchmark. Finally, we investigate the method's ability to generate realistic virtual populations of cardiac anatomies and find good alignment between the synthesized and gold standard mesh populations in terms of multiple clinical metrics.

CyCoSeg: A Cyclic Collaborative Framework for Automated Medical Image Segmentation

Deep neural networks have been tremendously successful at segmenting objects in images. However, it has been shown they still have limitations on challenging problems such as the segmentation of medical images. The main reason behind this lower success resides in the reduced size of the object in the image. In this paper we overcome this limitation through a cyclic collaborative framework, CyCoSeg. The proposed framework is based on a deep active shape model (D-ASM), which provides prior information about the shape of the object, and a semantic segmentation network (SSN). These two models collaborate to reach the desired segmentation by influencing each other: SSN helps D-ASM identify relevant keypoints in the image through an Expectation Maximization formulation, while D-ASM provides a segmentation proposal that guides the SSN. This cycle is repeated until both models converge. Extensive experimental evaluation shows CyCoSeg boosts the performance of the baseline models, including several popular SSNs, while avoiding major architectural modifications. The effectiveness of our method is demonstrated on the left ventricle segmentation on two benchmark datasets, where our approach achieves one of the most competitive results in segmentation accuracy. Furthermore, its generalization is demonstrated for lungs and kidneys segmentation in CT scans.

A Review on Multiscale-Deep-Learning Applications

In general, most of the existing convolutional neural network (CNN)-based deep-learning models suffer from spatial-information loss and inadequate feature-representation issues. This is due to their inability to capture multiscale-context information and the exclusion of semantic information throughout the pooling operations. In the early layers of a CNN, the network encodes simple semantic representations, such as edges and corners, while, in the latter part of the CNN, the network encodes more complex semantic features, such as complex geometric shapes. Theoretically, it is better for a CNN to extract features from different levels of semantic representation because tasks such as classification and segmentation work better when both simple and complex feature maps are utilized. Hence, it is also crucial to embed multiscale capability throughout the network so that the various scales of the features can be optimally captured to represent the intended task. Multiscale representation enables the network to fuse low-level and high-level features from a restricted receptive field to enhance the deep-model performance. The main novelty of this review is the comprehensive novel taxonomy of multiscale-deep-learning methods, which includes details of several architectures and their strengths that have been implemented in the existing works. Predominantly, multiscale approaches in deep-learning networks can be classed into two categories: multiscale feature learning and multiscale feature fusion. Multiscale feature learning refers to the method of deriving feature maps by examining kernels over several sizes to collect a larger range of relevant features and predict the input images' spatial mapping. Multiscale feature fusion uses features with different resolutions to find patterns over short and long distances, without a deep network. Additionally, several examples of the techniques are also discussed according to their applications in satellite imagery, medical imaging, agriculture, and industrial and manufacturing systems.

Crosslink-Net: Double-Branch Encoder Network via Fusing Vertical and Horizontal Convolutions for Medical Image Segmentation

Accurate image segmentation plays a crucial role in medical image analysis, yet it faces great challenges caused by various shapes, diverse sizes, and blurry boundaries. To address these difficulties, square kernel-based encoder-decoder architectures have been proposed and widely used, but their performance remains unsatisfactory. To further address these challenges, we present a novel double-branch encoder architecture. Our architecture is inspired by two observations. (1) Since the discrimination of the features learned via square convolutional kernels needs to be further improved, we propose utilizing nonsquare vertical and horizontal convolutional kernels in a double-branch encoder so that the features learned by both branches can be expected to complement each other. (2) Considering that spatial attention can help models to better focus on the target region in a large-sized image, we develop an attention loss to further emphasize the segmentation of small-sized targets. With the above two schemes, we develop a novel double-branch encoder-based segmentation framework for medical image segmentation, namely, Crosslink-Net, and validate its effectiveness on five datasets with experiments. The code is released at https://github.com/Qianyu1226/Crosslink-Net.

Knowledge distillation with ensembles of convolutional neural networks for medical image segmentation

Purpose: Ensembles of convolutional neural networks (CNNs) often outperform a single CNN in medical image segmentation tasks, but inference is computationally more expensive and makes ensembles unattractive for some applications. We compared the performance of differently constructed ensembles with the performance of CNNs derived from these ensembles using knowledge distillation, a technique for reducing the footprint of large models such as ensembles. Approach: We investigated two different types of ensembles, namely, diverse ensembles of networks with three different architectures and two different loss-functions, and uniform ensembles of networks with the same architecture but initialized with different random seeds. For each ensemble, additionally, a single student network was trained to mimic the class probabilities predicted by the teacher model, the ensemble. We evaluated the performance of each network, the ensembles, and the corresponding distilled networks across three different publicly available datasets. These included chest computed tomography scans with four annotated organs of interest, brain magnetic resonance imaging (MRI) with six annotated brain structures, and cardiac cine-MRI with three annotated heart structures. Results: Both uniform and diverse ensembles obtained better results than any of the individual networks in the ensemble. Furthermore, applying knowledge distillation resulted in a single network that was smaller and faster without compromising performance compared with the ensemble it learned from. The distilled networks significantly outperformed the same network trained with reference segmentation instead of knowledge distillation. Conclusion: Knowledge distillation can compress segmentation ensembles of uniform or diverse composition into a single CNN while maintaining the performance of the ensemble.

Comparison of two-dimensional and three-dimensional U-Net architectures for segmentation of adipose tissue in cardiac magnetic resonance images

The process of identifying cardiac adipose tissue (CAT) from volumetric magnetic resonance imaging of the heart is tedious, time-consuming, and often dependent on observer interpretation. Many 2-dimensional (2D) convolutional neural networks (CNNs) have been implemented to automate the cardiac segmentation process, but none have attempted to identify CAT. Furthermore, the results from automatic segmentation of other cardiac structures leave room for improvement. This study investigated the viability of a 3-dimensional (3D) CNN in comparison to a similar 2D CNN. Both models used a U-Net architecture to simultaneously classify CAT, left myocardium, left ventricle, and right myocardium. The multi-phase model trained with multiple observers' segmentations reached a whole-volume Dice similarity coefficient (DSC) of 0.925 across all classes and 0.640 for CAT specifically; the corresponding 2D model's DSC across all classes was 0.902 and 0.590 for CAT specifically. This 3D model also achieved a higher level of CAT-specific DSC agreement with a group of observers with a Williams Index score of 0.973 in comparison to the 2D model's score of 0.822.

Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency

The interest in machine learning (ML) has grown tremendously in recent years, partly due to the performance leap that occurred with new techniques of deep learning, convolutional neural networks for images, increased computational power, and wider availability of large datasets. Most fields of medicine follow that popular trend and, notably, radiation oncology is one of those that are at the forefront, with already a long tradition in using digital images and fully computerized workflows. ML models are driven by data, and in contrast with many statistical or physical models, they can be very large and complex, with countless generic parameters. This inevitably raises two questions, namely, the tight dependence between the models and the datasets that feed them, and the interpretability of the models, which scales with its complexity. Any problems in the data used to train the model will be later reflected in their performance. This, together with the low interpretability of ML models, makes their implementation into the clinical workflow particularly difficult. Building tools for risk assessment and quality assurance of ML models must involve then two main points: interpretability and data-model dependency. After a joint introduction of both radiation oncology and ML, this paper reviews the main risks and current solutions when applying the latter to workflows in the former. Risks associated with data and models, as well as their interaction, are detailed. Next, the core concepts of interpretability, explainability, and data-model dependency are formally defined and illustrated with examples. Afterwards, a broad discussion goes through key applications of ML in workflows of radiation oncology as well as vendors' perspectives for the clinical implementation of ML.

From Accuracy to Reliability and Robustness in Cardiac Magnetic Resonance Image Segmentation: A Review

Since the rise of deep learning (DL) in the mid-2010s, cardiac magnetic resonance (CMR) image segmentation has achieved state-of-the-art performance. Despite achieving inter-observer variability in terms of different accuracy performance measures, visual inspections reveal errors in most segmentation results, indicating a lack of reliability and robustness of DL segmentation models, which can be critical if a model was to be deployed into clinical practice. In this work, we aim to bring attention to reliability and robustness, two unmet needs of cardiac image segmentation methods, which are hampering their translation into practice. To this end, we first study the performance accuracy evolution of CMR segmentation, illustrate the improvements brought by DL algorithms and highlight the symptoms of performance stagnation. Afterwards, we provide formal definitions of reliability and robustness. Based on the two definitions, we identify the factors that limit the reliability and robustness of state-of-the-art deep learning CMR segmentation techniques. Finally, we give an overview of the current set of works that focus on improving the reliability and robustness of CMR segmentation, and we categorize them into two families of methods: quality control methods and model improvement techniques. The first category corresponds to simpler strategies that only aim to flag situations where a model may be incurring poor reliability or robustness. The second one, instead, directly tackles the problem by bringing improvements into different aspects of the CMR segmentation model development process. We aim to bring the attention of more researchers towards these emerging trends regarding the development of reliable and robust CMR segmentation frameworks, which can guarantee the safe use of DL in clinical routines and studies.

An Improved 3D Deep Learning-Based Segmentation of Left Ventricular Myocardial Diseases from Delayed-Enhancement MRI with Inclusion and Classification Prior Information U-Net (ICPIU-Net)

Accurate segmentation of the myocardial scar may supply relevant advancements in predicting and controlling deadly ventricular arrhythmias in subjects with cardiovascular disease. In this paper, we propose the architecture of inclusion and classification of prior information U-Net (ICPIU-Net) to efficiently segment the left ventricle (LV) myocardium, myocardial infarction (MI), and microvascular-obstructed (MVO) tissues from late gadolinium enhancement magnetic resonance (LGE-MR) images. Our approach was developed using two subnets cascaded to first segment the LV cavity and myocardium. Then, we used inclusion and classification constraint networks to improve the resulting segmentation of the diseased regions within the pre-segmented LV myocardium. This network incorporates the inclusion and classification information of the LGE-MRI to maintain topological constraints of pathological areas. In the testing stage, the outputs of each segmentation network obtained with specific estimated parameters from training were fused using the majority voting technique for the final label prediction of each voxel in the LGE-MR image. The proposed method was validated by comparing its results to manual drawings by experts from 50 LGE-MR images. Importantly, compared to various deep learning-based methods participating in the EMIDEC challenge, the results of our approach have a more significant agreement with manual contouring in segmenting myocardial diseases.

Modality specific U-Net variants for biomedical image segmentation: a survey

With the advent of advancements in deep learning approaches, such as deep convolution neural network, residual neural network, adversarial network; U-Net architectures are most widely utilized in biomedical image segmentation to address the automation in identification and detection of the target regions or sub-regions. In recent studies, U-Net based approaches have illustrated state-of-the-art performance in different applications for the development of computer-aided diagnosis systems for early diagnosis and treatment of diseases such as brain tumor, lung cancer, alzheimer, breast cancer, etc., using various modalities. This article contributes in presenting the success of these approaches by describing the U-Net framework, followed by the comprehensive analysis of the U-Net variants by performing (1) inter-modality, and (2) intra-modality categorization to establish better insights into the associated challenges and solutions. Besides, this article also highlights the contribution of U-Net based frameworks in the ongoing pandemic, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) also known as COVID-19. Finally, the strengths and similarities of these U-Net variants are analysed along with the challenges involved in biomedical image segmentation to uncover promising future research directions in this area.

Myocardial Segmentation of Cardiac MRI Sequences With Temporal Consistency for Coronary Artery Disease Diagnosis

Coronary artery disease (CAD) is the most common cause of death globally, and its diagnosis is usually based on manual myocardial (MYO) segmentation of MRI sequences. As manual segmentation is tedious, time-consuming, and with low replicability, automatic MYO segmentation using machine learning techniques has been widely explored recently. However, almost all the existing methods treat the input MRI sequences independently, which fails to capture the temporal information between sequences, e.g., the shape and location information of the myocardium in sequences along time. In this article, we propose a MYO segmentation framework for sequence of cardiac MRI (CMR) scanning images of the left ventricular (LV) cavity, right ventricular (RV) cavity, and myocardium. Specifically, we propose to combine conventional neural networks and recurrent neural networks to incorporate temporal information between sequences to ensure temporal consistency. We evaluated our framework on the automated cardiac diagnosis challenge (ACDC) dataset. The experiment results demonstrate that our framework can improve the segmentation accuracy by up to 2% in the Dice coefficient.

A Novel Framework with Weighted Decision Map Based on Convolutional Neural Network for Cardiac MR Segmentation

For diagnosing cardiovascular disease, an accurate segmentation method is needed. There are several unre-solved issues in the complex field of cardiac magnetic resonance imaging, some of which have been partially addressed by using deep neural networks. To solve two problems of over-segmentation and under-segmentation of anatomical shapes in the short-axis view from different cardiac magnetic resonance sequences, we propose a novel two-stage framework with a weighted decision map based on convolutional neural networks to segment the myocardium (Myo), left ventricle (LV), and right ventricle (RV) simultaneously. The framework comprises a deci-sion map extractor and a cardiac segmenter. A cascaded U-Net++ is used as a decision map extractor to acquire the decision map that decides the category of each pixel. Cardiac segmenter is a multiscale dual-path feature ag-gregation network (MDFA-Net) which consists of a densely connected network and an asymmetric encoding and decoding network. The input to the cardiac seg-menter is derived from processed original images weighted by the output of the decision map extractor. We conducted experiments on two datasets of mul-ti-sequence cardiac magnetic resonance segmentation challenge 2019 (MS-CMRSeg 2019) and myocardial pa-thology segmentation challenge 2020 (MyoPS 2020). Test results obtained on MyoPS 2020 show that proposed method with average Dice coefficient of 84.70%, 86.00% and 86.31% in the segmentation task of Myo, LV, and RV, respectively.

Light-Weight Localization and Scale-Independent Multi-gate UNET Segmentation of Left and Right Ventricles in MRI Images

PURPOSE

Heart segmentation in cardiac magnetic resonance images is heavily used during the assessment of left ventricle global function. Automation of the segmentation is crucial to standardize the analysis. This study aims at developing a CNN-based framework to aid the clinical measurements of the left ventricle and right ventricle in cardiac magnetic resonance images.

METHODS

We propose a fully automated framework for localization and segmentation of the left ventricle and right ventricle in both short- and long-axis views from cardiac magnetic resonance images. The localization module utilizes a light-weight model that detects the region of interest and feeds it to the segmentation model. Also, we propose the Multi-Gate block as an extension to the UNet to boost the segmentation performance by aggregating multi-scale features. Comparison between our proposed method and the baseline UNet was performed to show the gain in the overall performance. The reliability of the model was assessed by testing the method against cardiac magnetic resonance images with different levels of noise and deformations.

RESULTS

Heart localization accuracy was 0.59 and 1.75 pixels in both short- and long-axis views respectively. Left and right ventricle blood-pool segmentation Dice was (0.93, 0.90) in end-systole and (0.97, 0.95) in end-diastole. The left ventricle myocardium was segmented accurately with Dice of 0.91 and 0.90 in end-systole and end-diastole respectively. Left ventricle ejection fraction was found to be highly correlated with the gold standard with r = 0.987. Moreover, the proposed pipeline is fast, achieving 0.002 sec per image on average.

CONCLUSION

Adding the Multi-Gate Dilated Inception Block has boosted the performance of UNet architecture and has shown generalization ability when tested on noisy and deformed cardiac magnetic resonance images. The proposed method has proven its wide applicability and reliability for heart detection when tested on different datasets.

A multiphase texture-based model of active contours assisted by a convolutional neural network for automatic CT and MRI heart ventricle segmentation

BACKGROUND

Left and right ventricle automatic segmentation remains one of the more important tasks in computed aided diagnosis. Active contours have shown to be efficient for this task, however they often require user interaction to provide the initial position, which drives the tool substantially dependent on a prior knowledge and a manual process.

METHODS

We propose to overcome this limitation with a Convolutional Neural Network (CNN) to reach the assumed target locations. This is followed by a novel multiphase active contour method based on texture that enhances whole heart patterns leading to an accurate identification of distinct regions, mainly left (LV) and right ventricle (RV) for the purposes of this work.

RESULTS

Experiments reveal that the initial location and estimated shape provided by the CNN are of great concern for the subsequent active contour stage. We assessed our method on two short data sets with Dice scores of 93% (LV-CT), 91% (LV-MRI), 0.86% (RV-CT) and 0.85% (RV-MRI).

CONCLUSION

Our approach overcomes the performance of other techniques by means of a multiregion segmentation assisted by a CNN trained with a limited data set, a typical issue in medical imaging.

Integrating multiple MRI sequences for pelvic organs segmentation via the attention mechanism

PURPOSE

To create a network which fully utilizes multi-sequence MRI and compares favorably with manual human contouring.

METHODS

We retrospectively collected 89 MRI studies of the pelvic cavity from patients with prostate cancer and cervical cancer. The dataset contained 89 samples from 87 patients with a total of 84 valid samples. MRI was performed with T1-weighted (T1), T2-weighted (T2), and Enhanced Dixon T1-weighted (T1DIXONC) sequences. There were two cohorts. The training cohort contained 55 samples and the testing cohort contained 29 samples. The MRI images in the training cohort contained contouring data from radiotherapist α. The MRI images in the testing cohort contained contouring data from radiotherapist α and contouring data from another radiotherapist: radiotherapist β. The training cohort was used to optimize the convolution neural networks, which included the attention mechanism through the proposed activation module and the blended module into multiple MRI sequences, to perform autodelineation. The testing cohort was used to assess the networks' autodelineation performance. The contoured organs at risk (OAR) were the anal canal, bladder, rectum, femoral head (L), and femoral head (R).

RESULTS

We compared our proposed network with UNet and FuseUNet using our dataset. When T1 was the main sequence, we input three sequences to segment five organs and evaluated the results using four metrics: the DSC (Dice similarity coefficient), the JSC (Jaccard similarity coefficient), the ASD (average mean distance), and the 95% HD (robust Hausdorff distance). The proposed network achieved improved results compared with the baselines among all metrics. The DSC were 0.834±0.029, 0.818±0.037, and 0.808±0.050 for our proposed network, FuseUNet, and UNet, respectively. The 95% HD were 7.256±2.748 mm, 8.404±3.297 mm, and 8.951±4.798 mm for our proposed network, FuseUNet, and UNet, respectively. Our proposed network also had superior performance on the JSC and ASD coefficients.

CONCLUSION

Our proposed activation module and blended module significantly improved the performance of FuseUNet for multi-sequence MRI segmentation. Our proposed network integrated multiple MRI sequences efficiently and autosegmented OAR rapidly and accurately. We also discovered that three-sequence fusion (T1-T1DIXONC-T2) was superior to two-sequence fusion (T1-T2 and T1-T1DIXONC, respectively). We infer that the more MRI sequences fused, the better the automatic segmentation results.

DeepStrain: A Deep Learning Workflow for the Automated Characterization of Cardiac Mechanics

Myocardial strain analysis from cinematic magnetic resonance imaging (cine-MRI) data provides a more thorough characterization of cardiac mechanics than volumetric parameters such as left-ventricular ejection fraction, but sources of variation including segmentation and motion estimation have limited its wider clinical use. We designed and validated a fast, fully-automatic deep learning (DL) workflow to generate both volumetric parameters and strain measures from cine-MRI data consisting of segmentation and motion estimation convolutional neural networks. The final motion network design, loss function, and associated hyperparameters are the result of a thorough ad hoc implementation that we carefully planned specific for strain quantification, tested, and compared to other potential alternatives. The optimal configuration was trained using healthy and cardiovascular disease (CVD) subjects (n = 150). DL-based volumetric parameters were correlated (>0.98) and without significant bias relative to parameters derived from manual segmentations in 50 healthy and CVD test subjects. Compared to landmarks manually-tracked on tagging-MRI images from 15 healthy subjects, landmark deformation using DL-based motion estimates from paired cine-MRI data resulted in an end-point-error of 2.9 ± 1.5 mm. Measures of end-systolic global strain from these cine-MRI data showed no significant biases relative to a tagging-MRI reference method. On 10 healthy subjects, intraclass correlation coefficient for intra-scanner repeatability was good to excellent (>0.75) for all global measures and most polar map segments. In conclusion, we developed and evaluated the first end-to-end learning-based workflow for automated strain analysis from cine-MRI data to quantitatively characterize cardiac mechanics of healthy and CVD subjects.