Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved?

Automated Measurement of Vascular Calcification in Femoral Endarterectomy Patients Using Deep Learning

Atherosclerosis, a chronic inflammatory disease affecting the large arteries, presents a global health risk. Accurate analysis of diagnostic images, like computed tomographic angiograms (CTAs), is essential for staging and monitoring the progression of atherosclerosis-related conditions, including peripheral arterial disease (PAD). However, manual analysis of CTA images is time-consuming and tedious. To address this limitation, we employed a deep learning model to segment the vascular system in CTA images of PAD patients undergoing femoral endarterectomy surgery and to measure vascular calcification from the left renal artery to the patella. Utilizing proprietary CTA images of 27 patients undergoing femoral endarterectomy surgery provided by Prisma Health Midlands, we developed a Deep Neural Network (DNN) model to first segment the arterial system, starting from the descending aorta to the patella, and second, to provide a metric of arterial calcification. Our designed DNN achieved 83.4% average Dice accuracy in segmenting arteries from aorta to patella, advancing the state-of-the-art by 0.8%. Furthermore, our work is the first to present a robust statistical analysis of automated calcification measurement in the lower extremities using deep learning, attaining a Mean Absolute Percentage Error (MAPE) of 9.5% and a correlation coefficient of 0.978 between automated and manual calcification scores. These findings underscore the potential of deep learning techniques as a rapid and accurate tool for medical professionals to assess calcification in the abdominal aorta and its branches above the patella.

Machine Learning-Based Segmentation of the Thoracic Aorta with Congenital Valve Disease Using MRI

Subjects with bicuspid aortic valves (BAV) are at risk of developing valve dysfunction and need regular clinical imaging surveillance. Management of BAV involves manual and time-consuming segmentation of the aorta for assessing left ventricular function, jet velocity, gradient, shear stress, and valve area with aortic valve stenosis. This paper aims to employ machine learning-based (ML) segmentation as a potential for improved BAV assessment and reducing manual bias. The focus is on quantifying the relationship between valve morphology and vortical structures, and analyzing how valve morphology influences the aorta's susceptibility to shear stress that may lead to valve incompetence. The ML-based segmentation that is employed is trained on whole-body Computed Tomography (CT). Magnetic Resonance Imaging (MRI) is acquired from six subjects, three with tricuspid aortic valves (TAV) and three functionally BAV, with right-left leaflet fusion. These are used for segmentation of the cardiovascular system and delineation of four-dimensional phase-contrast magnetic resonance imaging (4D-PCMRI) for quantification of vortical structures and wall shear stress. The ML-based segmentation model exhibits a high Dice score (0.86) for the heart organ, indicating a robust segmentation. However, the Dice score for the thoracic aorta is comparatively poor (0.72). It is found that wall shear stress is predominantly symmetric in TAVs. BAVs exhibit highly asymmetric wall shear stress, with the region opposite the fused coronary leaflets experiencing elevated tangential wall shear stress. This is due to the higher tangential velocity explained by helical flow, proximally of the sinutubal junction of the ascending aorta. ML-based segmentation not only reduces the runtime of assessing the hemodynamic effectiveness, but also identifies the significance of the tangential wall shear stress in addition to the axial wall shear stress that may lead to the progression of valve incompetence in BAVs, which could guide potential adjustments in surgical interventions.

Automated segmentation of 3D cine cardiovascular magnetic resonance imaging

Introduction

As the life expectancy of children with congenital heart disease (CHD) is rapidly increasing and the adult population with CHD is growing, there is an unmet need to improve clinical workflow and efficiency of analysis. Cardiovascular magnetic resonance (CMR) is a noninvasive imaging modality for monitoring patients with CHD. CMR exam is based on multiple breath-hold 2-dimensional (2D) cine acquisitions that should be precisely prescribed and is expert and institution dependent. Moreover, 2D cine images have relatively thick slices, which does not allow for isotropic delineation of ventricular structures. Thus, development of an isotropic 3D cine acquisition and automatic segmentation method is worthwhile to make CMR workflow straightforward and efficient, as the present work aims to establish.

Methods

Ninety-nine patients with many types of CHD were imaged using a non-angulated 3D cine CMR sequence covering the whole-heart and great vessels. Automatic supervised and semi-supervised deep-learning-based methods were developed for whole-heart segmentation of 3D cine images to separately delineate the cardiac structures, including both atria, both ventricles, aorta, pulmonary arteries, and superior and inferior vena cavae. The segmentation results derived from the two methods were compared with the manual segmentation in terms of Dice score, a degree of overlap agreement, and atrial and ventricular volume measurements.

Results

The semi-supervised method resulted in a better overlap agreement with the manual segmentation than the supervised method for all 8 structures (Dice score 83.23 ± 16.76% vs. 77.98 ± 19.64%; P-value ≤0.001). The mean difference error in atrial and ventricular volumetric measurements between manual segmentation and semi-supervised method was lower (bias ≤ 5.2 ml) than the supervised method (bias ≤ 10.1 ml).

Discussion

The proposed semi-supervised method is capable of cardiac segmentation and chamber volume quantification in a CHD population with wide anatomical variability. It accurately delineates the heart chambers and great vessels and can be used to accurately calculate ventricular and atrial volumes throughout the cardiac cycle. Such a segmentation method can reduce inter- and intra- observer variability and make CMR exams more standardized and efficient.

Semi-TMS: an efficient regularization-oriented triple-teacher semi-supervised medical image segmentation model

Objective. Although convolutional neural networks (CNN) and Transformers have performed well in many medical image segmentation tasks, they rely on large amounts of labeled data for training. The annotation of medical image data is expensive and time-consuming, so it is common to use semi-supervised learning methods that use a small amount of labeled data and a large amount of unlabeled data to improve the performance of medical imaging segmentation.Approach. This work aims to enhance the segmentation performance of medical images using a triple-teacher cross-learning semi-supervised medical image segmentation with shape perception and multi-scale consistency regularization. To effectively leverage the information from unlabeled data, we design a multi-scale semi-supervised method for three-teacher cross-learning based on shape perception, called Semi-TMS. The three teacher models engage in cross-learning with each other, where Teacher A and Teacher C utilize a CNN architecture, while Teacher B employs a transformer model. The cross-learning module consisting of Teacher A and Teacher C captures local and global information, generates pseudo-labels, and performs cross-learning using prediction results. Multi-scale consistency regularization is applied separately to the CNN and Transformer to improve accuracy. Furthermore, the low uncertainty output probabilities from Teacher A or Teacher C are utilized as input to Teacher B, enhancing the utilization of prior knowledge and overall segmentation robustness.Main results. Experimental evaluations on two public datasets demonstrate that the proposed method outperforms some existing semi-segmentation models, implicitly capturing shape information and effectively improving the utilization and accuracy of unlabeled data through multi-scale consistency.Significance. With the widespread utilization of medical imaging in clinical diagnosis, our method is expected to be a potential auxiliary tool, assisting clinicians and medical researchers in their diagnoses.

A Domain-Shift Invariant CNN Framework for Cardiac MRI Segmentation Across Unseen Domains

The emergence of various deep learning approaches in diagnostic medical image segmentation has made machines capable of accomplishing human-level accuracy. However, the generalizability of these architectures across patients from different countries, Magnetic Resonance Imaging (MRI) scans from distinct vendors, and varying imaging conditions remains questionable. In this work, we propose a translatable deep learning framework for diagnostic segmentation of cine MRI scans. This study aims to render the available SOTA (state-of-the-art) architectures domain-shift invariant by utilizing the heterogeneity of multi-sequence cardiac MRI. To develop and test our approach, we curated a diverse group of public datasets and a dataset obtained from private source. We evaluated 3 SOTA CNN (Convolution neural network) architectures i.e., U-Net, Attention-U-Net, and Attention-Res-U-Net. These architectures were first trained on a combination of three different cardiac MRI sequences. Next, we examined the M&M (multi-center & mutli-vendor) challenge dataset to investigate the effect of different training sets on translatability. The U-Net architecture, trained on the multi-sequence dataset, proved to be the most generalizable across multiple datasets during validation on unseen domains. This model attained mean dice scores of 0.81, 0.85, and 0.83 for myocardial wall segmentation after testing on unseen MyoPS (Myocardial Pathology Segmentation) 2020 dataset, AIIMS (All India Institute of Medical Sciences) dataset and M&M dataset, respectively. Our framework achieved Pearson's correlation values of 0.98, 0.99, and 0.95 between the observed and predicted parameters of end diastole volume, end systole volume, and ejection fraction, respectively, on the unseen Indian population dataset.

R2Net: Efficient and flexible diffeomorphic image registration using Lipschitz continuous residual networks

Classical diffeomorphic image registration methods, while being accurate, face the challenges of high computational costs. Deep learning based approaches provide a fast alternative to address these issues; however, most existing deep solutions either lose the good property of diffeomorphism or have limited flexibility to capture large deformations, under the assumption that deformations are driven by stationary velocity fields (SVFs). Also, the adopted squaring and scaling technique for integrating SVFs is time- and memory-consuming, hindering deep methods from handling large image volumes. In this paper, we present an unsupervised diffeomorphic image registration framework, which uses deep residual networks (ResNets) as numerical approximations of the underlying continuous diffeomorphic setting governed by ordinary differential equations, which is parameterized by either SVFs or time-varying (non-stationary) velocity fields. This flexible parameterization in our Residual Registration Network (R2Net) not only provides the model's ability to capture large deformation but also reduces the time and memory cost when integrating velocity fields for deformation generation. Also, we introduce a Lipschitz continuity constraint into the ResNet block to help achieve diffeomorphic deformations. To enhance the ability of our model for handling images with large volume sizes, we employ a hierarchical extension with a multi-phase learning strategy to solve the image registration task in a coarse-to-fine fashion. We demonstrate our models on four 3D image registration tasks with a wide range of anatomies, including brain MRIs, cine cardiac MRIs, and lung CT scans. Compared to classical methods SyN and diffeomorphic VoxelMorph, our models achieve comparable or better registration accuracy with much smoother deformations. Our source code is available online at https://github.com/ankitajoshi15/R2Net.

Unsupervised deep consistency learning adaptation network for cardiac cross-modality structural segmentation

Deep neural networks have recently been succeessful in the field of medical image segmentation; however, they are typically subject to performance degradation problems when well-trained models are tested in another new domain with different data distributions. Given that annotated cross-domain images may inaccessible, unsupervised domain adaptation methods that transfer learnable information from annotated source domains to unannotated target domains with different distributions have attracted substantial attention. Many methods leverage image-level or pixel-level translation networks to align domain-invariant information and mitigate domain shift issues. However, These methods rarely perform well when there is a large domain gap. A new unsupervised deep consistency learning adaptation network, which adopts input space consistency learning and output space consistency learning to realize unsupervised domain adaptation and cardiac structural segmentation, is introduced in this paper The framework mainly includes a domain translation path and a cross-modality segmentation path. In domain translation path, a symmetric alignment generator network with attention to cross-modality features and anatomy is introduced to align bidirectional domain features. In the segmentation path, entropy map minimization, output probability map minimization and segmentation prediction minimization are leveraged to align the output space features. The model conducts supervised learning to extract source domain features and conducts unsupervised deep consistency learning to extract target domain features. Through experimental testing on two challenging cross-modality segmentation tasks, our method has robust performance compared to that of previous methods. Furthermore, ablation experiments are conducted to confirm the effectiveness of our framework.

Min-Max Similarity: A Contrastive Semi-Supervised Deep Learning Network for Surgical Tools Segmentation

A common problem with segmentation of medical images using neural networks is the difficulty to obtain a significant number of pixel-level annotated data for training. To address this issue, we proposed a semi-supervised segmentation network based on contrastive learning. In contrast to the previous state-of-the-art, we introduce Min-Max Similarity (MMS), a contrastive learning form of dual-view training by employing classifiers and projectors to build all-negative, and positive and negative feature pairs, respectively, to formulate the learning as solving a MMS problem. The all-negative pairs are used to supervise the networks learning from different views and to capture general features, and the consistency of unlabeled predictions is measured by pixel-wise contrastive loss between positive and negative pairs. To quantitatively and qualitatively evaluate our proposed method, we test it on four public endoscopy surgical tool segmentation datasets and one cochlear implant surgery dataset, which we manually annotated. Results indicate that our proposed method consistently outperforms state-of-the-art semi-supervised and fully supervised segmentation algorithms. And our semi-supervised segmentation algorithm can successfully recognize unknown surgical tools and provide good predictions. Also, our MMS approach could achieve inference speeds of about 40 frames per second (fps) and is suitable to deal with the real-time video segmentation.

BayeSeg: Bayesian modeling for medical image segmentation with interpretable generalizability

Due to the cross-domain distribution shift aroused from diverse medical imaging systems, many deep learning segmentation methods fail to perform well on unseen data, which limits their real-world applicability. Recent works have shown the benefits of extracting domain-invariant representations on domain generalization. However, the interpretability of domain-invariant features remains a great challenge. To address this problem, we propose an interpretable Bayesian framework (BayeSeg) through Bayesian modeling of image and label statistics to enhance model generalizability for medical image segmentation. Specifically, we first decompose an image into a spatial-correlated variable and a spatial-variant variable, assigning hierarchical Bayesian priors to explicitly force them to model the domain-stable shape and domain-specific appearance information respectively. Then, we model the segmentation as a locally smooth variable only related to the shape. Finally, we develop a variational Bayesian framework to infer the posterior distributions of these explainable variables. The framework is implemented with neural networks, and thus is referred to as deep Bayesian segmentation. Quantitative and qualitative experimental results on prostate segmentation and cardiac segmentation tasks have shown the effectiveness of our proposed method. Moreover, we investigated the interpretability of BayeSeg by explaining the posteriors and analyzed certain factors that affect the generalization ability through further ablation studies. Our code is released via https://zmiclab.github.io/projects.html.

HDL: Hybrid Deep Learning for the Synthesis of Myocardial Velocity Maps in Digital Twins for Cardiac Analysis

Synthetic digital twins based on medical data accelerate the acquisition, labelling and decision making procedure in digital healthcare. A core part of digital healthcare twins is model-based data synthesis, which permits the generation of realistic medical signals without requiring to cope with the modelling complexity of anatomical and biochemical phenomena producing them in reality. Unfortunately, algorithms for cardiac data synthesis have been so far scarcely studied in the literature. An important imaging modality in the cardiac examination is three-directional CINE multi-slice myocardial velocity mapping (3Dir MVM), which provides a quantitative assessment of cardiac motion in three orthogonal directions of the left ventricle. The long acquisition time and complex acquisition produce make it more urgent to produce synthetic digital twins of this imaging modality. In this study, we propose a hybrid deep learning (HDL) network, especially for synthetic 3Dir MVM data. Our algorithm is featured by a hybrid UNet and a Generative Adversarial Network with a foreground-background generation scheme. The experimental results show that from temporally down-sampled magnitude CINE images (six times), our proposed algorithm can still successfully synthesise high temporal resolution 3Dir MVM CMR data (PSNR=42.32) with precise left ventricle segmentation (DICE=0.92). These performance scores indicate that our proposed HDL algorithm can be implemented in real-world digital twins for myocardial velocity mapping data simulation. To the best of our knowledge, this work is the first one investigating digital twins of the 3Dir MVM CMR, which has shown great potential for improving the efficiency of clinical studies via synthesised cardiac data.

Attractive deep morphology-aware active contour network for vertebral body contour extraction with extensions to heterogeneous and semi-supervised scenarios

Automatic vertebral body contour extraction (AVBCE) from heterogeneous spinal MRI is indispensable for the comprehensive diagnosis and treatment of spinal diseases. However, AVBCE is challenging due to data heterogeneity, image characteristics complexity, and vertebral body morphology variations, which may cause morphology errors in semantic segmentation. Deep active contour-based (deep ACM-based) methods provide a promising complement for tackling morphology errors by directly parameterizing the contour coordinates. Extending the target contours' capture range and providing morphology-aware parameter maps are crucial for deep ACM-based methods. For this purpose, we propose a novel Attractive Deep Morphology-aware actIve contouR nEtwork (ADMIRE) that embeds an elaborated contour attraction term (CAT) and a comprehensive contour quality (CCQ) loss into the deep ACM-based framework. The CAT adaptively extends the target contours' capture range by designing an all-to-all force field to enable the target contours' energy to contribute to farther locations. Furthermore, the CCQ loss is carefully designed to generate morphology-aware active contour parameters by simultaneously supervising the contour shape, tension, and smoothness. These designs, in cooperation with the deep ACM-based framework, enable robustness to data heterogeneity, image characteristics complexity, and target contour morphology variations. Furthermore, the deep ACM-based ADMIRE is able to cooperate well with semi-supervised strategies such as mean teacher, which enables its function in semi-supervised scenarios. ADMIRE is trained and evaluated on four challenging datasets, including three spinal datasets with more than 1000 heterogeneous images and more than 10000 vertebrae bodies, as well as a cardiac dataset with both normal and pathological cases. Results show ADMIRE achieves state-of-the-art performance on all datasets, which proves ADMIRE's accuracy, robustness, and generalization ability.

An artificial intelligence tool for automated analysis of large-scale unstructured clinical cine cardiac magnetic resonance databases

Aims

Artificial intelligence (AI) techniques have been proposed for automating analysis of short-axis (SAX) cine cardiac magnetic resonance (CMR), but no CMR analysis tool exists to automatically analyse large (unstructured) clinical CMR datasets. We develop and validate a robust AI tool for start-to-end automatic quantification of cardiac function from SAX cine CMR in large clinical databases.

Methods and results

Our pipeline for processing and analysing CMR databases includes automated steps to identify the correct data, robust image pre-processing, an AI algorithm for biventricular segmentation of SAX CMR and estimation of functional biomarkers, and automated post-analysis quality control to detect and correct errors. The segmentation algorithm was trained on 2793 CMR scans from two NHS hospitals and validated on additional cases from this dataset (n = 414) and five external datasets (n = 6888), including scans of patients with a range of diseases acquired at 12 different centres using CMR scanners from all major vendors. Median absolute errors in cardiac biomarkers were within the range of inter-observer variability: <8.4 mL (left ventricle volume), <9.2 mL (right ventricle volume), <13.3 g (left ventricular mass), and <5.9% (ejection fraction) across all datasets. Stratification of cases according to phenotypes of cardiac disease and scanner vendors showed good performance across all groups.

Conclusion

We show that our proposed tool, which combines image pre-processing steps, a domain-generalizable AI algorithm trained on a large-scale multi-domain CMR dataset and quality control steps, allows robust analysis of (clinical or research) databases from multiple centres, vendors, and cardiac diseases. This enables translation of our tool for use in fully automated processing of large multi-centre databases.

Emerging Roles for Artificial Intelligence in Heart Failure Imaging

Artificial intelligence (AI) applications are expanding in cardiac imaging. AI research has shown promise in workflow optimization, disease diagnosis, and integration of clinical and imaging data to predict patient outcomes. The diagnostic and prognostic paradigm of heart failure is heavily reliant on cardiac imaging. As AI becomes increasingly validated and integrated into clinical practice, AI influence on heart failure management will grow. This review discusses areas of current research and potential clinical applications in AI as applied to heart failure cardiac imaging.

ACTION++: Improving Semi-supervised Medical Image Segmentation with Adaptive Anatomical Contrast

Medical data often exhibits long-tail distributions with heavy class imbalance, which naturally leads to difficulty in classifying the minority classes (i.e., boundary regions or rare objects). Recent work has significantly improved semi-supervised medical image segmentation in long-tailed scenarios by equipping them with unsupervised contrastive criteria. However, it remains unclear how well they will perform in the labeled portion of data where class distribution is also highly imbalanced. In this work, we present ACTION++, an improved contrastive learning framework with adaptive anatomical contrast for semi-supervised medical segmentation. Specifically, we propose an adaptive supervised contrastive loss, where we first compute the optimal locations of class centers uniformly distributed on the embedding space (i.e., off-line), and then perform online contrastive matching training by encouraging different class features to adaptively match these distinct and uniformly distributed class centers. Moreover, we argue that blindly adopting a constant temperature τ in the contrastive loss on long-tailed medical data is not optimal, and propose to use a dynamic τ via a simple cosine schedule to yield better separation between majority and minority classes. Empirically, we evaluate ACTION++ on ACDC and LA benchmarks and show that it achieves state-of-the-art across two semi-supervised settings. Theoretically, we analyze the performance of adaptive anatomical contrast and confirm its superiority in label efficiency.

Fully Automatic initialization and segmentation of left and right ventricles for large-scale cardiac MRI using a deeply supervised network and 3D-ASM

BACKGROUND AND OBJECTIVE

The sheer volume of data generated by population imaging studies is unparalleled by current capabilities to extract objective and quantitative cardiac phenotypes; subjective and time-consuming manual image analysis remains the gold standard. Automated image analytics to compute quantitative imaging biomarkers of cardiac function are desperately needed. Data volumes and their variability pose a challenge to most state-of-the-art methods for endo- and epicardial contours, which lack robustness when applied to very large datasets. Our aim is to develop an analysis pipeline for the automatic quantification of cardiac function from cine magnetic resonance imaging data.

METHOD

This work adopt 4,638 cardiac MRI cases coming from UK Biobank with ground truth available for left and RV contours. A hybrid and robust algorithm is proposed to improve the accuracy of automatic left and right ventricle segmentation by harnessing the localization accuracy of deep learning and the morphological accuracy of 3D-ASM (three-dimensional active shape models). The contributions of this paper are three-fold. First, a fully automatic method is proposed for left and right ventricle initialization and cardiac MRI segmentation by taking full advantage of spatiotemporal constraint. Second, a deeply supervised network is introduced to train and segment the heart. Third, the 3D-ASM image search procedure is improved by combining image intensity models with convolutional neural network (CNN) derived distance maps improving endo- and epicardial edge localization.

RESULTS

The proposed architecture outperformed the state of the art for cardiac MRI segmentation from UK Biobank. The statistics of RV landmarks detection errors for Triscuspid valve and RV apex are 4.17 mm and 5.58 mm separately. The overlap metric, mean contour distance, Hausdorff distance and cardiac functional parameters are calculated for the LV (Left Ventricle) and RV (Right Ventricle) contour segmentation. Bland-Altman analysis for clinical parameters shows that the results from our automated image analysis pipelines are in good agreement with results from expert manual analysis.

CONCLUSIONS

Our hybrid scheme combines deep learning and statistical shape modeling for automatic segmentation of the LV/RV from cardiac MRI datasets is effective and robust and can compute cardiac functional indexes from population imaging.

Probabilistic Modeling for Image Registration Using Radial Basis Functions: Application to Cardiac Motion Estimation

Cardiovascular diseases (CVDs) are the leading cause of death, affecting the cardiac dynamics over the cardiac cycle. Estimation of cardiac motion plays an essential role in many medical clinical tasks. This article proposes a probabilistic framework for image registration using compact support radial basis functions (CSRBFs) to estimate cardiac motion. A variational inference-based generative model with convolutional neural networks (CNNs) is proposed to learn the probabilistic coefficients of CSRBFs used in image deformation. We designed two networks to estimate the deformation coefficients of CSRBFs: the first one solves the spatial transformation using given control points, and the second one models the transformation using drifting control points. The given-point-based network estimates the probabilistic coefficients of control points. In contrast, the drifting-point-based model predicts the probabilistic coefficients and spatial distribution of control points simultaneously. To regularize these coefficients, we derive the bending energy (BE) in the variational bound by defining the covariance of coefficients. The proposed framework has been evaluated on the cardiac motion estimation and the calculation of the myocardial strain. In the experiments, 1409 slice pairs of end-diastolic (ED) and end-systolic (ES) phase in 4-D cardiac magnetic resonance (MR) images selected from three public datasets are employed to evaluate our networks. The experimental results show that our framework outperforms the state-of-the-art registration methods concerning the deformation smoothness and registration accuracy.

Deep Learning Based Parameterization of Diffeomorphic Image Registration for Cardiac Image Segmentation

Cardiac segmentation from magnetic resonance imaging (MRI) is one of the essential tasks in analyzing the anatomy and function of the heart for the assessment and diagnosis of cardiac diseases. However, cardiac MRI generates hundreds of images per scan, and manual annotation of them is challenging and time-consuming, and therefore processing these images automatically is of interest. This study proposes a novel end-to-end supervised cardiac MRI segmentation framework based on a diffeomorphic deformable registration that can segment cardiac chambers from 2D and 3D images or volumes. To represent actual cardiac deformation, the method parameterizes the transformation using radial and rotational components computed via deep learning, with a set of paired images and segmentation masks used for training. The formulation guarantees transformations that are invertible and prevents mesh folding, which is essential for preserving the topology of the segmentation results. A physically plausible transformation is achieved by employing diffeomorphism in computing the transformations and activation functions that constrain the range of the radial and rotational components. The method was evaluated over three different data sets and showed significant improvements compared to exacting learning and non-learning based methods in terms of the Dice score and Hausdorff distance metrics.

Selecting the best optimizers for deep learning-based medical image segmentation

Purpose

The goal of this work is to explore the best optimizers for deep learning in the context of medical image segmentation and to provide guidance on how to design segmentation networks with effective optimization strategies.

Approach

Most successful deep learning networks are trained using two types of stochastic gradient descent (SGD) algorithms: adaptive learning and accelerated schemes. Adaptive learning helps with fast convergence by starting with a larger learning rate (LR) and gradually decreasing it. Momentum optimizers are particularly effective at quickly optimizing neural networks within the accelerated schemes category. By revealing the potential interplay between these two types of algorithms [LR and momentum optimizers or momentum rate (MR) in short], in this article, we explore the two variants of SGD algorithms in a single setting. We suggest using cyclic learning as the base optimizer and integrating optimal values of learning rate and momentum rate. The new optimization function proposed in this work is based on the Nesterov accelerated gradient optimizer, which is more efficient computationally and has better generalization capabilities compared to other adaptive optimizers.

Results

We investigated the relationship of LR and MR under an important problem of medical image segmentation of cardiac structures from MRI and CT scans. We conducted experiments using the cardiac imaging dataset from the ACDC challenge of MICCAI 2017, and four different architectures were shown to be successful for cardiac image segmentation problems. Our comprehensive evaluations demonstrated that the proposed optimizer achieved better results (over a 2% improvement in the dice metric) than other optimizers in the deep learning literature with similar or lower computational cost in both single and multi-object segmentation settings.

Conclusions

We hypothesized that the combination of accelerated and adaptive optimization methods can have a drastic effect in medical image segmentation performances. To this end, we proposed a new cyclic optimization method (Cyclic Learning/Momentum Rate) to address the efficiency and accuracy problems in deep learning-based medical image segmentation. The proposed strategy yielded better generalization in comparison to adaptive optimizers.

Quality control-driven deep ensemble for accountable automated segmentation of cardiac magnetic resonance LGE and VNE images

Background

Late gadolinium enhancement (LGE) cardiovascular magnetic resonance (CMR) imaging is the gold standard for non-invasive myocardial tissue characterisation. However, accurate segmentation of the left ventricular (LV) myocardium remains a challenge due to limited training data and lack of quality control. This study addresses these issues by leveraging generative adversarial networks (GAN)-generated virtual native enhancement (VNE) images to expand the training set and incorporating an automated quality control-driven (QCD) framework to improve segmentation reliability.

Methods

A dataset comprising 4,716 LGE images (from 1,363 patients with hypertrophic cardiomyopathy and myocardial infarction) was used for development. To generate additional clinically validated data, LGE data were augmented with a GAN-based generator to produce VNE images. LV was contoured on these images manually by clinical observers. To create diverse candidate segmentations, the QCD framework involved multiple U-Nets, which were combined using statistical rank filters. The framework predicted the Dice Similarity Coefficient (DSC) for each candidate segmentation, with the highest predicted DSC indicating the most accurate and reliable result. The performance of the QCD ensemble framework was evaluated on both LGE and VNE test datasets (309 LGE/VNE images from 103 patients), assessing segmentation accuracy (DSC) and quality prediction (mean absolute error (MAE) and binary classification accuracy).

Results

The QCD framework effectively and rapidly segmented the LV myocardium (<1 s per image) on both LGE and VNE images, demonstrating robust performance on both test datasets with similar mean DSC (LGE: 0.845 ± 0.075 ; VNE: 0.845 ± 0.071 ; p = n s ). Incorporating GAN-generated VNE data into the training process consistently led to enhanced performance for both individual models and the overall framework. The quality control mechanism yielded a high performance (MAE = 0.043 , accuracy = 0.951 ) emphasising the accuracy of the quality control-driven strategy in predicting segmentation quality in clinical settings. Overall, no statistical difference (p = n s ) was found when comparing the LGE and VNE test sets across all experiments.

Conclusions

The QCD ensemble framework, leveraging GAN-generated VNE data and an automated quality control mechanism, significantly improved the accuracy and reliability of LGE segmentation, paving the way for enhanced and accountable diagnostic imaging in routine clinical use.

Multi-task contrastive learning for semi-supervised medical image segmentation with multi-scale uncertainty estimation

Objective. Automated medical image segmentation is vital for the prevention and treatment of disease. However, medical data commonly exhibit class imbalance in practical applications, which may lead to unclear boundaries of specific classes and make it difficult to effectively segment certain tail classes in the results of semi-supervised medical image segmentation.Approach. We propose a novel multi-task contrastive learning framework for semi-supervised medical image segmentation with multi-scale uncertainty estimation. Specifically, the framework includes a student-teacher model. We introduce global image-level contrastive learning in the encoder to address the class imbalance and local pixel-level contrastive learning in the decoder to achieve intra-class aggregation and inter-class separation. Furthermore, we propose a multi-scale uncertainty-aware consistency loss to reduce noise caused by pseudo-label bias.Main results. Experiments on three public datasets ACDC, LA and LiTs show that our method achieves higher segmentation performance compared with state-of-the-art semi-supervised segmentation methods.Significance. The multi-task contrastive learning in our method facilitates the negative impact of class imbalance and achieves better classification results. The multi-scale uncertainty estimation encourages consistent predictions for the same input under different perturbations, motivating the teacher model to generate high-quality pseudo-labels. Code is available athttps://github.com/msctransu/MCSSMU.git.

H2Former: An Efficient Hierarchical Hybrid Transformer for Medical Image Segmentation

Accurate medical image segmentation is of great significance for computer aided diagnosis. Although methods based on convolutional neural networks (CNNs) have achieved good results, it is weak to model the long-range dependencies, which is very important for segmentation task to build global context dependencies. The Transformers can establish long-range dependencies among pixels by self-attention, providing a supplement to the local convolution. In addition, multi-scale feature fusion and feature selection are crucial for medical image segmentation tasks, which is ignored by Transformers. However, it is challenging to directly apply self-attention to CNNs due to the quadratic computational complexity for high-resolution feature maps. Therefore, to integrate the merits of CNNs, multi-scale channel attention and Transformers, we propose an efficient hierarchical hybrid vision Transformer (H2Former) for medical image segmentation. With these merits, the model can be data-efficient for limited medical data regime. The experimental results show that our approach exceeds previous Transformer, CNNs and hybrid methods on three 2D and two 3D medical image segmentation tasks. Moreover, it keeps computational efficiency in model parameters, FLOPs and inference time. For example, H2Former outperforms TransUNet by 2.29% in IoU score on KVASIR-SEG dataset with 30.77% parameters and 59.23% FLOPs.

A global-frequency-domain network for medical image segmentation

The UNet series networks have been a leader in the field of medical image segmentation since their introduction. However, encoder and decoder structures of the traditional UNet series network are complex, with a large number of parameters and floating-point operations. This requires a large amount of data as support for model training, but most medical datasets only contain limited numbers of samples. To address this issue, we propose a global frequency domain UNet (GFUNet), a novel architecture for fast medical image segmentation. Inspired by recent modified Multi-Layer Perceptron(MLP)-like models, we combine Fourier Transform with UNet structure to achieve more efficient and effective encoding and decoding processes. Meanwhile, A dual-domain encoding module is designed to improve the performance of the encoder and decoder by fully used frequency domain feature. Furthermore, due to the excellent property of the Fourier Transform and its optimization, our network greatly reduces the number of parameters compared to other UNets. We evaluate GFUNet on several medical segmentation tasks, achieving improved segmentation performance compared to state-of-the-art network architectures for medical image segmentation. Compared to the original UNet, the results show that we reduce the number of parameters by 46 times, reduce computational complexity by 114 times, and improved the considerable dice score.

Reconstruction and completion of high-resolution 3D cardiac shapes using anisotropic CMRI segmentations and continuous implicit neural representations

Since the onset of computer-aided diagnosis in medical imaging, voxel-based segmentation has emerged as the primary methodology for automatic analysis of left ventricle (LV) function and morphology in cardiac magnetic resonance images (CMRI). In standard clinical practice, simultaneous multi-slice 2D cine short-axis MR imaging is performed under multiple breath-holds resulting in highly anisotropic 3D images. Furthermore, sparse-view CMRI often lacks whole heart coverage caused by large slice thickness and often suffers from inter-slice misalignment induced by respiratory motion. Therefore, these volumes only provide limited information about the true 3D cardiac anatomy which may hamper highly accurate assessment of functional and anatomical abnormalities. To address this, we propose a method that learns a continuous implicit function representing 3D LV shapes by training an auto-decoder. For training, high-resolution segmentations from cardiac CT angiography are used. The ability of our approach to reconstruct and complete high-resolution shapes from manually or automatically obtained sparse-view cardiac shape information is evaluated by using paired high- and low-resolution CMRI LV segmentations. The results show that the reconstructed LV shapes have an unconstrained subvoxel resolution and appear smooth and plausible in through-plane direction. Furthermore, Bland-Altman analysis reveals that reconstructed high-resolution ventricle volumes are closer to the corresponding reference volumes than reference low-resolution volumes with bias of [limits of agreement] -3.51 [-18.87, 11.85] mL, and 12.96 [-10.01, 35.92] mL respectively. Finally, the results demonstrate that the proposed approach allows recovering missing shape information and can indirectly correct for limited motion-induced artifacts.

Siamese pyramidal deep learning network for strain estimation in 3D cardiac cine-MR

Strain represents the quantification of regional tissue deformation within a given area. Myocardial strain has demonstrated considerable utility as an indicator for the assessment of cardiac function. Notably, it exhibits greater sensitivity in detecting subtle myocardial abnormalities compared to conventional cardiac function indices, like left ventricle ejection fraction (LVEF). Nonetheless, the estimation of strain poses considerable challenges due to the necessity for precise tracking of myocardial motion throughout the complete cardiac cycle. This study introduces a novel deep learning-based pipeline, designed to automatically and accurately estimate myocardial strain from three-dimensional (3D) cine-MR images. Consequently, our investigation presents a comprehensive pipeline for the precise quantification of local and global myocardial strain. This pipeline incorporates a supervised Convolutional Neural Network (CNN) for accurate segmentation of the cardiac muscle and an unsupervised CNN for robust left ventricle motion tracking, enabling the estimation of strain in both artificial phantoms and real cine-MR images. Our investigation involved a comprehensive comparison of our findings with those obtained from two commonly utilized commercial software in this field. This analysis encompassed the examination of both intra- and inter-user variability. The proposed pipeline exhibited demonstrable reliability and reduced divergence levels when compared to alternative systems. Additionally, our approach is entirely independent of previous user data, effectively eliminating any potential user bias that could influence the strain analyses.

Recent progress in transformer-based medical image analysis

The transformer is primarily used in the field of natural language processing. Recently, it has been adopted and shows promise in the computer vision (CV) field. Medical image analysis (MIA), as a critical branch of CV, also greatly benefits from this state-of-the-art technique. In this review, we first recap the core component of the transformer, the attention mechanism, and the detailed structures of the transformer. After that, we depict the recent progress of the transformer in the field of MIA. We organize the applications in a sequence of different tasks, including classification, segmentation, captioning, registration, detection, enhancement, localization, and synthesis. The mainstream classification and segmentation tasks are further divided into eleven medical image modalities. A large number of experiments studied in this review illustrate that the transformer-based method outperforms existing methods through comparisons with multiple evaluation metrics. Finally, we discuss the open challenges and future opportunities in this field. This task-modality review with the latest contents, detailed information, and comprehensive comparison may greatly benefit the broad MIA community.

Cardiac Magnetic Resonance Imaging (CMRI) Applications in Patients with Chest Pain in the Emergency Department: A Narrative Review

CMRI is the exclusive imaging technique capable of identifying myocardial edema, endomyocardial fibrosis, pericarditis accompanied by pericardial effusions, and apical thrombi within either the left or right ventricle. In this work, we examine the research literature on the use of CMRI in the diagnosis of chest discomfort, employing randomized controlled trials (RCTs) to evaluate its effectiveness. The research outlines the disorders of the chest and the machine learning approaches for detecting them. In conclusion, the study ends with an examination of a fundamental illustration of CMRI analysis. To find a comprehensive review, the Scopus scientific resource is analyzed. The issue, based on the findings, is to distinguish ischemia from non-ischemic cardiac causes of chest pain in individuals presenting with sudden chest pain or discomfort upon arrival at the emergency department (ED). Due to the failure of conventional methods in accurately diagnosing acute cardiac ischemia, individuals are still being inappropriately discharged from the ED, resulting in a heightened death rate.

Three-dimensional printing in modelling mitral valve interventions

Mitral interventions remain technically challenging owing to the anatomical complexity and heterogeneity of mitral pathologies. As such, multi-disciplinary pre-procedural planning assisted by advanced cardiac imaging is pivotal to successful outcomes. Modern imaging techniques offer accurate 3D renderings of cardiac anatomy; however, users are required to derive a spatial understanding of complex mitral pathologies from a 2D projection thus generating an 'imaging gap' which limits procedural planning. Physical mitral modelling using 3D printing has the potential to bridge this gap and is increasingly being employed in conjunction with other transformative technologies to assess feasibility of intervention, direct prosthesis choice and avoid complications. Such platforms have also shown value in training and patient education. Despite important limitations, the pace of innovation and synergistic integration with other technologies is likely to ensure that 3D printing assumes a central role in the journey towards delivering personalised care for patients undergoing mitral valve interventions.

Machine learning in cardiology: Clinical application and basic research

Machine learning is a subfield of artificial intelligence. The quality and versatility of machine learning have been rapidly improving and playing a critical role in many aspects of social life. This trend is also observed in the medical field. Generally, there are three main types of machine learning: supervised, unsupervised, and reinforcement learning. Each type of learning is adequately selected for the purpose and type of data. In the field of medicine, various types of information are collected and used, and research using machine learning is becoming increasingly relevant. Many clinical studies are conducted using electronic health and medical records, including in the cardiovascular area. Machine learning has also been applied in basic research. Machine learning has been widely used for several types of data analysis, such as clustering of microarray analysis and RNA sequence analysis. Machine learning is essential for genome and multi-omics analyses. This review summarizes the recent advancements in the use of machine learning in clinical applications and basic cardiovascular research.

Coarse-Refined Consistency Learning Using Pixel-Level Features for Semi-Supervised Medical Image Segmentation

Pixel-level annotations are extremely expensive for medical image segmentation tasks as both expertise and time are needed to generate accurate annotations. Semi-supervised learning (SSL) for medical image segmentation has recently attracted growing attention because it can alleviate the exhausting manual annotations for clinicians by leveraging unlabeled data. However, most of the existing SSL methods do not take pixel-level information (e.g., pixel-level features) of labeled data into account, i.e., the labeled data are underutilized. Hence, in this work, an innovative Coarse-Refined Network with pixel-wise Intra-patch ranked loss and patch-wise Inter-patch ranked loss (CRII-Net) is proposed. It provides three advantages: i) it can produce stable targets for unlabeled data, as a simple yet effective coarse-refined consistency constraint is designed; ii) it is very effective for the extreme case where very scarce labeled data are available, as the pixel-level and patch-level features are extracted by our CRII-Net; and iii) it can output fine-grained segmentation results for hard regions (e.g., blurred object boundaries and low-contrast lesions), as the proposed Intra-Patch Ranked Loss (Intra-PRL) focuses on object boundaries and Inter-Patch Ranked loss (Inter-PRL) mitigates the adverse impact of low-contrast lesions. Experimental results on two common SSL tasks for medical image segmentation demonstrate the superiority of our CRII-Net. Specifically, when there are only 4% labeled data, our CRII-Net improves the Dice similarity coefficient (DSC) score by at least 7.49% when compared to five classical or state-of-the-art (SOTA) SSL methods. For hard samples/regions, our CRII-Net also significantly outperforms other compared methods in both quantitative and visualization results.

SequenceMorph: A Unified Unsupervised Learning Framework for Motion Tracking on Cardiac Image Sequences

Modern medical imaging techniques, such as ultrasound (US) and cardiac magnetic resonance (MR) imaging, have enabled the evaluation of myocardial deformation directly from an image sequence. While many traditional cardiac motion tracking methods have been developed for the automated estimation of the myocardial wall deformation, they are not widely used in clinical diagnosis, due to their lack of accuracy and efficiency. In this paper, we propose a novel deep learning-based fully unsupervised method, SequenceMorph, for in vivo motion tracking in cardiac image sequences. In our method, we introduce the concept of motion decomposition and recomposition. We first estimate the inter-frame (INF) motion field between any two consecutive frames, by a bi-directional generative diffeomorphic registration neural network. Using this result, we then estimate the Lagrangian motion field between the reference frame and any other frame, through a differentiable composition layer. Our framework can be extended to incorporate another registration network, to further reduce the accumulated errors introduced in the INF motion tracking step, and to refine the Lagrangian motion estimation. By utilizing temporal information to perform reasonable estimations of spatio-temporal motion fields, this novel method provides a useful solution for image sequence motion tracking. Our method has been applied to US (echocardiographic) and cardiac MR (untagged and tagged cine) image sequences; the results show that SequenceMorph is significantly superior to conventional motion tracking methods, in terms of the cardiac motion tracking accuracy and inference efficiency.

Diffusion models in medical imaging: A comprehensive survey

Denoising diffusion models, a class of generative models, have garnered immense interest lately in various deep-learning problems. A diffusion probabilistic model defines a forward diffusion stage where the input data is gradually perturbed over several steps by adding Gaussian noise and then learns to reverse the diffusion process to retrieve the desired noise-free data from noisy data samples. Diffusion models are widely appreciated for their strong mode coverage and quality of the generated samples in spite of their known computational burdens. Capitalizing on the advances in computer vision, the field of medical imaging has also observed a growing interest in diffusion models. With the aim of helping the researcher navigate this profusion, this survey intends to provide a comprehensive overview of diffusion models in the discipline of medical imaging. Specifically, we start with an introduction to the solid theoretical foundation and fundamental concepts behind diffusion models and the three generic diffusion modeling frameworks, namely, diffusion probabilistic models, noise-conditioned score networks, and stochastic differential equations. Then, we provide a systematic taxonomy of diffusion models in the medical domain and propose a multi-perspective categorization based on their application, imaging modality, organ of interest, and algorithms. To this end, we cover extensive applications of diffusion models in the medical domain, including image-to-image translation, reconstruction, registration, classification, segmentation, denoising, 2/3D generation, anomaly detection, and other medically-related challenges. Furthermore, we emphasize the practical use case of some selected approaches, and then we discuss the limitations of the diffusion models in the medical domain and propose several directions to fulfill the demands of this field. Finally, we gather the overviewed studies with their available open-source implementations at our GitHub.1 We aim to update the relevant latest papers within it regularly.

CAT: Constrained Adversarial Training for Anatomically-Plausible Semi-Supervised Segmentation

Deep learning models for semi-supervised medical image segmentation have achieved unprecedented performance for a wide range of tasks. Despite their high accuracy, these models may however yield predictions that are considered anatomically impossible by clinicians. Moreover, incorporating complex anatomical constraints into standard deep learning frameworks remains challenging due to their non-differentiable nature. To address these limitations, we propose a Constrained Adversarial Training (CAT) method that learns how to produce anatomically plausible segmentations. Unlike approaches focusing solely on accuracy measures like Dice, our method considers complex anatomical constraints like connectivity, convexity, and symmetry which cannot be easily modeled in a loss function. The problem of non-differentiable constraints is solved using a Reinforce algorithm which enables to obtain a gradient for violated constraints. To generate constraint-violating examples on the fly, and thereby obtain useful gradients, our method adopts an adversarial training strategy which modifies training images to maximize the constraint loss, and then updates the network to be robust to these adversarial examples. The proposed method offers a generic and efficient way to add complex segmentation constraints on top of any segmentation network. Experiments on synthetic data and four clinically-relevant datasets demonstrate the effectiveness of our method in terms of segmentation accuracy and anatomical plausibility.

Transformers in medical imaging: A survey

Following unprecedented success on the natural language tasks, Transformers have been successfully applied to several computer vision problems, achieving state-of-the-art results and prompting researchers to reconsider the supremacy of convolutional neural networks (CNNs) as de facto operators. Capitalizing on these advances in computer vision, the medical imaging field has also witnessed growing interest for Transformers that can capture global context compared to CNNs with local receptive fields. Inspired from this transition, in this survey, we attempt to provide a comprehensive review of the applications of Transformers in medical imaging covering various aspects, ranging from recently proposed architectural designs to unsolved issues. Specifically, we survey the use of Transformers in medical image segmentation, detection, classification, restoration, synthesis, registration, clinical report generation, and other tasks. In particular, for each of these applications, we develop taxonomy, identify application-specific challenges as well as provide insights to solve them, and highlight recent trends. Further, we provide a critical discussion of the field's current state as a whole, including the identification of key challenges, open problems, and outlining promising future directions. We hope this survey will ignite further interest in the community and provide researchers with an up-to-date reference regarding applications of Transformer models in medical imaging. Finally, to cope with the rapid development in this field, we intend to regularly update the relevant latest papers and their open-source implementations at https://github.com/fahadshamshad/awesome-transformers-in-medical-imaging.

Semi-Supervised Medical Image Segmentation with Co-Distribution Alignment

Medical image segmentation has made significant progress when a large amount of labeled data are available. However, annotating medical image segmentation datasets is expensive due to the requirement of professional skills. Additionally, classes are often unevenly distributed in medical images, which severely affects the classification performance on minority classes. To address these problems, this paper proposes Co-Distribution Alignment (Co-DA) for semi-supervised medical image segmentation. Specifically, Co-DA aligns marginal predictions on unlabeled data to marginal predictions on labeled data in a class-wise manner with two differently initialized models before using the pseudo-labels generated by one model to supervise the other. Besides, we design an over-expectation cross-entropy loss for filtering the unlabeled pixels to reduce noise in their pseudo-labels. Quantitative and qualitative experiments on three public datasets demonstrate that the proposed approach outperforms existing state-of-the-art semi-supervised medical image segmentation methods on both the 2D CaDIS dataset and the 3D LGE-MRI and ACDC datasets, achieving an mIoU of 0.8515 with only 24% labeled data on CaDIS, and a Dice score of 0.8824 and 0.8773 with only 20% data on LGE-MRI and ACDC, respectively.

Estimation of Left and Right Ventricular Ejection Fractions from cine-MRI Using 3D-CNN

Cardiac function indices must be calculated using tracing from short-axis images in cine-MRI. A 3D-CNN (convolutional neural network) that adds time series information to images can estimate cardiac function indices without tracing using images with known values and cardiac cycles as the input. Since the short-axis image depicts the left and right ventricles, it is unclear which motion feature is captured. This study aims to estimate the indices by learning the short-axis images and the known left and right ventricular ejection fractions and to confirm the accuracy and whether each index is captured as a feature. A total of 100 patients with publicly available short-axis cine images were used. The dataset was divided into training:test = 8:2, and a regression model was built by training with the 3D-ResNet50. Accuracy was assessed using a five-fold cross-validation. The correlation coefficient, MAE (mean absolute error), and RMSE (root mean squared error) were determined as indices of accuracy evaluation. The mean correlation coefficient of the left ventricular ejection fraction was 0.80, MAE was 9.41, and RMSE was 12.26. The mean correlation coefficient of the right ventricular ejection fraction was 0.56, MAE was 11.35, and RMSE was 14.95. The correlation coefficient was considerably higher for the left ventricular ejection fraction. Regression modeling using the 3D-CNN indicated that the left ventricular ejection fraction was estimated more accurately, and left ventricular systolic function was captured as a feature.

nnFormer: Volumetric Medical Image Segmentation via a 3D Transformer

Transformer, the model of choice for natural language processing, has drawn scant attention from the medical imaging community. Given the ability to exploit long-term dependencies, transformers are promising to help atypical convolutional neural networks to learn more contextualized visual representations. However, most of recently proposed transformer-based segmentation approaches simply treated transformers as assisted modules to help encode global context into convolutional representations. To address this issue, we introduce nnFormer (i.e., not-another transFormer), a 3D transformer for volumetric medical image segmentation. nnFormer not only exploits the combination of interleaved convolution and self-attention operations, but also introduces local and global volume-based self-attention mechanism to learn volume representations. Moreover, nnFormer proposes to use skip attention to replace the traditional concatenation/summation operations in skip connections in U-Net like architecture. Experiments show that nnFormer significantly outperforms previous transformer-based counterparts by large margins on three public datasets. Compared to nnUNet, the most widely recognized convnet-based 3D medical segmentation model, nnFormer produces significantly lower HD95 and is much more computationally efficient. Furthermore, we show that nnFormer and nnUNet are highly complementary to each other in model ensembling. Codes and models of nnFormer are available at https://git.io/JSf3i.

Generalizable Deep Learning Method for Suppressing Unseen and Multiple MRI Artifacts Using Meta-learning

Magnetic Resonance (MR) images suffer from various types of artifacts due to motion, spatial resolution, and under-sampling. Conventional deep learning methods deal with removing a specific type of artifact, leading to separately trained models for each artifact type that lack the shared knowledge generalizable across artifacts. Moreover, training a model for each type and amount of artifact is a tedious process that consumes more training time and storage of models. On the other hand, the shared knowledge learned by jointly training the model on multiple artifacts might be inadequate to generalize under deviations in the types and amounts of artifacts. Model-agnostic meta-learning (MAML), a nested bi-level optimization framework is a promising technique to learn common knowledge across artifacts in the outer level of optimization, and artifact-specific restoration in the inner level. We propose curriculum-MAML (CMAML), a learning process that integrates MAML with curriculum learning to impart the knowledge of variable artifact complexity to adaptively learn restoration of multiple artifacts during training. Comparative studies against Stochastic Gradient Descent and MAML, using two cardiac datasets reveal that CMAML exhibits (i) better generalization with improved PSNR for 83% of unseen types and amounts of artifacts and improved SSIM in all cases, and (ii) better artifact suppression in 4 out of 5 cases of composite artifacts (scans with multiple artifacts).Clinical relevance- Our results show that CMAML has the potential to minimize the number of artifact-specific models; which is essential to deploy deep learning models for clinical use. Furthermore, we have also taken another practical scenario of an image affected by multiple artifacts and show that our method performs better in 80% of cases.

Radiomics-Based Quality Control System for Automatic Cardiac Segmentation: A Feasibility Study

PURPOSE

In the past decade, there has been a rapid increase in the development of automatic cardiac segmentation methods. However, the automatic quality control (QC) of these segmentation methods has received less attention. This study aims to address this gap by developing an automatic pipeline that incorporates DL-based cardiac segmentation and radiomics-based quality control.

METHODS

In the DL-based localization and segmentation part, the entire heart was first located and cropped. Then, the cropped images were further utilized for the segmentation of the right ventricle cavity (RVC), myocardium (MYO), and left ventricle cavity (LVC). As for the radiomics-based QC part, a training radiomics dataset was created with segmentation tasks of various quality. This dataset was used for feature extraction, selection, and QC model development. The model performance was then evaluated using both internal and external testing datasets.

RESULTS

In the internal testing dataset, the segmentation model demonstrated a great performance with a dice similarity coefficient (DSC) of 0.954 for whole heart segmentations. Images were then appropriately cropped to 160 × 160 pixels. The models also performed well for cardiac substructure segmentations. The DSC values were 0.863, 0.872, and 0.940 for RVC, MYO, and LVC for 2D masks and 0.928, 0.886, and 0.962 for RVC, MYO, and LVC for 3D masks with an attention-UNet. After feature selection with the radiomics dataset, we developed a series of models to predict the automatic segmentation quality and its DSC value for the RVC, MYO, and LVC structures. The mean absolute values for our best prediction models were 0.060, 0.032, and 0.021 for 2D segmentations and 0.027, 0.017, and 0.011 for 3D segmentations, respectively. Additionally, the radiomics-based classification models demonstrated a high negative detection rate of >0.85 in all 2D groups. In the external dataset, models showed similar results.

CONCLUSIONS

We developed a pipeline including cardiac substructure segmentation and QC at both the slice (2D) and subject (3D) levels. Our results demonstrate that the radiomics method possesses great potential for the automatic QC of cardiac segmentation.

BATFormer: Towards Boundary-Aware Lightweight Transformer for Efficient Medical Image Segmentation

OBJECTIVE

Transformers, born to remedy the inadequate receptive fields of CNNs, have drawn explosive attention recently. However, the daunting computational complexity of global representation learning, together with rigid window partitioning, hinders their deployment in medical image segmentation. This work aims to address the above two issues in transformers for better medical image segmentation.

METHODS

We propose a boundary-aware lightweight transformer (BATFormer) that can build cross-scale global interaction with lower computational complexity and generate windows flexibly under the guidance of entropy. Specifically, to fully explore the benefits of transformers in long-range dependency establishment, a cross-scale global transformer (CGT) module is introduced to jointly utilize multiple small-scale feature maps for richer global features with lower computational complexity. Given the importance of shape modeling in medical image segmentation, a boundary-aware local transformer (BLT) module is constructed. Different from rigid window partitioning in vanilla transformers which would produce boundary distortion, BLT adopts an adaptive window partitioning scheme under the guidance of entropy for both computational complexity reduction and shape preservation.

RESULTS

BATFormer achieves the best performance in Dice of 92.84 %, 91.97 %, 90.26 %, and 96.30 % for the average, right ventricle, myocardium, and left ventricle respectively on the ACDC dataset and the best performance in Dice, IoU, and ACC of 90.76 %, 84.64 %, and 96.76 % respectively on the ISIC 2018 dataset. More importantly, BATFormer requires the least amount of model parameters and the lowest computational complexity compared to the state-of-the-art approaches.

CONCLUSION AND SIGNIFICANCE

Our results demonstrate the necessity of developing customized transformers for efficient and better medical image segmentation. We believe the design of BATFormer is inspiring and extendable to other applications/frameworks.

Deep Learning Segmentation of the Right Ventricle in Cardiac MRI: The M&Ms Challenge

In recent years, several deep learning models have been proposed to accurately quantify and diagnose cardiac pathologies. These automated tools heavily rely on the accurate segmentation of cardiac structures in MRI images. However, segmentation of the right ventricle is challenging due to its highly complex shape and ill-defined borders. Hence, there is a need for new methods to handle such structure's geometrical and textural complexities, notably in the presence of pathologies such as Dilated Right Ventricle, Tricuspid Regurgitation, Arrhythmogenesis, Tetralogy of Fallot, and Inter-atrial Communication. The last MICCAI challenge on right ventricle segmentation was held in 2012 and included only 48 cases from a single clinical center. As part of the 12th Workshop on Statistical Atlases and Computational Models of the Heart (STACOM 2021), the M&Ms-2 challenge was organized to promote the interest of the research community around right ventricle segmentation in multi-disease, multi-view, and multi-center cardiac MRI. Three hundred sixty CMR cases, including short-axis and long-axis 4-chamber views, were collected from three Spanish hospitals using nine different scanners from three different vendors, and included a diverse set of right and left ventricle pathologies. The solutions provided by the participants show that nnU-Net achieved the best results overall. However, multi-view approaches were able to capture additional information, highlighting the need to integrate multiple cardiac diseases, views, scanners, and acquisition protocols to produce reliable automatic cardiac segmentation algorithms.

3D Shape-Based Myocardial Infarction Prediction Using Point Cloud Classification Networks

Myocardial infarction (MI) is one of the most prevalent cardiovascular diseases with associated clinical decision-making typically based on single-valued imaging biomarkers. However, such metrics only approximate the complex 3D structure and physiology of the heart and hence hinder a better understanding and prediction of MI outcomes. In this work, we investigate the utility of complete 3D cardiac shapes in the form of point clouds for an improved detection of MI events. To this end, we propose a fully automatic multi-step pipeline consisting of a 3D cardiac surface reconstruction step followed by a point cloud classification network. Our method utilizes recent advances in geometric deep learning on point clouds to enable direct and efficient multi-scale learning on high-resolution surface models of the cardiac anatomy. We evaluate our approach on 1068 UK Biobank subjects for the tasks of prevalent MI detection and incident MI prediction and find improvements of ∼13% and ∼5% respectively over clinical benchmarks. Furthermore, we analyze the role of each ventricle and cardiac phase for 3D shape-based MI detection and conduct a visual analysis of the morphological and physiological patterns typically associated with MI outcomes.Clinical relevance- The presented approach enables the fast and fully automatic pathology-specific analysis of full 3D cardiac shapes. It can thus be employed as a real-time diagnostic tool in clinical practice to discover and visualize more intricate biomarkers than currently used single-valued metrics and improve predictive accuracy of myocardial infarction.

Deformable Image Registration Using Vision Transformers for Cardiac Motion Estimation from Cine Cardiac MRI Images

Accurate cardiac motion estimation is a crucial step in assessing the kinematic and contractile properties of the cardiac chambers, thereby directly quantifying the regional cardiac function, which plays an important role in understanding myocardial diseases and planning their treatment. Since the cine cardiac magnetic resonance imaging (MRI) provides dynamic, high-resolution 3D images of the heart that depict cardiac motion throughout the cardiac cycle, cardiac motion can be estimated by finding the optical flow representation between the consecutive 3D volumes from a 4D cine cardiac MRI dataset, thereby formulating it as an image registration problem. Therefore, we propose a hybrid convolutional neural network (CNN) and Vision Transformer (ViT) architecture for deformable image registration of 3D cine cardiac MRI images for consistent cardiac motion estimation. We compare the image registration results of our proposed method with those of the VoxelMorph CNN model and conventional B-spline free form deformation (FFD) non-rigid image registration algorithm. We conduct all our experiments on the open-source Automated Cardiac Diagnosis Challenge (ACDC) dataset. Our experiments show that the deformable image registration results obtained using the proposed method outperform the CNN model and the traditional FFD image registration method.

Bootstrapping Semi-supervised Medical Image Segmentation with Anatomical-Aware Contrastive Distillation

Contrastive learning has shown great promise over annotation scarcity problems in the context of medical image segmentation. Existing approaches typically assume a balanced class distribution for both labeled and unlabeled medical images. However, medical image data in reality is commonly imbalanced (i.e., multi-class label imbalance), which naturally yields blurry contours and usually incorrectly labels rare objects. Moreover, it remains unclear whether all negative samples are equally negative. In this work, we present ACTION, an Anatomical-aware ConTrastive dIstillatiON framework, for semi-supervised medical image segmentation. Specifically, we first develop an iterative contrastive distillation algorithm by softly labeling the negatives rather than binary supervision between positive and negative pairs. We also capture more semantically similar features from the randomly chosen negative set compared to the positives to enforce the diversity of the sampled data. Second, we raise a more important question: Can we really handle imbalanced samples to yield better performance? Hence, the key innovation in ACTION is to learn global semantic relationship across the entire dataset and local anatomical features among the neighbouring pixels with minimal additional memory footprint. During the training, we introduce anatomical contrast by actively sampling a sparse set of hard negative pixels, which can generate smoother segmentation boundaries and more accurate predictions. Extensive experiments across two benchmark datasets and different unlabeled settings show that ACTION significantly outperforms the current state-of-the-art semi-supervised methods.

Estimating Uncertainty in Neural Networks for Cardiac MRI Segmentation: A Benchmark Study

OBJECTIVE

Convolutional neural networks (CNNs) have demonstrated promise in automated cardiac magnetic resonance image segmentation. However, when using CNNs in a large real-world dataset, it is important to quantify segmentation uncertainty and identify segmentations which could be problematic. In this work, we performed a systematic study of Bayesian and non-Bayesian methods for estimating uncertainty in segmentation neural networks.

METHODS

We evaluated Bayes by Backprop, Monte Carlo Dropout, Deep Ensembles, and Stochastic Segmentation Networks in terms of segmentation accuracy, probability calibration, uncertainty on out-of-distribution images, and segmentation quality control.

RESULTS

We observed that Deep Ensembles outperformed the other methods except for images with heavy noise and blurring distortions. We showed that Bayes by Backprop is more robust to noise distortions while Stochastic Segmentation Networks are more resistant to blurring distortions. For segmentation quality control, we showed that segmentation uncertainty is correlated with segmentation accuracy for all the methods. With the incorporation of uncertainty estimates, we were able to reduce the percentage of poor segmentation to 5% by flagging 31-48% of the most uncertain segmentations for manual review, substantially lower than random review without using neural network uncertainty (reviewing 75-78% of all images).

CONCLUSION

This work provides a comprehensive evaluation of uncertainty estimation methods and showed that Deep Ensembles outperformed other methods in most cases.

SIGNIFICANCE

Neural network uncertainty measures can help identify potentially inaccurate segmentations and alert users for manual review.

Assessment and diagnosis of right ventricular failure-retrospection and future directions

The right ventricle (RV) has a critical role in hemodynamics and right ventricular failure (RVF) often leads to poor clinical outcome. Despite the clinical importance of RVF, its definition and recognition currently rely on patients' symptoms and signs, rather than on objective parameters from quantifying RV dimensions and function. A key challenge is the geometrical complexity of the RV, which often makes it difficult to assess RV function accurately. There are several assessment modalities currently utilized in the clinical settings. Each diagnostic investigation has both advantages and limitations according to its characteristics. The purpose of this review is to reflect on the current diagnostic tools, consider the potential technological advancements and propose how to improve the assessment of right ventricular failure. Advanced technique such as automatic evaluation with artificial intelligence and 3-dimensional assessment for the complex RV structure has a potential to improve RV assessment by increasing accuracy and reproducibility of the measurements. Further, noninvasive assessments for RV-pulmonary artery coupling and right and left ventricular interaction are also warranted to overcome the load-related limitations for the accurate evaluation of RV contractile function. Future studies to cross-validate the advanced technologies in various populations are required.

Automatic Multiple Articulator Segmentation in Dynamic Speech MRI Using a Protocol Adaptive Stacked Transfer Learning U-NET Model

Dynamic magnetic resonance imaging has emerged as a powerful modality for investigating upper-airway function during speech production. Analyzing the changes in the vocal tract airspace, including the position of soft-tissue articulators (e.g., the tongue and velum), enhances our understanding of speech production. The advent of various fast speech MRI protocols based on sparse sampling and constrained reconstruction has led to the creation of dynamic speech MRI datasets on the order of 80-100 image frames/second. In this paper, we propose a stacked transfer learning U-NET model to segment the deforming vocal tract in 2D mid-sagittal slices of dynamic speech MRI. Our approach leverages (a) low- and mid-level features and (b) high-level features. The low- and mid-level features are derived from models pre-trained on labeled open-source brain tumor MR and lung CT datasets, and an in-house airway labeled dataset. The high-level features are derived from labeled protocol-specific MR images. The applicability of our approach to segmenting dynamic datasets is demonstrated in data acquired from three fast speech MRI protocols: Protocol 1: 3 T-based radial acquisition scheme coupled with a non-linear temporal regularizer, where speakers were producing French speech tokens; Protocol 2: 1.5 T-based uniform density spiral acquisition scheme coupled with a temporal finite difference (FD) sparsity regularization, where speakers were producing fluent speech tokens in English, and Protocol 3: 3 T-based variable density spiral acquisition scheme coupled with manifold regularization, where speakers were producing various speech tokens from the International Phonetic Alphabetic (IPA). Segments from our approach were compared to those from an expert human user (a vocologist), and the conventional U-NET model without transfer learning. Segmentations from a second expert human user (a radiologist) were used as ground truth. Evaluations were performed using the quantitative DICE similarity metric, the Hausdorff distance metric, and segmentation count metric. This approach was successfully adapted to different speech MRI protocols with only a handful of protocol-specific images (e.g., of the order of 20 images), and provided accurate segmentations similar to those of an expert human.

Lazy Luna: Extendible software for multilevel reader comparison in cardiovascular magnetic resonance imaging

BACKGROUND AND OBJECTIVES

Cardiovascular Magnetic Resonance (CMR) imaging is a growing field with increasing diagnostic utility in clinical routine. Quantitative diagnostic parameters are typically calculated based on contours or points provided by readers, e.g. natural intelligences (NI) such as clinicians or researchers, and artificial intelligences (AI). As clinical applications multiply, evaluating the precision and reproducibility of quantitative parameters becomes increasingly important. Although segmentation challenges for AIs and guidelines for clinicians provide quality assessments and regulation, the methods ought to be combined and streamlined for clinical applications. The goal of the developed software, Lazy Luna (LL), is to offer a flexible evaluation tool that is readily extendible to new sequences and scientific endeavours.

METHODS

An interface was designed for LL, which allows for comparing annotated CMR images. Geometric objects ensure precise calculations of metric values and clinical results regardless of whether annotations originate from AIs or NIs. A graphical user interface (GUI) is provided to make the software available to non-programmers. The GUI allows for an interactive inspection of image datasets as well as implementing tracing procedures, which follow statistical reader differences in clinical results to their origins in individual image contours. The backend software builds on a set of meta-classes, which can be extended to new imaging sequences and clinical parameters. Following an agile development procedure with clinical feedback allows for a quick implementation of new classes, figures and tables for evaluation.

RESULTS

Two application cases present LL's extendibility to clinical evaluation and AI development contexts. The first concerns T1 parametric mapping images segmented by two expert readers. Quantitative result differences are traced to reveal typical segmentation dissimilarities from which these differences originate. The meta-classes are extended to this new application scenario. The second applies to the open source Late Gadolinium Enhancement (LGE) quantification challenge for AI developers "Emidec", which illustrates LL's usability as open source software.

CONCLUSION

The presented software Lazy Luna allows for an automated multilevel comparison of readers as well as identifying qualitative reasons for statistical reader differences. The open source software LL can be extended to new application cases in the future.

CRV-NET: Robust Intensity Recognition of Coronavirus in Lung Computerized Tomography Scan Images

The early diagnosis of infectious diseases is demanded by digital healthcare systems. Currently, the detection of the new coronavirus disease (COVID-19) is a major clinical requirement. For COVID-19 detection, deep learning models are used in various studies, but the robustness is still compromised. In recent years, deep learning models have increased in popularity in almost every area, particularly in medical image processing and analysis. The visualization of the human body's internal structure is critical in medical analysis; many imaging techniques are in use to perform this job. A computerized tomography (CT) scan is one of them, and it has been generally used for the non-invasive observation of the human body. The development of an automatic segmentation method for lung CT scans showing COVID-19 can save experts time and can reduce human error. In this article, the CRV-NET is proposed for the robust detection of COVID-19 in lung CT scan images. A public dataset (SARS-CoV-2 CT Scan dataset), is used for the experimental work and customized according to the scenario of the proposed model. The proposed modified deep-learning-based U-Net model is trained on a custom dataset with 221 training images and their ground truth, which was labeled by an expert. The proposed model is tested on 100 test images, and the results show that the model segments COVID-19 with a satisfactory level of accuracy. Moreover, the comparison of the proposed CRV-NET with different state-of-the-art convolutional neural network models (CNNs), including the U-Net Model, shows better results in terms of accuracy (96.67%) and robustness (low epoch value in detection and the smallest training data size).