Deep Learning Techniques for Automatic MRI Cardiac Multi-Structures Segmentation and Diagnosis: Is the Problem Solved?

SC-SSL: Self-Correcting Collaborative and Contrastive Co-Training Model for Semi-Supervised Medical Image Segmentation

Image segmentation achieves significant improvements with deep neural networks at the premise of a large scale of labeled training data, which is laborious to assure in medical image tasks. Recently, semi-supervised learning (SSL) has shown great potential in medical image segmentation. However, the influence of the learning target quality for unlabeled data is usually neglected in these SSL methods. Therefore, this study proposes a novel self-correcting co-training scheme to learn a better target that is more similar to ground-truth labels from collaborative network outputs. Our work has three-fold highlights. First, we advance the learning target generation as a learning task, improving the learning confidence for unannotated data with a self-correcting module. Second, we impose a structure constraint to encourage the shape similarity further between the improved learning target and the collaborative network outputs. Finally, we propose an innovative pixel-wise contrastive learning loss to boost the representation capacity under the guidance of an improved learning target, thus exploring unlabeled data more efficiently with the awareness of semantic context. We have extensively evaluated our method with the state-of-the-art semi-supervised approaches on four public-available datasets, including the ACDC dataset, M&Ms dataset, Pancreas-CT dataset, and Task_07 CT dataset. The experimental results with different labeled-data ratios show our proposed method's superiority over other existing methods, demonstrating its effectiveness in semi-supervised medical image segmentation.

Multi-network approach for image segmentation in non-contrast enhanced cardiac 3D MRI of arrhythmic patients

Left atrial appendage (LAA) is the source of thrombi formation in more than 90% of strokes in patients with nonvalvular atrial fibrillation. Catheter-based LAA occlusion is being increasingly applied as a treatment strategy to prevent stroke. Anatomical complexity of LAA makes percutaneous occlusion commonly performed under transesophageal echocardiography (TEE) and X-ray (XR) guidance especially challenging. Image fusion techniques integrating 3D anatomical models derived from pre-procedural imaging into the live XR fluoroscopy can be applied to guide each step of the LAA closure. Cardiac magnetic resonance (CMR) imaging gains in importance for radiation-free evaluation of cardiac morphology as alternative to gold-standard TEE or computed tomography angiography (CTA). Manual delineation of cardiac structures from non-contrast enhanced CMR is, however, labor-intensive, tedious, and challenging due to the rather low contrast. Additionally, arrhythmia often impairs the image quality in ECG synchronized acquisitions causing blurring and motion artifacts. Thus, for cardiac segmentation in arrhythmic patients, there is a strong need for an automated image segmentation method. Deep learning-based methods have shown great promise in medical image analysis achieving superior performance in various imaging modalities and different clinical applications. Fully-convolutional neural networks (CNNs), especially U-Net, have become the method of choice for cardiac segmentation. In this paper, we propose an approach for automatic segmentation of cardiac structures from non-contrast enhanced CMR images of arrhythmic patients based on CNNs implemented in a multi-stage pipeline. Two-stage implementation allows subdividing the task into localization of the relevant cardiac structures and segmentation of these structures from the cropped sub-regions obtained from previous step leading to efficient and effective way of automated cardiac segmentation.

Semi-Supervised Medical Image Segmentation Using Cross-Style Consistency With Shape-Aware and Local Context Constraints

Despite the remarkable progress in semi-supervised medical image segmentation methods based on deep learning, their application to real-life clinical scenarios still faces considerable challenges. For example, insufficient labeled data often makes it difficult for networks to capture the complexity and variability of the anatomical regions to be segmented. To address these problems, we design a new semi-supervised segmentation framework that aspires to produce anatomically plausible predictions. Our framework comprises two parallel networks: shape-agnostic and shape-aware networks. These networks learn from each other, enabling effective utilization of unlabeled data. Our shape-aware network implicitly introduces shape guidance to capture shape fine-grained information. Meanwhile, shape-agnostic networks employ uncertainty estimation to further obtain reliable pseudo-labels for the counterpart. We also employ a cross-style consistency strategy to enhance the network's utilization of unlabeled data. It enriches the dataset to prevent overfitting and further eases the coupling of the two networks that learn from each other. Our proposed architecture also incorporates a novel loss term that facilitates the learning of the local context of segmentation by the network, thereby enhancing the overall accuracy of prediction. Experiments on three different datasets of medical images show that our method outperforms many excellent semi-supervised segmentation methods and outperforms them in perceiving shape. The code can be seen at https://github.com/igip-liu/SLC-Net.

Physics informed contour selection for rapid image segmentation

Effective training of deep image segmentation models is challenging due to the need for abundant, high-quality annotations. To facilitate image annotation, we introduce Physics Informed Contour Selection (PICS)-an interpretable, physics-informed algorithm for rapid image segmentation without relying on labeled data. PICS draws inspiration from physics-informed neural networks (PINNs) and an active contour model called snake. It is fast and computationally lightweight because it employs cubic splines instead of a deep neural network as a basis function. Its training parameters are physically interpretable because they directly represent control knots of the segmentation curve. Traditional snakes involve minimization of the edge-based loss functionals by deriving the Euler-Lagrange equation followed by its numerical solution. However, PICS directly minimizes the loss functional, bypassing the Euler Lagrange equations. It is the first snake variant to minimize a region-based loss function instead of traditional edge-based loss functions. PICS uniquely models the three-dimensional (3D) segmentation process with an unsteady partial differential equation (PDE), which allows accelerated segmentation via transfer learning. To demonstrate its effectiveness, we apply PICS for 3D segmentation of the left ventricle on a publicly available cardiac dataset. We also demonstrate PICS's capacity to encode the prior shape information as a loss term by proposing a new convexity-preserving loss term for left ventricle. Overall, PICS presents several novelties in network architecture, transfer learning, and physics-inspired losses for image segmentation, thereby showing promising outcomes and potential for further refinement.

SCANeXt: Enhancing 3D medical image segmentation with dual attention network and depth-wise convolution

Existing approaches to 3D medical image segmentation can be generally categorized into convolution-based or transformer-based methods. While convolutional neural networks (CNNs) demonstrate proficiency in extracting local features, they encounter challenges in capturing global representations. In contrast, the consecutive self-attention modules present in vision transformers excel at capturing long-range dependencies and achieving an expanded receptive field. In this paper, we propose a novel approach, termed SCANeXt, for 3D medical image segmentation. Our method combines the strengths of dual attention (Spatial and Channel Attention) and ConvNeXt to enhance representation learning for 3D medical images. In particular, we propose a novel self-attention mechanism crafted to encompass spatial and channel relationships throughout the entire feature dimension. To further extract multiscale features, we introduce a depth-wise convolution block inspired by ConvNeXt after the dual attention block. Extensive evaluations on three benchmark datasets, namely Synapse, BraTS, and ACDC, demonstrate the effectiveness of our proposed method in terms of accuracy. Our SCANeXt model achieves a state-of-the-art result with a Dice Similarity Score of 95.18% on the ACDC dataset, significantly outperforming current methods.

MDT: semi-supervised medical image segmentation with mixup-decoupling training

Objective. In the field of medicine, semi-supervised segmentation algorithms hold crucial research significance while also facing substantial challenges, primarily due to the extreme scarcity of expert-level annotated medical image data. However, many existing semi-supervised methods still process labeled and unlabeled data in inconsistent ways, which can lead to knowledge learned from labeled data being discarded to some extent. This not only lacks a variety of perturbations to explore potential robust information in unlabeled data but also ignores the confirmation bias and class imbalance issues in pseudo-labeling methods.Approach. To solve these problems, this paper proposes a semi-supervised medical image segmentation method 'mixup-decoupling training (MDT)' that combines the idea of consistency and pseudo-labeling. Firstly, MDT introduces a new perturbation strategy 'mixup-decoupling' to fully regularize training data. It not only mixes labeled and unlabeled data at the data level but also performs decoupling operations between the output predictions of mixed target data and labeled data at the feature level to obtain strong version predictions of unlabeled data. Then it establishes a dual learning paradigm based on consistency and pseudo-labeling. Secondly, MDT employs a novel categorical entropy filtering approach to pick high-confidence pseudo-labels for unlabeled data, facilitating more refined supervision.Main results. This paper compares MDT with other advanced semi-supervised methods on 2D and 3D datasets separately. A large number of experimental results show that MDT achieves competitive segmentation performance and outperforms other state-of-the-art semi-supervised segmentation methods.Significance. This paper proposes a semi-supervised medical image segmentation method MDT, which greatly reduces the demand for manually labeled data and eases the difficulty of data annotation to a great extent. In addition, MDT not only outperforms many advanced semi-supervised image segmentation methods in quantitative and qualitative experimental results, but also provides a new and developable idea for semi-supervised learning and computer-aided diagnosis technology research.

Quaternion Cross-Modality Spatial Learning for Multi-Modal Medical Image Segmentation

Recently, the Deep Neural Networks (DNNs) have had a large impact on imaging process including medical image segmentation, and the real-valued convolution of DNN has been extensively utilized in multi-modal medical image segmentation to accurately segment lesions via learning data information. However, the weighted summation operation in such convolution limits the ability to maintain spatial dependence that is crucial for identifying different lesion distributions. In this paper, we propose a novel Quaternion Cross-modality Spatial Learning (Q-CSL) which explores the spatial information while considering the linkage between multi-modal images. Specifically, we introduce to quaternion to represent data and coordinates that contain spatial information. Additionally, we propose Quaternion Spatial-association Convolution to learn the spatial information. Subsequently, the proposed De-level Quaternion Cross-modality Fusion (De-QCF) module excavates inner space features and fuses cross-modality spatial dependency. Our experimental results demonstrate that our approach compared to the competitive methods perform well with only 0.01061 M parameters and 9.95G FLOPs.

Light-M: An efficient lightweight medical image segmentation framework for resource-constrained IoMT

The Internet of Medical Things (IoMT) is being incorporated into current healthcare systems. This technology intends to connect patients, IoMT devices, and hospitals over mobile networks, allowing for more secure, quick, and convenient health monitoring and intelligent healthcare services. However, existing intelligent healthcare applications typically rely on large-scale AI models, and standard IoMT devices have significant resource constraints. To alleviate this paradox, in this paper, we propose a Knowledge Distillation (KD)-based IoMT end-edge-cloud orchestrated architecture for medical image segmentation tasks, called Light-M, aiming to deploy a lightweight medical model in resource-constrained IoMT devices. Specifically, Light-M trains a large teacher model in the cloud server and employs computation in local nodes through imitation of the performance of the teacher model using knowledge distillation. Light-M contains two KD strategies: (1) active exploration and passive transfer (AEPT) and (2) self-attention-based inter-class feature variation (AIFV) distillation for the medical image segmentation task. The AEPT encourages the student model to learn undiscovered knowledge/features of the teacher model without additional feature layers, aiming to explore new features and outperform the teacher. To improve the distinguishability of the student for different classes, the student learns the self-attention-based feature variation (AIFV) between classes. Since the proposed AEPT and AIFV only appear in the training process, our framework does not involve any additional computation burden for a student model during the segmentation task deployment. Extensive experiments on cardiac images and public real-scene datasets demonstrate that our approach improves student model learning representations and outperforms state-of-the-art methods by combining two knowledge distillation strategies. Moreover, when deployed on the IoT device, the distilled student model takes only 29.6 ms for one sample at the inference step.

Evidence-based uncertainty-aware semi-supervised medical image segmentation

Semi-Supervised Learning (SSL) has demonstrated great potential to reduce the dependence on a large set of annotated data, which is challenging to collect in clinical practice. One of the most important SSL methods is to generate pseudo labels from the unlabeled data using a network model trained with labeled data, which will inevitably introduce false pseudo labels into the training process and potentially jeopardize performance. To address this issue, uncertainty-aware methods have emerged as a promising solution and have gained considerable attention recently. However, current uncertainty-aware methods usually face the dilemma of balancing the additional computational cost, uncertainty estimation accuracy, and theoretical basis in a unified training paradigm. To address this issue, we propose to integrate the Dempster-Shafer Theory of Evidence (DST) into SSL-based medical image segmentation, dubbed EVidential Inference Learning (EVIL). EVIL performs as a novel consistency regularization-based training paradigm, which enforces consistency on predictions perturbed by two networks with different parameters to enhance generalization Additionally, EVIL provides a theoretically assured solution for precise uncertainty quantification within a single forward pass. By discarding highly unreliable pseudo labels after uncertainty estimation, trustworthy pseudo labels can be generated and incorporated into subsequent model training. The experimental results demonstrate that the proposed approach performs competitively when benchmarked against several state-of-the-art methods on public datasets, i.e., ACDC, MM-WHS, and MonuSeg. The code can be found at https://github.com/CYYukio/EVidential-Inference-Learning.

Cooperative-Net: An end-to-end multi-task interaction network for unified reconstruction and segmentation of MR image

BACKGROUND AND OBJECTIVE

In clinical applications, there is an increasing demand for rapid acquisition and automated analysis of magnetic resonance imaging (MRI) data. However, most existing methods focus on either MR image reconstruction from undersampled data or segmentation using fully sampled data, hardly considering MR image segmentation in fast imaging scenarios. Consequently, it is imperative to investigate a multi-task approach that can simultaneously achieve high scanning acceleration and accurate segmentation results.

METHODS

In this paper, we propose a novel end-to-end multi-task interaction network, termed as the Cooperative-Net, which integrates accelerated MR imaging and multi-class tissue segmentation into a unified framework. The Cooperative-Net consists of alternating reconstruction modules and segmentation modules. To facilitate effective interaction between the two tasks, we introduce the spatial-adaptive semantic guidance module, which leverages the semantic map as a structural prior to guide MR image reconstruction. Furthermore, we propose a novel unrolling network with a multi-path shrinkage structure for MR image reconstruction. This network consists of parallel learnable shrinkage paths to handle varying degrees of degradation across different frequency components in the undersampled MR image, effectively improving the quality of the recovered image.

RESULTS

We use two publicly available datasets, including the cardiac and knee MR datasets, to validate the efficacy of our proposed Cooperative-Net. Through qualitative and quantitative analysis, we demonstrate that our method outperforms existing state-of-the-art multi-task approaches for joint MR image reconstruction and segmentation.

CONCLUSIONS

The proposed Cooperative-Net is capable of achieving both high accelerated MR imaging and accurate multi-class tissue segmentation.

MS-TCNet: An effective Transformer-CNN combined network using multi-scale feature learning for 3D medical image segmentation

Medical image segmentation is a fundamental research problem in the field of medical image processing. Recently, the Transformer have achieved highly competitive performance in computer vision. Therefore, many methods combining Transformer with convolutional neural networks (CNNs) have emerged for segmenting medical images. However, these methods cannot effectively capture the multi-scale features in medical images, even though texture and contextual information embedded in the multi-scale features are extremely beneficial for segmentation. To alleviate this limitation, we propose a novel Transformer-CNN combined network using multi-scale feature learning for three-dimensional (3D) medical image segmentation, which is called MS-TCNet. The proposed model utilizes a shunted Transformer and CNN to construct an encoder and pyramid decoder, allowing six different scale levels of feature learning. It captures multi-scale features with refinement at each scale level. Additionally, we propose a novel lightweight multi-scale feature fusion (MSFF) module that can fully fuse the different-scale semantic features generated by the pyramid decoder for each segmentation class, resulting in a more accurate segmentation output. We conducted experiments on three widely used 3D medical image segmentation datasets. The experimental results indicated that our method outperformed state-of-the-art medical image segmentation methods, suggesting its effectiveness, robustness, and superiority. Meanwhile, our model has a smaller number of parameters and lower computational complexity than conventional 3D segmentation networks. The results confirmed that the model is capable of effective multi-scale feature learning and that the learned multi-scale features are useful for improving segmentation performance. We open-sourced our code, which can be found at https://github.com/AustinYuAo/MS-TCNet.

Dual consistency regularization with subjective logic for semi-supervised medical image segmentation

Semi-supervised learning plays a vital role in computer vision tasks, particularly in medical image analysis. It significantly reduces the time and cost involved in labeling data. Current methods primarily focus on consistency regularization and the generation of pseudo labels. However, due to the model's poor awareness of unlabeled data, aforementioned methods may misguide the model. To alleviate this problem, we propose a dual consistency regularization with subjective logic for semi-supervised medical image segmentation. Specifically, we introduce subjective logic into our semi-supervised medical image segmentation task to estimate uncertainty, and based on the consistency hypothesis, we construct dual consistency regularization under weak and strong perturbations to guide the model's learning from unlabeled data. To evaluate the performance of the proposed method, we performed experiments on three widely used datasets: ACDC, LA, and Pancreas. Experiments show that the proposed method achieved improvement compared with other state-of-the-art (SOTA) methods.

A Deep Learning-Based Integrated Framework for Quality-Aware Undersampled Cine Cardiac MRI Reconstruction and Analysis

Cine cardiac magnetic resonance (CMR) imaging is considered the gold standard for cardiac function evaluation. However, cine CMR acquisition is inherently slow and in recent decades considerable effort has been put into accelerating scan times without compromising image quality or the accuracy of derived results. In this article, we present a fully-automated, quality-controlled integrated framework for reconstruction, segmentation and downstream analysis of undersampled cine CMR data. The framework produces high quality reconstructions and segmentations, leading to undersampling factors that are optimised on a scan-by-scan basis. This results in reduced scan times and automated analysis, enabling robust and accurate estimation of functional biomarkers. To demonstrate the feasibility of the proposed approach, we perform simulations of radial k-space acquisitions using in-vivo cine CMR data from 270 subjects from the UK Biobank (with synthetic phase) and in-vivo cine CMR data from 16 healthy subjects (with real phase). The results demonstrate that the optimal undersampling factor varies for different subjects by approximately 1 to 2 seconds per slice. We show that our method can produce quality-controlled images in a mean scan time reduced from 12 to 4 seconds per slice, and that image quality is sufficient to allow clinically relevant parameters to be automatically estimated to lie within 5% mean absolute difference.

A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis

Disease diagnosis represents a critical and arduous endeavor within the medical field. Artificial intelligence (AI) techniques, spanning from machine learning and deep learning to large model paradigms, stand poised to significantly augment physicians in rendering more evidence-based decisions, thus presenting a pioneering solution for clinical practice. Traditionally, the amalgamation of diverse medical data modalities (e.g., image, text, speech, genetic data, physiological signals) is imperative to facilitate a comprehensive disease analysis, a topic of burgeoning interest among both researchers and clinicians in recent times. Hence, there exists a pressing need to synthesize the latest strides in multi-modal data and AI technologies in the realm of medical diagnosis. In this paper, we narrow our focus to five specific disorders (Alzheimer's disease, breast cancer, depression, heart disease, epilepsy), elucidating advanced endeavors in their diagnosis and treatment through the lens of artificial intelligence. Our survey not only delineates detailed diagnostic methodologies across varying modalities but also underscores commonly utilized public datasets, the intricacies of feature engineering, prevalent classification models, and envisaged challenges for future endeavors. In essence, our research endeavors to contribute to the advancement of diagnostic methodologies, furnishing invaluable insights for clinical decision making.

Assessment of deep learning segmentation for real-time free-breathing cardiac magnetic resonance imaging at rest and under exercise stress

In recent years, a variety of deep learning networks for cardiac MRI (CMR) segmentation have been developed and analyzed. However, nearly all of them are focused on cine CMR under breathold. In this work, accuracy of deep learning methods is assessed for volumetric analysis (via segmentation) of the left ventricle in real-time free-breathing CMR at rest and under exercise stress. Data from healthy volunteers (n = 15) for cine and real-time free-breathing CMR at rest and under exercise stress were analyzed retrospectively. Exercise stress was performed using an ergometer in the supine position. Segmentations of two deep learning methods, a commercially available technique (comDL) and an openly available network (nnU-Net), were compared to a reference model created via the manual correction of segmentations obtained with comDL. Segmentations of left ventricular endocardium (LV), left ventricular myocardium (MYO), and right ventricle (RV) are compared for both end-systolic and end-diastolic phases and analyzed with Dice's coefficient. The volumetric analysis includes the cardiac function parameters LV end-diastolic volume (EDV), LV end-systolic volume (ESV), and LV ejection fraction (EF), evaluated with respect to both absolute and relative differences. For cine CMR, nnU-Net and comDL achieve Dice's coefficients above 0.95 for LV and 0.9 for MYO, and RV. For real-time CMR, the accuracy of nnU-Net exceeds that of comDL overall. For real-time CMR at rest, nnU-Net achieves Dice's coefficients of 0.94 for LV, 0.89 for MYO, and 0.90 for RV and the mean absolute differences between nnU-Net and the reference are 2.9 mL for EDV, 3.5 mL for ESV, and 2.6% for EF. For real-time CMR under exercise stress, nnU-Net achieves Dice's coefficients of 0.92 for LV, 0.85 for MYO, and 0.83 for RV and the mean absolute differences between nnU-Net and reference are 11.4 mL for EDV, 2.9 mL for ESV, and 3.6% for EF. Deep learning methods designed or trained for cine CMR segmentation can perform well on real-time CMR. For real-time free-breathing CMR at rest, the performance of deep learning methods is comparable to inter-observer variability in cine CMR and is usable for fully automatic segmentation. For real-time CMR under exercise stress, the performance of nnU-Net could promise a higher degree of automation in the future.

Advantages of transformer and its application for medical image segmentation: a survey

PURPOSE

Convolution operator-based neural networks have shown great success in medical image segmentation over the past decade. The U-shaped network with a codec structure is one of the most widely used models. Transformer, a technology used in natural language processing, can capture long-distance dependencies and has been applied in Vision Transformer to achieve state-of-the-art performance on image classification tasks. Recently, researchers have extended transformer to medical image segmentation tasks, resulting in good models.

METHODS

This review comprises publications selected through a Web of Science search. We focused on papers published since 2018 that applied the transformer architecture to medical image segmentation. We conducted a systematic analysis of these studies and summarized the results.

RESULTS

To better comprehend the benefits of convolutional neural networks and transformers, the construction of the codec and transformer modules is first explained. Second, the medical image segmentation model based on transformer is summarized. The typically used assessment markers for medical image segmentation tasks are then listed. Finally, a large number of medical segmentation datasets are described.

CONCLUSION

Even if there is a pure transformer model without any convolution operator, the sample size of medical picture segmentation still restricts the growth of the transformer, even though it can be relieved by a pretraining model. More often than not, researchers are still designing models using transformer and convolution operators.

Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation

Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches. Recent success of deep learning-based segmentation methods usually relies on a large amount of labeled data, which is particularly difficult and costly to obtain, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. Semi-supervised learning has emerged as an appealing strategy and been widely applied to medical image segmentation tasks to train deep models with limited annotations. In this paper, we present a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarize both the technical novelties and empirical results. Furthermore, we analyze and discuss the limitations and several unsolved problems of existing approaches. We hope this review can inspire the research community to explore solutions to this challenge and further advance the field of medical image segmentation.

HPFG: semi-supervised medical image segmentation framework based on hybrid pseudo-label and feature-guiding

Semi-supervised learning methods have been attracting much attention in medical image segmentation due to the lack of high-quality annotation. To cope with the noise problem of pseudo-label in semi-supervised medical image segmentation and the limitations of contrastive learning applications, we propose a semi-supervised medical image segmentation framework, HPFG, based on hybrid pseudo-label and feature-guiding, which consists of a hybrid pseudo-label strategy and two different feature-guiding modules. The hybrid pseudo-label strategy uses the CutMix operation and an auxiliary network to enable the labeled images to guide the unlabeled images to generate high-quality pseudo-label and reduce the impact of pseudo-label noise. In addition, a feature-guiding encoder module based on feature-level contrastive learning is designed to guide the encoder to mine useful local and global image features, thus effectively enhancing the feature extraction capability of the model. At the same time, a feature-guiding decoder module based on adaptive class-level contrastive learning is designed to guide the decoder in better extracting class information, achieving intra-class affinity and inter-class separation, and effectively alleviating the class imbalance problem in medical datasets. Extensive experimental results show that the segmentation performance of the HPFG framework proposed in this paper outperforms existing semi-supervised medical image segmentation methods on three public datasets: ACDC, LIDC, and ISIC. Code is available at https://github.com/fakerlove1/HPFG .

Deep learning based synthesis of MRI, CT and PET: Review and analysis

Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis.

Segment anything model for medical images?

The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: (1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. (2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. (3) SAM performed better with manual hints, especially box, than the Everything mode. (4) SAM could help human annotation with high labeling quality and less time. (5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. (6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. (7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. (8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. Codes and models are available at: https://github.com/yuhoo0302/Segment-Anything-Model-for-Medical-Images. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.

STANet: Spatio-Temporal Adaptive Network and Clinical Prior Embedding Learning for 3D+T CMR Segmentation

The segmentation of cardiac structure in magnetic resonance images (CMR) is paramount in diagnosing and managing cardiovascular illnesses, given its 3D+Time (3D+T) sequence. The existing deep learning methods are constrained in their ability to 3D+T CMR segmentation, due to: (1) Limited motion perception. The complexity of heart beating renders the motion perception in 3D+T CMR, including the long-range and cross-slice motions. The existing methods' local perception and slice-fixed perception directly limit the performance of 3D+T CMR perception. (2) Lack of labels. Due to the expensive labeling cost of the 3D+T CMR sequence, the labels of 3D+T CMR only contain the end-diastolic and end-systolic frames. The incomplete labeling scheme causes inefficient supervision. Hence, we propose a novel spatio-temporal adaptation network with clinical prior embedding learning (STANet) to ensure efficient spatio-temporal perception and optimization on 3D+T CMR segmentation. (1) A spatio-temporal adaptive convolution (STAC) treats the 3D+T CMR sequence as a whole for perception. The long-distance motion correlation is embedded into the structural perception by learnable weight regularization to balance long-range motion perception. The structural similarity is measured by cross-attention to adaptively correlate the cross-slice motion. (2) A clinical prior embedding learning strategy (CPE) is proposed to optimize the partially labeled 3D+T CMR segmentation dynamically by embedding clinical priors into optimization. STANet achieves outstanding performance with Dice of 0.917 and 0.94 on two public datasets (ACDC and STACOM), which indicates STANet has the potential to be incorporated into computer-aided diagnosis tools for clinical application.

Comparison of manual and artificial intelligence based quantification of myocardial strain by feature tracking-a cardiovascular MR study in health and disease

OBJECTIVES

The analysis of myocardial deformation using feature tracking in cardiovascular MR allows for the assessment of global and segmental strain values. The aim of this study was to compare strain values derived from artificial intelligence (AI)-based contours with manually derived strain values in healthy volunteers and patients with cardiac pathologies.

MATERIALS AND METHODS

A cohort of 136 subjects (60 healthy volunteers and 76 patients; of those including 46 cases with left ventricular hypertrophy (LVH) of varying etiology and 30 cases with chronic myocardial infarction) was analyzed. Comparisons were based on quantitative strain analysis and on a geometric level by the Dice similarity coefficient (DSC) of the segmentations. Strain quantification was performed in 3 long-axis slices and short-axis (SAX) stack with epi- and endocardial contours in end-diastole. AI contours were checked for plausibility and potential errors in the tracking algorithm.

RESULTS

AI-derived strain values overestimated radial strain (+ 1.8 ± 1.7% (mean difference ± standard deviation); p = 0.03) and underestimated circumferential (- 0.8 ± 0.8%; p = 0.02) and longitudinal strain (- 0.1 ± 0.8%; p = 0.54). Pairwise group comparisons revealed no significant differences for global strain. The DSC showed good agreement for healthy volunteers (85.3 ± 10.3% for SAX) and patients (80.8 ± 9.6% for SAX). In 27 cases (27/76; 35.5%), a tracking error was found, predominantly (24/27; 88.9%) in the LVH group and 22 of those (22/27; 81.5%) at the insertion of the papillary muscle in lateral segments.

CONCLUSIONS

Strain analysis based on AI-segmented images shows good results in healthy volunteers and in most of the patient groups. Hypertrophied ventricles remain a challenge for contouring and feature tracking.

CLINICAL RELEVANCE STATEMENT

AI-based segmentations can help to streamline and standardize strain analysis by feature tracking.

KEY POINTS

• Assessment of strain in cardiovascular magnetic resonance by feature tracking can generate global and segmental strain values. • Commercially available artificial intelligence algorithms provide segmentation for strain analysis comparable to manual segmentation. • Hypertrophied ventricles are challenging in regards of strain analysis by feature tracking.

Artificial Intelligence in Heart Failure: Friend or Foe?

In recent times, there have been notable changes in cardiovascular medicine, propelled by the swift advancements in artificial intelligence (AI). The present work provides an overview of the current applications and challenges of AI in the field of heart failure. It emphasizes the "garbage in, garbage out" issue, where AI systems can produce inaccurate results with skewed data. The discussion covers issues in heart failure diagnostic algorithms, particularly discrepancies between existing models. Concerns about the reliance on the left ventricular ejection fraction (LVEF) for classification and treatment are highlighted, showcasing differences in current scientific perceptions. This review also delves into challenges in implementing AI, including variable considerations and biases in training data. It underscores the limitations of current AI models in real-world scenarios and the difficulty in interpreting their predictions, contributing to limited physician trust in AI-based models. The overarching suggestion is that AI can be a valuable tool in clinicians' hands for treating heart failure patients, as far as existing medical inaccuracies have been addressed before integrating AI into these frameworks.

Promoting fast MR imaging pipeline by full-stack AI

Magnetic resonance imaging (MRI) is a widely used imaging modality in clinics for medical disease diagnosis, staging, and follow-up. Deep learning has been extensively used to accelerate k-space data acquisition, enhance MR image reconstruction, and automate tissue segmentation. However, these three tasks are usually treated as independent tasks and optimized for evaluation by radiologists, thus ignoring the strong dependencies among them; this may be suboptimal for downstream intelligent processing. Here, we present a novel paradigm, full-stack learning (FSL), which can simultaneously solve these three tasks by considering the overall imaging process and leverage the strong dependence among them to further improve each task, significantly boosting the efficiency and efficacy of practical MRI workflows. Experimental results obtained on multiple open MR datasets validate the superiority of FSL over existing state-of-the-art methods on each task. FSL has great potential to optimize the practical workflow of MRI for medical diagnosis and radiotherapy.

Artificial intelligence generated content (AIGC) in medicine: A narrative review

Recently, artificial intelligence generated content (AIGC) has been receiving increased attention and is growing exponentially. AIGC is generated based on the intentional information extracted from human-provided instructions by generative artificial intelligence (AI) models. AIGC quickly and automatically generates large amounts of high-quality content. Currently, there is a shortage of medical resources and complex medical procedures in medicine. Due to its characteristics, AIGC can help alleviate these problems. As a result, the application of AIGC in medicine has gained increased attention in recent years. Therefore, this paper provides a comprehensive review on the recent state of studies involving AIGC in medicine. First, we present an overview of AIGC. Furthermore, based on recent studies, the application of AIGC in medicine is reviewed from two aspects: medical image processing and medical text generation. The basic generative AI models, tasks, target organs, datasets and contribution of studies are considered and summarized. Finally, we also discuss the limitations and challenges faced by AIGC and propose possible solutions with relevant studies. We hope this review can help readers understand the potential of AIGC in medicine and obtain some innovative ideas in this field.

Adaptive t-vMF dice loss: An effective expansion of dice loss for medical image segmentation

Dice loss is widely used for medical image segmentation, and many improved loss functions have been proposed. However, further Dice loss improvements are still possible. In this study, we reconsidered the use of Dice loss and discovered that Dice loss can be rewritten in the loss function using the cosine similarity through a simple equation transformation. Using this knowledge, we present a novel t-vMF Dice loss based on the t-vMF similarity instead of the cosine similarity. Based on the t-vMF similarity, our proposed Dice loss is formulated in a more compact similarity loss function than the original Dice loss. Furthermore, we present an effective algorithm that automatically determines the parameter κ for the t-vMF similarity using a validation accuracy, called Adaptive t-vMF Dice loss. Using this algorithm, it is possible to apply more compact similarities for easy classes and wider similarities for difficult classes, and we are able to achieve adaptive training based on the accuracy of each class. We evaluated binary segmentation datasets of CVC-ClinicDB and Kvasir-SEG, and multi-class segmentation datasets of Automated Cardiac Diagnosis Challenge and Synapse multi-organ segmentation. Through experiments conducted on four datasets using a five-fold cross-validation, we confirmed that the Dice score coefficient (DSC) was further improved in comparison with the original Dice loss and other loss functions.

Attention-UNet architectures with pretrained backbones for multi-class cardiac MR image segmentation

Segmentation architectures based on deep learning proficient extraordinary results in medical imaging technologies. Computed tomography (CT) images and Magnetic Resonance Imaging (MRI) in diagnosis and treatment are increasing and significantly support the diagnostic process by removing the bottlenecks of manual segmentation. Cardiac Magnetic Resonance Imaging (CMRI) is a state-of-the-art imaging technique used to acquire vital heart measurements and has received extensive attention from researchers for automatic segmentation. Deep learning methods offer high-precision segmentation but still pose several difficulties, such as pixel homogeneity in nearby organs. The motivated study using the attention mechanism approach was introduced for medical images for automated algorithms. The experiment focuses on observing the impact of the attention mechanism with and without pretrained backbone networks on the UNet model. For the same, three networks are considered: Attention-UNet, Attention-UNet with resnet50 pretrained backbone and Attention-UNet with densenet121 pretrained backbone. The experiments are performed on the ACDC Challenge 2017 dataset. The performance is evaluated by conducting a comparative analysis based on the Dice Coefficient, IoU Coefficient, and cross-entropy loss calculations. The Attention-UNet, Attention-UNet with resnet50 pretrained backbone, and Attention-UNet with densenet121 pretrained backbone networks obtained Dice Coefficients of 0.9889, 0.9720, and 0.9801, respectively, along with corresponding IoU scores of 0.9781, 0.9457, and 0.9612. Results compared with the state-of-the-art methods indicate that the methods are on par with, or even superior in terms of both the Dice coefficient and Intersection-over-union.

Advances in medical image analysis with vision Transformers: A comprehensive review

The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.

Joint shape/texture representation learning for cardiovascular disease diagnosis from magnetic resonance imaging

Aims

Cardiovascular diseases (CVDs) are the leading cause of mortality worldwide. Cardiac image and mesh are two primary modalities to present the shape and structure of the heart and have been demonstrated to be efficient in CVD prediction and diagnosis. However, previous research has been generally focussed on a single modality (image or mesh), and few of them have tried to jointly consider the image and mesh representations of heart. To obtain efficient and explainable biomarkers for CVD prediction and diagnosis, it is needed to jointly consider both representations.

Methods and results

We design a novel multi-channel variational auto-encoder, mesh-image variational auto-encoder, to learn joint representation of paired mesh and image. After training, the shape-aware image representation (SAIR) can be learned directly from the raw images and applied for further CVD prediction and diagnosis. We demonstrate our method on data from UK Biobank study and two other datasets via extensive experiments. In acute myocardial infarction prediction, SAIR achieves 81.43% accuracy, significantly higher than traditional biomarkers like metadata and clinical indices (left ventricle and right ventricle clinical indices of cardiac function like chamber volume, mass, and ejection fraction).

Conclusion

Our mesh-image variational auto-encoder provides a novel approach for 3D cardiac mesh reconstruction from images. The extraction of SAIR is fast and without need of segmentation masks, and its focussing can be visualized in the corresponding cardiac meshes. SAIR archives better performance than traditional biomarkers and can be applied as an efficient supplement to them, which is of significant potential in CVD analysis.

Clair N, Miller GF, Dorfman AL, Fogel MA, Ghelani S, Krishnamurthy R, Lam CZ, Quail M, Robinson JD, Schidlow D, Slesnick TC, Weigand J, Steeden JA, Rathod RH, Muthurangu V. A Deep Learning Pipeline for Assessing Ventricular Volumes from a Cardiac MRI Registry of Patients with Single Ventricle Physiology

Purpose To develop an end-to-end deep learning (DL) pipeline for automated ventricular segmentation of cardiac MRI data from a multicenter registry of patients with Fontan circulation (Fontan Outcomes Registry Using CMR Examinations [FORCE]). Materials and Methods This retrospective study used 250 cardiac MRI examinations (November 2007-December 2022) from 13 institutions for training, validation, and testing. The pipeline contained three DL models: a classifier to identify short-axis cine stacks and two U-Net 3+ models for image cropping and segmentation. The automated segmentations were evaluated on the test set (n = 50) by using the Dice score. Volumetric and functional metrics derived from DL and ground truth manual segmentations were compared using Bland-Altman and intraclass correlation analysis. The pipeline was further qualitatively evaluated on 475 unseen examinations. Results There were acceptable limits of agreement (LOA) and minimal biases between the ground truth and DL end-diastolic volume (EDV) (bias: -0.6 mL/m2, LOA: -20.6 to 19.5 mL/m2) and end-systolic volume (ESV) (bias: -1.1 mL/m2, LOA: -18.1 to 15.9 mL/m2), with high intraclass correlation coefficients (ICCs > 0.97) and Dice scores (EDV, 0.91 and ESV, 0.86). There was moderate agreement for ventricular mass (bias: -1.9 g/m2, LOA: -17.3 to 13.5 g/m2) and an ICC of 0.94. There was also acceptable agreement for stroke volume (bias: 0.6 mL/m2, LOA: -17.2 to 18.3 mL/m2) and ejection fraction (bias: 0.6%, LOA: -12.2% to 13.4%), with high ICCs (>0.81). The pipeline achieved satisfactory segmentation in 68% of the 475 unseen examinations, while 26% needed minor adjustments, 5% needed major adjustments, and in 0.4%, the cropping model failed. Conclusion The DL pipeline can provide fast standardized segmentation for patients with single ventricle physiology across multiple centers. This pipeline can be applied to all cardiac MRI examinations in the FORCE registry. Keywords: Cardiac, Adults and Pediatrics, MR Imaging, Congenital, Volume Analysis, Segmentation, Quantification Supplemental material is available for this article. © RSNA, 2023.

CSAM: A 2.5D Cross-Slice Attention Module for Anisotropic Volumetric Medical Image Segmentation

A large portion of volumetric medical data, especially magnetic resonance imaging (MRI) data, is anisotropic, as the through-plane resolution is typically much lower than the in-plane resolution. Both 3D and purely 2D deep learning-based segmentation methods are deficient in dealing with such volumetric data since the performance of 3D methods suffers when confronting anisotropic data, and 2D methods disregard crucial volumetric information. Insufficient work has been done on 2.5D methods, in which 2D convolution is mainly used in concert with volumetric information. These models focus on learning the relationship across slices, but typically have many parameters to train. We offer a Cross-Slice Attention Module (CSAM) with minimal trainable parameters, which captures information across all the slices in the volume by applying semantic, positional, and slice attention on deep feature maps at different scales. Our extensive experiments using different network architectures and tasks demonstrate the usefulness and generalizability of CSAM. Associated code is available at https://github.com/aL3x-O-o-Hung/CSAM.

Present and Future Innovations in AI and Cardiac MRI

Cardiac MRI is used to diagnose and treat patients with a multitude of cardiovascular diseases. Despite the growth of clinical cardiac MRI, complicated image prescriptions and long acquisition protocols limit the specialty and restrain its impact on the practice of medicine. Artificial intelligence (AI)-the ability to mimic human intelligence in learning and performing tasks-will impact nearly all aspects of MRI. Deep learning (DL) primarily uses an artificial neural network to learn a specific task from example data sets. Self-driving scanners are increasingly available, where AI automatically controls cardiac image prescriptions. These scanners offer faster image collection with higher spatial and temporal resolution, eliminating the need for cardiac triggering or breath holding. In the future, fully automated inline image analysis will most likely provide all contour drawings and initial measurements to the reader. Advanced analysis using radiomic or DL features may provide new insights and information not typically extracted in the current analysis workflow. AI may further help integrate these features with clinical, genetic, wearable-device, and "omics" data to improve patient outcomes. This article presents an overview of AI and its application in cardiac MRI, including in image acquisition, reconstruction, and processing, and opportunities for more personalized cardiovascular care through extraction of novel imaging markers.

D-SAT: dual semantic aggregation transformer with dual attention for medical image segmentation

Objective. Medical image segmentation is significantly essential to assist clinicians in facilitating a quick and accurate diagnoses. However, most of the existing methods are still challenged by the loss of semantic information, blurred boundaries and the huge semantic gap between the encoder and decoder.Approach. To tackle these issues, a dual semantic aggregation transformer with dual attention is proposed for medical image segmentation. Firstly, the dual-semantic feature aggregation module is designed to build a bridge between convolutional neural network (CNN) and Transformer, effectively aggregating CNN's local feature detail ability and Transformer's long-range modeling ability to mitigate semantic information loss. Thereafter, the strip spatial attention mechanism is put forward to alleviate the blurred boundaries during encoding by constructing pixel-level feature relations across CSWin Transformer blocks from different spatial dimensions. Finally, a feature distribution gated attention module is constructed in the skip connection between the encoder and decoder to decrease the large semantic gap by filtering out the noise in low-level semantic information when fusing low-level and high-level semantic features during decoding.Main results. Comprehensive experiments conducted on abdominal multi-organ segmentation, cardiac diagnosis, polyp segmentation and skin lesion segmentation serve to validate the generalization and effectiveness of the proposed dual semantic aggregation transformer with dual attention (D-SAT). The superiority of D-SAT over current state-of-the-art methods is substantiated by both subjective and objective evaluations, revealing its remarkable performance in terms of segmentation accuracy and quality.Significance. The proposed method subtly preserves the local feature details and global context information in medical image segmentation, providing valuable support to improve diagnostic efficiency for clinicians and early disease control for patients. Code is available athttps://github.com/Dxkm/D-SAT.

Combining generative modelling and semi-supervised domain adaptation for whole heart cardiovascular magnetic resonance angiography segmentation

BACKGROUND

Quantification of three-dimensional (3D) cardiac anatomy is important for the evaluation of cardiovascular diseases. Changes in anatomy are indicative of remodeling processes as the heart tissue adapts to disease. Although robust segmentation methods exist for computed tomography angiography (CTA), few methods exist for whole-heart cardiovascular magnetic resonance angiograms (CMRA) which are more challenging due to variable contrast, lower signal to noise ratio and a limited amount of labeled data.

METHODS

Two state-of-the-art unsupervised generative deep learning domain adaptation architectures, generative adversarial networks and variational auto-encoders, were applied to 3D whole heart segmentation of both conventional (n = 20) and high-resolution (n = 45) CMRA (target) images, given segmented CTA (source) images for training. An additional supervised loss function was implemented to improve performance given 10%, 20% and 30% segmented CMRA cases. A fully supervised nn-UNet trained on the given CMRA segmentations was used as the benchmark.

RESULTS

The addition of a small number of segmented CMRA training cases substantially improved performance in both generative architectures in both standard and high-resolution datasets. Compared with the nn-UNet benchmark, the generative methods showed substantially better performance in the case of limited labelled cases. On the standard CMRA dataset, an average 12% (adversarial method) and 10% (variational method) improvement in Dice score was obtained.

CONCLUSIONS

Unsupervised domain-adaptation methods for CMRA segmentation can be boosted by the addition of a small number of supervised target training cases. When only few labelled cases are available, semi-supervised generative modelling is superior to supervised methods.

Dynamic Corrected Split Federated Learning With Homomorphic Encryption for U-Shaped Medical Image Networks

U-shaped networks have become prevalent in various medical image tasks such as segmentation, and restoration. However, most existing U-shaped networks rely on centralized learning which raises privacy concerns. To address these issues, federated learning (FL) and split learning (SL) have been proposed. However, achieving a balance between the local computational cost, model privacy, and parallel training remains a challenge. In this articler, we propose a novel hybrid learning paradigm called Dynamic Corrected Split Federated Learning (DC-SFL) for U-shaped medical image networks. To preserve data privacy, including the input, model parameters, label and output simultaneously, we propose to split the network into three parts hosted by different parties. We propose a Dynamic Weight Correction Strategy (DWCS) to stabilize the training process and avoid the model drift problem due to data heterogeneity. To further enhance privacy protection and establish a trustworthy distributed learning paradigm, we propose to introduce additively homomorphic encryption into the aggregation process of client-side model, which helps prevent potential collusion between parties and provides a better privacy guarantee for our proposed method. The proposed DC-SFL is evaluated on various medical image tasks, and the experimental results demonstrate its effectiveness. In comparison with state-of-the-art distributed learning methods, our method achieves competitive performance.

Dual CNN cross-teaching semi-supervised segmentation network with multi-kernels and global contrastive loss in ACDC

The cross-teaching based on Convolutional Neural Network (CNN) and Transformer has been successful in semi-supervised learning; however, the information interaction between local and global relations ignores the semantic features of the medium scale, and at the same time, the information in the process of feature coding is not fully utilized. To solve these problems, we proposed a new semi-supervised segmentation network. Based on the principle of complementary modeling information of different kernel convolutions, we design a dual CNN cross-supervised network with different kernel sizes under cross-teaching. We introduce global feature contrastive learning and generate contrast samples with the help of dual CNN architecture to make efficient use of coding features. We conducted plenty of experiments on the Automated Cardiac Diagnosis Challenge (ACDC) dataset to evaluate our approach. Our method achieves an average Dice Similarity Coefficient (DSC) of 87.2% and Hausdorff distance ([Formula: see text]) of 6.1 mm on 10% labeled data, which is significantly improved compared with many current popular models. Supervised learning is performed on the labeled data, and dual CNN cross-teaching supervised learning is performed on the unlabeled data. All data would be mapped by the two CNNs to generate features, which are used for contrastive learning to optimize the parameters.

A clinically applicable AI system for diagnosis of congenital heart diseases based on computed tomography images

Congenital heart disease (CHD) is the most common type of birth defect. Without timely detection and treatment, approximately one-third of children with CHD would die in the infant period. However, due to the complicated heart structures, early diagnosis of CHD and its types is quite challenging, even for experienced radiologists. Here, we present an artificial intelligence (AI) system that achieves a comparable performance of human experts in the critical task of classifying 17 categories of CHD types. We collected the first-large CT dataset from three different CT machines, including more than 3750 CHD patients over 14 years. Experimental results demonstrate that it can achieve diagnosis accuracy (86.03%) comparable with junior cardiovascular radiologists (86.27%) in a World Health Organization appointed research and cooperation center in China on most types of CHD, and obtains a higher sensitivity (82.91%) than junior cardiovascular radiologists (76.18%). The accuracy of the combination of our AI system (97.20%) and senior radiologists achieves comparable results to that of junior radiologists and senior radiologists (97.16%) which is the current clinical routine. Our AI system can further provide 3D visualization of hearts to senior radiologists for interpretation and flexible review, surgeons for precise intuition of heart structures, and clinicians for more precise outcome prediction. We demonstrate the potential of our model to be integrated into current clinic practice to improve the diagnosis of CHD globally, especially in regions where experienced radiologists can be scarce.

Development of an Expert-Level Right Ventricular Abnormality Detection Algorithm Based on Deep Learning

PURPOSE

Studies relating to the right ventricle (RV) are inadequate, and specific diagnostic algorithms still need to be improved. This essay is designed to make exploration and verification on an algorithm of deep learning based on imaging and clinical data to detect RV abnormalities.

METHODS

The Automated Cardiac Diagnosis Challenge dataset includes 20 subjects with RV abnormalities (an RV cavity volume which is higher than 110 mL/m2 or RV ejection fraction which is lower than 40%) and 20 normal subjects who suffered from both cardiac MRI. The subjects were separated into training and validation sets in a ratio of 7:3 and were modeled by utilizing a nerve net of deep-learning and six machine-learning algorithms. Eight MRI specialists from multiple centers independently determined whether each subject in the validation group had RV abnormalities. Model performance was evaluated based on the AUC, accuracy, recall, sensitivity and specificity. Furthermore, a preliminary assessment of patient disease risk was performed based on clinical information using a nomogram.

RESULTS

The deep-learning neural network outperformed the other six machine-learning algorithms, with an AUC value of 1 (95% confidence interval: 1-1) on both training group and validation group. This algorithm surpassed most human experts (87.5%). In addition, the nomogram model could evaluate a population with a disease risk of 0.2-0.8.

CONCLUSIONS

A deep-learning algorithm could effectively identify patients with RV abnormalities. This AI algorithm developed specifically for right ventricular abnormalities will improve the detection of right ventricular abnormalities at all levels of care units and facilitate the timely diagnosis and treatment of related diseases. In addition, this study is the first to validate the algorithm's ability to classify RV abnormalities by comparing it with human experts.

SUnet: A multi-organ segmentation network based on multiple attention

Organ segmentation in abdominal or thoracic computed tomography (CT) images plays a crucial role in medical diagnosis as it enables doctors to locate and evaluate organ abnormalities quickly, thereby guiding surgical planning, and aiding treatment decision-making. This paper proposes a novel and efficient medical image segmentation method called SUnet for multi-organ segmentation in the abdomen and thorax. SUnet is a fully attention-based neural network. Firstly, an efficient spatial reduction attention (ESRA) module is introduced not only to extract image features better, but also to reduce overall model parameters, and to alleviate overfitting. Secondly, SUnet's multiple attention-based feature fusion module enables effective cross-scale feature integration. Additionally, an enhanced attention gate (EAG) module is considered by using grouped convolution and residual connections, providing richer semantic features. We evaluate the performance of the proposed model on synapse multiple organ segmentation dataset and automated cardiac diagnostic challenge dataset. SUnet achieves an average Dice of 84.29% and 92.25% on these two datasets, respectively, outperforming other models of similar complexity and size, and achieving state-of-the-art results.

MPVF: 4D Medical Image Inpainting by Multi-Pyramid Voxel Flows

Generatinga detailed 4D medical image usually accompanies with prolonged examination time and increased radiation exposure risk. Modern deep learning solutions have exploited interpolation mechanisms to generate a complete 4D image with fewer 3D volumes. However, existing solutions focus more on 2D-slice information, thus missing the changes on the z-axis. This article tackles the 4D cardiac and lung image interpolation problem by synthesizing 3D volumes directly. Although heart and lung only account for a fraction of chest, they constantly undergo periodical motions of varying magnitudes in contrast to the rest of the chest volume, which is more stationary. This poses big challenges to existing models. In order to handle various magnitudes of motions, we propose a Multi-Pyramid Voxel Flows (MPVF) model that takes multiple multi-scale voxel flows into account. This renders our generation network rich information during interpolation, both globally and regionally. Focusing on periodic medical imaging, MPVF takes the maximal and the minimal phases of an organ motion cycle as inputs and can restore a 3D volume at any time point in between. MPVF is featured by a Bilateral Voxel Flow (BVF) module for generating multi-pyramid voxel flows in an unsupervised manner and a Pyramid Fusion (PyFu) module for fusing multiple pyramids of 3D volumes. The model is validated to outperform the state-of-the-art model in several indices with significantly less synthesis time.

Semi-Supervised Representation Learning for Segmentation on Medical Volumes and Sequences

Benefiting from the massive labeled samples, deep learning-based segmentation methods have achieved great success for two dimensional natural images. However, it is still a challenging task to segment high dimensional medical volumes and sequences, due to the considerable efforts for clinical expertise to make large scale annotations. Self/semi-supervised learning methods have been shown to improve the performance by exploiting unlabeled data. However, they are still lack of mining local semantic discrimination and exploitation of volume/sequence structures. In this work, we propose a semi-supervised representation learning method with two novel modules to enhance the features in the encoder and decoder, respectively. For the encoder, based on the continuity between slices/frames and the common spatial layout of organs across subjects, we propose an asymmetric network with an attention-guided predictor to enable prediction between feature maps of different slices of unlabeled data. For the decoder, based on the semantic consistency between labeled data and unlabeled data, we introduce a novel semantic contrastive learning to regularize the feature maps in the decoder. The two parts are trained jointly with both labeled and unlabeled volumes/sequences in a semi-supervised manner. When evaluated on three benchmark datasets of medical volumes and sequences, our model outperforms existing methods with a large margin of 7.3% DSC on ACDC, 6.5% on Prostate, and 3.2% on CAMUS when only a few labeled data is available. Further, results on the M&M dataset show that the proposed method yields improvement without using any domain adaption techniques for data from unknown domain. Intensive evaluations reveal the effectiveness of representation mining, and superiority on performance of our method. The code is available at https://github.com/CcchenzJ/BootstrapRepresentation.

Genome-wide association analysis of left ventricular imaging-derived phenotypes identifies 72 risk loci and yields genetic insights into hypertrophic cardiomyopathy

Left ventricular regional wall thickness (LVRWT) is an independent predictor of morbidity and mortality in cardiovascular diseases (CVDs). To identify specific genetic influences on individual LVRWT, we established a novel deep learning algorithm to calculate 12 LVRWTs accurately in 42,194 individuals from the UK Biobank with cardiac magnetic resonance (CMR) imaging. Genome-wide association studies of CMR-derived 12 LVRWTs identified 72 significant genetic loci associated with at least one LVRWT phenotype (P < 5 × 10-8), which were revealed to actively participate in heart development and contraction pathways. Significant causal relationships were observed between the LVRWT traits and hypertrophic cardiomyopathy (HCM) using genetic correlation and Mendelian randomization analyses (P < 0.01). The polygenic risk score of inferoseptal LVRWT at end systole exhibited a notable association with incident HCM, facilitating the identification of high-risk individuals. The findings yield insights into the genetic determinants of LVRWT phenotypes and shed light on the biological basis for HCM etiology.

Cross-Parallel Transformer: Parallel ViT for Medical Image Segmentation

Medical image segmentation primarily utilizes a hybrid model consisting of a Convolutional Neural Network and sequential Transformers. The latter leverage multi-head self-attention mechanisms to achieve comprehensive global context modelling. However, despite their success in semantic segmentation, the feature extraction process is inefficient and demands more computational resources, which hinders the network's robustness. To address this issue, this study presents two innovative methods: PTransUNet (PT model) and C-PTransUNet (C-PT model). The C-PT module refines the Vision Transformer by substituting a sequential design with a parallel one. This boosts the feature extraction capabilities of Multi-Head Self-Attention via self-correlated feature attention and channel feature interaction, while also streamlining the Feed-Forward Network to lower computational demands. On the Synapse public dataset, the PT and C-PT models demonstrate improvements in DSC accuracy by 0.87% and 3.25%, respectively, in comparison with the baseline model. As for the parameter count and FLOPs, the PT model aligns with the baseline model. In contrast, the C-PT model shows a decrease in parameter count by 29% and FLOPs by 21.4% relative to the baseline model. The proposed segmentation models in this study exhibit benefits in both accuracy and efficiency.

Enhancing Medical Image Denoising with Innovative Teacher-Student Model-Based Approaches for Precision Diagnostics

The realm of medical imaging is a critical frontier in precision diagnostics, where the clarity of the image is paramount. Despite advancements in imaging technology, noise remains a pervasive challenge that can obscure crucial details and impede accurate diagnoses. Addressing this, we introduce a novel teacher-student network model that leverages the potency of our bespoke NoiseContextNet Block to discern and mitigate noise with unprecedented precision. This innovation is coupled with an iterative pruning technique aimed at refining the model for heightened computational efficiency without compromising the fidelity of denoising. We substantiate the superiority and effectiveness of our approach through a comprehensive suite of experiments, showcasing significant qualitative enhancements across a multitude of medical imaging modalities. The visual results from a vast array of tests firmly establish our method's dominance in producing clearer, more reliable images for diagnostic purposes, thereby setting a new benchmark in medical image denoising.

Enhancing the Super-Resolution of Medical Images: Introducing the Deep Residual Feature Distillation Channel Attention Network for Optimized Performance and Efficiency

In the advancement of medical image super-resolution (SR), the Deep Residual Feature Distillation Channel Attention Network (DRFDCAN) marks a significant step forward. This work presents DRFDCAN, a model that innovates traditional SR approaches by introducing a channel attention block that is tailored for high-frequency features-crucial for the nuanced details in medical diagnostics-while streamlining the network structure for enhanced computational efficiency. DRFDCAN's architecture adopts a residual-within-residual design to facilitate faster inference and reduce memory demands without compromising the integrity of the image reconstruction. This design strategy, combined with an innovative feature extraction method that emphasizes the utility of the initial layer features, allows for improved image clarity and is particularly effective in optimizing the peak signal-to-noise ratio (PSNR). The proposed work redefines efficiency in SR models, outperforming established frameworks like RFDN by improving model compactness and accelerating inference. The meticulous crafting of a feature extractor that effectively captures edge and texture information exemplifies the model's capacity to render detailed images, necessary for accurate medical analysis. The implications of this study are two-fold: it presents a viable solution for deploying SR technology in real-time medical applications, and it sets a precedent for future models that address the delicate balance between computational efficiency and high-fidelity image reconstruction. This balance is paramount in medical applications where the clarity of images can significantly influence diagnostic outcomes. The DRFDCAN model thus stands as a transformative contribution to the field of medical image super-resolution.

Addressing image misalignments in multi-parametric prostate MRI for enhanced computer-aided diagnosis of prostate cancer

Prostate cancer (PCa) diagnosis on multi-parametric magnetic resonance images (MRI) requires radiologists with a high level of expertise. Misalignments between the MRI sequences can be caused by patient movement, elastic soft-tissue deformations, and imaging artifacts. They further increase the complexity of the task prompting radiologists to interpret the images. Recently, computer-aided diagnosis (CAD) tools have demonstrated potential for PCa diagnosis typically relying on complex co-registration of the input modalities. However, there is no consensus among research groups on whether CAD systems profit from using registration. Furthermore, alternative strategies to handle multi-modal misalignments have not been explored so far. Our study introduces and compares different strategies to cope with image misalignments and evaluates them regarding to their direct effect on diagnostic accuracy of PCa. In addition to established registration algorithms, we propose 'misalignment augmentation' as a concept to increase CAD robustness. As the results demonstrate, misalignment augmentations can not only compensate for a complete lack of registration, but if used in conjunction with registration, also improve the overall performance on an independent test set.

Compete to Win: Enhancing Pseudo Labels for Barely-Supervised Medical Image Segmentation

This study investigates barely-supervised medical image segmentation where only few labeled data, i.e., single-digit cases are available. We observe the key limitation of the existing state-of-the-art semi-supervised solution cross pseudo supervision is the unsatisfactory precision of foreground classes, leading to a degenerated result under barely-supervised learning. In this paper, we propose a novel Compete-to-Win method (ComWin) to enhance the pseudo label quality. In contrast to directly using one model's predictions as pseudo labels, our key idea is that high-quality pseudo labels should be generated by comparing multiple confidence maps produced by different networks to select the most confident one (a compete-to-win strategy). To further refine pseudo labels at near-boundary areas, an enhanced version of ComWin, namely, ComWin + , is proposed by integrating a boundary-aware enhancement module. Experiments show that our method can achieve the best performance on three public medical image datasets for cardiac structure segmentation, pancreas segmentation and colon tumor segmentation, respectively. The source code is now available at https://github.com/Huiimin5/comwin.

Cardiovascular magnetic resonance (CMR) and positron emission tomography (PET) imaging in the diagnosis and follow-up of patients with acute myocarditis and chronic inflammatory cardiomyopathy : A review paper with practical recommendations on behalf of the European Society of Cardiovascular Radiology (ESCR)

Advanced cardiac imaging techniques such as cardiovascular magnetic resonance (CMR) and positron emission tomography (PET) are widely used in clinical practice in patients with acute myocarditis and chronic inflammatory cardiomyopathies (I-CMP). We aimed to provide a review article with practical recommendations from the European Society of Cardiovascular Radiology (ESCR), in order to guide physicians in the use and interpretation of CMR and PET in clinical practice both for acute myocarditis and follow-up in chronic forms of I-CMP.

Optimal MRI undersampling patterns for ultimate benefit of medical vision tasks

Compressed sensing is commonly concerned with optimizing the image quality after a partial undersampling of the measurable k-space to accelerate MRI. In this article, we propose to change the focus from the quality of the reconstructed image to the quality of the downstream image analysis outcome. Specifically, we propose to optimize the patterns according to how well a sought-after pathology could be detected or localized in the reconstructed images. We find the optimal undersampling patterns in k-space that maximize target value functions of interest in commonplace medical vision problems (reconstruction, segmentation, and classification) and propose a new iterative gradient sampling routine universally suitable for these tasks. We validate the proposed MRI acceleration paradigm on three classical medical datasets, demonstrating a noticeable improvement of the target metrics at the high acceleration factors (for the segmentation problem at ×16 acceleration, we report up to 12% improvement in Dice score over the other undersampling patterns).

Automatic right ventricular segmentation for cine cardiac magnetic resonance images based on a new deep atlas network

BACKGROUND

The high morbidity and mortality of heart disease present a significant threat to human health. The development of methods for the quick and accurate diagnosis of heart diseases, enabling their effective treatment, has become a key issue of concern. Right ventricular (RV) segmentation from cine cardiac magnetic resonance (CMR) images plays a significant role in evaluating cardiac function for clinical diagnosis and prognosis. However, due to the complex structure of the RV, traditional segmentation methods are ineffective for RV segmentation.

PURPOSE

In this paper, we propose a new deep atlas network to improve the learning efficiency and segmentation accuracy of a deep learning network by integrating multi-atlas.

METHODS

First, a dense multi-scale U-net (DMU-net) is presented to acquire transformation parameters from atlas images to target images. The transformation parameters map the atlas image labels to the target image labels. Second, using a spatial transformation layer, the atlas images are deformed based on these parameters. Finally, the network is optimized by backpropagation with two loss functions where the mean squared error function (MSE) is used to measure the similarity of the input images and transformed images. Further, the Dice metric (DM) is used to quantify the overlap between the predicted contours and the ground truth. In our experiments, 15 datasets are used in testing, and 20 cine CMR images are selected as atlas.

RESULTS

The mean values and standard deviations for the DM and Hausdorff distance are 0.871 and 4.67 mm, 0.104 and 2.528 mm, respectively. The correlation coefficients of endo-diastolic volume, endo-systolic volume, ejection fraction, and stroke volume are 0.984, 0.926, 0.980, and 0.991, respectively, and the mean differences between all of the mentioned parameters are 3.2, -1.7, 0.02, and 4.9, respectively. Most of these differences are within the allowable range of 95%, indicating that the results are acceptable and show good consistency. The segmentation results obtained in this method are compared with those obtained by other methods that provide satisfactory performance. The other methods provide better segmentation effects at the base, but either no segmentation or the wrong segmentation at the top, which demonstrate that the deep atlas network can improve top-area segmentation accuracy.

CONCLUSION

Our results indicate that the proposed method can achieve better segmentation results than the previous methods, with both high relevance and consistency, and has the potential for clinical application.