1
|
Guan H, Yap PT, Bozoki A, Liu M. Federated learning for medical image analysis: A survey. Pattern Recognit 2024; 151:110424. [PMID: 38559674 PMCID: PMC10976951 DOI: 10.1016/j.patcog.2024.110424] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Machine learning in medical imaging often faces a fundamental dilemma, namely, the small sample size problem. Many recent studies suggest using multi-domain data pooled from different acquisition sites/centers to improve statistical power. However, medical images from different sites cannot be easily shared to build large datasets for model training due to privacy protection reasons. As a promising solution, federated learning, which enables collaborative training of machine learning models based on data from different sites without cross-site data sharing, has attracted considerable attention recently. In this paper, we conduct a comprehensive survey of the recent development of federated learning methods in medical image analysis. We have systematically gathered research papers on federated learning and its applications in medical image analysis published between 2017 and 2023. Our search and compilation were conducted using databases from IEEE Xplore, ACM Digital Library, Science Direct, Springer Link, Web of Science, Google Scholar, and PubMed. In this survey, we first introduce the background of federated learning for dealing with privacy protection and collaborative learning issues. We then present a comprehensive review of recent advances in federated learning methods for medical image analysis. Specifically, existing methods are categorized based on three critical aspects of a federated learning system, including client end, server end, and communication techniques. In each category, we summarize the existing federated learning methods according to specific research problems in medical image analysis and also provide insights into the motivations of different approaches. In addition, we provide a review of existing benchmark medical imaging datasets and software platforms for current federated learning research. We also conduct an experimental study to empirically evaluate typical federated learning methods for medical image analysis. This survey can help to better understand the current research status, challenges, and potential research opportunities in this promising research field.
Collapse
Affiliation(s)
- Hao Guan
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| | - Pew-Thian Yap
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| | - Andrea Bozoki
- Department of Neurology, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| | - Mingxia Liu
- Department of Radiology and Biomedical Research Imaging Center, University of North Carolina at Chapel Hill, Chapel Hill, NC 27599, USA
| |
Collapse
|
2
|
Chang Q, Wang Y. Structure-aware independently trained multi-scale registration network for cardiac images. Med Biol Eng Comput 2024; 62:1795-1808. [PMID: 38381202 DOI: 10.1007/s11517-024-03039-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 01/31/2024] [Indexed: 02/22/2024]
Abstract
Image registration is a primary task in various medical image analysis applications. However, cardiac image registration is difficult due to the large non-rigid deformation of the heart and the complex anatomical structure. This paper proposes a structure-aware independently trained multi-scale registration network (SIMReg) to address this challenge. Using image pairs of different resolutions, independently train each registration network to extract image features of large deformation image pairs at different resolutions. In the testing stage, the large deformation registration is decomposed into a multi-scale registration process, and the deformation fields of different resolutions are fused by a step-by-step deformation method, thus solving the difficulty of directly processing large deformation. Meanwhile, the targeted introduction of MIND (modality independent neighborhood descriptor) structural features to guide network training enhances the registration of cardiac structural contours and improves the registration effect of local details. Experiments were carried out on the open cardiac dataset ACDC (automated cardiac diagnosis challenge), and the average Dice value of the experimental results of the proposed method was 0.833. Comparative experiments showed that the proposed SIMReg could better solve the problem of heart image registration and achieve a better registration effect on cardiac images.
Collapse
Affiliation(s)
- Qing Chang
- School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China
| | - Yaqi Wang
- School of Information Science and Engineering, East China University of Science and Technology, Shanghai, China.
| |
Collapse
|
3
|
Petmezas G, Papageorgiou VE, Vassilikos V, Pagourelias E, Tsaklidis G, Katsaggelos AK, Maglaveras N. Recent advancements and applications of deep learning in heart failure: Α systematic review. Comput Biol Med 2024; 176:108557. [PMID: 38728995 DOI: 10.1016/j.compbiomed.2024.108557] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Revised: 04/12/2024] [Accepted: 05/05/2024] [Indexed: 05/12/2024]
Abstract
BACKGROUND Heart failure (HF), a global health challenge, requires innovative diagnostic and management approaches. The rapid evolution of deep learning (DL) in healthcare necessitates a comprehensive review to evaluate these developments and their potential to enhance HF evaluation, aligning clinical practices with technological advancements. OBJECTIVE This review aims to systematically explore the contributions of DL technologies in the assessment of HF, focusing on their potential to improve diagnostic accuracy, personalize treatment strategies, and address the impact of comorbidities. METHODS A thorough literature search was conducted across four major electronic databases: PubMed, Scopus, Web of Science and IEEE Xplore, yielding 137 articles that were subsequently categorized into five primary application areas: cardiovascular disease (CVD) classification, HF detection, image analysis, risk assessment, and other clinical analyses. The selection criteria focused on studies utilizing DL algorithms for HF assessment, not limited to HF detection but extending to any attempt in analyzing and interpreting HF-related data. RESULTS The analysis revealed a notable emphasis on CVD classification and HF detection, with DL algorithms showing significant promise in distinguishing between affected individuals and healthy subjects. Furthermore, the review highlights DL's capacity to identify underlying cardiomyopathies and other comorbidities, underscoring its utility in refining diagnostic processes and tailoring treatment plans to individual patient needs. CONCLUSIONS This review establishes DL as a key innovation in HF management, highlighting its role in advancing diagnostic accuracy and personalized care. The insights provided advocate for the integration of DL in clinical settings and suggest directions for future research to enhance patient outcomes in HF care.
Collapse
Affiliation(s)
- Georgios Petmezas
- 2nd Department of Obstetrics and Gynecology, Medical School, Aristotle University of Thessaloniki, Thessaloniki, Greece; Centre for Research and Technology Hellas, Thessaloniki, Greece.
| | | | - Vasileios Vassilikos
- 3rd Department of Cardiology, Medical School, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Efstathios Pagourelias
- 3rd Department of Cardiology, Medical School, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - George Tsaklidis
- Department of Mathematics, Aristotle University of Thessaloniki, Thessaloniki, Greece
| | - Aggelos K Katsaggelos
- Department of Electrical and Computer Engineering, Northwestern University, Evanston, IL, USA
| | - Nicos Maglaveras
- 2nd Department of Obstetrics and Gynecology, Medical School, Aristotle University of Thessaloniki, Thessaloniki, Greece
| |
Collapse
|
4
|
Peng J, Wang P, Pedersoli M, Desrosiers C. Boundary-aware information maximization for self-supervised medical image segmentation. Med Image Anal 2024; 94:103150. [PMID: 38574545 DOI: 10.1016/j.media.2024.103150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 02/24/2024] [Accepted: 03/20/2024] [Indexed: 04/06/2024]
Abstract
Self-supervised representation learning can boost the performance of a pre-trained network on downstream tasks for which labeled data is limited. A popular method based on this paradigm, known as contrastive learning, works by constructing sets of positive and negative pairs from the data, and then pulling closer the representations of positive pairs while pushing apart those of negative pairs. Although contrastive learning has been shown to improve performance in various classification tasks, its application to image segmentation has been more limited. This stems in part from the difficulty of defining positive and negative pairs for dense feature maps without having access to pixel-wise annotations. In this work, we propose a novel self-supervised pre-training method that overcomes the challenges of contrastive learning in image segmentation. Our method leverages Information Invariant Clustering (IIC) as an unsupervised task to learn a local representation of images in the decoder of a segmentation network, but addresses three important drawbacks of this approach: (i) the difficulty of optimizing the loss based on mutual information maximization; (ii) the lack of clustering consistency for different random transformations of the same image; (iii) the poor correspondence of clusters obtained by IIC with region boundaries in the image. Toward this goal, we first introduce a regularized mutual information maximization objective that encourages the learned clusters to be balanced and consistent across different image transformations. We also propose a boundary-aware loss based on cross-correlation, which helps the learned clusters to be more representative of important regions in the image. Compared to contrastive learning applied in dense features, our method does not require computing positive and negative pairs and also enhances interpretability through the visualization of learned clusters. Comprehensive experiments involving four different medical image segmentation tasks reveal the high effectiveness of our self-supervised representation learning method. Our results show the proposed method to outperform by a large margin several state-of-the-art self-supervised and semi-supervised approaches for segmentation, reaching a performance close to full supervision with only a few labeled examples.
Collapse
Affiliation(s)
- Jizong Peng
- ETS Montréal, 1100 Notre-Dame St W, Montreal H3C 1K3, QC, Canada.
| | - Ping Wang
- ETS Montréal, 1100 Notre-Dame St W, Montreal H3C 1K3, QC, Canada
| | - Marco Pedersoli
- ETS Montréal, 1100 Notre-Dame St W, Montreal H3C 1K3, QC, Canada
| | | |
Collapse
|
5
|
Onnis C, van Assen M, Muscogiuri E, Muscogiuri G, Gershon G, Saba L, De Cecco CN. The Role of Artificial Intelligence in Cardiac Imaging. Radiol Clin North Am 2024; 62:473-488. [PMID: 38553181 DOI: 10.1016/j.rcl.2024.01.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/02/2024]
Abstract
Artificial intelligence (AI) is having a significant impact in medical imaging, advancing almost every aspect of the field, from image acquisition and postprocessing to automated image analysis with outreach toward supporting decision making. Noninvasive cardiac imaging is one of the main and most exciting fields for AI development. The aim of this review is to describe the main applications of AI in cardiac imaging, including CT and MR imaging, and provide an overview of recent advancements and available clinical applications that can improve clinical workflow, disease detection, and prognostication in cardiac disease.
Collapse
Affiliation(s)
- Carlotta Onnis
- Translational Laboratory for Cardiothoracic Imaging and Artificial Intelligence, Department of Radiology and Imaging Sciences, Emory University, 100 Woodruff Circle, Atlanta, GA 30322, USA; Department of Radiology, Azienda Ospedaliero Universitaria (A.O.U.), di Cagliari-Polo di Monserrato, SS 554 km 4,500 Monserrato, Cagliari 09042, Italy. https://twitter.com/CarlottaOnnis
| | - Marly van Assen
- Translational Laboratory for Cardiothoracic Imaging and Artificial Intelligence, Department of Radiology and Imaging Sciences, Emory University, 100 Woodruff Circle, Atlanta, GA 30322, USA. https://twitter.com/marly_van_assen
| | - Emanuele Muscogiuri
- Translational Laboratory for Cardiothoracic Imaging and Artificial Intelligence, Department of Radiology and Imaging Sciences, Emory University, 100 Woodruff Circle, Atlanta, GA 30322, USA; Division of Thoracic Imaging, Department of Radiology, University Hospitals Leuven, Herestraat 49, Leuven 3000, Belgium
| | - Giuseppe Muscogiuri
- Department of Diagnostic and Interventional Radiology, Papa Giovanni XXIII Hospital, Piazza OMS, 1, Bergamo BG 24127, Italy. https://twitter.com/GiuseppeMuscog
| | - Gabrielle Gershon
- Translational Laboratory for Cardiothoracic Imaging and Artificial Intelligence, Department of Radiology and Imaging Sciences, Emory University, 100 Woodruff Circle, Atlanta, GA 30322, USA. https://twitter.com/gabbygershon
| | - Luca Saba
- Department of Radiology, Azienda Ospedaliero Universitaria (A.O.U.), di Cagliari-Polo di Monserrato, SS 554 km 4,500 Monserrato, Cagliari 09042, Italy. https://twitter.com/lucasabaITA
| | - Carlo N De Cecco
- Translational Laboratory for Cardiothoracic Imaging and Artificial Intelligence, Department of Radiology and Imaging Sciences, Emory University, 100 Woodruff Circle, Atlanta, GA 30322, USA; Division of Cardiothoracic Imaging, Department of Radiology and Imaging Sciences, Emory University, Emory University Hospital, 1365 Clifton Road Northeast, Suite AT503, Atlanta, GA 30322, USA.
| |
Collapse
|
6
|
Zhang T, Wei D, Zhu M, Gu S, Zheng Y. Self-supervised learning for medical image data with anatomy-oriented imaging planes. Med Image Anal 2024; 94:103151. [PMID: 38527405 DOI: 10.1016/j.media.2024.103151] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/07/2023] [Revised: 12/29/2023] [Accepted: 03/20/2024] [Indexed: 03/27/2024]
Abstract
Self-supervised learning has emerged as a powerful tool for pretraining deep networks on unlabeled data, prior to transfer learning of target tasks with limited annotation. The relevance between the pretraining pretext and target tasks is crucial to the success of transfer learning. Various pretext tasks have been proposed to utilize properties of medical image data (e.g., three dimensionality), which are more relevant to medical image analysis than generic ones for natural images. However, previous work rarely paid attention to data with anatomy-oriented imaging planes, e.g., standard cardiac magnetic resonance imaging views. As these imaging planes are defined according to the anatomy of the imaged organ, pretext tasks effectively exploiting this information can pretrain the networks to gain knowledge on the organ of interest. In this work, we propose two complementary pretext tasks for this group of medical image data based on the spatial relationship of the imaging planes. The first is to learn the relative orientation between the imaging planes and implemented as regressing their intersecting lines. The second exploits parallel imaging planes to regress their relative slice locations within a stack. Both pretext tasks are conceptually straightforward and easy to implement, and can be combined in multitask learning for better representation learning. Thorough experiments on two anatomical structures (heart and knee) and representative target tasks (semantic segmentation and classification) demonstrate that the proposed pretext tasks are effective in pretraining deep networks for remarkably boosted performance on the target tasks, and superior to other recent approaches.
Collapse
Affiliation(s)
- Tianwei Zhang
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Dong Wei
- Jarvis Research Center, Tencent YouTu Lab, Shenzhen 518057, China
| | - Mengmeng Zhu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Shi Gu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu 611731, China.
| | - Yefeng Zheng
- Jarvis Research Center, Tencent YouTu Lab, Shenzhen 518057, China
| |
Collapse
|
7
|
Lyu J, Wang S, Tian Y, Zou J, Dong S, Wang C, Aviles-Rivero AI, Qin J. STADNet: Spatial-Temporal Attention-Guided Dual-Path Network for cardiac cine MRI super-resolution. Med Image Anal 2024; 94:103142. [PMID: 38492252 DOI: 10.1016/j.media.2024.103142] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2023] [Revised: 02/29/2024] [Accepted: 03/05/2024] [Indexed: 03/18/2024]
Abstract
Cardiac cine magnetic resonance imaging (MRI) is a commonly used clinical tool for evaluating cardiac function and morphology. However, its diagnostic accuracy may be compromised by the low spatial resolution. Current methods for cine MRI super-resolution reconstruction still have limitations. They typically rely on 3D convolutional neural networks or recurrent neural networks, which may not effectively capture long-range or non-local features due to their limited receptive fields. Optical flow estimators are also commonly used to align neighboring frames, which may cause information loss and inaccurate motion estimation. Additionally, pre-warping strategies may involve interpolation, leading to potential loss of texture details and complicated anatomical structures. To overcome these challenges, we propose a novel Spatial-Temporal Attention-Guided Dual-Path Network (STADNet) for cardiac cine MRI super-resolution. We utilize transformers to model long-range dependencies in cardiac cine MR images and design a cross-frame attention module in the location-aware spatial path, which enhances the spatial details of the current frame by using complementary information from neighboring frames. We also introduce a recurrent flow-enhanced attention module in the motion-aware temporal path that exploits the correlation between cine MRI frames and extracts the motion information of the heart. Experimental results demonstrate that STADNet outperforms SOTA approaches and has significant potential for clinical practice.
Collapse
Affiliation(s)
- Jun Lyu
- School of Computer and Control Engineering, Yantai University, Yantai, China
| | - Shuo Wang
- School of Basic Medical Sciences, Fudan University, Shanghai, China
| | - Yapeng Tian
- Department of Computer Science, The University of Texas at Dallas, Richardson, TX, USA
| | - Jing Zou
- Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University, Hong Kong
| | - Shunjie Dong
- College of Information Science & Electronic Engineering, Zhejiang University, Hangzhou, China
| | - Chengyan Wang
- Human Phenome Institute, Fudan University, Shanghai, China.
| | - Angelica I Aviles-Rivero
- Department of Applied Mathematics and Theoretical Physics, University of Cambridge, Cambridge, UK
| | - Jing Qin
- Centre for Smart Health, School of Nursing, The Hong Kong Polytechnic University, Hong Kong
| |
Collapse
|
8
|
Baccouch W, Oueslati S, Solaiman B, Lahidheb D, Labidi S. Automatic left ventricle volume and mass quantification from 2D cine-MRI: Investigating papillary muscle influence. Med Eng Phys 2024; 127:104162. [PMID: 38692762 DOI: 10.1016/j.medengphy.2024.104162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2023] [Revised: 03/01/2024] [Accepted: 03/27/2024] [Indexed: 05/03/2024]
Abstract
OBJECTIVE Early detection of cardiovascular diseases is based on accurate quantification of the left ventricle (LV) function parameters. In this paper, we propose a fully automatic framework for LV volume and mass quantification from 2D-cine MR images already segmented using U-Net. METHODS The general framework consists of three main steps: Data preparation including automatic LV localization using a convolution neural network (CNN) and application of morphological operations to exclude papillary muscles from the LV cavity. The second step consists in automatically extracting the LV contours using U-Net architecture. Finally, by integrating temporal information which is manifested by a spatial motion of myocytes as a third dimension, we calculated LV volume, LV ejection fraction (LVEF) and left ventricle mass (LVM). Based on these parameters, we detected and quantified cardiac contraction abnormalities using Python software. RESULTS CNN was trained with 35 patients and tested on 15 patients from the ACDC database with an accuracy of 99,15 %. U-Net architecture was trained using ACDC database and evaluated using local dataset with a Dice similarity coefficient (DSC) of 99,78 % and a Hausdorff Distance (HD) of 4.468 mm (p < 0,001). Quantification results showed a strong correlation with physiological measures with a Pearson correlation coefficient (PCC) of 0,991 for LV volume, 0.962 for LVEF, 0.98 for stroke volume (SV) and 0.923 for LVM after pillars' elimination. Clinically, our method allows regional and accurate identification of pathological myocardial segments and can serve as a diagnostic aid tool of cardiac contraction abnormalities. CONCLUSION Experimental results prove the usefulness of the proposed method for LV volume and function quantification and verify its potential clinical applicability.
Collapse
Affiliation(s)
- Wafa Baccouch
- University of Tunis El Manar, Higher institute of Medical Technologies of Tunis, Research laboratory of Biophysics and Medical Technologies LR13ES07, Tunis, 1006, Tunisia.
| | - Sameh Oueslati
- University of Tunis El Manar, Higher institute of Medical Technologies of Tunis, Research laboratory of Biophysics and Medical Technologies LR13ES07, Tunis, 1006, Tunisia
| | - Basel Solaiman
- Image & Information Processing Department (iTi), IMT-Atlantique, Technopôle Brest Iroise CS 83818, 29238, Brest Cedex, France
| | - Dhaker Lahidheb
- University of Tunis El Manar, Faculty of Medicine of Tunis, Tunis, Tunisia; Department of Cardiology, Military Hospital of Tunis, Tunis, Tunisia
| | - Salam Labidi
- University of Tunis El Manar, Higher institute of Medical Technologies of Tunis, Research laboratory of Biophysics and Medical Technologies LR13ES07, Tunis, 1006, Tunisia
| |
Collapse
|
9
|
Li B, Xu Y, Wang Y, Li L, Zhang B. The student-teacher framework guided by self-training and consistency regularization for semi-supervised medical image segmentation. PLoS One 2024; 19:e0300039. [PMID: 38648206 PMCID: PMC11034649 DOI: 10.1371/journal.pone.0300039] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Accepted: 02/20/2024] [Indexed: 04/25/2024] Open
Abstract
Due to the high suitability of semi-supervised learning for medical image segmentation, a plethora of valuable research has been conducted and has achieved noteworthy success in this field. However, many approaches tend to confine their focus to a singular semi-supervised framework, thereby overlooking the potential enhancements in segmentation performance offered by integrating several frameworks. In this paper, we propose a novel semi-supervised framework named Pesudo-Label Mean Teacher (PLMT), which synergizes the self-training pipeline with pseudo-labeling and consistency regularization techniques. In particular, we integrate the student-teacher structure with consistency loss into the self-training pipeline to facilitate a mutually beneficial enhancement between the two methods. This structure not only generates remarkably accurate pseudo-labels for the self-training pipeline but also furnishes additional pseudo-label supervision for the student-teacher framework. Moreover, to explore the impact of different semi-supervised losses on the segmentation performance of the PLMT framework, we introduce adaptive loss weights. The PLMT could dynamically adjust the weights of different semi-supervised losses during the training process. Extension experiments on three public datasets demonstrate that our framework achieves the best performance and outperforms the other five semi-supervised methods. The PLMT is an initial exploration of the framework that melds the self-training pipeline with consistency regularization and offers a comparatively innovative perspective in semi-supervised image segmentation.
Collapse
Affiliation(s)
- Boliang Li
- Department of control science and engineering, Harbin Institute of Technology, Harbin, Heilongjiang, China
| | - Yaming Xu
- Department of control science and engineering, Harbin Institute of Technology, Harbin, Heilongjiang, China
| | - Yan Wang
- Department of control science and engineering, Harbin Institute of Technology, Harbin, Heilongjiang, China
| | - Luxiu Li
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Bo Zhang
- Sergeant schools of Army Academy of Armored Forces, Changchun, Jilin, China
| |
Collapse
|
10
|
Aghapanah H, Rasti R, Kermani S, Tabesh F, Banaem HY, Aliakbar HP, Sanei H, Segars WP. CardSegNet: An adaptive hybrid CNN-vision transformer model for heart region segmentation in cardiac MRI. Comput Med Imaging Graph 2024; 115:102382. [PMID: 38640619 DOI: 10.1016/j.compmedimag.2024.102382] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/01/2023] [Revised: 03/08/2024] [Accepted: 04/10/2024] [Indexed: 04/21/2024]
Abstract
Cardiovascular MRI (CMRI) is a non-invasive imaging technique adopted for assessing the blood circulatory system's structure and function. Precise image segmentation is required to measure cardiac parameters and diagnose abnormalities through CMRI data. Because of anatomical heterogeneity and image variations, cardiac image segmentation is a challenging task. Quantification of cardiac parameters requires high-performance segmentation of the left ventricle (LV), right ventricle (RV), and left ventricle myocardium from the background. The first proposed solution here is to manually segment the regions, which is a time-consuming and error-prone procedure. In this context, many semi- or fully automatic solutions have been proposed recently, among which deep learning-based methods have revealed high performance in segmenting regions in CMRI data. In this study, a self-adaptive multi attention (SMA) module is introduced to adaptively leverage multiple attention mechanisms for better segmentation. The convolutional-based position and channel attention mechanisms with a patch tokenization-based vision transformer (ViT)-based attention mechanism in a hybrid and end-to-end manner are integrated into the SMA. The CNN- and ViT-based attentions mine the short- and long-range dependencies for more precise segmentation. The SMA module is applied in an encoder-decoder structure with a ResNet50 backbone named CardSegNet. Furthermore, a deep supervision method with multi-loss functions is introduced to the CardSegNet optimizer to reduce overfitting and enhance the model's performance. The proposed model is validated on the ACDC2017 (n=100), M&Ms (n=321), and a local dataset (n=22) using the 10-fold cross-validation method with promising segmentation results, demonstrating its outperformance versus its counterparts.
Collapse
Affiliation(s)
- Hamed Aghapanah
- School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan, Iran
| | - Reza Rasti
- Department of Biomedical Engineering, Faculty of Engineering, University of Isfahan, Isfahan, Iran; Department of Biomedical Engineering, Duke University, Durham, NC 27708, USA.
| | - Saeed Kermani
- School of Advanced Technologies in Medicine, Isfahan University of Medical Sciences, Isfahan, Iran.
| | - Faezeh Tabesh
- Cardiovascular Research Institute, Isfahan University of Medical Sciences, Isfahan, Iran
| | - Hossein Yousefi Banaem
- Skull Base Research Center, Loghman Hakim Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Hamidreza Pour Aliakbar
- Rajaie Cardiovascular Medical and Research Center, Iran University of Medical Sciences, Tehran, Iran
| | - Hamid Sanei
- Cardiovascular Research Institute, Isfahan University of Medical Sciences, Isfahan, Iran
| | - William Paul Segars
- Department of Biomedical Engineering, Faculty of Engineering, University of Isfahan, Isfahan, Iran
| |
Collapse
|
11
|
Jiang N, Zhang Y, Li Q, Fu X, Fang D. A cardiac MRI motion artifact reduction method based on edge enhancement network. Phys Med Biol 2024; 69:095004. [PMID: 38537303 DOI: 10.1088/1361-6560/ad3884] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Accepted: 03/26/2024] [Indexed: 04/16/2024]
Abstract
Cardiac magnetic resonance imaging (MRI) usually requires a long acquisition time. The movement of the patients during MRI acquisition will produce image artifacts. Previous studies have shown that clear MR image texture edges are of great significance for pathological diagnosis. In this paper, a motion artifact reduction method for cardiac MRI based on edge enhancement network is proposed. Firstly, the four-plane normal vector adaptive fractional differential mask is applied to extract the edge features of blurred images. The four-plane normal vector method can reduce the noise information in the edge feature maps. The adaptive fractional order is selected according to the normal mean gradient and the local Gaussian curvature entropy of the images. Secondly, the extracted edge feature maps and blurred images are input into the de-artifact network. In this network, the edge fusion feature extraction network and the edge fusion transformer network are specially designed. The former combines the edge feature maps with the fuzzy feature maps to extract the edge feature information. The latter combines the edge attention network and the fuzzy attention network, which can focus on the blurred image edges. Finally, extensive experiments show that the proposed method can obtain higher peak signal-to-noise ratio and structural similarity index measure compared to state-of-art methods. The de-artifact images have clear texture edges.
Collapse
Affiliation(s)
- Nanhe Jiang
- School of Electrical Engineering, Yanshan University, Qinhuangdao, 066004, Hebei, People's Republic of China
| | - Yucun Zhang
- School of Electrical Engineering, Yanshan University, Qinhuangdao, 066004, Hebei, People's Republic of China
| | - Qun Li
- School of Mechanical Engineering, Yanshan University, Qinhuangdao, 066004, Hebei, People's Republic of China
| | - Xianbin Fu
- Hebei University of Environmental Engineering, Qinhuangdao, 066102, Hebei, People's Republic of China
| | - Dongqing Fang
- Capital Aerospace Machinery Co, Ltd, Fengtai, 100076, Beijing, People's Republic of China
| |
Collapse
|
12
|
Zheng S, Sun Q, Ye X, Li W, Yu L, Yang C. Multi-scale adversarial learning with difficult region supervision learning models for primary tumor segmentation. Phys Med Biol 2024; 69:085009. [PMID: 38471170 DOI: 10.1088/1361-6560/ad3321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Accepted: 03/12/2024] [Indexed: 03/14/2024]
Abstract
Objective.Recently, deep learning techniques have found extensive application in accurate and automated segmentation of tumor regions. However, owing to the variety of tumor shapes, complex types, and unpredictability of spatial distribution, tumor segmentation still faces major challenges. Taking cues from the deep supervision and adversarial learning, we have devised a cascade-based methodology incorporating multi-scale adversarial learning and difficult-region supervision learning in this study to tackle these challenges.Approach.Overall, the method adheres to a coarse-to-fine strategy, first roughly locating the target region, and then refining the target object with multi-stage cascaded binary segmentation which converts complex multi-class segmentation problems into multiple simpler binary segmentation problems. In addition, a multi-scale adversarial learning difficult supervised UNet (MSALDS-UNet) is proposed as our model for fine-segmentation, which applies multiple discriminators along the decoding path of the segmentation network to implement multi-scale adversarial learning, thereby enhancing the accuracy of network segmentation. Meanwhile, in MSALDS-UNet, we introduce a difficult region supervision loss to effectively utilize structural information for segmenting difficult-to-distinguish areas, such as blurry boundary areas.Main results.A thorough validation of three independent public databases (KiTS21, MSD's Brain and Pancreas datasets) shows that our model achieves satisfactory results for tumor segmentation in terms of key evaluation metrics including dice similarity coefficient, Jaccard similarity coefficient, and HD95.Significance.This paper introduces a cascade approach that combines multi-scale adversarial learning and difficult supervision to achieve precise tumor segmentation. It confirms that the combination can improve the segmentation performance, especially for small objects (our codes are publicly availabled onhttps://zhengshenhai.github.io/).
Collapse
Affiliation(s)
- Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Qiuyu Sun
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Xin Ye
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Weisheng Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Lei Yu
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, People's Republic of China
| | - Chaohui Yang
- Nanpeng Artificial Intelligence Research Institute, Chongqing, People's Republic of China
| |
Collapse
|
13
|
Pan NY, Huang TY, Yu JJ, Peng HH, Chuang TC, Lin YR, Chung HW, Wu MT. Virtual MOLLI Target: Generative Adversarial Networks Toward Improved Motion Correction in MRI Myocardial T1 Mapping. J Magn Reson Imaging 2024. [PMID: 38563660 DOI: 10.1002/jmri.29373] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 03/21/2024] [Accepted: 03/21/2024] [Indexed: 04/04/2024] Open
Abstract
BACKGROUND The modified Look-Locker inversion recovery (MOLLI) sequence is commonly used for myocardial T1 mapping. However, it acquires images with different inversion times, which causes difficulty in motion correction for respiratory-induced misregistration to a given target image. HYPOTHESIS Using a generative adversarial network (GAN) to produce virtual MOLLI images with consistent heart positions can reduce respiratory-induced misregistration of MOLLI datasets. STUDY TYPE Retrospective. POPULATION 1071 MOLLI datasets from 392 human participants. FIELD STRENGTH/SEQUENCE Modified Look-Locker inversion recovery sequence at 3 T. ASSESSMENT A GAN model with a single inversion time image as input was trained to generate virtual MOLLI target (VMT) images at different inversion times which were subsequently used in an image registration algorithm. Four VMT models were investigated and the best performing model compared with the standard vendor-provided motion correction (MOCO) technique. STATISTICAL TESTS The effectiveness of the motion correction technique was assessed using the fitting quality index (FQI), mutual information (MI), and Dice coefficients of motion-corrected images, plus subjective quality evaluation of T1 maps by three independent readers using Likert score. Wilcoxon signed-rank test with Bonferroni correction for multiple comparison. Significance levels were defined as P < 0.01 for highly significant differences and P < 0.05 for significant differences. RESULTS The best performing VMT model with iterative registration demonstrated significantly better performance (FQI 0.88 ± 0.03, MI 1.78 ± 0.20, Dice 0.84 ± 0.23, quality score 2.26 ± 0.95) compared to other approaches, including the vendor-provided MOCO method (FQI 0.86 ± 0.04, MI 1.69 ± 0.25, Dice 0.80 ± 0.27, quality score 2.16 ± 1.01). DATA CONCLUSION Our GAN model generating VMT images improved motion correction, which may assist reliable T1 mapping in the presence of respiratory motion. Its robust performance, even with considerable respiratory-induced heart displacements, may be beneficial for patients with difficulties in breath-holding. LEVEL OF EVIDENCE 3 TECHNICAL EFFICACY: Stage 1.
Collapse
Affiliation(s)
- Nai-Yu Pan
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Teng-Yi Huang
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Jui-Jung Yu
- Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Hsu-Hsia Peng
- Department of Biomedical Engineering and Environmental Sciences, National Tsing Hua University, Hsinchu, Taiwan
| | - Tzu-Chao Chuang
- Department of Electrical Engineering, National Sun Yat-Sen University, Kaohsiung, Taiwan
| | - Yi-Ru Lin
- Department of Electronic and Computer Engineering, National Taiwan University of Science and Technology, Taipei, Taiwan
| | - Hsiao-Wen Chung
- Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
| | - Ming-Ting Wu
- Department of Radiology, Kaohsiung Veterans General Hospital, Kaohsiung, Taiwan
- School of Medicine, College of Medicine, National Yang Ming Chiao Tung University, Taipei, Taiwan
| |
Collapse
|
14
|
Miao J, Zhou SP, Zhou GQ, Wang KN, Yang M, Zhou S, Chen Y. SC-SSL: Self-Correcting Collaborative and Contrastive Co-Training Model for Semi-Supervised Medical Image Segmentation. IEEE Trans Med Imaging 2024; 43:1347-1364. [PMID: 37995173 DOI: 10.1109/tmi.2023.3336534] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/25/2023]
Abstract
Image segmentation achieves significant improvements with deep neural networks at the premise of a large scale of labeled training data, which is laborious to assure in medical image tasks. Recently, semi-supervised learning (SSL) has shown great potential in medical image segmentation. However, the influence of the learning target quality for unlabeled data is usually neglected in these SSL methods. Therefore, this study proposes a novel self-correcting co-training scheme to learn a better target that is more similar to ground-truth labels from collaborative network outputs. Our work has three-fold highlights. First, we advance the learning target generation as a learning task, improving the learning confidence for unannotated data with a self-correcting module. Second, we impose a structure constraint to encourage the shape similarity further between the improved learning target and the collaborative network outputs. Finally, we propose an innovative pixel-wise contrastive learning loss to boost the representation capacity under the guidance of an improved learning target, thus exploring unlabeled data more efficiently with the awareness of semantic context. We have extensively evaluated our method with the state-of-the-art semi-supervised approaches on four public-available datasets, including the ACDC dataset, M&Ms dataset, Pancreas-CT dataset, and Task_07 CT dataset. The experimental results with different labeled-data ratios show our proposed method's superiority over other existing methods, demonstrating its effectiveness in semi-supervised medical image segmentation.
Collapse
|
15
|
Vernikouskaya I, Bertsche D, Metze P, Schneider LM, Rasche V. Multi-network approach for image segmentation in non-contrast enhanced cardiac 3D MRI of arrhythmic patients. Comput Med Imaging Graph 2024; 113:102340. [PMID: 38277768 DOI: 10.1016/j.compmedimag.2024.102340] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 01/16/2024] [Accepted: 01/16/2024] [Indexed: 01/28/2024]
Abstract
Left atrial appendage (LAA) is the source of thrombi formation in more than 90% of strokes in patients with nonvalvular atrial fibrillation. Catheter-based LAA occlusion is being increasingly applied as a treatment strategy to prevent stroke. Anatomical complexity of LAA makes percutaneous occlusion commonly performed under transesophageal echocardiography (TEE) and X-ray (XR) guidance especially challenging. Image fusion techniques integrating 3D anatomical models derived from pre-procedural imaging into the live XR fluoroscopy can be applied to guide each step of the LAA closure. Cardiac magnetic resonance (CMR) imaging gains in importance for radiation-free evaluation of cardiac morphology as alternative to gold-standard TEE or computed tomography angiography (CTA). Manual delineation of cardiac structures from non-contrast enhanced CMR is, however, labor-intensive, tedious, and challenging due to the rather low contrast. Additionally, arrhythmia often impairs the image quality in ECG synchronized acquisitions causing blurring and motion artifacts. Thus, for cardiac segmentation in arrhythmic patients, there is a strong need for an automated image segmentation method. Deep learning-based methods have shown great promise in medical image analysis achieving superior performance in various imaging modalities and different clinical applications. Fully-convolutional neural networks (CNNs), especially U-Net, have become the method of choice for cardiac segmentation. In this paper, we propose an approach for automatic segmentation of cardiac structures from non-contrast enhanced CMR images of arrhythmic patients based on CNNs implemented in a multi-stage pipeline. Two-stage implementation allows subdividing the task into localization of the relevant cardiac structures and segmentation of these structures from the cropped sub-regions obtained from previous step leading to efficient and effective way of automated cardiac segmentation.
Collapse
Affiliation(s)
- Ina Vernikouskaya
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Dagmar Bertsche
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Patrick Metze
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Leonhard M Schneider
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| | - Volker Rasche
- Department of Internal Medicine II, Ulm University Medical Center, Ulm, Germany.
| |
Collapse
|
16
|
Liu J, Desrosiers C, Yu D, Zhou Y. Semi-Supervised Medical Image Segmentation Using Cross-Style Consistency With Shape-Aware and Local Context Constraints. IEEE Trans Med Imaging 2024; 43:1449-1461. [PMID: 38032771 DOI: 10.1109/tmi.2023.3338269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/02/2023]
Abstract
Despite the remarkable progress in semi-supervised medical image segmentation methods based on deep learning, their application to real-life clinical scenarios still faces considerable challenges. For example, insufficient labeled data often makes it difficult for networks to capture the complexity and variability of the anatomical regions to be segmented. To address these problems, we design a new semi-supervised segmentation framework that aspires to produce anatomically plausible predictions. Our framework comprises two parallel networks: shape-agnostic and shape-aware networks. These networks learn from each other, enabling effective utilization of unlabeled data. Our shape-aware network implicitly introduces shape guidance to capture shape fine-grained information. Meanwhile, shape-agnostic networks employ uncertainty estimation to further obtain reliable pseudo-labels for the counterpart. We also employ a cross-style consistency strategy to enhance the network's utilization of unlabeled data. It enriches the dataset to prevent overfitting and further eases the coupling of the two networks that learn from each other. Our proposed architecture also incorporates a novel loss term that facilitates the learning of the local context of segmentation by the network, thereby enhancing the overall accuracy of prediction. Experiments on three different datasets of medical images show that our method outperforms many excellent semi-supervised segmentation methods and outperforms them in perceiving shape. The code can be seen at https://github.com/igip-liu/SLC-Net.
Collapse
|
17
|
Liu Z, Kainth K, Zhou A, Deyer TW, Fayad ZA, Greenspan H, Mei X. A review of self-supervised, generative, and few-shot deep learning methods for data-limited magnetic resonance imaging segmentation. NMR Biomed 2024:e5143. [PMID: 38523402 DOI: 10.1002/nbm.5143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 02/15/2024] [Accepted: 02/16/2024] [Indexed: 03/26/2024]
Abstract
Magnetic resonance imaging (MRI) is a ubiquitous medical imaging technology with applications in disease diagnostics, intervention, and treatment planning. Accurate MRI segmentation is critical for diagnosing abnormalities, monitoring diseases, and deciding on a course of treatment. With the advent of advanced deep learning frameworks, fully automated and accurate MRI segmentation is advancing. Traditional supervised deep learning techniques have advanced tremendously, reaching clinical-level accuracy in the field of segmentation. However, these algorithms still require a large amount of annotated data, which is oftentimes unavailable or impractical. One way to circumvent this issue is to utilize algorithms that exploit a limited amount of labeled data. This paper aims to review such state-of-the-art algorithms that use a limited number of annotated samples. We explain the fundamental principles of self-supervised learning, generative models, few-shot learning, and semi-supervised learning and summarize their applications in cardiac, abdomen, and brain MRI segmentation. Throughout this review, we highlight algorithms that can be employed based on the quantity of annotated data available. We also present a comprehensive list of notable publicly available MRI segmentation datasets. To conclude, we discuss possible future directions of the field-including emerging algorithms, such as contrastive language-image pretraining, and potential combinations across the methods discussed-that can further increase the efficacy of image segmentation with limited labels.
Collapse
Affiliation(s)
- Zelong Liu
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Komal Kainth
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Alexander Zhou
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Timothy W Deyer
- East River Medical Imaging, New York, New York, USA
- Department of Radiology, Cornell Medicine, New York, New York, USA
| | - Zahi A Fayad
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Hayit Greenspan
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Xueyan Mei
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| |
Collapse
|
18
|
Dwivedi V, Srinivasan B, Krishnamurthi G. Physics informed contour selection for rapid image segmentation. Sci Rep 2024; 14:6996. [PMID: 38523137 PMCID: PMC10961308 DOI: 10.1038/s41598-024-57281-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2023] [Accepted: 03/15/2024] [Indexed: 03/26/2024] Open
Abstract
Effective training of deep image segmentation models is challenging due to the need for abundant, high-quality annotations. To facilitate image annotation, we introduce Physics Informed Contour Selection (PICS)-an interpretable, physics-informed algorithm for rapid image segmentation without relying on labeled data. PICS draws inspiration from physics-informed neural networks (PINNs) and an active contour model called snake. It is fast and computationally lightweight because it employs cubic splines instead of a deep neural network as a basis function. Its training parameters are physically interpretable because they directly represent control knots of the segmentation curve. Traditional snakes involve minimization of the edge-based loss functionals by deriving the Euler-Lagrange equation followed by its numerical solution. However, PICS directly minimizes the loss functional, bypassing the Euler Lagrange equations. It is the first snake variant to minimize a region-based loss function instead of traditional edge-based loss functions. PICS uniquely models the three-dimensional (3D) segmentation process with an unsteady partial differential equation (PDE), which allows accelerated segmentation via transfer learning. To demonstrate its effectiveness, we apply PICS for 3D segmentation of the left ventricle on a publicly available cardiac dataset. We also demonstrate PICS's capacity to encode the prior shape information as a loss term by proposing a new convexity-preserving loss term for left ventricle. Overall, PICS presents several novelties in network architecture, transfer learning, and physics-inspired losses for image segmentation, thereby showing promising outcomes and potential for further refinement.
Collapse
Affiliation(s)
- Vikas Dwivedi
- Atmospheric Science Research Center, State University of New York, Albany, NY, 12222, USA.
| | - Balaji Srinivasan
- Department of Mechanical Engineering, Indian Institute of Technology, Madras, Chennai, 600036, India
- Wadhwani School of Data Science and AI, Indian Institute of Technology, Madras, Chennai, 600036, India
| | - Ganapathy Krishnamurthi
- Department of Engineering Design, Indian Institute of Technology, Madras, Chennai, 600036, India
- Wadhwani School of Data Science and AI, Indian Institute of Technology, Madras, Chennai, 600036, India
| |
Collapse
|
19
|
Liu Y, Zhang Z, Yue J, Guo W. SCANeXt: Enhancing 3D medical image segmentation with dual attention network and depth-wise convolution. Heliyon 2024; 10:e26775. [PMID: 38439873 PMCID: PMC10909707 DOI: 10.1016/j.heliyon.2024.e26775] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2024] [Revised: 02/19/2024] [Accepted: 02/20/2024] [Indexed: 03/06/2024] Open
Abstract
Existing approaches to 3D medical image segmentation can be generally categorized into convolution-based or transformer-based methods. While convolutional neural networks (CNNs) demonstrate proficiency in extracting local features, they encounter challenges in capturing global representations. In contrast, the consecutive self-attention modules present in vision transformers excel at capturing long-range dependencies and achieving an expanded receptive field. In this paper, we propose a novel approach, termed SCANeXt, for 3D medical image segmentation. Our method combines the strengths of dual attention (Spatial and Channel Attention) and ConvNeXt to enhance representation learning for 3D medical images. In particular, we propose a novel self-attention mechanism crafted to encompass spatial and channel relationships throughout the entire feature dimension. To further extract multiscale features, we introduce a depth-wise convolution block inspired by ConvNeXt after the dual attention block. Extensive evaluations on three benchmark datasets, namely Synapse, BraTS, and ACDC, demonstrate the effectiveness of our proposed method in terms of accuracy. Our SCANeXt model achieves a state-of-the-art result with a Dice Similarity Score of 95.18% on the ACDC dataset, significantly outperforming current methods.
Collapse
Affiliation(s)
- Yajun Liu
- Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, China
| | - Zenghui Zhang
- Shanghai Key Laboratory of Intelligent Sensing and Recognition, Shanghai Jiao Tong University, China
| | - Jiang Yue
- Department of Endocrinology and Metabolism, Renji Hospital, School of Medicine, Shanghai Jiao Tong University, China
| | - Weiwei Guo
- Center for Digital Innovation, Tongji University, China
| |
Collapse
|
20
|
Long J, Ren Y, Yang C, Ren P, Zeng Z. MDT: semi-supervised medical image segmentation with mixup-decoupling training. Phys Med Biol 2024; 69:065012. [PMID: 38324897 DOI: 10.1088/1361-6560/ad2715] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/07/2023] [Accepted: 02/07/2024] [Indexed: 02/09/2024]
Abstract
Objective. In the field of medicine, semi-supervised segmentation algorithms hold crucial research significance while also facing substantial challenges, primarily due to the extreme scarcity of expert-level annotated medical image data. However, many existing semi-supervised methods still process labeled and unlabeled data in inconsistent ways, which can lead to knowledge learned from labeled data being discarded to some extent. This not only lacks a variety of perturbations to explore potential robust information in unlabeled data but also ignores the confirmation bias and class imbalance issues in pseudo-labeling methods.Approach. To solve these problems, this paper proposes a semi-supervised medical image segmentation method 'mixup-decoupling training (MDT)' that combines the idea of consistency and pseudo-labeling. Firstly, MDT introduces a new perturbation strategy 'mixup-decoupling' to fully regularize training data. It not only mixes labeled and unlabeled data at the data level but also performs decoupling operations between the output predictions of mixed target data and labeled data at the feature level to obtain strong version predictions of unlabeled data. Then it establishes a dual learning paradigm based on consistency and pseudo-labeling. Secondly, MDT employs a novel categorical entropy filtering approach to pick high-confidence pseudo-labels for unlabeled data, facilitating more refined supervision.Main results. This paper compares MDT with other advanced semi-supervised methods on 2D and 3D datasets separately. A large number of experimental results show that MDT achieves competitive segmentation performance and outperforms other state-of-the-art semi-supervised segmentation methods.Significance. This paper proposes a semi-supervised medical image segmentation method MDT, which greatly reduces the demand for manually labeled data and eases the difficulty of data annotation to a great extent. In addition, MDT not only outperforms many advanced semi-supervised image segmentation methods in quantitative and qualitative experimental results, but also provides a new and developable idea for semi-supervised learning and computer-aided diagnosis technology research.
Collapse
Affiliation(s)
- Jianwu Long
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, People's Republic of China
| | - Yan Ren
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, People's Republic of China
| | - Chengxin Yang
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, People's Republic of China
| | - Pengcheng Ren
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, People's Republic of China
| | - Ziqin Zeng
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing 400054, People's Republic of China
| |
Collapse
|
21
|
Daudé P, Ramasawmy R, Javed A, Lederman RJ, Chow K, Campbell-Washburn AE. Inline automatic quality control of 2D phase-contrast flow MRI for subject-specific scan time adaptation. Magn Reson Med 2024. [PMID: 38469944 DOI: 10.1002/mrm.30083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Revised: 02/01/2024] [Accepted: 02/24/2024] [Indexed: 03/13/2024]
Abstract
PURPOSE To develop an inline automatic quality control to achieve consistent diagnostic image quality with subject-specific scan time, and to demonstrate this method for 2D phase-contrast flow MRI to reach a predetermined SNR. METHODS We designed a closed-loop feedback framework between image reconstruction and data acquisition to intermittently check SNR (every 20 s) and automatically stop the acquisition when a target SNR is achieved. A free-breathing 2D pseudo-golden-angle spiral phase-contrast sequence was modified to listen for image-quality messages from the reconstructions. Ten healthy volunteers and 1 patient were imaged at 0.55 T. Target SNR was selected based on retrospective analysis of cardiac output error, and performance of the automatic SNR-driven "stop" was assessed inline. RESULTS SNR calculation and automated segmentation was feasible within 20 s with inline deployment. The SNR-driven acquisition time was 2 min 39 s ± 67 s (aorta) and 3 min ± 80 s (main pulmonary artery) with a min/max acquisition time of 1 min 43 s/4 min 52 s (aorta) and 1 min 43 s/5 min 50 s (main pulmonary artery) across 6 healthy volunteers, while ensuring a diagnostic measurement with relative absolute error in quantitative flow measurement lower than 2.1% (aorta) and 6.3% (main pulmonary artery). CONCLUSION The inline quality control enables subject-specific optimized scan times while ensuring consistent diagnostic image quality. The distribution of automated stopping times across the population revealed the value of a subject-specific scan time.
Collapse
Affiliation(s)
- Pierre Daudé
- Laboratory of Imaging Technology, Cardiovascular Branch, Division of Intramural Research, National Heart Lung & Blood Institute, National Institutes of Health, Bethesda, Maryland, USA
| | - Rajiv Ramasawmy
- Laboratory of Imaging Technology, Cardiovascular Branch, Division of Intramural Research, National Heart Lung & Blood Institute, National Institutes of Health, Bethesda, Maryland, USA
| | - Ahsan Javed
- Laboratory of Imaging Technology, Cardiovascular Branch, Division of Intramural Research, National Heart Lung & Blood Institute, National Institutes of Health, Bethesda, Maryland, USA
| | - Robert J Lederman
- Laboratory of Cardiovascular Intervention, Cardiovascular Branch, Division of Intramural Research, National Heart Lung & Blood Institute, National Institutes of Health, Bethesda, Maryland, USA
| | - Kelvin Chow
- Siemens Healthcare Ltd., Calgary, Alberta, Canada
| | - Adrienne E Campbell-Washburn
- Laboratory of Imaging Technology, Cardiovascular Branch, Division of Intramural Research, National Heart Lung & Blood Institute, National Institutes of Health, Bethesda, Maryland, USA
| |
Collapse
|
22
|
Zhang Y, Chen Z, Yang X. Light-M: An efficient lightweight medical image segmentation framework for resource-constrained IoMT. Comput Biol Med 2024; 170:108088. [PMID: 38320339 DOI: 10.1016/j.compbiomed.2024.108088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Revised: 12/22/2023] [Accepted: 01/27/2024] [Indexed: 02/08/2024]
Abstract
The Internet of Medical Things (IoMT) is being incorporated into current healthcare systems. This technology intends to connect patients, IoMT devices, and hospitals over mobile networks, allowing for more secure, quick, and convenient health monitoring and intelligent healthcare services. However, existing intelligent healthcare applications typically rely on large-scale AI models, and standard IoMT devices have significant resource constraints. To alleviate this paradox, in this paper, we propose a Knowledge Distillation (KD)-based IoMT end-edge-cloud orchestrated architecture for medical image segmentation tasks, called Light-M, aiming to deploy a lightweight medical model in resource-constrained IoMT devices. Specifically, Light-M trains a large teacher model in the cloud server and employs computation in local nodes through imitation of the performance of the teacher model using knowledge distillation. Light-M contains two KD strategies: (1) active exploration and passive transfer (AEPT) and (2) self-attention-based inter-class feature variation (AIFV) distillation for the medical image segmentation task. The AEPT encourages the student model to learn undiscovered knowledge/features of the teacher model without additional feature layers, aiming to explore new features and outperform the teacher. To improve the distinguishability of the student for different classes, the student learns the self-attention-based feature variation (AIFV) between classes. Since the proposed AEPT and AIFV only appear in the training process, our framework does not involve any additional computation burden for a student model during the segmentation task deployment. Extensive experiments on cardiac images and public real-scene datasets demonstrate that our approach improves student model learning representations and outperforms state-of-the-art methods by combining two knowledge distillation strategies. Moreover, when deployed on the IoT device, the distilled student model takes only 29.6 ms for one sample at the inference step.
Collapse
Affiliation(s)
- Yifan Zhang
- Shenzhen University, 3688 Nanhai Ave., Shenzhen, 518060, Guangdong, China
| | - Zhuangzhuang Chen
- Shenzhen University, 3688 Nanhai Ave., Shenzhen, 518060, Guangdong, China
| | - Xuan Yang
- Shenzhen University, 3688 Nanhai Ave., Shenzhen, 518060, Guangdong, China.
| |
Collapse
|
23
|
Chen Y, Yang Z, Shen C, Wang Z, Zhang Z, Qin Y, Wei X, Lu J, Liu Y, Zhang Y. Evidence-based uncertainty-aware semi-supervised medical image segmentation. Comput Biol Med 2024; 170:108004. [PMID: 38277924 DOI: 10.1016/j.compbiomed.2024.108004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Revised: 12/21/2023] [Accepted: 01/13/2024] [Indexed: 01/28/2024]
Abstract
Semi-Supervised Learning (SSL) has demonstrated great potential to reduce the dependence on a large set of annotated data, which is challenging to collect in clinical practice. One of the most important SSL methods is to generate pseudo labels from the unlabeled data using a network model trained with labeled data, which will inevitably introduce false pseudo labels into the training process and potentially jeopardize performance. To address this issue, uncertainty-aware methods have emerged as a promising solution and have gained considerable attention recently. However, current uncertainty-aware methods usually face the dilemma of balancing the additional computational cost, uncertainty estimation accuracy, and theoretical basis in a unified training paradigm. To address this issue, we propose to integrate the Dempster-Shafer Theory of Evidence (DST) into SSL-based medical image segmentation, dubbed EVidential Inference Learning (EVIL). EVIL performs as a novel consistency regularization-based training paradigm, which enforces consistency on predictions perturbed by two networks with different parameters to enhance generalization Additionally, EVIL provides a theoretically assured solution for precise uncertainty quantification within a single forward pass. By discarding highly unreliable pseudo labels after uncertainty estimation, trustworthy pseudo labels can be generated and incorporated into subsequent model training. The experimental results demonstrate that the proposed approach performs competitively when benchmarked against several state-of-the-art methods on public datasets, i.e., ACDC, MM-WHS, and MonuSeg. The code can be found at https://github.com/CYYukio/EVidential-Inference-Learning.
Collapse
Affiliation(s)
- Yingyu Chen
- College of Computer Science, Sichuan University, China; The Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University, China
| | - Ziyuan Yang
- College of Computer Science, Sichuan University, China
| | - Chenyu Shen
- College of Computer Science, Sichuan University, China
| | - Zhiwen Wang
- College of Computer Science, Sichuan University, China
| | | | - Yang Qin
- College of Computer Science, Sichuan University, China
| | - Xin Wei
- Department of Ophthalmology, West China Hospital, Sichuan University, China
| | - Jingfeng Lu
- School of Cyber Science and Engineering, Sichuan University, China
| | - Yan Liu
- College of Electrical Engineering, Sichuan University, China.
| | - Yi Zhang
- School of Cyber Science and Engineering, Sichuan University, China; The Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Sichuan University, China
| |
Collapse
|
24
|
Li X, Hu Y. Cooperative-Net: An end-to-end multi-task interaction network for unified reconstruction and segmentation of MR image. Comput Methods Programs Biomed 2024; 245:108045. [PMID: 38290292 DOI: 10.1016/j.cmpb.2024.108045] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Revised: 01/12/2024] [Accepted: 01/21/2024] [Indexed: 02/01/2024]
Abstract
BACKGROUND AND OBJECTIVE In clinical applications, there is an increasing demand for rapid acquisition and automated analysis of magnetic resonance imaging (MRI) data. However, most existing methods focus on either MR image reconstruction from undersampled data or segmentation using fully sampled data, hardly considering MR image segmentation in fast imaging scenarios. Consequently, it is imperative to investigate a multi-task approach that can simultaneously achieve high scanning acceleration and accurate segmentation results. METHODS In this paper, we propose a novel end-to-end multi-task interaction network, termed as the Cooperative-Net, which integrates accelerated MR imaging and multi-class tissue segmentation into a unified framework. The Cooperative-Net consists of alternating reconstruction modules and segmentation modules. To facilitate effective interaction between the two tasks, we introduce the spatial-adaptive semantic guidance module, which leverages the semantic map as a structural prior to guide MR image reconstruction. Furthermore, we propose a novel unrolling network with a multi-path shrinkage structure for MR image reconstruction. This network consists of parallel learnable shrinkage paths to handle varying degrees of degradation across different frequency components in the undersampled MR image, effectively improving the quality of the recovered image. RESULTS We use two publicly available datasets, including the cardiac and knee MR datasets, to validate the efficacy of our proposed Cooperative-Net. Through qualitative and quantitative analysis, we demonstrate that our method outperforms existing state-of-the-art multi-task approaches for joint MR image reconstruction and segmentation. CONCLUSIONS The proposed Cooperative-Net is capable of achieving both high accelerated MR imaging and accurate multi-class tissue segmentation.
Collapse
Affiliation(s)
- Xiaodi Li
- School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, 150001, China
| | - Yue Hu
- School of Electronics and Information Engineering, Harbin Institute of Technology, Harbin, 150001, China.
| |
Collapse
|
25
|
Ao Y, Shi W, Ji B, Miao Y, He W, Jiang Z. MS-TCNet: An effective Transformer-CNN combined network using multi-scale feature learning for 3D medical image segmentation. Comput Biol Med 2024; 170:108057. [PMID: 38301516 DOI: 10.1016/j.compbiomed.2024.108057] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Revised: 12/31/2023] [Accepted: 01/26/2024] [Indexed: 02/03/2024]
Abstract
Medical image segmentation is a fundamental research problem in the field of medical image processing. Recently, the Transformer have achieved highly competitive performance in computer vision. Therefore, many methods combining Transformer with convolutional neural networks (CNNs) have emerged for segmenting medical images. However, these methods cannot effectively capture the multi-scale features in medical images, even though texture and contextual information embedded in the multi-scale features are extremely beneficial for segmentation. To alleviate this limitation, we propose a novel Transformer-CNN combined network using multi-scale feature learning for three-dimensional (3D) medical image segmentation, which is called MS-TCNet. The proposed model utilizes a shunted Transformer and CNN to construct an encoder and pyramid decoder, allowing six different scale levels of feature learning. It captures multi-scale features with refinement at each scale level. Additionally, we propose a novel lightweight multi-scale feature fusion (MSFF) module that can fully fuse the different-scale semantic features generated by the pyramid decoder for each segmentation class, resulting in a more accurate segmentation output. We conducted experiments on three widely used 3D medical image segmentation datasets. The experimental results indicated that our method outperformed state-of-the-art medical image segmentation methods, suggesting its effectiveness, robustness, and superiority. Meanwhile, our model has a smaller number of parameters and lower computational complexity than conventional 3D segmentation networks. The results confirmed that the model is capable of effective multi-scale feature learning and that the learned multi-scale features are useful for improving segmentation performance. We open-sourced our code, which can be found at https://github.com/AustinYuAo/MS-TCNet.
Collapse
Affiliation(s)
- Yu Ao
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, 130022, China
| | - Weili Shi
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, 130022, China; Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, 528437, China
| | - Bai Ji
- Department of Hepatobiliary and Pancreatic Surgery, The First Hospital of Jilin University, Changchun, 130061, China
| | - Yu Miao
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, 130022, China; Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, 528437, China
| | - Wei He
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, 130022, China; Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, 528437, China
| | - Zhengang Jiang
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, 130022, China; Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, 528437, China.
| |
Collapse
|
26
|
Lu S, Yan Z, Chen W, Cheng T, Zhang Z, Yang G. Dual consistency regularization with subjective logic for semi-supervised medical image segmentation. Comput Biol Med 2024; 170:107991. [PMID: 38242016 DOI: 10.1016/j.compbiomed.2024.107991] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2023] [Revised: 12/18/2023] [Accepted: 01/13/2024] [Indexed: 01/21/2024]
Abstract
Semi-supervised learning plays a vital role in computer vision tasks, particularly in medical image analysis. It significantly reduces the time and cost involved in labeling data. Current methods primarily focus on consistency regularization and the generation of pseudo labels. However, due to the model's poor awareness of unlabeled data, aforementioned methods may misguide the model. To alleviate this problem, we propose a dual consistency regularization with subjective logic for semi-supervised medical image segmentation. Specifically, we introduce subjective logic into our semi-supervised medical image segmentation task to estimate uncertainty, and based on the consistency hypothesis, we construct dual consistency regularization under weak and strong perturbations to guide the model's learning from unlabeled data. To evaluate the performance of the proposed method, we performed experiments on three widely used datasets: ACDC, LA, and Pancreas. Experiments show that the proposed method achieved improvement compared with other state-of-the-art (SOTA) methods.
Collapse
Affiliation(s)
- Shanfu Lu
- Perception Vision Medical Technologies Co., Ltd, Guangzhou, 510530, China.
| | - Ziye Yan
- Perception Vision Medical Technologies Co., Ltd, Guangzhou, 510530, China
| | - Wei Chen
- The radiotherapy department of second peoples' hospital, neijiang, 641000, China
| | - Tingting Cheng
- Department of Oncology, National Clinical Research Center for Geriatric Disorders and Xiangya Lung Cancer Center, Xiangya Hospital, Central South University, Changsha, 41000, China.
| | - Zijian Zhang
- Department of Oncology, National Clinical Research Center for Geriatric Disorders and Xiangya Lung Cancer Center, Xiangya Hospital, Central South University, Changsha, 41000, China.
| | - Guang Yang
- Bioengineering Department and Imperial-X, Imperial College London, London, UK; National Heart and Lung Institute, Imperial College London, London, UK; Cardiovascular Research Centre, Royal Brompton Hospital, London, UK; School of Biomedical Engineering & Imaging Sciences, King's College London, London, UK
| |
Collapse
|
27
|
Machado I, Puyol-Anton E, Hammernik K, Cruz G, Ugurlu D, Olakorede I, Oksuz I, Ruijsink B, Castelo-Branco M, Young A, Prieto C, Schnabel J, King A. A Deep Learning-Based Integrated Framework for Quality-Aware Undersampled Cine Cardiac MRI Reconstruction and Analysis. IEEE Trans Biomed Eng 2024; 71:855-865. [PMID: 37782583 DOI: 10.1109/tbme.2023.3321431] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/04/2023]
Abstract
Cine cardiac magnetic resonance (CMR) imaging is considered the gold standard for cardiac function evaluation. However, cine CMR acquisition is inherently slow and in recent decades considerable effort has been put into accelerating scan times without compromising image quality or the accuracy of derived results. In this article, we present a fully-automated, quality-controlled integrated framework for reconstruction, segmentation and downstream analysis of undersampled cine CMR data. The framework produces high quality reconstructions and segmentations, leading to undersampling factors that are optimised on a scan-by-scan basis. This results in reduced scan times and automated analysis, enabling robust and accurate estimation of functional biomarkers. To demonstrate the feasibility of the proposed approach, we perform simulations of radial k-space acquisitions using in-vivo cine CMR data from 270 subjects from the UK Biobank (with synthetic phase) and in-vivo cine CMR data from 16 healthy subjects (with real phase). The results demonstrate that the optimal undersampling factor varies for different subjects by approximately 1 to 2 seconds per slice. We show that our method can produce quality-controlled images in a mean scan time reduced from 12 to 4 seconds per slice, and that image quality is sufficient to allow clinically relevant parameters to be automatically estimated to lie within 5% mean absolute difference.
Collapse
|
28
|
Xu X, Li J, Zhu Z, Zhao L, Wang H, Song C, Chen Y, Zhao Q, Yang J, Pei Y. A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis. Bioengineering (Basel) 2024; 11:219. [PMID: 38534493 DOI: 10.3390/bioengineering11030219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Revised: 02/15/2024] [Accepted: 02/21/2024] [Indexed: 03/28/2024] Open
Abstract
Disease diagnosis represents a critical and arduous endeavor within the medical field. Artificial intelligence (AI) techniques, spanning from machine learning and deep learning to large model paradigms, stand poised to significantly augment physicians in rendering more evidence-based decisions, thus presenting a pioneering solution for clinical practice. Traditionally, the amalgamation of diverse medical data modalities (e.g., image, text, speech, genetic data, physiological signals) is imperative to facilitate a comprehensive disease analysis, a topic of burgeoning interest among both researchers and clinicians in recent times. Hence, there exists a pressing need to synthesize the latest strides in multi-modal data and AI technologies in the realm of medical diagnosis. In this paper, we narrow our focus to five specific disorders (Alzheimer's disease, breast cancer, depression, heart disease, epilepsy), elucidating advanced endeavors in their diagnosis and treatment through the lens of artificial intelligence. Our survey not only delineates detailed diagnostic methodologies across varying modalities but also underscores commonly utilized public datasets, the intricacies of feature engineering, prevalent classification models, and envisaged challenges for future endeavors. In essence, our research endeavors to contribute to the advancement of diagnostic methodologies, furnishing invaluable insights for clinical decision making.
Collapse
Affiliation(s)
- Xi Xu
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Jianqiang Li
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Zhichao Zhu
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Linna Zhao
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Huina Wang
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Changwei Song
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Yining Chen
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Qing Zhao
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Jijiang Yang
- Tsinghua National Laboratory for Information Science and Technology, Tsinghua University, Beijing 100084, China
| | - Yan Pei
- School of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu 965-8580, Japan
| |
Collapse
|
29
|
Schilling M, Unterberg-Buchwald C, Lotz J, Uecker M. Assessment of deep learning segmentation for real-time free-breathing cardiac magnetic resonance imaging at rest and under exercise stress. Sci Rep 2024; 14:3754. [PMID: 38355969 PMCID: PMC10866998 DOI: 10.1038/s41598-024-54164-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Accepted: 02/09/2024] [Indexed: 02/16/2024] Open
Abstract
In recent years, a variety of deep learning networks for cardiac MRI (CMR) segmentation have been developed and analyzed. However, nearly all of them are focused on cine CMR under breathold. In this work, accuracy of deep learning methods is assessed for volumetric analysis (via segmentation) of the left ventricle in real-time free-breathing CMR at rest and under exercise stress. Data from healthy volunteers (n = 15) for cine and real-time free-breathing CMR at rest and under exercise stress were analyzed retrospectively. Exercise stress was performed using an ergometer in the supine position. Segmentations of two deep learning methods, a commercially available technique (comDL) and an openly available network (nnU-Net), were compared to a reference model created via the manual correction of segmentations obtained with comDL. Segmentations of left ventricular endocardium (LV), left ventricular myocardium (MYO), and right ventricle (RV) are compared for both end-systolic and end-diastolic phases and analyzed with Dice's coefficient. The volumetric analysis includes the cardiac function parameters LV end-diastolic volume (EDV), LV end-systolic volume (ESV), and LV ejection fraction (EF), evaluated with respect to both absolute and relative differences. For cine CMR, nnU-Net and comDL achieve Dice's coefficients above 0.95 for LV and 0.9 for MYO, and RV. For real-time CMR, the accuracy of nnU-Net exceeds that of comDL overall. For real-time CMR at rest, nnU-Net achieves Dice's coefficients of 0.94 for LV, 0.89 for MYO, and 0.90 for RV and the mean absolute differences between nnU-Net and the reference are 2.9 mL for EDV, 3.5 mL for ESV, and 2.6% for EF. For real-time CMR under exercise stress, nnU-Net achieves Dice's coefficients of 0.92 for LV, 0.85 for MYO, and 0.83 for RV and the mean absolute differences between nnU-Net and reference are 11.4 mL for EDV, 2.9 mL for ESV, and 3.6% for EF. Deep learning methods designed or trained for cine CMR segmentation can perform well on real-time CMR. For real-time free-breathing CMR at rest, the performance of deep learning methods is comparable to inter-observer variability in cine CMR and is usable for fully automatic segmentation. For real-time CMR under exercise stress, the performance of nnU-Net could promise a higher degree of automation in the future.
Collapse
Affiliation(s)
- Martin Schilling
- Institute for Diagnostic and Interventional Radiology, Universitätsmedizin Göttingen, Göttingen, Germany
| | - Christina Unterberg-Buchwald
- Institute for Diagnostic and Interventional Radiology, Universitätsmedizin Göttingen, Göttingen, Germany
- German Centre for Cardiovascular Research (DZHK), Partner Site Göttingen, Göttingen, Germany
- Clinic of Cardiology and Pneumology, Universitätsmedizin Göttingen, Göttingen, Germany
| | - Joachim Lotz
- Institute for Diagnostic and Interventional Radiology, Universitätsmedizin Göttingen, Göttingen, Germany
| | - Martin Uecker
- Institute for Diagnostic and Interventional Radiology, Universitätsmedizin Göttingen, Göttingen, Germany.
- German Centre for Cardiovascular Research (DZHK), Partner Site Göttingen, Göttingen, Germany.
- Institute of Biomedical Imaging, Graz University of Technology, Graz, Austria.
| |
Collapse
|
30
|
Pu Q, Xi Z, Yin S, Zhao Z, Zhao L. Advantages of transformer and its application for medical image segmentation: a survey. Biomed Eng Online 2024; 23:14. [PMID: 38310297 PMCID: PMC10838005 DOI: 10.1186/s12938-024-01212-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2023] [Accepted: 01/22/2024] [Indexed: 02/05/2024] Open
Abstract
PURPOSE Convolution operator-based neural networks have shown great success in medical image segmentation over the past decade. The U-shaped network with a codec structure is one of the most widely used models. Transformer, a technology used in natural language processing, can capture long-distance dependencies and has been applied in Vision Transformer to achieve state-of-the-art performance on image classification tasks. Recently, researchers have extended transformer to medical image segmentation tasks, resulting in good models. METHODS This review comprises publications selected through a Web of Science search. We focused on papers published since 2018 that applied the transformer architecture to medical image segmentation. We conducted a systematic analysis of these studies and summarized the results. RESULTS To better comprehend the benefits of convolutional neural networks and transformers, the construction of the codec and transformer modules is first explained. Second, the medical image segmentation model based on transformer is summarized. The typically used assessment markers for medical image segmentation tasks are then listed. Finally, a large number of medical segmentation datasets are described. CONCLUSION Even if there is a pure transformer model without any convolution operator, the sample size of medical picture segmentation still restricts the growth of the transformer, even though it can be relieved by a pretraining model. More often than not, researchers are still designing models using transformer and convolution operators.
Collapse
Affiliation(s)
- Qiumei Pu
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
| | - Zuoxin Xi
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
- CAS Key Laboratory for Biomedical Effects of Nanomaterials and Nanosafety Institute of High Energy Physics, Chinese Academy of Sciences, Beijing, 100049, China
| | - Shuai Yin
- School of Information Engineering, Minzu University of China, Beijing, 100081, China
| | - Zhe Zhao
- The Fourth Medical Center of PLA General Hospital, Beijing, 100039, China
| | - Lina Zhao
- CAS Key Laboratory for Biomedical Effects of Nanomaterials and Nanosafety Institute of High Energy Physics, Chinese Academy of Sciences, Beijing, 100049, China.
| |
Collapse
|
31
|
Jiao R, Zhang Y, Ding L, Xue B, Zhang J, Cai R, Jin C. Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation. Comput Biol Med 2024; 169:107840. [PMID: 38157773 DOI: 10.1016/j.compbiomed.2023.107840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 10/30/2023] [Accepted: 12/07/2023] [Indexed: 01/03/2024]
Abstract
Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches. Recent success of deep learning-based segmentation methods usually relies on a large amount of labeled data, which is particularly difficult and costly to obtain, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. Semi-supervised learning has emerged as an appealing strategy and been widely applied to medical image segmentation tasks to train deep models with limited annotations. In this paper, we present a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarize both the technical novelties and empirical results. Furthermore, we analyze and discuss the limitations and several unsolved problems of existing approaches. We hope this review can inspire the research community to explore solutions to this challenge and further advance the field of medical image segmentation.
Collapse
Affiliation(s)
- Rushi Jiao
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; School of Engineering Medicine, Beihang University, Beijing, 100191, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Yichi Zhang
- School of Data Science, Fudan University, Shanghai, 200433, China; Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, 200433, China.
| | - Le Ding
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China.
| | - Bingsen Xue
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Jicong Zhang
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China; Hefei Innovation Research Institute, Beihang University, Hefei, 230012, China.
| | - Rong Cai
- School of Engineering Medicine, Beihang University, Beijing, 100191, China; Key Laboratory for Biomechanics and Mechanobiology of Ministry of Education, Beihang University, Beijing, 100191, China.
| | - Cheng Jin
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China; Beijing Anding Hospital, Capital Medical University, Beijing, 100088, China.
| |
Collapse
|
32
|
Li F, Jiang A, Li M, Xiao C, Ji W. HPFG: semi-supervised medical image segmentation framework based on hybrid pseudo-label and feature-guiding. Med Biol Eng Comput 2024; 62:405-421. [PMID: 37875739 DOI: 10.1007/s11517-023-02946-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2023] [Accepted: 10/07/2023] [Indexed: 10/26/2023]
Abstract
Semi-supervised learning methods have been attracting much attention in medical image segmentation due to the lack of high-quality annotation. To cope with the noise problem of pseudo-label in semi-supervised medical image segmentation and the limitations of contrastive learning applications, we propose a semi-supervised medical image segmentation framework, HPFG, based on hybrid pseudo-label and feature-guiding, which consists of a hybrid pseudo-label strategy and two different feature-guiding modules. The hybrid pseudo-label strategy uses the CutMix operation and an auxiliary network to enable the labeled images to guide the unlabeled images to generate high-quality pseudo-label and reduce the impact of pseudo-label noise. In addition, a feature-guiding encoder module based on feature-level contrastive learning is designed to guide the encoder to mine useful local and global image features, thus effectively enhancing the feature extraction capability of the model. At the same time, a feature-guiding decoder module based on adaptive class-level contrastive learning is designed to guide the decoder in better extracting class information, achieving intra-class affinity and inter-class separation, and effectively alleviating the class imbalance problem in medical datasets. Extensive experimental results show that the segmentation performance of the HPFG framework proposed in this paper outperforms existing semi-supervised medical image segmentation methods on three public datasets: ACDC, LIDC, and ISIC. Code is available at https://github.com/fakerlove1/HPFG .
Collapse
Affiliation(s)
- Feixiang Li
- College of Computer Science and Technology, Taiyuan University of Technology, Jinzhong, 030600, China
| | - Ailian Jiang
- College of Computer Science and Technology, Taiyuan University of Technology, Jinzhong, 030600, China.
| | - Mengyang Li
- College of Computer Science and Technology, Taiyuan University of Technology, Jinzhong, 030600, China
| | - Cimei Xiao
- College of Computer Science and Technology, Taiyuan University of Technology, Jinzhong, 030600, China
| | - Wei Ji
- College of Computer Science and Technology, Taiyuan University of Technology, Jinzhong, 030600, China
| |
Collapse
|
33
|
Huang Y, Yang X, Liu L, Zhou H, Chang A, Zhou X, Chen R, Yu J, Chen J, Chen C, Liu S, Chi H, Hu X, Yue K, Li L, Grau V, Fan DP, Dong F, Ni D. Segment anything model for medical images? Med Image Anal 2024; 92:103061. [PMID: 38086235 DOI: 10.1016/j.media.2023.103061] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2023] [Revised: 09/28/2023] [Accepted: 12/05/2023] [Indexed: 01/12/2024]
Abstract
The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: (1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. (2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. (3) SAM performed better with manual hints, especially box, than the Everything mode. (4) SAM could help human annotation with high labeling quality and less time. (5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. (6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. (7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. (8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. Codes and models are available at: https://github.com/yuhoo0302/Segment-Anything-Model-for-Medical-Images. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.
Collapse
Affiliation(s)
- Yuhao Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Lian Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Han Zhou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Ao Chang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Xinrui Zhou
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Rusi Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Jiongquan Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Chaoyu Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Sijing Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | | | - Xindi Hu
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, China
| | - Kejuan Yue
- Hunan First Normal University, Changsha, China
| | - Lei Li
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Vicente Grau
- Department of Engineering Science, University of Oxford, Oxford, UK
| | - Deng-Ping Fan
- Computer Vision Lab (CVL), ETH Zurich, Zurich, Switzerland
| | - Fajin Dong
- Ultrasound Department, the Second Clinical Medical College, Jinan University, China; First Affiliated Hospital, Southern University of Science and Technology, Shenzhen People's Hospital, Shenzhen, China.
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China; Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China.
| |
Collapse
|
34
|
Dayarathna S, Islam KT, Uribe S, Yang G, Hayat M, Chen Z. Deep learning based synthesis of MRI, CT and PET: Review and analysis. Med Image Anal 2024; 92:103046. [PMID: 38052145 DOI: 10.1016/j.media.2023.103046] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 11/14/2023] [Accepted: 11/29/2023] [Indexed: 12/07/2023]
Abstract
Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis.
Collapse
Affiliation(s)
- Sanuwani Dayarathna
- Department of Data Science and AI, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia.
| | | | - Sergio Uribe
- Department of Medical Imaging and Radiation Sciences, Faculty of Medicine, Monash University, Clayton VIC 3800, Australia
| | - Guang Yang
- Bioengineering Department and Imperial-X, Imperial College London, W12 7SL, United Kingdom
| | - Munawar Hayat
- Department of Data Science and AI, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia
| | - Zhaolin Chen
- Department of Data Science and AI, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia; Monash Biomedical Imaging, Clayton VIC 3800, Australia
| |
Collapse
|
35
|
Qi X, He Y, Qi Y, Kong Y, Yang G, Li S. STANet: Spatio-Temporal Adaptive Network and Clinical Prior Embedding Learning for 3D+T CMR Segmentation. IEEE J Biomed Health Inform 2024; 28:881-892. [PMID: 38048234 DOI: 10.1109/jbhi.2023.3337521] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/06/2023]
Abstract
The segmentation of cardiac structure in magnetic resonance images (CMR) is paramount in diagnosing and managing cardiovascular illnesses, given its 3D+Time (3D+T) sequence. The existing deep learning methods are constrained in their ability to 3D+T CMR segmentation, due to: (1) Limited motion perception. The complexity of heart beating renders the motion perception in 3D+T CMR, including the long-range and cross-slice motions. The existing methods' local perception and slice-fixed perception directly limit the performance of 3D+T CMR perception. (2) Lack of labels. Due to the expensive labeling cost of the 3D+T CMR sequence, the labels of 3D+T CMR only contain the end-diastolic and end-systolic frames. The incomplete labeling scheme causes inefficient supervision. Hence, we propose a novel spatio-temporal adaptation network with clinical prior embedding learning (STANet) to ensure efficient spatio-temporal perception and optimization on 3D+T CMR segmentation. (1) A spatio-temporal adaptive convolution (STAC) treats the 3D+T CMR sequence as a whole for perception. The long-distance motion correlation is embedded into the structural perception by learnable weight regularization to balance long-range motion perception. The structural similarity is measured by cross-attention to adaptively correlate the cross-slice motion. (2) A clinical prior embedding learning strategy (CPE) is proposed to optimize the partially labeled 3D+T CMR segmentation dynamically by embedding clinical priors into optimization. STANet achieves outstanding performance with Dice of 0.917 and 0.94 on two public datasets (ACDC and STACOM), which indicates STANet has the potential to be incorporated into computer-aided diagnosis tools for clinical application.
Collapse
|
36
|
Gröschel J, Kuhnt J, Viezzer D, Hadler T, Hormes S, Barckow P, Schulz-Menger J, Blaszczyk E. Comparison of manual and artificial intelligence based quantification of myocardial strain by feature tracking-a cardiovascular MR study in health and disease. Eur Radiol 2024; 34:1003-1015. [PMID: 37594523 PMCID: PMC10853310 DOI: 10.1007/s00330-023-10127-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 06/27/2023] [Accepted: 07/04/2023] [Indexed: 08/19/2023]
Abstract
OBJECTIVES The analysis of myocardial deformation using feature tracking in cardiovascular MR allows for the assessment of global and segmental strain values. The aim of this study was to compare strain values derived from artificial intelligence (AI)-based contours with manually derived strain values in healthy volunteers and patients with cardiac pathologies. MATERIALS AND METHODS A cohort of 136 subjects (60 healthy volunteers and 76 patients; of those including 46 cases with left ventricular hypertrophy (LVH) of varying etiology and 30 cases with chronic myocardial infarction) was analyzed. Comparisons were based on quantitative strain analysis and on a geometric level by the Dice similarity coefficient (DSC) of the segmentations. Strain quantification was performed in 3 long-axis slices and short-axis (SAX) stack with epi- and endocardial contours in end-diastole. AI contours were checked for plausibility and potential errors in the tracking algorithm. RESULTS AI-derived strain values overestimated radial strain (+ 1.8 ± 1.7% (mean difference ± standard deviation); p = 0.03) and underestimated circumferential (- 0.8 ± 0.8%; p = 0.02) and longitudinal strain (- 0.1 ± 0.8%; p = 0.54). Pairwise group comparisons revealed no significant differences for global strain. The DSC showed good agreement for healthy volunteers (85.3 ± 10.3% for SAX) and patients (80.8 ± 9.6% for SAX). In 27 cases (27/76; 35.5%), a tracking error was found, predominantly (24/27; 88.9%) in the LVH group and 22 of those (22/27; 81.5%) at the insertion of the papillary muscle in lateral segments. CONCLUSIONS Strain analysis based on AI-segmented images shows good results in healthy volunteers and in most of the patient groups. Hypertrophied ventricles remain a challenge for contouring and feature tracking. CLINICAL RELEVANCE STATEMENT AI-based segmentations can help to streamline and standardize strain analysis by feature tracking. KEY POINTS • Assessment of strain in cardiovascular magnetic resonance by feature tracking can generate global and segmental strain values. • Commercially available artificial intelligence algorithms provide segmentation for strain analysis comparable to manual segmentation. • Hypertrophied ventricles are challenging in regards of strain analysis by feature tracking.
Collapse
Affiliation(s)
- Jan Gröschel
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany.
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany.
- DZHK (German Centre for Cardiovascular Research), Partner Site Berlin, Berlin, Germany.
| | - Johanna Kuhnt
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany
- DZHK (German Centre for Cardiovascular Research), Partner Site Berlin, Berlin, Germany
| | - Darian Viezzer
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany
- DZHK (German Centre for Cardiovascular Research), Partner Site Berlin, Berlin, Germany
| | - Thomas Hadler
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany
- DZHK (German Centre for Cardiovascular Research), Partner Site Berlin, Berlin, Germany
| | - Sophie Hormes
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany
| | | | - Jeanette Schulz-Menger
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany
- DZHK (German Centre for Cardiovascular Research), Partner Site Berlin, Berlin, Germany
| | - Edyta Blaszczyk
- Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt-Universität Zu Berlin, Berlin, Germany.
- Working Group On Cardiovascular Magnetic Resonance, Experimental and Clinical Research Center, a joint cooperation between the Charité Medical Faculty and the Max-Delbrück Center for Molecular Medicine and HELIOS Hospital Berlin-Buch, Department of Cardiology and Nephrology, Berlin, Germany.
- DZHK (German Centre for Cardiovascular Research), Partner Site Berlin, Berlin, Germany.
| |
Collapse
|
37
|
Bourazana A, Xanthopoulos A, Briasoulis A, Magouliotis D, Spiliopoulos K, Athanasiou T, Vassilopoulos G, Skoularigis J, Triposkiadis F. Artificial Intelligence in Heart Failure: Friend or Foe? Life (Basel) 2024; 14:145. [PMID: 38276274 PMCID: PMC10817517 DOI: 10.3390/life14010145] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Revised: 01/08/2024] [Accepted: 01/17/2024] [Indexed: 01/27/2024] Open
Abstract
In recent times, there have been notable changes in cardiovascular medicine, propelled by the swift advancements in artificial intelligence (AI). The present work provides an overview of the current applications and challenges of AI in the field of heart failure. It emphasizes the "garbage in, garbage out" issue, where AI systems can produce inaccurate results with skewed data. The discussion covers issues in heart failure diagnostic algorithms, particularly discrepancies between existing models. Concerns about the reliance on the left ventricular ejection fraction (LVEF) for classification and treatment are highlighted, showcasing differences in current scientific perceptions. This review also delves into challenges in implementing AI, including variable considerations and biases in training data. It underscores the limitations of current AI models in real-world scenarios and the difficulty in interpreting their predictions, contributing to limited physician trust in AI-based models. The overarching suggestion is that AI can be a valuable tool in clinicians' hands for treating heart failure patients, as far as existing medical inaccuracies have been addressed before integrating AI into these frameworks.
Collapse
Affiliation(s)
- Angeliki Bourazana
- Department of Cardiology, University Hospital of Larissa, 41110 Larissa, Greece
| | - Andrew Xanthopoulos
- Department of Cardiology, University Hospital of Larissa, 41110 Larissa, Greece
| | - Alexandros Briasoulis
- Division of Cardiovascular Medicine, Section of Heart Failure and Transplantation, University of Iowa, Iowa City, IA 52242, USA
| | - Dimitrios Magouliotis
- Department of Cardiothoracic Surgery, University of Thessaly, 41110 Larissa, Greece; (D.M.); (K.S.)
| | - Kyriakos Spiliopoulos
- Department of Cardiothoracic Surgery, University of Thessaly, 41110 Larissa, Greece; (D.M.); (K.S.)
| | - Thanos Athanasiou
- Department of Surgery and Cancer, Imperial College London, St Mary’s Hospital, London W2 1NY, UK
| | - George Vassilopoulos
- Department of Hematology, University Hospital of Larissa, University of Thessaly Medical School, 41110 Larissa, Greece
| | - John Skoularigis
- Department of Cardiology, University Hospital of Larissa, 41110 Larissa, Greece
| | | |
Collapse
|
38
|
Wang Z, Li B, Yu H, Zhang Z, Ran M, Xia W, Yang Z, Lu J, Chen H, Zhou J, Shan H, Zhang Y. Promoting fast MR imaging pipeline by full-stack AI. iScience 2024; 27:108608. [PMID: 38174317 PMCID: PMC10762466 DOI: 10.1016/j.isci.2023.108608] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2023] [Revised: 10/17/2023] [Accepted: 11/29/2023] [Indexed: 01/05/2024] Open
Abstract
Magnetic resonance imaging (MRI) is a widely used imaging modality in clinics for medical disease diagnosis, staging, and follow-up. Deep learning has been extensively used to accelerate k-space data acquisition, enhance MR image reconstruction, and automate tissue segmentation. However, these three tasks are usually treated as independent tasks and optimized for evaluation by radiologists, thus ignoring the strong dependencies among them; this may be suboptimal for downstream intelligent processing. Here, we present a novel paradigm, full-stack learning (FSL), which can simultaneously solve these three tasks by considering the overall imaging process and leverage the strong dependence among them to further improve each task, significantly boosting the efficiency and efficacy of practical MRI workflows. Experimental results obtained on multiple open MR datasets validate the superiority of FSL over existing state-of-the-art methods on each task. FSL has great potential to optimize the practical workflow of MRI for medical diagnosis and radiotherapy.
Collapse
Affiliation(s)
- Zhiwen Wang
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Bowen Li
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Hui Yu
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Zhongzhou Zhang
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Maosong Ran
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Wenjun Xia
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Ziyuan Yang
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Jingfeng Lu
- School of Cyber Science and Engineering, Sichuan University, Chengdu, Sichuan, China
| | - Hu Chen
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Jiliu Zhou
- School of Computer Science, Sichuan University, Chengdu, Sichuan, China
| | - Hongming Shan
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai, China
| | - Yi Zhang
- School of Cyber Science and Engineering, Sichuan University, Chengdu, Sichuan, China
| |
Collapse
|
39
|
Shao L, Chen B, Zhang Z, Zhang Z, Chen X. Artificial intelligence generated content (AIGC) in medicine: A narrative review. Math Biosci Eng 2024; 21:1672-1711. [PMID: 38303483 DOI: 10.3934/mbe.2024073] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/03/2024]
Abstract
Recently, artificial intelligence generated content (AIGC) has been receiving increased attention and is growing exponentially. AIGC is generated based on the intentional information extracted from human-provided instructions by generative artificial intelligence (AI) models. AIGC quickly and automatically generates large amounts of high-quality content. Currently, there is a shortage of medical resources and complex medical procedures in medicine. Due to its characteristics, AIGC can help alleviate these problems. As a result, the application of AIGC in medicine has gained increased attention in recent years. Therefore, this paper provides a comprehensive review on the recent state of studies involving AIGC in medicine. First, we present an overview of AIGC. Furthermore, based on recent studies, the application of AIGC in medicine is reviewed from two aspects: medical image processing and medical text generation. The basic generative AI models, tasks, target organs, datasets and contribution of studies are considered and summarized. Finally, we also discuss the limitations and challenges faced by AIGC and propose possible solutions with relevant studies. We hope this review can help readers understand the potential of AIGC in medicine and obtain some innovative ideas in this field.
Collapse
Affiliation(s)
- Liangjing Shao
- Academy for Engineering & Technology, Fudan University, Shanghai 200433, China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University, Shanghai 200032, China
| | - Benshuang Chen
- Academy for Engineering & Technology, Fudan University, Shanghai 200433, China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University, Shanghai 200032, China
| | - Ziqun Zhang
- Information office, Fudan University, Shanghai 200032, China
| | - Zhen Zhang
- Baoshan Branch of Ren Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200444, China
| | - Xinrong Chen
- Academy for Engineering & Technology, Fudan University, Shanghai 200433, China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Fudan University, Shanghai 200032, China
| |
Collapse
|
40
|
Kato S, Hotta K. Adaptive t-vMF dice loss: An effective expansion of dice loss for medical image segmentation. Comput Biol Med 2024; 168:107695. [PMID: 38061152 DOI: 10.1016/j.compbiomed.2023.107695] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Revised: 10/30/2023] [Accepted: 11/06/2023] [Indexed: 01/10/2024]
Abstract
Dice loss is widely used for medical image segmentation, and many improved loss functions have been proposed. However, further Dice loss improvements are still possible. In this study, we reconsidered the use of Dice loss and discovered that Dice loss can be rewritten in the loss function using the cosine similarity through a simple equation transformation. Using this knowledge, we present a novel t-vMF Dice loss based on the t-vMF similarity instead of the cosine similarity. Based on the t-vMF similarity, our proposed Dice loss is formulated in a more compact similarity loss function than the original Dice loss. Furthermore, we present an effective algorithm that automatically determines the parameter κ for the t-vMF similarity using a validation accuracy, called Adaptive t-vMF Dice loss. Using this algorithm, it is possible to apply more compact similarities for easy classes and wider similarities for difficult classes, and we are able to achieve adaptive training based on the accuracy of each class. We evaluated binary segmentation datasets of CVC-ClinicDB and Kvasir-SEG, and multi-class segmentation datasets of Automated Cardiac Diagnosis Challenge and Synapse multi-organ segmentation. Through experiments conducted on four datasets using a five-fold cross-validation, we confirmed that the Dice score coefficient (DSC) was further improved in comparison with the original Dice loss and other loss functions.
Collapse
Affiliation(s)
- Sota Kato
- Department of Electrical, Information, Materials and Materials Engineering, Meijo University, Tempaku-ku, Nagoya, 468-8502, Aichi, Japan.
| | - Kazuhiro Hotta
- Department of Electrical and Electronic Engineering, Meijo University, Nagoya, Japan.
| |
Collapse
|
41
|
Das N, Das S. Attention-UNet architectures with pretrained backbones for multi-class cardiac MR image segmentation. Curr Probl Cardiol 2024; 49:102129. [PMID: 37866419 DOI: 10.1016/j.cpcardiol.2023.102129] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/11/2023] [Revised: 10/05/2023] [Accepted: 10/14/2023] [Indexed: 10/24/2023]
Abstract
Segmentation architectures based on deep learning proficient extraordinary results in medical imaging technologies. Computed tomography (CT) images and Magnetic Resonance Imaging (MRI) in diagnosis and treatment are increasing and significantly support the diagnostic process by removing the bottlenecks of manual segmentation. Cardiac Magnetic Resonance Imaging (CMRI) is a state-of-the-art imaging technique used to acquire vital heart measurements and has received extensive attention from researchers for automatic segmentation. Deep learning methods offer high-precision segmentation but still pose several difficulties, such as pixel homogeneity in nearby organs. The motivated study using the attention mechanism approach was introduced for medical images for automated algorithms. The experiment focuses on observing the impact of the attention mechanism with and without pretrained backbone networks on the UNet model. For the same, three networks are considered: Attention-UNet, Attention-UNet with resnet50 pretrained backbone and Attention-UNet with densenet121 pretrained backbone. The experiments are performed on the ACDC Challenge 2017 dataset. The performance is evaluated by conducting a comparative analysis based on the Dice Coefficient, IoU Coefficient, and cross-entropy loss calculations. The Attention-UNet, Attention-UNet with resnet50 pretrained backbone, and Attention-UNet with densenet121 pretrained backbone networks obtained Dice Coefficients of 0.9889, 0.9720, and 0.9801, respectively, along with corresponding IoU scores of 0.9781, 0.9457, and 0.9612. Results compared with the state-of-the-art methods indicate that the methods are on par with, or even superior in terms of both the Dice coefficient and Intersection-over-union.
Collapse
Affiliation(s)
- Niharika Das
- Department of Mathematics & Computer Application, Maulana Azad National Institute of Technology, India.
| | - Sujoy Das
- Department of Mathematics & Computer Application, Maulana Azad National Institute of Technology, India
| |
Collapse
|
42
|
Azad R, Kazerouni A, Heidari M, Aghdam EK, Molaei A, Jia Y, Jose A, Roy R, Merhof D. Advances in medical image analysis with vision Transformers: A comprehensive review. Med Image Anal 2024; 91:103000. [PMID: 37883822 DOI: 10.1016/j.media.2023.103000] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Revised: 09/30/2023] [Accepted: 10/11/2023] [Indexed: 10/28/2023]
Abstract
The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.
Collapse
Affiliation(s)
- Reza Azad
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Amirhossein Kazerouni
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Moein Heidari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | | | - Amirali Molaei
- School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Yiwei Jia
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Abin Jose
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Rijo Roy
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | - Dorit Merhof
- Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| |
Collapse
|
43
|
Yao T, St. Clair N, Miller GF, Dorfman AL, Fogel MA, Ghelani S, Krishnamurthy R, Lam CZ, Quail M, Robinson JD, Schidlow D, Slesnick TC, Weigand J, Steeden JA, Rathod RH, Muthurangu V. A Deep Learning Pipeline for Assessing Ventricular Volumes from a Cardiac MRI Registry of Patients with Single Ventricle Physiology. Radiol Artif Intell 2024; 6:e230132. [PMID: 38166332 PMCID: PMC10831511 DOI: 10.1148/ryai.230132] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2023] [Revised: 10/05/2023] [Accepted: 10/30/2023] [Indexed: 01/04/2024]
Abstract
Purpose To develop an end-to-end deep learning (DL) pipeline for automated ventricular segmentation of cardiac MRI data from a multicenter registry of patients with Fontan circulation (Fontan Outcomes Registry Using CMR Examinations [FORCE]). Materials and Methods This retrospective study used 250 cardiac MRI examinations (November 2007-December 2022) from 13 institutions for training, validation, and testing. The pipeline contained three DL models: a classifier to identify short-axis cine stacks and two U-Net 3+ models for image cropping and segmentation. The automated segmentations were evaluated on the test set (n = 50) by using the Dice score. Volumetric and functional metrics derived from DL and ground truth manual segmentations were compared using Bland-Altman and intraclass correlation analysis. The pipeline was further qualitatively evaluated on 475 unseen examinations. Results There were acceptable limits of agreement (LOA) and minimal biases between the ground truth and DL end-diastolic volume (EDV) (bias: -0.6 mL/m2, LOA: -20.6 to 19.5 mL/m2) and end-systolic volume (ESV) (bias: -1.1 mL/m2, LOA: -18.1 to 15.9 mL/m2), with high intraclass correlation coefficients (ICCs > 0.97) and Dice scores (EDV, 0.91 and ESV, 0.86). There was moderate agreement for ventricular mass (bias: -1.9 g/m2, LOA: -17.3 to 13.5 g/m2) and an ICC of 0.94. There was also acceptable agreement for stroke volume (bias: 0.6 mL/m2, LOA: -17.2 to 18.3 mL/m2) and ejection fraction (bias: 0.6%, LOA: -12.2% to 13.4%), with high ICCs (>0.81). The pipeline achieved satisfactory segmentation in 68% of the 475 unseen examinations, while 26% needed minor adjustments, 5% needed major adjustments, and in 0.4%, the cropping model failed. Conclusion The DL pipeline can provide fast standardized segmentation for patients with single ventricle physiology across multiple centers. This pipeline can be applied to all cardiac MRI examinations in the FORCE registry. Keywords: Cardiac, Adults and Pediatrics, MR Imaging, Congenital, Volume Analysis, Segmentation, Quantification Supplemental material is available for this article. © RSNA, 2023.
Collapse
Affiliation(s)
| | | | - Gabriel F. Miller
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Adam L. Dorfman
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Mark A. Fogel
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Sunil Ghelani
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Rajesh Krishnamurthy
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Christopher Z. Lam
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Michael Quail
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Joshua D. Robinson
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - David Schidlow
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Timothy C. Slesnick
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Justin Weigand
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Jennifer A. Steeden
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Rahul H. Rathod
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| | - Vivek Muthurangu
- From the Institutes of Health Informatics (T.Y.) and Cardiovascular Science (M.Q., J.A.S., V.M.), University College London, 20c Guilford Street, London WC1N 1DZ, England; Department of Cardiology, Boston Children's Hospital, Boston, Mass (N.S.C., G.F.M., S.G., D.S., R.H.R.); Department of Pediatrics, University of Michigan, Ann Arbor, Mich (A.L.D.); Division of Cardiology, The Children's Hospital of Philadelphia, Philadelphia, Pa (M.A.F.); Department of Radiology, Nationwide Children's Hospital, Columbus, Ohio (R.K.); Department of Diagnostic Imaging, Hospital for Sick Children, Toronto, Canada (C.Z.L.); Department of Pediatrics, Ann and Robert H Lurie Children's Hospital of Chicago, Chicago, Ill (J.D.R.); Department of Pediatric Cardiology, Emory University School of Medicine, Atlanta, Ga (T.C.S.); and Department of Cardiology, Texas Children's Hospital, Houston, Tex (J.W.)
| |
Collapse
|
44
|
Morales MA, Manning WJ, Nezafat R. Present and Future Innovations in AI and Cardiac MRI. Radiology 2024; 310:e231269. [PMID: 38193835 PMCID: PMC10831479 DOI: 10.1148/radiol.231269] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2023] [Revised: 10/21/2023] [Accepted: 10/26/2023] [Indexed: 01/10/2024]
Abstract
Cardiac MRI is used to diagnose and treat patients with a multitude of cardiovascular diseases. Despite the growth of clinical cardiac MRI, complicated image prescriptions and long acquisition protocols limit the specialty and restrain its impact on the practice of medicine. Artificial intelligence (AI)-the ability to mimic human intelligence in learning and performing tasks-will impact nearly all aspects of MRI. Deep learning (DL) primarily uses an artificial neural network to learn a specific task from example data sets. Self-driving scanners are increasingly available, where AI automatically controls cardiac image prescriptions. These scanners offer faster image collection with higher spatial and temporal resolution, eliminating the need for cardiac triggering or breath holding. In the future, fully automated inline image analysis will most likely provide all contour drawings and initial measurements to the reader. Advanced analysis using radiomic or DL features may provide new insights and information not typically extracted in the current analysis workflow. AI may further help integrate these features with clinical, genetic, wearable-device, and "omics" data to improve patient outcomes. This article presents an overview of AI and its application in cardiac MRI, including in image acquisition, reconstruction, and processing, and opportunities for more personalized cardiovascular care through extraction of novel imaging markers.
Collapse
Affiliation(s)
- Manuel A. Morales
- From the Department of Medicine, Cardiovascular Division (M.A.M.,
W.J.M., R.N.), and Department of Radiology (W.J.M.), Beth Israel Deaconess
Medical Center and Harvard Medical School, 330 Brookline Ave, Boston, MA
02215
| | - Warren J. Manning
- From the Department of Medicine, Cardiovascular Division (M.A.M.,
W.J.M., R.N.), and Department of Radiology (W.J.M.), Beth Israel Deaconess
Medical Center and Harvard Medical School, 330 Brookline Ave, Boston, MA
02215
| | - Reza Nezafat
- From the Department of Medicine, Cardiovascular Division (M.A.M.,
W.J.M., R.N.), and Department of Radiology (W.J.M.), Beth Israel Deaconess
Medical Center and Harvard Medical School, 330 Brookline Ave, Boston, MA
02215
| |
Collapse
|
45
|
Li H, Ding J, Shi X, Zhang Q, Yu P, Li H. D-SAT: dual semantic aggregation transformer with dual attention for medical image segmentation. Phys Med Biol 2023; 69:015013. [PMID: 37607559 DOI: 10.1088/1361-6560/acf2e5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/15/2023] [Accepted: 08/22/2023] [Indexed: 08/24/2023]
Abstract
Objective. Medical image segmentation is significantly essential to assist clinicians in facilitating a quick and accurate diagnoses. However, most of the existing methods are still challenged by the loss of semantic information, blurred boundaries and the huge semantic gap between the encoder and decoder.Approach. To tackle these issues, a dual semantic aggregation transformer with dual attention is proposed for medical image segmentation. Firstly, the dual-semantic feature aggregation module is designed to build a bridge between convolutional neural network (CNN) and Transformer, effectively aggregating CNN's local feature detail ability and Transformer's long-range modeling ability to mitigate semantic information loss. Thereafter, the strip spatial attention mechanism is put forward to alleviate the blurred boundaries during encoding by constructing pixel-level feature relations across CSWin Transformer blocks from different spatial dimensions. Finally, a feature distribution gated attention module is constructed in the skip connection between the encoder and decoder to decrease the large semantic gap by filtering out the noise in low-level semantic information when fusing low-level and high-level semantic features during decoding.Main results. Comprehensive experiments conducted on abdominal multi-organ segmentation, cardiac diagnosis, polyp segmentation and skin lesion segmentation serve to validate the generalization and effectiveness of the proposed dual semantic aggregation transformer with dual attention (D-SAT). The superiority of D-SAT over current state-of-the-art methods is substantiated by both subjective and objective evaluations, revealing its remarkable performance in terms of segmentation accuracy and quality.Significance. The proposed method subtly preserves the local feature details and global context information in medical image segmentation, providing valuable support to improve diagnostic efficiency for clinicians and early disease control for patients. Code is available athttps://github.com/Dxkm/D-SAT.
Collapse
Affiliation(s)
- Haiyan Li
- School of Information Science and Engineering, Yunnan University, Kunming 650504, People's Republic of China
| | - Jiayu Ding
- School of Information Science and Engineering, Yunnan University, Kunming 650504, People's Republic of China
| | - Xin Shi
- Department of Urology Surgery, The Second Hospital Affiliated to the Medical University of Kunming, Kunming, People's Republic of China
- Faculty of Life Science and Technology, Kunming University of Science and Technology, Kunming, People's Republic of China
| | - Qi Zhang
- School of Environmental and Chemical Engineering, Kunming Metallurgy College, Kunming, People's Republic of China
| | - Pengfei Yu
- School of Information Science and Engineering, Yunnan University, Kunming 650504, People's Republic of China
| | - Hongsong Li
- School of Information Science and Engineering, Yunnan University, Kunming 650504, People's Republic of China
| |
Collapse
|
46
|
Muffoletto M, Xu H, Kunze KP, Neji R, Botnar R, Prieto C, Rückert D, Young AA. Combining generative modelling and semi-supervised domain adaptation for whole heart cardiovascular magnetic resonance angiography segmentation. J Cardiovasc Magn Reson 2023; 25:80. [PMID: 38124106 PMCID: PMC10734115 DOI: 10.1186/s12968-023-00981-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2023] [Accepted: 11/12/2023] [Indexed: 12/23/2023] Open
Abstract
BACKGROUND Quantification of three-dimensional (3D) cardiac anatomy is important for the evaluation of cardiovascular diseases. Changes in anatomy are indicative of remodeling processes as the heart tissue adapts to disease. Although robust segmentation methods exist for computed tomography angiography (CTA), few methods exist for whole-heart cardiovascular magnetic resonance angiograms (CMRA) which are more challenging due to variable contrast, lower signal to noise ratio and a limited amount of labeled data. METHODS Two state-of-the-art unsupervised generative deep learning domain adaptation architectures, generative adversarial networks and variational auto-encoders, were applied to 3D whole heart segmentation of both conventional (n = 20) and high-resolution (n = 45) CMRA (target) images, given segmented CTA (source) images for training. An additional supervised loss function was implemented to improve performance given 10%, 20% and 30% segmented CMRA cases. A fully supervised nn-UNet trained on the given CMRA segmentations was used as the benchmark. RESULTS The addition of a small number of segmented CMRA training cases substantially improved performance in both generative architectures in both standard and high-resolution datasets. Compared with the nn-UNet benchmark, the generative methods showed substantially better performance in the case of limited labelled cases. On the standard CMRA dataset, an average 12% (adversarial method) and 10% (variational method) improvement in Dice score was obtained. CONCLUSIONS Unsupervised domain-adaptation methods for CMRA segmentation can be boosted by the addition of a small number of supervised target training cases. When only few labelled cases are available, semi-supervised generative modelling is superior to supervised methods.
Collapse
Affiliation(s)
- Marica Muffoletto
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK.
| | - Hao Xu
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Karl P Kunze
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
- MR Research Collaborations, Siemens Healthcare Limited, Frimley, UK
| | - Radhouene Neji
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
- MR Research Collaborations, Siemens Healthcare Limited, Frimley, UK
| | - René Botnar
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Claudia Prieto
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| | - Daniel Rückert
- Biomedical Image Analysis Group, Department of Computing, Imperial College London, London, UK
- Institute for Artificial Intelligence and Informatics in Medicine, Klinikum Rechts der Isar, Technical University of Munich, Munich, Germany
| | - Alistair A Young
- School of Biomedical Engineering and Imaging Sciences, King's College, St Thomas' Hospital, 4th Floor Lambeth Wing, Westminster Bridge, London, SW1 7EH, UK
| |
Collapse
|
47
|
Li K, Zhang G, Li K, Li J, Wang J, Yang Y. Dual CNN cross-teaching semi-supervised segmentation network with multi-kernels and global contrastive loss in ACDC. Med Biol Eng Comput 2023; 61:3409-3417. [PMID: 37684494 DOI: 10.1007/s11517-023-02920-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2023] [Accepted: 08/22/2023] [Indexed: 09/10/2023]
Abstract
The cross-teaching based on Convolutional Neural Network (CNN) and Transformer has been successful in semi-supervised learning; however, the information interaction between local and global relations ignores the semantic features of the medium scale, and at the same time, the information in the process of feature coding is not fully utilized. To solve these problems, we proposed a new semi-supervised segmentation network. Based on the principle of complementary modeling information of different kernel convolutions, we design a dual CNN cross-supervised network with different kernel sizes under cross-teaching. We introduce global feature contrastive learning and generate contrast samples with the help of dual CNN architecture to make efficient use of coding features. We conducted plenty of experiments on the Automated Cardiac Diagnosis Challenge (ACDC) dataset to evaluate our approach. Our method achieves an average Dice Similarity Coefficient (DSC) of 87.2% and Hausdorff distance ([Formula: see text]) of 6.1 mm on 10% labeled data, which is significantly improved compared with many current popular models. Supervised learning is performed on the labeled data, and dual CNN cross-teaching supervised learning is performed on the unlabeled data. All data would be mapped by the two CNNs to generate features, which are used for contrastive learning to optimize the parameters.
Collapse
Affiliation(s)
- Keming Li
- School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan, China
| | - Guangyuan Zhang
- School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan, China
| | - Kefeng Li
- School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan, China.
| | - Jindi Li
- School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan, China
| | - Jiaqi Wang
- School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan, China
| | - Yumin Yang
- School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan, China
| |
Collapse
|
48
|
Xu X, Jia Q, Yuan H, Qiu H, Dong Y, Xie W, Yao Z, Zhang J, Nie Z, Li X, Shi Y, Zou JY, Huang M, Zhuang J. A clinically applicable AI system for diagnosis of congenital heart diseases based on computed tomography images. Med Image Anal 2023; 90:102953. [PMID: 37734140 DOI: 10.1016/j.media.2023.102953] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2022] [Revised: 08/22/2023] [Accepted: 09/01/2023] [Indexed: 09/23/2023]
Abstract
Congenital heart disease (CHD) is the most common type of birth defect. Without timely detection and treatment, approximately one-third of children with CHD would die in the infant period. However, due to the complicated heart structures, early diagnosis of CHD and its types is quite challenging, even for experienced radiologists. Here, we present an artificial intelligence (AI) system that achieves a comparable performance of human experts in the critical task of classifying 17 categories of CHD types. We collected the first-large CT dataset from three different CT machines, including more than 3750 CHD patients over 14 years. Experimental results demonstrate that it can achieve diagnosis accuracy (86.03%) comparable with junior cardiovascular radiologists (86.27%) in a World Health Organization appointed research and cooperation center in China on most types of CHD, and obtains a higher sensitivity (82.91%) than junior cardiovascular radiologists (76.18%). The accuracy of the combination of our AI system (97.20%) and senior radiologists achieves comparable results to that of junior radiologists and senior radiologists (97.16%) which is the current clinical routine. Our AI system can further provide 3D visualization of hearts to senior radiologists for interpretation and flexible review, surgeons for precise intuition of heart structures, and clinicians for more precise outcome prediction. We demonstrate the potential of our model to be integrated into current clinic practice to improve the diagnosis of CHD globally, especially in regions where experienced radiologists can be scarce.
Collapse
Affiliation(s)
- Xiaowei Xu
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Qianjun Jia
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Catheterization Lab, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China
| | - Haiyun Yuan
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Hailong Qiu
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Yuhao Dong
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Catheterization Lab, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China
| | - Wen Xie
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Zeyang Yao
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Jiawei Zhang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Zhiqaing Nie
- Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China
| | - Xiaomeng Li
- Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong Special Administrative Region
| | - Yiyu Shi
- Computer Science and Engineering, University of Notre Dame, IN, 46656, USA
| | - James Y Zou
- Department of Computer Science, Stanford University, Stanford, CA, 94305, USA; Department of Electrical Engineering, Stanford University, Stanford, CA, 94305, USA.
| | - Meiping Huang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Catheterization Lab, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, China.
| | - Jian Zhuang
- Guangdong Provincial Key Laboratory of South China Structural Heart Disease, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Guangdong Cardiovascular Institute, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China; Department of Cardiovascular Surgery, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Southern Medical University, Guangzhou, 510080, China.
| |
Collapse
|
49
|
Li X, Qin X, Huang C, Lu Y, Cheng J, Wang L, Liu O, Shuai J, Yuan CA. SUnet: A multi-organ segmentation network based on multiple attention. Comput Biol Med 2023; 167:107596. [PMID: 37890423 DOI: 10.1016/j.compbiomed.2023.107596] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2023] [Revised: 09/13/2023] [Accepted: 10/17/2023] [Indexed: 10/29/2023]
Abstract
Organ segmentation in abdominal or thoracic computed tomography (CT) images plays a crucial role in medical diagnosis as it enables doctors to locate and evaluate organ abnormalities quickly, thereby guiding surgical planning, and aiding treatment decision-making. This paper proposes a novel and efficient medical image segmentation method called SUnet for multi-organ segmentation in the abdomen and thorax. SUnet is a fully attention-based neural network. Firstly, an efficient spatial reduction attention (ESRA) module is introduced not only to extract image features better, but also to reduce overall model parameters, and to alleviate overfitting. Secondly, SUnet's multiple attention-based feature fusion module enables effective cross-scale feature integration. Additionally, an enhanced attention gate (EAG) module is considered by using grouped convolution and residual connections, providing richer semantic features. We evaluate the performance of the proposed model on synapse multiple organ segmentation dataset and automated cardiac diagnostic challenge dataset. SUnet achieves an average Dice of 84.29% and 92.25% on these two datasets, respectively, outperforming other models of similar complexity and size, and achieving state-of-the-art results.
Collapse
Affiliation(s)
- Xiaosen Li
- School of Artificial Intelligence, Guangxi Minzu University, Nanning, 530006, China; Wenzhou Institute, University of Chinese Academy of Sciences, Wenzhou, 325105, China
| | - Xiao Qin
- Guangxi Key Lab of Human-machine Interaction and Intelligent Decision, Nanning Normal University, Nanning, 530023, China
| | - Chengliang Huang
- Academy of Artificial Intelligence, Zhejiang Dongfang Polytechnic, Wenzhou, 325025, China
| | - Yuer Lu
- Wenzhou Institute, University of Chinese Academy of Sciences, Wenzhou, 325105, China
| | - Jinyan Cheng
- Wenzhou Institute, University of Chinese Academy of Sciences, Wenzhou, 325105, China
| | - Liansheng Wang
- Department of Computer Science, Xiamen University, Xiamen, 361005, China
| | - Ou Liu
- Wenzhou Institute, University of Chinese Academy of Sciences, Wenzhou, 325105, China
| | - Jianwei Shuai
- Wenzhou Institute, University of Chinese Academy of Sciences, Wenzhou, 325105, China.
| | - Chang-An Yuan
- Guangxi Key Lab of Human-machine Interaction and Intelligent Decision, Nanning Normal University, Nanning, 530023, China; Guangxi Academy of Science, Nanning, 530007, China.
| |
Collapse
|
50
|
Liu Z, Li H, Li W, Zhang F, Ouyang W, Wang S, Zhi A, Pan X. Development of an Expert-Level Right Ventricular Abnormality Detection Algorithm Based on Deep Learning. Interdiscip Sci 2023; 15:653-662. [PMID: 37470945 DOI: 10.1007/s12539-023-00581-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2023] [Revised: 07/06/2023] [Accepted: 07/10/2023] [Indexed: 07/21/2023]
Abstract
PURPOSE Studies relating to the right ventricle (RV) are inadequate, and specific diagnostic algorithms still need to be improved. This essay is designed to make exploration and verification on an algorithm of deep learning based on imaging and clinical data to detect RV abnormalities. METHODS The Automated Cardiac Diagnosis Challenge dataset includes 20 subjects with RV abnormalities (an RV cavity volume which is higher than 110 mL/m2 or RV ejection fraction which is lower than 40%) and 20 normal subjects who suffered from both cardiac MRI. The subjects were separated into training and validation sets in a ratio of 7:3 and were modeled by utilizing a nerve net of deep-learning and six machine-learning algorithms. Eight MRI specialists from multiple centers independently determined whether each subject in the validation group had RV abnormalities. Model performance was evaluated based on the AUC, accuracy, recall, sensitivity and specificity. Furthermore, a preliminary assessment of patient disease risk was performed based on clinical information using a nomogram. RESULTS The deep-learning neural network outperformed the other six machine-learning algorithms, with an AUC value of 1 (95% confidence interval: 1-1) on both training group and validation group. This algorithm surpassed most human experts (87.5%). In addition, the nomogram model could evaluate a population with a disease risk of 0.2-0.8. CONCLUSIONS A deep-learning algorithm could effectively identify patients with RV abnormalities. This AI algorithm developed specifically for right ventricular abnormalities will improve the detection of right ventricular abnormalities at all levels of care units and facilitate the timely diagnosis and treatment of related diseases. In addition, this study is the first to validate the algorithm's ability to classify RV abnormalities by comparing it with human experts.
Collapse
Affiliation(s)
- Zeye Liu
- Department of Structural Heart Disease, National Center for Cardiovascular Disease, China and Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100037, China
- National Health Commission Key Laboratory of Cardiovascular Regeneration Medicine, Beijing, 100037, China
- Key Laboratory of Innovative Cardiovascular Devices, Chinese Academy of Medical Sciences, Beijing, 100037, China
- National Clinical Research Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences, Beijing, 100037, China
| | - Hang Li
- Department of Structural Heart Disease, National Center for Cardiovascular Disease, China and Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100037, China
- National Health Commission Key Laboratory of Cardiovascular Regeneration Medicine, Beijing, 100037, China
- Key Laboratory of Innovative Cardiovascular Devices, Chinese Academy of Medical Sciences, Beijing, 100037, China
- National Clinical Research Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences, Beijing, 100037, China
| | - Wenchao Li
- Pediatric Cardiac Surgery, Henan Provincial People's Hospital, Huazhong Fuwai Hospital, Zhengzhou University People's Hospital, Zhengzhou, 450000, China
| | - Fengwen Zhang
- Department of Structural Heart Disease, National Center for Cardiovascular Disease, China and Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100037, China
- National Health Commission Key Laboratory of Cardiovascular Regeneration Medicine, Beijing, 100037, China
- Key Laboratory of Innovative Cardiovascular Devices, Chinese Academy of Medical Sciences, Beijing, 100037, China
- National Clinical Research Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences, Beijing, 100037, China
| | - Wenbin Ouyang
- Department of Structural Heart Disease, National Center for Cardiovascular Disease, China and Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100037, China
- National Health Commission Key Laboratory of Cardiovascular Regeneration Medicine, Beijing, 100037, China
- Key Laboratory of Innovative Cardiovascular Devices, Chinese Academy of Medical Sciences, Beijing, 100037, China
- National Clinical Research Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences, Beijing, 100037, China
| | - Shouzheng Wang
- Department of Structural Heart Disease, National Center for Cardiovascular Disease, China and Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100037, China
- National Health Commission Key Laboratory of Cardiovascular Regeneration Medicine, Beijing, 100037, China
- Key Laboratory of Innovative Cardiovascular Devices, Chinese Academy of Medical Sciences, Beijing, 100037, China
- National Clinical Research Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences, Beijing, 100037, China
| | - Aihua Zhi
- Department of Medical Imaging, Fuwai Yunnan Cardiovascular Hospital, Kunming, 650000, China
| | - Xiangbin Pan
- Department of Structural Heart Disease, National Center for Cardiovascular Disease, China and Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing, 100037, China.
- National Health Commission Key Laboratory of Cardiovascular Regeneration Medicine, Beijing, 100037, China.
- Key Laboratory of Innovative Cardiovascular Devices, Chinese Academy of Medical Sciences, Beijing, 100037, China.
- National Clinical Research Center for Cardiovascular Diseases, Fuwai Hospital, Chinese Academy of Medical Sciences, Beijing, 100037, China.
| |
Collapse
|