1
|
Huang Y, Yang J, Sun Q, Yuan Y, Li H, Hou Y. Multi-residual 2D network integrating spatial correlation for whole heart segmentation. Comput Biol Med 2024; 172:108261. [PMID: 38508056 DOI: 10.1016/j.compbiomed.2024.108261] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Revised: 02/21/2024] [Accepted: 03/06/2024] [Indexed: 03/22/2024]
Abstract
Whole heart segmentation (WHS) has significant clinical value for cardiac anatomy, modeling, and analysis of cardiac function. This study aims to address the WHS accuracy on cardiac CT images, as well as the fast inference speed and low graphics processing unit (GPU) memory consumption required by practical clinical applications. Thus, we propose a multi-residual two-dimensional (2D) network integrating spatial correlation for WHS. The network performs slice-by-slice segmentation on three-dimensional cardiac CT images in a 2D encoder-decoder manner. In the network, a convolutional long short-term memory skip connection module is designed to perform spatial correlation feature extraction on the feature maps at different resolutions extracted by the sub-modules of the pre-trained ResNet-based encoder. Moreover, a decoder based on the multi-residual module is designed to analyze the extracted features from the perspectives of multi-scale and channel attention, thereby accurately delineating the various substructures of the heart. The proposed method is verified on a dataset of the multi-modality WHS challenge, an in-house WHS dataset, and a dataset of the abdominal organ segmentation challenge. The dice, Jaccard, average symmetric surface distance, Hausdorff distance, inference time, and maximum GPU memory of the WHS are 0.914, 0.843, 1.066 mm, 15.778 mm, 9.535 s, and 1905 MB, respectively. The proposed network has high accuracy, fast inference speed, minimal GPU memory consumption, strong robustness, and good generalization. It can be deployed to clinical practical applications for WHS and can be effectively extended and applied to other multi-organ segmentation fields. The source code is publicly available at https://github.com/nancy1984yan/MultiResNet-SC.
Collapse
Affiliation(s)
- Yan Huang
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Jinzhu Yang
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China; National Frontiers Science Center for Industrial Intelligence and Systems Optimization, Shenyang, Liaoning, China.
| | - Qi Sun
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Yuliang Yuan
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Honghe Li
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Northeastern University, Shenyang, Liaoning, China; School of Computer Science and Engineering, Northeastern University, Shenyang, Liaoning, China
| | - Yang Hou
- Department of Radiology, Shengjing Hospital of China Medical University, Shenyang, Liaoning, China
| |
Collapse
|
2
|
Pace DF, Dalca AV, Brosch T, Geva T, Powell AJ, Weese J, Moghari MH, Golland P. Learned iterative segmentation of highly variable anatomy from limited data: Applications to whole heart segmentation for congenital heart disease. Med Image Anal 2022; 80:102469. [PMID: 35640385 PMCID: PMC9617683 DOI: 10.1016/j.media.2022.102469] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2021] [Revised: 04/26/2022] [Accepted: 04/29/2022] [Indexed: 02/08/2023]
Abstract
Training deep learning models that segment an image in one step typically requires a large collection of manually annotated images that captures the anatomical variability in a cohort. This poses challenges when anatomical variability is extreme but training data is limited, as when segmenting cardiac structures in patients with congenital heart disease (CHD). In this paper, we propose an iterative segmentation model and show that it can be accurately learned from a small dataset. Implemented as a recurrent neural network, the model evolves a segmentation over multiple steps, from a single user click until reaching an automatically determined stopping point. We develop a novel loss function that evaluates the entire sequence of output segmentations, and use it to learn model parameters. Segmentations evolve predictably according to growth dynamics encapsulated by training data, which consists of images, partially completed segmentations, and the recommended next step. The user can easily refine the final segmentation by examining those that are earlier or later in the output sequence. Using a dataset of 3D cardiac MR scans from patients with a wide range of CHD types, we show that our iterative model offers better generalization to patients with the most severe heart malformations.
Collapse
Affiliation(s)
- Danielle F Pace
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA; A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA.
| | - Adrian V Dalca
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA; A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Harvard Medical School, Boston, MA, USA
| | - Tom Brosch
- Philips Research Laboratories, Hamburg, Germany
| | - Tal Geva
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA; Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | - Andrew J Powell
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA; Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | | | - Mehdi H Moghari
- Department of Cardiology, Boston Children's Hospital, Boston, MA, USA; Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | - Polina Golland
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA
| |
Collapse
|
3
|
Kong F, Wilson N, Shadden S. A deep-learning approach for direct whole-heart mesh reconstruction. Med Image Anal 2021; 74:102222. [PMID: 34543913 DOI: 10.1016/j.media.2021.102222] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/17/2021] [Revised: 07/14/2021] [Accepted: 08/31/2021] [Indexed: 01/16/2023]
Abstract
Automated construction of surface geometries of cardiac structures from volumetric medical images is important for a number of clinical applications. While deep-learning-based approaches have demonstrated promising reconstruction precision, these approaches have mostly focused on voxel-wise segmentation followed by surface reconstruction and post-processing techniques. However, such approaches suffer from a number of limitations including disconnected regions or incorrect surface topology due to erroneous segmentation and stair-case artifacts due to limited segmentation resolution. We propose a novel deep-learning-based approach that directly predicts whole heart surface meshes from volumetric CT and MR image data. Our approach leverages a graph convolutional neural network to predict deformation on mesh vertices from a pre-defined mesh template to reconstruct multiple anatomical structures in a 3D image volume. Our method demonstrated promising performance of generating whole heart reconstructions with as good or better accuracy than prior deep-learning-based methods on both CT and MR data. Furthermore, by deforming a template mesh, our method can generate whole heart geometries with better anatomical consistency and produce high-resolution geometries from lower resolution input image data. Our method was also able to produce temporally-consistent surface mesh predictions for heart motion from CT or MR cine sequences, and therefore can potentially be applied for efficiently constructing 4D whole heart dynamics. Our code and pre-trained networks are available at https://github.com/fkong7/MeshDeformNet.
Collapse
|
4
|
Sharobeem S, Le Breton H, Lalys F, Lederlin M, Lagorce C, Bedossa M, Boulmier D, Leurent G, Haigron P, Auffret V. Validation of a Whole Heart Segmentation from Computed Tomography Imaging Using a Deep-Learning Approach. J Cardiovasc Transl Res 2021; 15:427-437. [PMID: 34448116 DOI: 10.1007/s12265-021-10166-0] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 04/19/2021] [Accepted: 08/09/2021] [Indexed: 11/28/2022]
Abstract
The aim of this study is to develop an automated deep-learning-based whole heart segmentation of ECG-gated computed tomography data. After 21 exclusions, CT acquired before transcatheter aortic valve implantation in 71 patients were reviewed and randomly split in a training (n = 55 patients), validation (n = 8 patients), and a test set (n = 8 patients). A fully automatic deep-learning method combining two convolutional neural networks performed segmentation of 10 cardiovascular structures, which was compared with the manually segmented reference by the Dice index. Correlations and agreement between myocardial volumes and mass were assessed. The algorithm demonstrated high accuracy (Dice score = 0.920; interquartile range: 0.906-0.925) and a low computing time (13.4 s, range 11.9-14.9). Correlations and agreement of volumes and mass were satisfactory for most structures. Six of ten structures were well segmented. Deep-learning-based method allowed automated WHS from ECG-gated CT data with a high accuracy. Challenges remain to improve right-sided structures segmentation and achieve daily clinical application.
Collapse
Affiliation(s)
- Sam Sharobeem
- LTSI - UMR 1099, Inserm, CHU Rennes, Univ Rennes, 35000, Rennes, France.,Service de Cardiologie, CHU Rennes, 35000, Rennes, France
| | - Hervé Le Breton
- LTSI - UMR 1099, Inserm, CHU Rennes, Univ Rennes, 35000, Rennes, France.,Service de Cardiologie, CHU Rennes, 35000, Rennes, France
| | | | - Mathieu Lederlin
- LTSI - UMR 1099, Inserm, CHU Rennes, Univ Rennes, 35000, Rennes, France.,Service de Radiologie, CHU Rennes, 35000, Rennes, France
| | | | - Marc Bedossa
- Service de Cardiologie, CHU Rennes, 35000, Rennes, France
| | - Dominique Boulmier
- LTSI - UMR 1099, Inserm, CHU Rennes, Univ Rennes, 35000, Rennes, France.,Service de Cardiologie, CHU Rennes, 35000, Rennes, France
| | | | - Pascal Haigron
- LTSI - UMR 1099, Inserm, CHU Rennes, Univ Rennes, 35000, Rennes, France
| | - Vincent Auffret
- LTSI - UMR 1099, Inserm, CHU Rennes, Univ Rennes, 35000, Rennes, France. .,Service de Cardiologie, CHU Rennes, 35000, Rennes, France. .,Service de Cardiologie, CHU Pontchaillou, 2 rue Henri Le Guilloux, 35000, Rennes, France.
| |
Collapse
|
5
|
Dormer JD, Ma L, Halicek M, Reilly CM, Schreibmann E, Fei B. Heart Chamber Segmentation from CT Using Convolutional Neural Networks. Proc SPIE Int Soc Opt Eng 2018; 10578. [PMID: 30197464 DOI: 10.1117/12.2293554] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/14/2022]
Abstract
CT is routinely used for radiotherapy planning with organs and regions of interest being segmented for diagnostic evaluation and parameter optimization. For cardiac segmentation, many methods have been proposed for left ventricular segmentation, but few for simultaneous segmentation of the entire heart. In this work, we present a convolutional neural networks (CNN)-based cardiac chamber segmentation method for 3D CT with 5 classes: left ventricle, right ventricle, left atrium, right atrium, and background. We achieved an overall accuracy of 87.2% ± 3.3% and an overall chamber accuracy of 85.6 ± 6.1%. The deep learning based segmentation method may provide an automatic tool for cardiac segmentation on CT images.
Collapse
Affiliation(s)
- James D Dormer
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA
| | - Ling Ma
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA
| | - Martin Halicek
- Medical College of Georgia, Augusta, GA.,Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA
| | - Carolyn M Reilly
- Nell Hodgson Woodruff School of Nursing, Emory University, Atlanta, GA
| | | | - Baowei Fei
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA.,Department of Biomedical Engineering, Emory University and Georgia Institute of Technology, Atlanta, GA.,Winship Cancer Institute of Emory University, Atlanta, Georgia
| |
Collapse
|