Heart Chamber Segmentation from CT Using Convolutional Neural Networks

Role of Deep Learning in Computed Tomography

Computed tomography has played an instrumental role in the understanding of the pathophysiology of atherosclerosis in coronary artery disease. It enables visualization of the plaque obstruction and vessel stenosis in a comprehensive manner. As technology for computed tomography is constantly evolving, coronary applications and possibilities are constantly expanding. This influx of information can overwhelm a physician's ability to interpret information in this era of big data. Machine learning is a revolutionary approach that can help provide limitless pathways in patient management. Within these machine algorithms, deep learning has tremendous potential and can revolutionize computed tomography and cardiovascular imaging. In this review article, we highlight the role of deep learning in various aspects of computed tomography.

Heart and bladder detection and segmentation on FDG PET/CT by deep learning

Purpose

Positron emission tomography (PET)/ computed tomography (CT) has been extensively used to quantify metabolically active tumors in various oncology indications. However, FDG-PET/CT often encounters false positives in tumor detection due to ¹⁸fluorodeoxyglucose (FDG) accumulation from the heart and bladder that often exhibit similar FDG uptake as tumors. Thus, it is necessary to eliminate this source of physiological noise. Major challenges for this task include: (1) large inter-patient variability in the appearance for the heart and bladder. (2) The size and shape of bladder or heart may appear different on PET and CT. (3) Tumors can be very close or connected to the heart or bladder.

Approach

A deep learning based approach is proposed to segment the heart and bladder on whole body PET/CT automatically. Two 3D U-Nets were developed separately to segment the heart and bladder, where each network receives the PET and CT as a multi-modal input. Data sets were obtained from retrospective clinical trials and include 575 PET/CT for heart segmentation and 538 for bladder segmentation.

Results

The models were evaluated on a test set from an independent trial and achieved a Dice Similarity Coefficient (DSC) of 0.96 for heart segmentation and 0.95 for bladder segmentation, Average Surface Distance (ASD) of 0.44 mm on heart and 0.90 mm on bladder.

Conclusions

This methodology could be a valuable component to the FDG-PET/CT data processing chain by removing FDG physiological noise associated with heart and/or bladder accumulation prior to image analysis by manual, semi- or automated tumor analysis methods.

Semi-automated three-dimensional segmentation for cardiac CT images using deep learning and randomly distributed points

Given the prevalence of cardiovascular diseases (CVDs), the segmentation of the heart on cardiac computed tomography (CT) remains of great importance. Manual segmentation is time-consuming and intra-and inter-observer variabilities yield inconsistent and inaccurate results. Computer-assisted, and in particular, deep learning approaches to segmentation continue to potentially offer an accurate, efficient alternative to manual segmentation. However, fully automated methods for cardiac segmentation have yet to achieve accurate enough results to compete with expert segmentation. Thus, we focus on a semi-automated deep learning approach to cardiac segmentation that bridges the divide between a higher accuracy from manual segmentation and higher efficiency from fully automated methods. In this approach, we selected a fixed number of points along the surface of the cardiac region to mimic user interaction. Points-distance maps were then generated from these points selections, and a three-dimensional (3D) fully convolutional neural network (FCNN) was trained using points-distance maps to provide a segmentation prediction. Testing our method with different numbers of selected points, we achieved a Dice score from 0.742 to 0.917 across the four chambers. Specifically. Dice scores averaged 0.846 ± 0.059, 0.857 ± 0.052, 0.826 ± 0.062, and 0.824 ± 0.062 for the left atrium, left ventricle, right atrium, and right ventricle, respectively across all points selections. This point-guided, image-independent, deep learning segmentation approach illustrated a promising performance for chamber-by-chamber delineation of the heart in CT images.

Generalizable Framework for Atrial Volume Estimation for Cardiac CT Images Using Deep Learning With Quality Control Assessment

Objectives

Cardiac computed tomography (CCT) is a common pre-operative imaging modality to evaluate pulmonary vein anatomy and left atrial appendage thrombus in patients undergoing catheter ablation (CA) for atrial fibrillation (AF). These images also allow for full volumetric left atrium (LA) measurement for recurrence risk stratification, as larger LA volume (LAV) is associated with higher recurrence rates. Our objective is to apply deep learning (DL) techniques to fully automate the computation of LAV and assess the quality of the computed LAV values.

Methods

Using a dataset of 85,477 CCT images from 337 patients, we proposed a framework that consists of several processes that perform a combination of tasks including the selection of images with LA from all other images using a ResNet50 classification model, the segmentation of images with LA using a UNet image segmentation model, the assessment of the quality of the image segmentation task, the estimation of LAV, and quality control (QC) assessment.

Results

Overall, the proposed LAV estimation framework achieved accuracies of 98% (precision, recall, and F1 score metrics) in the image classification task, 88.5% (mean dice score) in the image segmentation task, 82% (mean dice score) in the segmentation quality prediction task, and R² (the coefficient of determination) value of 0.968 in the volume estimation task. It correctly identified 9 out of 10 poor LAV estimations from a total of 337 patients as poor-quality estimates.

Conclusions

We proposed a generalizable framework that consists of DL models and computational methods for LAV estimation. The framework provides an efficient and robust strategy for QC assessment of the accuracy for DL-based image segmentation and volume estimation tasks, allowing high-throughput extraction of reproducible LAV measurements to be possible.

Automatic Aortic Valve Cusps Segmentation from CT Images Based on the Cascading Multiple Deep Neural Networks

Accurate morphological information on aortic valve cusps is critical in treatment planning. Image segmentation is necessary to acquire this information, but manual segmentation is tedious and time consuming. In this paper, we propose a fully automatic aortic valve cusps segmentation method from CT images by combining two deep neural networks, spatial configuration-Net for detecting anatomical landmarks and U-Net for segmentation of aortic valve components. A total of 258 CT volumes of end systolic and end diastolic phases, which include cases with and without severe calcifications, were collected and manually annotated for each aortic valve component. The collected CT volumes were split 6:2:2 for the training, validation and test steps, and our method was evaluated by five-fold cross validation. The segmentation was successful for all CT volumes with 69.26 s as mean processing time. For the segmentation results of the aortic root, the right-coronary cusp, the left-coronary cusp and the non-coronary cusp, mean Dice Coefficient were 0.95, 0.70, 0.69, and 0.67, respectively. There were strong correlations between measurement values automatically calculated based on the annotations and those based on the segmentation results. The results suggest that our method can be used to automatically obtain measurement values for aortic valve morphology.

Deep learning for segmentation in radiation therapy planning: a review

Segmentation of organs and structures, as either targets or organs-at-risk, has a significant influence on the success of radiation therapy. Manual segmentation is a tedious and time-consuming task for clinicians, and inter-observer variability can affect the outcomes of radiation therapy. The recent hype over deep neural networks has added many powerful auto-segmentation methods as variations of convolutional neural networks (CNN). This paper presents a descriptive review of the literature on deep learning techniques for segmentation in radiation therapy planning. The most common CNN architecture across the four clinical sub sites considered was U-net, with the majority of deep learning segmentation articles focussed on head and neck normal tissue structures. The most common data sets were CT images from an inhouse source, along with some public data sets. N-fold cross-validation was commonly employed; however, not all work separated training, test and validation data sets. This area of research is expanding rapidly. To facilitate comparisons of proposed methods and benchmarking, consistent use of appropriate metrics and independent validation should be carefully considered.

Effects of sample size and data augmentation on U-Net-based automatic segmentation of various organs

Deep learning has demonstrated high efficacy for automatic segmentation in contour delineation, which is crucial in radiation therapy planning. However, the collection, labeling, and management of medical imaging data can be challenging. This study aims to elucidate the effects of sample size and data augmentation on the automatic segmentation of computed tomography images using U-Net, a deep learning method. For the chest and pelvic regions, 232 and 556 cases are evaluated, respectively. We investigate multiple conditions by changing the sum of the training and validation datasets across a broad range of values: 10-200 and 10-500 cases for the chest and pelvic regions, respectively. A U-Net is constructed, and horizontal-flip data augmentation, which produces left and right inverse images resulting in twice the number of images, is compared with no augmentation for each training session. All lung cases and more than 100 prostate, bladder, and rectum cases indicate that adding horizontal-flip data augmentation is almost as effective as doubling the number of cases. The slope of the Dice similarity coefficient (DSC) in all organs decreases rapidly until approximately 100 cases, stabilizes after 200 cases, and shows minimal changes as the number of cases is increased further. The DSCs stabilize at a smaller sample size with the incorporation of data augmentation in all organs except the heart. This finding is applicable to the automation of radiation therapy for rare cancers, where large datasets may be difficult to obtain.

Clinical evaluation of atlas- and deep learning-based automatic segmentation of multiple organs and clinical target volumes for breast cancer

Manual segmentation is the gold standard method for radiation therapy planning; however, it is time-consuming and prone to inter- and intra-observer variation, giving rise to interests in auto-segmentation methods. We evaluated the feasibility of deep learning-based auto-segmentation (DLBAS) in comparison to commercially available atlas-based segmentation solutions (ABAS) for breast cancer radiation therapy. This study used contrast-enhanced planning computed tomography scans from 62 patients with breast cancer who underwent breast-conservation surgery. Contours of target volumes (CTVs), organs, and heart substructures were generated using two commercial ABAS solutions and DLBAS using fully convolutional DenseNet. The accuracy of the segmentation was assessed using 14 test patients using the Dice Similarity Coefficient and Hausdorff Distance referencing the expert contours. A sensitivity analysis was performed using non-contrast planning CT from 14 additional patients. Compared to ABAS, the proposed DLBAS model yielded more consistent results and the highest average Dice Similarity Coefficient values and lowest Hausdorff Distances, especially for CTVs and the substructures of the heart. ABAS showed limited performance in soft-tissue-based regions, such as the esophagus, cardiac arteries, and smaller CTVs. The results of sensitivity analysis between contrast and non-contrast CT test sets showed little difference in the performance of DLBAS and conversely, a large discrepancy for ABAS. The proposed DLBAS algorithm was more consistent and robust in its performance than ABAS across the majority of structures when examining both CTVs and normal organs. DLBAS has great potential to aid a key process in the radiation therapy workflow, helping optimise and reduce the clinical workload.

Cardio-pulmonary substructure segmentation of radiotherapy computed tomography images using convolutional neural networks for clinical outcomes analysis

Background and purpose

Radiation dose to the cardio-pulmonary system is critical for radiotherapy-induced mortality in non-small cell lung cancer. Our goal was to automatically segment substructures of the cardio-pulmonary system for use in outcomes analyses for thoracic cancers. We built and validated a multi-label Deep Learning Segmentation (DLS) model for accurate auto-segmentation of twelve cardio-pulmonary substructures.

Materials and methods

The DLS model utilized a convolutional neural network for segmenting substructures from 217 thoracic radiotherapy Computed Tomography (CT) scans. The model was built in the presence of variable image characteristics such as the absence/presence of contrast. We quantitatively evaluated the final model against expert contours for a hold-out dataset of 24 CT scans using Dice Similarity Coefficient (DSC), 95th Percentile of Hausdorff Distance and Dose-volume Histograms (DVH). DLS contours of an additional 25 scans were qualitatively evaluated by a radiation oncologist to determine their clinical acceptability.

Results

The DLS model reduced segmentation time per patient from about one hour to 10 s. Quantitatively, the highest accuracy was observed for the Heart (median DSC = (0.96 (0.95–0.97)). The median DSC for the remaining structures was between 0.81 and 0.93. No statistically significant difference was found between DVH metrics of the auto-generated and manual contours (p-value ⩾ 0.69). The expert judged that, on average, 85% of contours were qualitatively equivalent to state-of-the-art manual contouring.

Conclusion

The cardio-pulmonary DLS model performed well both quantitatively and qualitatively for all structures. This model has been incorporated into an open-source tool for the community to use for treatment planning and clinical outcomes analysis.

Fully automated segmentation of the right ventricle in patients with repaired Tetralogy of Fallot using U-Net

Cardiac magnetic resonance (CMR) imaging is considered the standard imaging modality for volumetric analysis of the right ventricle (RV), an especially important practice in the evaluation of heart structure and function in patients with repaired Tetralogy of Fallot (rTOF). In clinical practice, however, this requires time-consuming manual delineation of the RV endocardium in multiple 2-dimensional (2D) slices at multiple phases of the cardiac cycle. In this work, we employed a U-Net based 2D convolutional neural network (CNN) classifier in the fully automatic segmentation of the RV blood pool. Our dataset was comprised of 5,729 short-axis cine CMR slices taken from 100 individuals with rTOF. Training of our CNN model was performed on images from 50 individuals while validation was performed on images from 10 individuals. Segmentation results were evaluated by Dice similarity coefficient (DSC) and Hausdorff distance (HD). Use of the CNN model on our testing group of 40 individuals yielded a median DSC of 90% and a median 95^th percentile HD of 5.1 mm, demonstrating good performance in these metrics when compared to literature results. Our preliminary results suggest that our deep learning-based method can be effective in automating RV segmentation.

Deep Learning for Cardiac Image Segmentation: A Review

Deep learning has become the most widely used approach for cardiac image segmentation in recent years. In this paper, we provide a review of over 100 cardiac image segmentation papers using deep learning, which covers common imaging modalities including magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound and major anatomical structures of interest (ventricles, atria, and vessels). In addition, a summary of publicly available cardiac image datasets and code repositories are included to provide a base for encouraging reproducible research. Finally, we discuss the challenges and limitations with current deep learning-based approaches (scarcity of labels, model generalizability across different domains, interpretability) and suggest potential directions for future research.

A primer in artificial intelligence in cardiovascular medicine

Driven by recent developments in computational power, algorithms and web-based storage resources, machine learning (ML)-based artificial intelligence (AI) has quickly gained ground as the solution for many technological and societal challenges. AI education has become very popular and is oversubscribed at Dutch universities. Major investments were made in 2018 to develop and build the first AI-driven hospitals to improve patient care and reduce healthcare costs. AI has the potential to greatly enhance traditional statistical analyses in many domains and has been demonstrated to allow the discovery of 'hidden' information in highly complex datasets. As such, AI can also be of significant value in the diagnosis and treatment of cardiovascular disease, and the first applications of AI in the cardiovascular field are promising. However, many professionals in the cardiovascular field involved in patient care, education or science are unaware of the basics behind AI and the existing and expected applications in their field. In this review, we aim to introduce the broad cardiovascular community to the basics of modern ML-based AI and explain several of the commonly used algorithms. We also summarise their initial and future applications relevant to the cardiovascular field.