Standardized Evaluation System for Left Ventricular Segmentation Algorithms in 3D Echocardiography

Deep-DM: Deep-Driven Deformable Model for 3D Image Segmentation Using Limited Data

Objective - Medical image segmentation is essential for several clinical tasks, including diagnosis, surgical and treatment planning, and image-guided interventions. Deep Learning (DL) methods have become the state-of-the-art for several image segmentation scenarios. However, a large and well-annotated dataset is required to effectively train a DL model, which is usually difficult to obtain in clinical practice, especially for 3D images. Methods - In this paper, we proposed Deep-DM, a learning-guided deformable model framework for 3D medical imaging segmentation using limited training data. In the proposed method, an energy function is learned by a Convolutional Neural Network (CNN) and integrated into an explicit deformable model to drive the evolution of an initial surface towards the object to segment. Specifically, the learning-based energy function is iteratively retrieved from localized anatomical representations of the image containing the image information around the evolving surface at each iteration. By focusing on localized regions of interest, this representation excludes irrelevant image information, facilitating the learning process. Results and conclusion - The performance of the proposed method is demonstrated for the tasks of left ventricle and fetal head segmentation in ultrasound, left atrium segmentation in Magnetic Resonance, and bladder segmentation in Computed Tomography, using different numbers of training volumes in each study. The results obtained showed the feasibility of the proposed method to segment different anatomical structures in different imaging modalities. Moreover, the results also showed that the proposed approach is less dependent on the size of the training dataset in comparison with state-of-the-art DL-based segmentation methods, outperforming them for all tasks when a low number of samples is available. Significance - Overall, by offering a more robust and less data-intensive approach to accurately segmenting anatomical structures, the proposed method has the potential to enhance clinical tasks that require image segmentation strategies.

BEAS-Net: A Shape-Prior-Based Deep Convolutional Neural Network for Robust Left Ventricular Segmentation in 2-D Echocardiography

Left ventricle (LV) segmentation of 2-D echocardiography images is an essential step in the analysis of cardiac morphology and function and-more generally-diagnosis of cardiovascular diseases (CVD). Several deep learning (DL) algorithms have recently been proposed for the automatic segmentation of the LV, showing significant performance improvement over the traditional segmentation algorithms. However, unlike the traditional methods, prior information about the segmentation problem, e.g., anatomical shape information, is not usually incorporated for training the DL algorithms. This can degrade the generalization performance of the DL models on unseen images if their characteristics are somewhat different from those of the training images, e.g., low-quality testing images. In this study, a new shape-constrained deep convolutional neural network (CNN)-called B-spline explicit active surface (BEAS)-Net-is introduced for automatic LV segmentation. The BEAS-Net learns how to associate the image features, encoded by its convolutional layers, with anatomical shape-prior information derived by the BEAS algorithm to generate physiologically meaningful segmentation contours when dealing with artifactual or low-quality images. The performance of the proposed network was evaluated using three different in vivo datasets and was compared with a deep segmentation algorithm based on the U-Net model. Both the networks yielded comparable results when tested on images of acceptable quality, but the BEAS-Net outperformed the benchmark DL model on artifactual and low-quality images.

US2Mask: Image-to-mask generation learning via a conditional GAN for cardiac ultrasound image segmentation

Cardiac ultrasound (US) image segmentation is vital for evaluating clinical indices, but it often demands a large dataset and expert annotations, resulting in high costs for deep learning algorithms. To address this, our study presents a framework utilizing artificial intelligence generation technology to produce multi-class RGB masks for cardiac US image segmentation. The proposed approach directly performs semantic segmentation of the heart's main structures in US images from various scanning modes. Additionally, we introduce a novel learning approach based on conditional generative adversarial networks (CGAN) for cardiac US image segmentation, incorporating a conditional input and paired RGB masks. Experimental results from three cardiac US image datasets with diverse scan modes demonstrate that our approach outperforms several state-of-the-art models, showcasing improvements in five commonly used segmentation metrics, with lower noise sensitivity. Source code is available at https://github.com/energy588/US2mask.

Left Ventricle Segmentation in Echocardiography with Transformer

Left ventricular ejection fraction (LVEF) plays as an essential role in the assessment of cardiac function, providing quantitative data support for the medical diagnosis of heart disease. Robust evaluation of the ejection fraction relies on accurate left ventricular (LV) segmentation of echocardiograms. Because human bias and expensive labor cost exist in manual echocardiographic analysis, computer algorithms of deep-learning have been developed to help human experts in segmentation tasks. Most of the previous work is based on the convolutional neural networks (CNN) structure and has achieved good results. However, the region occupied by the left ventricle is large for echocardiography. Therefore, the limited receptive field of CNN leaves much room for improvement in the effectiveness of LV segmentation. In recent years, Vision Transformer models have demonstrated their effectiveness and universality in traditional semantic segmentation tasks. Inspired by this, we propose two models that use two different pure Transformers as the basic framework for LV segmentation in echocardiography: one combines Swin Transformer and K-Net, and the other uses Segformer. We evaluate these two models on the EchoNet-Dynamic dataset of LV segmentation and compare the quantitative metrics with other models for LV segmentation. The experimental results show that the mean Dice similarity of the two models scores are 92.92% and 92.79%, respectively, which outperform most of the previous mainstream CNN models. In addition, we found that for some samples that were not easily segmented, whereas both our models successfully recognized the valve region and separated left ventricle and left atrium, the CNN model segmented them together as a single part. Therefore, it becomes possible for us to obtain accurate segmentation results through simple post-processing, by filtering out the parts with the largest circumference or pixel square. These promising results prove the effectiveness of the two models and reveal the potential of Transformer structure in echocardiographic segmentation.

Dual consistency loss for contour-aware segmentation in medical images

Medical image segmentation is a paramount task for several clinical applications, namely for the diagnosis of pathologies, for treatment planning, and for aiding image-guided surgeries. With the development of deep learning, Convolutional Neural Networks (CNN) have become the state-of-the-art for medical image segmentation. However, issues are still raised concerning the precise object boundary delineation, since traditional CNNs can produce non-smooth segmentations with boundary discontinuities. In this work, a U-shaped CNN architecture is proposed to generate both pixel-wise segmentation and probabilistic contour maps of the object to segment, in order to generate reliable segmentations at the object's boundaries. Moreover, since the segmentation and contour maps must be inherently related to each other, a dual consistency loss that relates the two outputs of the network is proposed. Thus, the network is enforced to consistently learn the segmentation and contour delineation tasks during the training. The proposed method was applied and validated on a public dataset of cardiac 3D ultrasound images of the left ventricle. The results obtained showed the good performance of the method and its applicability for the cardiac dataset, showing its potential to be used in clinical practice for medical image segmentation.Clinical Relevance- The proposed network with dual consistency loss scheme can improve the performance of state-of-the-art CNNs for medical image segmentation, proving its value to be applied for computer-aided diagnosis.

Cross-dimensional transfer learning in medical image segmentation with deep learning

Over the last decade, convolutional neural networks have emerged and advanced the state-of-the-art in various image analysis and computer vision applications. The performance of 2D image classification networks is constantly improving and being trained on databases made of millions of natural images. Conversely, in the field of medical image analysis, the progress is also remarkable but has mainly slowed down due to the relative lack of annotated data and besides, the inherent constraints related to the acquisition process. These limitations are even more pronounced given the volumetry of medical imaging data. In this paper, we introduce an efficient way to transfer the efficiency of a 2D classification network trained on natural images to 2D, 3D uni- and multi-modal medical image segmentation applications. In this direction, we designed novel architectures based on two key principles: weight transfer by embedding a 2D pre-trained encoder into a higher dimensional U-Net, and dimensional transfer by expanding a 2D segmentation network into a higher dimension one. The proposed networks were tested on benchmarks comprising different modalities: MR, CT, and ultrasound images. Our 2D network ranked first on the CAMUS challenge dedicated to echo-cardiographic data segmentation and surpassed the state-of-the-art. Regarding 2D/3D MR and CT abdominal images from the CHAOS challenge, our approach largely outperformed the other 2D-based methods described in the challenge paper on Dice, RAVD, ASSD, and MSSD scores and ranked third on the online evaluation platform. Our 3D network applied to the BraTS 2022 competition also achieved promising results, reaching an average Dice score of 91.69% (91.22%) for the whole tumor, 83.23% (84.77%) for the tumor core and 81.75% (83.88%) for enhanced tumor using the approach based on weight (dimensional) transfer. Experimental and qualitative results illustrate the effectiveness of our methods for multi-dimensional medical image segmentation.

MITEA: A dataset for machine learning segmentation of the left ventricle in 3D echocardiography using subject-specific labels from cardiac magnetic resonance imaging

Segmentation of the left ventricle (LV) in echocardiography is an important task for the quantification of volume and mass in heart disease. Continuing advances in echocardiography have extended imaging capabilities into the 3D domain, subsequently overcoming the geometric assumptions associated with conventional 2D acquisitions. Nevertheless, the analysis of 3D echocardiography (3DE) poses several challenges associated with limited spatial resolution, poor contrast-to-noise ratio, complex noise characteristics, and image anisotropy. To develop automated methods for 3DE analysis, a sufficiently large, labeled dataset is typically required. However, ground truth segmentations have historically been difficult to obtain due to the high inter-observer variability associated with manual analysis. We address this lack of expert consensus by registering labels derived from higher-resolution subject-specific cardiac magnetic resonance (CMR) images, producing 536 annotated 3DE images from 143 human subjects (10 of which were excluded). This heterogeneous population consists of healthy controls and patients with cardiac disease, across a range of demographics. To demonstrate the utility of such a dataset, a state-of-the-art, self-configuring deep learning network for semantic segmentation was employed for automated 3DE analysis. Using the proposed dataset for training, the network produced measurement biases of -9 ± 16 ml, -1 ± 10 ml, -2 ± 5 %, and 5 ± 23 g, for end-diastolic volume, end-systolic volume, ejection fraction, and mass, respectively, outperforming an expert human observer in terms of accuracy as well as scan-rescan reproducibility. As part of the Cardiac Atlas Project, we present here a large, publicly available 3DE dataset with ground truth labels that leverage the higher resolution and contrast of CMR, to provide a new benchmark for automated 3DE analysis. Such an approach not only reduces the effect of observer-specific bias present in manual 3DE annotations, but also enables the development of analysis techniques which exhibit better agreement with CMR compared to conventional methods. This represents an important step for enabling more efficient and accurate diagnostic and prognostic information to be obtained from echocardiography.

Detection of Cardiac Tissues using K-means Analysis Methods in Nuclear Medicine Images

BACKGROUND: Nuclear cardiology uses to diagnose the cardiac disorders such as ischemic and inflammation disorders. In cardiac scintigraphy, unraveling closely adjacent tissues in the image are challenging issue. AIM: The aim of the study is to detect of cardiac tissues using K-means analysis methods in nuclear medicine images. This study also aimed to reduce the existence of fleck noise that disturbs the contrast and make its analysis more difficult. METHODS: Thus, digital image processing uses to increase the detection rate of myocardium easily using its color-based algorithms. In this study, color-based K-means was used. The scintographs were converted into color space presentation. Then, each pixel in the image was segmented using color analysis algorithms. RESULTS: The segmented scintograph was displayed in distinct fresh image. The proposed technique defines the myocardial tissues and borders precisely. Both exactness rate and recall reckoning were calculated. The results were 97.3 + 8.46 (p > 0.05). CONCLUSION: The proposed technique offered recognition of the heart tissue with high exactness amount.

Advanced Ultrasound and Photoacoustic Imaging in Cardiology

Cardiovascular diseases (CVDs) remain the leading cause of death worldwide. An effective management and treatment of CVDs highly relies on accurate diagnosis of the disease. As the most common imaging technique for clinical diagnosis of the CVDs, US imaging has been intensively explored. Especially with the introduction of deep learning (DL) techniques, US imaging has advanced tremendously in recent years. Photoacoustic imaging (PAI) is one of the most promising new imaging methods in addition to the existing clinical imaging methods. It can characterize different tissue compositions based on optical absorption contrast and thus can assess the functionality of the tissue. This paper reviews some major technological developments in both US (combined with deep learning techniques) and PA imaging in the application of diagnosis of CVDs.

Mitral Valve Segmentation Using Robust Nonnegative Matrix Factorization

Analyzing and understanding the movement of the mitral valve is of vital importance in cardiology, as the treatment and prevention of several serious heart diseases depend on it. Unfortunately, large amounts of noise as well as a highly varying image quality make the automatic tracking and segmentation of the mitral valve in two-dimensional echocardiographic videos challenging. In this paper, we present a fully automatic and unsupervised method for segmentation of the mitral valve in two-dimensional echocardiographic videos, independently of the echocardiographic view. We propose a bias-free variant of the robust non-negative matrix factorization (RNMF) along with a window-based localization approach, that is able to identify the mitral valve in several challenging situations. We improve the average f1-score on our dataset of 10 echocardiographic videos by 0.18 to a f1-score of 0.56.

Systematic Comparison of Left Ventricular Geometry Between 3D-Echocardiography and Cardiac Magnetic Resonance Imaging

Aims: Left ventricular (LV) volumes estimated using three-dimensional echocardiography (3D-echo) have been reported to be smaller than those measured using cardiac magnetic resonance (CMR) imaging, but the underlying causes are not well-understood. We investigated differences in regional LV anatomy derived from these modalities and related subsequent findings to image characteristics. Methods and Results: Seventy participants (18 patients and 52 healthy participants) were imaged with 3D-echo and CMR (<1 h apart). Three-dimensional left ventricular models were constructed at end-diastole (ED) and end-systole (ES) from both modalities using previously validated software, enabling the fusion of CMR with 3D-echo by rigid registration. Regional differences were evaluated as mean surface distances for each of the 17 American Heart Association segments, and by comparing contours superimposed on images from each modality. In comparison to CMR-derived models, 3D-echo models underestimated LV end-diastolic volume (EDV) by -16 ± 22, -1 ± 25, and -18 ± 24 ml across three independent analysis methods. Average surface distance errors were largest in the basal-anterolateral segment (11-15 mm) and smallest in the mid-inferoseptal segment (6 mm). Larger errors were associated with signal dropout in anterior regions and the appearance of trabeculae at the lateral wall. Conclusions: Fusion of CMR and 3D-echo provides insight into the causes of volume underestimation by 3D-echo. Systematic signal dropout and differences in appearances of trabeculae lead to discrepancies in the delineation of LV geometry at anterior and lateral regions. A better understanding of error sources across modalities may improve correlation of clinical indices between 3D-echo and CMR.

A New Semi-automated Algorithm for Volumetric Segmentation of the Left Ventricle in Temporal 3D Echocardiography Sequences

PURPOSE

Echocardiography is commonly used as a non-invasive imaging tool in clinical practice for the assessment of cardiac function. However, delineation of the left ventricle is challenging due to the inherent properties of ultrasound imaging, such as the presence of speckle noise and the low signal-to-noise ratio.

METHODS

We propose a semi-automated segmentation algorithm for the delineation of the left ventricle in temporal 3D echocardiography sequences. The method requires minimal user interaction and relies on a diffeomorphic registration approach. Advantages of the method include no dependence on prior geometrical information, training data, or registration from an atlas.

RESULTS

The method was evaluated using three-dimensional ultrasound scan sequences from 18 patients from the Mazankowski Alberta Heart Institute, Edmonton, Canada, and compared to manual delineations provided by an expert cardiologist and four other registration algorithms. The segmentation approach yielded the following results over the cardiac cycle: a mean absolute difference of 1.01 (0.21) mm, a Hausdorff distance of 4.41 (1.43) mm, and a Dice overlap score of 0.93 (0.02).

CONCLUSION

The method performed well compared to the four other registration algorithms.

Harnessing Machine Intelligence in Automatic Echocardiogram Analysis: Current Status, Limitations, and Future Directions

Echocardiography (echo) is a critical tool in diagnosing various cardiovascular diseases. Despite its diagnostic and prognostic value, interpretation and analysis of echo images are still widely performed manually by echocardiographers. A plethora of algorithms has been proposed to analyze medical ultrasound data using signal processing and machine learning techniques. These algorithms provided opportunities for developing automated echo analysis and interpretation systems. The automated approach can significantly assist in decreasing the variability and burden associated with manual image measurements. In this paper, we review the state-of-the-art automatic methods for analyzing echocardiography data. Particularly, we comprehensively and systematically review existing methods of four major tasks: echo quality assessment, view classification, boundary segmentation, and disease diagnosis. Our review covers three echo imaging modes, which are B-mode, M-mode, and Doppler. We also discuss the challenges and limitations of current methods and outline the most pressing directions for future research. In summary, this review presents the current status of automatic echo analysis and discusses the challenges that need to be addressed to obtain robust systems suitable for efficient use in clinical settings or point-of-care testing.

Real-Time Automatic Ejection Fraction and Foreshortening Detection Using Deep Learning

Volume and ejection fraction (EF) measurements of the left ventricle (LV) in 2-D echocardiography are associated with a high uncertainty not only due to interobserver variability of the manual measurement, but also due to ultrasound acquisition errors such as apical foreshortening. In this work, a real-time and fully automated EF measurement and foreshortening detection method is proposed. The method uses several deep learning components, such as view classification, cardiac cycle timing, segmentation and landmark extraction, to measure the amount of foreshortening, LV volume, and EF. A data set of 500 patients from an outpatient clinic was used to train the deep neural networks, while a separate data set of 100 patients from another clinic was used for evaluation, where LV volume and EF were measured by an expert using clinical protocols and software. A quantitative analysis using 3-D ultrasound showed that EF is considerably affected by apical foreshortening, and that the proposed method can detect and quantify the amount of apical foreshortening. The bias and standard deviation of the automatic EF measurements were -3.6 ± 8.1%, while the mean absolute difference was measured at 7.2% which are all within the interobserver variability and comparable with related studies. The proposed real-time pipeline allows for a continuous acquisition and measurement workflow without user interaction, and has the potential to significantly reduce the time spent on the analysis and measurement error due to foreshortening, while providing quantitative volume measurements in the everyday echo lab.

Segmentation of coronary arteries images using global feature embedded network with active contour loss

Coronary heart disease (CHD) is a serious disease that endangers human health and life. In recent years, the morbidity and mortality of CHD are increasing significantly. Because of the particularity and complexity of medical image, it is challenging to segment coronary artery accurately and efficiently. This paper proposes a novel global feature embedded network for better coronary arteries segmentation in 3D coronary computed tomography angiography (CTA) data. The global feature combines multi-level layers from various stages of the network, which contains semantic information and detailed features, aiming to accurately segment target with precise boundary. In addition, we integrate a group of improved noisy activating functions with parameters into our network to eliminate the impact of noise in CTA data. And we improve the learning active contour model, which obtains a refined segmentation result with smooth boundary based on the high-quality score map produced by the networks. The experimental results show that the proposed framework achieved the state-of-the-art performance intuitively and quantitively.

Clinical ultrasound image standardization using histogram specification

This article presents a novel ultrasound image standardization approach. This method aims to preserve the non-linear relationship in the echo-textures while ensuring the endurance in the transformed image. This is achieved by utilizing the concept of histogram specification. A reference cumulative distribution function (CDF) of a considered distribution is used to process the test images. Initially, the shape and scale parameter of the distribution are estimated for each type of echo-texture from the reference ultrasound images of a particular organ. These parameters are used to estimate the prototype parameter set. The obtained prototype parameter set, along with a distribution function, is then used to construct a reference CDF. This CDF, in turn, is used as a transfer function in the histogram specification technique for standardizing the given input image. The efficiency and stability of the proposed approach are investigated and compared with the linear scaling technique. Four measures are used to evaluate the algorithms on three data sets. The results show that the proposed approach provides better standardization of images and is invariant to the gain of the scanning device as opposed to linear scaling.

Spatio-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data

Prognostic tumor growth modeling via volumetric medical imaging observations can potentially lead to better outcomes of tumor treatment management and surgical planning. Recent advances of convolutional networks (ConvNets) have demonstrated higher accuracy than traditional mathematical models can be achieved in predicting future tumor volumes. This indicates that deep learning based data-driven techniques may have great potentials on addressing such problem. However, current 2D image patch based modeling approaches can not make full use of the spatio-temporal imaging context of the tumor's longitudinal 4D (3D + time) patient data. Moreover, they are incapable to predict clinically-relevant tumor properties, other than the tumor volumes. In this paper, we exploit to formulate the tumor growth process through convolutional Long Short-Term Memory (ConvLSTM) that extract tumor's static imaging appearances and simultaneously capture its temporal dynamic changes within a single network. We extend ConvLSTM into the spatio-temporal domain (ST-ConvLSTM) by jointly learning the inter-slice 3D contexts and the longitudinal or temporal dynamics from multiple patient studies. Our approach can incorporate other non-imaging patient information in an end-to-end trainable manner. Experiments are conducted on the largest 4D longitudinal tumor dataset of 33 patients to date. Results validate that the proposed ST-ConvLSTM model produces a Dice score of 83.2%±5.1% and a RVD of 11.2%±10.8%, both statistically significantly outperforming (p < 0.05) other compared methods of traditional linear model, ConvLSTM, and generative adversarial network (GAN) under the metric of predicting future tumor volumes. Additionally, our new method enables the prediction of both cell density and CT intensity numbers. Last, we demonstrate the generalizability of ST-ConvLSTM by employing it in 4D medical image segmentation task, which achieves an averaged Dice score of 86.3%±1.2% for left-ventricle segmentation in 4D ultrasound with 3 seconds per patient case.

Deep Learning for Cardiac Image Segmentation: A Review

Deep learning has become the most widely used approach for cardiac image segmentation in recent years. In this paper, we provide a review of over 100 cardiac image segmentation papers using deep learning, which covers common imaging modalities including magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound and major anatomical structures of interest (ventricles, atria, and vessels). In addition, a summary of publicly available cardiac image datasets and code repositories are included to provide a base for encouraging reproducible research. Finally, we discuss the challenges and limitations with current deep learning-based approaches (scarcity of labels, model generalizability across different domains, interpretability) and suggest potential directions for future research.

DW-Net: A cascaded convolutional neural network for apical four-chamber view segmentation in fetal echocardiography

Fetal echocardiography (FE) is a widely used medical examination for early diagnosis of congenital heart disease (CHD). The apical four-chamber view (A4C) is an important view among early FE images. Accurate segmentation of crucial anatomical structures in the A4C view is a useful and important step for early diagnosis and timely treatment of CHDs. However, it is a challenging task due to several unfavorable factors: (a) artifacts and speckle noise produced by ultrasound imaging. (b) category confusion caused by the similarity of anatomical structures and variations of scanning angles. (c) missing boundaries. In this paper, we propose an end-to-end DW-Net for accurate segmentation of seven important anatomical structures in the A4C view. The network comprises two components: 1) a Dilated Convolutional Chain (DCC) for "gridding issue" reduction, multi-scale contextual information aggregation and accurate localization of cardiac chambers. 2) a W-Net for gaining more precise boundaries and yielding refined segmentation results. Extensive experiments of the proposed method on a dataset of 895 A4C views have demonstrated that DW-Net can achieve good segmentation results, including the Dice Similarity Coefficient (DSC) of 0.827, the Pixel Accuracy (PA) of 0.933, the AUC of 0.990 and it substantially outperformed some well-known segmentation methods. Our work was highly valued by experienced clinicians. The accurate and automatic segmentation of the A4C view using the proposed DW-Net can benefit further extractions of useful clinical indicators in early FE and improve the prenatal diagnostic accuracy and efficiency of CHDs.

Spatio-Temporal Segmentation in 3-D Echocardiographic Sequences Using Fractional Brownian Motion

An important aspect for an improved cardiac functional analysis is the accurate segmentation of the left ventricle (LV). A novel approach for fully-automated segmentation of the LV endocardium and epicardium contours is presented. This is mainly based on the natural physical characteristics of the LV shape structure. Both sides of the LV boundaries exhibit natural elliptical curvatures by having details on various scales, i.e. exhibiting fractal-like characteristics. The fractional Brownian motion (fBm), which is a non-stationary stochastic process, integrates well with the stochastic nature of ultrasound echoes. It has the advantage of representing a wide range of non-stationary signals and can quantify statistical local self-similarity throughout the time-sequence ultrasound images. The locally characterized boundaries of the fBm segmented LV were further iteratively refined using global information by means of second-order moments. The method is benchmarked using synthetic 3D+time echocardiographic sequences for normal and different ischemic cardiomyopathy, and results compared with state-of-the-art LV segmentation. Furthermore, the framework was validated against real data from canine cases with expert-defined segmentations and demonstrated improved accuracy. The fBm-based segmentation algorithm is fully automatic and has the potential to be used clinically together with 3D echocardiography for improved cardiovascular disease diagnosis.

Deep Learning for Segmentation Using an Open Large-Scale Dataset in 2D Echocardiography

Delineation of the cardiac structures from 2D echocardiographic images is a common clinical task to establish a diagnosis. Over the past decades, the automation of this task has been the subject of intense research. In this paper, we evaluate how far the state-of-the-art encoder-decoder deep convolutional neural network methods can go at assessing 2D echocardiographic images, i.e., segmenting cardiac structures and estimating clinical indices, on a dataset, especially, designed to answer this objective. We, therefore, introduce the cardiac acquisitions for multi-structure ultrasound segmentation dataset, the largest publicly-available and fully-annotated dataset for the purpose of echocardiographic assessment. The dataset contains two and four-chamber acquisitions from 500 patients with reference measurements from one cardiologist on the full dataset and from three cardiologists on a fold of 50 patients. Results show that encoder-decoder-based architectures outperform state-of-the-art non-deep learning methods and faithfully reproduce the expert analysis for the end-diastolic and end-systolic left ventricular volumes, with a mean correlation of 0.95 and an absolute mean error of 9.5 ml. Concerning the ejection fraction of the left ventricle, results are more contrasted with a mean correlation coefficient of 0.80 and an absolute mean error of 5.6%. Although these results are below the inter-observer scores, they remain slightly worse than the intra-observer's ones. Based on this observation, areas for improvement are defined, which open the door for accurate and fully-automatic analysis of 2D echocardiographic images.

Benchmarking Wilms' tumor in multisequence MRI data: why does current clinical practice fail? Which popular segmentation algorithms perform well?

Wilms' tumor is one of the most frequent malignant solid tumors in childhood. Accurate segmentation of tumor tissue is a key step during therapy and treatment planning. Since it is difficult to obtain a comprehensive set of tumor data of children, there is no benchmark so far allowing evaluation of the quality of human or computer-based segmentations. The contributions in our paper are threefold: (i) we present the first heterogeneous Wilms' tumor benchmark data set. It contains multisequence MRI data sets before and after chemotherapy, along with ground truth annotation, approximated based on the consensus of five human experts. (ii) We analyze human expert annotations and interrater variability, finding that the current clinical practice of determining tumor volume is inaccurate and that manual annotations after chemotherapy may differ substantially. (iii) We evaluate six computer-based segmentation methods, ranging from classical approaches to recent deep-learning techniques. We show that the best ones offer a quality comparable to human expert annotations.

Deep learning for cardiovascular medicine: a practical primer

Deep learning (DL) is a branch of machine learning (ML) showing increasing promise in medicine, to assist in data classification, novel disease phenotyping and complex decision making. Deep learning is a form of ML typically implemented via multi-layered neural networks. Deep learning has accelerated by recent advances in computer hardware and algorithms and is increasingly applied in e-commerce, finance, and voice and image recognition to learn and classify complex datasets. The current medical literature shows both strengths and limitations of DL. Strengths of DL include its ability to automate medical image interpretation, enhance clinical decision-making, identify novel phenotypes, and select better treatment pathways in complex diseases. Deep learning may be well-suited to cardiovascular medicine in which haemodynamic and electrophysiological indices are increasingly captured on a continuous basis by wearable devices as well as image segmentation in cardiac imaging. However, DL also has significant weaknesses including difficulties in interpreting its models (the 'black-box' criticism), its need for extensive adjudicated ('labelled') data in training, lack of standardization in design, lack of data-efficiency in training, limited applicability to clinical trials, and other factors. Thus, the optimal clinical application of DL requires careful formulation of solvable problems, selection of most appropriate DL algorithms and data, and balanced interpretation of results. This review synthesizes the current state of DL for cardiovascular clinicians and investigators, and provides technical context to appreciate the promise, pitfalls, near-term challenges, and opportunities for this exciting new area.

Why rankings of biomedical image analysis competitions should be interpreted with care

International challenges have become the standard for validation of biomedical image analysis methods. Given their scientific impact, it is surprising that a critical analysis of common practices related to the organization of challenges has not yet been performed. In this paper, we present a comprehensive analysis of biomedical image analysis challenges conducted up to now. We demonstrate the importance of challenges and show that the lack of quality control has critical consequences. First, reproducibility and interpretation of the results is often hampered as only a fraction of relevant information is typically provided. Second, the rank of an algorithm is generally not robust to a number of variables such as the test data used for validation, the ranking scheme applied and the observers that make the reference annotations. To overcome these problems, we recommend best practice guidelines and define open research questions to be addressed in the future.

Cloud Deployment of High-Resolution Medical Image Analysis With TOMAAT

BACKGROUND

Deep learning has been recently applied to a multitude of computer vision and medical image analysis problems. Although recent research efforts have improved the state of the art, most of the methods cannot be easily accessed, compared or used by other researchers or clinicians. Even if developers publish their code and pre-trained models on the internet, integration in stand-alone applications and existing workflows is often not straightforward, especially for clinical research partners. In this paper, we propose an open-source framework to provide AI-enabled medical image analysis through the network.

METHODS

TOMAAT provides a cloud environment for general medical image analysis, composed of three basic components: (i) an announcement service, maintaining a public registry of (ii) multiple distributed server nodes offering various medical image analysis solutions, and (iii) client software offering simple interfaces for users. Deployment is realized through HTTP-based communication, along with an API and wrappers for common image manipulations during pre- and post-processing.

RESULTS

We demonstrate the utility and versatility of TOMAAT on several hallmark medical image analysis tasks: segmentation, diffeomorphic deformable atlas registration, landmark localization, and workflow integration. Through TOMAAT, the high hardware demands, setup and model complexity of demonstrated approaches are transparent to users, who are provided with simple client interfaces. We present example clients in three-dimensional Slicer, in the web browser, on iOS devices and in a commercially available, certified medical image analysis suite.

CONCLUSION

TOMAAT enables deployment of state-of-the-art image segmentation in the cloud, fostering interaction among deep learning researchers and medical collaborators in the clinic. Currently, a public announcement service is hosted by the authors, and several ready-to-use services are registered and enlisted at http://tomaat.cloud.

Artificial intelligence and echocardiography

Echocardiography plays a crucial role in the diagnosis and management of cardiovascular disease. However, interpretation remains largely reliant on the subjective expertise of the operator. As a result inter-operator variability and experience can lead to incorrect diagnoses. Artificial intelligence (AI) technologies provide new possibilities for echocardiography to generate accurate, consistent and automated interpretation of echocardiograms, thus potentially reducing the risk of human error. In this review, we discuss a subfield of AI relevant to image interpretation, called machine learning, and its potential to enhance the diagnostic performance of echocardiography. We discuss recent applications of these methods and future directions for AI-assisted interpretation of echocardiograms. The research suggests it is feasible to apply machine learning models to provide rapid, highly accurate and consistent assessment of echocardiograms, comparable to clinicians. These algorithms are capable of accurately quantifying a wide range of features, such as the severity of valvular heart disease or the ischaemic burden in patients with coronary artery disease. However, the applications and their use are still in their infancy within the field of echocardiography. Research to refine methods and validate their use for automation, quantification and diagnosis are in progress. Widespread adoption of robust AI tools in clinical echocardiography practice should follow and have the potential to deliver significant benefits for patient outcome.

Automatic Assessment of Full Left Ventricular Coverage in Cardiac Cine Magnetic Resonance Imaging with Fisher Discriminative 3D CNN

Cardiac magnetic resonance (CMR) images play a growing role in the diagnostic imaging of cardiovascular diseases. Full coverage of the left ventricle (LV), from base to apex, is a basic criterion for CMR image quality and necessary for accurate measurement of cardiac volume and functional assessment. Incomplete coverage of the LV is identified through visual inspection, which is time-consuming and usually done retrospectively in the assessment of large imaging cohorts. This paper proposes a novel automatic method for determining LV coverage from CMR images by using Fisher-discriminative three-dimensional (FD3D) convolutional neural networks (CNNs). In contrast to our previous method employing 2D CNNs, this approach utilizes spatial contextual information in CMR volumes, extracts more representative high-level features and enhances the discriminative capacity of the baseline 2D CNN learning framework, thus achieving superior detection accuracy. A two-stage framework is proposed to identify missing basal and apical slices in measurements of CMR volume. First, the FD3D CNN extracts high-level features from the CMR stacks. These image representations are then used to detect the missing basal and apical slices. Compared to the traditional 3D CNN strategy, the proposed FD3D CNN minimizes within-class scatter and maximizes between-class scatter. We performed extensive experiments to validate the proposed method on more than 5,000 independent volumetric CMR scans from the UK Biobank study, achieving low error rates for missing basal/apical slice detection (4.9%/4.6%). The proposed method can also be adopted for assessing LV coverage for other types of CMR image data.

A Combined Fully Convolutional Networks and Deformable Model for Automatic Left Ventricle Segmentation Based on 3D Echocardiography

Segmentation of the left ventricle (LV) from three-dimensional echocardiography (3DE) plays a key role in the clinical diagnosis of the LV function. In this work, we proposed a new automatic method for the segmentation of LV, based on the fully convolutional networks (FCN) and deformable model. This method implemented a coarse-to-fine framework. Firstly, a new deep fusion network based on feature fusion and transfer learning, combining the residual modules, was proposed to achieve coarse segmentation of LV on 3DE. Secondly, we proposed a method of geometrical model initialization for a deformable model based on the results of coarse segmentation. Thirdly, the deformable model was implemented to further optimize the segmentation results with a regularization item to avoid the leakage between left atria and left ventricle to achieve the goal of fine segmentation of LV. Numerical experiments have demonstrated that the proposed method outperforms the state-of-the-art methods on the challenging CETUS benchmark in the segmentation accuracy and has a potential for practical applications.

Convolutional Neural Network With Shape Prior Applied to Cardiac MRI Segmentation

In this paper, we present a novel convolutional neural network architecture to segment images from a series of short-axis cardiac magnetic resonance slices (CMRI). The proposed model is an extension of the U-net that embeds a cardiac shape prior and involves a loss function tailored to the cardiac anatomy. Since the shape prior is computed offline only once, the execution of our model is not limited by its calculation. Our system takes as input raw magnetic resonance images, requires no manual preprocessing or image cropping and is trained to segment the endocardium and epicardium of the left ventricle, the endocardium of the right ventricle, as well as the center of the left ventricle. With its multiresolution grid architecture, the network learns both high and low-level features useful to register the shape prior as well as accurately localize the borders of the cardiac regions. Experimental results obtained on the Automatic Cardiac Diagnostic Challenge - Medical Image Computing and Computer Assisted Intervention (ACDC-MICCAI) 2017 dataset show that our model segments multislices CMRI (left and right ventricle contours) in 0.18 s with an average Dice coefficient of [Formula: see text] and an average 3-D Hausdorff distance of [Formula: see text] mm.

Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation

Incorporation of prior knowledge about organ shape and location is key to improve performance of image analysis approaches. In particular, priors can be useful in cases where images are corrupted and contain artefacts due to limitations in image acquisition. The highly constrained nature of anatomical objects can be well captured with learning-based techniques. However, in most recent and promising techniques such as CNN-based segmentation it is not obvious how to incorporate such prior knowledge. State-of-the-art methods operate as pixel-wise classifiers where the training objectives do not incorporate the structure and inter-dependencies of the output. To overcome this limitation, we propose a generic training strategy that incorporates anatomical prior knowledge into CNNs through a new regularisation model, which is trained end-to-end. The new framework encourages models to follow the global anatomical properties of the underlying anatomy (e.g. shape, label structure) via learnt non-linear representations of the shape. We show that the proposed approach can be easily adapted to different analysis tasks (e.g. image enhancement, segmentation) and improve the prediction accuracy of the state-of-the-art models. The applicability of our approach is shown on multi-modal cardiac data sets and public benchmarks. In addition, we demonstrate how the learnt deep models of 3-D shapes can be interpreted and used as biomarkers for classification of cardiac pathologies.

Patient-specific model-based segmentation of brain tumors in 3D intraoperative ultrasound images

PURPOSE

Intraoperative ultrasound (iUS) imaging is commonly used to support brain tumor operation. The tumor segmentation in the iUS images is a difficult task and still under improvement because of the low signal-to-noise ratio. The success of automatic methods is also limited due to the high noise sensibility. Therefore, an alternative brain tumor segmentation method in 3D-iUS data using a tumor model obtained from magnetic resonance (MR) data for local MR-iUS registration is presented in this paper. The aim is to enhance the visualization of the brain tumor contours in iUS.

METHODS

A multistep approach is proposed. First, a region of interest (ROI) based on the specific patient tumor model is defined. Second, hyperechogenic structures, mainly tumor tissues, are extracted from the ROI of both modalities by using automatic thresholding techniques. Third, the registration is performed over the extracted binary sub-volumes using a similarity measure based on gradient values, and rigid and affine transformations. Finally, the tumor model is aligned with the 3D-iUS data, and its contours are represented.

RESULTS

Experiments were successfully conducted on a dataset of 33 patients. The method was evaluated by comparing the tumor segmentation with expert manual delineations using two binary metrics: contour mean distance and Dice index. The proposed segmentation method using local and binary registration was compared with two grayscale-based approaches. The outcomes showed that our approach reached better results in terms of computational time and accuracy than the comparative methods.

CONCLUSION

The proposed approach requires limited interaction and reduced computation time, making it relevant for intraoperative use. Experimental results and evaluations were performed offline. The developed tool could be useful for brain tumor resection supporting neurosurgeons to improve tumor border visualization in the iUS volumes.

Fast left ventricle tracking using localized anatomical affine optical flow

In daily clinical cardiology practice, left ventricle (LV) global and regional function assessment is crucial for disease diagnosis, therapy selection, and patient follow-up. Currently, this is still a time-consuming task, spending valuable human resources. In this work, a novel fast methodology for automatic LV tracking is proposed based on localized anatomically constrained affine optical flow. This novel method can be combined to previously proposed segmentation frameworks or manually delineated surfaces at an initial frame to obtain fully delineated datasets and, thus, assess both global and regional myocardial function. Its feasibility and accuracy were investigated in 3 distinct public databases, namely in realistically simulated 3D ultrasound, clinical 3D echocardiography, and clinical cine cardiac magnetic resonance images. The method showed accurate tracking results in all databases, proving its applicability and accuracy for myocardial function assessment. Moreover, when combined to previous state-of-the-art segmentation frameworks, it outperformed previous tracking strategies in both 3D ultrasound and cardiac magnetic resonance data, automatically computing relevant cardiac indices with smaller biases and narrower limits of agreement compared to reference indices. Simultaneously, the proposed localized tracking method showed to be suitable for online processing, even for 3D motion assessment. Importantly, although here evaluated for LV tracking only, this novel methodology is applicable for tracking of other target structures with minimal adaptations.

Fast and Fully Automatic Left Ventricular Segmentation and Tracking in Echocardiography Using Shape-Based B-Spline Explicit Active Surfaces

Cardiac volume/function assessment remains a critical step in daily cardiology, and 3-D ultrasound plays an increasingly important role. Fully automatic left ventricular segmentation is, however, a challenging task due to the artifacts and low contrast-to-noise ratio of ultrasound imaging. In this paper, a fast and fully automatic framework for the full-cycle endocardial left ventricle segmentation is proposed. This approach couples the advantages of the B-spline explicit active surfaces framework, a purely image information approach, to those of statistical shape models to give prior information about the expected shape for an accurate segmentation. The segmentation is propagated throughout the heart cycle using a localized anatomical affine optical flow. It is shown that this approach not only outperforms other state-of-the-art methods in terms of distance metrics with a mean average distances of 1.81±0.59 and 1.98±0.66 mm at end-diastole and end-systole, respectively, but is computationally efficient (in average 11 s per 4-D image) and fully automatic.

The Role of Automated 3D Echocardiography for Left Ventricular Ejection Fraction Assessment

Ejection fraction is one of the most powerful determinants of prognosis and is a crucial parameter for the determination of cardiovascular therapies in conditions such as heart failure, valvular conditions and ischaemic heart disease. Among echocardiographic methods, 3D echocardiography has been attributed as the preferred one for its assessment, given an increased accuracy and reproducibility. Full-volume multi-beat acquisitions are prone to stitching artefacts due to arrhythmias and require prolonged breath holds. Single-beat acquisitions exhibit a lower temporal resolution, but address the limitations of multi-beat acquisitions. If not fully automated, 3D echocardiography remains time-consuming and resource-intensive, with suboptimal observer variability, preventing its implementation in routine practice. Further developments in hardware and software, including fully automated knowledge-based algorithms for left ventricular quantification, may bring 3D echocardiography to a definite turning point.

Fully Automatic 3-D-TEE Segmentation for the Planning of Transcatheter Aortic Valve Implantation

A novel fully automatic framework for aortic valve (AV) trunk segmentation in three-dimensional (3-D) transesophageal echocardiography (TEE) datasets is proposed. The methodology combines a previously presented semiautomatic segmentation strategy by using shape-based B-spline Explicit Active Surfaces with two novel algorithms to automate the quantification of relevant AV measures. The first combines a fast rotation-invariant 3-D generalized Hough transform with a vessel-like dark tube detector to initialize the segmentation. After segmenting the AV wall, the second algorithm focuses on aligning this surface with the reference ones in order to estimate the short-axis (SAx) planes (at the left ventricular outflow tract, annulus, sinuses of Valsalva, and sinotubular junction) in which to perform the measurements. The framework has been tested in 20 3-D-TEE datasets with both stenotic and nonstenotic AVs. The initialization algorithm presented a median error of around 3 mm for the AV axis endpoints, with an overall feasibility of 90%. In its turn, the SAx detection algorithm showed to be highly reproducible, with indistinguishable results compared with the variability found between the experts' defined planes. Automatically extracted measures at the four levels showed a good agreement with the experts' ones, with limits of agreement similar to the interobserver variability. Moreover, a validation set of 20 additional stenotic AV datasets corroborated the method's applicability and accuracy. The proposed approach mitigates the variability associated with the manual quantification while significantly reducing the required analysis time (12 s versus 5 to 10 min), which shows its appeal for automatic dimensioning of the AV morphology in 3-D-TEE for the planning of transcatheter AV implantation.

Aortic Valve Tract Segmentation From 3D-TEE Using Shape-Based B-Spline Explicit Active Surfaces

A novel semi-automatic algorithm for aortic valve (AV) wall segmentation is presented for 3D transesophageal echocardiography (TEE) datasets. The proposed methodology uses a 3D cylindrical formulation of the B-spline Explicit Active Surfaces (BEAS) framework in a dual-stage energy evolution process, comprising a threshold-based and a localized region-based stage. Hereto, intensity and shape-based features are combined to accurately delineate the AV wall from the ascending aorta (AA) to the left ventricular outflow tract (LVOT). Shape-prior information is included using a profile-based statistical shape model (SSM), and embedded in BEAS through two novel regularization terms: one confining the segmented AV profiles to shapes seen in the SSM (hard regularization) and another penalizing according to the profile's degree of likelihood (soft regularization). The proposed energy functional takes thus advantage of the intensity data in regions with strong image content, while complementing it with shape knowledge in regions with nearly absent image data. The proposed algorithm has been validated in 20 3D-TEE datasets with both stenotic and non-stenotic valves. It was shown to be accurate, robust and computationally efficient, taking less than 1 second to segment the AV wall from the AA to the LVOT with an average accuracy of 0.78 mm. Semi-automatically extracted measurements at four relevant anatomical levels (LVOT, aortic annulus, sinuses of Valsalva and sinotubular junction) showed an excellent agreement with experts' ones, with a higher reproducibility than manually-extracted measures.

Detailed Evaluation of Five 3D Speckle Tracking Algorithms Using Synthetic Echocardiographic Recordings

A plethora of techniques for cardiac deformation imaging with 3D ultrasound, typically referred to as 3D speckle tracking techniques, are available from academia and industry. Although the benefits of single methods over alternative ones have been reported in separate publications, the intrinsic differences in the data and definitions used makes it hard to compare the relative performance of different solutions. To address this issue, we have recently proposed a framework to simulate realistic 3D echocardiographic recordings and used it to generate a common set of ground-truth data for 3D speckle tracking algorithms, which was made available online. The aim of this study was therefore to use the newly developed database to contrast non-commercial speckle tracking solutions from research groups with leading expertise in the field. The five techniques involved cover the most representative families of existing approaches, namely block-matching, radio-frequency tracking, optical flow and elastic image registration. The techniques were contrasted in terms of tracking and strain accuracy. The feasibility of the obtained strain measurements to diagnose pathology was also tested for ischemia and dyssynchrony.