LU-Net: A Multistage Attention Network to Improve the Robustness of Segmentation of Left Ventricular Structures in 2-D Echocardiography

Deep learning supported echocardiogram analysis: A comprehensive review

An echocardiogram is a sophisticated ultrasound imaging technique employed to diagnose heart conditions. The transthoracic echocardiogram, one of the most prevalent types, is instrumental in evaluating significant cardiac diseases. However, interpreting its results heavily relies on the clinician's expertise. In this context, artificial intelligence has emerged as a vital tool for helping clinicians. This study critically analyzes key state-of-the-art research that uses deep learning techniques to automate transthoracic echocardiogram analysis and support clinical judgments. We have systematically organized and categorized articles that proffer solutions for view classification, enhancement of image quality and dataset, segmentation and identification of cardiac structures, detection of cardiac function abnormalities, and quantification of cardiac functions. We compared the performance of various deep learning approaches within each category, identifying the most promising methods. Additionally, we highlight limitations in current research and explore promising avenues for future exploration. These include addressing generalizability issues, incorporating novel AI approaches, and tackling the analysis of rare cardiac diseases.

Harnessing the Power of AI: A Comprehensive Review of Left Ventricular Ejection Fraction Assessment With Echocardiography

The quantification of left ventricular ejection fraction (LVEF) has important clinical utility in the assessment of cardiac function and is vital for the diagnosis of cardiovascular diseases. A transthoracic echocardiogram serves as the most commonly used tool for LVEF assessment for several reasons, including, its noninvasive nature, great safety profile, real-time image processing ability, portability, and cost-effectiveness. However, transthoracic echocardiogram is highly dependent on the clinical skill of the sonographer and interpreting physician. Moreover, even amongst well-trained clinicians, significant interobserver variability exists in the quantification of LVEF. In search of possible solutions, the usage of artificial intelligence (AI) has been increasingly tested in the clinical setting. While AI-derived ejection fraction is in the preliminary stages of development, it has shown promise in its ability to rapidly quantify LVEF, decrease variability, increase accuracy, and utilize higher-order processing capabilities. This review will delineate the latest advancements of AI in evaluating LVEF through echocardiography and explore the challenges and future trajectory of this emerging domain.

Attention-UNet architectures with pretrained backbones for multi-class cardiac MR image segmentation

Segmentation architectures based on deep learning proficient extraordinary results in medical imaging technologies. Computed tomography (CT) images and Magnetic Resonance Imaging (MRI) in diagnosis and treatment are increasing and significantly support the diagnostic process by removing the bottlenecks of manual segmentation. Cardiac Magnetic Resonance Imaging (CMRI) is a state-of-the-art imaging technique used to acquire vital heart measurements and has received extensive attention from researchers for automatic segmentation. Deep learning methods offer high-precision segmentation but still pose several difficulties, such as pixel homogeneity in nearby organs. The motivated study using the attention mechanism approach was introduced for medical images for automated algorithms. The experiment focuses on observing the impact of the attention mechanism with and without pretrained backbone networks on the UNet model. For the same, three networks are considered: Attention-UNet, Attention-UNet with resnet50 pretrained backbone and Attention-UNet with densenet121 pretrained backbone. The experiments are performed on the ACDC Challenge 2017 dataset. The performance is evaluated by conducting a comparative analysis based on the Dice Coefficient, IoU Coefficient, and cross-entropy loss calculations. The Attention-UNet, Attention-UNet with resnet50 pretrained backbone, and Attention-UNet with densenet121 pretrained backbone networks obtained Dice Coefficients of 0.9889, 0.9720, and 0.9801, respectively, along with corresponding IoU scores of 0.9781, 0.9457, and 0.9612. Results compared with the state-of-the-art methods indicate that the methods are on par with, or even superior in terms of both the Dice coefficient and Intersection-over-union.

Segmentation of Arm Ultrasound Images in Breast Cancer-Related Lymphedema: A Database and Deep Learning Algorithm

OBJECTIVE

Breast cancer treatment often causes the removal of or damage to lymph nodes of the patient's lymphatic drainage system. This side effect is the origin of Breast Cancer-Related Lymphedema (BCRL), referring to a noticeable increase in excess arm volume. Ultrasound imaging is a preferred modality for the diagnosis and progression monitoring of BCRL because of its low cost, safety, and portability. As the affected and unaffected arms look similar in B-mode ultrasound images, the thickness of the skin, subcutaneous fat, and muscle have been shown to be important biomarkers for this task. The segmentation masks are also helpful in monitoring the longitudinal changes in morphology and mechanical properties of tissue layers.

METHODS

For the first time, a publicly available ultrasound dataset containing the Radio-Frequency (RF) data of 39 subjects and manual segmentation masks by two experts, are provided. Inter- and intra-observer reproducibility studies performed on the segmentation maps show a high Dice Score Coefficient (DSC) of 0.94±0.08 and 0.92±0.06, respectively. Gated Shape Convolutional Neural Network (GSCNN) is modified for precise automatic segmentation of tissue layers, and its generalization performance is improved by the CutMix augmentation strategy.

RESULTS

We got an average DSC of 0.87±0.11 on the test set, which confirms the high performance of the method.

CONCLUSION

Automatic segmentation can pave the way for convenient and accessible staging of BCRL, and our dataset can facilitate development and validation of those methods.

SIGNIFICANCE

Timely diagnosis and treatment of BCRL have crucial importance in preventing irreversible damage.

EchoEFNet: Multi-task deep learning network for automatic calculation of left ventricular ejection fraction in 2D echocardiography

Left ventricular ejection fraction (LVEF) is essential for evaluating left ventricular systolic function. However, its clinical calculation requires the physician to interactively segment the left ventricle and obtain the mitral annulus and apical landmarks. This process is poorly reproducible and error prone. In this study, we propose a multi-task deep learning network EchoEFNet. The network use ResNet50 with dilated convolution as the backbone to extract high-dimensional features while maintaining spatial features. The branching network used our designed multi-scale feature fusion decoder to segment the left ventricle and detect landmarks simultaneously. The LVEF was then calculated automatically and accurately using the biplane Simpson's method. The model was tested for performance on the public dataset CAMUS and private dataset CMUEcho. The experimental results showed that the geometrical metrics and percentage of correct keypoints of EchoEFNet outperformed other deep learning methods. The correlation between the predicted LVEF and true values on the CAMUS and CMUEcho datasets was 0.854 and 0.916, respectively.

Fully Automatic Left Ventricle Segmentation Using Bilateral Lightweight Deep Neural Network

The segmentation of the left ventricle (LV) is one of the fundamental procedures that must be performed to obtain quantitative measures of the heart, such as its volume, area, and ejection fraction. In clinical practice, the delineation of LV is still often conducted semi-automatically, leaving it open to operator subjectivity. The automatic LV segmentation from echocardiography images is a challenging task due to poorly defined boundaries and operator dependency. Recent research has demonstrated that deep learning has the capability to employ the segmentation process automatically. However, the well-known state-of-the-art segmentation models still lack in terms of accuracy and speed. This study aims to develop a single-stage lightweight segmentation model that precisely and rapidly segments the LV from 2D echocardiography images. In this research, a backbone network is used to acquire both low-level and high-level features. Two parallel blocks, known as the spatial feature unit and the channel feature unit, are employed for the enhancement and improvement of these features. The refined features are merged by an integrated unit to segment the LV. The performance of the model and the time taken to segment the LV are compared to other established segmentation models, DeepLab, FCN, and Mask RCNN. The model achieved the highest values of the dice similarity index (0.9446), intersection over union (0.8445), and accuracy (0.9742). The evaluation metrics and processing time demonstrate that the proposed model not only provides superior quantitative results but also trains and segments the LV in less time, indicating its improved performance over competing segmentation models.

Echocardiography Segmentation With Enforced Temporal Consistency

Convolutional neural networks (CNN) have demonstrated their ability to segment 2D cardiac ultrasound images. However, despite recent successes according to which the intra-observer variability on end-diastole and end-systole images has been reached, CNNs still struggle to leverage temporal information to provide accurate and temporally consistent segmentation maps across the whole cycle. Such consistency is required to accurately describe the cardiac function, a necessary step in diagnosing many cardiovascular diseases. In this paper, we propose a framework to learn the 2D+time apical long-axis cardiac shape such that the segmented sequences can benefit from temporal and anatomical consistency constraints. Our method is a post-processing that takes as input segmented echocardiographic sequences produced by any state-of-the-art method and processes it in two steps to (i) identify spatio-temporal inconsistencies according to the overall dynamics of the cardiac sequence and (ii) correct the inconsistencies. The identification and correction of cardiac inconsistencies relies on a constrained autoencoder trained to learn a physiologically interpretable embedding of cardiac shapes, where we can both detect and fix anomalies. We tested our framework on 98 full-cycle sequences from the CAMUS dataset, which are available alongside this paper. Our temporal regularization method not only improves the accuracy of the segmentation across the whole sequences, but also enforces temporal and anatomical consistency.

Automated analysis of limited echocardiograms: Feasibility and relationship to outcomes in COVID-19

Background

As automated echocardiographic analysis is increasingly utilized, continued evaluation within hospital settings is important to further understand its potential value. The importance of cardiac involvement in patients hospitalized with COVID-19 provides an opportunity to evaluate the feasibility and clinical relevance of automated analysis applied to limited echocardiograms.

Methods

In this multisite US cohort, the feasibility of automated AI analysis was evaluated on 558 limited echocardiograms in patients hospitalized with COVID-19. Reliability of automated assessment of left ventricular (LV) volumes, ejection fraction (EF), and LV longitudinal strain (LS) was assessed against clinically obtained measures and echocardiographic findings. Automated measures were evaluated against patient outcomes using ROC analysis, survival modeling, and logistic regression for the outcomes of 30-day mortality and in-hospital sequelae.

Results

Feasibility of automated analysis for both LVEF and LS was 87.5% (488/558 patients). AI analysis was performed with biplane method in 300 (61.5%) and single plane apical 4- or 2-chamber analysis in 136 (27.9%) and 52 (10.7%) studies, respectively. Clinical LVEF was assessed using visual estimation in 192 (39.3%), biplane in 163 (33.4%), and single plane or linear methods in 104 (21.2%) of the 488 studies; 29 (5.9%) studies did not have clinically reported LVEF. LV LS was clinically reported in 80 (16.4%). Consistency between automated and clinical values demonstrated Pearson's R, root mean square error (RMSE) and intraclass correlation coefficient (ICC) of 0.61, 11.3% and 0.72, respectively, for LVEF; 0.73, 3.9% and 0.74, respectively for LS; 0.76, 24.4ml and 0.87, respectively, for end-diastolic volume; and 0.82, 12.8 ml, and 0.91, respectively, for end-systolic volume. Abnormal automated measures of LVEF and LS were associated with LV wall motion abnormalities, left atrial enlargement, and right ventricular dysfunction. Automated analysis was associated with outcomes, including survival.

Conclusion

Automated analysis was highly feasible on limited echocardiograms using abbreviated protocols, consistent with equivalent clinically obtained metrics, and associated with echocardiographic abnormalities and patient outcomes.

Artificial Intelligence-Enhanced Echocardiography for Systolic Function Assessment

The accurate assessment of left ventricular systolic function is crucial in the diagnosis and treatment of cardiovascular diseases. Left ventricular ejection fraction (LVEF) and global longitudinal strain (GLS) are the most critical indexes of cardiac systolic function. Echocardiography has become the mainstay of cardiac imaging for measuring LVEF and GLS because it is non-invasive, radiation-free, and allows for bedside operation and real-time processing. However, the human assessment of cardiac function depends on the sonographer’s experience, and despite their years of training, inter-observer variability exists. In addition, GLS requires post-processing, which is time consuming and shows variability across different devices. Researchers have turned to artificial intelligence (AI) to address these challenges. The powerful learning capabilities of AI enable feature extraction, which helps to achieve accurate identification of cardiac structures and reliable estimation of the ventricular volume and myocardial motion. Hence, the automatic output of systolic function indexes can be achieved based on echocardiographic images. This review attempts to thoroughly explain the latest progress of AI in assessing left ventricular systolic function and differential diagnosis of heart diseases by echocardiography and discusses the challenges and promises of this new field.

Explicit and automatic ejection fraction assessment on 2D cardiac ultrasound with a deep learning-based approach

BACKGROUND

Ejection fraction (EF) is a key parameter for assessing cardiovascular functions in cardiac ultrasound, but its manual assessment is time-consuming and subject to high inter and intra-observer variability. Deep learning-based methods have the potential to perform accurate fully automatic EF predictions but suffer from a lack of explainability and interpretability. This study proposes a fully automatic method to reliably and explicitly evaluate the biplane left ventricular EF on 2D echocardiography following the recommended modified Simpson's rule.

METHODS

A deep learning model was trained on apical 4 and 2-chamber echocardiography to segment the left ventricle and locate the mitral valve. Predicted segmentations are then validated with a statistical shape model, which detects potential failures that could impact the EF evaluation. Finally, the end-diastolic and end-systolic frames are identified based on the remaining LV segmentations' areas and EF is estimated on all available cardiac cycles.

RESULTS

Our approach was trained on a dataset of 783 patients. Its performances were evaluated on an internal and external dataset of respectively 200 and 450 patients. On the internal dataset, EF assessment achieved a mean absolute error of 6.10% and a bias of 1.56 ± 7.58% using multiple cardiac cycles. The approach evaluated EF with a mean absolute error of 5.39% and a bias of -0.74 ± 7.12% on the external dataset.

CONCLUSION

Following the recommended guidelines, we proposed an end-to-end fully automatic approach that achieves state-of-the-art performance in biplane EF evaluation while giving explicit details to clinicians.

Full-volume three-component intraventricular vector flow mapping by triplane color Doppler

Objective. Intraventricular vector flow mapping (iVFM) is a velocimetric technique for retrieving two-dimensional velocity vector fields of blood flow in the left ventricular cavity. This method is based on conventional color Doppler imaging, which makes iVFM compatible with the clinical setting. We have generalized the iVFM for a three-dimensional reconstruction (3D-iVFM). Approach. 3D-iVFM is able to recover three-component velocity vector fields in a full intraventricular volume by using a clinical echocardiographic triplane mode. The 3D-iVFM problem was written in the spherical (radial, polar, azimuthal) coordinate system associated to the six half-planes produced by the triplane mode. As with the 2D version, the method is based on the mass conservation, and free-slip boundary conditions on the endocardial wall. These mechanical constraints were imposed in a least-squares minimization problem that was solved through the method of Lagrange multipliers. We validated 3D-iVFM in silico in a patient-specific CFD (computational fluid dynamics) model of cardiac flow and tested its clinical feasibility in vivo in patients and in one volunteer. Main results. The radial and polar components of the velocity were recovered satisfactorily in the CFD setup (correlation coefficients, r  = 0.99 and 0.78). The azimuthal components were estimated with larger errors ( r  = 0.57) as only six samples were available in this direction. In both in silico and in vivo investigations, the dynamics of the intraventricular vortex that forms during diastole was deciphered by 3D-iVFM. In particular, the CFD results showed that the mean vorticity can be estimated accurately by 3D-iVFM. Significance. Our results tend to indicate that 3D-iVFM could provide full-volume echocardiographic information on left intraventricular hemodynamics from the clinical modality of triplane color Doppler.

Investigating Shift Variance of Convolutional Neural Networks in Ultrasound Image Segmentation

While accuracy is an evident criterion for ultrasound image segmentation, output consistency across different tests is equally crucial for tracking changes in regions of interest in applications such as monitoring the patients' response to treatment, measuring the progression or regression of the disease, reaching a diagnosis, or treatment planning. Convolutional neural networks (CNNs) have attracted rapidly growing interest in automatic ultrasound image segmentation recently. However, CNNs are not shift-equivariant, meaning that, if the input translates, e.g., in the lateral direction by one pixel, the output segmentation may drastically change. To the best of our knowledge, this problem has not been studied in ultrasound image segmentation or even more broadly in ultrasound images. Herein, we investigate and quantify the shift-variance problem of CNNs in this application and further evaluate the performance of a recently published technique, called BlurPooling, for addressing the problem. In addition, we propose the Pyramidal BlurPooling method that outperforms BlurPooling in both output consistency and segmentation accuracy. Finally, we demonstrate that data augmentation is not a replacement for the proposed method. Source code is available at http://code.sonography.ai.

Fully automated quantification of cardiac chamber and function assessment in 2-D echocardiography: clinical feasibility of deep learning-based algorithms

We aimed to compare the segmentation performance of the current prominent deep learning (DL) algorithms with ground-truth segmentations and to validate the reproducibility of the manually created 2D echocardiographic four cardiac chamber ground-truth annotation. Recently emerged DL based fully-automated chamber segmentation and function assessment methods have shown great potential for future application in aiding image acquisition, quantification, and suggestion for diagnosis. However, the performance of current DL algorithms have not previously been compared with each other. In addition, the reproducibility of ground-truth annotations which are the basis of these algorithms have not yet been fully validated. We retrospectively enrolled 500 consecutive patients who underwent transthoracic echocardiogram (TTE) from December 2019 to December 2020. Simple U-net, Res-U-net, and Dense-U-net algorithms were compared for the segmentation performances and clinical indices such as left atrial volume (LAV), left ventricular end diastolic volume (LVEDV), left ventricular end systolic volume (LVESV), LV mass, and ejection fraction (EF) were evaluated. The inter- and intra-observer variability analysis was performed by two expert sonographers for a randomly selected echocardiographic view in 100 patients (apical 2-chamber, apical 4-chamber, and parasternal short axis views). The overall performance of all DL methods was excellent [average dice similarity coefficient (DSC) 0.91 to 0.95 and average Intersection over union (IOU) 0.83 to 0.90], with the exception of LV wall area on PSAX view (average DSC of 0.83, IOU 0.72). In addition, there were no significant difference in clinical indices between ground truth and automated DL measurements. For inter- and intra-observer variability analysis, the overall intra observer reproducibility was excellent: LAV (ICC = 0.995), LVEDV (ICC = 0.996), LVESV (ICC = 0.997), LV mass (ICC = 0.991) and EF (ICC = 0.984). The inter-observer reproducibility was slightly lower as compared to intraobserver agreement: LAV (ICC = 0.976), LVEDV (ICC = 0.982), LVESV (ICC = 0.970), LV mass (ICC = 0.971), and EF (ICC = 0.899). The three current prominent DL-based fully automated methods are able to reliably perform four-chamber segmentation and quantification of clinical indices. Furthermore, we were able to validate the four cardiac chamber ground-truth annotation and demonstrate an overall excellent reproducibility, but still with some degree of inter-observer variability.

Physics-constrained intraventricular vector flow mapping by color Doppler

Color Doppler by transthoracic echocardiography creates two-dimensional fan-shaped maps of blood velocities in the cardiac cavities. It is a one-component velocimetric technique since it only returns the velocity components parallel to the ultrasound beams. Intraventricular vector flow mapping (iVFM) is a method to recover the blood velocity vectors from the Doppler scalar fields in an echocardiographic three-chamber view. We improved ouriVFM numerical scheme by imposing physical constraints. TheiVFM consisted in minimizing regularized Doppler residuals subject to the condition that two fluid-dynamics constraints were satisfied, namely planar mass conservation, and free-slip boundary conditions. The optimization problem was solved by using the Lagrange multiplier method. A finite-difference discretization of the optimization problem, written in the polar coordinate system centered on the cardiac ultrasound probe, led to a sparse linear system. The single regularization parameter was determined automatically for non-supervision considerations. The physics-constrained method was validated using realistic intracardiac flow data from a patient-specific computational fluid dynamics (CFD) model. The numerical evaluations showed that theiVFM-derived velocity vectors were in very good agreement with the CFD-based original velocities, with relative errors ranged between 0.3% and 12%. We calculated two macroscopic measures of flow in the cardiac region of interest, the mean vorticity and mean stream function, and observed an excellent concordance between physics-constrainediVFM and CFD. The capability of physics-constrainediVFM was finally tested within vivocolor Doppler data acquired in patients routinely examined in the echocardiographic laboratory. The vortex that forms during the rapid filling was deciphered. The physics-constrainediVFM algorithm is ready for pilot clinical studies and is expected to have a significant clinical impact on the assessment of diastolic function.

Artificial intelligence applied to support medical decisions for the automatic analysis of echocardiogram images: A systematic review

The echocardiogram is a test that is widely used in Heart Disease Diagnoses. However, its analysis is largely dependent on the physician's experience. In this regard, artificial intelligence has become an essential technology to assist physicians. This study is a Systematic Literature Review (SLR) of primary state-of-the-art studies that used Artificial Intelligence (AI) techniques to automate echocardiogram analyses. Searches on the leading scientific article indexing platforms using a search string returned approximately 1400 articles. After applying the inclusion and exclusion criteria, 118 articles were selected to compose the detailed SLR. This SLR presents a thorough investigation of AI applied to support medical decisions for the main types of echocardiogram (Transthoracic, Transesophageal, Doppler, Stress, and Fetal). The article's data extraction indicated that the primary research interest of the studies comprised four groups: 1) Improvement of image quality; 2) identification of the cardiac window vision plane; 3) quantification and analysis of cardiac functions, and; 4) detection and classification of cardiac diseases. The articles were categorized and grouped to show the main contributions of the literature to each type of ECHO. The results indicate that the Deep Learning (DL) methods presented the best results for the detection and segmentation of the heart walls, right and left atrium and ventricles, and classification of heart diseases using images/videos obtained by echocardiography. The models that used Convolutional Neural Network (CNN) and its variations showed the best results for all groups. The evidence produced by the results presented in the tabulation of the studies indicates that the DL contributed significantly to advances in echocardiogram automated analysis processes. Although several solutions were presented regarding the automated analysis of ECHO, this area of research still has great potential for further studies to improve the accuracy of results already known in the literature.

Dual attention enhancement feature fusion network for segmentation and quantitative analysis of paediatric echocardiography

Paediatric echocardiography is a standard method for screening congenital heart disease (CHD). The segmentation of paediatric echocardiography is essential for subsequent extraction of clinical parameters and interventional planning. However, it remains a challenging task due to (1) the considerable variation of key anatomic structures, (2) the poor lateral resolution affecting accurate boundary definition, (3) the existence of speckle noise and artefacts in echocardiographic images. In this paper, we propose a novel deep network to address these challenges comprehensively. We first present a dual-path feature extraction module (DP-FEM) to extract rich features via a channel attention mechanism. A high- and low-level feature fusion module (HL-FFM) is devised based on spatial attention, which selectively fuses rich semantic information from high-level features with spatial cues from low-level features. In addition, a hybrid loss is designed to deal with pixel-level misalignment and boundary ambiguities. Based on the segmentation results, we derive key clinical parameters for diagnosis and treatment planning. We extensively evaluate the proposed method on 4,485 two-dimensional (2D) paediatric echocardiograms from 127 echocardiographic videos. The proposed method consistently achieves better segmentation performance than other state-of-the-art methods, whichdemonstratesfeasibility for automatic segmentation and quantitative analysis of paediatric echocardiography. Our code is publicly available at https://github.com/end-of-the-century/Cardiac.