A SEMI-SUPERVISED JOINT LEARNING APPROACH TO LEFT VENTRICULAR SEGMENTATION AND MOTION TRACKING IN ECHOCARDIOGRAPHY

Multi-Task Learning for Motion Analysis and Segmentation in 3D Echocardiography

Characterizing left ventricular deformation and strain using 3D+time echocardiography provides useful insights into cardiac function and can be used to detect and localize myocardial injury. To achieve this, it is imperative to obtain accurate motion estimates of the left ventricle. In many strain analysis pipelines, this step is often accompanied by a separate segmentation step; however, recent works have shown both tasks to be highly related and can be complementary when optimized jointly. In this work, we present a multi-task learning network that can simultaneously segment the left ventricle and track its motion between multiple time frames. Two task-specific networks are trained using a composite loss function. Cross-stitch units combine the activations of these networks by learning shared representations between the tasks at different levels. We also propose a novel shape-consistency unit that encourages motion propagated segmentations to match directly predicted segmentations. Using a combined synthetic and in-vivo 3D echocardiography dataset, we demonstrate that our proposed model can achieve excellent estimates of left ventricular motion displacement and myocardial segmentation. Additionally, we observe strong correlation of our image-based strain measurements with crystal-based strain measurements as well as good correspondence with SPECT perfusion mappings. Finally, we demonstrate the clinical utility of the segmentation masks in estimating ejection fraction and sphericity indices that correspond well with benchmark measurements.

Motion Estimation by Deep Learning in 2D Echocardiography: Synthetic Dataset and Validation

Motion estimation in echocardiography plays an important role in the characterization of cardiac function, allowing the computation of myocardial deformation indices. However, there exist limitations in clinical practice, particularly with regard to the accuracy and robustness of measurements extracted from images. We therefore propose a novel deep learning solution for motion estimation in echocardiography. Our network corresponds to a modified version of PWC-Net which achieves high performance on ultrasound sequences. In parallel, we designed a novel simulation pipeline allowing the generation of a large amount of realistic B-mode sequences. These synthetic data, together with strategies during training and inference, were used to improve the performance of our deep learning solution, which achieved an average endpoint error of 0.07 ± 0.06 mm per frame and 1.20 ± 0.67 mm between ED and ES on our simulated dataset. The performance of our method was further investigated on 30 patients from a publicly available clinical dataset acquired from a GE system. The method showed promise by achieving a mean absolute error of the global longitudinal strain of 2.5 ± 2.1% and a correlation of 0.77 compared to GLS derived from manual segmentation, much better than one of the most efficient methods in the state-of-the-art (namely the FFT-Xcorr block-matching method). We finally evaluated our method on an auxiliary dataset including 30 patients from another center and acquired with a different system. Comparable results were achieved, illustrating the ability of our method to maintain high performance regardless of the echocardiographic data processed.

Simultaneous Segmentation and Motion Estimation of Left Ventricular Myocardium in 3D Echocardiography Using Multi-task Learning

Motion estimation and segmentation are both critical steps in identifying and assessing myocardial dysfunction, but are traditionally treated as unique tasks and solved as separate steps. However, many motion estimation techniques rely on accurate segmentations. It has been demonstrated in the computer vision and medical image analysis literature that both these tasks may be mutually beneficial when solved simultaneously. In this work, we propose a multi-task learning network that can concurrently predict volumetric segmentations of the left ventricle and estimate motion between 3D echocardiographic image pairs. The model exploits complementary latent features between the two tasks using a shared feature encoder with task-specific decoding branches. Anatomically inspired constraints are incorporated to enforce realistic motion patterns. We evaluate our proposed model on an in vivo 3D echocardiographic canine dataset. Results suggest that coupling these two tasks in a learning framework performs favorably when compared against single task learning and other alternative methods.

Advanced Ultrasound and Photoacoustic Imaging in Cardiology

Cardiovascular diseases (CVDs) remain the leading cause of death worldwide. An effective management and treatment of CVDs highly relies on accurate diagnosis of the disease. As the most common imaging technique for clinical diagnosis of the CVDs, US imaging has been intensively explored. Especially with the introduction of deep learning (DL) techniques, US imaging has advanced tremendously in recent years. Photoacoustic imaging (PAI) is one of the most promising new imaging methods in addition to the existing clinical imaging methods. It can characterize different tissue compositions based on optical absorption contrast and thus can assess the functionality of the tissue. This paper reviews some major technological developments in both US (combined with deep learning techniques) and PA imaging in the application of diagnosis of CVDs.

SHAPE-REGULARIZED UNSUPERVISED LEFT VENTRICULAR MOTION NETWORK WITH SEGMENTATION CAPABILITY IN 3D+TIME ECHOCARDIOGRAPHY

Accurate motion estimation and segmentation of the left ventricle from medical images are important tasks for quantitative evaluation of cardiovascular health. Echocardiography offers a cost-efficient and non-invasive modality for examining the heart, but provides additional challenges for automated analyses due to the low signal-to-noise ratio inherent in ultrasound imaging. In this work, we propose a shape regularized convolutional neural network for estimating dense displacement fields between sequential 3D B-mode echocardiography images with the capability of also predicting left ventricular segmentation masks. Manually traced segmentations are used as a guide to assist in the unsupervised estimation of displacement between a source and a target image while also serving as labels to train the network to additionally predict segmentations. To enforce realistic cardiac motion patterns, a flow incompressibility term is also incorporated to penalize divergence. Our proposed network is evaluated on an in vivo canine 3D+t B-mode echocardiographic dataset. It is shown that the shape regularizer improves the motion estimation performance of the network and our overall model performs favorably against competing methods.

Multi-frame Attention Network for Left Ventricle Segmentation in 3D Echocardiography

Echocardiography is one of the main imaging modalities used to assess the cardiovascular health of patients. Among the many analyses performed on echocardiography, segmentation of left ventricle is crucial to quantify the clinical measurements like ejection fraction. However, segmentation of left ventricle in 3D echocardiography remains a challenging and tedious task. In this paper, we propose a multi-frame attention network to improve the performance of segmentation of left ventricle in 3D echocardiography. The multi-frame attention mechanism allows highly correlated spatiotemporal features in a sequence of images that come after a target image to be used to augment the performance of segmentation. Experimental results shown on 51 in vivo porcine 3D+time echocardiography images show that utilizing correlated spatiotemporal features significantly improves the performance of left ventricle segmentation when compared to other standard deep learning-based medical image segmentation models.

A Semi-supervised Joint Network for Simultaneous Left Ventricular Motion Tracking and Segmentation in 4D Echocardiography

This work presents a novel deep learning method to combine segmentation and motion tracking in 4D echocardiography. The network iteratively trains a motion branch and a segmentation branch. The motion branch is initially trained entirely unsupervised and learns to roughly map the displacements between a source and a target frame. The estimated displacement maps are then used to generate pseudo-ground truth labels to train the segmentation branch. The labels predicted by the trained segmentation branch are fed back into the motion branch and act as landmarks to help retrain the branch to produce smoother displacement estimations. These smoothed out displacements are then used to obtain smoother pseudo-labels to retrain the segmentation branch. Additionally, a biomechanically-inspired incompressibility constraint is implemented in order to encourage more realistic cardiac motion. The proposed method is evaluated against other approaches using synthetic and in-vivo canine studies. Both the segmentation and motion tracking results of our model perform favorably against competing methods.