Deep learning architecture for 3D image super-resolution of late gadolinium enhanced cardiac MRI

Purpose

High-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) volumes are difficult to acquire due to the limitations of the maximal breath-hold time achievable by the patient. This results in anisotropic 3D volumes of the heart with high in-plane resolution, but low-through-plane resolution. Thus, we propose a 3D convolutional neural network (CNN) approach to improve the through-plane resolution of the cardiac LGE-MRI volumes.

Approach

We present a 3D CNN-based framework with two branches: a super-resolution branch to learn the mapping between low-resolution and high-resolution LGE-MRI volumes, and a gradient branch that learns the mapping between the gradient map of low-resolution LGE-MRI volumes and the gradient map of high-resolution LGE-MRI volumes. The gradient branch provides structural guidance to the CNN-based super-resolution framework. To assess the performance of the proposed CNN-based framework, we train two CNN models with and without gradient guidance, namely, dense deep back-projection network (DBPN) and enhanced deep super-resolution network. We train and evaluate our method on the 2018 atrial segmentation challenge dataset. Additionally, we also evaluate these trained models on the left atrial and scar quantification and segmentation challenge 2022 dataset to assess their generalization ability. Finally, we investigate the effect of the proposed CNN-based super-resolution framework on the 3D segmentation of the left atrium (LA) from these cardiac LGE-MRI image volumes.

Results

Experimental results demonstrate that our proposed CNN method with gradient guidance consistently outperforms bicubic interpolation and the CNN models without gradient guidance. Furthermore, the segmentation results, evaluated using Dice score, obtained using the super-resolved images generated by our proposed method are superior to the segmentation results obtained using the images generated by bicubic interpolation (p<0.01) and the CNN models without gradient guidance (p<0.05).

Conclusion

The presented CNN-based super-resolution method with gradient guidance improves the through-plane resolution of the LGE-MRI volumes and the structure guidance provided by the gradient branch can be useful to aid the 3D segmentation of cardiac chambers, such as LA, from the 3D LGE-MRI images.

Disparity refinement framework for learning-based stereo matching methods in cross-domain setting for laparoscopic images

Purpose

Stereo matching methods that enable depth estimation are crucial for visualization enhancement applications in computer-assisted surgery. Learning-based stereo matching methods have shown great promise in making accurate predictions on laparoscopic images. However, they require a large amount of training data, and their performance may be degraded due to domain shifts.

Approach

Maintaining robustness and improving the accuracy of learning-based methods are still open problems. To overcome the limitations of learning-based methods, we propose a disparity refinement framework consisting of a local disparity refinement method and a global disparity refinement method to improve the results of learning-based stereo matching methods in a cross-domain setting. Those learning-based stereo matching methods are pre-trained on a large public dataset of natural images and are tested on two datasets of laparoscopic images.

Results

Qualitative and quantitative results suggest that our proposed disparity framework can effectively refine disparity maps when they are noise-corrupted on an unseen dataset, without compromising prediction accuracy when the network can generalize well on an unseen dataset.

Conclusions

Our proposed disparity refinement framework could work with learning-based methods to achieve robust and accurate disparity prediction. Yet, as a large laparoscopic dataset for training learning-based methods does not exist and the generalization ability of networks remains to be improved, the incorporation of the proposed disparity refinement framework into existing networks will contribute to improving their overall accuracy and robustness associated with depth estimation.

Inpainting surgical occlusion from laparoscopic video sequences for robot-assisted interventions

Purpose

Medical technology for minimally invasive surgery has undergone a paradigm shift with the introduction of robot-assisted surgery. However, it is very difficult to track the position of the surgical tools in a surgical scene, so it is crucial to accurately detect and identify surgical tools. This task can be aided by deep learning-based semantic segmentation of surgical video frames. Furthermore, due to the limited working and viewing areas of these surgical instruments, there is a higher chance of complications from tissue injuries (e.g., tissue scars and tears).

Approach

With the aid of digital inpainting algorithms, we present an application that uses image segmentation to remove surgical instruments from laparoscopic/endoscopic video. We employ a modified U-Net architecture (U-NetPlus) to segment the surgical instruments. It consists of a redesigned decoder and a pre-trained VGG11 or VGG16 encoder. The decoder was modified by substituting an up-sampling operation based on nearest-neighbor interpolation for the transposed convolution operation. Furthermore, these interpolation weights do not need to be learned to perform upsampling, which eliminates the artifacts generated by the transposed convolution. In addition, we use a very fast and adaptable data augmentation technique to further enhance performance. The instrument segmentation mask is filled in (i.e., inpainted) by the tool removal algorithms using the previously acquired tool segmentation masks and either previous instrument-containing frames or instrument-free reference frames.

Results

We have shown the effectiveness of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 and 2017 EndoVis Challenge. We report a 90.20% DICE for binary segmentation, a 76.26% DICE for instrument part segmentation, and a 46.07% DICE for instrument type (i.e., all instruments) segmentation on the MICCAI 2017 challenge dataset using our U-NetPlus architecture, outperforming the results of earlier techniques used and tested on these data. In addition, we demonstrated the successful execution of the tool removal algorithm from surgical tool-free videos that contained moving surgical tools that were generated artificially.

Conclusions

Our application successfully separates and eliminates the surgical tool to reveal a view of the background tissue that was otherwise hidden by the tool, producing results that are visually similar to the actual data.

Learning feature descriptors for pre- and intra-operative point cloud matching for laparoscopic liver registration

PURPOSE

In laparoscopic liver surgery, preoperative information can be overlaid onto the intra-operative scene by registering a 3D preoperative model to the intra-operative partial surface reconstructed from the laparoscopic video. To assist with this task, we explore the use of learning-based feature descriptors, which, to our best knowledge, have not been explored for use in laparoscopic liver registration. Furthermore, a dataset to train and evaluate the use of learning-based descriptors does not exist.

METHODS

We present the LiverMatch dataset consisting of 16 preoperative models and their simulated intra-operative 3D surfaces. We also propose the LiverMatch network designed for this task, which outputs per-point feature descriptors, visibility scores, and matched points.

RESULTS

We compare the proposed LiverMatch network with a network closest to LiverMatch and a histogram-based 3D descriptor on the testing split of the LiverMatch dataset, which includes two unseen preoperative models and 1400 intra-operative surfaces. Results suggest that our LiverMatch network can predict more accurate and dense matches than the other two methods and can be seamlessly integrated with a RANSAC-ICP-based registration algorithm to achieve an accurate initial alignment.

CONCLUSION

The use of learning-based feature descriptors in laparoscopic liver registration (LLR) is promising, as it can help achieve an accurate initial rigid alignment, which, in turn, serves as an initialization for subsequent non-rigid registration.

Investigating the impact of class-dependent label noise in medical image classification

Label noise is inevitable in medical image databases developed for deep learning due to the inter-observer variability caused by the different levels of expertise of the experts annotating the images, and, in some cases, the automated methods that generate labels from medical reports. It is known that incorrect annotations or label noise can degrade the actual performance of supervised deep learning models and can bias the model's evaluation. Existing literature show that noise in one class has minimal impact on the model's performance for another class in natural image classification problems where different target classes have a relatively distinct shape and share minimal visual cues for knowledge transfer among the classes. However, it is not clear how class-dependent label noise affects the model's performance when operating on medical images, for which different output classes can be difficult to distinguish even for experts, and there is a high possibility of knowledge transfer across classes during the training period. We hypothesize that for medical image classification tasks where the different classes share a very similar shape with differences only in texture, the noisy label for one class might affect the performance across other classes, unlike the case when the target classes have different shapes and are visually distinct. In this paper, we study this hypothesis using two publicly available datasets: a 2D organ classification dataset with target organ classes being visually distinct, and a histopathology image classification dataset where the target classes look very similar visually. Our results show that the label noise in one class has a much higher impact on the model's performance on other classes for the histopathology dataset compared to the organ dataset.

A Deep Learning Framework to Estimate Elastic Modulus from Ultrasound Measured Displacement Fields

Ultrasound (US) elastography is a technique that enables non-invasive quantification of material properties, such as stiffness, from ultrasound images of deforming tissue. The displacement field is measured from the US images using image matching algorithms, and then a parameter, often the elastic modulus, is inferred or subsequently measured to identify potential tissue pathologies, such as cancerous tissues. Several traditional inverse problem approaches, loosely grouped as either direct or iterative, have been explored to estimate the elastic modulus. Nevertheless, the iterative techniques are typically slow and computationally intensive, while the direct techniques, although more computationally efficient, are very sensitive to measurement noise and require the full displacement field data (i.e., both vector components). In this work, we propose a deep learning approach to solve the inverse problem and recover the spatial distribution of the elastic modulus from one component of the US measured displacement field. The neural network used here is trained using only simulated data obtained via a forward finite element (FE) model with known variations in the modulus field, thus avoiding the reliance on large measurement data sets that may be challenging to acquire. A U-net based neural network is then used to predict the modulus distribution (i.e., solve the inverse problem) using the simulated forward data as input. We quantitatively evaluated our trained model with a simulated test dataset and observed a 0.0018 mean squared error (MSE) and a 1.14% mean absolute percent error (MAPE) between the reconstructed and ground truth elastic modulus. Moreover, we also qualitatively compared the output of our U-net model to experimentally measured displacement data acquired using a US elastography tissue-mimicking calibration phantom.

Learning Deep Representations of Cardiac Structures for 4D Cine MRI Image Segmentation through Semi-Supervised Learning

Learning good data representations for medical imaging tasks ensures the preservation of relevant information and the removal of irrelevant information from the data to improve the interpretability of the learned features. In this paper, we propose a semi-supervised model-namely, combine-all in semi-supervised learning (CqSL)-to demonstrate the power of a simple combination of a disentanglement block, variational autoencoder (VAE), generative adversarial network (GAN), and a conditioning layer-based reconstructor for performing two important tasks in medical imaging: segmentation and reconstruction. Our work is motivated by the recent progress in image segmentation using semi-supervised learning (SSL), which has shown good results with limited labeled data and large amounts of unlabeled data. A disentanglement block decomposes an input image into a domain-invariant spatial factor and a domain-specific non-spatial factor. We assume that medical images acquired using multiple scanners (different domain information) share a common spatial space but differ in non-spatial space (intensities, contrast, etc.). Hence, we utilize our spatial information to generate segmentation masks from unlabeled datasets using a generative adversarial network (GAN). Finally, to reconstruct the original image, our conditioning layer-based reconstruction block recombines spatial information with random non-spatial information sampled from the generative models. Our ablation study demonstrates the benefits of disentanglement in holding domain-invariant (spatial) as well as domain-specific (non-spatial) information with high accuracy. We further apply a structured L 2 similarity ( S L 2 SIM ) loss along with a mutual information minimizer (MIM) to improve the adversarially trained generative models for better reconstruction. Experimental results achieved on the STACOM 2017 ACDC cine cardiac magnetic resonance (MR) dataset suggest that our proposed (CqSL) model outperforms fully supervised and semi-supervised models, achieving an 83.2% performance accuracy even when using only 1% labeled data. We hypothesize that our proposed model has the potential to become an efficient semantic segmentation tool that may be used for domain adaptation in data-limited medical imaging scenarios, where annotations are expensive. Code, and experimental configurations will be made available publicly.

Toward a Quasi-dynamic Pulsed Field Electroporation Numerical Model for Cardiac Ablation: Predicting Tissue Conductance Changes and Ablation Lesion Patterns

Pulsed field ablation (PFA) has the potential to evolve into an efficient alternative to traditional RF ablation for atrial fibrillation treatment. However, achieving irreversible tissue electroporation is critical to suppressing arrhythmic pathways, raising the need for accurate lesion characterization. To understand the physics behind the tissue response PFA, we propose a quasi-dynamic model that quantifies tissue conductance at end-electroporation and identifies regions that have undergone fully irreversible electroporation (IRE). The model uses several parameters and numerically solves the electrical field diffusion into the tissue by iteratively updating the tissue conductance until equilibrium at end-electroporation. The model yields a steady-state tissue conductance map used to identify the irreversible lesion. We conducted numerical experiments mimicking a lasso catheter featuring nine 3-mm electrodes spaced circumferentially at 3.75 mm and fired sequentially using a 1500 V and 3000 V pulse amplitude. The IRE lesion region has a surface area and volume of 780 mm² and 1411 mm³, respectively, at 1500 V, and 1178 mm² and 2760 mm³, respectively, at 3000 V. Lesion discontinuity was observed at 5.0 mm depth with 1500 V, and 7.2 mm depth with 3000 V. This quasi-dynamic model yields tissue conductance maps, predicts irreversible lesion and lesion penumbra at end-electroporation, and confirms larger lesions with higher pulse amplitudes.

Joint Segmentation and Uncertainty Estimation of Ventricular Structures from Cardiac MRI using a Bayesian CondenseUNet

While convolutional neural networks (CNNs) have shown potential in segmenting cardiac structures from magnetic resonance (MR) images, their clinical applications still fall short of providing reliable cardiac segmentation. As a result, it is critical to quantify segmentation uncertainty in order to identify which segmentations might be troublesome. Moreover, quantifying uncertainty is critical in real-world scenarios, where input distributions are frequently moved from the training distribution due to sample bias and non-stationarity. Therefore, well-calibrated uncertainty estimates provide information on whether a model's output should (or should not) be trusted in such situations. In this work, we used a Bayesian version of our previously proposed CondenseUNet [1] framework featuring both a learned group structure and a regularized weight-pruner to reduce the computational cost in volumetric image segmentation and help quantify predictive uncertainty. Our study further showcases the potential of our deep-learning framework to evaluate the correlation between the uncertainty and the segmentation errors for a given model. The proposed model was trained and tested on the Automated Cardiac Diagnosis Challenge (ACDC) dataset featuring 150 cine cardiac MRI patient dataset for the segmentation and uncertainty estimation of the left ventricle (LV), right ventricle (RV), and myocardium (Myo) at end-diastole (ED) and end-systole (ES) phases.

Endoscope Localization and Dense Surgical Scene Reconstruction for Stereo Endoscopy by Unsupervised Optical Flow and Kanade-Lucas-Tomasi Tracking

In image-guided surgery, endoscope tracking and surgical scene reconstruction are critical, yet equally challenging tasks. We present a hybrid visual odometry and reconstruction framework for stereo endoscopy that leverages unsupervised learning-based and traditional optical flow methods to enable concurrent endoscope tracking and dense scene reconstruction. More specifically, to reconstruct texture-less tissue surfaces, we use an unsupervised learning-based optical flow method to estimate dense depth maps from stereo images. Robust 3D landmarks are selected from the dense depth maps and tracked via the Kanade-Lucas-Tomasi tracking algorithm. The hybrid visual odometry also benefits from traditional visual odometry modules, such as keyframe insertion and local bundle adjustment. We evaluate the proposed framework on endoscopic video sequences openly available via the SCARED dataset against both ground truth data, as well as two other state-of-the-art methods - ORB-SLAM2 and Endo-depth. Our proposed method achieved comparable results in terms of both RMS Absolute Trajectory Error and Cloud-to-Mesh RMS Error, suggesting its potential to enable accurate endoscope tracking and scene reconstruction.

A 3D Convolutional Neural Network with Gradient Guidance for Image Super-Resolution of Late Gadolinium Enhanced Cardiac MRI

In this paper, we describe a 3D convolutional neural network (CNN) framework to compute and generate super-resolution late gadolinium enhanced (LGE) cardiac magnetic resonance imaging (MRI) images. The proposed CNN framework consists of two branches: a super-resolution branch with a 3D dense deep back-projection network (DBPN) as the backbone to learn the mapping of low-resolution LGE cardiac volumes to high-resolution LGE cardiac volumes, and a gradient branch that learns the mapping of the gradient map of low resolution LGE cardiac volumes to the gradient map of their high-resolution counterparts. The gradient branch of the CNN provides additional cardiac structure information to the super-resolution branch to generate structurally more accurate super-resolution LGE MRI images. We conducted our experiments on the 2018 atrial segmentation challenge dataset. The proposed CNN framework achieved a mean peak signal-to-noise ratio (PSNR) of 30.91 and 25.66 and a mean structural similarity index measure (SSIM) of 0.91 and 0.75 on training the model on low-resolution images downsamp led by a scale factor of 2 and 4, respectively.

SPIE Medical Imaging 50th anniversary: historical review of the Image-Guided Procedures, Robotic Interventions, and Modeling conference

Purpose

Among the conferences comprising the Medical Imaging Symposium is the MI104 conference currently titled Image-Guided Procedures, Robotic Interventions, and Modeling, although its name has evolved through at least nine iterations over the last 30 years. Here, we discuss the important role that this forum has presented for researchers in the field during this time.

Approach

The origins of the conference are traced from its roots in Image Capture and Display in the late 1980s, and some of the major themes for which the conference and its proceedings have provided a valuable forum are highlighted.

Results

These major themes include image display/visualization, surgical tracking/navigation, surgical robotics, interventional imaging, image registration, and modeling. Exceptional work from the conference is highlighted by summarizing keynote lectures, the top 50 most downloaded proceedings papers over the last 30 years, the most downloaded paper each year, and the papers earning student paper and young scientist awards.

Conclusions

Looking forward and considering the burgeoning technologies, algorithms, and markets related to image-guided and robot-assisted interventions, we anticipate growth and ever increasing quality of the conference as well as increased interaction with sister conferences within the symposium.

Calibration of cine MRI segmentation probability for uncertainty estimation using a multi-task cross-task learning architecture

While deep learning has shown potential in solving a variety of medical image analysis problems including segmentation, registration, motion estimation, etc., their applications in the real-world clinical setting are still not affluent due to the lack of reliability caused by the failures of deep learning models in prediction. Furthermore, deep learning models need a large number of labeled datasets. In this work, we propose a novel method that incorporates uncertainty estimation to detect failures in the segmentation masks generated by CNNs. Our study further showcases the potential of our model to evaluate the correlation between the uncertainty and the segmentation errors for a given model. Furthermore, we introduce a multi-task cross-task learning consistency approach to enforce the correlation between the pixel-level (segmentation) and the geometric-level (distance map) tasks. Our extensive experimentation with varied quantities of labeled data in the training sets justifies the effectiveness of our model for the segmentation and uncertainty estimation of the left ventricle (LV), right ventricle (RV), and myocardium (Myo) at end-diastole (ED) and end-systole (ES) phases from cine MRI images available through the MICCAI 2017 ACDC Challenge Dataset. Our study serves as a proof-of-concept of how uncertainty measure correlates with the erroneous segmentation generated by different deep learning models, further showcasing the potential of our model to flag low-quality segmentation from a given model in our future study.

Pairwise versus Transitive Inverse Consistent Longitudinal Rigid Registration of Magnetic Resonance Images of Athletes With Repetitive Non-Concussive Head Injuries: Effects on Regional Distributions of Diffusion Measures

Accurate alignment of longitudinal diffusion weighted imaging (DWI) scans of a subject is necessary to investigate longitudinal changes in DWI-derived diffusion measures such as fractional anisotropy (FA), mean diffusivity (MD), and quantitative anisotropy (QA). Currently, studies investigating these changes in the context of repetitive non-concussive head injuries (RHIs) perform pairwise rigid registration of all scans of a subject to the first scan or any other reference scan or template. Prajapati et.al 1 show that this strategy of performing pairwise rigid registration lead to a discrepancy in the rigid transformations. To eliminate this discrepancy, they propose performing transitive inverse consistent rigid registration of the longitudinal scans, and they analyze the impact of this approach on the mean values of the local/regional estimates of these diffusion measures. In this work, we further analyze the impact of transitive inverse consistent rigid registration on the distributions (CDFs) of the local/regional estimates of diffusion measures. We identify the regions (among the 48 anatomically defined regions by the JHU DTI-based white matter atlas2,3) that show significant differences in the CDFs obtained using pairwise inverse consistent and transitive inverse consistent rigid registration by performing the two sided Kolmogorov-Smirnov(KS) hypothesis test. We find that for MD and QA, there are certain subjects that have five or more regions with significant differences in the CDFs. Further, these are the same subjects for which Prajapati et.al 1 found regions with 2%-4% differences in the mean values of these diffusion measures. Thus, our results further strengthen the recommendation made by Prajapati et.al 1 to employ transitive inverse consistent rigid registration when investigating local/regional longitudinal changes in diffusion measures.

A Feature-based Affine Registration Method for Capturing Background Lung Tissue Deformation for Ground Glass Nodule Tracking

Lung nodule tracking assessment relies on cross-sectional measurements of the largest lesion profile depicted in initial and follow-up computed tomography (CT) images. However, apparent changes in nodule size assessed via simple image-based measurements may also be compromised by the effect of the background lung tissue deformation on the GGN between the initial and follow-up images, leading to erroneous conclusions about nodule changes due to disease. To compensate for the lung deformation and enable consistent nodule tracking, here we propose a feature-based affine registration method and study its performance vis-a-vis several other registration methods. We implement and test each registration method using both a lung- and a lesion-centered region of interest on ten patient CT datasets featuring twelve nodules, including both benign and malignant GGO lesions containing pure GGNs, part-solid, or solid nodules. We evaluate each registration method according to the target registration error (TRE) computed across 30 - 50 homologous fiducial landmarks surrounding the lesions and selected by expert radiologists in both the initial and follow-up patient CT images. Our results show that the proposed feature-based affine lesion-centered registration yielded a 1.1 ± 1.2 mm TRE, while a Symmetric Normalization deformable registration yielded a 1.2 ± 1.2 mm TRE, and a least-square fit registration of the 30-50 validation fiducial landmark set yielded a 1.5 ± 1.2 mm TRE. Although the deformable registration yielded a slightly higher registration accuracy than the feature-based affine registration, it is significantly more computationally efficient, eliminates the need for ambiguous segmentation of GGNs featuring ill-defined borders, and reduces the susceptibility of artificial deformations introduced by the deformable registration, which may lead to increased similarity between the registered initial and follow-up images, over-compensating for the background lung tissue deformation, and, in turn, compromising the true disease-induced nodule change assessment. We also assessed the registration qualitatively, by visual inspection of the subtraction images, and conducted a pilot pre-clinical study that showed the proposed feature-based lesion-centered affine registration effectively compensates for the background lung tissue deformation between the initial and follow-up images and also serves as a reliable baseline registration method prior to assessing lung nodule changes due to disease.

Transitive Inverse Consistent Rigid Longitudinal Registration of Diffusion Weighted Magnetic Resonance Imaging: A Case Study in Athletes With Repetitive Non-Concussive Head Injuries

Significant longitudinal changes in metrics derived from diffusion weighted magnetic resonance (MR) images of the brain have been observed in athletes subject to repetitive non-concussive head injuries (RHIs). Accurate alignment of longitudinal scans of a subject is an important step in detecting and quantifying these changes. Currently, tools such as DSI Studio [1], FreeSurfer [2], and FSL [3] perform pairwise rigid registration of all scans in a longitudinal sequence to the first time-point scan (or to another reference scan or template). While the rigid transformations obtained using this strategy can be computed in a manner that enforces inverse consistency, for the case of three or more scans, the transformations are not transitive. This can lead to discrepancy in the rigid transformations that can be measured in physical units. Using a diffusion MRI dataset collected and analyzed as part of a larger study in [4], [5], [6], we illustrate this discrepancy, and we show how it can lead to uncertainty in local/regional estimates of diffusion metrics including fractional anistropy (FA), mean diffusivity (MD), and quantitatve anisotropy (QA). Additionally, we propose a method to perform transitive longitudinal rigid registration of a sequence of scans in a manner that guarantees that the discrepancy in the transformations will be eliminated.Clinical relevance- This paper establishes that standard processing pipelines for performing longitudinal analysis of diffusion MR images of the brain exhibit registration discrepancies that can be eliminated.

High-Order Cardiomyopathy Human Heart Model and Mesh Generation

Faithful, accurate, and successful cardiac biomechanics and electrophysiological simulations require patient-specific geometric models of the heart. Since the cardiac geometry consists of highly-curved boundaries, the use of high-order meshes with curved elements would ensure that the various curves and features present in the cardiac geometry are well-captured and preserved in the corresponding mesh. Most other existing mesh generation techniques require computer-aided design files to represent the geometric boundary, which are often not available for biomedical applications. Unlike such methods, our technique takes a high-order surface mesh, generated from patient medical images, as input and generates a high-order volume mesh directly from the curved surface mesh. In this paper, we use our direct high-order curvilinear tetrahedral mesh generation method [1] to generate several second-order cardiac meshes. Our meshes include the left ventricle myocardia of a healthy heart and hearts with dilated and hypertrophic cardiomyopathy. We show that our high-order cardiac meshes do not contain inverted elements and are of sufficiently high quality for use in cardiac finite element simulations.

A Deep Learning Framework for Image Super-Resolution for Late Gadolinium Enhanced Cardiac MRI

Cardiac magnetic resonance imaging (MRI) provides 3D images with high-resolution in-plane information, however, they are known to have low through-plane resolution due to the trade-off between resolution, image acquisition time and signal-to-noise ratio. This results in anisotropic 3D images which could lead to difficulty in diagnosis, especially in late gadolinium enhanced (LGE) cardiac MRI, which is the reference imaging modality for locating the extent of myocardial fibrosis in various cardiovascular diseases like myocardial infarction and atrial fibrillation. To address this issue, we propose a self-supervised deep learning-based approach to enhance the through-plane resolution of the LGE MRI images. We train a convolutional neural network (CNN) model on randomly extracted patches of short-axis LGE MRI images and this trained CNN model is used to leverage the information learnt from the high-resolution in-plane data to improve the through-plane resolution. We conducted experiments on LGE MRI dataset made available through the 2018 atrial segmentation challenge. Our proposed method achieved a mean peak signal-to-noise-ratio (PSNR) of 36.99 and 35.92 and a mean structural similarity index measure (SSIM) of 0.9 and 0.84 on training the CNN model using low-resolution images downsampled by a scale factor of 2 and 4, respectively.

A Multi-Task Cross-Task Learning Architecture for Ad Hoc Uncertainty Estimation in 3D Cardiac MRI Image Segmentation

Medical image segmentation has significantly benefitted thanks to deep learning architectures. Furthermore, semi-supervised learning (SSL) has recently been a growing trend for improving a model's overall performance by leveraging abundant unlabeled data. Moreover, learning multiple tasks within the same model further improves model generalizability. To generate smooth and accurate segmentation masks from 3D cardiac MR images, we present a Multi-task Cross-task learning consistency approach to enforce the correlation between the pixel-level (segmentation) and the geometric-level (distance map) tasks. Our extensive experimentation with varied quantities of labeled data in the training sets justifies the effectiveness of our model for the segmentation of the left atrial cavity from Gadolinium-enhanced magnetic resonance (GE-MR) images. With the incorporation of uncertainty estimates to detect failures in the segmentation masks generated by CNNs, our study further showcases the potential of our model to flag low-quality segmentation from a given model.

Dense Depth Estimation from Stereo Endoscopy Videos Using Unsupervised Optical Flow Methods

In the context of Minimally Invasive Surgery, estimating depth from stereo endoscopy plays a crucial role in three-dimensional (3D) reconstruction, surgical navigation, and augmentation reality (AR) visualization. However, the challenges associated with this task are three-fold: 1) feature-less surface representations, often polluted by artifacts, pose difficulty in identifying correspondence; 2) ground truth depth is difficult to estimate; and 3) an endoscopy image acquisition accompanied by accurately calibrated camera parameters is rare, as the camera is often adjusted during an intervention. To address these difficulties, we propose an unsupervised depth estimation framework (END-flow) based on an unsupervised optical flow network trained on un-rectified binocular videos without calibrated camera parameters. The proposed END-flow architecture is compared with traditional stereo matching, self-supervised depth estimation, unsupervised optical flow, and supervised methods implemented on the Stereo Correspondence and Reconstruction of Endoscopic Data (SCARED) Challenge dataset. Experimental results show that our method outperforms several state-of-the-art techniques and achieves a close performance to that of supervised methods.

CNN-Based Cardiac Motion Extraction to Generate Deformable Geometric Left Ventricle Myocardial Models from Cine MRI

Patient-specific left ventricle (LV) myocardial models have the potential to be used in a variety of clinical scenarios for improved diagnosis and treatment plans. Cine cardiac magnetic resonance (MR) imaging provides high resolution images to reconstruct patient-specific geometric models of the LV myocardium. With the advent of deep learning, accurate segmentation of cardiac chambers from cine cardiac MR images and unsupervised learning for image registration for cardiac motion estimation on a large number of image datasets is attainable. Here, we propose a deep leaning-based framework for the development of patient-specific geometric models of LV myocardium from cine cardiac MR images, using the Automated Cardiac Diagnosis Challenge (ACDC) dataset. We use the deformation field estimated from the VoxelMorph-based convolutional neural network (CNN) to propagate the isosurface mesh and volume mesh of the end-diastole (ED) frame to the subsequent frames of the cardiac cycle. We assess the CNN-based propagated models against segmented models at each cardiac phase, as well as models propagated using another traditional nonrigid image registration technique. Additionally, we generate dynamic LV myocardial volume meshes at all phases of the cardiac cycle using the log barrier-based mesh warping (LBWARP) method and compare them with the CNN-propagated volume meshes.

Segmentation and Removal of Surgical Instruments for Background Scene Visualization from Endoscopic / Laparoscopic Video

Surgical tool segmentation is becoming imperative to provide detailed information during intra-operative execution. These tools can obscure surgeons' dexterity control due to narrow working space and visual field-of-view, which increases the risk of complications resulting from tissue injuries (e.g. tissue scars and tears). This paper demonstrates a novel application of segmenting and removing surgical instruments from laparoscopic/endoscopic video using digital inpainting algorithms. To segment the surgical instruments, we use a modified U-Net architecture (U-NetPlus) composed of a pre-trained VGG11 or VGG16 encoder and redesigned decoder. The decoder is modified by replacing the transposed convolution operation with an up-sampling operation based on nearest-neighbor (NN) interpolation. This modification removes the artifacts generated by the transposed convolution, and, furthermore, these new interpolation weights require no learning for the upsampling operation. The tool removal algorithms use the tool segmentation mask and either the instrument-free reference frames or previous instrument-containing frames to fill-in (i.e., inpaint) the instrument segmentation mask with the background tissue underneath. We have demonstrated the performance of the proposed surgical tool segmentation/removal algorithms on a robotic instrument dataset from the MICCAI 2015 EndoVis Challenge. We also showed successful performance of the tool removal algorithm from synthetically generated surgical instruments-containing videos obtained by embedding a moving surgical tool into surgical tool-free videos. Our application successfully segments and removes the surgical tool to unveil the background tissue view otherwise obstructed by the tool, producing visually comparable results to the ground truth.

Joint Deep Learning Framework for Image Registration and Segmentation of Late Gadolinium Enhanced MRI and Cine Cardiac MRI

Late gadolinium enhanced (LGE) cardiac magnetic resonance (CMR) imaging, the current benchmark for assessment of myocardium viability, enables the identification and quantification of the compromised myocardial tissue regions, as they appear hyper-enhanced compared to the surrounding, healthy myocardium. However, in LGE CMR images, the reduced contrast between the left ventricle (LV) myocardium and LV blood-pool hampers the accurate delineation of the LV myocardium. On the other hand, the balanced-Steady State Free Precession (bSSFP) cine CMR imaging provides high resolution images ideal for accurate segmentation of the cardiac chambers. In the interest of generating patient-specific hybrid 3D and 4D anatomical models of the heart, to identify and quantify the compromised myocardial tissue regions for revascularization therapy planning, in our previous work, we presented a spatial transformer network (STN) based convolutional neural network (CNN) architecture for registration of LGE and bSSFP cine CMR image datasets made available through the 2019 Multi-Sequence Cardiac Magnetic Resonance Segmentation Challenge (MS-CMRSeg). We performed a supervised registration by leveraging the region of interest (RoI) information using the manual annotations of the LV blood-pool, LV myocardium and right ventricle (RV) blood-pool provided for both the LGE and the bSSFP cine CMR images. In order to reduce the reliance on the number of manual annotations for training such network, we propose a joint deep learning framework consisting of three branches: a STN based RoI guided CNN for registration of LGE and bSSFP cine CMR images, an U-Net model for segmentation of bSSFP cine CMR images, and an U-Net model for segmentation of LGE CMR images. This results in learning of a joint multi-scale feature encoder by optimizing all three branches of the network architecture simultaneously. Our experiments show that the registration results obtained by training 25 of the available 45 image datasets in a joint deep learning framework is comparable to the registration results obtained by stand-alone STN based CNN model by training 35 of the available 45 image datasets and also shows significant improvement in registration performance when compared to the results achieved by the stand-alone STN based CNN model by training 25 of the available 45 image datasets.

L-CO-Net: Learned Condensation-Optimization Network for Segmentation and Clinical Parameter Estimation from Cardiac Cine MRI

In this work, we implement a fully convolutional segmenter featuring both a learned group structure and a regularized weight-pruner to reduce the high computational cost in volumetric image segmentation. We validated our framework on the ACDC dataset featuring one healthy and four pathology patient groups imaged throughout the cardiac cycle. Our technique achieved Dice scores of 96.8% (LV blood-pool), 93.3% (RV blood-pool), and 90.0% (LV Myocardium) with five-fold cross-validation and yielded similar clinical parameters as those estimated from the ground-truth segmentation data. Based on these results, this technique has the potential to become an efficient and competitive cardiac image segmentation tool that may be used for cardiac computer-aided diagnosis, planning, and guidance applications.

A Convolutional Neural Network-based Deformable Image Registration Method for Cardiac Motion Estimation from Cine Cardiac MR Images

In this work, we describe an unsupervised deep learning framework featuring a Laplacian-based operator as smoothing loss for deformable registration of 3D cine cardiac magnetic resonance (CMR) images. Before registration, the input 3D images are corrected for slice misalignment by segmenting the left ventricle (LV) blood-pool, LV myocardium and right ventricle (RV) blood-pool using a U-Net model and aligning the 2D slices along the center of the LV blood-pool. We conducted experiments using the Automated Cardiac Diagnosis Challenge (ACDC) dataset. We used the registration deformation field to warp the manually segmented LV blood-pool, LV myocardium and RV blood-pool labels from end-diastole (ED) frame to the other frames in the cardiac cycle. We achieved a mean Dice score of 94.84%, 85.22% and 84.36%, and Hausdorff distance (HD) of 2.74 mm, 5.88 mm and 9.04 mm, for the LV blood-pool, LV myocardium and RV blood-pool, respectively. We also introduce a pipeline to estimate patient tractography using the proposed CNN-based cardiac motion estimation.

Reconstructing Cardiac Wave Dynamics From Myocardial Motion Data

Various models exist to predict the active stresses and membrane potentials within cardiac muscle tissue. However, there exist no methods to reliably measure active stresses, nor do there exist ways to measure transmural membrane potentials that are suitable for in vivo usage. Prior work has devised a linear model to map from the active stresses within the tissue to displacements [1]. In situations where measurements of tissue displacements are entirely precise, we are able to naively solve for the active stresses from the measurements with ease. However, real measurement processes always carry some associated random error and, in the presence of this error, our naive solution to this inverse problem fails. In this work we propose the use of the Ensemble Transform Kalman Filter to more reliably solve this inverse problem. This technique is faster than other related Kalman Filter techniques while still generating high quality estimates which improve on our naive solution. We demonstrate, using in silico simulations, that the Ensemble Transform Kalman Filter produces errors whose standard deviation is an order of magnitude smaller than the least-squares solution.

A Supervised Image Registration Approach for Late Gadolinium Enhanced MRI and Cine Cardiac MRI Using Convolutional Neural Networks

Late gadolinium enhanced (LGE) cardiac magnetic resonance (CMR) imaging is the current gold standard for assessing myocardium viability for patients diagnosed with myocardial infarction, myocarditis or cardiomyopathy. This imaging method enables the identification and quantification of myocardial tissue regions that appear hyper-enhanced. However, the delineation of the myocardium is hampered by the reduced contrast between the myocardium and the left ventricle (LV) blood-pool due to the gadolinium-based contrast agent. The balanced-Steady State Free Precession (bSSFP) cine CMR imaging provides high resolution images with superior contrast between the myocardium and the LV blood-pool. Hence, the registration of the LGE CMR images and the bSSFP cine CMR images is a vital step for accurate localization and quantification of the compromised myocardial tissue. Here, we propose a Spatial Transformer Network (STN) inspired convolutional neural network (CNN) architecture to perform supervised registration of bSSFP cine CMR and LGE CMR images. We evaluate our proposed method on the 2019 Multi-Sequence Cardiac Magnetic Resonance Segmentation Challenge (MS-CMRSeg) dataset and use several evaluation metrics, including the center-to-center LV and right ventricle (RV) blood-pool distance, and the contour-to-contour blood-pool and myocardium distance between the LGE and bSSFP CMR images. Specifically, we showed that our registration method reduced the bSSFP to LGE LV blood-pool center distance from 3.28mm before registration to 2.27mm post registration and RV blood-pool center distance from 4.35mm before registration to 2.52mm post registration. We also show that the average surface distance (ASD) between bSSFP and LGE is reduced from 2.53mm to 2.09mm, 1.78mm to 1.40mm and 2.42mm to 1.73mm for LV blood-pool, LV myocardium and RV blood-pool, respectively.

CondenseUNet: A memory-efficient condensely-connected architecture for bi-ventricular blood pool and myocardium segmentation

With the advent of Cardiac Cine Magnetic Resonance (CMR) Imaging, there has been a paradigm shift in medical technology, thanks to its capability of imaging different structures within the heart without ionizing radiation. However, it is very challenging to conduct pre-operative planning of minimally invasive cardiac procedures without accurate segmentation and identification of the left ventricle (LV), right ventricle (RV) blood-pool, and LV-myocardium. Manual segmentation of those structures, nevertheless, is time-consuming and often prone to error and biased outcomes. Hence, automatic and computationally efficient segmentation techniques are paramount. In this work, we propose a novel memory-efficient Convolutional Neural Network (CNN) architecture as a modification of both CondenseNet, as well as DenseNet for ventricular blood-pool segmentation by introducing a bottleneck block and an upsampling path. Our experiments show that the proposed architecture runs on the Automated Cardiac Diagnosis Challenge (ACDC) dataset using half (50%) the memory requirement of DenseNet and one-twelfth (~ 8%) of the memory requirements of U-Net, while still maintaining excellent accuracy of cardiac segmentation. We validated the framework on the ACDC dataset featuring one healthy and four pathology groups whose heart images were acquired throughout the cardiac cycle and achieved the mean dice scores of 96.78% (LV blood-pool), 93.46% (RV blood-pool) and 90.1% (LV-Myocardium). These results are promising and promote the proposed methods as a competitive tool for cardiac image segmentation and clinical parameter estimation that has the potential to provide fast and accurate results, as needed for pre-procedural planning and / or pre-operative applications.

Combining Statistical Shape Model and Principal Component Analysis to Estimate Left Ventricular Volume and Ejection Fraction

Left ventricular ejection fraction (LVEF) assessment is instrumental for cardiac health diagnosis, patient management, and patient eligibility for participation in clinical studies. Due to its non-invasiveness and low operational cost, ultrasound (US) imaging is the most commonly used imaging modality to image the heart and assess LVEF. Even though 3D US imaging technology is becoming more available, cardiologists dominantly use 2D US imaging to visualize the LV blood pool and interpret its area changes between end-systole and end-diastole. Our previous work showed that LVEF estimates based on area changes are significantly lower than the true volume-based estimates by as much as 13%,¹ which could lead to unnecessary and costly therapeutic decisions. Acquiring volumetric information about the LV blood pool necessitates either time-consuming 3D reconstruction or 3D US image acquisition. Here, we propose a method that leverages on a statistical shape model (SSM) constructed from 13 landmarks depicting the LV endocardial border to estimate a new patient's LV volume and LVEF. Two methods to estimate the 3D LV geometry with and without size normalization were employed. The SSM was built using the 13 landmarks from 50 training patient image datasets. Subsequently, the Mahalanobis distance (with size normalization) or the vector distance (without size normalization) between an incoming patient's LV landmarks and each shape in the SSM were used to determine the weights each training patient contributed to describing the new, incoming patient's LV geometry and associated blood pool volume. We tested the proposed method to estimate the LV volumes and LVEF for 16 new test patients. The estimated LVEFs based on Mahalanobis distance and vector distance were within 2.9% and 1.1%, respectively, of the ground truth LVEFs calculated from the 3D reconstructed LV volumes. Furthermore, the viability of using fewer principal components (PCs) to estimate the LV volume was explored by reducing the number of PCs retained when projecting landmarks onto PCA space. LVEF estimated based on 3 PCs, 5 PCs, and 10 PCs are within 6.6%, 5.4%, and 3.3%, respectively, of LVEF estimates using the full set of 39 PCs.

A machine leaning approach for abdominal aortic aneurysm severity assessment using geometric, biomechanical, and patient-specific historical clinical features

Recent studies monitoring severity of abdominal aortic aneurysm (AAA) suggested that reliance on only the maximum transverse diameter ( D _max ) may be insufficient to predict AAA rupture risk. Moreover, geometric indices, biomechanical parameters, material properties, and patient-specific historical data affect AAA morphology, indicating the need for an integrative approach that incorporates all factors for more accurate estimation of AAA severity. We implemented a machine learning algorithm using 45 features extracted from 66 patients. The model was generated using the J48 decision tree algorithm with the aim of maximizing model accuracy. Three different feature sets were used to assess the prediction rate: i) using D _max as a single-feature set, ii) using a set of all features, and, lastly iii) using a feature set selected via the BestFirst feature selection algorithm. Our results indicate that BestFirst feature selection yielded the highest prediction accuracy. These results indicate that a combination of several specific parameters that comprehensively capture AAA behavior may enable a suitable assessment of AAA severity, suggesting the potential benefit of machine learning for this application.

Automated Segmentation of Cardiac Chambers from Cine Cardiac MRI Using an Adversarial Network Architecture

Cine cardiac magnetic resonance imaging (CMRI), the current gold standard for cardiac function analysis, provides images with high spatio-temporal resolution. Computing clinical cardiac parameters like ventricular blood-pool volumes, ejection fraction and myocardial mass from these high resolution images is an important step in cardiac disease diagnosis, therapy planning and monitoring cardiac health. An accurate segmentation of left ventricle blood-pool, myocardium and right ventricle blood-pool is crucial for computing these clinical cardiac parameters. U-Net inspired models are the current state-of-the-art for medical image segmentation. SegAN, a novel adversarial network architecture with multi-scale loss function, has shown superior segmentation performance over U-Net models with single-scale loss function. In this paper, we compare the performance of stand-alone U-Net models and U-Net models in SegAN framework for segmentation of left ventricle blood-pool, myocardium and right ventricle blood-pool from the 2017 ACDC segmentation challenge dataset. The mean Dice scores achieved by training U-Net models was on the order of 89.03%, 89.32% and 88.71% for left ventricle blood-pool, myocardium and right ventricle blood-pool, respectively. The mean Dice scores achieved by training the U-Net models in SegAN framework are 91.31%, 88.68% and 90.93% for left ventricle blood-pool, myocardium and right ventricle blood-pool, respectively.

A distance map regularized CNN for cardiac cine MR image segmentation

PURPOSE

Cardiac image segmentation is a critical process for generating personalized models of the heart and for quantifying cardiac performance parameters. Fully automatic segmentation of the left ventricle (LV), the right ventricle (RV), and the myocardium from cardiac cine MR images is challenging due to variability of the normal and abnormal anatomy, as well as the imaging protocols. This study proposes a multi-task learning (MTL)-based regularization of a convolutional neural network (CNN) to obtain accurate segmenation of the cardiac structures from cine MR images.

METHODS

We train a CNN network to perform the main task of semantic segmentation, along with the simultaneous, auxiliary task of pixel-wise distance map regression. The network also predicts uncertainties associated with both tasks, such that their losses are weighted by the inverse of their corresponding uncertainties. As a result, during training, the task featuring a higher uncertainty is weighted less and vice versa. The proposed distance map regularizer is a decoder network added to the bottleneck layer of an existing CNN architecture, facilitating the network to learn robust global features. The regularizer block is removed after training, so that the original number of network parameters does not change. The trained network outputs per-pixel segmentation when a new patient cine MR image is provided as an input.

RESULTS

We show that the proposed regularization method improves both binary and multi-class segmentation performance over the corresponding state-of-the-art CNN architectures. The evaluation was conducted on two publicly available cardiac cine MRI datasets, yielding average Dice coefficients of 0.84 ± 0.03 and 0.91 ± 0.04. We also demonstrate improved generalization performance of the distance map regularized network on cross-dataset segmentation, showing as much as 42% improvement in myocardium Dice coefficient from 0.56 ± 0.28 to 0.80 ± 0.14.

CONCLUSIONS

We have presented a method for accurate segmentation of cardiac structures from cine MR images. Our experiments verify that the proposed method exceeds the segmentation performance of three existing state-of-the-art methods. Furthermore, several cardiac indices that often serve as diagnostic biomarkers, specifically blood pool volume, myocardial mass, and ejection fraction, computed using our method are better correlated with the indices computed from the reference, ground truth segmentation. Hence, the proposed method has the potential to become a non-invasive screening and diagnostic tool for the clinical assessment of various cardiac conditions, as well as a reliable aid for generating patient specific models of the cardiac anatomy for therapy planning, simulation, and guidance.

A Statistical Shape Model Approach for Computing Left Ventricle Volume and Ejection Fraction Using Multi-plane Ultrasound Images

Assessing the left ventricular ejection fraction (LVEF) accurately requires 3D volumetric data of the LV. Cardiologists either have no access to 3D ultrasound (US) systems or prefer to visually estimate LVEF based on 2D US images. To facilitate the consistent estimation of the end-diastolic and end-systolic blood pool volume and LVEF based on 3D data without extensive complicated manual input, we propose a statistical shape model (SSM) based on 13 key anchor points-the LV apex (1), mitral valve hinges (6), and the midpoints of the endocardial contours (6)-identified from the LV endocardial contour of the tri-plane 2D US images. We use principal component analysis (PCA) to identify the principle modes of variation needed to represent the LV shapes, which enables us to estimate an incoming LV as a linear combination of the principle components (PC). For a new, incoming patient image, its 13 anchor points are projected onto the PC space; its shape is compared to each LV shape in the SSM based on Mahalanobis distance, which is normalized with respect to the LV size, as well as direct vector distance (i.e., PCA distance), without any size normalization. These distances are used to determine the weight each training shape in the SSM contributes to the description of the new patient LV shape. Finally, the new patient's LV systolic and diastolic volumes are estimated as the weighted average of the training volumes in the SSM. To assess our proposed method, we compared the SSM-based estimates of diastolic, systolic, stroke volumes, and LVEF with those computed directly from 16 tri-plane 2D US imaging datasets using the GE Echo-Pac PC clinical platform. The estimated LVEF based on Mahalanobis distance and PCA distance were within 6.8% and 1.7% of the reference LVEF computed using the GE Echo-Pac PC clinical platform.

Toward an Affine Feature-Based Registration Method for Ground Glass Lung Nodule Tracking

Lung nodule progression assessment from medical imaging is a critical biomarker for assessing the course of the disease or the patient's response to therapy. CT images are routinely used to identify the location and size and rack the progression of lung nodules. However, nodule segmentation is challenging and prone to error, due to the irregular nodule boundaries, therefore introducing error in the lung nodule quantification process. Here, we describe the development and evaluation of a feature-based affine image registration framework that enables us to register two time point thoracic CT images as a means to account for the back-ground lung tissue deformation, then use digital subtraction images to assess tumor progression/regression. We have demonstrated this method on twelve de-identified patient datasets and showed that the proposed method yielded a better than 1.5mm registration accuracy vis-à-vis the widely accepted non-rigid image registration techniques. To demonstrate the potential clinical value of our described technique, we conducted a study in which our collaborating clinician was asked to provide an assessment of nodule progression/regression using the digital subtraction images post-registration. This assessment was consistent, yet provided more confidence, than the traditional lung nodule tracking based on visual analysis of the CT images.

Toward Quantification and Visualization of Active Stress Waves for Myocardial Biomechanical Function Assessment

Estimating and visualizing myocardial active stress wave patterns is crucial to understanding the mechanical activity of the heart and provides a potential non-invasive method to assess myocardial function. These patterns can be reconstructed by analyzing 2D and/or 3D tissue displacement data acquired using medical imaging. Here we describe an application that utilizes a 3D finite element formulation to reconstruct active stress from displacement data. As a proof of concept, a simple cubic mesh was used to represent a myocardial tissue "sample" consisting of a 10 × 10 × 10 lattice of nodes featuring different fiber directions that rotate with depth, mimicking cardiac transverse isotropy. In the forward model, tissue deformation was generated using a test wave with active stresses that mimic the myocardial contractile forces. The generated deformation field was used as input to an inverse model designed to reconstruct the original active stress distribution. We numerically simulated malfunctioning tissue regions (experiencing limited contractility and hence active stress) within the healthy tissue. We also assessed model sensitivity by adding noise to the deformation field generated using the forward model. The difference image between the original and reconstructed active stress distribution suggests that the model accurately estimates active stress from tissue deformation data with a high signal-to-noise ratio.

(PEAK) WALL STRESS AS AN INDICATOR OF ABDOMINAL AORTIC ANEURYSM SEVERITY

Abdominal aortic aneurysms, which consist of dilatations of the infra-renal aorta by at least 1.5 times of its normal diameter, are becoming a leading cause of death worldwide. Rupture often occurs unexpectedly, before a repair procedure is conducted. The AAA maximum diameter has been used as a clinical criterion to monitor AAA severity. However, assessment of AAA rupture risk requires knowledge of wall stress and wall strength at the potential rupture location. We conducted a study on 37 patient specific CT datasets to investigate the benefits of using peak wall stress instead of D_max for AAA rupture severity. Correlation between PWS and 24 geometric indices and biomechanical factors was studied where eleven of them showed a statistically significant correlation with PWS. A Finite Element Analysis Rupture Index was used to conclude that the use of D _max as a single predictor of AAA behavior and severity may be insufficient based on our patient population with a D_max smaller than the 5.5 cm, clinically recommended repair threshold.

U-NetPlus: A Modified Encoder-Decoder U-Net Architecture for Semantic and Instance Segmentation of Surgical Instruments from Laparoscopic Images

With the advent of robot-assisted surgery, there has been a paradigm shift in medical technology for minimally invasive surgery. However, it is very challenging to track the position of the surgical instruments in a surgical scene, and accurate detection & identification of surgical tools is paramount. Deep learning-based semantic segmentation in frames of surgery videos has the potential to facilitate this task. In this work, we modify the U-Net architecture by introducing a pre-trained encoder and re-design the decoder part, by replacing the transposed convolution operation with an upsampling operation based on nearest-neighbor (NN) interpolation. To further improve performance, we also employ a very fast and flexible data augmentation technique. We trained the framework on 8 × 225 frame sequences of robotic surgical videos available through the MICCAI 2017 EndoVis Challenge dataset and tested it on 8 × 75 frame and 2 × 300 frame videos. Using our U-NetPlus architecture, we report a 90.20% DICE for binary segmentation, 76.26% DICE for instrument part segmentation, and 46.07% for instrument type (i.e., all instruments) segmentation, outperforming the results of previous techniques implemented and tested on these data.

Therapeutic Systems and Technologies: State-of-the-Art Applications, Opportunities, and Challenges

In this review, we present current state-of-the-art developments and challenges in the areas of thermal therapy, ultrasound tomography, image-guided therapies, ocular drug delivery, and robotic devices in neurorehabilitation. Additionally, intellectual property and regulatory aspects pertaining to therapeutic systems and technologies are addressed.

Left Ventricular Ejection Fraction Assessment: Unraveling the Bias between Area- and Volume-based Estimates

Calculating left ventricular ejection fraction (LVEF) accurately is crucial for the clinical diagnosis of cardiac disease, patient management, or other therapeutic treatment decisions. The measure of a patient's LVEF often affects their candidacy for cardiovascular intervention. Ultrasound (US) is one of the imaging modalities used to non-invasively assess LVEF, and it is the most common and least expensive. Despite the advances in 3D US transducer technology, only limited US machines are equipped with such transducer to enable true 3D US image acquisition. Thus, 2D US images remain to be widely used by cardiologists to image the heart and their interpretation is inherently based on two dimensional information immediately available in the US images. Past knowledge indicates that visual estimation of the LVEF based on the area changes of the left ventricle blood pool between systole and diastole (as depicted in 2D ultrasound images) may significantly underestimate the ejection fraction, rendering some patients as suitable candidates for potentially unnecessary interventions or implantation of assistive devices. True LVEF should be calculated based on changes in LV volumes, but equipment and time constraint limit the current technique to assess 3D LV geometry. The estimation of the systolic and diastolic blood pool volumes requires additional work beyond a simple visual assessment of the blood pool area changed in the 2D US images. Specifically, following the manual segmentation of the endocardial LV border, 3D volume would be assessed by reconstructing a LV volume from multiple tomographic views. In this work, we leverage on two idealized mathematical models of the left ventricle - a truncated prolate spheroid (TPS) and a paraboloid geometric model to characterize the LV shape according to the range of possible dimensions gathered from our patient-specific multi-plane US imaging data. The objective of this work is to reveal the necessity of calculating LVEFs based on volumes by showing that LVEF estimated using area changes underestimate the LVEF computed using volume changes. Additionally, we present a method to reconstruct the LV volume from 2D blood pool representations identified in the multi-plane 2D US images and use the reconstructed 3D volume throughout the cardiac cycle to estimate the LVEF. Our preliminary results show that the area-based LVEF significantly underestimates the true volume-based LVEF across both the theoretical simulations using idealized geometric models of the LV shape, as well as the patient-specific US imaging data. Specifically, both the TPS and paraboloid model showed an area-based LVEF of 41.3 ± 4.7% and a volume-based LVEF of 55.4 ± 5.7%, while the US image data showed an area-based LVEF of 34.7 ± 11.9% and a volume-based LVEF of 48.0 ± 14.0%. In summary, the area-based LVEF estimations using both the idealized TPS and paraboloid models was 14.1% lower than volume- based LVEF calculations using corresponding models. Furthermore, the area-based LVEF based on reconstructed LV volumes are 13.3% lower than volume-based estimates. Evidently, there is a need to further investigate a method to enable practical volume-based LVEF calculations to avoid the need for clinicians to estimate LVEF based on visual, holistic assessment of the blood pool area changes that improperly infer volumetric blood pool changes.

Left Ventricle Segmentation and Quantification from Cardiac Cine MR Images via Multi-task Learning

Segmentation of the left ventricle and quantification of various cardiac contractile functions is crucial for the timely diagnosis and treatment of cardiovascular diseases. Traditionally, the two tasks have been tackled independently. Here we propose a convolutional neural network based multi-task learning approach to perform both tasks simultaneously, such that, the network learns better representation of the data with improved generalization performance. Probabilistic formulation of the problem enables learning the task uncertainties during the training, which are used to automatically compute the weights for the tasks. We performed a five fold cross-validation of the myocardium segmentation obtained from the proposed multi-task network on 97 patient 4-dimensional cardiac cine-MRI datasets available through the STA-COM LV segmentation challenge against the provided gold-standard myocardium segmentation, obtaining a Dice overlap of 0.849 ± 0.036 and mean surface distance of 0.274 ± 0.083 mm, while simultaneously estimating the myocardial area with mean absolute difference error of 205 ± 198 mm2.

A marker-free registration method for standing X-ray panorama reconstruction for hip-knee-ankle axis deformity assessment

Accurate measurement of knee alignment, quantified by the hip-knee-ankle (HKA) angle (varus-valgus), serves as an essential biomarker in the diagnosis of various orthopaedic conditions and selection of appropriate therapies. Such angular deformities are assessed from standing X-ray panoramas. However, the limited field-of-view of traditional X-ray imaging systems necessitates the acquisition of several sector images to capture an individual's standing posture, and their subsequent 'stitching' to reconstruct a panoramic image. Such panoramas are typically constructed manually by an X-ray imaging technician, often using various external markers attached to the individual's clothing and visible in two adjacent sector images. To eliminate human error, user-induced variability, improve consistency and reproducibility, and reduce the time associated with the traditional manual 'stitching' protocol, here we propose an automatic panorama construction method that only relies on anatomical features reliably detected in the images, eliminating the need for any external markers or manual input from the technician. The method first performs a rough segmentation of the femur and the tibia, then the sector images are registered by evaluating a distance metric between the corresponding bones along their medial edge. The identified translations are then used to generate the standing panorama image. The method was evaluated on 95 patient image datasets from a database of X-ray images acquired across 10 clinical sites as part of the screening process for a multi-site clinical trial. The panorama reconstruction parameters yielded by the proposed method were compared to those used for the manual panorama construction, which served as gold-standard. The horizontal translation differences were 0:43 ± 1:95 mm 0:26 ± 1:43 mm for the femur and tibia respectively, while the vertical translation differences were 3:76 ± 22:35 mm and 1:85 ± 6:79 mm for the femur and tibia, respectively. Our results showed no statistically significant differences between the HKA angles measured using the automated vs. the manually generated panoramas, and also led to similar decisions with regards to the patient inclusion/exclusion in the clinical trial. Thus, the proposed method was shown to provide comparable performance to manual panorama construction, with increased efficiency, consistency and robustness.

A Modified U-Net Convolutional Network Featuring a Nearest-neighbor Re-sampling-based Elastic-Transformation for Brain Tissue Characterization and Segmentation

The detection and segmentation of brain tumors from Magnetic Resonance Imaging (MRI) is a very challenging task, despite the availability of modern medical image processing tools. Neuro-radiologists still diagnose deadly brain cancers such as even glioblastoma using manual segmentation. This approach is not only tedious, but also highly variable, featuring limited accuracy and precision, and hence raising the need for more robust, automated techniques. Deep learning methods such as the U-Net deep convolutional neural networks have been widely used in biomedical image segmentation. Although this model was demonstrated to yield desirable results on the BRATS 2015 dataset by using a pixel-wise segmentation map of the input image as an auto-encoder, which assures best segmentation accuracy, the output only showed limited accuracy and robustness for a number of cases. The goal of this work was to improve the U-net model by replacing the de-convolution component with an up-sampled by the Nearest-neighbor algorithm and also employing an elastic transformation to augment the training dataset to render the model more robust, especially for the segmentation of low-grade tumors. The proposed Nearest-Neighbor Re-sampling Based Elastic-Transformed (NNRET) U-net Deep CNN framework has been trained on 285 glioma patients BRATS 2017 MR dataset available through the MICCAI 2017 grand challenge. The framework has been tested on 146 patients using Dice similarity coefficient (DSC) & Intersection over Union (IoU) performance metrics and outweighed the classic U-net model.

LEFT VENTRICULAR EJECTION FRACTION: COMPARISON BETWEEN TRUE VOLUME-BASED MEASUREMENTS AND AREA-BASED ESTIMATES

Left ventricular ejection fraction (LVEF) is a critical measure of cardiac health commonly acquired in clinical practice, which serves as the basis for cardiovascular therapeutic treatment. Ultrasound (US) imaging of the heart is the most common, least expensive, reliable and non-invasive modality to assess LVEF. Cardiologists, in practice, persistently use 2D US images to provide visual estimates of LVEF, which are based on 2D information embedded in the US images by examining the area changes in LV blood pool between diastole and systole. There has been some anecdotal evidence that visual estimation of the LVEF based on the area changes of the LV blood pool significantly underestimate true LVEF. True LVEF should be calculated based on changes in LV volumes between diastole and systole. In this project, we utilized both idealized models of the LV geometry - a truncated prolate spheroid (TPS) and a paraboloid model - to represent the LV anatomy. Cross-sectional areas and volumes of simulated LV shapes using both models were calculated to compare the LVEF. Further, a LV reconstruction algorithm was employed to build the LV blood pool volume in both systole and diastole from multi-plane 2D US imaging data. Our mathematical models yielded an area-based LVEF of 41 4.7% and a volume-based LVEF of 55 ±5.7%, while the 3D recon-struction model showed an area-based LVEF of 35 11.9% and a volume-based LVEF of 48.0 ± 14.0%. In summary, the area-based LVEF using all three models ±underestimate the volume-based LVEF using corresponding models by 13% to 14%. This preliminary study confirms both mathematically and empirically that area-based LVEF estimates indeed underestimate the true volume-based LVEF measurements and suggests that true volumetric measurements of the LV blood pool must be computed to correctly assess cardiac LVEF.

Lesion modeling, characterization, and visualization for image-guided cardiac ablation therapy monitoring

In spite of significant efforts to improve image-guided ablation therapy, a large number of patients undergoing ablation therapy to treat cardiac arrhythmic conditions require repeat procedures. The delivery of insufficient thermal dose is a significant contributor to incomplete tissue ablation, in turn leading to the arrhythmia recurrence. Ongoing research efforts aim to better characterize and visualize RF delivery to monitor the induced tissue damage during therapy. Here, we propose a method that entails modeling and visualization of the lesions in real-time. The described image-based ablation model relies on classical heat transfer principles to estimate tissue temperature in response to the ablation parameters, tissue properties, and duration. The ablation lesion quality, geometry, and overall progression are quantified on a voxel-by-voxel basis according to each voxel's cumulative temperature and time exposure. The model was evaluated both numerically under different parameter conditions, as well as experimentally, using ex vivo bovine tissue samples undergoing ex vivo clinically relevant ablation protocols. The studies demonstrated less than 5°C difference between the model-predicted and experimentally measured end-ablation temperatures. The model predicted lesion patterns were within 0.5 to 1 mm from the observed lesion patterns, suggesting sufficiently accurate modeling of the ablation lesions. Lastly, our proposed method enables therapy delivery feedback with no significant workflow latency. This study suggests that the proposed technique provides reasonably accurate and sufficiently fast visualizations of the delivered ablation lesions.

Toward modeling the effects of regional material properties on the wall stress distribution of abdominal aortic aneurysms

The overall geometry and different biomechanical parameters of an abdominal aortic aneurysm (AAA), contribute to its severity and risk of rupture, therefore they could be used to track its progression. Previous and ongoing research efforts have resorted to using uniform material properties to model the behavior of AAA. However, it has been recently illustrated that different regions of the AAA wall exhibit different behavior due to the effect of the biological activities in the metalloproteinase matrix that makes up the wall at the aneurysm site. In this work, we introduce a non-invasive patient-specific regional material property model to help us better understand and investigate the AAA wall stress distribution, peak wall stress (PWS) severity, and potential rupture risk. Our results indicate that the PWS and the overall wall stress distribution predicted using the proposed regional material property model, are higher than those predicted using the traditional homogeneous, hyper-elastic model (p <1.43E-07). Our results also show that to investigate AAA, the overall geometry, presence of intra-luminal thrombus (ILT), and loading condition in a patient specific manner may be critical for capturing the biomechanical complexity of AAAs.

Technical Note: On Cardiac Ablation Lesion Visualization for Image-guided Therapy Monitoring

The delivery of insufficient thermal dose is a significant contributor to incomplete tissue ablation and leads to arrhythmia recurrence and a large number of patients requiring repeat procedures. In concert with ongoing research efforts aimed at better characterizing the RF energy delivery, here we propose a method that entails modeling and visualization of the lesions in real time. The described image-based ablation model relies on classical heat transfer principles to estimate tissue temperature in response to the ablation parameters, tissue properties, and duration. The ablation lesion quality, geometry, and overall progression is quantified on a voxel-by-voxel basis according to each voxel's cumulative temperature and time exposure. The model was evaluated both numerically under different parameter conditions, as well as experimentally, using ex vivo bovine tissue samples. This study suggests that the proposed technique provides reasonably accurate and sufficiently fast visualizations of the delivered ablation lesions.

Cine Cardiac MRI Slice Misalignment Correction Towards Full 3D Left Ventricle Segmentation

Accurate segmentation of the left ventricle (LV) blood-pool and myocardium is required to compute cardiac function assessment parameters or generate personalized cardiac models for pre-operative planning of minimally invasive therapy. Cardiac Cine Magnetic Resonance Imaging (MRI) is the preferred modality for high resolution cardiac imaging thanks to its capability of imaging the heart throughout the cardiac cycle, while providing tissue contrast superior to other imaging modalities without ionizing radiation. However, there exists an inevitable misalignment between the slices in cine MRI due to the 2D + time acquisition, rendering 3D segmentation methods ineffective. A large part of published work on cardiac MR image segmentation focuses on 2D segmentation methods that yield good results in mid-slices, however with less accurate results for the apical and basal slices. Here, we propose an algorithm to correct for the slice misalignment using a Convolutional Neural Network (CNN)-based regression method, and then perform a 3D graph-cut based segmentation of the LV using atlas shape prior. Our algorithm is able to reduce the median slice misalignment error from 3.13 to 2.07 pixels, and obtain the blood-pool segmentation with an accuracy characterized by a 0.904 mean dice overlap and 0.56 mm mean surface distance with respect to the gold-standard blood-pool segmentation for 9 test cine MR datasets.