Evaluation of Six Registration Methods for the Human Abdomen on Clinically Acquired CT

Lightweight cross-resolution coarse-to-fine network for efficient deformable medical image registration

BACKGROUND

Accurate and efficient deformable medical image registration is crucial in medical image analysis. While recent deep learning-based registration methods have achieved state-of-the-art accuracy, they often suffer from extensive network parameters and slow inference times, leading to inefficiency. Efforts to reduce model size and input resolution can improve computational efficiency but frequently result in suboptimal accuracy.

PURPOSE

To address the trade-off between high accuracy and efficiency, we propose a Lightweight Cross-Resolution Coarse-to-Fine registration framework, termed LightCRCF.

METHODS

Our method is built on an ultra-lightweight U-Net architecture with only 0.1 million parameters, offering remarkable efficiency. To mitigate accuracy degradation resulting from fewer parameters while preserving the lightweight nature of the networks, LightCRCF introduces three key innovations as follows: (1) selecting an efficient cross-resolution coarse-to-fine (C2F) registration strategy and integrating it into the lightweight network to progressively decompose the deformation fields into multiresolution subfields to capture fine-grained deformations; (2) a Texture-aware Reparameterization (TaRep) module that integrates Sobel and Laplacian operators to extract rich textural information; (3) a Group-flow Reparameterization (GfRep) module that captures diverse deformation modes by decomposing the deformation field into multiple groups. Furthermore, we introduce a structural reparameterization technique that enhances training accuracy through multibranch structures of the TaRep and GfRep modules, while maintaining efficient inference by equivalently transforming these multibranch structures into single-branch standard convolutions.

RESULTS

We evaluate LightCRCF against various methods on the three public MRI datasets (LPBA, OASIS, and ACDC) and one CT dataset (abdomen CT). Following the previous data division methods, the LPBA dataset comprises 30 training image pairs and nine testing image pairs. For the OASIS dataset, the training, validation, and testing data consist of 1275, 110, and 660 image pairs, respectively. Similarly, for the ACDC dataset, the training, validation, and testing data include 180, 20, and 100 image pairs, respectively. For intersubject registration of the abdomen CT dataset, there are 380 training pairs, six validation pairs, and 42 testing pairs. Compared to state-of-the-art C2F methods, LightCRCF achieves comparable accuracy scores (DSC, HD95, and MSE), while demonstrating significantly superior performance across all efficiency metrics (Params, VRAM, FLOPs, and inference time). Relative to efficiency-first approaches, LightCRCF significantly outperforms these methods in accuracy metrics.

CONCLUSIONS

Our LightCRCF method offers a favorable trade-off between accuracy and efficiency, maintaining high accuracy while achieving superior efficiency, thereby highlighting its potential for clinical applications. The code will be available at https://github.com/PerceptionComputingLab/LightCRCF.

Automatic colon segmentation on T1-FS MR images

The volume and distribution of the colonic contents provides valuable insights into the effects of diet on gut microbiotica involving both clinical diagnosis and research. In terms of Magnetic Resonance Imaging modalities, T2-weighted images allow the segmentation of the colon lumen, while fecal and gas contents can be only distinguished on the T1-weighted Fat-Sat modality. However, the manual segmentation of T1-weighted Fat-Sat is challenging, and no automatic segmentation methods are known. This paper proposed a non-supervised algorithm providing an accurate T1-weighted Fat-Sat colon segmentation via the registration of an existing colon segmentation in T2-weighted modality. The algorithm consists of two phases. It starts with a registration process based on a classical deformable registration method, followed by a novel Iterative Colon Registration process that utilizes a mesh deformation approach. This approach is guided by a probabilistic model that provides the likelihood of the colon boundary, followed by a shape preservation process of the colon segmentation on T2-weighted images. The iterative process converges to achieve an optimal fit for colon segmentation in T1-weighted Fat-Sat images. The segmentation algorithm has been tested on multiple datasets (154 scans) and acquisition machines (3) as part of the proof of concept for the proposed methodology. The quantitative evaluation was based on two metrics: the percentage of ground truth labeled feces correctly identified by our proposal (93±5%), and the volume variation between the existing colon segmentation in the T2-weighted modality and the colon segmentation computed in T1-weighted Fat-Sat images. Quantitative and medical evaluations demonstrated a degree of accuracy, usability, and stability concerning the acquisition hardware, making the algorithm suitable for clinical application and research.

Influence of early through late fusion on pancreas segmentation from imperfectly registered multimodal magnetic resonance imaging

Purpose

Combining different types of medical imaging data, through multimodal fusion, promises better segmentation of anatomical structures, such as the pancreas. Strategic implementation of multimodal fusion could improve our ability to study diseases such as diabetes. However, where to perform fusion in deep learning models is still an open question. It is unclear if there is a single best location to fuse information when analyzing pairs of imperfectly aligned images or if the optimal fusion location depends on the specific model being used. Two main challenges when using multiple imaging modalities to study the pancreas are (1) the pancreas and surrounding abdominal anatomy have a deformable structure, making it difficult to consistently align the images and (2) breathing by the individual during image collection further complicates the alignment between multimodal images. Even after using state-of-the-art deformable image registration techniques, specifically designed to align abdominal images, multimodal images of the abdomen are often not perfectly aligned. We examine how the choice of different fusion points, ranging from early in the image processing pipeline to later stages, impacts the segmentation of the pancreas on imperfectly registered multimodal magnetic resonance (MR) images.

Approach

Our dataset consists of 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas segmentation labels drawn mainly based on the T2w images. Because the T2w images were acquired in an interleaved manner across two breath holds and the T1w images on one breath hold, there were three different breath holds impacting the alignment of each pair of images. We used deeds, a state-of-the-art deformable abdominal image registration method to align the image pairs. Then, we trained a collection of basic UNets with different fusion points, spanning from early to late layers in the model, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. To investigate whether performance differences on key fusion points are generalized to other architectures, we expanded our experiments to nnUNet.

Results

The single-modality T2w baseline using a basic UNet model had a median Dice score of 0.766, whereas the same baseline on the nnUNet model achieved 0.824. For each fusion approach, we analyzed the differences in performance with Dice residuals, by subtracting the baseline score from the fusion score for each datapoint. For the basic UNet, the best fusion approach was from early/mid fusion and occurred in the middle of the encoder with a median Dice residual of + 0.012 compared with the baseline. For the nnUNet, the best fusion approach was early fusion through naïve image concatenation before the model, with a median Dice residual of + 0.004 compared with the baseline. After Bonferroni correction, the distributions of the Dice scores for these best fusion approaches were found to be statistically significant ( p < 0.05 ) via the paired Wilcoxon signed-rank test against the baseline.

Conclusions

Fusion in specific blocks can improve performance, but the best blocks for fusion are model-specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs. The code associated with this project is available here https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation.

Multi-contrast computed tomography atlas of healthy pancreas with dense displacement sampling registration

Purpose

Diverse population demographics can lead to substantial variation in the human anatomy. Therefore, standard anatomical atlases are needed for interpreting organ-specific analyses. Among abdominal organs, the pancreas exhibits notable variability in volumetric morphology, shape, and appearance, complicating the generalization of population-wide features. Understanding the common features of a healthy pancreas is crucial for identifying biomarkers and diagnosing pancreatic diseases.

Approach

We propose a high-resolution CT atlas framework optimized for the healthy pancreas. We introduce a deep-learning-based preprocessing technique to extract abdominal ROIs and leverage a hierarchical registration pipeline to align pancreatic anatomy across populations. Briefly, DEEDS affine and non-rigid registration techniques are employed to transfer patient abdominal volumes to a fixed high-resolution atlas template. To generate and evaluate the pancreas atlas, multi-phase contrast CT scans of 443 subjects (aged 15 to 50 years, with no reported history of pancreatic disease) were processed.

Results

The two-stage DEEDS affine and non-rigid registration outperforms other state-of-the-art tools, achieving the highest scores for pancreas label transfer across all phases (non-contrast: 0.497, arterial: 0.505, portal venous: 0.494, delayed: 0.497). External evaluation with 100 portal venous scans and 13 labeled abdominal organs shows a mean Dice score of 0.504. The low variance between the pancreases of registered subjects and the obtained pancreas atlas further illustrates the generalizability of the proposed method.

Conclusion

We introduce a high-resolution pancreas atlas framework to generalize healthy biomarkers across populations with multi-contrast abdominal CT. The atlases and the associated pancreas organ labels are publicly available through the Human Biomolecular Atlas Program (HuBMAP).

ConvexAdam: Self-Configuring Dual-Optimization-Based 3D Multitask Medical Image Registration

Registration of medical image data requires methods that can align anatomical structures precisely while applying smooth and plausible transformations. Ideally, these methods should furthermore operate quickly and apply to a wide variety of tasks. Deep learning-based image registration methods usually entail an elaborate learning procedure with the need for extensive training data. However, they often struggle with versatility when aiming to apply the same approach across various anatomical regions and different imaging modalities. In this work, we present a method that extracts semantic or hand-crafted image features and uses a coupled convex optimisation followed by Adam-based instance optimisation for multitask medical image registration. We make use of pre-trained semantic feature extraction models for the individual datasets and combine them with our fast dual optimisation procedure for deformation field computation. Furthermore, we propose a very fast automatic hyperparameter selection procedure that explores many settings and ranks them on validation data to provide a self-configuring image registration framework. With our approach, we can align image data for various tasks with little learning. We conduct experiments on all available Learn2Reg challenge datasets and obtain results that are to be positioned in the upper ranks of the challenge leaderboards. http://github.com/multimodallearning/convexAdam.

A survey on deep learning in medical image registration: New technologies, uncertainty, evaluation metrics, and beyond

Deep learning technologies have dramatically reshaped the field of medical image registration over the past decade. The initial developments, such as regression-based and U-Net-based networks, established the foundation for deep learning in image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, network architectures, and uncertainty estimation. These advancements have not only enriched the field of image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration.

A Vessel Bifurcation Landmark Pair Dataset for Abdominal CT Deformable Image Registration (DIR) Validation

Purpose

Deformable image registration (DIR) is an enabling technology in many diagnostic and therapeutic tasks. Despite this, DIR algorithms have limited clinical use, largely due to a lack of benchmark datasets for quality assurance during development. DIRs of intra-patient abdominal CTs are among the most challenging registration scenarios due to significant organ deformations and inconsistent image content. To support future algorithm development, here we introduce our first-of-its-kind abdominal CT DIR benchmark dataset, comprising large numbers of highly accurate landmark pairs on matching blood vessel bifurcations.

Acquisition and Validation Methods

Abdominal CT image pairs of 30 patients were acquired from several publicly available repositories as well as the authors' institution with IRB approval. The two CTs of each pair were originally acquired for the same patient but on different days. An image processing workflow was developed and applied to each CT image pair: 1) Abdominal organs were segmented with a deep learning model, and image intensity within organ masks was overwritten. 2) Matching image patches were manually identified between two CTs of each image pair. 3) Vessel bifurcation landmarks were labeled on one image of each image patch pair. 4) Image patches were deformably registered, and landmarks were projected onto the second image 5) Landmark pair locations were refined manually or with an automated process. This workflow resulted in 1895 total landmark pairs, or 63 per case on average. Estimates of the landmark pair accuracy using digital phantoms were 0.7mm +/- 1.2 mm.

Data Format and Usage Notes

The data is published in Zenodo at https://doi.org/10.5281/zenodo.14362785. Instructions for use can be found at https://github.com/deshanyang/Abdominal-DIR-QA.

Potential Applications

This dataset is a first-of-its-kind for abdominal DIR validation. The number, accuracy, and distribution of landmark pairs will allow for robust validation of DIR algorithms with precision beyond what is currently available.

BucketAugment: Reinforced Domain Generalisation in Abdominal CT Segmentation

Goal: In recent years, deep neural networks have consistently outperformed previously proposed methods in the domain of medical segmentation. However, due to their nature, these networks often struggle to delineate desired structures in data that fall outside their training distribution. The goal of this study is to address the challenges associated with domain generalization in CT segmentation by introducing a novel method called BucketAugment for deep neural networks. Methods: BucketAugment leverages principles from the Q-learning algorithm and employs validation loss to search for an optimal policy within a search space comprised of distributed stacks of 3D volumetric augmentations, termed 'buckets.' These buckets have tunable parameters and can be seamlessly integrated into existing neural network architectures, offering flexibility for customization. Results: In our experiments, we focus on segmenting kidney and liver structures across three distinct medical datasets, each containing CT scans of the abdominal region collected from various clinical institutions and scanner vendors. Our results indicate that BucketAugment significantly enhances domain generalization across diverse medical datasets, requiring only minimal modifications to existing network architectures. Conclusions: The introduction of BucketAugment provides a promising solution to the challenges of domain generalization in CT segmentation. By leveraging Q-learning principles and distributed stacks of 3D augmentations, this method improves the performance of deep neural networks on medical segmentation tasks, demonstrating its potential to enhance the applicability of such models across different datasets and clinical scenarios.

MS-TCNet: An effective Transformer-CNN combined network using multi-scale feature learning for 3D medical image segmentation

Medical image segmentation is a fundamental research problem in the field of medical image processing. Recently, the Transformer have achieved highly competitive performance in computer vision. Therefore, many methods combining Transformer with convolutional neural networks (CNNs) have emerged for segmenting medical images. However, these methods cannot effectively capture the multi-scale features in medical images, even though texture and contextual information embedded in the multi-scale features are extremely beneficial for segmentation. To alleviate this limitation, we propose a novel Transformer-CNN combined network using multi-scale feature learning for three-dimensional (3D) medical image segmentation, which is called MS-TCNet. The proposed model utilizes a shunted Transformer and CNN to construct an encoder and pyramid decoder, allowing six different scale levels of feature learning. It captures multi-scale features with refinement at each scale level. Additionally, we propose a novel lightweight multi-scale feature fusion (MSFF) module that can fully fuse the different-scale semantic features generated by the pyramid decoder for each segmentation class, resulting in a more accurate segmentation output. We conducted experiments on three widely used 3D medical image segmentation datasets. The experimental results indicated that our method outperformed state-of-the-art medical image segmentation methods, suggesting its effectiveness, robustness, and superiority. Meanwhile, our model has a smaller number of parameters and lower computational complexity than conventional 3D segmentation networks. The results confirmed that the model is capable of effective multi-scale feature learning and that the learned multi-scale features are useful for improving segmentation performance. We open-sourced our code, which can be found at https://github.com/AustinYuAo/MS-TCNet.

In vivo strain measurements in the human buttock during sitting using MR-based digital volume correlation

Advancements in systems for prevention and management of pressure ulcers require a more detailed understanding of the complex response of soft tissues to compressive loads. This study aimed at quantifying the progressive deformation of the buttock based on 3D measurements of soft tissue displacements from MR scans of 10 healthy subjects in a semi-recumbent position. Measurements were obtained using digital volume correlation (DVC) and released as a public dataset. A first parametric optimisation of the global registration step aimed at aligning skeletal elements showed acceptable values of Dice coefficient (around 80%). A second parametric optimisation on the deformable registration method showed errors of 0.99mm and 1.78mm against two simulated fields with magnitude 7.30±3.15mm and 19.37±9.58mm, respectively, generated with a finite element model of the buttock under sitting loads. Measurements allowed the quantification of the slide of the gluteus maximus away from the ischial tuberosity (IT, average 13.74 mm) that was only qualitatively identified in the literature, highlighting the importance of the ischial bursa in allowing sliding. Spatial evolution of the maximus shear strain on a path from the IT to the seating interface showed a peak of compression in the fat, close to the interface with the muscle. Obtained peak values were above the proposed damage threshold in the literature. Results in the study showed the complexity of the deformation of the soft tissues in the buttock and the need for further investigations aimed at isolating factors such as tissue geometry, duration and extent of load, sitting posture and tissue properties.

Tailored multi-organ segmentation with model adaptation and ensemble

Multi-organ segmentation, which identifies and separates different organs in medical images, is a fundamental task in medical image analysis. Recently, the immense success of deep learning motivated its wide adoption in multi-organ segmentation tasks. However, due to expensive labor costs and expertise, the availability of multi-organ annotations is usually limited and hence poses a challenge in obtaining sufficient training data for deep learning-based methods. In this paper, we aim to address this issue by combining off-the-shelf single-organ segmentation models to develop a multi-organ segmentation model on the target dataset, which helps get rid of the dependence on annotated data for multi-organ segmentation. To this end, we propose a novel dual-stage method that consists of a Model Adaptation stage and a Model Ensemble stage. The first stage enhances the generalization of each off-the-shelf segmentation model on the target domain, while the second stage distills and integrates knowledge from multiple adapted single-organ segmentation models. Extensive experiments on four abdomen datasets demonstrate that our proposed method can effectively leverage off-the-shelf single-organ segmentation models to obtain a tailored model for multi-organ segmentation with high accuracy.

Deep Learning for Fully Automated Prediction of Overall Survival in Patients Undergoing Resection for Pancreatic Cancer: A Retrospective Multicenter Study

OBJECTIVE

To develop an imaging-derived biomarker for prediction of overall survival (OS) of pancreatic cancer by analyzing preoperative multiphase contrast-enhanced computed topography (CECT) using deep learning.

BACKGROUND

Exploiting prognostic biomarkers for guiding neoadjuvant and adjuvant treatment decisions may potentially improve outcomes in patients with resectable pancreatic cancer.

METHODS

This multicenter, retrospective study included 1516 patients with resected pancreatic ductal adenocarcinoma (PDAC) from 5 centers located in China. The discovery cohort (n=763), which included preoperative multiphase CECT scans and OS data from 2 centers, was used to construct a fully automated imaging-derived prognostic biomarker-DeepCT-PDAC-by training scalable deep segmentation and prognostic models (via self-learning) to comprehensively model the tumor-anatomy spatial relations and their appearance dynamics in multiphase CECT for OS prediction. The marker was independently tested using internal (n=574) and external validation cohorts (n=179, 3 centers) to evaluate its performance, robustness, and clinical usefulness.

RESULTS

Preoperatively, DeepCT-PDAC was the strongest predictor of OS in both internal and external validation cohorts [hazard ratio (HR) for high versus low risk 2.03, 95% confidence interval (CI): 1.50-2.75; HR: 2.47, CI: 1.35-4.53] in a multivariable analysis. Postoperatively, DeepCT-PDAC remained significant in both cohorts (HR: 2.49, CI: 1.89-3.28; HR: 2.15, CI: 1.14-4.05) after adjustment for potential confounders. For margin-negative patients, adjuvant chemoradiotherapy was associated with improved OS in the subgroup with DeepCT-PDAC low risk (HR: 0.35, CI: 0.19-0.64), but did not affect OS in the subgroup with high risk.

CONCLUSIONS

Deep learning-based CT imaging-derived biomarker enabled the objective and unbiased OS prediction for patients with resectable PDAC. This marker is applicable across hospitals, imaging protocols, and treatments, and has the potential to tailor neoadjuvant and adjuvant treatments at the individual level.

Contour-aware network with class-wise convolutions for 3D abdominal multi-organ segmentation

Accurate delineation of multiple organs is a critical process for various medical procedures, which could be operator-dependent and time-consuming. Existing organ segmentation methods, which were mainly inspired by natural image analysis techniques, might not fully exploit the traits of the multi-organ segmentation task and could not accurately segment the organs with various shapes and sizes simultaneously. In this work, the characteristics of multi-organ segmentation are considered: the global count, position and scale of organs are generally predictable, while their local shape and appearance are volatile. Thus, we supplement the region segmentation backbone with a contour localization task to increase the certainty along delicate boundaries. Meantime, each organ has exclusive anatomical traits, which motivates us to deal with class variability with class-wise convolutions to highlight organ-specific features and suppress irrelevant responses at different field-of-views. To validate our method with adequate amounts of patients and organs, we constructed a multi-center dataset, which contains 110 3D CT scans with 24,528 axial slices, and provided voxel-level manual segmentations of 14 abdominal organs, which adds up to 1,532 3D structures in total. Extensive ablation and visualization studies on it validate the effectiveness of the proposed method. Quantitative analysis shows that we achieve state-of-the-art performance for most abdominal organs, and obtain 3.63 mm 95% Hausdorff Distance and 83.32% Dice Similarity Coefficient on an average.

High-Resolution Swin Transformer for Automatic Medical Image Segmentation

The resolution of feature maps is a critical factor for accurate medical image segmentation. Most of the existing Transformer-based networks for medical image segmentation adopt a U-Net-like architecture, which contains an encoder that converts the high-resolution input image into low-resolution feature maps using a sequence of Transformer blocks and a decoder that gradually generates high-resolution representations from low-resolution feature maps. However, the procedure of recovering high-resolution representations from low-resolution representations may harm the spatial precision of the generated segmentation masks. Unlike previous studies, in this study, we utilized the high-resolution network (HRNet) design style by replacing the convolutional layers with Transformer blocks, continuously exchanging feature map information with different resolutions generated by the Transformer blocks. The proposed Transformer-based network is named the high-resolution Swin Transformer network (HRSTNet). Extensive experiments demonstrated that the HRSTNet can achieve performance comparable with that of the state-of-the-art Transformer-based U-Net-like architecture on the 2021 Brain Tumor Segmentation dataset, the Medical Segmentation Decathlon's liver dataset, and the BTCV multi-organ segmentation dataset.

Learn2Reg: Comprehensive Multi-Task Medical Image Registration Challenge, Dataset and Evaluation in the Era of Deep Learning

Image registration is a fundamental medical image analysis task, and a wide variety of approaches have been proposed. However, only a few studies have comprehensively compared medical image registration approaches on a wide range of clinically relevant tasks. This limits the development of registration methods, the adoption of research advances into practice, and a fair benchmark across competing approaches. The Learn2Reg challenge addresses these limitations by providing a multi-task medical image registration data set for comprehensive characterisation of deformable registration algorithms. A continuous evaluation will be possible at https://learn2reg.grand-challenge.org. Learn2Reg covers a wide range of anatomies (brain, abdomen, and thorax), modalities (ultrasound, CT, MR), availability of annotations, as well as intra- and inter-patient registration evaluation. We established an easily accessible framework for training and validation of 3D registration methods, which enabled the compilation of results of over 65 individual method submissions from more than 20 unique teams. We used a complementary set of metrics, including robustness, accuracy, plausibility, and runtime, enabling unique insight into the current state-of-the-art of medical image registration. This paper describes datasets, tasks, evaluation methods and results of the challenge, as well as results of further analysis of transferability to new datasets, the importance of label supervision, and resulting bias. While no single approach worked best across all tasks, many methodological aspects could be identified that push the performance of medical image registration to new state-of-the-art performance. Furthermore, we demystified the common belief that conventional registration methods have to be much slower than deep-learning-based methods.

DiffeoRaptor: diffeomorphic inter-modal image registration using RaPTOR

PURPOSE

Diffeomorphic image registration is essential in many medical imaging applications. Several registration algorithms of such type have been proposed, but primarily for intra-contrast alignment. Currently, efficient inter-modal/contrast diffeomorphic registration, which is vital in numerous applications, remains a challenging task.

METHODS

We proposed a novel inter-modal/contrast registration algorithm that leverages Robust PaTch-based cOrrelation Ratio metric to allow inter-modal/contrast image alignment and bandlimited geodesic shooting demonstrated in Fourier-Approximated Lie Algebras (FLASH) algorithm for fast diffeomorphic registration.

RESULTS

The proposed algorithm, named DiffeoRaptor, was validated with three public databases for the tasks of brain and abdominal image registration while comparing the results against three state-of-the-art techniques, including FLASH, NiftyReg, and Symmetric image Normalization (SyN).

CONCLUSIONS

Our results demonstrated that DiffeoRaptor offered comparable or better registration performance in terms of registration accuracy. Moreover, DiffeoRaptor produces smoother deformations than SyN in inter-modal and contrast registration. The code for DiffeoRaptor is publicly available at https://github.com/nimamasoumi/DiffeoRaptor .

An End-to-End Data-Adaptive Pancreas Segmentation System with an Image Quality Control Toolbox

With the development of radiology and computer technology, diagnosis by medical imaging is heading toward precision and automation. Due to complex anatomy around the pancreatic tissue and high demands for clinical experience, the assisted pancreas segmentation system will greatly promote clinical efficiency. However, the existing segmentation model suffers from poor generalization among images from multiple hospitals. In this paper, we propose an end-to-end data-adaptive pancreas segmentation system to tackle the problems of lack of annotations and model generalizability. The system employs adversarial learning to transfer features from labeled domains to unlabeled domains, seeking a dynamic balance between domain discrimination and unsupervised segmentation. The image quality control toolbox is embedded in the system, which standardizes image quality in terms of intensity, field of view, and so on, to decrease heterogeneity among image domains. In addition, the system implements a data-adaptive process end-to-end without complex operations by doctors. The experiments are conducted on an annotated public dataset and an unannotated in-hospital dataset. The results indicate that after data adaptation, the segmentation performance measured by the dice similarity coefficient on unlabeled images improves from 58.79% to 75.43%, with a gain of 16.64%. Furthermore, the system preserves quantitatively structured information such as the pancreas’ size and volume, as well as objective and accurate visualized images, which assists clinicians in diagnosing and formulating treatment plans in a timely and accurate manner.

Complete region of interest reconstruction by fusing multiview deformable three-dimensional transesophageal echocardiography images

BACKGROUND

While three-dimensional transesophageal echocardiography (3D TEE) has been increasingly used for assessing cardiac anatomy and function, it still suffers from a limited field of view (FoV) of the ultrasound transducer. Therefore, it is difficult to examine a complete region of interest without moving the transducer. Existing methods extend the FoV of 3D TEE images by mosaicing multiview static images, which requires synchronization between 3D TEE images and electrocardiogram (ECG) signal to avoid deformations in the images and can only get the widened image at a specific phase.

PURPOSE

This work aims to develop a novel multiview nonrigid registration and fusion method to extend the FoV of 3D TEE images at different cardiac phases, avoiding the bias toward the specifically chosen phase.

METHODS

A multiview nonrigid registration and fusion method is proposed to enlarge the FoV of 3D TEE images by fusing dynamic images captured from different viewpoints sequentially. The deformation field for registering images is defined by a collection of affine transformations organized in a graph structure and is estimated by a direct (intensity-based) method. The accuracy of the proposed method is evaluated by comparing it with two B-spline-based methods, two Demons-based methods, and one learning-based method VoxelMorph. Twenty-nine sequences of in vivo 3D TEE images captured from four patients are used for the comparative experiments. Four performance metrics including checkerboard volumes, signed distance, mean absolute distance (MAD), and Dice similarity coefficient (DSC) are used jointly to evaluate the accuracy of the results. Additionally, paired t-tests are performed to examine the significance of the results.

RESULTS

The qualitative results show that the proposed method can align images more accurately and obtain the fused images with higher quality than the other five methods. Additionally, in the evaluation of the segmented left atrium (LA) walls for the pairwise registration and sequential fusion experiments, the proposed method achieves the MAD of (0.07 ± 0.03) mm for pairwise registration and (0.19 ± 0.02) mm for sequential fusion. Paired t-tests indicate that the results obtained from the proposed method are more accurate than those obtained by the state-of-the-art VoxelMorph and the diffeomorphic Demons methods at the significance level of 0.05. In the evaluation of left ventricle (LV) segmentations for the sequential fusion experiments, the proposed method achieves a DSC of (0.88 ± 0.08), which is also significantly better than diffeomorphic Demons at the 0.05 level. The FoVs of the final fused 3D TEE images obtained by our method are enlarged around two times compared with the original images.

CONCLUSIONS

Without selecting the static (ECG-gated) images from the same cardiac phase, this work addressed the problem of limited FoV of 3D TEE images in the deformable scenario, obtaining the fused images with high accuracy and good quality. The proposed method could provide an alternative to the conventional fusion methods that are biased toward the specifically chosen phase.

SAM: Self-Supervised Learning of Pixel-Wise Anatomical Embeddings in Radiological Images

Radiological images such as computed tomography (CT) and X-rays render anatomy with intrinsic structures. Being able to reliably locate the same anatomical structure across varying images is a fundamental task in medical image analysis. In principle it is possible to use landmark detection or semantic segmentation for this task, but to work well these require large numbers of labeled data for each anatomical structure and sub-structure of interest. A more universal approach would learn the intrinsic structure from unlabeled images. We introduce such an approach, called Self-supervised Anatomical eMbedding (SAM). SAM generates semantic embeddings for each image pixel that describes its anatomical location or body part. To produce such embeddings, we propose a pixel-level contrastive learning framework. A coarse-to-fine strategy ensures both global and local anatomical information are encoded. Negative sample selection strategies are designed to enhance the embedding's discriminability. Using SAM, one can label any point of interest on a template image and then locate the same body part in other images by simple nearest neighbor searching. We demonstrate the effectiveness of SAM in multiple tasks with 2D and 3D image modalities. On a chest CT dataset with 19 landmarks, SAM outperforms widely-used registration algorithms while only taking 0.23 seconds for inference. On two X-ray datasets, SAM, with only one labeled template image, surpasses supervised methods trained on 50 labeled images. We also apply SAM on whole-body follow-up lesion matching in CT and obtain an accuracy of 91%. SAM can also be applied for improving image registration and initializing CNN weights.

Automated Segmentation of Spinal Muscles From Upright Open MRI Using a Multiscale Pyramid 2D Convolutional Neural Network

STUDY DESIGN

Randomized trial.

OBJECTIVE

To implement an algorithm enabling the automated segmentation of spinal muscles from open magnetic resonance images in healthy volunteers and patients with adult spinal deformity (ASD).

SUMMARY OF BACKGROUND DATA

Understanding spinal muscle anatomy is critical to diagnosing and treating spinal deformity.Muscle boundaries can be extrapolated from medical images using segmentation, which is usually done manually by clinical experts and remains complicated and time-consuming.

METHODS

Three groups were examined: two healthy volunteer groups (N = 6 for each group) and one ASD group (N = 8 patients) were imaged at the lumbar and thoracic regions of the spine in an upright open magnetic resonance imaging scanner while maintaining different postures (various seated, standing, and supine). For each group and region, a selection of regions of interest (ROIs) was manually segmented. A multiscale pyramid two-dimensional convolutional neural network was implemented to automatically segment all defined ROIs. A five-fold crossvalidation method was applied and distinct models were trained for each resulting set and group and evaluated using Dice coefficients calculated between the model output and the manually segmented target.

RESULTS

Good to excellent results were found across all ROIs for the ASD (Dice coefficient >0.76) and healthy (dice coefficient > 0.86) groups.

CONCLUSION

This study represents a fundamental step toward the development of an automated spinal muscle properties extraction pipeline, which will ultimately allow clinicians to have easier access to patient-specific simulations, diagnosis, and treatment.

Learning-based three-dimensional registration with weak bounding box supervision

Purpose: Image registration is the process of aligning images, and it is a fundamental task in medical image analysis. While many tasks in the field of image analysis, such as image segmentation, are handled almost entirely with deep learning and exceed the accuracy of conventional algorithms, currently available deformable image registration methods are often still conventional. Deep learning methods for medical image registration have recently reached the accuracy of conventional algorithms. However, they are often based on a weakly supervised learning scheme using multilabel image segmentations during training. The creation of such detailed annotations is very time-consuming. Approach: We propose a weakly supervised learning scheme for deformable image registration. By calculating the loss function based on only bounding box labels, we are able to train an image registration network for large displacement deformations without using densely labeled images. We evaluate our model on interpatient three-dimensional abdominal CT and MRI images. Results: The results show an improvement of ∼ 10 % (for CT images) and 20% (for MRI images) in comparison to the unsupervised method. When taking into account the reduced annotation effort, the performance also exceeds the performance of weakly supervised training using detailed image segmentations. Conclusion: We show that the performance of image registration methods can be enhanced with little annotation effort using our proposed method.

Salient deformable network for abdominal multiorgan registration

BACKGROUND

Image registration has long been an active research area in the society of medical image computing, which is to perform spatial transformation between a pair of images and establish a point-wise correspondence to achieve spatial consistency.

PURPOSE

Previous work mainly focused on learning complicated deformation fields by maximizing the global-level (i.e., foreground plus background) image similarity. We argue that taking the background similarity into account may not be a good solution, if we only seek the accurate alignment of target organs/regions in real clinical practice.

METHODS

We, therefore, propose a novel concept of S a l i e n t $Salient$ R e g i s t r a t i o n $Registration$ and introduce a novel deformable network equipped with a saliency module. Specifically, a multitask learning-based saliency module is proposed to discriminate the salient regions-of-registration in a semisupervised manner. Then, our deformable network analyzes the intensity and anatomical similarity of salient regions, and finally conducts the salient deformable registration.

RESULTS

We evaluate the efficacy of the proposed network on challenging abdominal multiorgan CT scans. The experimental results demonstrate that the proposed registration network outperforms other state-of-the-art methods, achieving a mean Dice similarity coefficient (DSC) of 40.2%, Hausdorff distance (95 HD) of 20.8 mm, and average symmetric surface distance (ASSD) of 4.58 mm. Moreover, even by training using one labeled data, our network can still attain satisfactory registration performance, with a mean DSC of 39.2%, 95 HD of 21.2 mm, and ASSD of 4.78 mm.

CONCLUSIONS

The proposed network provides an accurate solution for multiorgan registration and has the potential to be used for improving other registration applications. The code is publicly available at https://github.com/Rrrfrr/Salient-Deformable-Network.

Fully Automated Abdominal CT Biomarkers for Type 2 Diabetes Using Deep Learning

Background CT biomarkers both inside and outside the pancreas can potentially be used to diagnose type 2 diabetes mellitus. Previous studies on this topic have shown significant results but were limited by manual methods and small study samples. Purpose To investigate abdominal CT biomarkers for type 2 diabetes mellitus in a large clinical data set using fully automated deep learning. Materials and Methods For external validation, noncontrast abdominal CT images were retrospectively collected from consecutive patients who underwent routine colorectal cancer screening with CT colonography from 2004 to 2016. The pancreas was segmented using a deep learning method that outputs measurements of interest, including CT attenuation, volume, fat content, and pancreas fractal dimension. Additional biomarkers assessed included visceral fat, atherosclerotic plaque, liver and muscle CT attenuation, and muscle volume. Univariable and multivariable analyses were performed, separating patients into groups based on time between type 2 diabetes diagnosis and CT date and including clinical factors such as sex, age, body mass index (BMI), BMI greater than 30 kg/m², and height. The best set of predictors for type 2 diabetes were determined using multinomial logistic regression. Results A total of 8992 patients (mean age, 57 years ± 8 [SD]; 5009 women) were evaluated in the test set, of whom 572 had type 2 diabetes mellitus. The deep learning model had a mean Dice similarity coefficient for the pancreas of 0.69 ± 0.17, similar to the interobserver Dice similarity coefficient of 0.69 ± 0.09 (P = .92). The univariable analysis showed that patients with diabetes had, on average, lower pancreatic CT attenuation (mean, 18.74 HU ± 16.54 vs 29.99 HU ± 13.41; P < .0001) and greater visceral fat volume (mean, 235.0 mL ± 108.6 vs 130.9 mL ± 96.3; P < .0001) than those without diabetes. Patients with diabetes also showed a progressive decrease in pancreatic attenuation with greater duration of disease. The final multivariable model showed pairwise areas under the receiver operating characteristic curve (AUCs) of 0.81 and 0.85 between patients without and patients with diabetes who were diagnosed 0-2499 days before and after undergoing CT, respectively. In the multivariable analysis, adding clinical data did not improve upon CT-based AUC performance (AUC = 0.67 for the CT-only model vs 0.68 for the CT and clinical model). The best predictors of type 2 diabetes mellitus included intrapancreatic fat percentage, pancreatic fractal dimension, plaque severity between the L1 and L4 vertebra levels, average liver CT attenuation, and BMI. Conclusion The diagnosis of type 2 diabetes mellitus was associated with abdominal CT biomarkers, especially measures of pancreatic CT attenuation and visceral fat. © RSNA, 2022 Online supplemental material is available for this article.

SynthMorph: Learning Contrast-Invariant Registration Without Acquired Images

We introduce a strategy for learning image registration without acquired imaging data, producing powerful networks agnostic to contrast introduced by magnetic resonance imaging (MRI). While classical registration methods accurately estimate the spatial correspondence between images, they solve an optimization problem for every new image pair. Learning-based techniques are fast at test time but limited to registering images with contrasts and geometric content similar to those seen during training. We propose to remove this dependency on training data by leveraging a generative strategy for diverse synthetic label maps and images that exposes networks to a wide range of variability, forcing them to learn more invariant features. This approach results in powerful networks that accurately generalize to a broad array of MRI contrasts. We present extensive experiments with a focus on 3D neuroimaging, showing that this strategy enables robust and accurate registration of arbitrary MRI contrasts even if the target contrast is not seen by the networks during training. We demonstrate registration accuracy surpassing the state of the art both within and across contrasts, using a single model. Critically, training on arbitrary shapes synthesized from noise distributions results in competitive performance, removing the dependency on acquired data of any kind. Additionally, since anatomical label maps are often available for the anatomy of interest, we show that synthesizing images from these dramatically boosts performance, while still avoiding the need for real intensity images. Our code is available at doic https://w3id.org/synthmorph.

Supervised Deep Generation of High-Resolution Arterial Phase Computed Tomography Kidney Substructure Atlas

The Human BioMolecular Atlas Program (HuBMAP) provides an opportunity to contextualize findings across cellular to organ systems levels. Constructing an atlas target is the primary endpoint for generalizing anatomical information across scales and populations. An initial target of HuBMAP is the kidney organ and arterial phase contrast-enhanced computed tomography (CT) provides distinctive appearance and anatomical context on the internal substructure of kidney organs such as renal context, medulla, and pelvicalyceal system. With the confounding effects of demographics and morphological characteristics of the kidney across large-scale imaging surveys, substantial variation is demonstrated with the internal substructure morphometry and the intensity contrast due to the variance of imaging protocols. Such variability increases the level of difficulty to localize the anatomical features of the kidney substructure in a well-defined spatial reference for clinical analysis. In order to stabilize the localization of kidney substructures in the context of this variability, we propose a high-resolution CT kidney substructure atlas template. Briefly, we introduce a deep learning preprocessing technique to extract the volumetric interest of the abdominal regions and further perform a deep supervised registration pipeline to stably adapt the anatomical context of the kidney internal substructure. To generate and evaluate the atlas template, arterial phase CT scans of 500 control subjects are de-identified and registered to the atlas template with a complete end-to-end pipeline. With stable registration to the abdominal wall and kidney organs, the internal substructure of both left and right kidneys are substantially localized in the high-resolution atlas space. The atlas average template successfully demonstrated the contextual details of the internal structure and was applicable to generalize the morphological variation of internal substructure across patients.

Comparison of an in-house hybrid DIR method to NiftyReg on CBCT and CT images for head and neck cancer

An in-house hybrid deformable image registration (DIR) method, which combines free-form deformation (FFD) and the viscous fluid registration method, is proposed. Its results on the planning computed tomography (CT) and the day 1 treatment cone-beam CT (CBCT) image from 68 head and neck cancer patients are compared with the results of NiftyReg, which uses B-spline FFD alone. Several similarity metrics, the target registration error (TRE) of annotated points, as well as the Dice similarity coefficient (DSC) and Hausdorff distance (HD) of the propagated organs at risk are employed to analyze their registration accuracy. According to quantitative analysis on mutual information, normalized cross-correlation, and the absolute pixel value differences, the results of the proposed DIR are more similar to the CBCT images than the NiftyReg results. Smaller TRE of the annotated points is observed in the proposed method, and the overall mean TRE for the proposed method and NiftyReg was 2.34 and 2.98 mm, respectively (p < 0.001). The mean DSC in the larynx, spinal cord, oral cavity, mandible, and parotid given by the proposed method ranged from 0.78 to 0.91, significantly higher than the NiftyReg results (ranging from 0.77 to 0.90), and the HD was significantly lower compared to NiftyReg. Furthermore, the proposed method did not suffer from unrealistic deformations as the NiftyReg did in the visual evaluation. Meanwhile, the execution time of the proposed method was much higher than NiftyReg (96.98 ± 11.88 s vs. 4.60 ± 0.49 s). In conclusion, the in-house hybrid method gave better accuracy and more stable performance than NiftyReg.

Radiomics Based on Contrast-Enhanced MRI in Differentiation Between Fat-Poor Angiomyolipoma and Hepatocellular Carcinoma in Noncirrhotic Liver: A Multicenter Analysis

Objective

This study aims to develop and externally validate a contrast-enhanced magnetic resonance imaging (CE-MRI) radiomics-based model for preoperative differentiation between fat-poor angiomyolipoma (fp-AML) and hepatocellular carcinoma (HCC) in patients with noncirrhotic livers and to compare the diagnostic performance with that of two radiologists.

Methods

This retrospective study was performed with 165 patients with noncirrhotic livers from three medical centers. The dataset was divided into a training cohort (n = 99), a time-independent internal validation cohort (n = 24) from one center, and an external validation cohort (n = 42) from the remaining two centers. The volumes of interest were contoured on the arterial phase (AP) images and then registered to the venous phase (VP) and delayed phase (DP), and a total of 3,396 radiomics features were extracted from the three phases. After the joint mutual information maximization feature selection procedure, four radiomics logistic regression classifiers, including the AP model, VP model, DP model, and combined model, were built. The area under the receiver operating characteristic curve (AUC), diagnostic accuracy, sensitivity, and specificity of each radiomics model and those of two radiologists were evaluated and compared.

Results

The AUCs of the combined model reached 0.789 (95%CI, 0.579-0.999) in the internal validation cohort and 0.730 (95%CI, 0.563-0.896) in the external validation cohort, higher than the AP model (AUCs, 0.711 and 0.638) and significantly higher than the VP model (AUCs, 0.594 and 0.610) and the DP model (AUCs, 0.547 and 0.538). The diagnostic accuracy, sensitivity, and specificity of the combined model were 0.708, 0.625, and 0.750 in the internal validation cohort and 0.619, 0.786, and 0.536 in the external validation cohort, respectively. The AUCs for the two radiologists were 0.656 and 0.594 in the internal validation cohort and 0.643 and 0.500 in the external validation cohort. The AUCs of the combined model surpassed those of the two radiologists and were significantly higher than that of the junior one in both validation cohorts.

Conclusions

The proposed radiomics model based on triple-phase CE-MRI images was proven to be useful for differentiating between fp-AML and HCC and yielded comparable or better performance than two radiologists in different centers, with different scanners and different scanning parameters.

A Unified Framework for Generalized Low-Shot Medical Image Segmentation With Scarce Data

Medical image segmentation has achieved remarkable advancements using deep neural networks (DNNs). However, DNNs often need big amounts of data and annotations for training, both of which can be difficult and costly to obtain. In this work, we propose a unified framework for generalized low-shot (one- and few-shot) medical image segmentation based on distance metric learning (DML). Unlike most existing methods which only deal with the lack of annotations while assuming abundance of data, our framework works with extreme scarcity of both, which is ideal for rare diseases. Via DML, the framework learns a multimodal mixture representation for each category, and performs dense predictions based on cosine distances between the pixels' deep embeddings and the category representations. The multimodal representations effectively utilize the inter-subject similarities and intraclass variations to overcome overfitting due to extremely limited data. In addition, we propose adaptive mixing coefficients for the multimodal mixture distributions to adaptively emphasize the modes better suited to the current input. The representations are implicitly embedded as weights of the fc layer, such that the cosine distances can be computed efficiently via forward propagation. In our experiments on brain MRI and abdominal CT datasets, the proposed framework achieves superior performances for low-shot segmentation towards standard DNN-based (3D U-Net) and classical registration-based (ANTs) methods, e.g., achieving mean Dice coefficients of 81%/69% for brain tissue/abdominal multi-organ segmentation using a single training sample, as compared to 52%/31% and 72%/35% by the U-Net and ANTs, respectively.

An Unsupervised Learning-Based Multi-Organ Registration Method for 3D Abdominal CT Images

Medical image registration is an essential technique to achieve spatial consistency geometric positions of different medical images obtained from single- or multi-sensor, such as computed tomography (CT), magnetic resonance (MR), and ultrasound (US) images. In this paper, an improved unsupervised learning-based framework is proposed for multi-organ registration on 3D abdominal CT images. First, the explored coarse-to-fine recursive cascaded network (RCN) modules are embedded into a basic U-net framework to achieve more accurate multi-organ registration results from 3D abdominal CT images. Then, a topology-preserving loss is added in the total loss function to avoid a distortion of the predicted transformation field. Four public databases are selected to validate the registration performances of the proposed method. The experimental results show that the proposed method is superior to some existing traditional and deep learning-based methods and is promising to meet the real-time and high-precision clinical registration requirements of 3D abdominal CT images.

3D deformable registration of longitudinal abdominopelvic CT images using unsupervised deep learning

BACKGROUND AND OBJECTIVES

Deep learning is being increasingly used for deformable image registration and unsupervised approaches, in particular, have shown great potential. However, the registration of abdominopelvic Computed Tomography (CT) images remains challenging due to the larger displacements compared to those in brain or prostate Magnetic Resonance Imaging datasets that are typically considered as benchmarks. In this study, we investigate the use of the commonly used unsupervised deep learning framework VoxelMorph for the registration of a longitudinal abdominopelvic CT dataset acquired in patients with bone metastases from breast cancer.

METHODS

As a pre-processing step, the abdominopelvic CT images were refined by automatically removing the CT table and all other extra-corporeal components. To improve the learning capabilities of the VoxelMorph framework when only a limited amount of training data is available, a novel incremental training strategy is proposed based on simulated deformations of consecutive CT images in the longitudinal dataset. This devised training strategy was compared against training on simulated deformations of a single CT volume. A widely used software toolbox for deformable image registration called NiftyReg was used as a benchmark. The evaluations were performed by calculating the Dice Similarity Coefficient (DSC) between manual vertebrae segmentations and the Structural Similarity Index (SSIM).

RESULTS

The CT table removal procedure allowed both VoxelMorph and NiftyReg to achieve significantly better registration performance. In a 4-fold cross-validation scheme, the incremental training strategy resulted in better registration performance compared to training on a single volume, with a mean DSC of 0.929±0.037 and 0.883±0.033, and a mean SSIM of 0.984±0.009 and 0.969±0.007, respectively. Although our deformable image registration method did not outperform NiftyReg in terms of DSC (0.988±0.003) or SSIM (0.995±0.002), the registrations were approximately 300 times faster.

CONCLUSIONS

This study showed the feasibility of deep learning based deformable registration of longitudinal abdominopelvic CT images via a novel incremental training strategy based on simulated deformations.

A Medical Image Registration Method Based on Progressive Images

Background

Medical image registration is an essential task for medical image analysis in various applications. In this work, we develop a coarse-to-fine medical image registration method based on progressive images and SURF algorithm (PI-SURF) for higher registration accuracy.

Methods

As a first step, the reference image and the floating image are fused to generate multiple progressive images. Thereafter, the floating image and progressive image are registered to get the coarse registration result based on the SURF algorithm. For further improvement, the coarse registration result and the reference image are registered to perform fine image registration. The appropriate progressive image has been investigated by experiments. The mutual information (MI), normal mutual information (NMI), normalized correlation coefficient (NCC), and mean square difference (MSD) similarity metrics are used to demonstrate the potential of the PI-SURF method.

Results

For the unimodal and multimodal registration, the PI-SURF method achieves the best results compared with the mutual information method, Demons method, Demons+B-spline method, and SURF method. The MI, NMI, and NCC of PI-SURF are improved by 15.5%, 1.31%, and 7.3%, respectively, while MSD decreased by 13.2% for the multimodal registration compared with the optimal result of the state-of-the-art methods.

Conclusions

The extensive experiments show that the proposed PI-SURF method achieves higher quality of registration.

Incorporating the hybrid deformable model for improving the performance of abdominal CT segmentation via multi-scale feature fusion network

Automated multi-organ abdominal Computed Tomography (CT) image segmentation can assist the treatment planning, diagnosis, and improve many clinical workflows' efficiency. The 3-D Convolutional Neural Network (CNN) recently attained state-of-the-art accuracy, which typically relies on supervised training with many manual annotated data. Many methods used the data augmentation strategy with a rigid or affine spatial transformation to alleviate the over-fitting problem and improve the network's robustness. However, the rigid or affine spatial transformation fails to capture the complex voxel-based deformation in the abdomen, filled with many soft organs. We developed a novel Hybrid Deformable Model (HDM), which consists of the inter-and intra-patient deformation for more effective data augmentation to tackle this issue. The inter-patient deformations were extracted from the learning-based deformable registration between different patients, while the intra-patient deformations were formed using the random 3-D Thin-Plate-Spline (TPS) transformation. Incorporating the HDM enabled the network to capture many of the subtle deformations of abdominal organs. To find a better solution and achieve faster convergence for network training, we fused the pre-trained multi-scale features into the a 3-D attention U-Net. We directly compared the segmentation accuracy of the proposed method to the previous techniques on several centers' datasets via cross-validation. The proposed method achieves the average Dice Similarity Coefficient (DSC) 0.852, which outperformed the other state-of-the-art on multi-organ abdominal CT segmentation results.

DeepPrognosis: Preoperative prediction of pancreatic cancer survival and surgical margin via comprehensive understanding of dynamic contrast-enhanced CT imaging and tumor-vascular contact parsing

Pancreatic ductal adenocarcinoma (PDAC) is one of the most lethal cancers and carries a dismal prognosis of ∼10% in five year survival rate. Surgery remains the best option of a potential cure for patients who are evaluated to be eligible for initial resection of PDAC. However, outcomes vary significantly even among the resected patients who were the same cancer stage and received similar treatments. Accurate quantitative preoperative prediction of primary resectable PDACs for personalized cancer treatment is thus highly desired. Nevertheless, there are a very few automated methods yet to fully exploit the contrast-enhanced computed tomography (CE-CT) imaging for PDAC prognosis assessment. CE-CT plays a critical role in PDAC staging and resectability evaluation. In this work, we propose a novel deep neural network model for the survival prediction of primary resectable PDAC patients, named as 3D Contrast-Enhanced Convolutional Long Short-Term Memory network (CE-ConvLSTM), which can derive the tumor attenuation signatures or patterns from patient CE-CT imaging studies. Tumor-vascular relationships, which might indicate the resection margin status, have also been proven to hold strong relationships with the overall survival of PDAC patients. To capture such relationships, we propose a self-learning approach for automated pancreas and peripancreatic anatomy segmentation without requiring any annotations on our PDAC datasets. We then employ a multi-task convolutional neural network (CNN) to accomplish both tasks of survival outcome and margin prediction where the network benefits from learning the resection margin related image features to improve the survival prediction. Our presented framework can improve overall survival prediction performances compared with existing state-of-the-art survival analysis approaches. The new staging biomarker integrating both the proposed risk signature and margin prediction has evidently added values to be combined with the current clinical staging system.

An Abdominal Registration Technology for Integration of Nanomaterial Imaging-Aided Diagnosis and Treatment

Image registration technology is a key technology used in the process of nanomaterial imaging-aided diagnosis and targeted therapy effect monitoring for abdominal diseases. Recently, the deep-learning based methods have been increasingly used for large-scale medical image registration, because their iteration is much less than those of traditional ones. In this paper, a coarse-to-fine unsupervised learning-based three-dimensional (3D) abdominal CT image registration method is presented. Firstly, an affine transformation was used as an initial step to deal with large deformation between two images. Secondly, an unsupervised total loss function containing similarity, smoothness, and topology preservation measures was proposed to achieve better registration performances during convolutional neural network (CNN) training and testing. The experimental results demonstrated that the proposed method severally obtains the average MSE, PSNR, and SSIM values of 0.0055, 22.7950, and 0.8241, which outperformed some existing traditional and unsupervised learning-based methods. Moreover, our method can register 3D abdominal CT images with shortest time and is expected to become a real-time method for clinical application.

Domain adaptation for segmentation of critical structures for prostate cancer therapy

Preoperative assessment of the proximity of critical structures to the tumors is crucial in avoiding unnecessary damage during prostate cancer treatment. A patient-specific 3D anatomical model of those structures, namely the neurovascular bundles (NVB) and the external urethral sphincters (EUS), can enable physicians to perform such assessments intuitively. As a crucial step to generate a patient-specific anatomical model from preoperative MRI in a clinical routine, we propose a multi-class automatic segmentation based on an anisotropic convolutional network. Our specific challenge is to train the network model on a unique source dataset only available at a single clinical site and deploy it to another target site without sharing the original images or labels. As network models trained on data from a single source suffer from quality loss due to the domain shift, we propose a semi-supervised domain adaptation (DA) method to refine the model's performance in the target domain. Our DA method combines transfer learning and uncertainty guided self-learning based on deep ensembles. Experiments on the segmentation of the prostate, NVB, and EUS, show significant performance gain with the combination of those techniques compared to pure TL and the combination of TL with simple self-learning ([Formula: see text] for all structures using a Wilcoxon's signed-rank test). Results on a different task and data (Pancreas CT segmentation) demonstrate our method's generic application capabilities. Our method has the advantage that it does not require any further data from the source domain, unlike the majority of recent domain adaptation strategies. This makes our method suitable for clinical applications, where the sharing of patient data is restricted.

Abdominal multi-organ segmentation with cascaded convolutional and adversarial deep networks

Abdominal anatomy segmentation is crucial for numerous applications from computer-assisted diagnosis to image-guided surgery. In this context, we address fully-automated multi-organ segmentation from abdominal CT and MR images using deep learning. The proposed model extends standard conditional generative adversarial networks. Additionally to the discriminator which enforces the model to create realistic organ delineations, it embeds cascaded partially pre-trained convolutional encoder-decoders as generator. Encoder fine-tuning from a large amount of non-medical images alleviates data scarcity limitations. The network is trained end-to-end to benefit from simultaneous multi-level segmentation refinements using auto-context. Employed for healthy liver, kidneys and spleen segmentation, our pipeline provides promising results by outperforming state-of-the-art encoder-decoder schemes. Followed for the Combined Healthy Abdominal Organ Segmentation (CHAOS) challenge organized in conjunction with the IEEE International Symposium on Biomedical Imaging 2019, it gave us the first rank for three competition categories: liver CT, liver MR and multi-organ MR segmentation. Combining cascaded convolutional and adversarial networks strengthens the ability of deep learning pipelines to automatically delineate multiple abdominal organs, with good generalization capability. The comprehensive evaluation provided suggests that better guidance could be achieved to help clinicians in abdominal image interpretation and clinical decision making.

F3RNet: full-resolution residual registration network for deformable image registration

PURPOSE

Deformable image registration (DIR) is essential for many image-guided therapies. Recently, deep learning approaches have gained substantial popularity and success in DIR. Most deep learning approaches use the so-called mono-stream high-to-low, low-to-high network structure and can achieve satisfactory overall registration results. However, accurate alignments for some severely deformed local regions, which are crucial for pinpointing surgical targets, are often overlooked. Consequently, these approaches are not sensitive to some hard-to-align regions, e.g., intra-patient registration of deformed liver lobes.

METHODS

We propose a novel unsupervised registration network, namely full-resolution residual registration network (F3RNet), for deformable registration of severely deformed organs. The proposed method combines two parallel processing streams in a residual learning fashion. One stream takes advantage of the full-resolution information that facilitates accurate voxel-level registration. The other stream learns the deep multi-scale residual representations to obtain robust recognition. We also factorize the 3D convolution to reduce the training parameters and enhance network efficiency.

RESULTS

We validate the proposed method on a clinically acquired intra-patient abdominal CT-MRI dataset and a public inspiratory and expiratory thorax CT dataset. Experiments on both multimodal and unimodal registration demonstrate promising results compared to state-of-the-art approaches.

CONCLUSION

By combining the high-resolution information and multi-scale representations in a highly interactive residual learning fashion, the proposed F3RNet can achieve accurate overall and local registration. The run time for registering a pair of images is less than 3 s using a GPU. In future works, we will investigate how to cost-effectively process high-resolution information and fuse multi-scale representations.

Non-Rigid Liver Registration in Liver Computed Tomography Images Using Elastic Method with Global and Local Deformations

The study of follow-up liver computed tomography (CT) images is required for the early diagnosis and treatment evaluation of liver cancer. Although this requirement has been manually performed by doctors, the demands on computer-aided diagnosis are dramatically growing according to the increased amount of medical image data by the recent development of CT. However, conventional image segmentation, registration, and skeletonization methods cannot be directly applied to clinical data due to the characteristics of liver CT images varying largely by patients and contrast agents. In this paper, we propose non-rigid liver segmentation using elastic method with global and local deformation for follow-up liver CT images. To manage intensity differences between two scans, we extract the liver vessel and parenchyma in each scan. And our method binarizes the segmented liver parenchyma and vessel, and performs the registration to minimize the intensity difference between these binarized images of follow-up CT images. The global movements between follow-up CT images are corrected by rigid registration based on liver surface. The local deformations between follow-up CT images are modeled by non-rigid registration, which aligns images using non-rigid transformation, based on locally deformable model. Our method can model the global and local deformation between follow-up liver CT scans by considering the deformation of both the liver surface and vessel. In experimental results using twenty clinical datasets, our method matches the liver effectively between follow-up portal phase CT images, enabling the accurate assessment of the volume change of the liver cancer. The proposed registration method can be applied to the follow-up study of various organ diseases, including cardiovascular diseases and lung cancer.

Statistical parametric mapping of three-dimensional local activity distribution of skeletal muscle using magnetic resonance imaging (MRI)

Analysis of the internal local activity distribution in human skeletal muscles is important for managing muscle fatigue/pain and dysfunction. However, no method is established for three-dimensional (3D) statistical analysis of features of activity regions common to multiple subjects during voluntary motor tasks. We investigated the characteristics of muscle activity distribution from the data of ten healthy subjects (29 ± 1 year old, 2 women) during voluntary teeth clenching under two different occlusal conditions by applying spatial normalization and statistical parametric mapping (SPM) to analysis of muscle functional magnetic resonance imaging (mfMRI) using increase in transverse relaxation time (T2) of the skeletal muscle induced by exercise. The expansion of areas with significant T2 increase was observed in the masticatory muscles after clenching with molar loss comparing with intact dentition. The muscle activity distribution characteristics common to a group of subjects, i.e., the active region in the temporal muscle ipsilateral to the side with the molar loss and medial pterygoid muscle contralateral to the side with the molar loss, were clarified in 3D by applying spatial normalization and SPM to mfMRI analysis. This method might elucidate the functional distribution within the muscles and the localized muscular activity related to skeletal muscle disorders.

Measure Partial Liver Volumetric Variations from Paired Inspiratory-expiratory Chest CT Scans

Liver stiffness is an essential clinical biomarker for diagnosing liver fibrosis and cirrhosis. In current clinical practice, elastography techniques are standard non-invasive diagnosis tools to assess stiffness of liver, using either Ultrasound (US) or magnetic resonance imaging (MRI). However, the US elastography yields ≈ 10 % failure rate and degraded performance on obese patients, while the MR elastography is costlier and less available. Compared with US and MRI, the computerized tomography (CT) imaging has not been widely used in measuring liver stiffness. In this paper, we performed a pilot study to assess if volumetric variations of liver can be captured from paired inspiratory-expiratory chest (PIEC) CT scans. To enable the assessment, we propose a Hierarchical Intra-Patient Organ-specific (HIPO) registration pipeline to quantify the partial liver volumetric variations with lung pressure from a respiratory cycle. The PIEC protocol is employed since it naturally provides two paired CT scans with liver deformation from regulated respiratory motions. For the subjects whose registration results passed both an automatic quantitative quality assurance (QA) and another visual qualitative QA, 6.0% average volumetric variations of liver were measured, from inspiratory phase to expiratory phase. Future clinical validations will be required to validate the findings in this pilot study.

Development and Characterization of a Chest CT Atlas

A major goal of lung cancer screening is to identify individuals with particular phenotypes that are associated with high risk of cancer. Identifying relevant phenotypes is complicated by the variation in body position and body composition. In the brain, standardized coordinate systems (e.g., atlases) have enabled separate consideration of local features from gross/global structure. To date, no analogous standard atlas has been presented to enable spatial mapping and harmonization in chest computational tomography (CT). In this paper, we propose a thoracic atlas built upon a large low dose CT (LDCT) database of lung cancer screening program. The study cohort includes 466 male and 387 female subjects with no screening detected malignancy (age 46-79 years, mean 64.9 years). To provide spatial mapping, we optimize a multi-stage inter-subject non-rigid registration pipeline for the entire thoracic space. Briefly, with 50 scans of 50 randomly selected female subjects as fine tuning dataset, we search for the optimal configuration of the non-rigid registration module in a range of adjustable parameters including: registration searching radius, degree of keypoint dispersion, regularization coefficient and similarity patch size, to minimize the registration failure rate approximated by the number of samples with low Dice similarity score (DSC) for lung and body segmentation. We evaluate the optimized pipeline on a separate cohort (100 scans of 50 female and 50 male subjects) relative to two baselines with alternative non-rigid registration module: the same software with default parameters and an alternative software. We achieve a significant improvement in terms of registration success rate based on manual QA. For the entire study cohort, the optimized pipeline achieves a registration success rate of 91.7%. The application validity of the developed atlas is evaluated in terms of discriminative capability for different anatomic phenotypes, including body mass index (BMI), chronic obstructive pulmonary disease (COPD), and coronary artery calcification (CAC).

Construction of a Multi-Phase Contrast Computed Tomography Kidney Atlas

The Human BioMolecular Atlas Program (HuBMAP) seeks to create a molecular atlas at the cellular level of the human body to spur interdisciplinary innovations across spatial and temporal scales. While the preponderance of effort is allocated towards cellular and molecular scale mapping, differentiating and contextualizing findings within tissues, organs and systems are essential for the HuBMAP efforts. The kidney is an initial organ target of HuBMAP, and constructing a framework (or atlas) for integrating information across scales is needed for visualizing and integrating information. However, there is no abdominal atlas currently available in the public domain. Substantial variation in healthy kidneys exists with sex, body size, and imaging protocols. With the integration of clinical archives for secondary research use, we are able to build atlases based on a diverse population and clinically relevant protocols. In this study, we created a computed tomography (CT) phase-specific atlas for the abdomen, which is optimized for the kidney organ. A two-stage registration pipeline was used by registering extracted abdominal volume of interest from body part regression, to a high-resolution CT. Affine and non-rigid registration were performed to all scans hierarchically. To generate and evaluate the atlas, multiphase CT scans of 500 control subjects (age: 15 - 50, 250 males, 250 females) are registered to the atlas target through the complete pipeline. The abdominal body and kidney registration are shown to be stable with the variance map computed from the result average template. Both left and right kidneys are substantially localized in the high-resolution target space, which successfully demonstrated the sharp details of its anatomical characteristics across each phase. We illustrated the applicability of the atlas template for integrating across normal kidney variation from 64 cm³ to 302 cm³.

Faster dense deformable image registration by utilizing both CPU and GPU

Purpose: Image registration is an important aspect of medical image analysis and a key component in many analysis concepts. Applications include fusion of multimodal images, multi-atlas segmentation, and whole-body analysis. Deformable image registration is often computationally expensive, and the need for efficient registration methods is highlighted by the emergence of large-scale image databases, e.g., the UK Biobank, providing imaging from 100,000 participants.

Approach: We present a heterogeneous computing approach, utilizing both the CPU and the graphics processing unit (GPU), to accelerate a previously proposed image registration method. The parallelizable task of computing the matching criterion is offloaded to the GPU, where it can be computed efficiently, while the more complex optimization task is performed on the CPU. To lessen the impact of data synchronization between the CPU and GPU, we propose a pipeline model, effectively overlapping computational tasks with data synchronization. The performance is evaluated on a brain labeling task and compared with a CPU implementation of the same method and the popular advanced normalization tools (ANTs) software.

Results: The proposed method presents a speed-up by factors of 4 and 8 against the CPU implementation and the ANTs software, respectively. A significant improvement in labeling quality was also observed, with measured mean Dice overlaps of 0.712 and 0.701 for our method and ANTs, respectively.

Conclusions: We showed that the proposed method compares favorably to the ANTs software yielding both a significant speed-up and an improvement in labeling quality. The registration method together with the proposed parallelization strategy is implemented as an open-source software package, deform.

Automated Contouring of Contrast and Noncontrast Computed Tomography Liver Images With Fully Convolutional Networks

PURPOSE

The deformable nature of the liver can make focal treatment challenging and is not adequately addressed with simple rigid registration techniques. More advanced registration techniques can take deformations into account (eg, biomechanical modeling) but require segmentations of the whole liver for each scan, which is a time-intensive process. We hypothesize that fully convolutional networks can be used to rapidly and accurately autosegment the liver, removing the temporal bottleneck for biomechanical modeling.

METHODS AND MATERIALS

Manual liver segmentations on computed tomography scans from 183 patients treated at our institution and 30 scans from the Medical Image Computing & Computer Assisted Intervention challenges were collected for this study. Three architectures were investigated for rapid automated segmentation of the liver (VGG-16, DeepLabv3 +, and a 3-dimensional UNet). Fifty-six cases were set aside as a final test set for quantitative model evaluation. Accuracy of the autosegmentations was assessed using Dice similarity coefficient and mean surface distance. Qualitative evaluation was also performed by 3 radiation oncologists on 50 independent cases with previously clinically treated liver contours.

RESULTS

The mean (minimum-maximum) mean surface distance for the test groups with the final model, DeepLabv3 +, were as follows: μContrast(N = 17): 0.99 mm (0.47-2.2), μNon_Contrast(N = 19)l: 1.12 mm (0.41-2.87), and μMiccai(N = 30)t: 1.48 mm (0.82-3.96). The qualitative evaluation showed that 30 of 50 autosegmentations (60%) were preferred to manual contours (majority voting) in a blinded comparison, and 48 of 50 autosegmentations (96%) were deemed clinically acceptable by at least 1 reviewing physician.

CONCLUSIONS

The autosegmentations were preferred compared with manually defined contours in the majority of cases. The ability to rapidly segment the liver with high accuracy achieved in this investigation has the potential to enable the efficient integration of biomechanical model-based registration into a clinical workflow.