Region-adaptive Deformable Registration of CT/MRI Pelvic Images via Learning-based Image Synthesis

RTF: Recursive TransFusion for Multi-Modal Image Synthesis

Multi-modal image synthesis is crucial for obtaining complete modalities due to the imaging restrictions in reality. Current methods, primarily CNN-based models, find it challenging to extract global representations because of local inductive bias, leading to synthetic structure deformation or color distortion. Despite the significant global representation ability of transformer in capturing long-range dependencies, its huge parameter size requires considerable training data. Multi-modal synthesis solely based on one of the two structures makes it hard to extract comprehensive information from each modality with limited data. To tackle this dilemma, we propose a simple yet effective Recursive TransFusion (RTF) framework for multi-modal image synthesis. Specifically, we develop a TransFusion unit to integrate local knowledge extracted from the individual modality by connecting a CNN-based local representation block (LRB) and a transformer-based global fusion block (GFB) via a feature translating gate (FTG). Considering the numerous parameters introduced by the transformer, we further unfold a TransFusion unit with recursive constraint repeatedly, forming recursive TransFusion (RTF), which progressively extracts multi-modal information at different depths. Our RTF remarkably reduces network parameters while maintaining superior performance. Extensive experiments validate our superiority against the competing methods on multiple benchmarks. The source code will be available at https://github.com/guoliangq/RTF.

Complex Large-Deformation Multimodality Image Registration Network for Image-Guided Radiotherapy of Cervical Cancer

In recent years, image-guided brachytherapy for cervical cancer has become an important treatment method for patients with locally advanced cervical cancer, and multi-modality image registration technology is a key step in this system. However, due to the patient's own movement and other factors, the deformation between the different modalities of images is discontinuous, which brings great difficulties to the registration of pelvic computed tomography (CT/) and magnetic resonance (MR) images. In this paper, we propose a multimodality image registration network based on multistage transformation enhancement features (MTEF) to maintain the continuity of the deformation field. The model uses wavelet transform to extract different components of the image and performs fusion and enhancement processing as the input to the model. The model performs multiple registrations from local to global regions. Then, we propose a novel shared pyramid registration network that can accurately extract features from different modalities, optimizing the predicted deformation field through progressive refinement. In order to improve the registration performance, we also propose a deep learning similarity measurement method combined with bistructural morphology. On the basis of deep learning, bistructural morphology is added to the model to train the pelvic area registration evaluator, and the model can obtain parameters covering large deformation for loss function. The model was verified by the actual clinical data of cervical cancer patients. After a large number of experiments, our proposed model achieved the highest dice similarity coefficient (DSC) metric compared with the state-of-the-art registration methods. The DSC index of the MTEF algorithm is 5.64% higher than that of the TransMorph algorithm. It will effectively integrate multi-modal image information, improve the accuracy of tumor localization, and benefit more cervical cancer patients.

Detail-preserving image warping by enforcing smooth image sampling

Multi-phase dynamic contrast-enhanced magnetic resonance imaging image registration makes a substantial contribution to medical image analysis. However, existing methods (e.g., VoxelMorph, CycleMorph) often encounter the problem of image information misalignment in deformable registration tasks, posing challenges to the practical application. To address this issue, we propose a novel smooth image sampling method to align full organic information to realize detail-preserving image warping. In this paper, we clarify that the phenomenon about image information mismatch is attributed to imbalanced sampling. Then, a sampling frequency map constructed by sampling frequency estimators is utilized to instruct smooth sampling by reducing the spatial gradient and discrepancy between all-ones matrix and sampling frequency map. In addition, our estimator determines the sampling frequency of a grid voxel in the moving image by aggregating the sum of interpolation weights from warped non-grid sampling points in its vicinity and vectorially constructs sampling frequency map through projection and scatteration. We evaluate the effectiveness of our approach through experiments on two in-house datasets. The results showcase that our method preserves nearly complete details with ideal registration accuracy compared with several state-of-the-art registration methods. Additionally, our method exhibits a statistically significant difference in the regularity of the registration field compared to other methods, at a significance level of p < 0.05. Our code will be released at https://github.com/QingRui-Sha/SFM.

TAI-GAN: A Temporally and Anatomically Informed Generative Adversarial Network for early-to-late frame conversion in dynamic cardiac PET inter-frame motion correction

Inter-frame motion in dynamic cardiac positron emission tomography (PET) using rubidium-82 (82Rb) myocardial perfusion imaging impacts myocardial blood flow (MBF) quantification and the diagnosis accuracy of coronary artery diseases. However, the high cross-frame distribution variation due to rapid tracer kinetics poses a considerable challenge for inter-frame motion correction, especially for early frames where intensity-based image registration techniques often fail. To address this issue, we propose a novel method called Temporally and Anatomically Informed Generative Adversarial Network (TAI-GAN) that utilizes an all-to-one mapping to convert early frames into those with tracer distribution similar to the last reference frame. The TAI-GAN consists of a feature-wise linear modulation layer that encodes channel-wise parameters generated from temporal information and rough cardiac segmentation masks with local shifts that serve as anatomical information. Our proposed method was evaluated on a clinical 82Rb PET dataset, and the results show that our TAI-GAN can produce converted early frames with high image quality, comparable to the real reference frames. After TAI-GAN conversion, the motion estimation accuracy and subsequent myocardial blood flow (MBF) quantification with both conventional and deep learning-based motion correction methods were improved compared to using the original frames. The code is available at https://github.com/gxq1998/TAI-GAN.

TAI-GAN: Temporally and Anatomically Informed GAN for Early-to-Late Frame Conversion in Dynamic Cardiac PET Motion Correction

The rapid tracer kinetics of rubidium-82 (82Rb) and high variation of cross-frame distribution in dynamic cardiac positron emission tomography (PET) raise significant challenges for inter-frame motion correction, particularly for the early frames where conventional intensity-based image registration techniques are not applicable. Alternatively, a promising approach utilizes generative methods to handle the tracer distribution changes to assist existing registration methods. To improve frame-wise registration and parametric quantification, we propose a Temporally and Anatomically Informed Generative Adversarial Network (TAI-GAN) to transform the early frames into the late reference frame using an all-to-one mapping. Specifically, a feature-wise linear modulation layer encodes channel-wise parameters generated from temporal tracer kinetics information, and rough cardiac segmentations with local shifts serve as the anatomical information. We validated our proposed method on a clinical 82Rb PET dataset and found that our TAI-GAN can produce converted early frames with high image quality, comparable to the real reference frames. After TAI-GAN conversion, motion estimation accuracy and clinical myocardial blood flow (MBF) quantification were improved compared to using the original frames. Our code is published at https://github.com/gxq1998/TAI-GAN.

Automated Learning for Deformable Medical Image Registration by Jointly Optimizing Network Architectures and Objective Functions

Deformable image registration plays a critical role in various tasks of medical image analysis. A successful registration algorithm, either derived from conventional energy optimization or deep networks, requires tremendous efforts from computer experts to well design registration energy or to carefully tune network architectures with respect to medical data available for a given registration task/scenario. This paper proposes an automated learning registration algorithm (AutoReg) that cooperatively optimizes both architectures and their corresponding training objectives, enabling non-computer experts to conveniently find off-the-shelf registration algorithms for various registration scenarios. Specifically, we establish a triple-level framework to embrace the searching for both network architectures and objectives with a cooperating optimization. Extensive experiments on multiple volumetric datasets and various registration scenarios demonstrate that AutoReg can automatically learn an optimal deep registration network for given volumes and achieve state-of-the-art performance. The automatically learned network also improves computational efficiency over the mainstream UNet architecture from 0.558 to 0.270 seconds for a volume pair on the same configuration.

Is image-to-image translation the panacea for multimodal image registration? A comparative study

Despite current advancement in the field of biomedical image processing, propelled by the deep learning revolution, multimodal image registration, due to its several challenges, is still often performed manually by specialists. The recent success of image-to-image (I2I) translation in computer vision applications and its growing use in biomedical areas provide a tempting possibility of transforming the multimodal registration problem into a, potentially easier, monomodal one. We conduct an empirical study of the applicability of modern I2I translation methods for the task of rigid registration of multimodal biomedical and medical 2D and 3D images. We compare the performance of four Generative Adversarial Network (GAN)-based I2I translation methods and one contrastive representation learning method, subsequently combined with two representative monomodal registration methods, to judge the effectiveness of modality translation for multimodal image registration. We evaluate these method combinations on four publicly available multimodal (2D and 3D) datasets and compare with the performance of registration achieved by several well-known approaches acting directly on multimodal image data. Our results suggest that, although I2I translation may be helpful when the modalities to register are clearly correlated, registration of modalities which express distinctly different properties of the sample are not well handled by the I2I translation approach. The evaluated representation learning method, which aims to find abstract image-like representations of the information shared between the modalities, manages better, and so does the Mutual Information maximisation approach, acting directly on the original multimodal images. We share our complete experimental setup as open-source (https://github.com/MIDA-group/MultiRegEval), including method implementations, evaluation code, and all datasets, for further reproducing and benchmarking.

Multimodal MRI Reconstruction Assisted With Spatial Alignment Network

In clinical practice, multi-modal magnetic resonance imaging (MRI) with different contrasts is usually acquired in a single study to assess different properties of the same region of interest in the human body. The whole acquisition process can be accelerated by having one or more modalities under-sampled in the k -space. Recent research has shown that, considering the redundancy between different modalities, a target MRI modality under-sampled in the k -space can be more efficiently reconstructed with a fully-sampled reference MRI modality. However, we find that the performance of the aforementioned multi-modal reconstruction can be negatively affected by subtle spatial misalignment between different modalities, which is actually common in clinical practice. In this paper, we improve the quality of multi-modal reconstruction by compensating for such spatial misalignment with a spatial alignment network. First, our spatial alignment network estimates the displacement between the fully-sampled reference and the under-sampled target images, and warps the reference image accordingly. Then, the aligned fully-sampled reference image joins the multi-modal reconstruction of the under-sampled target image. Also, considering the contrast difference between the target and reference images, we have designed a cross-modality-synthesis-based registration loss in combination with the reconstruction loss, to jointly train the spatial alignment network and the reconstruction network. The experiments on both clinical MRI and multi-coil k -space raw data demonstrate the superiority and robustness of the multi-modal MRI reconstruction empowered with our spatial alignment network. Our code is publicly available at https://github.com/woxuankai/SpatialAlignmentNetwork.

Artificial intelligence in multi-parametric magnetic resonance imaging: A review

Multi-parametric magnetic resonance imaging (mpMRI) is an indispensable tool in the clinical workflow for the diagnosis and treatment planning of various diseases. Machine learning-based artificial intelligence (AI) methods, especially those adopting the deep learning technique, have been extensively employed to perform mpMRI image classification, segmentation, registration, detection, reconstruction, and super-resolution. The current availability of increasing computational power and fast-improving AI algorithms have empowered numerous computer-based systems for applying mpMRI to disease diagnosis, imaging-guided radiotherapy, patient risk and overall survival time prediction, and the development of advanced quantitative imaging technology for magnetic resonance fingerprinting. However, the wide application of these developed systems in the clinic is still limited by a number of factors, including robustness, reliability, and interpretability. This survey aims to provide an overview for new researchers in the field as well as radiologists with the hope that they can understand the general concepts, main application scenarios, and remaining challenges of AI in mpMRI. This article is protected by copyright. All rights reserved.

End to End Unsupervised Rigid Medical Image Registration by Using Convolutional Neural Networks

In this paper, we focus on the issue of rigid medical image registration using deep learning. Under ultrasound, the moving of some organs, e.g., liver and kidney, can be modeled as rigid motion. Therefore, when the ultrasound probe keeps stationary, the registration between frames can be modeled as rigid registration. We propose an unsupervised method with Convolutional Neural Networks. The network estimates from the input image pair the transform parameters first then the moving image is wrapped using the parameters. The loss is calculated between the registered image and the fixed image. Experiments on ultrasound data of kidney and liver verified that the method is capable of achieve higher accuracy compared with traditional methods and is much faster.

A Coarse-to-Fine Deformable Transformation Framework for Unsupervised Multi-Contrast MR Image Registration with Dual Consistency Constraint

Multi-contrast magnetic resonance (MR) image registration is useful in the clinic to achieve fast and accurate imaging-based disease diagnosis and treatment planning. Nevertheless, the efficiency and performance of the existing registration algorithms can still be improved. In this paper, we propose a novel unsupervised learning-based framework to achieve accurate and efficient multi-contrast MR image registration. Specifically, an end-to-end coarse-to-fine network architecture consisting of affine and deformable transformations is designed to improve the robustness and achieve end-to-end registration. Furthermore, a dual consistency constraint and a new prior knowledge-based loss function are developed to enhance the registration performances. The proposed method has been evaluated on a clinical dataset containing 555 cases, and encouraging performances have been achieved. Compared to the commonly utilized registration methods, including VoxelMorph, SyN, and LT-Net, the proposed method achieves better registration performance with a Dice score of 0.8397± 0.0756 in identifying stroke lesions. With regards to the registration speed, our method is about 10 times faster than the most competitive method of SyN (Affine) when testing on a CPU. Moreover, we prove that our method can still perform well on more challenging tasks with lacking scanning information data, showing the high robustness for the clinical application.

Segmenting Vitiligo on Clinical Face Images Using CNN Trained on Synthetic and Internet Images

Accurately diagnosing and describing the severity of vitiligo is crucial for prognostication, treatment selection and comparison. Currently, disease severity scores require dermatologists to estimate percentage area of involvement, which is subjected to inter and intra-assessor variability. Previous studies focus on pure skin but vitiligo on the face, which has a more serious impact on patients' quality of life, was completely neglected. Convolutional neural networks (CNNs) have good performance on many segmentation tasks. However, due to data privacy, it is hard to have a large clinical vitiligo face image dataset to train a CNN. To address this challenge, images from two different sources, the Internet and the proposed vitiligo face synthesis algorithm, are employed in training. 843 vitiligo images taken from different viewpoints were collected from the Internet. These images are hugely different from the target clinical images collected according to a newly established international standard. To have more vitiligo face images similar to the target clinical images to enhance segmentation performance, an image synthesis algorithm is proposed. Both synthetic and Internet images are used to train a CNN which is modified from the fully convolutional network (FCN) to segment face vitiligo lesions. The results show that 1) the synthetic images effectively improve segmentation performance; 2) the proposed algorithm achieves 1.06 % error for the face vitiligo area estimation and 3) it is more accurate than two dermatologists and all the previous automated vitiligo segmentation methods, which were designed for segmentation vitiligo on pure skin.

Multi-Task Learning for Registering Images With Large Deformation

Accurate registration of prostate magnetic resonance imaging (MRI) images of the same subject acquired at different time points helps diagnose cancer and monitor the tumor progress. However, it is very challenging especially when one image was acquired with the use of endorectal coil (ERC) but the other was not, which causes significant deformation. Classical iterative image registration methods are also computationally intensive. Deep learning based registration frameworks have recently been developed and demonstrated promising performance. However, the lack of proper constraints often results in unrealistic registration. In this paper, we propose a multi-task learning based registration network with anatomical constraint to address these issues. The proposed approach uses a cycle constraint loss to achieve forward/backward registration and an inverse constraint loss to encourage diffeomorphic registration. In addition, an adaptive anatomical constraint aiming for regularizing the registration network with the use of anatomical labels is introduced through weak supervision. Our experiments on registering prostate MR images of the same subject obtained at different time points with and without ERC show that the proposed method achieves very promising performance under different measures in dealing with the large deformation. Compared with other existing methods, our approach works more efficiently with average running time less than a second and is able to obtain more visually realistic results.

Research on obtaining pseudo CT images based on stacked generative adversarial network

Background

To investigate the feasibility of using a stacked generative adversarial network (sGAN) to synthesize pseudo computed tomography (CT) images based on ultrasound (US) images.

Methods

The pre-radiotherapy US and CT images of 75 patients with cervical cancer were selected for the training set of pseudo-image synthesis. In the first stage, labeled US images were used as the first conditional GAN input to obtain low-resolution pseudo CT images, and in the second stage, a super-resolution reconstruction GAN was used. The pseudo CT image obtained in the first stage was used as an input, following which a high-resolution pseudo CT image with clear texture and accurate grayscale information was obtained. Five cross validation tests were performed to verify our model. The mean absolute error (MAE) was used to compare each pseudo CT with the same patient's real CT image. Also, another 10 cases of patients with cervical cancer, before radiotherapy, were selected for testing, and the pseudo CT image obtained using the neural style transfer (NSF) and CycleGAN methods were compared with that obtained using the sGAN method proposed in this study. Finally, the dosimetric accuracy of pseudo CT images was verified by phantom experiments.

Results

The MAE metric values between the pseudo CT obtained based on sGAN, and the real CT in five-fold cross validation are 66.82±1.59 HU, 66.36±1.85 HU, 67.26±2.37 HU, 66.34±1.75 HU, and 67.22±1.30 HU, respectively. The results of the metrics, namely, normalized mutual information (NMI), structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR), between the pseudo CT images obtained using the sGAN method and the ground truth CT (CTgt) images were compared with those of the other two methods via the paired t-test, and the differences were statistically significant. The dice similarity coefficient (DSC) measurement results showed that the pseudo CT images obtained using the sGAN method were more similar to the CTgt images of organs at risk. The dosimetric phantom experiments also showed that the dose distribution between the pseudo CT images synthesized by the new method was similar to that of the CTgt images.

Conclusions

Compared with NSF and CycleGAN methods, the sGAN method can obtain more accurate pseudo CT images, thereby providing a new method for image guidance in radiotherapy for cervical cancer.

Pelvic Reconstruction With a Novel Three-Dimensional-Printed, Multimodality Imaging Based Endoprosthesis Following Enneking Type I + IV Resection

BACKGROUND AND PURPOSE

Pelvic tumor involving Type I + IV resections are technically challenging, along with various reconstructions methods presenting unsatisfactory outcomes and high complication rates. Since predominating studies preferred adopting pedicle screw-rod system (PRSS) to address this issue, we designed a novel three-dimensional-printed, multimodality imaging (3DMMI) based endoprosthesis with patient-specific instrument (PSI) assistance to facilitate the surgical reconstruction of pelvic tumor involving Enneking Type I + IV resection. We aimed to investigate the clinical effectiveness of this novel endoprosthesis and compare it with PRSS in Type I + IV reconstruction.

METHODS

We retrospective studied 28 patients for a median follow-up of 47 months (range, 10 to 128 months) in this study with either 3D-printed endoprosthesis reconstruction (n = 10) or PRSS reconstruction (n = 18) between January 2000 and December 2017. Preoperative 3DMMI technique was used for tumor evaluation, PSI design, virtual surgery, and endoprosthesis fabrication. Clinical, oncological outcomes, functional assessments, and complications were analyzed between the two groups.

RESULTS

Minor surgical trauma with mean operative duration of 251 ± 52.16 minutes (p = 0.034) and median intraoperative hemorrhage of 2000ml (range, 1600, 4000ml) (p = 0.032) was observed in endoprosthesis group. Wide margins were achieved in 9 patients of the endoprosthesis group compared with 10 in the PRSS group (p = 0.09). The 1993 version of the Musculoskeletal Tumor Society score (MSTS-93) was 23.9 ± 3.76 in endoprosthesis group, which was higher than PRSS group (p = 0.012). No statistical significance was found in relapse between two groups (p = 0.36). Complications were observed in two patients in endoprosthesis group compared with 12 patients in PRSS group (p = 0.046).

CONCLUSION

The novel design of this 3D-printed endoprosthesis, together with 3DMMI and PSI assisted, is technically accessible with favorable clinical outcomes compared with PRSS. Further study is essential to identify its long-term outcomes.

Deep-learning based multi-modal retinal image registration for the longitudinal analysis of patients with age-related macular degeneration

This work reports a deep-learning based registration algorithm that aligns multi-modal retinal images collected from longitudinal clinical studies to achieve accuracy and robustness required for analysis of structural changes in large-scale clinical data. Deep-learning networks that mirror the architecture of conventional feature-point-based registration were evaluated with different networks that solved for registration affine parameters, image patch displacements, and patch displacements within the region of overlap. The ground truth images for deep learning-based approaches were derived from successful conventional feature-based registration. Cross-sectional and longitudinal affine registrations were performed across color fundus photography (CFP), fundus autofluorescence (FAF), and infrared reflectance (IR) image modalities. For mono-modality longitudinal registration, the conventional feature-based registration method achieved mean errors in the range of 39-53 µm (depending on the modality) whereas the deep learning method with region overlap prediction exhibited mean errors in the range 54-59 µm. For cross-sectional multi-modality registration, the conventional method exhibited gross failures with large errors in more than 50% of the cases while the proposed deep-learning method achieved robust performance with no gross failures and mean errors in the range 66-69 µm. Thus, the deep learning-based method achieved superior overall performance across all modalities. The accuracy and robustness reported in this work provide important advances that will facilitate clinical research and enable a detailed study of the progression of retinal diseases such as age-related macular degeneration.

Deformable MR-CBCT prostate registration using biomechanically constrained deep learning networks

BACKGROUND AND PURPOSE

Radiotherapeutic dose escalation to dominant intraprostatic lesions (DIL) in prostate cancer could potentially improve tumor control. The purpose of this study was to develop a method to accurately register multiparametric magnetic resonance imaging (MRI) with CBCT images for improved DIL delineation, treatment planning, and dose monitoring in prostate radiotherapy.

METHODS AND MATERIALS

We proposed a novel registration framework which considers biomechanical constraint when deforming the MR to CBCT. The registration framework consists of two segmentation convolutional neural networks (CNN) for MR and CBCT prostate segmentation, and a three-dimensional (3D) point cloud (PC) matching network. Image intensity-based rigid registration was first performed to initialize the alignment between MR and CBCT prostate. The aligned prostates were then meshed into tetrahedron elements to generate volumetric PC representation of the prostate shapes. The 3D PC matching network was developed to predict a PC motion vector field which can deform the MRI prostate PC to match the CBCT prostate PC. To regularize the network's motion prediction with biomechanical constraints, finite element (FE) modeling-generated motion fields were used to train the network. MRI and CBCT images of 50 patients with intraprostatic fiducial markers were used in this study. Registration results were evaluated using three metrics including dice similarity coefficient (DSC), mean surface distance (MSD), and target registration error (TRE). In addition to spatial registration accuracy, Jacobian determinant and strain tensors were calculated to assess the physical fidelity of the deformation field.

RESULTS

The mean and standard deviation of our method were 0.93 ± 0.01, 1.66 ± 0.10 mm, and 2.68 ± 1.91 mm for DSC, MSD, and TRE, respectively. The mean TRE of the proposed method was reduced by 29.1%, 14.3%, and 11.6% as compared to image intensity-based rigid registration, coherent point drifting (CPD) nonrigid surface registration, and modality-independent neighborhood descriptor (MIND) registration, respectively.

CONCLUSION

We developed a new framework to accurately register the prostate on MRI to CBCT images for external beam radiotherapy. The proposed method could be used to aid DIL delineation on CBCT, treatment planning, dose escalation to DIL, and dose monitoring.

Artificial Intelligence (AI) Assisted CT/MRI Image Fusion Technique in Preoperative Evaluation of a Pelvic Bone Osteosarcoma

Surgeries of pelvic bone tumors are very challenging due to the complexity of anatomical structures and the irregular bone shape. CT and MRI are used in clinic for tumor evaluation, each with its own advantages and shortcomings. Combining the data of both CT and MRI images would take advantage of the merits of both images and provide better model for preoperative evaluation. We utilized an artificial intelligence (AI)-assisted CT/MRI image fusion technique and built a personalized 3-D model for preoperative tumor margin assessment. A young female patient with pelvic osteosarcoma was evaluated with our novel image fusion 3-D model in comparison with the 3-D model based solely on CT images. The fusion image model showed more detailed anatomical information and discovered multiple emboli within veins which were previously neglected. The discovery of emboli implied abysmal prognosis and discouraged any attempts for complex reconstruction after tumor resection. Based on the experience with this pelvic osteosarcoma, we believe that our image fusion model can be very informative with bone tumors. Though further validation with a large number of clinical cases is required, we propose that our model has the potential to benefit the clinic in the preoperative evaluation of bone tumors.

Magnetic resonance image (MRI) synthesis from brain computed tomography (CT) images based on deep learning methods for magnetic resonance (MR)-guided radiotherapy

Background

Precise patient setup is critical in radiation therapy. Medical imaging plays an essential role in patient setup. As compared to computed tomography (CT) images, magnetic resonance image (MRI) has high contrast for soft tissues, which becomes a promising imaging modality during treatment. In this paper, we proposed a method to synthesize brain MRI images from corresponding planning CT (pCT) images. The synthetic MRI (sMRI) images can be used to align with positioning MRI (pMRI) equipped by an MRI-guided accelerator to account for the disadvantages of multi-modality image registration.

Methods

Several deep learning network models were applied to implement this brain MRI synthesis task, including CycleGAN, Pix2Pix model, and U-Net. We evaluated these methods using several metrics, including mean absolute error (MAE), mean squared error (MSE), structural similarity index (SSIM), and peak signal-to-noise ratio (PSNR).

Results

In our experiments, U-Net with L1+L2 loss achieved the best results with the lowest overall average MAE of 74.19 and MSE of 1.035*10⁴, respectively, and produced the highest SSIM of 0.9440 and PSNR of 32.44.

Conclusions

Quantitative comparisons suggest that the performance of U-Net, a supervised deep learning method, is better than the performance of CycleGAN, a typical unsupervised method, in our brain MRI synthesis procedure. The proposed method can convert pCT/pMRI multi-modality registration into mono-modality registration, which can be used to reduce registration error and achieve a more accurate patient setup.

Synthesized b0 for diffusion distortion correction (Synb0-DisCo)

Diffusion magnetic resonance images typically suffer from spatial distortions due to susceptibility induced off-resonance fields, which may affect the geometric fidelity of the reconstructed volume and cause mismatches with anatomical images. State-of-the art susceptibility correction (for example, FSL's TOPUP algorithm) typically requires data acquired twice with reverse phase encoding directions, referred to as blip-up blip-down acquisitions, in order to estimate an undistorted volume. Unfortunately, not all imaging protocols include a blip-up blip-down acquisition, and cannot take advantage of the state-of-the art susceptibility and motion correction capabilities. In this study, we aim to enable TOPUP-like processing with historical and/or limited diffusion imaging data that include only a structural image and single blip diffusion image. We utilize deep learning to synthesize an undistorted non-diffusion weighted image from the structural image, and use the non-distorted synthetic image as an anatomical target for distortion correction. We evaluate the efficacy of this approach (named Synb0-DisCo) and show that our distortion correction process results in better matching of the geometry of undistorted anatomical images, reduces variation in diffusion modeling, and is practically equivalent to having both blip-up and blip-down non-diffusion weighted images.

Improved virtual surgical planning with 3D- multimodality image for malignant giant pelvic tumors

PURPOSE

We sought to assess the early clinical outcome of 3D-multimodality image (3DMMI)-based virtual surgical planning for resection and reconstruction of malignant giant pelvic tumors.

PATIENTS AND METHODS

In this retrospective case-control study, surgery was planned and performed with 3DMMI-based patient-specific instruments (PSI) in 13 patients with giant pelvic malignancy and without 3DMMI-based PSI in the other 13 patients. In the 3DMMI group, 3DMMI was utilized, taking advantages of computed tomography (CT), contrast-enhanced CT angiography (CTA), contrast-enhanced magnetic resonance imaging (MRI), contrast-enhanced magnetic resonance neurography (MRN), which could reveal the whole tumor and all adjacent vital structures. Based on these 3DMMI, virtual surgical planning was conducted and the corresponding PSI was then designed. The median follow-up was 8 (3-24) months. The median age at operation was 37.5 (17-64) years. The mean tumor size in maximum diameter was 13.3 cm. Surgical margins, intraoperative and postoperative complications, duration of surgery, and intra-operative blood loss were analyzed.

RESULTS

In the non-3DMMI group, the margins were wide in six patients (6/13), marginal in four (4/13), wide-contaminated in two (2/13), and intralesional in one (1/13). In the 3DMMI group, the margins were wide in 10 patients (10/13), marginal in three (3/13), and there were no wide-contaminated or intralesional margins. The 3DMMI group achieved shorter duration of surgery (P=0.354) and lower intraoperative blood loss (P=0.044) than the non-3DMMI group. Conclusion: The 3DMMI-based technique is advantageous to obtain negative surgical margin and decrease surgical complications related to critical structures injury for malignant giant pelvic tumor.

Deformable Image Registration Using a Cue-Aware Deep Regression Network

SIGNIFICANCE

Analysis of modern large-scale, multicenter or diseased data requires deformable registration algorithms that can cope with data of diverse nature.

OBJECTIVE

We propose a novel deformable registration method, which is based on a cue-aware deep regression network, to deal with multiple databases with minimal parameter tuning.

METHODS

Our method learns and predicts the deformation field between a reference image and a subject image. Specifically, given a set of training images, our method learns the displacement vector associated with a pair of reference-subject patches. To achieve this, we first introduce a key-point truncated-balanced sampling strategy to facilitate accurate learning from the image database of limited size. Then, we design a cue-aware deep regression network, where we propose to employ the contextual cue, i.e., the scale-adaptive local similarity, to more apparently guide the learning process. The deep regression network is aware of the contextual cue for accurate prediction of local deformation.

RESULTS AND CONCLUSION

Our experiments show that the proposed method can tackle various registration tasks on different databases, giving consistent good performance without the need of manual parameter tuning, which could be applicable to various clinical applications.

Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity

Non-rigid inter-modality registration can facilitate accurate information fusion from different modalities, but it is challenging due to the very different image appearances across modalities. In this paper, we propose to train a non-rigid inter-modality image registration network, which can directly predict the transformation field from the input multimodal images, such as CT and MR images. In particular, the training of our inter-modality registration network is supervised by intra-modality similarity metric based on the available paired data, which is derived from a pre-aligned CT and MR dataset. Specifically, in the training stage, to register the input CT and MR images, their similarity is evaluated on the warped MR image and the MR image that is paired with the input CT. So that, the intra-modality similarity metric can be directly applied to measure whether the input CT and MR images are well registered. Moreover, we use the idea of dual-modality fashion, in which we measure the similarity on both CT modality and MR modality. In this way, the complementary anatomies in both modalities can be jointly considered to more accurately train the inter-modality registration network. In the testing stage, the trained inter-modality registration network can be directly applied to register the new multimodal images without any paired data. Experimental results have shown that, the proposed method can achieve promising accuracy and efficiency for the challenging non-rigid inter-modality registration task and also outperforms the state-of-the-art approaches.

Structurally-sensitive Multi-scale Deep Neural Network for Low-Dose CT Denoising

Computed tomography (CT) is a popular medical imaging modality and enjoys wide clinical applications. At the same time, the x-ray radiation dose associated with CT scannings raises a public concern due to its potential risks to the patients. Over the past years, major efforts have been dedicated to the development of Low-Dose CT (LDCT) methods. However, the radiation dose reduction compromises the signal-to-noise ratio (SNR), leading to strong noise and artifacts that downgrade CT image quality. In this paper, we propose a novel 3D noise reduction method, called Structurally-sensitive Multi-scale Generative Adversarial Net (SMGAN), to improve the LDCT image quality. Specifically, we incorporate three-dimensional (3D) volumetric information to improve the image quality. Also, different loss functions for training denoising models are investigated. Experiments show that the proposed method can effectively preserve structural and textural information in reference to normal-dose CT (NDCT) images, and significantly suppress noise and artifacts. Qualitative visual assessments by three experienced radiologists demonstrate that the proposed method retrieves more information, and outperforms competing methods.