One-Shot Learning for Deformable Medical Image Registration and Periodic Motion Tracking

Encoding matching criteria for cross-domain deformable image registration

BACKGROUND

Most existing deep learning-based registration methods are trained on single-type images to address same-domain tasks, resulting in performance degradation when applied to new scenarios. Retraining a model for new scenarios requires extra time and data. Therefore, efficient and accurate solutions for cross-domain deformable registration are in demand.

PURPOSE

We argue that the tailor-made matching criteria in traditional registration methods is one of the main reason they are applicable in different domains. Motivated by this, we devise a registration-oriented encoder to model the matching criteria of image features and structural features, which is beneficial to boost registration accuracy and adaptability.

METHODS

Specifically, a general feature encoder (Encoder-G) is proposed to capture comprehensive medical image features, while a structural feature encoder (Encoder-S) is designed to encode the structural self-similarity into the global representation. Moreover, by updating Encoder-S using one-shot learning, our method can effectively adapt to different domains. The efficacy of our method is evaluated using MRI images from three different domains, including brain images (training/testing: 870/90 pairs), abdomen images (training/testing: 1406/90 pairs), and cardiac images (training/testing: 64770/870 pairs). The comparison methods include traditional method (SyN) and cutting-edge deep networks. The evaluation metrics contain dice similarity coefficient (DSC) and average symmetric surface distance (ASSD).

RESULTS

In the single-domain task, our method attains an average DSC of 68.9%/65.2%/72.8%, and ASSD of 9.75/3.82/1.30 mm on abdomen/cardiac/brain images, outperforming the second-best comparison methods by large margins. In the cross-domain task, without one-shot optimization, our method outperforms other deep networks in five out of six cross-domain scenarios and even surpasses symmetric image normalization method (SyN) in two scenarios. By conducting the one-shot optimization, our method successfully surpasses SyN in all six cross-domain scenarios.

CONCLUSIONS

Our method yields favorable results in the single-domain task while ensuring improved generalization and adaptation performance in the cross-domain task, showing its feasibility for the challenging cross-domain registration applications. The code is publicly available at https://github.com/JuliusWang-7/EncoderReg.

A survey on deep learning in medical image registration: New technologies, uncertainty, evaluation metrics, and beyond

Deep learning technologies have dramatically reshaped the field of medical image registration over the past decade. The initial developments, such as regression-based and U-Net-based networks, established the foundation for deep learning in image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, network architectures, and uncertainty estimation. These advancements have not only enriched the field of image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration.

Pulmonary CT Registration Network Based on Deformable Cross Attention

Current Transformer structure utilizes the self-attention mechanism to model global contextual relevance within image, which makes an impact on medical image registration. However, the use of Transformer in handling large deformation lung CT registration is relatively straightforwardly. These models only focus on single image feature representation neglecting to employ attention mechanism to capture the across image correspondence. This hinders further improvement in registration performance. To address the above limitations, we propose a novel registration method in a cascaded manner, Cascaded Swin Deformable Cross Attention Transformer based U-shape structure (SD-CATU), to address the challenge of large deformations in lung CT registration. In SD-CATU, we introduce a Cross Attention-based Transformer (CAT) block that incorporates the Shifted Regions Multihead Cross-attention (SR-MCA) mechanism to flexibly exchange feature information and thus reduce the computational complexity. Besides, a consistency constraint in the loss function is used to ensure the preservation of topology and inverse consistency of the transformations. Experiments with public lung datasets demonstrate that the Cascaded SD-CATU outperforms current state-of-the-art registration methods (Dice Similarity Coefficient of 93.19% and Target registration error of 0.98 mm). The results further highlight the potential for obtaining excellent registration accuracy while assuring desirable smoothness and consistency in the deformed images.

CCMNet: Cross-scale correlation-aware mapping network for 3D lung CT image registration

The lung is characterized by high elasticity and complex structure, which implies that the lung is capable of undergoing complex deformation and the shape variable is substantial. Large deformation estimation poses significant challenges to lung image registration. The traditional U-Net architecture is difficult to cover complex deformation due to its limited receptive field. Moreover, the relationship between voxels weakens as the number of downsampling times increases, that is, the long-range dependence issue. In this paper, we propose a novel multilevel registration framework which enhances the correspondence between voxels to improve the ability of estimating large deformations. Our approach consists of a convolutional neural network (CNN) with a two-stream registration structure and a cross-scale mapping attention (CSMA) mechanism. The former extracts the robust features of image pairs within layers, while the latter establishes frequent connections between layers to maintain the correlation of image pairs. This method fully utilizes the context information of different scales to establish the mapping relationship between low-resolution and high-resolution feature maps. We have achieved remarkable results on DIRLAB (TRE 1.56 ± 1.60) and POPI (NCC 99.72% SSIM 91.42%) dataset, demonstrating that this strategy can effectively address the large deformation issues, mitigate long-range dependence, and ultimately achieve more robust lung CT image registration.

A systematic review of few-shot learning in medical imaging

The lack of annotated medical images limits the performance of deep learning models, which usually need large-scale labelled datasets. Few-shot learning techniques can reduce data scarcity issues and enhance medical image analysis speed and robustness. This systematic review gives a comprehensive overview of few-shot learning methods for medical image analysis, aiming to establish a standard methodological pipeline for future research reference. With a particular emphasis on the role of meta-learning, we analysed 80 relevant articles published from 2018 to 2023, conducting a risk of bias assessment and extracting relevant information, especially regarding the employed learning techniques. From this, we delineated a comprehensive methodological pipeline shared among all studies. In addition, we performed a statistical analysis of the studies' results concerning the clinical task and the meta-learning method employed while also presenting supplemental information such as imaging modalities and model robustness evaluation techniques. We discussed the findings of our analysis, providing a deep insight into the limitations of the state-of-the-art methods and the most promising approaches. Drawing on our investigation, we yielded recommendations on potential future research directions aiming to bridge the gap between research and clinical practice.

Independently Trained Multi-Scale Registration Network Based on Image Pyramid

Image registration is a fundamental task in various applications of medical image analysis and plays a crucial role in auxiliary diagnosis, treatment, and surgical navigation. However, cardiac image registration is challenging due to the large non-rigid deformation of the heart and the complex anatomical structure. To address this challenge, this paper proposes an independently trained multi-scale registration network based on an image pyramid. By down-sampling the original input image multiple times, we can construct image pyramid pairs, and design a multi-scale registration network using image pyramid pairs of different resolutions as the training set. Using image pairs of different resolutions, train each registration network independently to extract image features from the image pairs at different resolutions. During the testing stage, the large deformation registration is decomposed into a multi-scale registration process. The deformation fields of different resolutions are fused by a step-by-step deformation method, thereby addressing the challenge of directly handling large deformations. Experiments were conducted on the open cardiac dataset ACDC (Automated Cardiac Diagnosis Challenge); the proposed method achieved an average Dice score of 0.828 in the experimental results. Through comparative experiments, it has been demonstrated that the proposed method effectively addressed the challenge of heart image registration and achieved superior registration results for cardiac images.

MACG-Net: Multi-axis cross gating network for deformable medical image registration

Deformable Image registration is a fundamental yet vital task for preoperative planning, intraoperative information fusion, disease diagnosis and follow-ups. It solves the non-rigid deformation field to align an image pair. Latest approaches such as VoxelMorph and TransMorph compute features from a simple concatenation of moving and fixed images. However, this often leads to weak alignment. Moreover, the convolutional neural network (CNN) or the hybrid CNN-Transformer based backbones are constrained to have limited sizes of receptive field and cannot capture long range relations while full Transformer based approaches are computational expensive. In this paper, we propose a novel multi-axis cross grating network (MACG-Net) for deformable medical image registration, which combats these limitations. MACG-Net uses a dual stream multi-axis feature fusion module to capture both long-range and local context relationships from the moving and fixed images. Cross gate blocks are integrated with the dual stream backbone to consider both independent feature extractions in the moving-fixed image pair and the relationship between features from the image pair. We benchmark our method on several different datasets including 3D atlas-based brain MRI, inter-patient brain MRI and 2D cardiac MRI. The results demonstrate that the proposed method has achieved state-of-the-art performance. The source code has been released at https://github.com/Valeyards/MACG.

Stop moving: MR motion correction as an opportunity for artificial intelligence

Subject motion is a long-standing problem of magnetic resonance imaging (MRI), which can seriously deteriorate the image quality. Various prospective and retrospective methods have been proposed for MRI motion correction, among which deep learning approaches have achieved state-of-the-art motion correction performance. This survey paper aims to provide a comprehensive review of deep learning-based MRI motion correction methods. Neural networks used for motion artifacts reduction and motion estimation in the image domain or frequency domain are detailed. Furthermore, besides motion-corrected MRI reconstruction, how estimated motion is applied in other downstream tasks is briefly introduced, aiming to strengthen the interaction between different research areas. Finally, we identify current limitations and point out future directions of deep learning-based MRI motion correction.

Structure-aware independently trained multi-scale registration network for cardiac images

Image registration is a primary task in various medical image analysis applications. However, cardiac image registration is difficult due to the large non-rigid deformation of the heart and the complex anatomical structure. This paper proposes a structure-aware independently trained multi-scale registration network (SIMReg) to address this challenge. Using image pairs of different resolutions, independently train each registration network to extract image features of large deformation image pairs at different resolutions. In the testing stage, the large deformation registration is decomposed into a multi-scale registration process, and the deformation fields of different resolutions are fused by a step-by-step deformation method, thus solving the difficulty of directly processing large deformation. Meanwhile, the targeted introduction of MIND (modality independent neighborhood descriptor) structural features to guide network training enhances the registration of cardiac structural contours and improves the registration effect of local details. Experiments were carried out on the open cardiac dataset ACDC (automated cardiac diagnosis challenge), and the average Dice value of the experimental results of the proposed method was 0.833. Comparative experiments showed that the proposed SIMReg could better solve the problem of heart image registration and achieve a better registration effect on cardiac images.

The Role of AI in Hospitals and Clinics: Transforming Healthcare in the 21st Century

As healthcare systems around the world face challenges such as escalating costs, limited access, and growing demand for personalized care, artificial intelligence (AI) is emerging as a key force for transformation. This review is motivated by the urgent need to harness AI's potential to mitigate these issues and aims to critically assess AI's integration in different healthcare domains. We explore how AI empowers clinical decision-making, optimizes hospital operation and management, refines medical image analysis, and revolutionizes patient care and monitoring through AI-powered wearables. Through several case studies, we review how AI has transformed specific healthcare domains and discuss the remaining challenges and possible solutions. Additionally, we will discuss methodologies for assessing AI healthcare solutions, ethical challenges of AI deployment, and the importance of data privacy and bias mitigation for responsible technology use. By presenting a critical assessment of AI's transformative potential, this review equips researchers with a deeper understanding of AI's current and future impact on healthcare. It encourages an interdisciplinary dialogue between researchers, clinicians, and technologists to navigate the complexities of AI implementation, fostering the development of AI-driven solutions that prioritize ethical standards, equity, and a patient-centered approach.

Multi-scale V-net architecture with deep feature CRF layers for brain extraction

BACKGROUND

Brain extraction is a computational necessity for researchers using brain imaging data. However, the complex structure of the interfaces between the brain, meninges and human skull have not allowed a highly robust solution to emerge. While previous methods have used machine learning with structural and geometric priors in mind, with the development of Deep Learning (DL), there has been an increase in Neural Network based methods. Most proposed DL models focus on improving the training data despite the clear gap between groups in the amount and quality of accessible training data between.

METHODS

We propose an architecture we call Efficient V-net with Additional Conditional Random Field Layers (EVAC+). EVAC+ has 3 major characteristics: (1) a smart augmentation strategy that improves training efficiency, (2) a unique way of using a Conditional Random Fields Recurrent Layer that improves accuracy and (3) an additional loss function that fine-tunes the segmentation output. We compare our model to state-of-the-art non-DL and DL methods.

RESULTS

Results show that even with limited training resources, EVAC+ outperforms in most cases, achieving a high and stable Dice Coefficient and Jaccard Index along with a desirable lower Surface (Hausdorff) Distance. More importantly, our approach accurately segmented clinical and pediatric data, despite the fact that the training dataset only contains healthy adults.

CONCLUSIONS

Ultimately, our model provides a reliable way of accurately reducing segmentation errors in complex multi-tissue interfacing areas of the brain. We expect our method, which is publicly available and open-source, to be beneficial to a wide range of researchers.

Uncertainty-aware network for fine-grained and imbalanced reflux esophagitis grading

Computer-aided diagnosis (CAD) assists endoscopists in analyzing endoscopic images, reducing misdiagnosis rates and enabling timely treatment. A few studies have focused on CAD for gastroesophageal reflux disease, but CAD studies on reflux esophagitis (RE) are still inadequate. This paper presents a CAD study on RE using a dataset collected from hospital, comprising over 3000 images. We propose an uncertainty-aware network with handcrafted features, utilizing representation and classifier decoupling with metric learning to address class imbalance and achieve fine-grained RE classification. To enhance interpretability, the network estimates uncertainty through test time augmentation. The experimental results demonstrate that the proposed network surpasses previous methods, achieving an accuracy of 90.2% and an F1 score of 90.1%.

ORRN: An ODE-Based Recursive Registration Network for Deformable Respiratory Motion Estimation With Lung 4DCT Images

OBJECTIVE

Deformable Image Registration (DIR) plays a significant role in quantifying deformation in medical data. Recent Deep Learning methods have shown promising accuracy and speedup for registering a pair of medical images. However, in 4D (3D + time) medical data, organ motion, such as respiratory motion and heart beating, can not be effectively modeled by pair-wise methods as they were optimized for image pairs but did not consider the organ motion patterns necessary when considering 4D data.

METHODS

This article presents ORRN, an Ordinary Differential Equations (ODE)-based recursive image registration network. Our network learns to estimate time-varying voxel velocities for an ODE that models deformation in 4D image data. It adopts a recursive registration strategy to progressively estimate a deformation field through ODE integration of voxel velocities.

RESULTS

We evaluate the proposed method on two publicly available lung 4DCT datasets, DIRLab and CREATIS, for two tasks: 1) registering all images to the extreme inhale image for 3D+t deformation tracking and 2) registering extreme exhale to inhale phase images. Our method outperforms other learning-based methods in both tasks, producing the smallest Target Registration Error of 1.24 mm and 1.26 mm, respectively. Additionally, it produces less than 0.001% unrealistic image folding, and the computation speed is less than 1 s for each CT volume.

CONCLUSION

ORRN demonstrates promising registration accuracy, deformation plausibility, and computation efficiency on group-wise and pair-wise registration tasks.

SIGNIFICANCE

It has significant implications in enabling fast and accurate respiratory motion estimation for treatment planning in radiation therapy or robot motion planning in thoracic needle insertion.

A multi-view assisted registration network for MRI registration pre- and post-therapy

Image registration of magnetic resonance imaging (MRI) pre- and post-therapy is an important part of evaluating the effect of therapy in tumor patients. The accuracy of evaluation results heavily relies on the alignment of the MRI image after registration. Although recent advancements have been made in medical image registration, applying these methods to MRI registration pre- and post-therapy remains challenging. Existing methods typically utilize single-view data for registration. However, when applied to MRI data where some slices are clear while others are blurred, these methods can be misled by erroneous spatial information in the blurred regions, leading to poor registration outcomes. To mitigate the interference caused by erroneous spatial information in single-view data, this paper proposes a multi-stream fusion-assisted registration network that incorporates different-view MRIs of the same patient at the same site. Additionally, a cross-attention guided fusion module is designed within the network to effectively utilize accurate spatial information from multi-view data. The proposed approach was evaluated on clinical data, and the experimental results demonstrated that incorporating multiple view data as auxiliary information significantly enhances the accuracy of MRI image registration before and after radiotherapy.

Unsupervised deep learning registration model for multimodal brain images

Multimodal image registration is a key for many clinical image-guided interventions. However, it is a challenging task because of complicated and unknown relationships between different modalities. Currently, deep supervised learning is the state-of-theart method at which the registration is conducted in end-to-end manner and one-shot. Therefore, a huge ground-truth data is required to improve the results of deep neural networks for registration. Moreover, supervised methods may yield models that bias towards annotated structures. Here, to deal with above challenges, an alternative approach is using unsupervised learning models. In this study, we have designed a novel deep unsupervised Convolutional Neural Network (CNN)-based model based on computer tomography/magnetic resonance (CT/MR) co-registration of brain images in an affine manner. For this purpose, we created a dataset consisting of 1100 pairs of CT/MR slices from the brain of 110 neuropsychic patients with/without tumor. At the next step, 12 landmarks were selected by a well-experienced radiologist and annotated on each slice resulting in the computation of series of metrics evaluation, target registration error (TRE), Dice similarity, Hausdorff, and Jaccard coefficients. The proposed method could register the multimodal images with TRE 9.89, Dice similarity 0.79, Hausdorff 7.15, and Jaccard 0.75 that are appreciable for clinical applications. Moreover, the approach registered the images in an acceptable time 203 ms and can be appreciable for clinical usage due to the short registration time and high accuracy. Here, the results illustrated that our proposed method achieved competitive performance against other related approaches from both reasonable computation time and the metrics evaluation.

Deep learning-based lung image registration: A review

Lung image registration can effectively describe the relative motion of lung tissues, thereby helping to solve series problems in clinical applications. Since the lungs are soft and fairly passive organs, they are influenced by respiration and heartbeat, resulting in discontinuity of lung motion and large deformation of anatomic features. This poses great challenges for accurate registration of lung image and its applications. The recent application of deep learning (DL) methods in the field of medical image registration has brought promising results. However, a versatile registration framework has not yet emerged due to diverse challenges of registration for different regions of interest (ROI). DL-based image registration methods used for other ROI cannot achieve satisfactory results in lungs. In addition, there are few review articles available on DL-based lung image registration. In this review, the development of conventional methods for lung image registration is briefly described and a more comprehensive survey of DL-based methods for lung image registration is illustrated. The DL-based methods are classified according to different supervision types, including fully-supervised, weakly-supervised and unsupervised. The contributions of researchers in addressing various challenges are described, as well as the limitations of these approaches. This review also presents a comprehensive statistical analysis of the cited papers in terms of evaluation metrics and loss functions. In addition, publicly available datasets for lung image registration are also summarized. Finally, the remaining challenges and potential trends in DL-based lung image registration are discussed.

Deformable medical image registration with global-local transformation network and region similarity constraint

Deformable medical image registration can achieve fast and accurate alignment between two images, enabling medical professionals to analyze images of different subjects in a unified anatomical space. As such, it plays an important role in many medical image studies. Current deep learning (DL)-based approaches for image registration directly learn spatial transformation from one image to another, relying on a convolutional neural network and ground truth or similarity metrics. However, these methods only use a global similarity energy function to evaluate the similarity of a pair of images, which ignores the similarity of regions of interest (ROIs) within the images. This can limit the accuracy of the image registration and affect the analysis of specific ROIs. Additionally, DL-based methods often estimate global spatial transformations of images directly, without considering local spatial transformations of ROIs within the images. To address this issue, we propose a novel global-local transformation network with a region similarity constraint that maximizes the similarity of ROIs within the images and estimates both global and local spatial transformations simultaneously. Experiments conducted on four public 3D MRI datasets demonstrate that the proposed method achieves the highest registration performance in terms of accuracy and generalization compared to other state-of-the-art methods.

Survey of methods and principles in three-dimensional reconstruction from two-dimensional medical images

Three-dimensional (3D) reconstruction of human organs has gained attention in recent years due to advances in the Internet and graphics processing units. In the coming years, most patient care will shift toward this new paradigm. However, development of fast and accurate 3D models from medical images or a set of medical scans remains a daunting task due to the number of pre-processing steps involved, most of which are dependent on human expertise. In this review, a survey of pre-processing steps was conducted, and reconstruction techniques for several organs in medical diagnosis were studied. Various methods and principles related to 3D reconstruction were highlighted. The usefulness of 3D reconstruction of organs in medical diagnosis was also highlighted.

Normal vibration distribution search-based differential evolution algorithm for multimodal biomedical image registration

In linear registration, a floating image is spatially aligned with a reference image after performing a series of linear metric transformations. Additionally, linear registration is mainly considered a preprocessing version of nonrigid registration. To better accomplish the task of finding the optimal transformation in pairwise intensity-based medical image registration, in this work, we present an optimization algorithm called the normal vibration distribution search-based differential evolution algorithm (NVSA), which is modified from the Bernstein search-based differential evolution (BSD) algorithm. We redesign the search pattern of the BSD algorithm and import several control parameters as part of the fine-tuning process to reduce the difficulty of the algorithm. In this study, 23 classic optimization functions and 16 real-world patients (resulting in 41 multimodal registration scenarios) are used in experiments performed to statistically investigate the problem solving ability of the NVSA. Nine metaheuristic algorithms are used in the conducted experiments. When compared to the commonly utilized registration methods, such as ANTS, Elastix, and FSL, our method achieves better registration performance on the RIRE dataset. Moreover, we prove that our method can perform well with or without its initial spatial transformation in terms of different evaluation indicators, demonstrating its versatility and robustness for various clinical needs and applications. This study establishes the idea that metaheuristic-based methods can better accomplish linear registration tasks than the frequently used approaches; the proposed method demonstrates promise that it can solve real-world clinical and service problems encountered during nonrigid registration as a preprocessing approach.The source code of the NVSA is publicly available at https://github.com/PengGui-N/NVSA.

A bidirectional registration neural network for cardiac motion tracking using cine MRI images

Using cine magnetic resonance imaging (cine MRI) images to track cardiac motion helps users to analyze the myocardial strain, and is of great importance in clinical applications. At present, most of the automatic deep learning-based motion tracking methods compare two images without considering temporal information between MRI frames, which easily leads to the lack of consistency of the generated motion fields. Even though a small number of works take into account the temporal factor, they are usually computationally intensive or have limitations on image length. To solve this problem, we propose a bidirectional convolution neural network for motion tracking of cardiac cine MRI images. This network leverages convolutional blocks to extract spatial features from three-dimensional (3D) image registration pairs, and models the temporal relations through a bidirectional recurrent neural network to obtain the Lagrange motion field between the reference image and other images. Compared with previous pairwise registration methods, the proposed method can automatically learn spatiotemporal information from multiple images with fewer parameters. We evaluated our model on three public cardiac cine MRI datasets. The experimental results demonstrated that the proposed method can significantly improve the motion tracking accuracy. The average Dice coefficient between estimated segmentation and manual segmentation has reached almost 0.85 on the widely used Automatic Cardiac Diagnostic Challenge (ACDC) dataset.

Segmentation by test-time optimization for CBCT-based adaptive radiation therapy

PURPOSE

Online adaptive radiotherapy (ART) requires accurate and efficient auto-segmentation of target volumes and organs-at-risk (OARs) in mostly cone-beam computed tomography (CBCT) images, which often have severe artifacts and lack soft-tissue contrast, making direct segmentation very challenging. Propagating expert-drawn contours from the pretreatment planning CT through traditional or deep learning (DL)-based deformable image registration (DIR) can achieve improved results in many situations. Typical DL-based DIR models are population based, that is, trained with a dataset for a population of patients, and so they may be affected by the generalizability problem.

METHODS

In this paper, we propose a method called test-time optimization (TTO) to refine a pretrained DL-based DIR population model, first for each individual test patient, and then progressively for each fraction of online ART treatment. Our proposed method is less susceptible to the generalizability problem and thus can improve overall performance of different DL-based DIR models by improving model accuracy, especially for outliers. Our experiments used data from 239 patients with head-and-neck squamous cell carcinoma to test the proposed method. First, we trained a population model with 200 patients and then applied TTO to the remaining 39 test patients by refining the trained population model to obtain 39 individualized models. We compared each of the individualized models with the population model in terms of segmentation accuracy.

RESULTS

The average improvement of the Dice similarity coefficient (DSC) and 95% Hausdorff distance (HD95) of segmentation can be up to 0.04 (5%) and 0.98 mm (25%), respectively, with the individualized models compared to the population model over 17 selected OARs and a target of 39 patients. Although the average improvement may seem mild, we found that the improvement for outlier patients with structures of large anatomical changes is significant. The number of patients with at least 0.05 DSC improvement or 2 mm HD95 improvement by TTO averaged over the 17 selected structures for the state-of-the-art architecture VoxelMorph is 10 out of 39 test patients. By deriving the individualized model using TTO from the pretrained population model, TTO models can be ready in about 1 min. We also generated the adapted fractional models for each of the 39 test patients by progressively refining the individualized models using TTO to CBCT images acquired at later fractions of online ART treatment. When adapting the individualized model to a later fraction of the same patient, the model can be ready in less than a minute with slightly improved accuracy.

CONCLUSIONS

The proposed TTO method is well suited for online ART and can boost segmentation accuracy for DL-based DIR models, especially for outlier patients where the pretrained models fail.

Multi-mode information fusion navigation system for robot-assisted vascular interventional surgery

BACKGROUND

Minimally invasive vascular intervention (MIVI) is a powerful technique for the treatment of cardiovascular diseases, such as abdominal aortic aneurysm (AAA), thoracic aortic aneurysm (TAA) and aortic dissection (AD). Navigation of traditional MIVI surgery mainly relies only on 2D digital subtraction angiography (DSA) images, which is hard to observe the 3D morphology of blood vessels and position the interventional instruments. The multi-mode information fusion navigation system (MIFNS) proposed in this paper combines preoperative CT images and intraoperative DSA images together to increase the visualization information during operations.

RESULTS

The main functions of MIFNS were evaluated by real clinical data and a vascular model. The registration accuracy of preoperative CTA images and intraoperative DSA images were less than 1 mm. The positioning accuracy of surgical instruments was quantitatively assessed using a vascular model and was also less than 1 mm. Real clinical data used to assess the navigation results of MIFNS on AAA, TAA and AD.

CONCLUSIONS

A comprehensive and effective navigation system was developed to facilitate the operation of surgeon during MIVI. The registration accuracy and positioning accuracy of the proposed navigation system were both less than 1 mm, which met the accuracy requirements of robot assisted MIVI.

An unsupervised image registration method employing chest computed tomography images and deep neural networks

BACKGROUND

Deformable image registration is crucial for multiple radiation therapy applications. Fast registration of computed tomography (CT) lung images is challenging because of the large and nonlinear deformation between inspiration and expiration. With advancements in deep learning techniques, learning-based registration methods are considered efficient alternatives to traditional methods in terms of accuracy and computational cost.

METHOD

In this study, an unsupervised lung registration network (LRN) with cycle-consistent training is proposed to align two acquired CT-derived lung datasets during breath-holds at inspiratory and expiratory levels without utilizing any ground-truth registration results. Generally, the LRN model uses three loss functions: image similarity, regularization, and Jacobian determinant. Here, LRN was trained on the CT datasets of 705 subjects and tested using 10 pairs of public CT DIR-Lab datasets. Furthermore, to evaluate the effectiveness of the registration technique, target registration errors (TREs) of the LRN model were compared with those of the conventional algorithm (sum of squared tissue volume difference; SSTVD) and a state-of-the-art unsupervised registration method (VoxelMorph).

RESULTS

The results showed that the LRN with an average TRE of 1.78 ± 1.56 mm outperformed VoxelMorph with an average TRE of 2.43 ± 2.43 mm, which is comparable to that of SSTVD with an average TRE of 1.66 ± 1.49 mm. In addition, estimating the displacement vector field without any folding voxel consumed less than 2 s, demonstrating the superiority of the learning-based method with respect to fiducial marker tracking and the overall soft tissue alignment with a nearly real-time speed.

CONCLUSIONS

Therefore, this proposed method shows significant potential for use in time-sensitive pulmonary studies, such as lung motion tracking and image-guided surgery.

Surface deformation tracking in monocular laparoscopic video

Image-guided surgery has been proven to enhance the accuracy and safety of minimally invasive surgery (MIS). Nonrigid deformation tracking of soft tissue is one of the main challenges in image-guided MIS owing to the existence of tissue deformation, homogeneous texture, smoke and instrument occlusion, etc. In this paper, we proposed a piecewise affine deformation model-based nonrigid deformation tracking method. A Markov random field based mask generation method is developed to eliminate tracking anomalies. The deformation information vanishes when the regular constraint is invalid, which further deteriorates the tracking accuracy. Atime-series deformation solidification mechanism is introduced to reduce the degradation of the deformation field of the model. For the quantitative evaluation of the proposed method, we synthesized nine laparoscopic videos mimicking instrument occlusion and tissue deformation. Quantitative tracking robustness was evaluated on the synthetic videos. Three real videos of MIS containing challenges of large-scale deformation, large-range smoke, instrument occlusion, and permanent changes in soft tissue texture were also used to evaluate the performance of the proposed method. Experimental results indicate the proposed method outperforms state-of-the-art methods in terms of accuracy and robustness, which shows good performance in image-guided MIS.

Predicting pulmonary ventilation damage after radiation therapy for nonsmall cell lung cancer using a ResNet generative adversarial network

BACKGROUND

Functional lung avoidance radiation therapy (RT) is a technique being investigated to preferentially avoid specific regions of the lung that are predicted to be more susceptible to radiation-induced damage. Reducing the dose delivered to high functioning regions may reduce the occurrence radiation-induced lung injuries (RILIs) and toxicities. However, in order to develop effective lung function-sparing plans, accurate predictions of post-RT ventilation change are needed to determine which regions of the lung should be spared.

PURPOSE

To predict pulmonary ventilation change following RT for nonsmall cell lung cancer using machine learning.

METHODS

A conditional generative adversarial network (cGAN) was developed with data from 82 human subjects enrolled in a randomized clinical trial approved by the institution's IRB to predict post-RT pulmonary ventilation change. The inputs to the network were the pre-RT pulmonary ventilation map and radiation dose distribution. The loss function was a combination of the binary cross-entropy loss and an asymmetrical structural similarity index measure (aSSIM) function designed to increase penalization of under-prediction of ventilation damage. Network performance was evaluated against a previously developed polynomial regression model using a paired sample t-test for comparison. Evaluation was performed using eight-fold cross-validation.

RESULTS

From the eight-fold cross-validation, we found that relative to the polynomial model, the cGAN model significantly improved predicting regions of ventilation damage following radiotherapy based on true positive rate (TPR), 0.14±0.15 to 0.72±0.21, and Dice similarity coefficient (DSC), 0.19±0.16 to 0.46±0.14, but significantly declined in true negative rate, 0.97±0.05 to 0.62±0.21, and accuracy, 0.79±0.08 to 0.65±0.14. Additionally, the average true positive volume increased from 104±119 cc in the POLY model to 565±332 cc in the cGAN model, and the average false negative volume decreased from 654±361 cc in the POLY model to 193±163 cc in the cGAN model.

CONCLUSIONS

The proposed cGAN model demonstrated significant improvement in TPR and DSC. The higher sensitivity of the cGAN model can improve the clinical utility of functional lung avoidance RT by identifying larger volumes of functional lung that can be spared and thus decrease the probability of the patient developing RILIs.

Enhancement of cone beam CT image registration by super-resolution pre-processing algorithm

In order to enhance cone-beam computed tomography (CBCT) image information and improve the registration accuracy for image-guided radiation therapy, we propose a super-resolution (SR) image enhancement method. This method uses super-resolution techniques to pre-process the CBCT prior to registration. Three rigid registration methods (rigid transformation, affine transformation, and similarity transformation) and a deep learning deformed registration (DLDR) method with and without SR were compared. The five evaluation indices, the mean squared error (MSE), mutual information, Pearson correlation coefficient (PCC), structural similarity index (SSIM), and PCC + SSIM, were used to validate the results of registration with SR. Moreover, the proposed method SR-DLDR was also compared with the VoxelMorph (VM) method. In rigid registration with SR, the registration accuracy improved by up to 6% in the PCC metric. In DLDR with SR, the registration accuracy was improved by up to 5% in PCC + SSIM. When taking the MSE as the loss function, the accuracy of SR-DLDR is equivalent to that of the VM method. In addition, when taking the SSIM as the loss function, the registration accuracy of SR-DLDR is 6% higher than that of VM. SR is a feasible method to be used in medical image registration for planning CT (pCT) and CBCT. The experimental results show that the SR algorithm can improve the accuracy and efficiency of CBCT image alignment regardless of which alignment algorithm is used.

Hierarchical anatomical structure-aware based thoracic CT images registration

Accurate thoracic CT image registration remains challenging due to complex joint deformations and different motion patterns in multiple organs/tissues during breathing. To combat this, we devise a hierarchical anatomical structure-aware based registration framework. It affords a coordination scheme necessary for constraining a general free-form deformation (FFD) during thoracic CT registration. The key is to integrate the deformations of different anatomical structures in a divide-and-conquer way. Specifically, a deformation ability-aware dissimilarity metric is proposed for complex joint deformations containing large-scale flexible deformation of the lung region, rigid displacement of the bone region, and small-scale flexible deformation of the rest region. Furthermore, a motion pattern-aware regularization is devised to handle different motion patterns, which contain sliding motion along the lung surface, almost no displacement of the spine and smooth deformation of other regions. Moreover, to accommodate large-scale deformation, a novel hierarchical strategy, wherein different anatomical structures are fused on the same control lattice, registers images from coarse to fine via elaborate Gaussian pyramids. Extensive experiments and comprehensive evaluations have been executed on the 4D-CT DIR and 3D DIR COPD datasets. It confirms that this newly proposed method is locally comparable to state-of-the-art registration methods specializing in local deformations, while guaranteeing overall accuracy. Additionally, in contrast to the current popular learning-based methods that typically require dozens of hours or more pre-training with powerful graphics cards, our method only takes an average of 63 s to register a case with an ordinary graphics card of RTX2080 SUPER, making our method still worth promoting. Our code is available at https://github.com/heluxixue/Structure_Aware_Registration/tree/master.

PLOSL: Population learning followed by one shot learning pulmonary image registration using tissue volume preserving and vesselness constraints

This paper presents the Population Learning followed by One Shot Learning (PLOSL) pulmonary image registration method. PLOSL is a fast unsupervised learning-based framework for 3D-CT pulmonary image registration algorithm based on combining population learning (PL) and one-shot learning (OSL). The PLOSL image registration has the advantages of the PL and OSL approaches while reducing their respective drawbacks. The advantages of PLOSL include improved performance over PL, substantially reducing OSL training time and reducing the likelihood of OSL getting stuck in local minima. PLOSL pulmonary image registration uses tissue volume preserving and vesselness constraints for registration of inspiration-to-expiration and expiration-to-inspiration pulmonary CT images. A coarse-to-fine convolution encoder-decoder CNN architecture is used to register large and small shape features. During training, the sum of squared tissue volume difference (SSTVD) compensates for intensity differences between inspiration and expiration computed tomography (CT) images and the sum of squared vesselness measure difference (SSVMD) helps match the lung vessel tree. Results show that the PLOSL (SSTVD+SSVMD) algorithm achieved subvoxel landmark error while preserving pulmonary topology on the SPIROMICS data set, the public DIR-LAB COPDGene and 4DCT data sets.

Towards a safe and efficient clinical implementation of machine learning in radiation oncology by exploring model interpretability, explainability and data-model dependency

The interest in machine learning (ML) has grown tremendously in recent years, partly due to the performance leap that occurred with new techniques of deep learning, convolutional neural networks for images, increased computational power, and wider availability of large datasets. Most fields of medicine follow that popular trend and, notably, radiation oncology is one of those that are at the forefront, with already a long tradition in using digital images and fully computerized workflows. ML models are driven by data, and in contrast with many statistical or physical models, they can be very large and complex, with countless generic parameters. This inevitably raises two questions, namely, the tight dependence between the models and the datasets that feed them, and the interpretability of the models, which scales with its complexity. Any problems in the data used to train the model will be later reflected in their performance. This, together with the low interpretability of ML models, makes their implementation into the clinical workflow particularly difficult. Building tools for risk assessment and quality assurance of ML models must involve then two main points: interpretability and data-model dependency. After a joint introduction of both radiation oncology and ML, this paper reviews the main risks and current solutions when applying the latter to workflows in the former. Risks associated with data and models, as well as their interaction, are detailed. Next, the core concepts of interpretability, explainability, and data-model dependency are formally defined and illustrated with examples. Afterwards, a broad discussion goes through key applications of ML in workflows of radiation oncology as well as vendors' perspectives for the clinical implementation of ML.

Artificial intelligence in functional imaging of the lung

Artificial intelligence (AI) is transforming the way we perform advanced imaging. From high-resolution image reconstruction to predicting functional response from clinically acquired data, AI is promising to revolutionize clinical evaluation of lung performance, pushing the boundary in pulmonary functional imaging for patients suffering from respiratory conditions. In this review, we overview the current developments and expound on some of the encouraging new frontiers. We focus on the recent advances in machine learning and deep learning that enable reconstructing images, quantitating, and predicting functional responses of the lung. Finally, we shed light on the potential opportunities and challenges ahead in adopting AI for functional lung imaging in clinical settings.

Deep learning in structural and functional lung image analysis

The recent resurgence of deep learning (DL) has dramatically influenced the medical imaging field. Medical image analysis applications have been at the forefront of DL research efforts applied to multiple diseases and organs, including those of the lungs. The aims of this review are twofold: (i) to briefly overview DL theory as it relates to lung image analysis; (ii) to systematically review the DL research literature relating to the lung image analysis applications of segmentation, reconstruction, registration and synthesis. The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. 479 studies were initially identified from the literature search with 82 studies meeting the eligibility criteria. Segmentation was the most common lung image analysis DL application (65.9% of papers reviewed). DL has shown impressive results when applied to segmentation of the whole lung and other pulmonary structures. DL has also shown great potential for applications in image registration, reconstruction and synthesis. However, the majority of published studies have been limited to structural lung imaging with only 12.9% of reviewed studies employing functional lung imaging modalities, thus highlighting significant opportunities for further research in this field. Although the field of DL in lung image analysis is rapidly expanding, concerns over inconsistent validation and evaluation strategies, intersite generalisability, transparency of methodological detail and interpretability need to be addressed before widespread adoption in clinical lung imaging workflow.

Artificial Intelligence in Radiation Therapy

Artificial intelligence (AI) has great potential to transform the clinical workflow of radiotherapy. Since the introduction of deep neural networks, many AI-based methods have been proposed to address challenges in different aspects of radiotherapy. Commercial vendors have started to release AI-based tools that can be readily integrated to the established clinical workflow. To show the recent progress in AI-aided radiotherapy, we have reviewed AI-based studies in five major aspects of radiotherapy including image reconstruction, image registration, image segmentation, image synthesis, and automatic treatment planning. In each section, we summarized and categorized the recently published methods, followed by a discussion of the challenges, concerns, and future development. Given the rapid development of AI-aided radiotherapy, the efficiency and effectiveness of radiotherapy in the future could be substantially improved through intelligent automation of various aspects of radiotherapy.

Self-supervised learning-based diffeomorphic non-rigid motion estimation for fast motion-compensated coronary MR angiography

PURPOSE

To accelerate non-rigid motion corrected coronary MR angiography (CMRA) reconstruction by developing a deep learning based non-rigid motion estimation network and combining this with an efficient implementation of the undersampled motion corrected reconstruction.

METHODS

Undersampled and respiratory motion corrected CMRA with overall short scans of 5 to 10 min have been recently proposed. However, image reconstruction with this approach remains lengthy, since it relies on several non-rigid image registrations to estimate the respiratory motion and on a subsequent iterative optimization to correct for motion during the undersampled reconstruction. Here we introduce a self-supervised diffeomorphic non-rigid respiratory motion estimation network, DiRespME-net, to speed up respiratory motion estimation. We couple this with an efficient GPU-based implementation of the subsequent motion-corrected iterative reconstruction. DiRespME-net is based on a U-Net architecture, and is trained in a self-supervised fashion, with a loss enforcing image similarity and spatial smoothness of the motion fields. Motion predicted by DiRespME-net was used for GPU-based motion-corrected CMRA in 12 test subjects and final images were compared to those produced by state-of-the-art reconstruction. Vessel sharpness and visible length of the right coronary artery (RCA) and the left anterior descending (LAD) coronary artery were used as metrics of image quality for comparison.

RESULTS

No statistically significant difference in image quality was found between images reconstructed with the proposed approach (MC:DiRespME-net) and a motion-corrected reconstruction using cubic B-splines (MC:Nifty-reg). Visible vessel length was not significantly different between methods (RCA: MC:Nifty-reg 5.7 ± 1.7 cm vs MC:DiRespME-net 5.8 ± 1.7 cm, P = 0.32; LAD: MC:Nifty-reg 7.0 ± 2.6 cm vs MC:DiRespME-net 6.9 ± 2.7 cm, P = 0.81). Similarly, no statistically significant difference between methods was observed in terms of vessel sharpness (RCA: MC:Nifty-reg 60.3 ± 7.2% vs MC:DiRespME-net 61.0 ± 6.8%, P = 0.19; LAD: MC:Nifty-reg 57.4 ± 7.9% vs MC:DiRespME-net 58.1 ± 7.5%, P = 0.27). The proposed approach achieved a 50-fold reduction in computation time, resulting in a total reconstruction time of approximately 20 s.

CONCLUSIONS

The proposed self-supervised learning-based motion corrected reconstruction enables fast motion-corrected CMRA image reconstruction, holding promise for integration in clinical routine.

Few-shot learning for deformable image registration in 4DCT images

OBJECTIVES

To develop a rapid and accurate 4D deformable image registration (DIR) approach for online adaptive radiotherapy.

METHODS

We propose a deep learning (DL)-based few-shot registration network (FR-Net) to generate deformation vector fields from each respiratory phase to an implicit reference image, thereby mitigating the bias introduced by the selection of reference images. The proposed FR-Net is pretrained with limited unlabeled 4D data and further optimized by maximizing the intensity similarity of one specific four-dimensional computed tomography (4DCT) scan. Because of the learning ability of DL models, the few-shot learning strategy facilitates the generalization of the model to other 4D data sets and the acceleration of the optimization process.

RESULTS

The proposed FR-Net is evaluated for 4D groupwise and 3D pairwise registration on thoracic 4DCT data sets DIR-Lab and POPI. FR-Net displays an averaged target registration error of 1.48 mm and 1.16 mm between the maximum inhalation and exhalation phases in the 4DCT of DIR-Lab and POPI, respectively, with approximately 2 min required to optimize one 4DCT. Overall, FR-Net outperforms state-of-the-art methods in terms of registration accuracy and exhibits a low computational time.

CONCLUSION

We develop a few-shot groupwise DIR algorithm for 4DCT images. The promising registration performance and computational efficiency demonstrate the prospective applications of this approach in registration tasks for online adaptive radiotherapy.

ADVANCES IN KNOWLEDGE

This work exploits DL models to solve the optimization problem in registering 4DCT scans while combining groupwise registration and few-shot learning strategy to solve the problem of consuming computational time and inferior registration accuracy.

Lung-CRNet: A convolutional recurrent neural network for lung 4DCT image registration

PURPOSE

Deformable image registration (DIR) of lung four-dimensional computed tomography (4DCT) plays a vital role in a wide range of clinical applications. Most of the existing deep learning-based lung 4DCT DIR methods focus on pairwise registration which aims to register two images with large deformation. However, the temporal continuities of deformation fields between phases are ignored. This paper proposes a fast and accurate deep learning-based lung 4DCT DIR approach that leverages the temporal component of 4DCT images.

METHODS

We present Lung-CRNet, an end-to-end convolutional recurrent registration neural network for lung 4DCT images and reformulate 4DCT DIR as a spatiotemporal sequence predicting problem in which the input is a sequence of three-dimensional computed tomography images from the inspiratory phase to the expiratory phase in a respiratory cycle. The first phase in the sequence is selected as the only reference image and the rest as moving images. Multiple convolutional gated recurrent units (ConvGRUs) are stacked to capture the temporal clues between images. The proposed network is trained in an unsupervised way using a spatial transformer layer. During inference, Lung-CRNet is able to yield the respective displacement field for each reference-moving image pair in the input sequence.

RESULTS

We have trained the proposed network using a publicly available lung 4DCT dataset and evaluated performance on the widely used the DIR-Lab dataset. The mean and standard deviation of target registration error are 1.56 ± 1.05 mm on the DIR-Lab dataset. The computation time for each forward prediction is less than 1 s on average.

CONCLUSIONS

The proposed Lung-CRNet is comparable to the existing state-of-the-art deep learning-based 4DCT DIR methods in both accuracy and speed. Additionally, the architecture of Lung-CRNet can be generalized to suit other groupwise registration tasks which align multiple images simultaneously.

Optimizing Few-Shot Learning Based on Variational Autoencoders

Despite the importance of few-shot learning, the lack of labeled training data in the real world makes it extremely challenging for existing machine learning methods because this limited dataset does not well represent the data variance. In this research, we suggest employing a generative approach using variational autoencoders (VAEs), which can be used specifically to optimize few-shot learning tasks by generating new samples with more intra-class variations on the Labeled Faces in the Wild (LFW) dataset. The purpose of our research is to increase the size of the training dataset using various methods to improve the accuracy and robustness of the few-shot face recognition. Specifically, we employ the VAE generator to increase the size of the training dataset, including the basic and the novel sets while utilizing transfer learning as the backend. Based on extensive experimental research, we analyze various data augmentation methods to observe how each method affects the accuracy of face recognition. The face generation method based on VAEs with perceptual loss can effectively improve the recognition accuracy rate to 96.47% using both the base and the novel sets.

Generalized Zero-Shot Chest X-Ray Diagnosis Through Trait-Guided Multi-View Semantic Embedding With Self-Training

Zero-shot learning (ZSL) is one of the most promising avenues of annotation-efficient machine learning. In the era of deep learning, ZSL techniques have achieved unprecedented success. However, the developments of ZSL methods have taken place mostly for natural images. ZSL for medical images has remained largely unexplored. We design a novel strategy for generalized zero-shot diagnosis of chest radiographs. In doing so, we leverage the potential of multi-view semantic embedding, a useful yet less-explored direction for ZSL. Our design also incorporates a self-training phase to tackle the problem of noisy labels alongside improving the performance for classes not seen during training. Through rigorous experiments, we show that our model trained on one dataset can produce consistent performance across test datasets from different sources including those with very different quality. Comparisons with a number of state-of-the-art techniques show the superiority of the proposed method for generalized zero-shot chest x-ray diagnosis.

Interactive Few-Shot Learning: Limited Supervision, Better Medical Image Segmentation

Many known supervised deep learning methods for medical image segmentation suffer an expensive burden of data annotation for model training. Recently, few-shot segmentation methods were proposed to alleviate this burden, but such methods often showed poor adaptability to the target tasks. By prudently introducing interactive learning into the few-shot learning strategy, we develop a novel few-shot segmentation approach called Interactive Few-shot Learning (IFSL), which not only addresses the annotation burden of medical image segmentation models but also tackles the common issues of the known few-shot segmentation methods. First, we design a new few-shot segmentation structure, called Medical Prior-based Few-shot Learning Network (MPrNet), which uses only a few annotated samples (e.g., 10 samples) as support images to guide the segmentation of query images without any pre-training. Then, we propose an Interactive Learning-based Test Time Optimization Algorithm (IL-TTOA) to strengthen our MPrNet on the fly for the target task in an interactive fashion. To our best knowledge, our IFSL approach is the first to allow few-shot segmentation models to be optimized and strengthened on the target tasks in an interactive and controllable manner. Experiments on four few-shot segmentation tasks show that our IFSL approach outperforms the state-of-the-art methods by more than 20% in the DSC metric. Specifically, the interactive optimization algorithm (IL-TTOA) further contributes ~10% DSC improvement for the few-shot segmentation models.

Using synthetic data generation to train a cardiac motion tag tracking neural network

A CNN based method for cardiac MRI tag tracking was developed and validated. A synthetic data simulator was created to generate large amounts of training data using natural images, a Bloch equation simulation, a broad range of tissue properties, and programmed ground-truth motion. The method was validated using both an analytical deforming cardiac phantom and in vivo data with manually tracked reference motion paths. In the analytical phantom, error was investigated relative to SNR, and accurate results were seen for SNR>10 (displacement error <0.3 mm). Excellent agreement was seen in vivo for tag locations (mean displacement difference = -0.02 pixels, 95% CI [-0.73, 0.69]) and calculated cardiac circumferential strain (mean difference = 0.006, 95% CI [-0.012, 0.024]). Automated tag tracking with a CNN trained on synthetic data is both accurate and precise.

Transfer Learning for Nonrigid 2D/3D Cardiovascular Images Registration

Cardiovascular image registration is an essential approach to combine the advantages of preoperative 3D computed tomography angiograph (CTA) images and intraoperative 2D X-ray/digital subtraction angiography (DSA) images together in minimally invasive vascular interventional surgery (MIVI). Recent studies have shown that convolutional neural network (CNN) regression model can be used to register these two modality vascular images with fast speed and satisfactory accuracy. However, CNN regression model trained by tens of thousands of images of one patient is often unable to be applied to another patient due to the large difference and deformation of vascular structure in different patients. To overcome this challenge, we evaluate the ability of transfer learning (TL) for the registration of 2D/3D deformable cardiovascular images. Frozen weights in the convolutional layers were optimized to find the best common feature extractors for TL. After TL, the training data set size was reduced to 200 for a randomly selected patient to get accurate registration results. We compared the effectiveness of our proposed nonrigid registration model after TL with not only that without TL but also some traditional intensity-based methods to evaluate that our nonrigid model after TL performs better on deformable cardiovascular image registration.

GraphRegNet: Deep Graph Regularisation Networks on Sparse Keypoints for Dense Registration of 3D Lung CTs

In the last two years learning-based methods have started to show encouraging results in different supervised and unsupervised medical image registration tasks. Deep neural networks enable (near) real time applications through fast inference times and have tremendous potential for increased registration accuracies by task-specific learning. However, estimation of large 3D deformations, for example present in inhale to exhale lung CT or interpatient abdominal MRI registration, is still a major challenge for the widely adopted U-Net-like network architectures. Even when using multi-level strategies, current state-of-the-art DL registration results do not yet reach the high accuracy of conventional frameworks. To overcome the problem of large deformations for deep learning approaches, in this work, we present GraphRegNet, a sparse keypoint-based geometric network for dense deformable medical image registration. Similar to the successful 2D optical flow estimation of FlowNet or PWC-Net we leverage discrete dense displacement maps to facilitate the registration process. In order to cope with enormously increasing memory requirements when working with displacement maps in 3D medical volumes and to obtain a well-regularised and accurate deformation field we 1) formulate the registration task as the prediction of displacement vectors on a sparse irregular grid of distinctive keypoints and 2) introduce our efficient GraphRegNet for displacement regularisation, a combination of convolutional and graph neural network layers in a unified architecture. In our experiments on exhale to inhale lung CT registration we demonstrate substantial improvements (TRE below 1.4 mm) over other deep learning methods. Our code is publicly available at https://github.com/multimodallearning/graphregnet.

Measuring breathing induced oesophageal motion and its dosimetric impact

PURPOSE

Stereotactic body radiation therapy allows for a precise dose delivery. Organ motion bears the risk of undetected high dose healthy tissue exposure. An organ very susceptible to high dose is the oesophagus. Its low contrast on CT and the oblong shape render motion estimation difficult. We tackle this issue by modern algorithms to measure oesophageal motion voxel-wise and estimate motion related dosimetric impacts.

METHODS

Oesophageal motion was measured using deformable image registration and 4DCT of 11 internal and 5 public datasets. Current clinical practice of contouring the organ on 3DCT was compared to timely resolved 4DCT contours. Dosimetric impacts of the motion were estimated by analysing the trajectory of each voxel in the 4D dose distribution. Finally an organ motion model for patient-wise comparisons was built.

RESULTS

Motion analysis showed mean absolute maximal motion amplitudes of 4.55 ± 1.81 mm left-right, 5.29 ± 2.67 mm anterior-posterior and 10.78 ± 5.30 mm superior-inferior. Motion between cohorts differed significantly. In around 50% of the cases the dosimetric passing criteria was violated. Contours created on 3DCT did not cover 14% of the organ for 50% of the respiratory cycle and were around 38% smaller than the union of all 4D contours. The motion model revealed that the maximal motion is not limited to the lower part of the organ. Our results showed motion amplitudes higher than most reported values in the literature and that motion is very heterogeneous across patients.

CONCLUSIONS

Individual motion information should be considered in contouring and planning.

High-precision Registration Algorithm and Parallel Design Method for High-Resolution Optical Remote Sensing Images

In optical remote sensing image reconstruction, image registration is an important issue to address in order to ensure satisfactory reconstruction performance. In this study, a multi-frame image registration algorithm for high-resolution images and its parallel design method are proposed. The algorithm realizes an improved feature point detection method based on an adaptive gradient bilateral tensor filter and carries out weighted Gaussian surface sub-pixel interpolation to obtain more accurate corner positions, which better guarantees the registration accuracy. On this basis, multi-scale expansion is carried out to generate descriptors for image registration. In addition, the operation-level parallel analysis and design are carried out on a GPU platform based on compute unified device architecture (CUDA), and the memory model of the GPU is utilized reasonably. The task-level parallel analysis and design are carried out based on the GPU stream model. Moreover, based on the open multi-processing (OpenMP) platform, a multi-core CPU carries out parallel design at the operation level and task level, which realizes post-processing operations such as optical remote sensing images loading, accurate matching, and coordinate mapping, thereby effectively improving registration speed. Compared with feature point algorithms and deep learning algorithm, our algorithm and its parallel design significantly improve the registration accuracy and speed of high-resolution optical remote sensing images.

Deep learning-based coronary artery motion estimation and compensation for short-scan cardiac CT

PURPOSE

During a typical cardiac short scan, the heart can move several millimeters. As a result, the corresponding CT reconstructions may be corrupted by motion artifacts. Especially the assessment of small structures, such as the coronary arteries, is potentially impaired by the presence of these artifacts. In order to estimate and compensate for coronary artery motion, this manuscript proposes the deep partial angle-based motion compensation (Deep PAMoCo).

METHODS

The basic principle of the Deep PAMoCo relies on the concept of partial angle reconstructions (PARs), that is, it divides the short scan data into several consecutive angular segments and reconstructs them separately. Subsequently, the PARs are deformed according to a motion vector field (MVF) such that they represent the same motion state and summed up to obtain the final motion-compensated reconstruction. However, in contrast to prior work that is based on the same principle, the Deep PAMoCo estimates and applies the MVF via a deep neural network to increase the computational performance as well as the quality of the motion compensated reconstructions.

RESULTS

Using simulated data, it could be demonstrated that the Deep PAMoCo is able to remove almost all motion artifacts independent of the contrast, the radius and the motion amplitude of the coronary artery. In any case, the average error of the CT values along the coronary artery is about 25 HU while errors of up to 300 HU can be observed if no correction is applied. Similar results were obtained for clinical cardiac CT scans where the Deep PAMoCo clearly outperforms state-of-the-art coronary artery motion compensation approaches in terms of processing time as well as accuracy.

CONCLUSIONS

The Deep PAMoCo provides an efficient approach to increase the diagnostic value of cardiac CT scans even if they are highly corrupted by motion.

GroupRegNet: a groupwise one-shot deep learning-based 4D image registration method

Accurate deformable four-dimensional (4D) (three-dimensional in space and time) medical images registration is essential in a variety of medical applications. Deep learning-based methods have recently gained popularity in this area for the significantly lower inference time. However, they suffer from drawbacks of non-optimal accuracy and the requirement of a large amount of training data. A new method named GroupRegNet is proposed to address both limitations. The deformation fields to warp all images in the group into a common template is obtained through one-shot learning. The use of the implicit template reduces bias and accumulated error associated with the specified reference image. The one-shot learning strategy is similar to the conventional iterative optimization method but the motion model and parameters are replaced with a convolutional neural network and the weights of the network. GroupRegNet also features a simpler network design and a more straightforward registration process, which eliminates the need to break up the input image into patches. The proposed method was quantitatively evaluated on two public respiratory-binned 4D-computed tomography datasets. The results suggest that GroupRegNet outperforms the latest published deep learning-based methods and is comparable to the top conventional method pTVreg. To facilitate future research, the source code is available at https://github.com/vincentme/GroupRegNet.

Spatiotemporal Free-Form Registration Method Assisted by a Minimum Spanning Tree During Discontinuous Transformations

The sliding motion along the boundaries of discontinuous regions has been actively studied in B-spline free-form deformation framework. This study focusses on the sliding motion for a velocity field-based 3D+t registration. The discontinuity of the tangent direction guides the deformation of the object region, and a separate control of two regions provides a better registration accuracy. The sliding motion under the velocity field-based transformation is conducted under the [Formula: see text]-Rényi entropy estimator using a minimum spanning tree (MST) topology. Moreover, a new topology changing method of the MST is proposed. The topology change is performed as follows: inserting random noise, constructing the MST, and removing random noise while preserving a local connection consistency of the MST. This random noise process (RNP) prevents the [Formula: see text]-Rényi entropy-based registration from degrading in sliding motion, because the RNP creates a small disturbance around special locations. Experiments were performed using two publicly available datasets: the DIR-Lab dataset, which consists of 4D pulmonary computed tomography (CT) images, and a benchmarking framework dataset for cardiac 3D ultrasound. For the 4D pulmonary CT images, RNP produced a significantly improved result for the original MST with sliding motion (p<0.05). For the cardiac 3D ultrasound dataset, only a discontinuity-based registration indicated activity of the RNP. In contrast, the single MST without sliding motion did not show any improvement. These experiments proved the effectiveness of the RNP for sliding motion.