Deformable medical image registration: a survey

Variation assessment of deformable registration in stereotactic radiosurgery

INTRODUCTION

The regular functions of CT-MRI registration include delineation of targets and organs-at-risk (OARs) in radiosurgery planning. The question of whether deformable image registration (DIR) could be applied to stereotactic radiosurgery (SRS) in its place remains a subject of debate.

METHODS

This study collected data regarding 16 patients who had undergone single-fraction SRS treatment. All lesions were located close to the brainstem. CT and MRI two image sets were registered by both rigid image registration (RIR) and DIR algorithms. The contours of the OARs were drawn individually on the rigid and deformable CT-MRI image sets by qualified radiation oncologists and dosimetrists. The evaluation metrics included volume overlapping (VO), Dice similarity coefficient (DSC), and dose. The modified demons deformable algorithm (VARIAN SmartAdapt) was used for evaluation in this study.

RESULTS

The mean range of VO for OARs was 0.84 ± 0.08, and DSC was 0.82 ± 0.07. The maximum average volume difference was at normal brain (17.18 ± 14.48 cm³) and the second highest was at brainstem (2.26 cm³ ± 1.18). Pearson correlation testing showed that all DIRs' OAR volumes were linearly and significantly correlated with RIRs' volume (0.679-0.992, two tailed, P << 0.001). The 100% dose was prescribed at gross tumor volume (GTV). The average maximum percent dose difference was observed in brainstem (26.54% ± 27.027), and the average mean dose difference has found at same organ (1.6% ± 1.66).

CONCLUSION

The change in image-registration method definitely produces dose variance, and is significantly more what depending on the target location. The volume size of OARs, however, was not statistical significantly correlated with dose variance.

A hybrid optimization strategy for registering images with large local deformations and intensity variations

PURPOSE

To develop a method for intra-patient registration of pre- and post-contrast abdominal MR images with large local deformations and large intensity variations.

METHOD

A hybrid method is proposed to deal with this problem. It consists of two coupled techniques: (1) descriptor matching (DM) at the original resolution using a discrete optimization strategy to avoid getting trapped in a local minimum; (2) continuous optimization to refine the registration outcome based on autocorrelation of local image structure (ALOST). Our method-called DM-ALOST-has become insensitive to the local uptake of contrast agent by exploiting the mean phase and the phase congruency extracted from the multi-scale monogenic signal. The method was extensively tested on abdominal MR data of 30 patients with Crohn's disease.

RESULTS

DM-ALOST produced significantly larger mean Dice coefficients than two state-of-the-art methods [Formula: see text].

CONCLUSION

Both qualitative and quantitative tests demonstrated improved registration using the proposed method compared to the state-of-the-art. The DM-ALOST method facilitates measurement of corresponding features from different abdominal MR images, which can aid to assess certain diseases, particularly Crohn's disease.

Image Registration Based on Low Rank Matrix: Rank-Regularized SSD

Similarity measure is a main core of image registration algorithms. Spatially varying intensity distortion is an important challenge, which affects the performance of similarity measures. Correlation among the pixels is the main characteristic of this distortion. Similarity measures such as sum-of-squared-differences (SSD) and mutual information ignore this correlation; hence, perfect registration cannot be achieved in the presence of this distortion. In this paper, we model this correlation with the aid of the low rank matrix theory. Based on this model, we compensate this distortion analytically and introduce rank-regularized SSD (RRSSD). This new similarity measure is a modified SSD based on singular values of difference image in mono-modal imaging. In fact, image registration and distortion correction are performed simultaneously in the proposed model. Based on our experiments, the RRSSD similarity measure achieves clinically acceptable registration results, and outperforms other state-of-the-art similarity measures, such as the well-known method of residual complexity.

Fast elastic registration of soft tissues under large deformations

A fast and accurate fusion of intra-operative images with a pre-operative data is a key component of computer-aided interventions which aim at improving the outcomes of the intervention while reducing the patient's discomfort. In this paper, we focus on the problematic of the intra-operative navigation during abdominal surgery, which requires an accurate registration of tissues undergoing large deformations. Such a scenario occurs in the case of partial hepatectomy: to facilitate the access to the pathology, e.g. a tumor located in the posterior part of the right lobe, the surgery is performed on a patient in lateral position. Due to the change in patient's position, the resection plan based on the pre-operative CT scan acquired in the supine position must be updated to account for the deformations. We suppose that an imaging modality, such as the cone-beam CT, provides the information about the intra-operative shape of an organ, however, due to the reduced radiation dose and contrast, the actual locations of the internal structures necessary to update the planning are not available. To this end, we propose a method allowing for fast registration of the pre-operative data represented by a detailed 3D model of the liver and its internal structure and the actual configuration given by the organ surface extracted from the intra-operative image. The algorithm behind the method combines the iterative closest point technique with a biomechanical model based on a co-rotational formulation of linear elasticity which accounts for large deformations of the tissue. The performance, robustness and accuracy of the method is quantitatively assessed on a control semi-synthetic dataset with known ground truth and a real dataset composed of nine pairs of abdominal CT scans acquired in supine and flank positions. It is shown that the proposed surface-matching method is capable of reducing the target registration error evaluated of the internal structures of the organ from more than 40 mm to less then 10 mm. Moreover, the control data is used to demonstrate the compatibility of the method with intra-operative clinical scenario, while the real datasets are utilized to study the impact of parametrization on the accuracy of the method. The method is also compared to a state-of-the art intensity-based registration technique in terms of accuracy and performance.

A Robust and Accurate Non-rigid Medical Image Registration Algorithm Based on Multi-level Deformable Model

BACKGROUND

Compared to the rigid image registration task, the non-rigid image registration task faces much more challenges due to its high degree of freedom and inherent requirement of smoothness in the deformation field. The purpose was to propose an efficient coarse-to-fine non-rigid medical image registration algorithm based on a multilevel deformable model.

METHODS

In this paper, a robust and efficient coarse-to-fine non-rigid medical image registration algorithm is proposed. It contains three level deformation models, i.e., the global homography model, the local mesh-level homography model, and the local B-spline FFD (Free-Form Deformation) model. The coarse registration is achieved by the first two level models. In the global homography model, a robust algorithm for simultaneous outliers (error matched feature points) removal and model estimation is applied. In the local mesh-level homography model, a new similarity measure is proposed to improve the robustness and accuracy of local mesh based registration. In the fine registration, a local B-spline FFD model with normalized mutual information gradient is employed.

RESULTS

We verified the effectiveness of each stage of the proposed registration algorithm with many non-rigid transformation image pairs, and quantitatively compared our proposed registration algorithm with the HBFFD method which is based on the control points of multi-resolution. The experimental results show that our algorithm is more accurate than the hierarchical local B-spline FFD method.

CONCLUSION

Our algorithm can achieve high precision registration by coarse-to-fine process based on multi-level deformable model, which ourperforms the state-of-the-art methods.

Learning-based deformable registration for infant MRI by integrating random forest with auto-context model

PURPOSE

Accurately analyzing the rapid structural evolution of human brain in the first year of life is a key step in early brain development studies, which requires accurate deformable image registration. However, due to (a) dynamic appearance and (b) large anatomical changes, very few methods in the literature can work well for the registration of two infant brain MR images acquired at two arbitrary development phases, such as birth and one-year-old.

METHODS

To address these challenging issues, we propose a learning-based registration method, which can handle the anatomical structures and the appearance changes between the two infant brain MR images with possible time gap. Specifically, in the training stage, we employ a multioutput random forest regression and auto-context model to learn the evolution of anatomical shape and appearance from a training set of longitudinal infant images. To make the learning procedure more robust, we further harness the multimodal MR imaging information. Then, in the testing stage, for registering the two new infant images scanned at two different development phases, the learned model will be used to predict both the deformation field and appearance changes between the images under registration. After that, it becomes much easier to deploy any conventional image registration method to complete the remaining registration since the above-mentioned challenges for state-of-the-art registration methods have been well addressed.

RESULTS

We have applied our proposed registration method to intersubject registration of infant brain MR images acquired at 2-week-old, 3-month-old, 6-month-old, and 9-month-old with the images acquired at 12-month-old. Promising registration results have been achieved in terms of registration accuracy, compared with the counterpart nonlearning based registration methods.

CONCLUSIONS

The proposed new learning-based registration method have tackled the challenging issues in registering infant brain images acquired from the first year of life, by leveraging the multioutput random forest regression with auto-context model, which can learn the evolution of shape and appearance from a training set of longitudinal infant images. Thus, for the new infant image, its deformation field to the template and also its template-like appearances can be predicted by the learned models. We have extensively compared our method with state-of-the-art deformable registration methods, as well as multiple variants of our method, which show that our method can achieve higher accuracy even for the difficult cases with large appearance and shape changes between subject and template images.

Evaluation of Deformable Image Registration for Three-Dimensional Temporal Subtraction of Chest Computed Tomography Images

Purpose

To perform lung image registration for reducing misregistration artifacts on three-dimensional (3D) temporal subtraction of chest computed tomography (CT) images, in order to enhance temporal changes in lung lesions and evaluate these changes after deformable image registration (DIR).

Methods

In 10 cases, mutual information (MI) lung mask affine mapping combined with cross-correlation (CC) lung diffeomorphic mapping was used to implement lung volume registration. With advanced normalization tools (ANTs), we used greedy symmetric normalization (greedy SyN) as a transformation model, which involved MI-CC-SyN implementation. The resulting displacement fields were applied to warp the previous (moving) image, which was subsequently subtracted from the current (fixed) image to obtain the lung subtraction image.

Results

The average minimum and maximum log-Jacobians were 0.31 and 3.74, respectively. When considering 3D landmark distance, the root-mean-square error changed from an average of 20.82 mm for P_fixed to P_moving to 0.5 mm for P_warped to P_fixed. Clear shadows were observed as enhanced lung nodules and lesions in subtraction images. The lesion shadows showed lesion shrinkage changes over time. Lesion tissue morphology was maintained after DIR.

Conclusions

DIR (greedy SyN) effectively and accurately enhanced temporal changes in chest CT images and decreased misregistration artifacts in temporal subtraction images.

A SEMI-LAGRANGIAN TWO-LEVEL PRECONDITIONED NEWTON-KRYLOV SOLVER FOR CONSTRAINED DIFFEOMORPHIC IMAGE REGISTRATION

We propose an efficient numerical algorithm for the solution of diffeomorphic image registration problems. We use a variational formulation constrained by a partial differential equation (PDE), where the constraints are a scalar transport equation. We use a pseudospectral discretization in space and second-order accurate semi-Lagrangian time stepping scheme for the transport equations. We solve for a stationary velocity field using a preconditioned, globalized, matrix-free Newton-Krylov scheme. We propose and test a two-level Hessian preconditioner. We consider two strategies for inverting the preconditioner on the coarse grid: a nested preconditioned conjugate gradient method (exact solve) and a nested Chebyshev iterative method (inexact solve) with a fixed number of iterations. We test the performance of our solver in different synthetic and real-world two-dimensional application scenarios. We study grid convergence and computational efficiency of our new scheme. We compare the performance of our solver against our initial implementation that uses the same spatial discretization but a standard, explicit, second-order Runge-Kutta scheme for the numerical time integration of the transport equations and a single-level preconditioner. Our improved scheme delivers significant speedups over our original implementation. As a highlight, we observe a 20× speedup for a two dimensional, real world multi-subject medical image registration problem.

Primal-dual convex optimization in large deformation diffeomorphic metric mapping: LDDMM meets robust regularizers

This paper proposes a method for primal-dual convex optimization in variational large deformation diffeomorphic metric mapping problems formulated with robust regularizers and robust image similarity metrics. The method is based on Chambolle and Pock primal-dual algorithm for solving general convex optimization problems. Diagonal preconditioning is used to ensure the convergence of the algorithm to the global minimum. We consider three robust regularizers liable to provide acceptable results in diffeomorphic registration: Huber, V-Huber and total generalized variation. The Huber norm is used in the image similarity term. The primal-dual equations are derived for the stationary and the non-stationary parameterizations of diffeomorphisms. The resulting algorithms have been implemented for running in the GPU using Cuda. For the most memory consuming methods, we have developed a multi-GPU implementation. The GPU implementations allowed us to perform an exhaustive evaluation study in NIREP and LPBA40 databases. The experiments showed that, for all the considered regularizers, the proposed method converges to diffeomorphic solutions while better preserving discontinuities at the boundaries of the objects compared to baseline diffeomorphic registration methods. In most cases, the evaluation showed a competitive performance for the robust regularizers, close to the performance of the baseline diffeomorphic registration methods.

Computational and mathematical methods in brain atlasing

Brain atlases have a wide range of use from education to research to clinical applications. Mathematical methods as well as computational methods and tools play a major role in the process of brain atlas building and developing atlas-based applications. Computational methods and tools cover three areas: dedicated editors for brain model creation, brain navigators supporting multiple platforms, and atlas-assisted specific applications. Mathematical methods in atlas building and developing atlas-aided applications deal with problems in image segmentation, geometric body modelling, physical modelling, atlas-to-scan registration, visualisation, interaction and virtual reality. Here I overview computational and mathematical methods in atlas building and developing atlas-assisted applications, and share my contribution to and experience in this field.

Geometric Graph Matching Using Monte Carlo Tree Search

We present an efficient matching method for generalized geometric graphs. Such graphs consist of vertices in space connected by curves and can represent many real world structures such as road networks in remote sensing, or vessel networks in medical imaging. Graph matching can be used for very fast and possibly multimodal registration of images of these structures. We formulate the matching problem as a single player game solved using Monte Carlo Tree Search, which automatically balances exploring new possible matches and extending existing matches. Our method can handle partial matches, topological differences, geometrical distortion, does not use appearance information and does not require an initial alignment. Moreover, our method is very efficient-it can match graphs with thousands of nodes, which is an order of magnitude better than the best competing method, and the matching only takes a few seconds.

Propagation of registration uncertainty during multi-fraction cervical cancer brachytherapy

Multi-fraction cervical cancer brachytherapy is a form of image-guided radiotherapy that heavily relies on 3D imaging during treatment planning, delivery, and quality control. In this context, deformable image registration can increase the accuracy of dosimetric evaluations, provided that one can account for the uncertainties associated with the registration process. To enable such capability, we propose a mathematical framework that first estimates the registration uncertainty and subsequently propagates the effects of the computed uncertainties from the registration stage through to the visualizations, organ segmentations, and dosimetric evaluations. To ensure the practicality of our proposed framework in real world image-guided radiotherapy contexts, we implemented our technique via a computationally efficient and generalizable algorithm that is compatible with existing deformable image registration software. In our clinical context of fractionated cervical cancer brachytherapy, we perform a retrospective analysis on 37 patients and present evidence that our proposed methodology for computing and propagating registration uncertainties may be beneficial during therapy planning and quality control. Specifically, we quantify and visualize the influence of registration uncertainty on dosimetric analysis during the computation of the total accumulated radiation dose on the bladder wall. We further show how registration uncertainty may be leveraged into enhanced visualizations that depict the quality of the registration and highlight potential deviations from the treatment plan prior to the delivery of radiation treatment. Finally, we show that we can improve the transfer of delineated volumetric organ segmentation labels from one fraction to the next by encoding the computed registration uncertainties into the segmentation labels.

Deformable image registration applied to lung SBRT: Usefulness and limitations

Radiation therapy (RT) of the lung requires deformation analysis. Deformable image registration (DIR) is the fundamental method to quantify deformations for various applications: motion compensation, contour propagation, dose accumulation, etc. DIR is therefore unavoidable in lung RT. DIR algorithms have been studied for decades and are now available both within commercial and academic packages. However, they are complex and have limitations that every user must be aware of before clinical implementation. In this paper, the main applications of DIR for lung RT with their associated uncertainties and their limitations are reviewed.

A method to detect landmark pairs accurately between intra-patient volumetric medical images

PURPOSES

An image processing procedure was developed in this study to detect large quantity of landmark pairs accurately in pairs of volumetric medical images. The detected landmark pairs can be used to evaluate of deformable image registration (DIR) methods quantitatively.

METHODS

Landmark detection and pair matching were implemented in a Gaussian pyramid multi-resolution scheme. A 3D scale-invariant feature transform (SIFT) feature detection method and a 3D Harris-Laplacian corner detection method were employed to detect feature points, i.e., landmarks. A novel feature matching algorithm, Multi-Resolution Inverse-Consistent Guided Matching or MRICGM, was developed to allow accurate feature pairs matching. MRICGM performs feature matching using guidance by the feature pairs detected at the lower resolution stage and the higher confidence feature pairs already detected at the same resolution stage, while enforces inverse consistency.

RESULTS

The proposed feature detection and feature pair matching algorithms were optimized to process 3D CT and MRI images. They were successfully applied between the inter-phase abdomen 4DCT images of three patients, between the original and the re-scanned radiation therapy simulation CT images of two head-neck patients, and between inter-fractional treatment MRIs of two patients. The proposed procedure was able to successfully detect and match over 6300 feature pairs on average. The automatically detected landmark pairs were manually verified and the mismatched pairs were rejected. The automatic feature matching accuracy before manual error rejection was 99.4%. Performance of MRICGM was also evaluated using seven digital phantom datasets with known ground truth of tissue deformation. On average, 11855 feature pairs were detected per digital phantom dataset with TRE = 0.77 ± 0.72 mm.

CONCLUSION

A procedure was developed in this study to detect large number of landmark pairs accurately between two volumetric medical images. It allows a semi-automatic way to generate the ground truth landmark datasets that allow quantitatively evaluation of DIR algorithms for radiation therapy applications.

Demons registration for in vivo and deformable laser scanning confocal endomicroscopy

A critical effect found in noninvasive in vivo endomicroscopic imaging modalities is image distortions due to sporadic movement exhibited by living organisms. In three-dimensional confocal imaging, this effect results in a dataset that is tilted across deeper slices. Apart from that, the sequential flow of the imaging-processing pipeline restricts real-time adjustments due to the unavailability of information obtainable only from subsequent stages. To solve these problems, we propose an approach to render Demons-registered datasets as they are being captured, focusing on the coupling between registration and visualization. To improve the acquisition process, we also propose a real-time visual analytics tool, which complements the imaging pipeline and the Demons registration pipeline with useful visual indicators to provide real-time feedback for immediate adjustments. We highlight the problem of deformation within the visualization pipeline for object-ordered and image-ordered rendering. Visualizations of critical information including registration forces and partial renderings of the captured data are also presented in the analytics system. We demonstrate the advantages of the algorithmic design through experimental results with both synthetically deformed datasets and actual in vivo, time-lapse tissue datasets expressing natural deformations. Remarkably, this algorithm design is for embedded implementation in intelligent biomedical imaging instrumentation with customizable circuitry.

Quicksilver: Fast predictive image registration - A deep learning approach

This paper introduces Quicksilver, a fast deformable image registration method. Quicksilver registration for image-pairs works by patch-wise prediction of a deformation model based directly on image appearance. A deep encoder-decoder network is used as the prediction model. While the prediction strategy is general, we focus on predictions for the Large Deformation Diffeomorphic Metric Mapping (LDDMM) model. Specifically, we predict the momentum-parameterization of LDDMM, which facilitates a patch-wise prediction strategy while maintaining the theoretical properties of LDDMM, such as guaranteed diffeomorphic mappings for sufficiently strong regularization. We also provide a probabilistic version of our prediction network which can be sampled during the testing time to calculate uncertainties in the predicted deformations. Finally, we introduce a new correction network which greatly increases the prediction accuracy of an already existing prediction network. We show experimental results for uni-modal atlas-to-image as well as uni-/multi-modal image-to-image registrations. These experiments demonstrate that our method accurately predicts registrations obtained by numerical optimization, is very fast, achieves state-of-the-art registration results on four standard validation datasets, and can jointly learn an image similarity measure. Quicksilver is freely available as an open-source software.

Accuracy of eight deformable image registration (DIR) methods for tomotherapy megavoltage computed tomography (MVCT) images

Introduction

The application of deformable image registration (DIR) to megavoltage computed tomography (MVCT) images benefits adaptive radiotherapy. This study aims to quantify the accuracy of DIR for MVCT images when using different deformation methods assessed in a cubic phantom and nasopharyngeal carcinoma (NPC) patients.

Methods

In the control studies, the DIR accuracy in air‐tissue and tissue‐tissue interface areas was observed using twelve shapes of acrylic and tissue‐equivalent material inserted in the phantom. In the clinical studies, the 1st and 20th fraction MVCT images of seven NPC patients were used to evaluate application of DIR. The eight DIR methods used in the DIRART software varied in (i) transformation framework (asymmetric or symmetric), (ii) DIR registration algorithm (Demons or Optical Flow) and (iii) mapping direction (forward or backward). The accuracy of the methods was compared using an intensity‐based criterion (correlation coefficient, CC) and volume‐based criterion (Dice's similarity coefficient, DSC).

Results

The asymmetric transformation with Optical Flow showed the best performance for air‐tissue interface areas, with a mean CC and DSC of 0.97 ± 0.03 and 0.79 ± 0.11 respectively. The symmetric transformation with Optical Flow showed good agreement for tissue‐tissue interface areas with a CC of (0.99 ± 0.01) and DSC of (0.89 ± 0.03). The sequences of target domains were significantly different in tissue‐tissue interface areas.

Conclusions

The deformation method and interface area affected the accuracy of DIR. The validation techniques showed satisfactory volume matching of greater than 0.7 with DSC analysis. The methods can yield acceptable results for clinical applications.

A structural similarity-inspired performance assessment model for multisensor image registration algorithms

In order to assess the performance of multisensor image registration algorithms that are used in the multirobot information fusion, we propose a model based on structural similarity whose name is vision registration assessment model. First of all, this article introduces a new image concept named superimposed image for testing subjective and objective assessment methods. Therefore, we assess the superimposed image but not the registered image, which is different from previous image registration assessment methods that usually use reference and sensed images. Then, we calculate eight assessment indicators from different aspects for superimposed images. After that, vision registration assessment model fuses the eight indicators using canonical correlation analysis, which is used for evaluating the quality of an image registration results in different aspects. Finally, three kinds of images which include optical images, infrared images, and SAR images are used to test vision registration assessment model. After evaluating three state-of-the-art image registration methods, experiments indict that the proposed structural similarity-motivated model achieved almost same evaluation results with that of the human object with the consistency rate of 98.3%, which shows that vision registration assessment model is efficient and robust for evaluating multisensor image registration algorithms. Moreover, vision registration assessment model is independent of the emotional factors and outside environment, which is different from the human.

A hybrid patient-specific biomechanical model based image registration method for the motion estimation of lungs

This paper presents a new hybrid biomechanical model-based non-rigid image registration method for lung motion estimation. In the proposed method, a patient-specific biomechanical modelling process captures major physically realistic deformations with explicit physical modelling of sliding motion, whilst a subsequent non-rigid image registration process compensates for small residuals. The proposed algorithm was evaluated with 10 4D CT datasets of lung cancer patients. The target registration error (TRE), defined as the Euclidean distance of landmark pairs, was significantly lower with the proposed method (TRE = 1.37 mm) than with biomechanical modelling (TRE = 3.81 mm) and intensity-based image registration without specific considerations for sliding motion (TRE = 4.57 mm). The proposed method achieved a comparable accuracy as several recently developed intensity-based registration algorithms with sliding handling on the same datasets. A detailed comparison on the distributions of TREs with three non-rigid intensity-based algorithms showed that the proposed method performed especially well on estimating the displacement field of lung surface regions (mean TRE = 1.33 mm, maximum TRE = 5.3 mm). The effects of biomechanical model parameters (such as Poisson's ratio, friction and tissue heterogeneity) on displacement estimation were investigated. The potential of the algorithm in optimising biomechanical models of lungs through analysing the pattern of displacement compensation from the image registration process has also been demonstrated.

Randomly Perturbed B-Splines for Nonrigid Image Registration

B-splines are commonly utilized to construct the transformation model in free-form deformation (FFD) based registration. B-splines become smoother with increasing spline order. However, a higher-order B-spline requires a larger support region involving more control points, which means higher computational cost. In general, the third-order B-spline is considered as a good compromise between spline smoothness and computational cost. A lower-order function is seldom used to construct the transformation model for registration since it is less smooth. In this research, we investigated whether lower-order B-spline functions can be utilized for more efficient registration, while preserving smoothness of the deformation by using a novel random perturbation technique. With the proposed perturbation technique, the expected value of the cost function given probability density function (PDF) of the perturbation is minimized by a stochastic gradient descent optimization. Extensive experiments on 2D synthetically deformed brain images, and real 3D lung and brain scans demonstrated that the novel randomly perturbed free-form deformation (RPFFD) approach improves the registration accuracy and transformation smoothness. Meanwhile, lower-order RPFFD methods reduce the computational cost substantially.

Deformable image registration in radiation therapy

The number of imaging data sets has significantly increased during radiation treatment after introducing a diverse range of advanced techniques into the field of radiation oncology. As a consequence, there have been many studies proposing meaningful applications of imaging data set use. These applications commonly require a method to align the data sets at a reference. Deformable image registration (DIR) is a process which satisfies this requirement by locally registering image data sets into a reference image set. DIR identifies the spatial correspondence in order to minimize the differences between two or among multiple sets of images. This article describes clinical applications, validation, and algorithms of DIR techniques. Applications of DIR in radiation treatment include dose accumulation, mathematical modeling, automatic segmentation, and functional imaging. Validation methods discussed are based on anatomical landmarks, physical phantoms, digital phantoms, and per application purpose. DIR algorithms are also briefly reviewed with respect to two algorithmic components: similarity index and deformation models.

Nonrigid Registration of Prostate Diffusion-Weighted MRI

Motion and deformation are common in prostate diffusion-weighted magnetic resonance imaging (DWI) during acquisition. These misalignments lead to errors in estimating an apparent diffusion coefficient (ADC) map fitted with DWI. To address this problem, we propose an image registration algorithm to align the prostate DWI and improve ADC map. First, we apply affine transformation to DWI to correct intraslice motions. Then, nonrigid registration based on free-form deformation (FFD) is used to compensate for intraimage deformations. To evaluate the influence of the proposed algorithm on ADC values, we perform statistical experiments in three schemes: no processing of the DWI, with the affine transform approach, and with FFD. The experimental results show that our proposed algorithm can correct the misalignment of prostate DWI and decrease the artifacts of ROI in the ADC maps. These ADC maps thus obtain sharper contours of lesions, which are helpful for improving the diagnosis and clinical staging of prostate cancer.

Deep Learning in Medical Image Analysis

This review covers computer-assisted analysis of images in the field of medical imaging. Recent advances in machine learning, especially with regard to deep learning, are helping to identify, classify, and quantify patterns in medical images. At the core of these advances is the ability to exploit hierarchical feature representations learned solely from data, instead of features designed by hand according to domain-specific knowledge. Deep learning is rapidly becoming the state of the art, leading to enhanced performance in various medical applications. We introduce the fundamentals of deep learning methods and review their successes in image registration, detection of anatomical and cellular structures, tissue segmentation, computer-aided disease diagnosis and prognosis, and so on. We conclude by discussing research issues and suggesting future directions for further improvement.

Stereo vision-based tracking of soft tissue motion with application to online ablation control in laser microsurgery

Recent research has revealed that image-based methods can enhance accuracy and safety in laser microsurgery. In this study, non-rigid tracking using surgical stereo imaging and its application to laser ablation is discussed. A recently developed motion estimation framework based on piecewise affine deformation modeling is extended by a mesh refinement step and considering texture information. This compensates for tracking inaccuracies potentially caused by inconsistent feature matches or drift. To facilitate online application of the method, computational load is reduced by concurrent processing and affine-invariant fusion of tracking and refinement results. The residual latency-dependent tracking error is further minimized by Kalman filter-based upsampling, considering a motion model in disparity space. Accuracy is assessed in laparoscopic, beating heart, and laryngeal sequences with challenging conditions, such as partial occlusions and significant deformation. Performance is compared with that of state-of-the-art methods. In addition, the online capability of the method is evaluated by tracking two motion patterns performed by a high-precision parallel-kinematic platform. Related experiments are discussed for tissue substitute and porcine soft tissue in order to compare performances in an ideal scenario and in a setup mimicking clinical conditions. Regarding the soft tissue trial, the tracking error can be significantly reduced from 0.72 mm to below 0.05 mm with mesh refinement. To demonstrate online laser path adaptation during ablation, the non-rigid tracking framework is integrated into a setup consisting of a surgical Er:YAG laser, a three-axis scanning unit, and a low-noise stereo camera. Regardless of the error source, such as laser-to-camera registration, camera calibration, image-based tracking, and scanning latency, the ablation root mean square error is kept below 0.21 mm when the sample moves according to the aforementioned patterns. Final experiments regarding motion-compensated laser ablation of structurally deforming tissue highlight the potential of the method for vision-guided laser surgery.

Using clinically acquired MRI to construct age-specific ADC atlases: Quantifying spatiotemporal ADC changes from birth to 6-year old

Diffusion imaging is critical for detecting acute brain injury. However, normal apparent diffusion coefficient (ADC) maps change rapidly in early childhood, making abnormality detection difficult. In this article, we explored clinical PACS and electronic healthcare records (EHR) to create age-specific ADC atlases for clinical radiology reference. Using the EHR and three rounds of multiexpert reviews, we found ADC maps from 201 children 0-6 years of age scanned between 2006 and 2013 who had brain MRIs with no reported abnormalities and normal clinical evaluations 2+ years later. These images were grouped in 10 age bins, densely sampling the first 1 year of life (5 bins, including neonates and 4 quarters) and representing the 1-6 year age range (an age bin per year). Unbiased group-wise registration was used to construct ADC atlases for 10 age bins. We used the atlases to quantify (a) cross-sectional normative ADC variations; (b) spatiotemporal heterogeneous ADC changes; and (c) spatiotemporal heterogeneous volumetric changes. The quantified age-specific whole-brain and region-wise ADC values were compared to those from age-matched individual subjects in our study and in multiple existing independent studies. The significance of this study is that we have shown that clinically acquired images can be used to construct normative age-specific atlases. These first of their kind age-specific normative ADC atlases quantitatively characterize changes of myelination-related water diffusion in the first 6 years of life. The quantified voxel-wise spatiotemporal ADC variations provide standard references to assist radiologists toward more objective interpretation of abnormalities in clinical images. Our atlases are available at https://www.nitrc.org/projects/mgh_adcatlases. Hum Brain Mapp 38:3052-3068, 2017. © 2017 Wiley Periodicals, Inc.

Three-dimensional finite-element based deformable image registration for evaluation of pleural cavity irradiation during photodynamic therapy

PURPOSE

Photodynamic therapy (PDT) is used after surgical resection to treat the microscopic disease for malignant pleural mesothelioma and to increase survival rates. As accurate light delivery is imperative to PDT efficacy, the deformation of the pleural volume during the surgery is studied on its impact on the delivered light fluence. In this study, a three-dimensional finite element-based (3D FEM) deformable image registration is proposed to directly match the volume of lung to the volume of pleural cavity obtained during PDT to have accurate representation of the light fluence accumulated in the lung, heart and liver (organs-at-risk) during treatment.

METHODS

A wand, comprised of a modified endotrachial tube filled with Intralipid and an optical fiber inside the tube, is used to deliver the treatment light. The position of the treatment is tracked using an optical tracking system with an attachment comprised of nine reflective passive markers that are seen by an infrared camera-based navigation system. This information is used to obtain the surface contours of the plural cavity and the cumulative light fluence on every point of the cavity surface that is being treated. The lung, heart, and liver geometry are also reconstructed from a series of computed tomography (CT) scans of the organs acquired in the same patient before and after the surgery. The contours obtained with the optical tracking system and CTs are imported into COMSOL Multiphysics, where the 3D FEM-based deformable image registration is obtained. The delivered fluence values are assigned to the respective positions (x, y, and z) on the optical tracking contour. The optical tracking contour is considered as the reference, and the CT contours are used as the target, which will be deformed. The data from three patients formed the basis for this study.

RESULTS

The physical correspondence between the CT and optical tracking geometries, taken at different times, from different imaging devices was established using the 3D FEM-based image deformable registration. The volume of lung was matched to the volume of pleural cavity and the distribution of light fluence on the surface of the heart, liver and deformed lung volumes was obtained.

CONCLUSION

The method used is appropriate for analyzing problems over complicated domains, such as when the domain changes (as in a solid-state reaction with a moving boundary), when the desired precision varies over the entire domain, or when the solution lacks smoothness. Implementing this method in real-time for clinical applications and in situ monitoring of the under- or over- exposed regions to light during PDT can significantly improve the treatment for mesothelioma.

REtroSpective Evaluation of Cerebral Tumors (RESECT): A clinical database of pre-operative MRI and intra-operative ultrasound in low-grade glioma surgeries

PURPOSE

The advancement of medical image processing techniques, such as image registration, can effectively help improve the accuracy and efficiency of brain tumor surgeries. However, it is often challenging to validate these techniques with real clinical data due to the rarity of such publicly available repositories.

ACQUISITION AND VALIDATION METHODS

Pre-operative magnetic resonance images (MRI), and intra-operative ultrasound (US) scans were acquired from 23 patients with low-grade gliomas who underwent surgeries at St. Olavs University Hospital between 2011 and 2016. Each patient was scanned by Gadolinium-enhanced T1w and T2-FLAIR MRI protocols to reveal the anatomy and pathology, and series of B-mode ultrasound images were obtained before, during, and after tumor resection to track the surgical progress and tissue deformation. Retrospectively, corresponding anatomical landmarks were identified across US images of different surgical stages, and between MRI and US, and can be used to validate image registration algorithms. Quality of landmark identification was assessed with intra- and inter-rater variability.

DATA FORMAT AND ACCESS

In addition to co-registered MRIs, each series of US scans are provided as a reconstructed 3D volume. All images are accessible in MINC2 and NIFTI formats, and the anatomical landmarks were annotated in MNI tag files. Both the imaging data and the corresponding landmarks are available online as the RESECT database at https://archive.norstore.no (search for "RESECT").

POTENTIAL IMPACT

The proposed database provides real high-quality multi-modal clinical data to validate and compare image registration algorithms that can potentially benefit the accuracy and efficiency of brain tumor resection. Furthermore, the database can also be used to test other image processing methods and neuro-navigation software platforms.

Mid-space-independent deformable image registration

Aligning images in a mid-space is a common approach to ensuring that deformable image registration is symmetric - that it does not depend on the arbitrary ordering of the input images. The results are, however, generally dependent on the mathematical definition of the mid-space. In particular, the set of possible solutions is typically restricted by the constraints that are enforced on the transformations to prevent the mid-space from drifting too far from the native image spaces. The use of an implicit atlas has been proposed as an approach to mid-space image registration. In this work, we show that when the atlas is aligned to each image in the native image space, the data term of implicit-atlas-based deformable registration is inherently independent of the mid-space. In addition, we show that the regularization term can be reformulated independently of the mid-space as well. We derive a new symmetric cost function that only depends on the transformation morphing the images to each other, rather than to the atlas. This eliminates the need for anti-drift constraints, thereby expanding the space of allowable deformations. We provide an implementation scheme for the proposed framework, and validate it through diffeomorphic registration experiments on brain magnetic resonance images.

Cardiac Image Reconstruction via Nonlinear Motion Correction Based on Partial Angle Reconstructed Images

Even though the X-ray Computed Tomography (CT) scan is considered suitable for fast imaging, motion-artifact-free cardiac imaging is still an important issue, because the gantry rotation speed is not fast enough compared with the heart motion. To obtain a heart image with less motion artifacts, a motion estimation (ME) and motion compensation (MC) approach is usually adopted. In this paper, we propose an ME/MC algorithm that can estimate a nonlinear heart motion model from a sinogram with a rotation angle of less than 360°. In this algorithm, we first assume the heart motion to be nonrigid but linear, and thereby estimate an initial 4-D motion vector field (MVF) during a half rotation by using conjugate partial angle reconstructed images, as in our previous ME/MC algorithm. We then refine the MVF to determine a more accurate nonlinear MVF by maximizing the information potential of a motion-compensated image. Finally, MC is performed by incorporating the determined MVF into the image reconstruction process, and a time-resolved heart image is obtained. By using a numerical phantom, a physical cardiac phantom, and an animal data set, we demonstrate that the proposed algorithm can noticeably improve the image quality by reducing motion artifacts throughout the image.

Thoracic CT-MRI coregistration for regional pulmonary structure-function measurements of obstructive lung disease

PURPOSE

Recent pulmonary imaging research has revealed that in patients with chronic obstructive pulmonary disease (COPD) and asthma, structural and functional abnormalities are spatially heterogeneous. This novel information may help optimize treatment in individual patients, monitor interventional efficacy, and develop new treatments. Moreover, by automating the measurement of regional biomarkers for the 19 different anatomical lung segments, there is an opportunity to embed imaging biomarkers into clinically acceptable clinical workflows and improve lung disease clinical care. Therefore, to exploit the regional structure-function information provided by thoracic imaging, and as a first step toward this goal, our objective was to develop a fully automated registration pipeline for thoracic x-ray computed tomography (CT) and inhaled gas functional magnetic resonance imaging (MRI) whole lung and segmental structure-function biomarkers.

METHODS

Thirty-five patients including 15 severe, poorly controlled asthmatics and 20 COPD patients [classified according to the global initiative for chronic obstructive lung disease (GOLD) criteria)] provided written informed consent to a study protocol approved by Health Canada and underwent pulmonary function tests, MRI, and CT during a single 2-hour visit. Using this diverse patient dataset, we developed and evaluated a joint deformable registration approach to simultaneously coregister CT with both ¹ H and ³ He MRI by enforcing the similarity of the deformation fields from the two individual registrations. We derived a simpler model that was equivalent to the original challenging optimization problem through variational analysis and the simpler model gave rise to an efficient numerical solver that was parallelized on a graphics processing unit. The coregistered CT-³ He MRI and whole lung/segmental lung masks were used to generate whole lung and segmental ³ He MRI ventilation defect percent (VDP). To estimate fiducial localization reproducibility, a single observer manually identified 109 pairs of CT and ³ He MRI fiducials for 35 patient images on five separate occasions and determined the fiducial localization error (FLE). CT-³ He MRI registration accuracy was evaluated using the target registration error (TRE). Whole lung VDP generated using the algorithm was compared with VDP generated using a previously validated semiautomated approach and computational efficiency was evaluated using run time.

RESULTS

In 35 patients including 15 with severe asthma and 20 with COPD, mean forced expiratory volume in 1 s (FEV₁ ) was 63±24%_pred and FEV₁ /forced vital capacity (FVC) was 54 ± 17%. FLE was 0.16 mm and 0.34 mm for ³ He MRI and CT, respectively. TRE was 4.5 ± 2.0 mm, 4.0 ± 1.7 mm, 4.8 ± 2.3 mm for asthma, COPD GOLD II, and GOLD III groups, respectively, with a mean of 4.4 ± 2.0 mm for the entire dataset. TRE was significantly improved for joint CT-¹ H/³ He MRI registration compared with CT-¹ H MRI rigid registration (P < 0.0001). Whole lung VDP generated using the pipeline was not significantly different (P = 0.37) compared to a semiautomated method with which it was strongly correlated (r = 0.93, P < 0.0001). The fully automated pipeline required 11 ± 0.4 min to generate whole lung and segmental VDP.

CONCLUSIONS

For a diverse group of patients with COPD and asthma, whole lung and segmental VDP was measured using an automated lung image analysis pipeline which provides a way to incorporate lung functional biomarkers into clinical research and patient care.