Deformable medical image registration: a survey

Detection of Focal Longitudinal Changes in the Brain by Subtraction of MR Images

BACKGROUND AND PURPOSE

The detection of new subtle brain pathology on MR imaging is a time-consuming and error-prone task for the radiologist. This article introduces and evaluates an image-registration and subtraction method for highlighting small changes in the brain with a view to minimizing the risk of missed pathology and reducing fatigue.

MATERIALS AND METHODS

We present a fully automated algorithm for highlighting subtle changes between multiple serially acquired brain MR images with a novel approach to registration and MR imaging bias field correction. The method was evaluated for the detection of new lesions in 77 patients undergoing cardiac surgery, by using pairs of fluid-attenuated inversion recovery MR images acquired 1-2 weeks before the operation and 6-8 weeks postoperatively. Three radiologists reviewed the images.

RESULTS

On the basis of qualitative comparison of pre- and postsurgery FLAIR images, radiologists identified 37 new ischemic lesions in 22 patients. When these images were accompanied by a subtraction image, 46 new ischemic lesions were identified in 26 patients. After we accounted for interpatient and interradiologist variability using a multilevel statistical model, the likelihood of detecting a lesion was 2.59 (95% CI, 1.18-5.67) times greater when aided by the subtraction algorithm (P = .017). Radiologists also reviewed the images significantly faster (P < .001) by using the subtraction image (mean, 42 seconds; 95% CI, 29-60 seconds) than through qualitative assessment alone (mean, 66 seconds; 95% CI, 46-96 seconds).

CONCLUSIONS

Use of this new subtraction algorithm would result in considerable savings in the time required to review images and in improved sensitivity to subtle focal pathology.

Nonrigid registration with corresponding points constraint for automatic segmentation of cardiac DSCT images

Background

Dual-source computed tomography (DSCT) is a very effective way for diagnosis and treatment of heart disease. The quantitative information of spatiotemporal DSCT images can be important for the evaluation of cardiac function. To avoid the shortcoming of manual delineation, it is imperative to develop an automatic segmentation technique for 4D cardiac images.

Methods

In this paper, we implement the heart segmentation-propagation framework based on nonrigid registration. The corresponding points of anatomical substructures are extracted by using the extension of n-dimensional scale invariant feature transform method. They are considered as a constraint term of nonrigid registration using the free-form deformation, in order to restrain the large variations and boundary ambiguity between subjects.

Results

We validate our method on 15 patients at ten time phases. Atlases are constructed by the training dataset from ten patients. On the remaining data the median overlap is shown to improve significantly compared to original mutual information, in particular from 0.4703 to 0.5015 (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ p = 5.0 \times 10^{ - 4} $$\end{document}p=5.0×10-4) for left ventricle myocardium and from 0.6307 to 0.6519 (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ p = 6.0 \times 10^{ - 4} $$\end{document}p=6.0×10-4) for right atrium.

Conclusions

The proposed method outperforms standard mutual information of intensity only. The segmentation errors had been significantly reduced at the left ventricle myocardium and the right atrium. The mean surface distance of using our framework is around 1.73 mm for the whole heart.

Deformable Registration-Based Super-resolution for Isotropic Reconstruction of 4-D MRI Volumes

Multi-plane super-resolution (SR) has been widely employed for resolution improvement of MR images. However, this has mostly been limited to MRI acquisitions with rigid motion. In cases of non-rigid motion, volumes are usually pre-registered using deformable registration methods before SR reconstruction. The pre-registered images are then used as input for the SR reconstruction. Since deformable registration involves smoothening of the inputs, using pre-registered inputs could lead to loss in information in SR reconstructions. Additionally, any registration errors present in pre-registered inputs could propagate throughout SR reconstructions leading to error accumulation. To address these limitations, in this study, we propose a deformable registration-based super-resolution reconstruction (DIRSR) reconstruction, which handles deformable registration as part of super-resolution. This approach has been demonstrated using 12 synthetic 4-D MRI lung datasets created using single plane (coronal) datasets of six patients and multi-plane (coronal and axial) 4-D lung MRI dataset of one patient. From our evaluation, DIRSR reconstructions are sharper and well aligned compared to reconstructions using SR of pre-registered inputs and rigid-registration SR. MSE, SNR and SSIM evaluations also indicate better reconstruction quality from DIRSR compared to reconstructions from SR of pre-registered inputs (p-value less than 0.0001). In conclusion, we found superior isotropic reconstructions of 4-D MR datasets from DIRSR reconstructions, which could benefit volumetric MR analyses.

Fast and robust multimodal image registration using a local derivative pattern

PURPOSE

Deformable multimodal image registration, which can benefit radiotherapy and image guided surgery by providing complementary information, remains a challenging task in the medical image analysis field due to the difficulty of defining a proper similarity measure. This article presents a novel, robust and fast binary descriptor, the discriminative local derivative pattern (dLDP), which is able to encode images of different modalities into similar image representations.

METHODS

dLDP calculates a binary string for each voxel according to the pattern of intensity derivatives in its neighborhood. The descriptor similarity is evaluated using the Hamming distance, which can be efficiently computed, instead of conventional L1 or L2 norms. For the first time, we validated the effectiveness and feasibility of the local derivative pattern for multimodal deformable image registration with several multi-modal registration applications.

RESULTS

dLDP was compared with three state-of-the-art methods in artificial image and clinical settings. In the experiments of deformable registration between different magnetic resonance imaging (MRI) modalities from BrainWeb, between computed tomography and MRI images from patient data, and between MRI and ultrasound images from BITE database, we show our method outperforms localized mutual information and entropy images in terms of both accuracy and time efficiency. We have further validated dLDP for the deformable registration of preoperative MRI and three-dimensional intraoperative ultrasound images. Our results indicate that dLDP reduces the average mean target registration error from 4.12 mm to 2.30 mm. This accuracy is statistically equivalent to the accuracy of the state-of-the-art methods in the study; however, in terms of computational complexity, our method significantly outperforms other methods and is even comparable to the sum of the absolute difference.

CONCLUSIONS

The results reveal that dLDP can achieve superior performance regarding both accuracy and time efficiency in general multimodal image registration. In addition, dLDP also indicates the potential for clinical ultrasound guided intervention.

Learning-based deformable image registration for infant MR images in the first year of life

PURPOSE

Many brain development studies have been devoted to investigate dynamic structural and functional changes in the first year of life. To quantitatively measure brain development in such a dynamic period, accurate image registration for different infant subjects with possible large age gap is of high demand. Although many state-of-the-art image registration methods have been proposed for young and elderly brain images, very few registration methods work for infant brain images acquired in the first year of life, because of (a) large anatomical changes due to fast brain development and (b) dynamic appearance changes due to white-matter myelination.

METHODS

To address these two difficulties, we propose a learning-based registration method to not only align the anatomical structures but also alleviate the appearance differences between two arbitrary infant MR images (with large age gap) by leveraging the regression forest to predict both the initial displacement vector and appearance changes. Specifically, in the training stage, two regression models are trained separately, with (a) one model learning the relationship between local image appearance (of one development phase) and its displacement toward the template (of another development phase) and (b) another model learning the local appearance changes between the two brain development phases. Then, in the testing stage, to register a new infant image to the template, we first predict both its voxel-wise displacement and appearance changes by the two learned regression models. Since such initializations can alleviate significant appearance and shape differences between new infant image and the template, it is easy to just use a conventional registration method to refine the remaining registration.

RESULTS

We apply our proposed registration method to align 24 infant subjects at five different time points (i.e., 2-week-old, 3-month-old, 6-month-old, 9-month-old, and 12-month-old), and achieve more accurate and robust registration results, compared to the state-of-the-art registration methods.

CONCLUSIONS

The proposed learning-based registration method addresses the challenging task of registering infant brain images and achieves higher registration accuracy compared with other counterpart registration methods.

Augmented reality for breast tumors visualization

3D visualization of breast tumors are shown to be effective by previous studies. In this paper, we introduce a new augmented reality application that can help doctors and surgeons to have a more accurate visualization of breast tumors; this system uses a marker-based image-processing technique to render a 3D model of the tumors on the body. The model can be created using a combination of breast 3D mammography by experts. We have tested the system using an Android smartphone and a head-mounted device. This proof of concept can be useful for oncologists to have a more effective screening, and surgeons to plan the surgery.

Automatic error correction using adaptive weighting for vessel-based deformable image registration

In this paper, we extend our previous work on deformable image registration to inhomogenous tissues. Inhomogenous tissues include the tissues with embedded tumors, which is common in clinical applications. It is a very challenging task since the registration method that works for homogenous tissues may not work well with inhomogenous tissues. The maximum error normally occurs in the regions with tumors and often exceeds the acceptable error threshold. In this paper, we propose a new error correction method with adaptive weighting to reduce the maximum registration error. Our previous fast deformable registration method is used in the inner loop. We have also proposed a new evaluation metric average error of deformation field (AEDF) to evaluate the registration accuracy in regions between vessels and bifurcation points. We have validated the proposed method using liver MR images from human subjects. AEDF results show that the proposed method can greatly reduce the maximum registration errors when compared with the previous method with no adaptive weighting. The proposed method has the potential to be used in clinical applications to reduce registration errors in regions with tumors.

Nonlinear image registration with bidirectional metric and reciprocal regularization

Nonlinear registration is an important technique to align two different images and widely applied in medical image analysis. In this paper, we develop a novel nonlinear registration framework based on the diffeomorphic demons, where a reciprocal regularizer is introduced to assume that the deformation between two images is an exact diffeomorphism. In detail, first, we adopt a bidirectional metric to improve the symmetry of the energy functional, whose variables are two reciprocal deformations. Secondly, we slack these two deformations into two independent variables and introduce a reciprocal regularizer to assure the deformations being the exact diffeomorphism. Then, we utilize an alternating iterative strategy to decouple the model into two minimizing subproblems, where a new closed form for the approximate velocity of deformation is calculated. Finally, we compare our proposed algorithm on two data sets of real brain MR images with two relative and conventional methods. The results validate that our proposed method improves accuracy and robustness of registration, as well as the gained bidirectional deformations are actually reciprocal.

Personalized heterogeneous deformable model for fast volumetric registration

BACKGROUND

Biomechanical deformable volumetric registration can help improve safety of surgical interventions by ensuring the operations are extremely precise. However, this technique has been limited by the accuracy and the computational efficiency of patient-specific modeling.

METHODS

This study presents a tissue-tissue coupling strategy based on penalty method to model the heterogeneous behavior of deformable body, and estimate the personalized tissue-tissue coupling parameters in a data-driven way. Moreover, considering that the computational efficiency of biomechanical model is highly dependent on the mechanical resolution, a practical coarse-to-fine scheme is proposed to increase runtime efficiency. Particularly, a detail enrichment database is established in an offline fashion to represent the mapping relationship between the deformation results of high-resolution hexahedral mesh extracted from the raw medical data and a newly constructed low-resolution hexahedral mesh. At runtime, the mechanical behavior of human organ under interactions is simulated with this low-resolution hexahedral mesh, then the microstructures are synthesized in virtue of the detail enrichment database.

RESULTS

The proposed method is validated by volumetric registration in an abdominal phantom compression experiments. Our personalized heterogeneous deformable model can well describe the coupling effects between different tissues of the phantom. Compared with high-resolution heterogeneous deformable model, the low-resolution deformable model with our detail enrichment database can achieve 9.4× faster, and the average target registration error is 3.42 mm, which demonstrates that the proposed method shows better volumetric registration performance than state-of-the-art.

CONCLUSIONS

Our framework can well balance the precision and efficiency, and has great potential to be adopted in the practical augmented reality image-guided robotic systems.

Classification of brain disease in magnetic resonance images using two-stage local feature fusion

BACKGROUND

Many classification methods have been proposed based on magnetic resonance images. Most methods rely on measures such as volume, the cerebral cortical thickness and grey matter density. These measures are susceptible to the performance of registration and limited in representation of anatomical structure. This paper proposes a two-stage local feature fusion method, in which deformable registration is not desired and anatomical information is represented from moderate scale.

METHODS

Keypoints are firstly extracted from scale-space to represent anatomical structure. Then, two kinds of local features are calculated around the keypoints, one for correspondence and the other for representation. Scores are assigned for keypoints to quantify their effect in classification. The sum of scores for all effective keypoints is used to determine which group the test subject belongs to.

RESULTS

We apply this method to magnetic resonance images of Alzheimer's disease and Parkinson's disease. The advantage of local feature in correspondence and representation contributes to the final classification. With the help of local feature (Scale Invariant Feature Transform, SIFT) in correspondence, the performance becomes better. Local feature (Histogram of Oriented Gradient, HOG) extracted from 16×16 cell block obtains better results compared with 4×4 and 8×8 cell block.

DISCUSSION

This paper presents a method which combines the effect of SIFT descriptor in correspondence and the representation ability of HOG descriptor in anatomical structure. This method has the potential in distinguishing patients with brain disease from controls.

Quantifying Registration Uncertainty With Sparse Bayesian Modelling

We investigate uncertainty quantification under a sparse Bayesian model of medical image registration. Bayesian modelling has proven powerful to automate the tuning of registration hyperparameters, such as the trade-off between the data and regularization functionals. Sparsity-inducing priors have recently been used to render the parametrization itself adaptive and data-driven. The sparse prior on transformation parameters effectively favors the use of coarse basis functions to capture the global trends in the visible motion while finer, highly localized bases are introduced only in the presence of coherent image information and motion. In earlier work, approximate inference under the sparse Bayesian model was tackled in an efficient Variational Bayes (VB) framework. In this paper we are interested in the theoretical and empirical quality of uncertainty estimates derived under this approximate scheme vs. under the exact model. We implement an (asymptotically) exact inference scheme based on reversible jump Markov Chain Monte Carlo (MCMC) sampling to characterize the posterior distribution of the transformation and compare the predictions of the VB and MCMC based methods. The true posterior distribution under the sparse Bayesian model is found to be meaningful: orders of magnitude for the estimated uncertainty are quantitatively reasonable, the uncertainty is higher in textureless regions and lower in the direction of strong intensity gradients.

Cross contrast multi-channel image registration using image synthesis for MR brain images

Multi-modal deformable registration is important for many medical image analysis tasks such as atlas alignment, image fusion, and distortion correction. Whereas a conventional method would register images with different modalities using modality independent features or information theoretic metrics such as mutual information, this paper presents a new framework that addresses the problem using a two-channel registration algorithm capable of using mono-modal similarity measures such as sum of squared differences or cross-correlation. To make it possible to use these same-modality measures, image synthesis is used to create proxy images for the opposite modality as well as intensity-normalized images from each of the two available images. The new deformable registration framework was evaluated by performing intra-subject deformation recovery, intra-subject boundary alignment, and inter-subject label transfer experiments using multi-contrast magnetic resonance brain imaging data. Three different multi-channel registration algorithms were evaluated, revealing that the framework is robust to the multi-channel deformable registration algorithm that is used. With a single exception, all results demonstrated improvements when compared against single channel registrations using the same algorithm with mutual information.

A Biomechanical Modeling Guided CBCT Estimation Technique

Two-dimensional-to-three-dimensional (2D-3D) deformation has emerged as a new technique to estimate cone-beam computed tomography (CBCT) images. The technique is based on deforming a prior high-quality 3D CT/CBCT image to form a new CBCT image, guided by limited-view 2D projections. The accuracy of this intensity-based technique, however, is often limited in low-contrast image regions with subtle intensity differences. The solved deformation vector fields (DVFs) can also be biomechanically unrealistic. To address these problems, we have developed a biomechanical modeling guided CBCT estimation technique (Bio-CBCT-est) by combining 2D-3D deformation with finite element analysis (FEA)-based biomechanical modeling of anatomical structures. Specifically, Bio-CBCT-est first extracts the 2D-3D deformation-generated displacement vectors at the high-contrast anatomical structure boundaries. The extracted surface deformation fields are subsequently used as the boundary conditions to drive structure-based FEA to correct and fine-tune the overall deformation fields, especially those at low-contrast regions within the structure. The resulting FEA-corrected deformation fields are then fed back into 2D-3D deformation to form an iterative loop, combining the benefits of intensity-based deformation and biomechanical modeling for CBCT estimation. Using eleven lung cancer patient cases, the accuracy of the Bio-CBCT-est technique has been compared to that of the 2D-3D deformation technique and the traditional CBCT reconstruction techniques. The accuracy was evaluated in the image domain, and also in the DVF domain through clinician-tracked lung landmarks.

Deformable image registration for tissues with large displacements

Image registration for internal organs and soft tissues is considered extremely challenging due to organ shifts and tissue deformation caused by patients' movements such as respiration and repositioning. In our previous work, we proposed a fast registration method for deformable tissues with small rotations. We extend our method to deformable registration of soft tissues with large displacements. We analyzed the deformation field of the liver by decomposing the deformation into shift, rotation, and pure deformation components and concluded that in many clinical cases, the liver deformation contains large rotations and small deformations. This analysis justified the use of linear elastic theory in our image registration method. We also proposed a region-based neuro-fuzzy transformation model to seamlessly stitch together local affine and local rigid models in different regions. We have performed the experiments on a liver MRI image set and showed the effectiveness of the proposed registration method. We have also compared the performance of the proposed method with the previous method on tissues with large rotations and showed that the proposed method outperformed the previous method when dealing with the combination of pure deformation and large rotations. Validation results show that we can achieve a target registration error of [Formula: see text] and an average centerline distance error of [Formula: see text]. The proposed technique has the potential to significantly improve registration capabilities and the quality of intraoperative image guidance. To the best of our knowledge, this is the first time that the complex displacement of the liver is explicitly separated into local pure deformation and rigid motion.

Deformable registration of CT and cone-beam CT with local intensity matching

Cone-beam CT (CBCT) is a widely used intra-operative imaging modality in image-guided radiotherapy and surgery. A short scan followed by a filtered-backprojection is typically used for CBCT reconstruction. While data on the mid-plane (plane of source-detector rotation) is complete, off-mid-planes undergo different information deficiency and the computed reconstructions are approximate. This causes different reconstruction artifacts at off-mid-planes depending on slice locations, and therefore impedes accurate registration between CT and CBCT. In this paper, we propose a method to accurately register CT and CBCT by iteratively matching local CT and CBCT intensities. We correct CBCT intensities by matching local intensity histograms slice by slice in conjunction with intensity-based deformable registration. The correction-registration steps are repeated in an alternating way until the result image converges. We integrate the intensity matching into three different deformable registration methods, B-spline, demons, and optical flow that are widely used for CT-CBCT registration. All three registration methods were implemented on a graphics processing unit for efficient parallel computation. We tested the proposed methods on twenty five head and neck cancer cases and compared the performance with state-of-the-art registration methods. Normalized cross correlation (NCC), structural similarity index (SSIM), and target registration error (TRE) were computed to evaluate the registration performance. Our method produced overall NCC of 0.96, SSIM of 0.94, and TRE of 2.26 → 2.27 mm, outperforming existing methods by 9%, 12%, and 27%, respectively. Experimental results also show that our method performs consistently and is more accurate than existing algorithms, and also computationally efficient.

The numerical stability of transformation-based CT ventilation

Abstract

Computed tomography (CT)-derived ventilation imaging utilizes deformable image registration (DIR) to recover respiratory-induced tissue volume changes from inhale/exhale 4DCT phases. While current strategies for validating CT ventilation rely on analyzing its correlation with existing functional imaging modalities, the numerical stability of the CT ventilation calculation has not been characterized.

Purpose

The purpose of this study is to examine how small changes in the DIR displacement field can affect the calculation of transformation-based CT ventilation.

Methods

First, we derive a mathematical theorem, which states that the change in ventilation metric induced by a perturbation to single displacement vector is bounded by the perturbation magnitude. Second, we introduce a novel Jacobian constrained optimization method for computing user-defined CT ventilation images.

Results

Using the Jacobian constrained method, we demonstrate that for the same inhale/exhale CT pair, it is possible to compute two DIR transformations that have similar spatial accuracies, but generate ventilation images with significantly different physical characteristics. In particular, we compute a CT ventilation image that perfectly correlates with a single-photon emission CT perfusion scan.

Conclusion

The analysis and experiments indicate that while transformation-based CT ventilation is a promising modality, small changes in the DIR displacement field can result in large relative changes in the ventilation image. As such, approaches for improving the reproducibility of CT ventilation are still needed.

Visualizing Shape Deformations with Variation of Geometric Spectrum

This paper presents a novel approach based on spectral geometry to quantify and visualize non-isometric deformations of 3D surfaces by mapping two manifolds. The proposed method can determine multi-scale, non-isometric deformations through the variation of Laplace-Beltrami spectrum of two shapes. Given two triangle meshes, the spectra can be varied from one to another with a scale function defined on each vertex. The variation is expressed as a linear interpolation of eigenvalues of the two shapes. In each iteration step, a quadratic programming problem is constructed, based on our derived spectrum variation theorem and smoothness energy constraint, to compute the spectrum variation. The derivation of the scale function is the solution of such a problem. Therefore, the final scale function can be solved by integral of the derivation from each step, which, in turn, quantitatively describes non-isometric deformations between two shapes. To evaluate the method, we conduct extensive experiments on synthetic and real data. We employ real epilepsy patient imaging data to quantify the shape variation between the left and right hippocampi in epileptic brains. In addition, we use longitudinal Alzheimer data to compare the shape deformation of diseased and healthy hippocampus. In order to show the accuracy and effectiveness of the proposed method, we also compare it with spatial registration-based methods, e.g., non-rigid Iterative Closest Point (ICP) and voxel-based method. These experiments demonstrate the advantages of our method.

A LAGRANGIAN GAUSS-NEWTON-KRYLOV SOLVER FOR MASS- AND INTENSITY-PRESERVING DIFFEOMORPHIC IMAGE REGISTRATION

We present an efficient solver for diffeomorphic image registration problems in the framework of Large Deformations Diffeomorphic Metric Mappings (LDDMM). We use an optimal control formulation, in which the velocity field of a hyperbolic PDE needs to be found such that the distance between the final state of the system (the transformed/transported template image) and the observation (the reference image) is minimized. Our solver supports both stationary and non-stationary (i.e., transient or time-dependent) velocity fields. As transformation models, we consider both the transport equation (assuming intensities are preserved during the deformation) and the continuity equation (assuming mass-preservation). We consider the reduced form of the optimal control problem and solve the resulting unconstrained optimization problem using a discretize-then-optimize approach. A key contribution is the elimination of the PDE constraint using a Lagrangian hyperbolic PDE solver. Lagrangian methods rely on the concept of characteristic curves. We approximate these curves using a fourth-order Runge-Kutta method. We also present an efficient algorithm for computing the derivatives of the final state of the system with respect to the velocity field. This allows us to use fast Gauss-Newton based methods. We present quickly converging iterative linear solvers using spectral preconditioners that render the overall optimization efficient and scalable. Our method is embedded into the image registration framework FAIR and, thus, supports the most commonly used similarity measures and regularization functionals. We demonstrate the potential of our new approach using several synthetic and real world test problems with up to 14.7 million degrees of freedom.

Molecular Surface Maps

We present Molecular Surface Maps, a novel, view-independent, and concise representation for molecular surfaces. It transfers the well-known world map metaphor to molecular visualization. Our application maps the complex molecular surface to a simple 2D representation through a spherical intermediate, the Molecular Surface Globe. The Molecular Surface Map concisely shows arbitrary attributes of the original molecular surface, such as biochemical properties or geometrical features. This results in an intuitive overview, which allows researchers to assess all molecular surface attributes at a glance. Our representation can be used as a visual summarization of a molecule's interface with its environment. In particular, Molecular Surface Maps simplify the analysis and comparison of different data sets or points in time. Furthermore, the map representation can be used in a Space-time Cube to analyze time-dependent data from molecular simulations without the need for animation. We show the feasibility of Molecular Surface Maps for different typical analysis tasks of biomolecular data.

iCellFusion

Temporal, multimodal microscopy imaging of live cells is becoming widely used in studies of cellular processes. In general, temporal sequences of images with functional and morphological data from live cells are acquired using multiple image sensors. The images from the different sources usually differ in resolution and have non-coincident fields of view, making the merging process complex. We present a new tool – iCellFusion – that performs data fusion of images from Phase-Contrast Microscopy and Fluorescence Microscopy in order to correlate the information on cell morphology, lineage and functionality. Prior to image fusion, iCellFusion performs automatic or computer-aided cell segmentation and establishes cell lineages. We exemplify its usage on time-lapse, multimodal microscopy images of bacteria producing fluorescent spots. We expect iCellFusion to assist research in Cell and Molecular Biology and the healthcare sector, where live-cell imaging is an increasingly important technique to detect and study diseases at the cellular level.

Combination of visual and symbolic knowledge: A survey in anatomy

In medicine, anatomy is considered as the most discussed field and results in a huge amount of knowledge, which is heterogeneous and covers aspects that are mostly independent in nature. Visual and symbolic modalities are mainly adopted for exemplifying knowledge about human anatomy and are crucial for the evolution of computational anatomy. In particular, a tight integration of visual and symbolic modalities is beneficial to support knowledge-driven methods for biomedical investigation. In this paper, we review previous work on the presentation and sharing of anatomical knowledge, and the development of advanced methods for computational anatomy, also focusing on the key research challenges for harmonizing symbolic knowledge and spatial 3D data.

A Parallel Nonrigid Registration Algorithm Based on B-Spline for Medical Images

The nonrigid registration algorithm based on B-spline Free-Form Deformation (FFD) plays a key role and is widely applied in medical image processing due to the good flexibility and robustness. However, it requires a tremendous amount of computing time to obtain more accurate registration results especially for a large amount of medical image data. To address the issue, a parallel nonrigid registration algorithm based on B-spline is proposed in this paper. First, the Logarithm Squared Difference (LSD) is considered as the similarity metric in the B-spline registration algorithm to improve registration precision. After that, we create a parallel computing strategy and lookup tables (LUTs) to reduce the complexity of the B-spline registration algorithm. As a result, the computing time of three time-consuming steps including B-splines interpolation, LSD computation, and the analytic gradient computation of LSD, is efficiently reduced, for the B-spline registration algorithm employs the Nonlinear Conjugate Gradient (NCG) optimization method. Experimental results of registration quality and execution efficiency on the large amount of medical images show that our algorithm achieves a better registration accuracy in terms of the differences between the best deformation fields and ground truth and a speedup of 17 times over the single-threaded CPU implementation due to the powerful parallel computing ability of Graphics Processing Unit (GPU).

A Greedy Algorithm for Brain MRI's Registration

This document presents a non-rigid registration algorithm for the use of brain magnetic resonance (MR) images comparison. More precisely, we want to compare pre-operative and post-operative MR images in order to assess the deformation due to a surgical removal. The proposed algorithm has been studied in Chesseboeuf et al. ((Non-rigid registration of magnetic resonance imaging of brain. IEEE, 385-390. doi: 10.1109/IPTA.2015.7367172 , 2015), following ideas of Trouvé (An infinite dimensional group approach for physics based models in patterns recognition. Technical Report DMI Ecole Normale Supérieure, Cachan, 1995), in which the author introduces the algorithm within a very general framework. Here we recalled this theory from a practical point of view. The emphasis is on illustrations and description of the numerical procedure. Our version of the algorithm is associated with a particular matching criterion. Then, a section is devoted to the description of this object. In the last section we focus on the construction of a statistical method of evaluation.

Multi-modal automatic montaging of adaptive optics retinal images

We present a fully automated adaptive optics (AO) retinal image montaging algorithm using classic scale invariant feature transform with random sample consensus for outlier removal. Our approach is capable of using information from multiple AO modalities (confocal, split detection, and dark field) and can accurately detect discontinuities in the montage. The algorithm output is compared to manual montaging by evaluating the similarity of the overlapping regions after montaging, and calculating the detection rate of discontinuities in the montage. Our results show that the proposed algorithm has high alignment accuracy and a discontinuity detection rate that is comparable (and often superior) to manual montaging. In addition, we analyze and show the benefits of using multiple modalities in the montaging process. We provide the algorithm presented in this paper as open-source and freely available to download.

Efficient Variational Approach to Multimodal Registration of Anatomical and Functional Intra-Patient Tumorous Brain Data

This paper addresses the functional localization of intra-patient images of the brain. Functional images of the brain (fMRI and PET) provide information about brain function and metabolism whereas anatomical images (MRI and CT) supply the localization of structures with high spatial resolution. The goal is to find the geometric correspondence between functional and anatomical images in order to complement and fuse the information provided by each imaging modality. The proposed approach is based on a variational formulation of the image registration problem in the frequency domain. It has been implemented as a C/C[Formula: see text] library which is invoked from a GUI. This interface is routinely used in the clinical setting by physicians for research purposes (Inscanner, Alicante, Spain), and may be used as well for diagnosis and surgical planning. The registration of anatomic and functional intra-patient images of the brain makes it possible to obtain a geometric correspondence which allows for the localization of the functional processes that occur in the brain. Through 18 clinical experiments, it has been demonstrated how the proposed approach outperforms popular state-of-the-art registration methods in terms of efficiency, information theory-based measures (such as mutual information) and actual registration error (distance in space of corresponding landmarks).

Synchronized navigation of current and prior studies using image registration improves radiologist's efficiency

PURPOSE

The purpose of this study was to investigate how the use of multi-modal rigid image registration integrated within a standard picture archiving and communication system affects the efficiency of a radiologist while performing routine interpretations of cases including prior examinations.

METHODS

Six radiologists were recruited to read a set of cases (either 16 neuroradiology or 14 musculoskeletal cases) during two crossover reading sessions. Each radiologist read each case twice, one time with synchronized navigation, which enables spatial synchronization across examinations from different study dates, and one time without. Efficiency was evaluated based upon time to read a case and amount of scrolling while browsing a case using Wilcoxon signed rank test.

RESULTS

Significant improvements in efficiency were found considering either all radiologists simultaneously, the two sections separately and the majority of individual radiologists for time to read and for amount of scrolling. The relative improvement for each individual radiologist ranged from 4 to 32% for time to read and from 14 to 38% for amount of scrolling.

CONCLUSION

Image registration providing synchronized navigation across examinations from different study dates provides a tool that enables radiologists to work more efficiently while reading cases with one or more prior examinations.

A novel approach for local abdominal aortic aneurysm growth quantification

Although aneurysm size still remains the most accepted predictor of rupture risk, abdominal aortic aneurysms (AAAs) with maximum diameter smaller than 5 cm may also rupture. Growth rate is an additional marker for rupture risk as it potentially reflects an undesirable wall remodeling that leads to fast regional growth. Currently, an indication for surgery is an expansion rate >10 mm/year, measured as change in maximum diameter over time. However, as AAA expansion is non-uniform, it is questionable whether measurement of maximum diameter change over time can capture increased localized remodeling activity. A method for estimating AAA surface area growth is introduced, providing a better measure of local wall deformation. The proposed approach is based on the non-rigid iterative closest point algorithm. Optimization and validation is performed using 12 patient-specific AAA geometries artificially deformed to produce a target surface with known nodal displacements. Mesh density sensitivity, range of uncertainty, and method limitations are discussed. Application to ten AAA patient-specific follow-ups suggested that maximum diameter growth does not correlate strongly with the maximum surface growth (R ² = 0.614), which is not always colocated with maximum diameter, or uniformly distributed. Surface growth quantification could reinforce the quality of aneurysm surveillance programs.

Lung motion estimation by robust point matching and spatiotemporal tracking for 4D CT

We propose a deformable registration approach to estimate patient-specific lung motion during free breathing for four-dimensional (4D) computed tomography (CT) based on point matching and tracking between images in different phases. First, a robust point matching (RPM) algorithm coarsely aligns the source phase image onto all other target phase images of 4D CT. Scale-invariant feature transform (SIFT) is introduced into the cost function in order to accelerate and stabilize the convergence of the point matching. Next, the temporal consistency of the estimated lung motion model is preserved by fitting the trajectories of the points in the respiratory phase using L₁ norm regularization. Then, the fitted positions of a point along the trajectory are used as the initial positions for the point tracking. Spatial mean-shift iteration is employed to track points in all phase images. The tracked positions in all phases are used to perform RPM again. These steps are repeated until the number of updated points is smaller than a given threshold σ. With this method, the correspondence between the source phase image and other target phase image is established more accurately. Trajectory fitting ensures the estimated trajectory does not fluctuate violently. We evaluated our method by using the public DIR-lab, POPI-model, CREATIS and COPDgene lung datasets. In the experimental results, the proposed method achieved satisfied accuracy for image registration. Our method also preserved the topology of the deformation fields well for image registration with large deformation.

MIND Demons: Symmetric Diffeomorphic Deformable Registration of MR and CT for Image-Guided Spine Surgery

Intraoperative localization of target anatomy and critical structures defined in preoperative MR/CT images can be achieved through the use of multimodality deformable registration. We propose a symmetric diffeomorphic deformable registration algorithm incorporating a modality-independent neighborhood descriptor (MIND) and a robust Huber metric for MR-to-CT registration. The method, called MIND Demons, finds a deformation field between two images by optimizing an energy functional that incorporates both the forward and inverse deformations, smoothness on the integrated velocity fields, a modality-insensitive similarity function suitable to multimodality images, and smoothness on the diffeomorphisms themselves. Direct optimization without relying on the exponential map and stationary velocity field approximation used in conventional diffeomorphic Demons is carried out using a Gauss-Newton method for fast convergence. Registration performance and sensitivity to registration parameters were analyzed in simulation, phantom experiments, and clinical studies emulating application in image-guided spine surgery, and results were compared to mutual information (MI) free-form deformation (FFD), local MI (LMI) FFD, normalized MI (NMI) Demons, and MIND with a diffusion-based registration method (MIND-elastic). The method yielded sub-voxel invertibility (0.008 mm) and nonzero-positive Jacobian determinants. It also showed improved registration accuracy in comparison to the reference methods, with mean target registration error (TRE) of 1.7 mm compared to 11.3, 3.1, 5.6, and 2.4 mm for MI FFD, LMI FFD, NMI Demons, and MIND-elastic methods, respectively. Validation in clinical studies demonstrated realistic deformations with sub-voxel TRE in cases of cervical, thoracic, and lumbar spine.

A Greedy Algorithm for Brain MRI's Registration

This document presents a non-rigid registration algorithm for the use of brain magnetic resonance (MR) images comparison. More precisely, we want to compare pre-operative and post-operative MR images in order to assess the deformation due to a surgical removal. The proposed algorithm has been studied in Chesseboeuf et al. ((Non-rigid registration of magnetic resonance imaging of brain. IEEE, 385-390. doi: 10.1109/IPTA.2015.7367172 , 2015), following ideas of Trouvé (An infinite dimensional group approach for physics based models in patterns recognition. Technical Report DMI Ecole Normale Supérieure, Cachan, 1995), in which the author introduces the algorithm within a very general framework. Here we recalled this theory from a practical point of view. The emphasis is on illustrations and description of the numerical procedure. Our version of the algorithm is associated with a particular matching criterion. Then, a section is devoted to the description of this object. In the last section we focus on the construction of a statistical method of evaluation.

A machine learning tool for re-planning and adaptive RT: A multicenter cohort investigation

PURPOSE

To predict patients who would benefit from adaptive radiotherapy (ART) and re-planning intervention based on machine learning from anatomical and dosimetric variations in a retrospective dataset.

MATERIALS AND METHODS

90 patients (pts) treated for head-neck cancer (H&N) formed a multicenter data-set. 41 H&N pts (45.6%) were considered for learning; 49 pts (54.4%) were used to test the tool. A homemade machine-learning classifier was developed to analyze volume and dose variations of parotid glands (PG). Using deformable image registration (DIR) and GPU, patients' conditions were analyzed automatically. Support Vector Machines (SVM) was used for time-series evaluation. "Inadequate" class identified patients that might benefit from replanning. Double-blind evaluation by two radiation oncologists (ROs) was carried out to validate day/week selected for re-planning by the classifier.

RESULTS

The cohort was affected by PG mean reduction of 23.7±8.8%. During the first 3weeks, 86.7% cases show PG deformation aligned with predefined tolerance, thus not requiring re-planning. From 4th week, an increased number of pts would potentially benefit from re-planning: a mean of 58% of cases, with an inter-center variability of 8.3%, showed "inadequate" conditions. 11% of cases showed "bias" due to DIR and script failure; 6% showed "warning" output due to potential positioning issues. Comparing re-planning suggested by tool with recommended by ROs, the 4th week seems the most favorable time in 70% cases.

CONCLUSIONS

SVM and decision-making tool was applied to overcome ART challenges. Pts would benefit from ART and ideal time for re-planning intervention was identified in this retrospective analysis.

Visualization of Deformable Image Registration Quality Using Local Image Dissimilarity

Deformable image registration (DIR) has the potential to improve modern radiotherapy in many aspects, including volume definition, treatment planning and image-guided adaptive radiotherapy. Studies have shown its possible clinical benefits. However, measuring DIR accuracy is difficult without known ground truth, but necessary before integration in the radiotherapy workflow. Visual assessment is an important step towards clinical acceptance. We propose a visualization framework which supports the exploration and the assessment of DIR accuracy. It offers different interaction and visualization features for exploration of candidate regions to simplify the process of visual assessment. The visualization is based on voxel-wise comparison of local image patches for which dissimilarity measures are computed and visualized to indicate locally the registration results. We performed an evaluation with three radiation oncologists to demonstrate the viability of our approach. In the evaluation, lung regions were rated by the participants with regards to their visual accuracy and compared to the registration error measured with expert defined landmarks. Regions rated as "accepted" had an average registration error of 1.8 mm, with the highest single landmark error being 3.3 mm. Additionally, survey results show that the proposed visualizations support a fast and intuitive investigation of DIR accuracy, and are suitable for finding even small errors.

Learning-Based Multimodal Image Registration for Prostate Cancer Radiation Therapy

Computed tomography (CT) is widely used for dose planning in the radiotherapy of prostate cancer. However, CT has low tissue contrast, thus making manual contouring difficult. In contrast, magnetic resonance (MR) image provides high tissue contrast and is thus ideal for manual contouring. If MR image can be registered to CT image of the same patient, the contouring accuracy of CT could be substantially improved, which could eventually lead to high treatment efficacy. In this paper, we propose a learning-based approach for multimodal image registration. First, to fill the appearance gap between modalities, a structured random forest with auto-context model is learnt to synthesize MRI from CT and vice versa. Then, MRI-to-CT registration is steered in a dual manner of registering images with same appearances, i.e., (1) registering the synthesized CT with CT, and (2) also registering MRI with the synthesized MRI. Next, a dual-core deformation fusion framework is developed to iteratively and effectively combine these two registration results. Experiments on pelvic CT and MR images have shown the improved registration performance by our proposed method, compared with the existing non-learning based registration methods.

Deformable Registration of Biomedical Images Using 2D Hidden Markov Models

Robust registration of unimodal and multimodal images is a key task in biomedical image analysis, and is often utilized as an initial step on which subsequent analysis techniques critically depend. We propose a novel probabilistic framework, based on a variant of the 2D hidden Markov model, namely, the turbo hidden Markov model, to capture the deformation between pairs of images. The hidden Markov model is tailored to capture spatial transformations across images via state transitions, and modality-specific data costs via emission probabilities. The method is derived for the unimodal setting (where simpler matching metrics may be used) as well as the multimodal setting, where different modalities may provide very different representations for a given class of objects, necessitating the use of advanced similarity measures. We utilize a rich model with hundreds of model parameters to describe the deformation relationships across such modalities. We also introduce a local edge-adaptive constraint to allow for varying degrees of smoothness between object boundaries and homogeneous regions. The parameters of the described method are estimated in a principled manner from training data via maximum likelihood learning, and the deformation is subsequently estimated using an efficient dynamic programming algorithm. Experimental results demonstrate the improved performance of the proposed approach over the state-of-the-art deformable registration techniques, on both unimodal and multimodal biomedical data sets.

Deformable image registration for adaptive radiotherapy with guaranteed local rigidity constraints

Background

Deformable image registration (DIR) is a key component in many radiotherapy applications. However, often resulting deformations are not satisfying, since varying deformation properties of different anatomical regions are not considered. To improve the plausibility of DIR in adaptive radiotherapy in the male pelvic area, this work integrates a local rigidity deformation model into a DIR algorithm.

Methods

A DIR framework is extended by constraints, enforcing locally rigid deformation behavior for arbitrary delineated structures. The approach restricts those structures to rigid deformations, while surrounding tissue is still allowed to deform elastically. The algorithm is tested on ten CT/CBCT male pelvis datasets with active rigidity constraints on bones and prostate and compared to the Varian SmartAdapt deformable registration (VSA) on delineations of bladder, prostate and bones.

Results

The approach with no rigid structures (REG0) obtains an average dice similarity coefficient (DSC) of 0.87 ± 0.06 and a Hausdorff-Distance (HD) of 8.74 ± 5.95 mm. The new approach with rigid bones (REG1) yields a DSC of 0.87 ± 0.07, HD 8.91 ± 5.89 mm. Rigid deformation of bones and prostate (REG2) obtains 0.87 ± 0.06, HD 8.73 ± 6.01 mm, while VSA yields a DSC of 0.86 ± 0.07, HD 10.22 ± 6.62 mm. No deformation grid foldings are observed for REG0 and REG1 in 7 of 10 cases; for REG2 in 8 of 10 cases, with no grid foldings in prostate, an average of 0.08 % in bladder (REG2: no foldings) and 0.01 % inside the body contour. VSA exhibits grid foldings in each case, with an average percentage of 1.81 % for prostate, 1.74 % for bladder and 0.12 % for the body contour. While REG1 and REG2 keep bones rigid, elastic bone deformations are observed with REG0 and VSA. An average runtime of 26.2 s was achieved with REG1; 31.1 s with REG2, compared to 10.5 s with REG0 and 10.7 s with VMS.

Conclusions

With accuracy in the range of VSA, the new approach with constraints delivers physically more plausible deformations in the pelvic area with guaranteed rigidity of arbitrary structures. Although the algorithm uses an advanced deformation model, clinically feasible runtimes are achieved.

Electronic supplementary material

The online version of this article (doi:10.1186/s13014-016-0697-4) contains supplementary material, which is available to authorized users.

Dense motion field estimation from myocardial boundary displacements

Minimally invasive cardiovascular interventions guided by multiple imaging modalities are rapidly gaining clinical acceptance for the treatment of several cardiovascular diseases. These images are typically fused with richly detailed pre-operative scans through registration techniques, enhancing the intra-operative clinical data and easing the image-guided procedures. Nonetheless, rigid models have been used to align the different modalities, not taking into account the anatomical variations of the cardiac muscle throughout the cardiac cycle. In the current study, we present a novel strategy to compensate the beat-to-beat physiological adaptation of the myocardium. Hereto, we intend to prove that a complete myocardial motion field can be quickly recovered from the displacement field at the myocardial boundaries, therefore being an efficient strategy to locally deform the cardiac muscle. We address this hypothesis by comparing three different strategies to recover a dense myocardial motion field from a sparse one, namely, a diffusion-based approach, thin-plate splines, and multiquadric radial basis functions. Two experimental setups were used to validate the proposed strategy. First, an in silico validation was carried out on synthetic motion fields obtained from two realistic simulated ultrasound sequences. Then, 45 mid-ventricular 2D sequences of cine magnetic resonance imaging were processed to further evaluate the different approaches. The results showed that accurate boundary tracking combined with dense myocardial recovery via interpolation/diffusion is a potentially viable solution to speed up dense myocardial motion field estimation and, consequently, to deform/compensate the myocardial wall throughout the cardiac cycle. Copyright © 2015 John Wiley & Sons, Ltd.