A unified non-rigid feature registration method for brain mapping

An Open-Source Package for Thermal and Multispectral Image Analysis for Plants in Glasshouse

Advanced plant phenotyping techniques to measure biophysical traits of crops are helping to deliver improved crop varieties faster. Phenotyping of plants using different sensors for image acquisition and its analysis with novel computational algorithms are increasingly being adapted to measure plant traits. Thermal and multispectral imagery provides novel opportunities to reliably phenotype crop genotypes tested for biotic and abiotic stresses under glasshouse conditions. However, optimization for image acquisition, pre-processing, and analysis is required to correct for optical distortion, image co-registration, radiometric rescaling, and illumination correction. This study provides a computational pipeline that optimizes these issues and synchronizes image acquisition from thermal and multispectral sensors. The image processing pipeline provides a processed stacked image comprising RGB, green, red, NIR, red edge, and thermal, containing only the pixels present in the object of interest, e.g., plant canopy. These multimodal outputs in thermal and multispectral imageries of the plants can be compared and analysed mutually to provide complementary insights and develop vegetative indices effectively. This study offers digital platform and analytics to monitor early symptoms of biotic and abiotic stresses and to screen a large number of genotypes for improved growth and productivity. The pipeline is packaged as open source and is hosted online so that it can be utilized by researchers working with similar sensors for crop phenotyping.

Deformable medical image registration: a survey

Deformable image registration is a fundamental task in medical image processing. Among its most important applications, one may cite: 1) multi-modality fusion, where information acquired by different imaging devices or protocols is fused to facilitate diagnosis and treatment planning; 2) longitudinal studies, where temporal structural or anatomical changes are investigated; and 3) population modeling and statistical atlases used to study normal anatomical variability. In this paper, we attempt to give an overview of deformable registration methods, putting emphasis on the most recent advances in the domain. Additional emphasis has been given to techniques applied to medical images. In order to study image registration methods in depth, their main components are identified and studied independently. The most recent techniques are presented in a systematic fashion. The contribution of this paper is to provide an extensive account of registration techniques in a systematic manner.

Nonrigid point registration for 2D curves and 3D surfaces and its various applications

A nonrigid B-spline-based point-matching (BPM) method is proposed to match dense surface points. The method solves both the point correspondence and nonrigid transformation without features extraction. The registration method integrates a motion model, which combines a global transformation and a B-spline-based local deformation, into a robust point-matching framework. The point correspondence and deformable transformation are estimated simultaneously by fuzzy correspondence and by a deterministic annealing technique. Prior information about global translation, rotation and scaling is incorporated into the optimization. A local B-spline motion model decreases the degrees of freedom for optimization and thus enables the registration of a larger number of feature points. The performance of the BPM method has been demonstrated and validated using synthesized 2D and 3D data, mouse MRI and micro-CT images. The proposed BPM method can be used to register feature point sets, 2D curves, 3D surfaces and various image data.

Deformable medical image registration: setting the state of the art with discrete methods

This review introduces a novel deformable image registration paradigm that exploits Markov random field formulation and powerful discrete optimization algorithms. We express deformable registration as a minimal cost graph problem, where nodes correspond to the deformation grid, a node's connectivity corresponds to regularization constraints, and labels correspond to 3D deformations. To cope with both iconic and geometric (landmark-based) registration, we introduce two graphical models, one for each subproblem. The two graphs share interconnected variables, leading to a modular, powerful, and flexible formulation that can account for arbitrary image-matching criteria, various local deformation models, and regularization constraints. To cope with the corresponding optimization problem, we adopt two optimization strategies: a computationally efficient one and a tight relaxation alternative. Promising results demonstrate the potential of this approach. Discrete methods are an important new trend in medical image registration, as they provide several improvements over the more traditional continuous methods. This is illustrated with several key examples where the presented framework outperforms existing general-purpose registration methods in terms of both performance and computational complexity. Our methods become of particular interest in applications where computation time is a critical issue, as in intraoperative imaging, or where the huge variation in data demands complex and application-specific matching criteria, as in large-scale multimodal population studies. The proposed registration framework, along with a graphical interface and corresponding publications, is available for download for research purposes (for Windows and Linux platforms) from http://www.mrf-registration.net.

Automatic range image registration in the Markov chain

In this paper, a novel entropy that can describe both long and short-tailed probability distributions of constituents of a thermodynamic system out of its thermodynamic limit is first derived from the Lyapunov function for a Markov chain. We then maximize this entropy for the estimation of the probabilities of possible correspondences established using the traditional closest point criterion between two overlapping range images. When we change our viewpoint to look carefully at the minimum solution to the probability estimate of the correspondences, the iterative range image registration process can also be modeled as a Markov chain in which lessons from past experience in estimating those probabilities are learned. To impose the two-way constraint, outliers are explicitly modeled due to the almost ubiquitous occurrence of occlusion, appearance, and disappearance of points in either image. The estimated probabilities of the correspondences are finally embedded into the powerful mean field annealing scheme for global optimization, leading the camera motion parameters to be estimated in the weighted least-squares sense. A comparative study using real images shows that the proposed algorithm usually outperforms the state-of-the-art ICP variants and the latest genetic algorithm for automatic overlapping range image registration.

Mjolnir: extending HAMMER using a diffusion transformation model and histogram equalization for deformable image registration

Image registration is a crucial step in many medical image analysis procedures such as image fusion, surgical planning, segmentation and labeling, and shape comparison in population or longitudinal studies. A new approach to volumetric intersubject deformable image registration is presented. The method, called Mjolnir, is an extension of the highly successful method HAMMER. New image features in order to better localize points of correspondence between the two images are introduced as well as a novel approach to generate a dense displacement field based upon the weighted diffusion of automatically derived feature correspondences. An extensive validation of the algorithm was performed on T1-weighted SPGR MR brain images from the NIREP evaluation database. The results were compared with results generated by HAMMER and are shown to yield significant improvements in cortical alignment as well as reduced computation time.

Nonrigid Registration of Brain Tumor Resection MR Images Based on Joint Saliency Map and Keypoint Clustering

This paper proposes a novel global-to-local nonrigid brain MR image registration to compensate for the brain shift and the unmatchable outliers caused by the tumor resection. The mutual information between the corresponding salient structures, which are enhanced by the joint saliency map (JSM), is maximized to achieve a global rigid registration of the two images. Being detected and clustered at the paired contiguous matching areas in the globally registered images, the paired pools of DoG keypoints in combination with the JSM provide a useful cluster-to-cluster correspondence to guide the local control-point correspondence detection and the outlier keypoint rejection. Lastly, a quasi-inverse consistent deformation is smoothly approximated to locally register brain images through the mapping the clustered control points by compact support radial basis functions. The 2D implementation of the method can model the brain shift in brain tumor resection MR images, though the theory holds for the 3D case.

Learning-based Deformation Estimation for Fast Non-rigid Registration

This paper presents a learning-based deformation estimation method for fast non-rigid registration. First, a PCA-based statistical deformation model is constructed using the deformation fields obtained by conventional registration algorithms between a template image and training subject images. Then, the constructed statistical model is used to generate a large number of sample deformation fields by resampling in the PCA space. In the meanwhile, by warping the template using these sample deformation fields, the respective sample images in the PCA space can be also generated. Finally, after learning the correlation between the features of the sample images and their deformation coefficients, given a new test image, we can immediately estimate its relative deformations to the template based on its image information. Using this estimated deformation, we can warp the template to generate an intermediate template close to the test image. Since the intermediate template is more similar to the test image compared to the original template, the deformable registration via the intermediate template becomes much easier and faster. Experimental results show that the proposed learning-based registration method can fast register MR brain image with robust performance.

More accurate Talairach coordinates for neuroimaging using non-linear registration

While the Talairach atlas remains the most commonly used system for reporting coordinates in neuroimaging studies, the absence of an actual 3-D image of the original brain used in its construction has severely limited the ability of researchers to automatically map locations from 3-D anatomical MRI images to the atlas. Previous work in this area attempted to circumvent this problem by constructing approximate linear and piecewise-linear mappings between standard brain templates (e.g. the MNI template) and Talairach space. These methods are limited in that they can only account for differences in overall brain size and orientation but cannot correct for the actual shape differences between the MNI template and the Talairach brain. In this paper we describe our work to digitize the Talairach atlas and generate a non-linear mapping between the Talairach atlas and the MNI template that attempts to compensate for the actual differences in shape between the two, resulting in more accurate coordinate transformations. We present examples in this paper and note that the method is available freely online as a Java applet.

Thoracoscopic surgical navigation system for cancer localization in collapsed lung based on estimation of lung deformation

We have developed a thoracoscopic surgical navigation system for lung cancer localization. In our system, the thoracic cage and mediastinum are localized using rigid registration between the intraoperatively digitized surface points and the preoperative CT surface model, and then the lung deformation field is estimated using nonrigid registration between the registered and digitized point datasets on the collapsed lung surface and the preoperative CT lung surface model to predict cancer locations. In this paper, improved methods on key components of the system are investigated to realize clinically acceptable usability and accuracy. Firstly, we implement a non-contact surface digitizer under thoracoscopic control using an optically tracked laser pointer. Secondly, we establish a rigid registration protocol which minimizes the influence of the deformation in different patient's positions by analyzing MR images of volunteers. These techniques were evaluated by in vitro and clinical experiments.

Non-Rigid Multi-Modal Image Registration Using Cross-Cumulative Residual Entropy

In this paper we present a new approach for the non-rigid registration of multi-modality images. Our approach is based on an information theoretic measure called the cumulative residual entropy (CRE), which is a measure of entropy defined using cumulative distributions. Cross-CRE between two images to be registered is defined and maximized over the space of smooth and unknown non-rigid transformations. For efficient and robust computation of the non-rigid deformations, a tri-cubic B-spline based representation of the deformation function is used. The key strengths of combining CCRE with the tri-cubic B-spline representation in addressing the non-rigid registration problem are that, not only do we achieve the robustness due to the nature of the CCRE measure, we also achieve computational efficiency in estimating the non-rigid registration. The salient features of our algorithm are: (i) it accommodates images to be registered of varying contrast+brightness, (ii) faster convergence speed compared to other information theory-based measures used for non-rigid registration in literature, (iii) analytic computation of the gradient of CCRE with respect to the non-rigid registration parameters to achieve efficient and accurate registration, (iv) it is well suited for situations where the source and the target images have field of views with large non-overlapping regions. We demonstrate these strengths via experiments on synthesized and real image data.

Brain functional localization: a survey of image registration techniques

Functional localization is a concept which involves the application of a sequence of geometrical and statistical image processing operations in order to define the location of brain activity or to produce functional/parametric maps with respect to the brain structure or anatomy. Considering that functional brain images do not normally convey detailed structural information and, thus, do not present an anatomically specific localization of functional activity, various image registration techniques are introduced in the literature for the purpose of mapping functional activity into an anatomical image or a brain atlas. The problems addressed by these techniques differ depending on the application and the type of analysis, i.e., single-subject versus group analysis. Functional to anatomical brain image registration is the core part of functional localization in most applications and is accompanied by intersubject and subject-to-atlas registration for group analysis studies. Cortical surface registration and automatic brain labeling are some of the other tools towards establishing a fully automatic functional localization procedure. While several previous survey papers have reviewed and classified general-purpose medical image registration techniques, this paper provides an overview of brain functional localization along with a survey and classification of the image registration techniques related to this problem.

A hierarchical elastic registration scheme based on deformable segmentation

In this paper a hierarchical scheme is presented for contour-based elastic image registration. An improved deformable model is used to simultaneously segment and register feature from multiple images. The objects in different images are segmented by evolving a single contour and meanwhile the parameters of affine registration transformation are found out. After that, a contour-constrained elastic registration is applied to register the images correctly. The experimental results indicate that the proposed approach is effective to segment and register medical images.

Comparison of fMRI coregistration results between human experts and software solutions in patients and healthy subjects

Functional magnetic resonance imaging (fMRI) performed by echo-planar imaging (EPI) is often highly distorted, and it is therefore necessary to coregister the functional to undistorted anatomical images, especially for clinical applications. This pilot study provides an evaluation of human and automatic coregistration results in the human motor cortex of normal and pathological brains. Ten healthy right-handed subjects and ten right-handed patients performed simple right hand movements during fMRI. A reference point chosen at a characteristic anatomical location within the fMRI sensorimotor activations was transferred to the high resolution anatomical MRI images by three human fMRI experts and by three automatic coregistration programs. The 3D distance between the median localizations of experts and programs was calculated and compared between patients and healthy subjects. Results show that fMRI localization on anatomical images was better with the experts than software in 70% of the cases and that software performance was worse for patients than healthy subjects (unpaired t-test: P = 0.040). With 45.6 mm the maximum disagreement between experts and software was quite large. The inter-rater consistency was better for the fMRI experts compared to the coregistration programs (ANOVA: P = 0.003). We conclude that results of automatic coregistration should be evaluated carefully, especially in case of clinical application.

3-D diffeomorphic shape registration on hippocampal data sets

Matching 3D shapes is important in many medical imaging applications. We show that a joint clustering and diffeomorphism estimation strategy is capable of simultaneously estimating correspondences and a diffeomorphism between unlabeled 3D point-sets. Correspondence is established between the cluster centers and this is coupled with a simultaneous estimation of a 3D diffeomorphism of space. The number of clusters can be estimated by minimizing the Jensen-Shannon divergence on the registered data. We apply our algorithm to both synthetically warped 3D hippocampal shapes as well as real 3D hippocampal shapes from different subjects.

A method to track cortical surface deformations using a laser range scanner

This paper reports a novel method to track brain shift using a laser-range scanner (LRS) and nonrigid registration techniques. The LRS used in this paper is capable of generating textured point-clouds describing the surface geometry/intensity pattern of the brain as presented during cranial surgery. Using serial LRS acquisitions of the brain's surface and two-dimensional (2-D) nonrigid image registration, we developed a method to track surface motion during neurosurgical procedures. A series of experiments devised to evaluate the performance of the developed shift-tracking protocol are reported. In a controlled, quantitative phantom experiment, the results demonstrate that the surface shift-tracking protocol is capable of resolving shift to an accuracy of approximately 1.6 mm given initial shifts on the order of 15 mm. Furthermore, in a preliminary in vivo case using the tracked LRS and an independent optical measurement system, the automatic protocol was able to reconstruct 50% of the brain shift with an accuracy of 3.7 mm while the manual measurement was able to reconstruct 77% with an accuracy of 2.1 mm. The results suggest that a LRS is an effective tool for tracking brain surface shift during neurosurgery.

Geometric strategies for neuroanatomic analysis from MRI

In this paper, we describe ongoing work in the Image Processing and Analysis Group (IPAG) at Yale University specifically aimed at the analysis of structural information as represented within magnetic resonance images (MRI) of the human brain. Specifically, we will describe our applied mathematical approaches to the segmentation of cortical and subcortical structure, the analysis of white matter fiber tracks using diffusion tensor imaging (DTI), and the intersubject registration of neuroanatomical (aMRI) data sets. Many of our methods rally around the use of geometric constraints, statistical (MAP) estimation, and the use of level set evolution strategies. The analysis of gray matter structure and connecting white matter paths combined with the ability to bring all information into a common space via intersubject registration should provide us with a rich set of data to investigate structure and variation in the human brain in neuropsychiatric disorders, as well as provide a basis for current work in the development of integrated brain function-structure analysis.

Validation of contour-driven thin-plate splines for tracking fraction-to-fraction changes in anatomy and radiation therapy dose mapping

The goal of this study is to validate a deformable model using contour-driven thin-plate splines for application to radiation therapy dose mapping. Our testing includes a virtual spherical phantom as well as real computed tomography (CT) data from ten prostate cancer patients with radio-opaque markers surgically implanted into the prostate and seminal vesicles. In the spherical mathematical phantom, homologous control points generated automatically given input contour data in CT slice geometry were compared to homologous control point placement using analytical geometry as the ground truth. The dose delivered to specific voxels driven by both sets of homologous control points were compared to determine the accuracy of dose tracking via the deformable model. A 3D analytical spherically symmetric dose distribution with a dose gradient of approximately 10% per mm was used for this phantom. This test showed that the uncertainty in calculating the delivered dose to a tissue element depends on slice thickness and the variation in defining homologous landmarks, where dose agreement of 3-4% in high dose gradient regions was achieved. In the patient data, radio-opaque marker positions driven by the thin-plate spline algorithm were compared to the actual marker positions as identified in the CT scans. It is demonstrated that the deformable model is accurate (approximately 2.5 mm) to within the intra-observer contouring variability. This work shows that the algorithm is appropriate for describing changes in pelvic anatomy and for the dose mapping application with dose gradients characteristic of conformal and intensity modulated radiation therapy.

Robust nonrigid multimodal image registration using local frequency maps

Automatic multi-modal image registration is central to numerous tasks in medical imaging today and has a vast range of applications e.g., image guidance, atlas construction, etc. In this paper, we present a novel multi-modal 3D non-rigid registration algorithm where in 3D images to be registered are represented by their corresponding local frequency maps efficiently computed using the Riesz transform as opposed to the popularly used Gabor filters. The non-rigid registration between these local frequency maps is formulated in a statistically robust framework involving the minimization of the integral squared error a.k.a. L2E (L2 error). This error is expressed as the squared difference between the true density of the residual (which is the squared difference between the non-rigidly transformed reference and the target local frequency representations) and a Gaussian or mixture of Gaussians density approximation of the same. The non-rigid transformation is expressed in a B-spline basis to achieve the desired smoothness in the transformation as well as computational efficiency. The key contributions of this work are (i) the use of Riesz transform to achieve better efficiency in computing the local frequency representation in comparison to Gabor filter-based approaches, (ii) new mathematical model for local-frequency based non-rigid registration, (iii) analytic computation of the gradient of the robust non-rigid registration cost function to achieve efficient and accurate registration. The proposed non-rigid L2E-based registration is a significant extension of research reported in literature to date. We present experimental results for registering several real data sets with synthetic and real non-rigid misalignments.

Articulated rigid registration for serial lower-limb mouse imaging

This paper describes a new piecewise rotational transformation model for capturing the articulation of joints such as the hip and the knee. While a simple piecewise rigid model can be applied, such models suffer from discontinuities at the motion boundary leading to both folding and stretching. Our model avoids both of these problems by constructing a provably continuous transformation along the motion interface. We embed this transformation model within the robust point matching framework and demonstrate its successful application to both synthetic data, and to serial x-ray CT mouse images. In the later case, our model captures the articulation of six joints, namely the left/right hip, the left/right knee and the left/right ankle. In the future such a model could be used to initialize non-rigid registrations of images from different subjects, as well as, be embedded in intensity-based and integrated registration algorithms. It could also be applied to human data in cases where articulated motion is an issue (e.g. image guided prostate radiotherapy, lower extremity CT angiography).

Object-based morphometry of the cerebral cortex

Most of the approaches dedicated to automatic morphometry rely on a point-by-point strategy based on warping each brain toward a reference coordinate system. In this paper, we describe an alternative object-based strategy dedicated to the cortex. This strategy relies on an artificial neuroanatomist performing automatic recognition of the main cortical sulci and parcellation of the cortical surface into gyral patches. A set of shape descriptors, which can be compared across subjects, is then attached to the sulcus and gyrus related objects segmented by this process. The framework is used to perform a study of 142 brains of the International Consortium for Brain Mapping (ICBM) database. This study reveals some correlates of handedness on the size of the sulci located in motor areas, which was not detected previously using standard voxel based morphometry.

Unsupervised learning of an atlas from unlabeled point-sets

One of the key challenges in deformable shape modeling is the problem of estimating a meaningful average or mean shape from a set of unlabeled shapes. We present a new joint clustering and matching algorithm that is capable of computing such a mean shape from multiple shape samples which are represented by unlabeled point-sets. An iterative bootstrap process is used wherein multiple shape sample point-sets are nonrigidly deformed to the emerging mean shape, with subsequent estimation of the mean shape based on these nonrigid alignments. The process is entirely symmetric with no bias toward any of the original shape sample point-sets. We believe that this method can be especially useful for creating atlases of various shapes present in medical images. We have applied the method to create mean shapes from nine hand-segmented 2D corpus callosum data sets and 10 hippocampal 3D point-sets.