Registration of translated and rotated images using finite fourier transforms

3-D rotation tracking from 2-D images of spherical colloids with textured surfaces

Tracking the three-dimensional rotation of colloidal particles is essential to elucidate many open questions, e.g. concerning the contact interactions between particles under flow, or the way in which obstacles and neighboring particles affect self-propulsion in active suspensions. In order to achieve rotational tracking, optically anisotropic particles are required. We synthesise here rough spherical colloids that present randomly distributed fluorescent asperities and track their motion under different experimental conditions. Specifically, we propose a new algorithm based on a 3-D rotation registration, which enables us to track the 3-D rotation of our rough colloids at short time-scales, using time series of 2-D images acquired at high frame rates with a conventional wide-field microscope. The method is based on the image correlation between a reference image and rotated 3-D prospective images to identify the most likely angular displacements between frames. We first validate our approach against simulated data and then apply it to the cases of: particles flowing through a capillary, freely diffusing at solid-liquid and liquid-liquid interfaces, and self-propelling above a substrate. By demonstrating the applicability of our algorithm and sharing the code, we hope to encourage further investigations in the rotational dynamics of colloidal systems.

Feature Point Detection and Description Networks Based on Asymmetric Convolution and the Cross-ResolutionImage-Matching Method

Image matching can be transformed into the problem of feature point detection and matching of images. The current neural network methods have a weak detection effect on feature points and cannot extract enough sparse and uniform feature points. In order to improve the detection and description ability of feature points, this paper proposes a self-supervised feature point detection and description network based on asymmetric convolution: ACPoint. Specifically, first, feature point pseudolabels are learned from an unlabeled dataset, and pseudolabels are used for supervised learning; then, the learned model is used to update pseudolabels. Through multiple iterations of model training and label updating, high-quality labels and high-accuracy models are obtained adaptively. The asymmetric convolution feature point (ACPoint) network adopts an asymmetric convolution module to simultaneously train three convolution branches to learn more feature information, which uses two one-dimensional convolutions to enhance the backbone of square convolution from both horizontal and vertical directions and improve the representation of local features during inference. Based on the ACPoint network, a cross-resolution image-matching method is proposed. Experiments show that our proposed network model has higher localization accuracy and homography estimation ability on the HPatches dataset.

An Improved Method of an Image Mosaic of a Tea Garden and Tea Tree Target Extraction

UAV may be limited by its flight height and camera resolution when aerial photography of a tea garden is carried out. The images of the tea garden contain trees and weeds whose vegetation information is similar to tea tree, which will affect tea tree extraction for further agricultural analysis. In order to obtain a high-definition large field-of-view tea garden image that contains tea tree targets, this paper (1) searches for the suture line based on the graph cut method in the image stitching technology; (2) improves the energy function to realize the image stitching of the tea garden; and (3) builds a feature vector to accurately extract tea tree vegetation information and remove unnecessary variables, such as trees and weeds. By comparing this with the manual extraction, the algorithm in this paper can effectively distinguish and eliminate most of the interference information. The IOU in a single mosaic image was more than 80% and the omissions account was 10%. The extraction results in accuracies that range from 84.91% to 93.82% at the different height levels (30 m, 60 m and 100 m height) of single images. Tea tree extraction accuracy rates in the mosaic images are 84.96% at a height of 30 m, and 79.94% at a height of 60 m.

Automated Crystal Orientation Mapping in py4DSTEM using Sparse Correlation Matching

Crystalline materials used in technological applications are often complex assemblies composed of multiple phases and differently oriented grains. Robust identification of the phases and orientation relationships from these samples is crucial, but the information extracted from the diffraction condition probed by an electron beam is often incomplete. We have developed an automated crystal orientation mapping (ACOM) procedure which uses a converged electron probe to collect diffraction patterns from multiple locations across a complex sample. We provide an algorithm to determine the orientation of each diffraction pattern based on a fast sparse correlation method. We demonstrate the speed and accuracy of our method by indexing diffraction patterns generated using both kinematical and dynamical simulations. We have also measured orientation maps from an experimental dataset consisting of a complex polycrystalline twisted helical AuAgPd nanowire. From these maps we identify twin planes between adjacent grains, which may be responsible for the twisted helical structure. All of our methods are made freely available as open source code, including tutorials which can be easily adapted to perform ACOM measurements on diffraction pattern datasets.

Multimodal Remote Sensing Image Registration Methods and Advancements: A Survey

With rapid advancements in remote sensing image registration algorithms, comprehensive imaging applications are no longer limited to single-modal remote sensing images. Instead, multi-modal remote sensing (MMRS) image registration has become a research focus in recent years. However, considering multi-source, multi-temporal, and multi-spectrum input introduces significant nonlinear radiation differences in MMRS images for which researchers need to develop novel solutions. At present, comprehensive reviews and analyses of MMRS image registration methods are inadequate in related fields. Thus, this paper introduces three theoretical frameworks: namely, area-based, feature-based and deep learning-based methods. We present a brief review of traditional methods and focus on more advanced methods for MMRS image registration proposed in recent years. Our review or comprehensive analysis is intended to provide researchers in related fields with advanced understanding to achieve further breakthroughs and innovations.

Mueller matrix imaging microscope using dual continuously rotating anisotropic mirrors

We demonstrate calibration and operation of a Mueller matrix imaging microscope using dual continuously rotating anisotropic mirrors for polarization state generation and analysis. The mirrors contain highly spatially coherent nanostructure slanted columnar titanium thin films deposited onto optically thick titanium layers on quartz substrates. The first mirror acts as polarization state image generator and the second mirror acts as polarization state image detector. The instrument is calibrated using samples consisting of laterally homogeneous properties such as straight-through-air, a clear aperture linear polarizer, and a clear aperture linear retarder waveplate. Mueller matrix images are determined for spatially varying anisotropic samples consisting of a commercially available (Thorlabs) birefringent resolution target and a spatially patterned titanium slanted columnar thin film deposited onto a glass substrate. Calibration and operation are demonstrated at a single wavelength (530 nm) only, while, in principle, the instrument can operate regardless of wavelength. We refer to this imaging ellipsometry configuration as rotating-anisotropic-mirror-sample-rotating-anisotropic-mirror ellipsometry (RAM-S-RAM-E).

Spatial light interference microscopy: principle and applications to biomedicine

In this paper, we review spatial light interference microscopy (SLIM), a common-path, phase-shifting interferometer, built onto a phase-contrast microscope, with white-light illumination. As one of the most sensitive quantitative phase imaging (QPI) methods, SLIM allows for speckle-free phase reconstruction with sub-nanometer path-length stability. We first review image formation in QPI, scattering, and full-field methods. Then, we outline SLIM imaging from theory and instrumentation to diffraction tomography. Zernike's phase-contrast microscopy, phase retrieval in SLIM, and halo removal algorithms are discussed. Next, we discuss the requirements for operation, with a focus on software developed in-house for SLIM that enables high-throughput acquisition, whole slide scanning, mosaic tile registration, and imaging with a color camera. We introduce two methods for solving the inverse problem using SLIM, white-light tomography, and Wolf phase tomography. Lastly, we review the applications of SLIM in basic science and clinical studies. SLIM can study cell dynamics, cell growth and proliferation, cell migration, mass transport, etc. In clinical settings, SLIM can assist with cancer studies, reproductive technology, blood testing, etc. Finally, we review an emerging trend, where SLIM imaging in conjunction with artificial intelligence brings computational specificity and, in turn, offers new solutions to outstanding challenges in cell biology and pathology.

Automatic Sub-Pixel Co-Registration of Remote Sensing Images Using Phase Correlation and Harris Detector

In this paper, we propose a new approach for sub-pixel co-registration based on Fourier phase correlation combined with the Harris detector. Due to the limitation of the standard phase correlation method to achieve only pixel-level accuracy, another approach is required to reach sub-pixel matching precision. We first applied the Harris corner detector to extract corners from both references and sensed images. Then, we identified their corresponding points using phase correlation between the image pairs. To achieve sub-pixel registration accuracy, two optimization algorithms were used. The effectiveness of the proposed method was tested with very high-resolution (VHR) remote sensing images, including Pleiades satellite images and aerial imagery. Compared with the speeded-up robust features (SURF)-based method, phase correlation with the Blackman window function produced 91% more matches with high reliability. Moreover, the results of the optimization analysis have revealed that Nelder–Mead algorithm performs better than the two-point step size gradient algorithm regarding localization accuracy and computation time. The proposed approach achieves better accuracy than 0.5 pixels and outperforms the speeded-up robust features (SURF)-based method. It can achieve sub-pixel accuracy in the presence of noise and produces large numbers of correct matching points.

Statistical downscaling with spatial misalignment: Application to wildland fire PM2.5 concentration forecasting

Fine particulate matter, PM_2.5, has been documented to have adverse health effects and wildland fires are a major contributor to PM_2.5 air pollution in the US. Forecasters use numerical models to predict PM_2.5 concentrations to warn the public of impending health risk. Statistical methods are needed to calibrate the numerical model forecast using monitor data to reduce bias and quantify uncertainty. Typical model calibration techniques do not allow for errors due to misalignment of geographic locations. We propose a spatiotemporal downscaling methodology that uses image registration techniques to identify the spatial misalignment and accounts for and corrects the bias produced by such warping. Our model is fitted in a Bayesian framework to provide uncertainty quantification of the misalignment and other sources of error. We apply this method to different simulated data sets and show enhanced performance of the method in presence of spatial misalignment. Finally, we apply the method to a large fire in Washington state and show that the proposed method provides more realistic uncertainty quantification than standard methods.

MRI image analysis methods and applications: an algorithmic perspective using brain tumors as an exemplar

The use of magnetic resonance imaging (MRI) in healthcare and the emergence of radiology as a practice are both relatively new compared with the classical specialties in medicine. Having its naissance in the 1970s and later adoption in the 1980s, the use of MRI has grown exponentially, consequently engendering exciting new areas of research. One such development is the use of computational techniques to analyze MRI images much like the way a radiologist would. With the advent of affordable, powerful computing hardware and parallel developments in computer vision, MRI image analysis has also witnessed unprecedented growth. Due to the interdisciplinary and complex nature of this subfield, it is important to survey the current landscape and examine the current approaches for analysis and trend trends moving forward.

RIFT: Multi-modal Image Matching Based on Radiation-variation Insensitive Feature Transform

Traditional feature matching methods, such as scale-invariant feature transform (SIFT), usually use image intensity or gradient information to detect and describe feature points; however, both intensity and gradient are sensitive to nonlinear radiation distortions (NRD). To solve this problem, this paper proposes a novel feature matching algorithm that is robust to large NRD. The proposed method is called radiation-variation insensitive feature transform (RIFT). There are three main contributions in RIFT. First, RIFT uses phase congruency (PC) instead of image intensity for feature point detection. RIFT considers both the number and repeatability of feature points and detects both corner points and edge points on the PC map. Second, RIFT originally proposes a maximum index map (MIM) for feature description. The MIM is constructed from the log-Gabor convolution sequence and is much more robust to NRD than traditional gradient map. Thus, RIFT not only largely improves the stability of feature detection but also overcomes the limitation of gradient information for feature description. Third, RIFT analyses the inherent influence of rotations on the values of the MIM and realises rotation invariance. We use six different types of multi-modal image datasets to evaluate RIFT, including optical-optical, infrared-optical, synthetic aperture radar (SAR)-optical, depth-optical, map-optical, and day-night datasets. Experimental results show that RIFT is superior to SIFT and SAR-SIFT on multi-modal images. To the best of our knowledge, RIFT is the first feature matching algorithm that can achieve good performance on all the abovementioned types of multi-modal images. The source code of RIFT and the multi-modal image datasets are publicly available1.

Comparison of Various Frequency Matching Schemes for Geometric Correction of Geostationary Ocean Color Imager

Current precise geometric correction of Geostationary Ocean Color Imager (GOCI) image slots is performed by shoreline matching. However, it is troublesome to handle slots with few or no shorelines, or slots covered by clouds. Geometric correction by frequency matching has been proposed to handle these slots. In this paper, we further extend previous research on frequency matching by comparing the performance of three frequency domain matching methods: phase correlation, gradient correlation, and orientation correlation. We compared the performance of each matching technique in terms of match success rate and geometric accuracy. We concluded that the three frequency domain matching method with peak search range limits was comparable to geometric correction performance with shoreline matching. The proposed method handles translation only, and assumes that rotation has been corrected. We need to do further work on how to handle rotation by frequency matching.

Rotation Estimation: A Closed-Form Solution Using Spherical Moments

Photometric moments are global descriptors of an image that can be used to recover motion information. This paper uses spherical photometric moments for a closed form estimation of 3D rotations from images. Since the used descriptors are global and not of the geometrical kind, they allow to avoid image processing as features extraction, matching, and tracking. The proposed scheme based on spherical projection can be used for the different vision sensors obeying the central unified model: conventional, fisheye, and catadioptric. Experimental results using both synthetic data and real images in different scenarios are provided to show the efficiency of the proposed method.

Evaluation of Indirect Methods for Motion Compensation in 2-D Focal Liver Lesion Contrast-Enhanced Ultrasound (CEUS) Imaging

This study investigates the application and evaluation of existing indirect methods, namely point-based registration techniques, for the estimation and compensation of observed motion included in the 2-D image plane of contrast-enhanced ultrasound (CEUS) cine-loops recorded for the characterization and diagnosis of focal liver lesions (FLLs). The value of applying motion compensation in the challenging modality of CEUS is to assist in the quantification of the perfusion dynamics of an FLL in relation to its parenchyma, allowing for a potentially accurate diagnostic suggestion. Towards this end, this study also proposes a novel quantitative multi-level framework for evaluating the quantification of FLLs, which to the best of our knowledge remains undefined, notwithstanding many relevant studies. Following quantitative evaluation of 19 indirect algorithms and configurations, while also considering the requirement for computational efficiency, our results suggest that the "compact and real-time descriptor" (CARD) is the optimal indirect motion compensation method in CEUS.

Motion Estimation by Hybrid Optical Flow Technology for UAV Landing in an Unvisited Area

The capability of landing on previously unvisited areas is a fundamental challenge for an unmanned aerial vehicle (UAV). In this paper, we developed a vision-based motion estimation as an aid to improve landing performance. As an alternative to the common scenarios accompanying by external infrastructures or well-defined marker, the proposed hybrid framework can successfully land on a new area without any prior information about guiding marks. The implementation was based on the optical flow technique associated with a multi-scale strategy to overcome the decreasing field-of-view during the UAV descending. Compared with a commercial Global Positioning System (GPS) through a sequence of flight trials, the vision-aided scheme can effectively minimize the possible sensing error, thus, leading to a more accurate result. Moreover, this work has potential to integrate the fast-growing image learning process and yields more practical versatility for UAV applications in the future.

Geometric Correction for the Geostationary Ocean Color Imager from a Combination of Shoreline Matching and Frequency Matching

Geometric correction is fundamental in producing high quality satellite data products. However, the geometric correction for ocean color sensors, e.g., Geostationary Ocean Color Imager (GOCI), is challenging because the traditional method based on ground control points (GCPs) cannot be applied when the shoreline is absent. In this study, we develop a hybrid geometric correction method, which applies shoreline matching and frequency matching on slots with shorelines and without shorelines, respectively. Frequency matching has been proposed to estimate the relative orientation between GOCI slots without a shoreline. In this paper, we extend our earlier research for absolute orientation and geometric correction by combining frequency matching results with shoreline matching ones. The proposed method consists of four parts: Initial sensor modeling of slots without shorelines, precise sensor modeling through shoreline matching, relative orientation modeling by frequency matching, and generation of geometric correction results using a combination of the two matching procedures. Initial sensor modeling uses the sensor model equation for GOCI and metadata in order to remove geometric distortion due to the Earth’s rotation and curvature in the slots without shorelines. Precise sensor modeling is performed with shoreline matching and random sample consensus (RANSAC) in the slots with shorelines. Frequency matching computes position shifts for slots without shorelines with respect to the precisely corrected slots with shorelines. GOCI Level 1B scenes are generated by combining the results from shoreline matching and frequency matching. We analyzed the accuracy of shoreline matching alone against that of the combination of shoreline matching and frequency matching. Both methods yielded a similar accuracy of 1.2 km, which supports the idea that frequency matching can replace traditional shoreline matching for slots without visible shorelines.

Thalamocortical Axonal Activity in Motor Cortex Exhibits Layer-Specific Dynamics during Motor Learning

The thalamus is the hub through which neural signals are transmitted from the basal ganglia and cerebellum to the neocortex. However, thalamocortical axonal activity during motor learning remains largely undescribed. We conducted two-photon calcium imaging of thalamocortical axonal activity in the motor cortex of mice learning a self-initiated lever-pull task. Layer 1 (L1) axons came to exhibit activity at lever-pull initiation and termination, while layer 3 (L3) axons did so at lever-pull initiation. L1 population activity had a sequence structure related to both lever-pull duration and reproducibility. Stimulation of the substantia nigra pars reticulata activated more L1 than L3 axons, whereas deep cerebellar nuclei (DCN) stimulation did the opposite. Lesions to either the dorsal striatum or the DCN impaired motor learning and disrupted temporal dynamics in both layers. Thus, layer-specific thalamocortical signals evolve with the progression of learning, which requires both the basal ganglia and cerebellar activities.

Two-photon imaging of neuronal activity in motor cortex of marmosets during upper-limb movement tasks

Two-photon imaging in behaving animals has revealed neuronal activities related to behavioral and cognitive function at single-cell resolution. However, marmosets have posed a challenge due to limited success in training on motor tasks. Here we report the development of protocols to train head-fixed common marmosets to perform upper-limb movement tasks and simultaneously perform two-photon imaging. After 2–5 months of training sessions, head-fixed marmosets can control a manipulandum to move a cursor to a target on a screen. We conduct two-photon calcium imaging of layer 2/3 neurons in the motor cortex during this motor task performance, and detect task-relevant activity from multiple neurons at cellular and subcellular resolutions. In a two-target reaching task, some neurons show direction-selective activity over the training days. In a short-term force-field adaptation task, some neurons change their activity when the force field is on. Two-photon calcium imaging in behaving marmosets may become a fundamental technique for determining the spatial organization of the cortical dynamics underlying action and cognition.

Marmosets are an important model organism in neuroscience but there has only been limited success in training them on behavioral tasks. Here the authors report their ability to train marmosets in various motor tasks and simultaneously image neural dynamics in motor cortex with 2-photon imaging.

Local All-Pass Geometric Deformations

This paper deals with the estimation of a deformation that describes the geometric transformation between two images. To solve this problem, we propose a novel framework that relies upon the brightness consistency hypothesis-a pixel's intensity is maintained throughout the transformation. Instead of assuming small distortion and linearizing the problem (e.g. via Taylor Series expansion), we propose to interpret the brightness hypothesis as an all-pass filtering relation between the two images. The key advantages of this new interpretation are that no restrictions are placed on the amplitude of the deformation or on the spatial variations of the images. Moreover, by converting the all-pass filtering to a linear forward-backward filtering relation, our solution to the estimation problem equates to solving a linear system of equations, which leads to a highly efficient implementation. Using this framework, we develop a fast algorithm that relates one image to another, on a local level, using an all-pass filter and then extracts the deformation from the filter-hence the name "Local All-Pass" (LAP) algorithm. The effectiveness of this algorithm is demonstrated on a variety of synthetic and real deformations that are found in applications, such as image registration and motion estimation. In particular, when compared with a selection of image registration algorithms, the LAP obtains very accurate results for significantly reduced computation time and is very robust to noise corruption.

A hybrid method for 3D mosaicing of OCT images of macula and Optic Nerve Head

A mosaiced image is the result of merging two or more images with overlapping area in order to generate a high resolution panorama of a large scene. A wide view of Optical Coherence Tomography (OCT) images can help clinicians in diagnosis by enabling simultaneous analysis of different portions of the gathered information. In this paper, we present a novel method for mosaicing of 3D OCT images of macula and Optic Nerve Head (ONH) that is carried out in two phases; registration of OCT projections and mosaicing of B-scans. In the first phase, in order to register the OCT projection images of macula and ONH, their corresponding color fundus image is considered as the main frame and the geometrical features of their curvelet-based extracted vessels are employed for registration. The registration parameters obtained are then applied on all x-y slices of the 3D OCT images of macula and ONH. In the B-scan mosaicing phase, the overlapping areas of corresponding reprojected B-scans are extracted and the best registration model is obtained based on line-by-line matching of corresponding A-scans in overlapping areas. This registration model is then applied to the remaining A-scans of the ONH-based B-scan. The aligned B-scans of macular OCT and OCT images of ONH are finally blended and 3D mosaiced OCT images are obtained. Two criteria are considered for assessment of mosaiced images; the quality of alignment/mosaicing of B-scans and the loss of clinical information from the B-scans after mosaicing. The average grading values of 3.5 ± 0.74 and 3.63 ± 0.55 (out of 4) are obtained for the first and second criteria, respectively.

Label-free tissue scanner for colorectal cancer screening

The current practice of surgical pathology relies on external contrast agents to reveal tissue architecture, which is then qualitatively examined by a trained pathologist. The diagnosis is based on the comparison with standardized empirical, qualitative assessments of limited objectivity. We propose an approach to pathology based on interferometric imaging of “unstained” biopsies, which provides unique capabilities for quantitative diagnosis and automation. We developed a label-free tissue scanner based on “quantitative phase imaging,” which maps out optical path length at each point in the field of view and, thus, yields images that are sensitive to the “nanoscale” tissue architecture. Unlike analysis of stained tissue, which is qualitative in nature and affected by color balance, staining strength and imaging conditions, optical path length measurements are intrinsically quantitative, i.e., images can be compared across different instruments and clinical sites. These critical features allow us to automate the diagnosis process. We paired our interferometric optical system with highly parallelized, dedicated software algorithms for data acquisition, allowing us to image at a throughput comparable to that of commercial tissue scanners while maintaining the nanoscale sensitivity to morphology. Based on the measured phase information, we implemented software tools for autofocusing during imaging, as well as image archiving and data access. To illustrate the potential of our technology for large volume pathology screening, we established an “intrinsic marker” for colorectal disease that detects tissue with dysplasia or colorectal cancer and flags specific areas for further examination, potentially improving the efficiency of existing pathology workflows.

Surface single-molecule dynamics controlled by entropy at low temperatures

Configuration transitions of individual molecules and atoms on surfaces are traditionally described using an Arrhenius equation with energy barrier and pre-exponential factor (attempt rate) parameters. Characteristic parameters can vary even for identical systems, and pre-exponential factors sometimes differ by orders of magnitude. Using low-temperature scanning tunnelling microscopy (STM) to measure an individual dibutyl sulfide molecule on Au(111), we show that the differences arise when the relative position of tip apex and molecule changes by a fraction of the molecule size. Altering the tip position on that scale modifies the transition's barrier and attempt rate in a highly correlated fashion, which results in a single-molecular enthalpy-entropy compensation. Conversely, appropriately positioning the STM tip allows selecting the operating point on the compensation line and modifying the transition rates. The results highlight the need to consider entropy in transition rates of single molecules, even at low temperatures.

STM is capable of imaging the configurations of molecules on surfaces and measuring the rate of transitions between them. Here the authors demonstrate that, controlled by the STM tip position, the entropic and conservative forces on the molecule can modify the rate by orders of magnitude.

Matching Aerial Images to 3D Building Models Using Context-Based Geometric Hashing

A city is a dynamic entity, which environment is continuously changing over time. Accordingly, its virtual city models also need to be regularly updated to support accurate model-based decisions for various applications, including urban planning, emergency response and autonomous navigation. A concept of continuous city modeling is to progressively reconstruct city models by accommodating their changes recognized in spatio-temporal domain, while preserving unchanged structures. A first critical step for continuous city modeling is to coherently register remotely sensed data taken at different epochs with existing building models. This paper presents a new model-to-image registration method using a context-based geometric hashing (CGH) method to align a single image with existing 3D building models. This model-to-image registration process consists of three steps: (1) feature extraction; (2) similarity measure; and matching, and (3) estimating exterior orientation parameters (EOPs) of a single image. For feature extraction, we propose two types of matching cues: edged corner features representing the saliency of building corner points with associated edges, and contextual relations among the edged corner features within an individual roof. A set of matched corners are found with given proximity measure through geometric hashing, and optimal matches are then finally determined by maximizing the matching cost encoding contextual similarity between matching candidates. Final matched corners are used for adjusting EOPs of the single airborne image by the least square method based on collinearity equations. The result shows that acceptable accuracy of EOPs of a single image can be achievable using the proposed registration approach as an alternative to a labor-intensive manual registration process.

A Framework for 3D Vessel Analysis using Whole Slide Images of Liver Tissue Sections

Three-dimensional (3D) high resolution microscopic images have high potential for improving the understanding of both normal and disease processes where structural changes or spatial relationship of disease features are significant. In this paper, we develop a complete framework applicable to 3D pathology analytical imaging, with an application to whole slide images of sequential liver slices for 3D vessel structure analysis. The analysis workflow consists of image registration, segmentation, vessel cross-section association, interpolation, and volumetric rendering. To identify biologically-meaningful correspondence across adjacent slides, we formulate a similarity function for four association cases. The optimal solution is then obtained by constrained Integer Programming. We quantitatively and qualitatively compare our vessel reconstruction results with human annotations. Validation results indicate a satisfactory concordance as measured both by region-based and distance-based metrics. These results demonstrate a promising 3D vessel analysis framework for whole slide images of liver tissue sections.

A Motion Detection Algorithm Using Local Phase Information

Previous research demonstrated that global phase alone can be used to faithfully represent visual scenes. Here we provide a reconstruction algorithm by using only local phase information. We also demonstrate that local phase alone can be effectively used to detect local motion. The local phase-based motion detector is akin to models employed to detect motion in biological vision, for example, the Reichardt detector. The local phase-based motion detection algorithm introduced here consists of two building blocks. The first building block measures/evaluates the temporal change of the local phase. The temporal derivative of the local phase is shown to exhibit the structure of a second order Volterra kernel with two normalized inputs. We provide an efficient, FFT-based algorithm for implementing the change of the local phase. The second processing building block implements the detector; it compares the maximum of the Radon transform of the local phase derivative with a chosen threshold. We demonstrate examples of applying the local phase-based motion detection algorithm on several video sequences. We also show how the locally detected motion can be used for segmenting moving objects in video scenes and compare our local phase-based algorithm to segmentation achieved with a widely used optic flow algorithm.

Mapping the stereotyped behaviour of freely moving fruit flies

A frequent assumption in behavioural science is that most of an animal's activities can be described in terms of a small set of stereotyped motifs. Here, we introduce a method for mapping an animal's actions, relying only upon the underlying structure of postural movement data to organize and classify behaviours. Applying this method to the ground-based behaviour of the fruit fly, Drosophila melanogaster, we find that flies perform stereotyped actions roughly 50% of the time, discovering over 100 distinguishable, stereotyped behavioural states. These include multiple modes of locomotion and grooming. We use the resulting measurements as the basis for identifying subtle sex-specific behavioural differences and revealing the low-dimensional nature of animal motions.

Stabilizing 3D in vivo intravital microscopy images with an iteratively refined soft-tissue model for immunology experiments

We describe a set of new algorithms and a software tool, StabiTissue, for stabilizing in vivo intravital microscopy images that suffer from soft-tissue background movement. Because these images lack predetermined anchors and are dominated by noise, we use a pixel weighted image alignment together with a correction for nonlinear tissue deformations. We call this correction a poor man׳s diffeomorphic map since it ascertains the nonlinear regions of the image without resorting to a full integral equation method. To determine the quality of the image stabilization, we developed an ensemble sampling method that quantifies the coincidence between image pairs from randomly distributed image regions. We obtain global stabilization alignment through an iterative constrained simulated annealing optimization procedure. To show the accuracy of our algorithm with existing software, we measured the misalignment error rate in datasets taken from two different organs and compared the results to a similar and popular open-source solution. Present open-source stabilization software tools perform poorly because they do not treat the specific needs of the IV-2pM datasets with soft-tissue deformation, speckle noise, full 5D inter- and intra-stack motion error correction, and undefined anchors. In contrast, the results of our tests demonstrate that our method is more immune to noise and provides better performance for datasets' possessing nonlinear tissue deformations. As a practical application of our software, we show how our stabilization improves cell tracking, where the presence of background movement would degrade track information. We also provide a qualitative comparison of our software with other open-source libraries/applications. Our software is freely available at the open source repository http://sourceforge.net/projects/stabitissue/.

On the role of spatial phase and phase correlation in vision, illusion, and cognition

Numerous findings indicate that spatial phase bears an important cognitive information. Distortion of phase affects topology of edge structures and makes images unrecognizable. In turn, appropriately phase-structured patterns give rise to various illusions of virtual image content and apparent motion. Despite a large body of phenomenological evidence not much is known yet about the role of phase information in neural mechanisms of visual perception and cognition. Here, we are concerned with analysis of the role of spatial phase in computational and biological vision, emergence of visual illusions and pattern recognition. We hypothesize that fundamental importance of phase information for invariant retrieval of structural image features and motion detection promoted development of phase-based mechanisms of neural image processing in course of evolution of biological vision. Using an extension of Fourier phase correlation technique, we show that the core functions of visual system such as motion detection and pattern recognition can be facilitated by the same basic mechanism. Our analysis suggests that emergence of visual illusions can be attributed to presence of coherently phase-shifted repetitive patterns as well as the effects of acuity compensation by saccadic eye movements. We speculate that biological vision relies on perceptual mechanisms effectively similar to phase correlation, and predict neural features of visual pattern (dis)similarity that can be used for experimental validation of our hypothesis of “cognition by phase correlation.”

Projection Operators and Moment Invariants to Image Blurring

In this paper we introduce a new theory of blur invariants. Blur invariants are image features which preserve their values if the image is convolved by a point-spread function (PSF) of a certain class. We present the invariants to convolution with an arbitrary N-fold symmetric PSF, both in Fourier and image domain. We introduce a notion of a primordial image as a canonical form of all blur-equivalent images. It is defined in spectral domain by means of projection operators. We prove that the moments of the primordial image are invariant to blur and we derive recursive formulae for their direct computation without actually constructing the primordial image. We further prove they form a complete set of invariants and show how to extent their invariance also to translation, rotation and scaling. We illustrate by simulated and real-data experiments their invariance and recognition power. Potential applications of this method are wherever one wants to recognize objects on blurred images.

Quantitative shear elasticity imaging from a complex elastic wavefield in soft solids with application to passive elastography

In passive elastography, the complex physiological noise present in the human body is used to conduct an elastography experiment. In the present work, quantitative shear elasticity imaging from a complex elastic wavefield is demonstrated in soft solids. By correlating the elastic field at different positions, which can be interpreted as a time-reversal experiment, shear waves are virtually focused on any point of the imaging plane. According to the Rayleigh criterion, the focus size is directly related to the shear wave speed and thus to the shear elasticity. To locally retrieve a shear wave speed estimation, analytical and empirical expressions that relate the focus size with the shear wave speed and the frequency band used in the correlation computation are derived. The validity of such expressions is demonstrated numerically and experimentally on a tissue-mimicking phantom consisting of two different elastic layers. The obtained results were in complete agreement with a prior shear wave speed estimation demonstrating the potential of the technique to quantitative shear elasticity assessment using a complex elastic wavefield. Finally, an ultraslow experiment at an imaging rate of 10 Hz shows the technique to be compatible with slow imaging devices such as standard echographs or MRI scanners.

Registration of images with N -fold dihedral blur

In this paper, we extend our recent registration method designed specifically for registering blurred images. The original method works for unknown blurs, assuming the blurring point-spread function (PSF) exhibits an N -fold rotational symmetry. Here, we also generalize the theory to the case of dihedrally symmetric blurs, which are produced by the PSFs having both rotational and axial symmetries. Such kind of blurs are often found in unfocused images acquired by digital cameras, as in out-of-focus shots the PSF typically mimics the shape of the shutter aperture. This makes our registration algorithm particularly well-suited in applications where blurred image registration must be used as a preprocess step of an image fusion algorithm, and where common registration methods fail, due to the amount of blur. We demonstrate that the proposed method leads to an improvement of the registration performance, and we show its applicability to real images by providing successful examples of blurred image registration followed by depth-of-field extension and multichannel blind deconvolution.

Registration of adaptive optics corrected retinal nerve fiber layer (RNFL) images

Glaucoma is the leading cause of preventable blindness in the western world. Investigation of high-resolution retinal nerve fiber layer (RNFL) images in patients may lead to new indicators of its onset. Adaptive optics (AO) can provide diffraction-limited images of the retina, providing new opportunities for earlier detection of neuroretinal pathologies. However, precise processing is required to correct for three effects in sequences of AO-assisted, flood-illumination images: uneven illumination, residual image motion and image rotation. This processing can be challenging for images of the RNFL due to their low contrast and lack of clearly noticeable features. Here we develop specific processing techniques and show that their application leads to improved image quality on the nerve fiber bundles. This in turn improves the reliability of measures of fiber texture such as the correlation of Gray-Level Co-occurrence Matrix (GLCM).

Markerless Lung Tumor Motion Tracking by Dynamic Decomposition of X-Ray Image Intensity

We propose a new markerless tracking technique of lung tumor motion by using an X-ray fluoroscopic image sequence for real-time image-guided radiation therapy (IGRT). A core innovation of the new technique is to extract a moving tumor intensity component from the fluoroscopic image intensity. The fluoroscopic intensity is the superimposition of intensity components of all the structures passed through by the X-ray. The tumor can then be extracted by decomposing the fluoroscopic intensity into the tumor intensity component and the others. The decomposition problem for more than two structures is ill posed, but it can be transformed into a well-posed one by temporally accumulating constraints that must be satisfied by the decomposed moving tumor component and the rest of the intensity components. The extracted tumor image can then be used to achieve accurate tumor motion tracking without implanted markers that are widely used in the current tracking techniques. The performance evaluation showed that the extraction error was sufficiently small and the extracted tumor tracking achieved a high and sufficient accuracy less than 1 mm for clinical datasets. These results clearly demonstrate the usefulness of the proposed method for markerless tumor motion tracking.

Automated image mosaics by non-automated light microscopes: the MicroMos software tool

Light widefield microscopes and digital imaging are the basis for most of the analyses performed in every biological laboratory. In particular, the microscope's user is typically interested in acquiring high-detailed images for analysing observed cells and tissues, meanwhile being representative of a wide area to have reliable statistics. The microscopist has to choose between higher magnification factor and extension of the observed area, due to the finite size of the camera's field of view. To overcome the need of arrangement, mosaicing techniques have been developed in the past decades for increasing the camera's field of view by stitching together more images. Nevertheless, these approaches typically work in batch mode and rely on motorized microscopes. Or alternatively, the methods are conceived just to provide visually pleasant mosaics not suitable for quantitative analyses. This work presents a tool for building mosaics of images acquired with nonautomated light microscopes. The method proposed is based on visual information only and the mosaics are built by incrementally stitching couples of images, making the approach available also for online applications. Seams in the stitching regions as well as tonal inhomogeneities are corrected by compensating the vignetting effect. In the experiments performed, we tested different registration approaches, confirming that the translation model is not always the best, despite the fact that the motion of the sample holder of the microscope is apparently translational and typically considered as such. The method's implementation is freely distributed as an open source tool called MicroMos. Its usability makes building mosaics of microscope images at subpixel accuracy easier. Furthermore, optional parameters for building mosaics according to different strategies make MicroMos an easy and reliable tool to compare different registration approaches, warping models and tonal corrections.

Blur invariant translational image registration for N-fold symmetric blurs

In this paper, we propose a new registration method designed particularly for registering differently blurred images. Such a task cannot be successfully resolved by traditional approaches. Our method is inspired by traditional phase correlation, which is now applied to certain blur-invariant descriptors instead of the original images. This method works for unknown blurs assuming the blurring PSF exhibits an N-fold rotational symmetry. It does not require any landmarks. We have experimentally proven its good performance, which is not dependent on the amount of blur. In this paper, we explicitly address only registration with respect to translation, but the method can be readily generalized to rotation and scaling.

Near-field effects in Green's function retrieval from cross-correlation of elastic fields: experimental study with application to elastography

In a lossless system, the causal and acausal Green's function for elastic waves can be retrieved by cross-correlating the elastic field at two positions. This field, composed of converging and diverging waves, is interpreted in the frame of a time-reversal process. In this work, the near-field effects on the spatio-temporal focusing of elastic waves are analyzed through the elastodynamic Green's function. Contrary to the scalar field case, the spatial focusing is not symmetric preserving the directivity pattern of a simple source. One important feature of the spatial asymmetry is its dependency on the Poisson ratio of the solid. Additionally, it is shown that the retrieval of the bulk wave speed values is affected by diffraction. The correction factor depends on the relative direction between the source and the observed field. Experimental verification of the analysis is carried out on the volume of a soft-solid. A low-frequency diffuse-like field is generated by random impacts at the sample's free surface. The displacement field is imaged using ultrasound by a standard speckle tracking technique. One important application of this work is in the estimation of the shear elastic modulus in soft biological tissues, whose quantification can be useful in non-invasive diagnosis of various diseases.

Spectral 6DOF registration of noisy 3D range data with partial overlap

We present Spectral Registration with Multilayer Resampling (SRMR) as a 6 Degrees Of Freedom (DOF) registration method for noisy 3D data with partial overlap. The algorithm is based on decoupling 3D rotation from 3D translation by a corresponding resampling process of the spectral magnitude of a 3D Fast Fourier Transform (FFT) calculation on discretized 3D range data. The registration of all 6DOF is then subsequently carried out with spectral registrations using Phase Only Matched Filtering (POMF). There are two main aspects for the fast and robust registration of Euler angles from spherical information in SRMR. First of all, there is the permanent use of phase matching. Second, based on the FFT on a discrete Cartesian grid, not only one spherical layer but also a complete stack of layers are processed in one step. Experiments are presented with challenging datasets with respect to interference and overlap. The results include the fast and robust registration of artificially transformed data for ground-truth comparison, scans from the Stanford Bunny dataset, high end 3D laser range finder (LRF) scans of a city center, and range data from a low-cost actuated LRF in a disaster response scenario.

Registration of plantar pressure images

In this work, five computational methodologies to register plantar pressure images are compared: (1) the first methodology is based on matching the external contours of the feet; (2) the second uses the phase correlation technique; (3) the third addresses the direct maximization of cross-correlation using the Fourier transform; (4) the fourth minimizes the sum of squared differences using the Fourier transform; and (5) the fifth methodology iteratively optimizes an intensity (dis)similarity measure based on Powell's method. The accuracy and robustness of the five methodologies were assessed by using images from three common plantar pressure acquisition devices: a Footscan system, an EMED system, and a light reflection system. Using the residual error as a measure of accuracy, all methodologies revealed to be very accurate even in the presence of noise. The most accurate was the methodology based on the iterative optimization, when the mean squared error was minimized. It achieved a residual error inferior to 0.01 mm and 0.6 mm for non-noisy and noisy images, respectively. On the other hand, the methodology based on image contour matching was the fastest, but its accuracy was the lowest.