Object tracking with particle filtering in fluorescence microscopy images: application to the motion of neurofilaments in axons

Deep probabilistic tracking of particles in fluorescence microscopy images

Tracking of particles in temporal fluorescence microscopy image sequences is of fundamental importance to quantify dynamic processes of intracellular structures as well as virus structures. We introduce a probabilistic deep learning approach for fluorescent particle tracking, which is based on a recurrent neural network that mimics classical Bayesian filtering. Compared to previous deep learning methods for particle tracking, our approach takes into account uncertainty, both aleatoric and epistemic uncertainty. Thus, information about the reliability of the computed trajectories is determined. Manual tuning of tracking parameters is not necessary and prior knowledge about the noise statistics is not required. Short and long-term temporal dependencies of individual object dynamics are exploited for state prediction, and assigned detections are used to update the predicted states. For correspondence finding, we introduce a neural network which computes assignment probabilities jointly across multiple detections as well as determines the probabilities of missing detections. Training requires only simulated data and therefore tedious manual annotation of ground truth is not needed. We performed a quantitative performance evaluation based on synthetic and real 2D as well as 3D fluorescence microscopy images. We used image data of the Particle Tracking Challenge as well as real time-lapse fluorescence microscopy images displaying virus structures and chromatin structures. It turned out that our approach yields state-of-the-art results or improves the tracking results compared to previous methods.

A Recurrent Neural Network for Particle Tracking in Microscopy Images Using Future Information, Track Hypotheses, and Multiple Detections

Automatic tracking of particles in time-lapse fluorescence microscopy images is essential for quantifying the dynamic behavior of subcellular structures and virus structures. We introduce a novel particle tracking approach based on a deep recurrent neural network architecture that exploits past and future information in both forward and backward direction. Assignment probabilities are determined jointly across multiple detections, and the probability of missing detections is computed. In addition, existence probabilities are determined by the network to handle track initiation and termination. For correspondence finding, track hypotheses are propagated to future time points so that information at later time points can be used to resolve ambiguities. A handcrafted similarity measure and handcrafted motion features are not necessary. Manually labeled data is not required for network training. We evaluated the performance of our approach using image data of the Particle Tracking Challenge as well as real fluorescence microscopy image sequences of virus structures. It turned out that the proposed approach outperforms previous methods.

Analyzing temporal dynamics of cell deformation and intracellular movement with video feature aggregation

BACKGROUND

The research and analysis of cellular physiological properties has been an essential approach to studying some biological and biomedical problems. Temporal dynamics of cells therein are used as a quantifiable indicator of cellular response to extracellular cues and physiological stimuli.

METHODS

This work presents a novel image-based framework to profile and model the cell dynamics in live-cell videos. In the framework, the cell dynamics between frames are represented as frame-level features from cell deformation and intracellular movement. On the one hand, shape context is introduced to enhance the robustness of measuring the deformation of cellular contours. On the other hand, we employ Scale-Invariant Feature Transform (SIFT) flow to simultaneously construct the complementary movement field and appearance change field for the cytoplasmic streaming. Then, time series modeling is performed on these frame-level features. Specifically, temporal feature aggregation is applied to capture the video-wide temporal evolution of cell dynamics.

RESULTS

Our results demonstrate that the proposed cell dynamic features can effectively capture the cell dynamics in videos. They also prove that the Movement Field and Appearance Change Field Feature (MFAFF) can more precisely model the cytoplasmic streaming. Besides, temporal aggregation of cell dynamic features brings a substantial absolute increase of classification performance.

CONCLUSION

Experimental results demonstrate that the proposed framework outperforms competing mainstreaming approaches on the aforementioned datasets. Thus, our method has potential for cell dynamics analysis in videos.

A Bayesian framework for the analog reconstruction of kymographs from fluorescence microscopy data

Kymographs are widely used to represent and analyse spatio-temporal dynamics of fluorescence markers along curvilinear biological compartments. These objects have a singular geometry, thus kymograph reconstruction is inherently an analog image processing task. However, the existing approaches are essentially digital: the kymograph photometry is sampled directly from the time-lapse images. As a result, such kymographs rely on raw image data that suffer from the degradations entailed by the image formation process and the spatio-temporal resolution of the imaging setup. In this work, we address these limitations and introduce a well-grounded Bayesian framework for the analog reconstruction of kymographs. To handle the movement of the object, we introduce an intrinsic description of kymographs using differential geometry: a kymograph is a photometry defined on a parameter space that is embedded in physical space by a time-varying map that follows the object geometry. We model the kymograph photometry as a Lévy innovation process, a flexible class of non-parametric signal priors. We account for the image formation process using the virtual microscope framework. We formulate a computationally tractable representation of the associated maximum a posteriori problem and solve it using a class of efficient and modular algorithms based on the alternating split Bregman. We assess the performance of our Bayesian framework on synthetic data and apply it to reconstruct the fluorescence dynamics along microtubules in vivo in the budding yeast S. cerevisiae. We demonstrate that our framework allows revealing patterns from single time-lapse data that are invisible on standard digital kymographs.

Computational Analysis of Cell Dynamics in Videos with Hierarchical-Pooled Deep-Convolutional Features

Computational analysis of cellular appearance and its dynamics is used to investigate physiological properties of cells in biomedical research. In consideration of the great success of deep learning in video analysis, we first introduce two-stream convolutional networks (ConvNets) to automatically learn the biologically meaningful dynamics from raw live-cell videos. However, the two-stream ConvNets lack the ability to capture long-range video evolution. Therefore, a novel hierarchical pooling strategy is proposed to model the cell dynamics in a whole video, which is composed of trajectory pooling for short-term dynamics and rank pooling for long-range ones. Experimental results demonstrate that the proposed pipeline effectively captures the spatiotemporal dynamics from the raw live-cell videos and outperforms existing methods on our cell video database.

Live-cell imaging of neurofilament transport in cultured neurons

Neurofilaments, which are the intermediate filaments of nerve cells, are space-filling cytoskeletal polymers that contribute to the growth of axonal caliber. In addition to their structural role, neurofilaments are cargos of axonal transport that move along microtubule tracks in a rapid, intermittent, and bidirectional manner. Though they measure just 10nm in diameter, which is well below the diffraction limit of optical microscopes, these polymers can reach 100 μm or more in length and are often packed densely, just tens of nanometers apart. These properties of neurofilaments present unique challenges for studies on their movement. In this article, we describe several live-cell fluorescence imaging strategies that we have developed to image neurofilament transport in axons of cultured neurons on short and long timescales. Together, these methods form a powerful set of complementary tools with which to study the axonal transport of these unique intracellular cargos.

Micro-object motion tracking based on the probability hypothesis density particle tracker

Tracking micro-objects in the noisy microscopy image sequences is important for the analysis of dynamic processes in biological objects. In this paper, an automated tracking framework is proposed to extract the trajectories of micro-objects. This framework uses a probability hypothesis density particle filtering (PF-PHD) tracker to implement a recursive state estimation and trajectories association. In order to increase the efficiency of this approach, an elliptical target model is presented to describe the micro-objects using shape parameters instead of point-like targets which may cause inaccurate tracking. A novel likelihood function, not only covering the spatiotemporal distance but also dealing with geometric shape function based on the Mahalanobis norm, is proposed to improve the accuracy of particle weight in the update process of the PF-PHD tracker. Using this framework, a larger number of tracks are obtained. The experiments are performed on simulated data of microtubule movements and real mouse stem cells. We compare the PF-PHD tracker with the nearest neighbor method and the multiple hypothesis tracking method. Our PF-PHD tracker can simultaneously track hundreds of micro-objects in the microscopy image sequence.

Multi-Target Tracking With Time-Varying Clutter Rate and Detection Profile: Application to Time-Lapse Cell Microscopy Sequences

Quantitative analysis of the dynamics of tiny cellular and sub-cellular structures, known as particles, in time-lapse cell microscopy sequences requires the development of a reliable multi-target tracking method capable of tracking numerous similar targets in the presence of high levels of noise, high target density, complex motion patterns and intricate interactions. In this paper, we propose a framework for tracking these structures based on the random finite set Bayesian filtering framework. We focus on challenging biological applications where image characteristics such as noise and background intensity change during the acquisition process. Under these conditions, detection methods usually fail to detect all particles and are often followed by missed detections and many spurious measurements with unknown and time-varying rates. To deal with this, we propose a bootstrap filter composed of an estimator and a tracker. The estimator adaptively estimates the required meta parameters for the tracker such as clutter rate and the detection probability of the targets, while the tracker estimates the state of the targets. Our results show that the proposed approach can outperform state-of-the-art particle trackers on both synthetic and real data in this regime.

Tracking multiple particles in fluorescence time-lapse microscopy images via probabilistic data association

Tracking subcellular structures as well as viral structures displayed as 'particles' in fluorescence microscopy images yields quantitative information on the underlying dynamical processes. We have developed an approach for tracking multiple fluorescent particles based on probabilistic data association. The approach combines a localization scheme that uses a bottom-up strategy based on the spot-enhancing filter as well as a top-down strategy based on an ellipsoidal sampling scheme that uses the Gaussian probability distributions computed by a Kalman filter. The localization scheme yields multiple measurements that are incorporated into the Kalman filter via a combined innovation, where the association probabilities are interpreted as weights calculated using an image likelihood. To track objects in close proximity, we compute the support of each image position relative to the neighboring objects of a tracked object and use this support to recalculate the weights. To cope with multiple motion models, we integrated the interacting multiple model algorithm. The approach has been successfully applied to synthetic 2-D and 3-D images as well as to real 2-D and 3-D microscopy images, and the performance has been quantified. In addition, the approach was successfully applied to the 2-D and 3-D image data of the recent Particle Tracking Challenge at the IEEE International Symposium on Biomedical Imaging (ISBI) 2012.

Intelligent vision guide for automatic ventilation grommet insertion into the tympanic membrane

BACKGROUND

Otitis media with effusion is a worldwide ear disease. The current treatment is to surgically insert a ventilation grommet into the tympanic membrane. A robotic device allowing automatic grommet insertion has been designed in a previous study; however, the part of the membrane where the malleus bone is attached to the inner surface is to be avoided during the insertion process.

METHODS

This paper proposes a synergy of optical flow technique and a gradient vector flow active contours algorithm to achieve an online tracking of the malleus under endoscopic vision, to guide the working channel to move efficiently during the surgery.

RESULTS

The proposed method shows a more stable and accurate tracking performance than the current tracking methods in preclinical tests.

CONCLUSION

With satisfactory tracking results, vision guidance of a suitable insertion spot can be provided to the device to perform the surgery in an automatic way.

Bridging the gap in connectomic studies: A particle filtering framework for estimating structural connectivity at network scale

The ultimate goal of neuroscience is understanding the brain at a functional level. This requires the investigation of the structural connectivity at multiple scales: from the single-neuron micro-connectomics to the brain-region macro-connectomics. In this work, we address the study of connectomics at the intermediate mesoscale, introducing a probabilistic approach capable of reconstructing complex topologies of large neuronal networks. Suitable directional features are designed to model the local neuritic architecture and a feature-based particle filtering framework is proposed which allows the spatial tracking of neurites on microscopy images. The experimental results on cultures of increasing complexity, grown on High-Density Micro Electrode Arrays, show good stability and performance as compared to ground truth annotations drawn by domain experts. We also show how the method can be used to dissect the structural connectivity of inhibitory and excitatory subnetworks opening new perspectives towards the investigation of functional interactions among multiple cellular populations.

A multiple model probability hypothesis density tracker for time-lapse cell microscopy sequences

Quantitative analysis of the dynamics of tiny cellular and subcellular structures in time-lapse cell microscopy sequences requires the development of a reliable multi-target tracking method capable of tracking numerous similar targets in the presence of high levels of noise, high target density, maneuvering motion patterns and intricate interactions. The linear Gaussian jump Markov system probability hypothesis density (LGJMS-PHD) filter is a recent Bayesian tracking filter that is well-suited for this task. However, the existing recursion equations for this filter do not consider a state-dependent transition probability matrix. As required in many biological applications, we propose a new closed-form recursion that incorporates this assumption and introduce a general framework for particle tracking using the proposed filter. We apply our scheme to multi-target tracking in total internal reflection fluorescence microscopy (TIRFM) sequences and evaluate the performance of our filter against the existing LGJMS-PHD and IMM-JPDA filters.