Software-Based Phase Control, Video-Rate Imaging, and Real-Time Mosaicing With a Lissajous-Scanned Confocal Microscope

CRASH2p: Closed-loop Two Photon Imaging in Freely Moving Animals

Direct measurement of neural activity in freely moving animals is essential for understanding how the brain controls and represents behaviors. Genetically encoded calcium indicators report neural activity as changes in fluorescence intensity, but brain motion confounds quantitative measurement of fluorescence. Translation, rotation, and deformation of the brain and the movements of intervening scattering or auto-fluorescent tissue all alter the amount of fluorescent light captured by a microscope. Compared to single-photon approaches, two photon microscopy is less sensitive to scattering and off-target fluorescence, but more sensitive to motion, and two photon imaging has always required anchoring the microscope to the brain. We developed a closed-loop resonant axial-scanning high-speed two photon (CRASH2p) microscope for real-time 3D motion correction in unrestrained animals, without implantation of reference markers. We complemented CRASH2p with a novel scanning strategy and a multistage registration pipeline. We performed volumetric ratiometrically corrected functional imaging in the CNS of freely moving Drosophila larvae and discovered previously unknown neural correlates of behavior.

Development of an Optoelectronic Integrated Sensor for a MEMS Mirror-Based Active Structured Light System

Micro-electro-mechanical system (MEMS) scanning micromirrors are playing an increasingly important role in active structured light systems. However, the initial phase error of the structured light generated by a scanning micromirror seriously affects the accuracy of the corresponding system. This paper reports an optoelectronic integrated sensor with high irradiance responsivity and high linearity that can be used to correct the phase error of the micromirror. The optoelectronic integrated sensor consists of a large-area photodetector (PD) and a receiving circuit, including a post amplifier, an operational amplifier, a bandgap reference, and a reference current circuit. The optoelectronic sensor chip is fabricated in a 180 nm CMOS process. Experimental results show that with a 5 V power supply, the optoelectronic sensor has an irradiance responsivity of 100 mV/(μW/cm²) and a -3 dB bandwidth of 2 kHz. The minimal detectable light power is about 19.4 nW, which satisfies the requirements of many active structured light systems. Through testing, the application of the chip effectively reduces the phase error of the micromirror to 2.5%.

Robust endoscopic image mosaicking via fusion of multimodal estimation

We propose an endoscopic image mosaicking algorithm that is robust to light conditioning changes, specular reflections, and feature-less scenes. These conditions are especially common in minimally invasive surgery where the light source moves with the camera to dynamically illuminate close range scenes. This makes it difficult for a single image registration method to robustly track camera motion and then generate consistent mosaics of the expanded surgical scene across different and heterogeneous environments. Instead of relying on one specialised feature extractor or image registration method, we propose to fuse different image registration algorithms according to their uncertainties, formulating the problem as affine pose graph optimisation. This allows to combine landmarks, dense intensity registration, and learning-based approaches in a single framework. To demonstrate our application we consider deep learning-based optical flow, hand-crafted features, and intensity-based registration, however, the framework is general and could take as input other sources of motion estimation, including other sensor modalities. We validate the performance of our approach on three datasets with very different characteristics to highlighting its generalisability, demonstrating the advantages of our proposed fusion framework. While each individual registration algorithm eventually fails drastically on certain surgical scenes, the fusion approach flexibly determines which algorithms to use and in which proportion to more robustly obtain consistent mosaics.

Capacitive Sensing for 2-D Electrostatic MEMS Scanner in a Clinical Endomicroscope

A flexible fiber-coupled confocal laser endomicroscope has been developed using an electrostatic micro-electromechanical system (MEMS) scanner located in at distal optics to collect in vivo images in human subjects. Long transmission lines are required that deliver drive and sense signals with limited bandwidth. Phase shifts have been observed between orthogonal X and Y scanner axes from environmental perturbations, which impede image reconstruction. Image processing algorithms used for correction depend on image content and quality, while scanner calibration in the clinic can be limited by potential patient exposure to lasers. We demonstrate a capacitive sensing method to track the motion of the electrostatically driven two-dimensional MEMS scanner and to extract phase information needed for image reconstruction. This circuit uses an amplitude modulation envelope detection method on shared drive and sensing electrodes of the scanner. Circuit parameters were optimized for performance given high scan frequencies, transmission line effects, and substantial parasitic coupling of drive signal to circuit output. Extraction of phase information further leverages nonlinear dynamics of the MEMS scanner. The sensing circuit was verified by comparing with data from a position sensing detector measurement. The phase estimation showed an accuracy of 2.18° and 0.79° in X and Y axes for motion sensing, respectively. The results indicate that the sensing circuit can be implemented with feedback control for pre-calibration of the scanner in clinical MEMS-based imaging systems.

Improved sampling scheme for LiDAR in Lissajous scanning mode

MEMS light detection and ranging (LiDAR) is becoming an indispensable sensor in vehicle environment sensing systems due to its low cost and high performance. The beam scanning trajectory, sampling scheme and gridding are the key technologies of MEMS LiDAR imaging. In Lissajous scanning mode, this paper improves the sampling scheme, through which a denser Cartesian grid of point cloud data at the same scanning frequency can be obtained. By summarizing the rules of the Cartesian grid, a general sampling scheme independent of the beam scanning trajectory patterns is proposed. Simulation and experiment results show that compared with the existing sampling scheme, the resolution and the number of points per frame are both increased by 2 times with the same hardware configuration and scanning frequencies for a MEMS scanning mirror (MEMS-SM). This is beneficial for improving the point cloud imaging performance of MEMS LiDAR.

Resonant scanning design and control for fast spatial sampling

Two-dimensional, resonant scanners have been utilized in a large variety of imaging modules due to their compact form, low power consumption, large angular range, and high speed. However, resonant scanners have problems with non-optimal and inflexible scanning patterns and inherent phase uncertainty, which limit practical applications. Here we propose methods for optimized design and control of the scanning trajectory of two-dimensional resonant scanners under various physical constraints, including high frame-rate and limited actuation amplitude. First, we propose an analytical design rule for uniform spatial sampling. We demonstrate theoretically and experimentally that by expanding the design space, the proposed designs outperform previous designs in terms of scanning range and fill factor. Second, we show that we can create flexible scanning patterns that allow focusing on user-defined Regions-of-Interest (RoI) by modulation of the scanning parameters. The scanning parameters are found by an optimization algorithm. In simulations, we demonstrate the benefits of these designs with standard metrics and higher-level computer vision tasks (LiDAR odometry and 3D object detection). Finally, we experimentally implement and verify both unmodulated and modulated scanning modes using a two-dimensional, resonant MEMS scanner. Central to the implementations is high bandwidth monitoring of the phase of the angular scans in both dimensions. This task is carried out with a position-sensitive photodetector combined with high-bandwidth electronics, enabling fast spatial sampling at [Formula: see text] Hz frame-rate.

Image processing metrics for phase identification of a multiaxis MEMS scanner used in single pixel imaging

This paper applies image processing metrics to tracking of perturbations in mechanical phase delay in a multi-axis microelectromechanical system (MEMS) scanner. The compact mirror is designed to scan a laser beam in a Lissajous pattern during the collection of endoscopic confocal fluorescence images, but environmental perturbations to the mirror dynamics can lead to image registration errors and blurry images. A binarized, threshold-based blur metric and variance-based sharpness metric are introduced for detecting scanner phase delay. Accuracy of local optima of the metric for identification of phase delay is examined, and relative advantages for processing accuracy and computational complexity are assessed. Image reconstruction is demonstrated using both generic images and sample tissue images, with significant improvement in image quality for tissue imaging. Implications of non-ideal Lissajous scan effects on phase detection and image reconstruction are discussed.

Real-time video mosaicking to guide handheld in vivo microscopy

Handheld and endoscopic optical-sectioning microscopes are being developed for noninvasive screening and intraoperative consultation. Imaging a large extent of tissue is often desired, but miniature in vivo microscopes tend to suffer from limited fields of view. To extend the imaging field during clinical use, we have developed a real-time video mosaicking method, which allows users to efficiently survey larger areas of tissue. Here, we modified a previous post-processing mosaicking method so that real-time mosaicking is possible at >30 frames/second when using a device that outputs images that are 400 × 400 pixels in size. Unlike other real-time mosaicking methods, our strategy can accommodate image rotations and deformations that often occur during clinical use of a handheld microscope. We perform a feasibility study to demonstrate that the use of real-time mosaicking is necessary to enable efficient sampling of a desired imaging field when using a handheld dual-axis confocal microscope.