Gigapixel imaging with a novel multi-camera array microscope

Recording dynamic facial micro-expressions with a multi-focus camera array

We present a multi-camera array for capturing dynamic high-resolution videos of the human face. Compared to traditional single-camera configurations, our array of 54 individual cameras allows stitching of high-resolution composite video frames (709 megapixels total). In our novel multi-focus strategy, each camera in the array focuses on a unique object plane to resolve non-planar surfaces at a higher resolution than a standard single-lens camera design. By overcoming the standard resolution and depth-of-field (DOF) tradeoffs, we use our array design to capture video of macroscopically curved surfaces such as the human face at a lateral resolution of 26.14 ± 5.8 µm across a composite DOF of ∼43 mm that covers the entire face (85 cm2+ FOV). Compared to a single-focus configuration, this is almost a 10-fold increase in effective DOF. We demonstrate how our multi-focus camera array can capture dynamic facial expressions at microscopic resolution with relevance in several biomedical applications.

Pan-cortical cellular imaging in freely behaving mice using a miniaturized micro-camera array microscope (mini-MCAM)

Understanding how circuits in the brain simultaneously coordinate their activity to mediate complex ethnologically relevant behaviors requires recording neural activities from distributed populations of neurons in freely behaving animals. Current miniaturized imaging microscopes are typically limited to imaging a relatively small field of view, precluding the measurement of neural activities across multiple brain regions. Here we present a miniaturized micro-camera array microscope (mini-MCAM) that consists of four fluorescence imaging micro-cameras, each capable of capturing neural activity across a 4.5 mm x 2.55 mm field of view (FOV). Cumulatively, the mini-MCAM images over 30 mm 2 area of sparsely expressed GCaMP6s neurons distributed throughout the dorsal cortex, in regions including the primary and secondary motor, somatosensory, visual, retrosplenial, and association cortices across both hemispheres. We demonstrate cortex-wide cellular resolution in vivo Calcium (Ca 2+ ) imaging using the mini-MCAM in both head-fixed and freely behaving mice.

Deep SBP+ 2.0: a physics-driven generation capability enhanced framework to reconstruct a space-bandwidth product expanded image from two image shots

The space-bandwidth product (SBP) limitation makes it difficult to obtain an image with both a high spatial resolution and a large field of view (FoV) through commonly used optical imaging systems. Although FoV and spectrum stitch provide solutions for SBP expansion, they rely on spatial and spectral scanning, which lead to massive image captures and a low processing speed. To solve the problem, we previously reported a physics-driven deep SBP-expanded framework (Deep SBP+) [J. Opt. Soc. Am. A40, 833 (2023)JOAOD60740-323210.1364/JOSAA.480920]. Deep SBP+ can reconstruct an image with both high spatial resolution and a large FoV from a low-spatial-resolution image in a large FoV and several high-spatial-resolution images in sub-FoVs. In physics, Deep SBP+ reconstructs the convolution kernel between the low- and high-spatial-resolution images and improves the spatial resolution through deconvolution. But Deep SBP+ needs multiple high-spatial-resolution images in different sub-FoVs, inevitably complicating the operations. To further reduce the image captures, we report an updated version of Deep SBP+ 2.0, which can reconstruct an SBP expanded image from a low-spatial-resolution image in a large FoV and another high-spatial-resolution image in a sub-FoV. Different from Deep SBP+, the assumption that the convolution kernel is a Gaussian distribution is added to Deep SBP+ 2.0 to make the kernel calculation simple and in line with physics. Moreover, improved deep neural networks have been developed to enhance the generation capability. Proven by simulations and experiments, the receptive field is analyzed to prove that a high-spatial-resolution image in the sub-FoV can also guide the generation of the entire FoV. Furthermore, we also discuss the requirement of the sub-FoV image to obtain an SBP-expanded image of high quality. Considering its SBP expansion capability and convenient operation, the updated Deep SBP+ 2.0 can be a useful tool to pursue images with both high spatial resolution and a large FoV.

Automated, high-throughput quantification of EGFP-expressing neutrophils in zebrafish by machine learning and a highly-parallelized microscope

Normal development of the immune system is essential for overall health and disease resistance. Bony fish, such as the zebrafish (Danio rerio), possess all the major immune cell lineages as mammals and can be employed to model human host response to immune challenge. Zebrafish neutrophils, for example, are present in the transparent larvae as early as 48 hours post fertilization and have been examined in numerous infection and immunotoxicology reports. One significant advantage of the zebrafish model is the ability to affordably generate high numbers of individual larvae that can be arrayed in multi-well plates for high throughput genetic and chemical exposure screens. However, traditional workflows for imaging individual larvae have been limited to low-throughput studies using traditional microscopes and manual analyses. Using a newly developed, parallelized microscope, the Multi-Camera Array Microscope (MCAM™), we have optimized a rapid, high-resolution algorithmic method to count fluorescently labeled cells in zebrafish larvae in vivo. Using transgenic zebrafish larvae, in which neutrophils express EGFP, we captured 18 gigapixels of images across a full 96-well plate, in 75 seconds, and processed the resulting datastream, counting individual fluorescent neutrophils in all individual larvae in 5 minutes. This automation is facilitated by a machine learning segmentation algorithm that defines the most in-focus view of each larva in each well after which pixel intensity thresholding and blob detection are employed to locate and count fluorescent cells. We validated this method by comparing algorithmic neutrophil counts to manual counts in larvae subjected to changes in neutrophil numbers, demonstrating the utility of this approach for high-throughput genetic and chemical screens where a change in neutrophil number is an endpoint metric. Using the MCAM™ we have been able to, within minutes, acquire both enough data to create an automated algorithm and execute a biological experiment with statistical significance. Finally, we present this open-source software package which allows the user to train and evaluate a custom machine learning segmentation model and use it to localize zebrafish and analyze cell counts within the segmented region of interest. This software can be modified as needed for studies involving other zebrafish cell lineages using different transgenic reporter lines and can also be adapted for studies using other amenable model species.

Automated, high-throughput quantification of EGFP-expressing neutrophils in zebrafish by machine learning and a highly-parallelized microscope

Normal development of the immune system is essential for overall health and disease resistance. Bony fish, such as the zebrafish (Danio rerio), possess all the major immune cell lineages as mammals and can be employed to model human host response to immune challenge. Zebrafish neutrophils, for example, are present in the transparent larvae as early as 48 hours post fertilization and have been examined in numerous infection and immunotoxicology reports. One significant advantage of the zebrafish model is the ability to affordably generate high numbers of individual larvae that can be arrayed in multi-well plates for high throughput genetic and chemical exposure screens. However, traditional workflows for imaging individual larvae have been limited to low-throughput studies using traditional microscopes and manual analyses. Using a newly developed, parallelized microscope, the Multi-Camera Array Microscope (MCAM™), we have optimized a rapid, high-resolution algorithmic method to count fluorescently labeled cells in zebrafish larvae in vivo. Using transgenic zebrafish larvae, in which neutrophils express EGFP, we captured 18 gigapixels of images across a full 96-well plate, in 75 seconds, and processed the resulting datastream, counting individual fluorescent neutrophils in all individual larvae in 5 minutes. This automation is facilitated by a machine learning segmentation algorithm that defines the most in-focus view of each larva in each well after which pixel intensity thresholding and blob detection are employed to locate and count fluorescent cells. We validated this method by comparing algorithmic neutrophil counts to manual counts in larvae subjected to changes in neutrophil numbers, demonstrating the utility of this approach for high-throughput genetic and chemical screens where a change in neutrophil number is an endpoint metric. Using the MCAM™ we have been able to, within minutes, acquire both enough data to create an automated algorithm and execute a biological experiment with statistical significance. Finally, we present this open-source software package which allows the user to train and evaluate a custom machine learning segmentation model and use it to localize zebrafish and analyze cell counts within the segmented region of interest. This software can be modified as needed for studies involving other zebrafish cell lineages using different transgenic reporter lines and can also be adapted for studies using other amenable model species.

SAMPL is a high-throughput solution to study unconstrained vertical behavior in small animals

Balance and movement are impaired in many neurological disorders. Recent advances in behavioral monitoring provide unprecedented access to posture and locomotor kinematics but without the throughput and scalability necessary to screen candidate genes/potential therapeutics. Here, we present a scalable apparatus to measure posture and locomotion (SAMPL). SAMPL includes extensible hardware and open-source software with real-time processing and can acquire data from D. melanogaster, C. elegans, and D. rerio as they move vertically. Using SAMPL, we define how zebrafish balance as they navigate vertically and discover small but systematic variations among kinematic parameters between genetic backgrounds. We demonstrate SAMPL's ability to resolve differences in posture and navigation as a function of effect size and data gathered, providing key data for screens. SAMPL is therefore both a tool to model balance and locomotor disorders and an exemplar of how to scale apparatus to support screens.

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving model organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising on speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over an area of 135 cm2, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The resulting reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms, including ants, fruit flies, and zebrafish larvae.

Scalable Apparatus to Measure Posture and Locomotion (SAMPL): a high-throughput solution to study unconstrained vertical behavior in small animals

Balance and movement are impaired in a wide variety of neurological disorders. Recent advances in behavioral monitoring provide unprecedented access to posture and locomotor kinematics, but without the throughput and scalability necessary to screen candidate genes / potential therapeutics. We present a powerful solution: a Scalable Apparatus to Measure Posture and Locomotion (SAMPL). SAMPL includes extensible imaging hardware and low-cost open-source acquisition software with real-time processing. We first demonstrate that SAMPL's hardware and acquisition software can acquire data from from D. melanogaster, C. elegans, and D. rerio as they move vertically. Next, we leverage SAMPL's throughput to rapidly (two weeks) gather a new zebrafish dataset. We use SAMPL's analysis and visualization tools to replicate and extend our current understanding of how zebrafish balance as they navigate through a vertical environment. Next, we discover (1) that key kinematic parameters vary systematically with genetic background, and (2) that such background variation is small relative to the changes that accompany early development. Finally, we simulate SAMPL's ability to resolve differences in posture or vertical navigation as a function of affect size and data gathered -- key data for screens. Taken together, our apparatus, data, and analysis provide a powerful solution for labs using small animals to investigate balance and locomotor disorders at scale. More broadly, SAMPL is both an adaptable resource for labs looking process videographic measures of behavior in real-time, and an exemplar of how to scale hardware to enable the throughput necessary for screening.