Real-time, deep-learning aided lensless microscope

Lensless imaging with a programmable Fresnel zone aperture

Optical imaging has long been dominated by traditional lens-based systems that, despite their success, are inherently limited by size, weight, and cost. Lensless imaging seeks to overcome these limitations by replacing lenses with thinner, lighter, and cheaper optical modulators and reconstructing images computationally, while facing trade-offs in image quality, artifacts, and flexibility inherent in traditional static modulation. Here, we propose a lensless imaging method with programmable Fresnel zone aperture (FZA), termed LIP. With a commercial liquid crystal display, we designed an integrated LIP module and demonstrated its capability of high-quality artifact-free reconstruction through dynamic modulation and offset-FZA parallel merging. Compared to static-modulation approaches, LIP achieves a 2.5× resolution enhancement and a 3 decibels improvement in signal-to-noise ratio in "static mode" while maintaining an interaction frame rate of 15 frames per second in "dynamic mode." Experimental results demonstrate LIP's potential as a miniaturized platform for versatile advanced imaging tasks like virtual reality and human-computer interaction.

Advances in Portable Optical Microscopy Using Cloud Technologies and Artificial Intelligence for Medical Applications

The need for faster and more accessible alternatives to laboratory microscopy is driving many innovations throughout the image and data acquisition chain in the biomedical field. Benchtop microscopes are bulky, lack communications capabilities, and require trained personnel for analysis. New technologies, such as compact 3D-printed devices integrated with the Internet of Things (IoT) for data sharing and cloud computing, as well as automated image processing using deep learning algorithms, can address these limitations and enhance the conventional imaging workflow. This review reports on recent advancements in microscope miniaturization, with a focus on emerging technologies such as photoacoustic microscopy and more established approaches like smartphone-based microscopy. The potential applications of IoT in microscopy are examined in detail. Furthermore, this review discusses the evolution of image processing in microscopy, transitioning from traditional to deep learning methods that facilitate image enhancement and data interpretation. Despite numerous advancements in the field, there is a noticeable lack of studies that holistically address the entire microscopy acquisition chain. This review aims to highlight the potential of IoT and artificial intelligence (AI) in combination with portable microscopy, emphasizing the importance of a comprehensive approach to the microscopy acquisition chain, from portability to image analysis.

Wide-field, high-resolution reconstruction in computational multi-aperture miniscope using a Fourier neural network

Traditional fluorescence microscopy is constrained by inherent trade-offs among resolution, field of view, and system complexity. To navigate these challenges, we introduce a simple and low-cost computational multi-aperture miniature microscope, utilizing a microlens array for single-shot wide-field, high-resolution imaging. Addressing the challenges posed by extensive view multiplexing and non-local, shift-variant aberrations in this device, we present SV-FourierNet, a multi-channel Fourier neural network. SV-FourierNet facilitates high-resolution image reconstruction across the entire imaging field through its learned global receptive field. We establish a close relationship between the physical spatially varying point-spread functions and the network's learned effective receptive field. This ensures that SV-FourierNet has effectively encapsulated the spatially varying aberrations in our system and learned a physically meaningful function for image reconstruction. Training of SV-FourierNet is conducted entirely on a physics-based simulator. We showcase wide-field, high-resolution video reconstructions on colonies of freely moving C. elegans and imaging of a mouse brain section. Our computational multi-aperture miniature microscope, augmented with SV-FourierNet, represents a major advancement in computational microscopy and may find broad applications in biomedical research and other fields requiring compact microscopy solutions.

Mesoscopic calcium imaging in a head-unrestrained male non-human primate using a lensless microscope

Mesoscopic calcium imaging enables studies of cell-type specific neural activity over large areas. A growing body of literature suggests that neural activity can be different when animals are free to move compared to when they are restrained. Unfortunately, existing systems for imaging calcium dynamics over large areas in non-human primates (NHPs) are table-top devices that require restraint of the animal's head. Here, we demonstrate an imaging device capable of imaging mesoscale calcium activity in a head-unrestrained male non-human primate. We successfully miniaturize our system by replacing lenses with an optical mask and computational algorithms. The resulting lensless microscope can fit comfortably on an NHP, allowing its head to move freely while imaging. We are able to measure orientation columns maps over a 20 mm2 field-of-view in a head-unrestrained macaque. Our work establishes mesoscopic imaging using a lensless microscope as a powerful approach for studying neural activity under more naturalistic conditions.