An integrated imaging sensor for aberration-corrected 3D photography

Long-term intravital subcellular imaging with confocal scanning light-field microscopy

Long-term observation of subcellular dynamics in living organisms is limited by background fluorescence originating from tissue scattering or dense labeling. Existing confocal approaches face an inevitable tradeoff among parallelization, resolution and phototoxicity. Here we present confocal scanning light-field microscopy (csLFM), which integrates axially elongated line-confocal illumination with the rolling shutter in scanning light-field microscopy (sLFM). csLFM enables high-fidelity, high-speed, three-dimensional (3D) imaging at near-diffraction-limit resolution with both optical sectioning and low phototoxicity. By simultaneous 3D excitation and detection, the excitation intensity can be reduced below 1 mW mm-2, with 15-fold higher signal-to-background ratio over sLFM. We imaged subcellular dynamics over 25,000 timeframes in optically challenging environments in different species, such as migrasome delivery in mouse spleen, retractosome generation in mouse liver and 3D voltage imaging in Drosophila. Moreover, csLFM facilitates high-fidelity, large-scale neural recording with reduced crosstalk, leading to high orientation selectivity to visual stimuli, similar to two-photon microscopy, which aids understanding of neural coding mechanisms.

Baseline-free structured light 3D imaging using a metasurface double-helix dot projector

Structured light is a widely used 3D imaging method with a drawback that it typically requires a long baseline length between the laser projector and the camera sensor, which hinders its utilization in space-constrained scenarios. On the other hand, the application of passive 3D imaging methods, such as depth from depth-dependent point spread functions (PSFs), is impeded by the challenge in measuring textureless scenes. Here, we combine the advantages of both structured light and depth-dependent PSFs and propose a baseline-free structured light 3D imaging system. A metasurface is designed to project a structured dot array and encode depth information in the double-helix pattern of each dot simultaneously. Combined with a straightforward and fast algorithm, we demonstrate accurate 3D point cloud acquisition for various real-world scenarios including multiple cardboard boxes and a living human face. Such a technique may find application in a broad range of areas including consumer electronics and precision metrology.

Dual Radar: A Multi-modal Dataset with Dual 4D Radar for Autononous Driving

4D radar has higher point cloud density and precise vertical resolution than conventional 3D radar, making it promising for adverse scenarios in the environmental perception of autonomous driving. However, 4D radar is more noisy than LiDAR and requires different filtering strategies that affect the point cloud density and noise level. Comparative analyses of different point cloud densities and noise levels are still lacking, mainly because the available datasets use only one type of 4D radar, making it difficult to compare different 4D radars in the same scenario. We introduce a novel large-scale multi-modal dataset that captures both types of 4D radar, consisting of 151 sequences, most of which are 20 seconds long and contain 10,007 synchronized and annotated frames. Our dataset captures a variety of challenging driving scenarios, including multiple road conditions, weather conditions, different lighting intensities and periods. It supports 3D object detection and tracking as well as multi-modal tasks. We experimentally validate the dataset, providing valuable insights for studying different types of 4D radar.

Blind aberration correction for light field photography

Aberration correction is critical for obtaining sharp images but remains a challenging task. Owing to its ability to record both spatial and angular information of light rays, light field imaging is a powerful method to measure and correct optical aberrations. However, current methods need extensive calibrations to obtain prior information about the camera, which is restrictive in real-world applications. In this work, we propose a two-stage blind aberration correction method for light field imaging, which leverages self-supervised learning for general blind aberration correction and low-rank approximation to exploit the specific correlations of light fields to further abate aberrations. We demonstrated experimentally the superiority of our method over current state-of-the-art.

Monocular meta-imaging camera sees depth

A novel monocular depth-sensing camera based on meta-imaging sensor technology has been developed, offering more precise depth sensing with millimeter-level accuracy and enhanced robustness compared to conventional 2D and light-field cameras.

Metasurface-Coated Liquid Microlens for Super Resolution Imaging

Inspired by metasurfaces' control over light fields, this study created a liquid microlens coated with a layer of Au@TiO2, Core-Shell nanospheres. Utilizing the surface plasmon resonance (SPR) effect of Au@TiO2, Core-Shell nanospheres, and the formation of photonic nanojets (PNJs), this study aimed to extend the imaging system's cutoff frequency, improve microlens focusing, enhance the capture capability of evanescent waves, and utilize nanospheres to improve the conversion of evanescent waves into propagating waves, thus boosting the liquid microlens's super-resolution capabilities. The finite difference time domain (FDTD) method analyzed the impact of parameters including nanosphere size, microlens sample contact width, and droplet's initial contact angle on super-resolution imaging. The results indicate that the full width at half maximum (FWHM) of the field distribution produced by the uncoated microlens is 1.083 times that of the field distribution produced by the Au@TiO2, Core-Shell nanospheres coated microlens. As the nanosphere radius, droplet contact angle, and droplet base diameter increased, the microlens's light intensity correspondingly increased. These findings confirm that metasurface coating enhances the super-resolution capabilities of the microlens.

Ultra-fast light-field microscopy with event detection

The event detection technique has been introduced to light-field microscopy, boosting its imaging speed in orders of magnitude with simultaneous axial resolution enhancement in scattering medium.

A broadband hyperspectral image sensor with high spatio-temporal resolution

Hyperspectral imaging provides high-dimensional spatial-temporal-spectral information showing intrinsic matter characteristics1-5. Here we report an on-chip computational hyperspectral imaging framework with high spatial and temporal resolution. By integrating different broadband modulation materials on the image sensor chip, the target spectral information is non-uniformly and intrinsically coupled to each pixel with high light throughput. Using intelligent reconstruction algorithms, multi-channel images can be recovered from each frame, realizing real-time hyperspectral imaging. Following this framework, we fabricated a broadband visible-near-infrared (400-1,700 nm) hyperspectral image sensor using photolithography, with an average light throughput of 74.8% and 96 wavelength channels. The demonstrated resolution is 1,024 × 1,024 pixels at 124 fps. We demonstrated its wide applications, including chlorophyll and sugar quantification for intelligent agriculture, blood oxygen and water quality monitoring for human health, textile classification and apple bruise detection for industrial automation, and remote lunar detection for astronomy. The integrated hyperspectral image sensor weighs only tens of grams and can be assembled on various resource-limited platforms or equipped with off-the-shelf optical systems. The technique transforms the challenge of high-dimensional imaging from a high-cost manufacturing and cumbersome system to one that is solvable through on-chip compression and agile computation.

Long-term mesoscale imaging of 3D intercellular dynamics across a mammalian organ

A comprehensive understanding of physio-pathological processes necessitates non-invasive intravital three-dimensional (3D) imaging over varying spatial and temporal scales. However, huge data throughput, optical heterogeneity, surface irregularity, and phototoxicity pose great challenges, leading to an inevitable trade-off between volume size, resolution, speed, sample health, and system complexity. Here, we introduce a compact real-time, ultra-large-scale, high-resolution 3D mesoscope (RUSH3D), achieving uniform resolutions of 2.6 × 2.6 × 6 μm3 across a volume of 8,000 × 6,000 × 400 μm3 at 20 Hz with low phototoxicity. Through the integration of multiple computational imaging techniques, RUSH3D facilitates a 13-fold improvement in data throughput and an orders-of-magnitude reduction in system size and cost. With these advantages, we observed premovement neural activity and cross-day visual representational drift across the mouse cortex, the formation and progression of multiple germinal centers in mouse inguinal lymph nodes, and heterogeneous immune responses following traumatic brain injury-all at single-cell resolution, opening up a horizon for intravital mesoscale study of large-scale intercellular interactions at the organ level.

Artificial Intelligence-Enhanced Metasurfaces for Instantaneous Measurements of Dispersive Refractive Index

Measurements of the refractive index of liquids are in high demand in numerous fields such as agriculture, food and beverages, and medicine. However, conventional ellipsometric refractive index measurements are too expensive and labor-intensive for consumer devices, while Abbe refractometry is limited to the measurement at a single wavelength. Here, a new approach is proposed using machine learning to unlock the potential of colorimetric metasurfaces for the real-time measurement of the dispersive refractive index of liquids over the entire visible spectrum. The platform with a proof-of-concept experiment for measuring the concentration of glucose is further demonstrated, which holds a profound impact in non-invasive medical sensing. High-index-dielectric metasurfaces are designed and fabricated, while their experimentally measured reflectance and reflected colors, through microscopy and a standard smartphone, are used to train deep-learning models to provide measurements of the dispersive background refractive index with a resolution of ≈10-4, which is comparable to the known index as measured with ellipsometry. These results show the potential of enabling the unique optical properties of metasurfaces with machine learning to create a platform for the quick, simple, and high-resolution measurement of the dispersive refractive index of liquids, without the need for highly specialized experts and optical procedures.

Stereo imaging inspired by bionic optics

Stereo imaging has been a focal point in fields such as robotics and autonomous driving. This Letter discusses the imaging mechanisms of jumping spiders and human eyes from a biomimetic perspective and proposes a monocular stereo imaging solution with low computational cost and high stability. The stereo imaging mechanism of jumping spiders enables monocular imaging without relying on multiple viewpoints, thus avoiding complex large-scale feature point matching and significantly conserving computational resources. The foveal imaging mechanism of the human eye allows for complex imaging tasks to be completed only on the locally interested regions, resulting in more efficient execution of various visual tasks. By combining these two advantages, we have developed a more computationally efficient monocular stereo imaging method that can achieve stereo imaging on only the locally interested regions without sacrificing the performance of wide field-of-view (FOV) imaging. Finally, through experimental validation, we demonstrate that the method proposed in this Letter exhibits excellent stereo imaging performance.

Aberration-robust monocular passive depth sensing using a meta-imaging camera

Depth sensing plays a crucial role in various applications, including robotics, augmented reality, and autonomous driving. Monocular passive depth sensing techniques have come into their own for the cost-effectiveness and compact design, offering an alternative to the expensive and bulky active depth sensors and stereo vision systems. While the light-field camera can address the defocus ambiguity inherent in 2D cameras and achieve unambiguous depth perception, it compromises the spatial resolution and usually struggles with the effect of optical aberration. In contrast, our previously proposed meta-imaging sensor1 has overcome such hurdles by reconciling the spatial-angular resolution trade-off and achieving the multi-site aberration correction for high-resolution imaging. Here, we present a compact meta-imaging camera and an analytical framework for the quantification of monocular depth sensing precision by calculating the Cramér-Rao lower bound of depth estimation. Quantitative evaluations reveal that the meta-imaging camera exhibits not only higher precision over a broader depth range than the light-field camera but also superior robustness against changes in signal-background ratio. Moreover, both the simulation and experimental results demonstrate that the meta-imaging camera maintains the capability of providing precise depth information even in the presence of aberrations. Showing the promising compatibility with other point-spread-function engineering methods, we anticipate that the meta-imaging camera may facilitate the advancement of monocular passive depth sensing in various applications.

Ultrahigh precision laser nanoprinting based on defect-compensated digital holography for fast-fabricating optical metalenses

The 3D structured light field manipulated by a digital-micromirror-device (DMD)-based digital hologram has demonstrated its superiority in fast-fabricating stereo nanostructures. However, this technique intrinsically suffers from defects of light intensity in generating modulated focal spots, which prevents from achieving high-precision micro/nanodevices. In this Letter, we have demonstrated a compensation approach based on adapting spatial voxel density for fabricating optical metalenses with ultrahigh precision. The modulated focal spot experiences intensity fluctuations of up to 3% by changing the spatial position, leading to a 20% variation of the structural dimension in fabrication. By altering the voxel density to improve the uniformity of the laser cumulative exposure dosage over the fabrication region, we achieved an increased dimensional uniformity from 94.4% to 97.6% in fabricated pillars. This approach enables fast fabrication of metalenses capable of sub-diffraction focusing of 0.44λ/NA with the increased mainlobe-sidelobe ratio from 1:0.34 to 1:0.14. A 6 × 5 supercritical lens array is fabricated within 2 min, paving a way for the fast fabrication of large-scale photonic devices.

Thermal active optical technology to achieve in-orbit wavefront aberration correction for optical remote sensing satellites

Image quality and resolution are important factors affecting the application value of remote sensing images. Although increasing the optical aperture of space optical remote sensors (SORSs) improves image resolution, it exacerbates the effects of the space environment on imaging quality. Thus, this study proposes thermal active optical technology (TAO) to enhance image quality while increasing the optical aperture of SORSs by actively correcting in-orbit wavefront aberrations. Replacing traditional wavefront detection and reconstruction with numerical calculation and simulation analysis, more realistic in-orbit SORS wavefront aberrations are obtained. Numerical and finite element analyses demonstrate that nonlinearities in TAO control lead to the failure of traditional wavefront correction algorithms. To address this, we use a neural network algorithm combining CNN and ResNet. Simulation results show that the residual of the systematic wavefront RMS error for SORS reduces to 1/100λ. The static and dynamic modular transfer functions are improved, and the structural similarity index is recovered by over 23%, highlighting the effectiveness of TAO in image quality enhancement. The static and thermal vacuum experiments demonstrate the wide applicability and engineering prospects of TAO.

Nonlocal, Flat-Band Meta-Optics for Monolithic, High-Efficiency, Compact Photodetectors

Miniaturized photodetectors are becoming increasingly sought-after components for next-generation technologies, such as autonomous vehicles, integrated wearable devices, or gadgets embedded on the Internet of Things. A major challenge, however, lies in shrinking the device footprint while maintaining high efficiency. This conundrum can be solved by realizing a nontrivial relation between the energy and momentum of photons, such as dispersion-free devices, known as flat bands. Here, we leverage flat-band meta-optics to simultaneously achieve critical absorption over a wide range of incidence angles. For a monolithic silicon meta-optical photodiode, we achieved an ∼10-fold enhancement in the photon-to-electron conversion efficiency. Such enhancement over a large angular range of ∼36° allows incoming light to be collected via a large-aperture lens and focused on a compact photodiode, potentially enabling high-speed and low-light operation. Our research unveils new possibilities for creating compact and efficient optoelectronic devices with far-reaching impact on various applications, including augmented reality and light detection and ranging.

Single femtosecond pulse writing of a bifocal lens

In this Letter, a method for the fabrication of bifocal lenses is presented by combining surface ablation and bulk modification in a single laser exposure followed by the wet etching processing step. The intensity of a single femtosecond laser pulse was modulated axially into two foci with a designed computer-generated hologram (CGH). Such pulse simultaneously induced an ablation region on the surface and a modified volume inside the fused silica. After etching in hydrofluoric acid (HF), the two exposed regions evolved into a bifocal lens. The area ratio (diameter) of the two lenses can be flexibly adjusted via control of the pulse energy distribution through the CGH. Besides, bifocal lenses with a center offset as well as convex lenses were obtained by a replication technique. This method simplifies the fabrication of micro-optical elements and opens a highly efficient and simple pathway for complex optical surfaces and integrated imaging systems.

Light field camera calibration and point spread function calculation based on differentiable ray tracing

The imaging process of the light field (LF) camera with a micro-lens array (MLA) may suffer from multiple aberrations. It is thus difficult to precisely calibrate the intrinsic hardware parameters and calculate the corresponding point spread function (PSF). To build an aberration-aware solution with better generalization, we propose an end-to-end imaging model based on the differentiable ray tracing. The input end is the point source location, and the output end is the rendered LF image, namely, PSF. Specially, a projection method is incorporated into the imaging model, eliminating the huge memory overhead induced by a large array of periodic elements. Taking captured PSF images as the ground truth, the LF camera is calibrated with the genetic algorithm initially and then the gradient-based optimization. This method is promising to be used in various LF camera applications, especially in challenging imaging conditions with severe aberrations.

Simultaneous detection of polarization states and wavefront by an angular variant micro-retarder-lens array

We have demonstrated simultaneous detection of the polarization states and wavefront of light using a 7 × 7 array of angular variant micro-retarder-lenses. Manipulating the angular variant polarization with our optical element allows us to determine the two-dimensional distribution of polarization states. We have also proposed a calibration method for polarization measurements using our micro-retarder-lens array, allowing accurate detection of polarization states with an ellipticity of ± 0.01 and an azimuth of ± 1.0°. We made wavefront measurements using the micro-retarder-lens array, achieving a resolution of 25 nm. We conducted simultaneous detection of the polarization states and wavefront on four types of structured beam as samples. The results show that the two-dimensional distributions of the polarization states and wavefront for the four types of structured light are radially and azimuthally polarized beams, as well as left- and right-hand optical vortices. Our sensing technology has the potential to enhance our understanding of the nature of light in the fields of laser sciences, astrophysics, and even ophthalmology.

Temporal structured illumination and vision-transformer enables large field-of-view binary snapshot ptychography

Ptychography, a widely used computational imaging method, generates images by processing coherent interference patterns scattered from an object of interest. In order to capture scenes with large field-of-view (FoV) and high spatial resolution simultaneously in a single shot, we propose a temporal-compressive structured-light Ptychography system. A novel three-step reconstruction algorithm composed of multi-frame spectra reconstruction, phase retrieval, and multi-frame image stitching is developed, where we employ the emerging Transformer-based network in the first step. Experimental results demonstrate that our system can expand the FoV by 20× without losing spatial resolution. Our results offer huge potential for enabling lensless imaging of molecules with large FoV as well as high spatial-temporal resolutions. We also notice that due to the loss of low-intensity information caused by the compressed sensing process, our method so far is only applicable to binary targets.

Quantitative real-time phase microscopy for extended depth-of-field imaging based on the 3D single-shot differential phase contrast (ssDPC) imaging method

Optical diffraction tomography (ODT) is a promising label-free imaging method capable of quantitatively measuring the three-dimensional (3D) refractive index distribution of transparent samples. In recent years, partially coherent ODT (PC-ODT) has attracted increasing attention due to its system simplicity and absence of laser speckle noise. Quantitative phase imaging (QPI) technologies represented by Fourier ptychographic microscopy (FPM), differential phase contrast (DPC) imaging and intensity diffraction tomography (IDT) need to collect several or hundreds of intensity images, which usually introduce motion artifacts when shooting fast-moving targets, leading to a decrease in image quality. Hence, a quantitative real-time phase microscopy (qRPM) for extended depth of field (DOF) imaging based on 3D single-shot differential phase contrast (ssDPC) imaging method is proposed in this research study. qRPM incorporates a microlens array (MLA) to simultaneously collect spatial information and angular information. In subsequent optical information processing, a deconvolution method is used to obtain intensity stacks under different illumination angles in a raw light field image. Importing the obtained intensity stack into the 3D DPC imaging model is able to finally obtain the 3D refractive index distribution. The captured four-dimensional light field information enables the reconstruction of 3D information in a single snapshot and extending the DOF of qRPM. The imaging capability of the proposed qRPM system is experimental verified on different samples, achieve single-exposure 3D label-free imaging with an extended DOF for 160 µm which is nearly 30 times higher than the traditional microscope system.

Single-shot deterministic complex amplitude imaging with a single-layer metalens

Conventional imaging systems can only capture light intensity. Meanwhile, the lost phase information may be critical for a variety of applications such as label-free microscopy and optical metrology. Existing phase retrieval techniques typically require a bulky setup, multiframe measurements, or prior information of the target scene. Here, we proposed an extremely compact system for complex amplitude imaging, leveraging the extreme versatility of a single-layer metalens to generate spatially multiplexed and polarization phase-shifted point spread functions. Combining the metalens with a polarization camera, the system can simultaneously record four polarization shearing interference patterns along both in-plane directions, thus allowing the deterministic reconstruction of the complex amplitude light field in a single shot. Using an incoherent light-emitting diode as the illumination, we experimentally demonstrated speckle-noise-free complex amplitude imaging for both static and moving objects with tailored magnification ratio and field of view. The miniaturized and robust system may open the door for complex amplitude imaging in portable devices for point-of-care applications.

Highly flexible and compact volumetric endoscope by integrating multiple micro-imaging devices

A light-field endoscope can simultaneously capture the three-dimensional information of in situ lesions and enables single-shot quantitative depth perception with minimal invasion for improving surgical and diagnostic accuracy. However, due to oversized rigid probes, clinical applications of current techniques are limited by their cumbersome devices. To minimize the size and enhance the flexibility, here we report a highly flexible and compact volumetric endoscope by employing precision-machined multiple micro-imaging devices (MIRDs). To further protect the flexibility, the designed MIRD with a diameter and height of 5 mm is packaged in pliable polyamide, using soft data cables for data transmission. It achieves the optimal lateral resolvability of 31 µm and axial resolvability of 255 µm, with an imaging volume over 2.3 × 2.3 × 10 mm3. Our technique allows easy access to the organism interior through the natural entrance, which has been verified through observational experiments of the stomach and rectum of a rabbit. Together, we expect this device can assist in the removal of tumors and polyps as well as the identification of certain early cancers of the digestive tract.

Compressive video via IR-pulsed illumination

We propose and demonstrate a compressive temporal imaging system based on pulsed illumination to encode temporal dynamics into the signal received by the imaging sensor during exposure time. Our approach enables >10x increase in effective frame rate without increasing camera complexity. To mitigate the complexity of the inverse problem during reconstruction, we introduce two keyframes: one before and one after the coded frame. We also craft what we believe to be a novel deep learning architecture for improved reconstruction of the high-speed scenes, combining specialized convolutional and transformer architectures. Simulation and experimental results clearly demonstrate the reconstruction of high-quality, high-speed videos from the compressed data.

All-analog photoelectronic chip for high-speed vision tasks

Photonic computing enables faster and more energy-efficient processing of vision data1-5. However, experimental superiority of deployable systems remains a challenge because of complicated optical nonlinearities, considerable power consumption of analog-to-digital converters (ADCs) for downstream digital processing and vulnerability to noises and system errors1,6-8. Here we propose an all-analog chip combining electronic and light computing (ACCEL). It has a systemic energy efficiency of 74.8 peta-operations per second per watt and a computing speed of 4.6 peta-operations per second (more than 99% implemented by optics), corresponding to more than three and one order of magnitude higher than state-of-the-art computing processors, respectively. After applying diffractive optical computing as an optical encoder for feature extraction, the light-induced photocurrents are directly used for further calculation in an integrated analog computing chip without the requirement of analog-to-digital converters, leading to a low computing latency of 72 ns for each frame. With joint optimizations of optoelectronic computing and adaptive training, ACCEL achieves competitive classification accuracies of 85.5%, 82.0% and 92.6%, respectively, for Fashion-MNIST, 3-class ImageNet classification and time-lapse video recognition task experimentally, while showing superior system robustness in low-light conditions (0.14 fJ μm-2 each frame). ACCEL can be used across a broad range of applications such as wearable devices, autonomous driving and industrial inspections.

Functionalizing nanophotonic structures with 2D van der Waals materials

The integration of two-dimensional (2D) van der Waals materials with nanostructures has triggered a wide spectrum of optical and optoelectronic applications. Photonic structures of conventional materials typically lack efficient reconfigurability or multifunctionality. Atomically thin 2D materials can thus generate new functionality and reconfigurability for a well-established library of photonic structures such as integrated waveguides, optical fibers, photonic crystals, and metasurfaces, to name a few. Meanwhile, the interaction between light and van der Waals materials can be drastically enhanced as well by leveraging micro-cavities or resonators with high optical confinement. The unique van der Waals surfaces of the 2D materials enable handiness in transfer and mixing with various prefabricated photonic templates with high degrees of freedom, functionalizing as the optical gain, modulation, sensing, or plasmonic media for diverse applications. Here, we review recent advances in synergizing 2D materials to nanophotonic structures for prototyping novel functionality or performance enhancements. Challenges in scalable 2D materials preparations and transfer, as well as emerging opportunities in integrating van der Waals building blocks beyond 2D materials are also discussed.

Neuron populations across layer 2-6 in the mouse visual cortex exhibit different coding abilities in the awake mice

Introduction

The visual cortex is a key region in the mouse brain, responsible for processing visual information. Comprised of six distinct layers, each with unique neuronal types and connections, the visual cortex exhibits diverse decoding properties across its layers. This study aimed to investigate the relationship between visual stimulus decoding properties and the cortical layers of the visual cortex while considering how this relationship varies across different decoders and brain regions.

Methods

This study reached the above conclusions by analyzing two publicly available datasets obtained through two-photon microscopy of visual cortex neuronal responses. Various types of decoders were tested for visual cortex decoding.

Results

Our findings indicate that the decoding accuracy of neuronal populations with consistent sizes varies among visual cortical layers for visual stimuli such as drift gratings and natural images. In particular, layer 4 neurons in VISp exhibited significantly higher decoding accuracy for visual stimulus identity compared to other layers. However, in VISm, the decoding accuracy of neuronal populations with the same size in layer 2/3 was higher than that in layer 4, despite the overall accuracy being lower than that in VISp and VISl. Furthermore, SVM surpassed other decoders in terms of accuracy, with the variation in decoding performance across layers being consistent among decoders. Additionally, we found that the difference in decoding accuracy across different imaging depths was not associated with the mean orientation selectivity index (OSI) and the mean direction selectivity index (DSI) neurons, but showed a significant positive correlation with the mean reliability and mean signal-to-noise ratio (SNR) of each layer's neuron population.

Discussion

These findings lend new insights into the decoding properties of the visual cortex, highlighting the role of different cortical layers and decoders in determining decoding accuracy. The correlations identified between decoding accuracy and factors such as reliability and SNR pave the way for more nuanced understandings of visual cortex functioning.

Alkaloid Precipitant Reaction Inspired Controllable Synthesis of Mesoporous Tungsten Oxide Spheres for Biomarker Sensing

Highly porous sensitive materials with well-defined structures and morphologies are extremely desirable for developing high-performance chemiresistive gas sensors. Herein, inspired by the classical alkaloid precipitant reaction, a robust and reliable active mesoporous nitrogen polymer sphere-directed synthesis method was demonstrated for the controllable construction of heteroatom-doped mesoporous tungsten oxide spheres. In the typical synthesis, P-doped mesoporous WO3 monodisperse spheres with radially oriented channels (P-mWO3-R) were obtained with a diameter of ∼180 nm, high specific surface area, and crystalline skeleton. The in situ-introduced P atoms could effectively adjust the coordination environment of W atoms (Wδ+-Ov), giving rise to dramatically enhanced active surface-adsorbed oxygen species and unusual metastable ε-WO3 crystallites. The P-mWO3-R spheres were applied for the sensing of 3-hydroxy-2-butanone (3H2B), a biomarker of foodborne pathogenic bacteria Listeria monocytogenes (LM). The sensor exhibited high sensitivity (Ra/Rg = 29 to 3 ppm), fast response dynamics (26/7 s), outstanding selectivity, and good long-term stability. Furthermore, the device was integrated into a wireless sensing module to realize remote real-time and precise detection of LM in practical applications, making it possible to evaluate food quality using gas sensors conveniently.

Large depth-of-field ultra-compact microscope by progressive optimization and deep learning

The optical microscope is customarily an instrument of substantial size and expense but limited performance. Here we report an integrated microscope that achieves optical performance beyond a commercial microscope with a 5×, NA 0.1 objective but only at 0.15 cm³ and 0.5 g, whose size is five orders of magnitude smaller than that of a conventional microscope. To achieve this, a progressive optimization pipeline is proposed which systematically optimizes both aspherical lenses and diffractive optical elements with over 30 times memory reduction compared to the end-to-end optimization. By designing a simulation-supervision deep neural network for spatially varying deconvolution during optical design, we accomplish over 10 times improvement in the depth-of-field compared to traditional microscopes with great generalization in a wide variety of samples. To show the unique advantages, the integrated microscope is equipped in a cell phone without any accessories for the application of portable diagnostics. We believe our method provides a new framework for the design of miniaturized high-performance imaging systems by integrating aspherical optics, computational optics, and deep learning.

NeuWS: Neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media

Diffraction-limited optical imaging through scattering media has the potential to transform many applications such as airborne and space-based imaging (through the atmosphere), bioimaging (through skin and human tissue), and fiber-based imaging (through fiber bundles). Existing wavefront shaping methods can image through scattering media and other obscurants by optically correcting wavefront aberrations using high-resolution spatial light modulators-but these methods generally require (i) guidestars, (ii) controlled illumination, (iii) point scanning, and/or (iv) statics scenes and aberrations. We propose neural wavefront shaping (NeuWS), a scanning-free wavefront shaping technique that integrates maximum likelihood estimation, measurement modulation, and neural signal representations to reconstruct diffraction-limited images through strong static and dynamic scattering media without guidestars, sparse targets, controlled illumination, nor specialized image sensors. We experimentally demonstrate guidestar-free, wide field-of-view, high-resolution, diffraction-limited imaging of extended, nonsparse, and static/dynamic scenes captured through static/dynamic aberrations.

Mold-free self-assembled scalable microlens arrays with ultrasmooth surface and record-high resolution

Microlens arrays (MLAs) based on the selective wetting have opened new avenues for developing compact and miniaturized imaging and display techniques with ultrahigh resolution beyond the traditional bulky and volumetric optics. However, the selective wetting lenses explored so far have been constrained by the lack of precisely defined pattern for highly controllable wettability contrast, thus limiting the available droplet curvature and numerical aperture, which is a major challenge towards the practical high-performance MLAs. Here we report a mold-free and self-assembly approach of mass-production of scalable MLAs, which can also have ultrasmooth surface, ultrahigh resolution, and the large tuning range of the curvatures. The selective surface modification based on tunable oxygen plasma can facilitate the precise pattern with adjusted chemical contrast, thus creating large-scale microdroplets array with controlled curvature. The numerical aperture of the MLAs can be up to 0.26 and precisely tuned by adjusting the modification intensity or the droplet dose. The fabricated MLAs have high-quality surface with subnanometer roughness and allow for record-high resolution imaging up to equivalently 10,328 ppi, as we demonstrated. This study shows a cost-effective roadmap for mass-production of high-performance MLAs, which may find applications in the rapid proliferating integral imaging industry and high-resolution display.

X-ray-to-visible light-field detection through pixelated colour conversion

Light-field detection measures both the intensity of light rays and their precise direction in free space. However, current light-field detection techniques either require complex microlens arrays or are limited to the ultraviolet-visible light wavelength ranges^1-4. Here we present a robust, scalable method based on lithographically patterned perovskite nanocrystal arrays that can be used to determine radiation vectors from X-rays to visible light (0.002-550 nm). With these multicolour nanocrystal arrays, light rays from specific directions can be converted into pixelated colour outputs with an angular resolution of 0.0018°. We find that three-dimensional light-field detection and spatial positioning of light sources are possible by modifying nanocrystal arrays with specific orientations. We also demonstrate three-dimensional object imaging and visible light and X-ray phase-contrast imaging by combining pixelated nanocrystal arrays with a colour charge-coupled device. The ability to detect light direction beyond optical wavelengths through colour-contrast encoding could enable new applications, for example, in three-dimensional phase-contrast imaging, robotics, virtual reality, tomographic biological imaging and satellite autonomous navigation.

Two-photon synthetic aperture microscopy for minimally invasive fast 3D imaging of native subcellular behaviors in deep tissue

Holistic understanding of physio-pathological processes requires noninvasive 3D imaging in deep tissue across multiple spatial and temporal scales to link diverse transient subcellular behaviors with long-term physiogenesis. Despite broad applications of two-photon microscopy (TPM), there remains an inevitable tradeoff among spatiotemporal resolution, imaging volumes, and durations due to the point-scanning scheme, accumulated phototoxicity, and optical aberrations. Here, we harnessed the concept of synthetic aperture radar in TPM to achieve aberration-corrected 3D imaging of subcellular dynamics at a millisecond scale for over 100,000 large volumes in deep tissue, with three orders of magnitude reduction in photobleaching. With its advantages, we identified direct intercellular communications through migrasome generation following traumatic brain injury, visualized the formation process of germinal center in the mouse lymph node, and characterized heterogeneous cellular states in the mouse visual cortex, opening up a horizon for intravital imaging to understand the organizations and functions of biological systems at a holistic level.

Virtual-scanning light-field microscopy for robust snapshot high-resolution volumetric imaging

High-speed three-dimensional (3D) intravital imaging in animals is useful for studying transient subcellular interactions and functions in health and disease. Light-field microscopy (LFM) provides a computational solution for snapshot 3D imaging with low phototoxicity but is restricted by low resolution and reconstruction artifacts induced by optical aberrations, motion and noise. Here, we propose virtual-scanning LFM (VsLFM), a physics-based deep learning framework to increase the resolution of LFM up to the diffraction limit within a snapshot. By constructing a 40 GB high-resolution scanning LFM dataset across different species, we exploit physical priors between phase-correlated angular views to address the frequency aliasing problem. This enables us to bypass hardware scanning and associated motion artifacts. Here, we show that VsLFM achieves ultrafast 3D imaging of diverse processes such as the beating heart in embryonic zebrafish, voltage activity in Drosophila brains and neutrophil migration in the mouse liver at up to 500 volumes per second.

Direct wavefront sensing with a plenoptic sensor based on deep learning

Traditional plenoptic wavefront sensors (PWS) suffer from the obvious step change of the slope response which leads to the poor performance of phase retrieval. In this paper, a neural network model combining the transformer architecture with the U-Net model is utilized to restore wavefront directly from the plenoptic image of PWS. The simulation results show that the averaged root mean square error (RMSE) of residual wavefront is less than 1/14λ (Marechal criterion), proving the proposed method successfully breaks through the non-linear problem existed in PWS wavefront sensing. In addition, our model performs better than the recently developed deep learning models and traditional modal approach. Furthermore, the robustness of our model to turbulence strength and signal level is also tested, proving the good generalizability of our model. To the best of our knowledge, it is the first time to perform direct wavefront detection with a deep-learning-based method in PWS-based applications and achieve the state-of-the-art performance.

Light field displays with computational vision correction for astigmatism and high-order aberrations with real-time implementation

Vision-correcting near-eye displays are necessary concerning the large population with refractive errors. However, varifocal optics cannot effectively address astigmatism (AST) and high-order aberration (HOAs); freeform optics has little prescription flexibility. Thus, a computational solution is desired to correct AST and HOA with high prescription flexibility and no increase in volume and hardware complexity. In addition, the computational complexity should support real-time rendering. We propose that the light field display can achieve such computational vision correction by manipulating sampling rays so that rays forming a voxel are re-focused on the retina. The ray manipulation merely requires updating the elemental image array (EIA), being a fully computational solution. The correction is first calculated based on an eye's wavefront map and then refined by a simulator performing iterative optimization with a schematic eye model. Using examples of HOA and AST, we demonstrate that corrected EIAs make sampling rays distributed within ±1 arcmin on the retina. Correspondingly, the synthesized image is recovered to nearly as clear as normal vision. We also propose a new voxel-based EIA generation method considering the computational complexity. All voxel positions and the mapping between voxels and their homogeneous pixels are acquired in advance and stored as a lookup table, bringing about an ultra-fast rendering speed of 10 ms per frame with no cost in computing hardware and rendering accuracy. Finally, experimental verification is carried out by introducing the HOA and AST with customized lenses in front of a camera. As a result, significantly recovered images are reported.