Fast non-line-of-sight imaging with high-resolution and wide field of view using synthetic wavelength holography

High-precision wavefront shaping based on high-pass filtering

Digital micromirror devices (DMDs) provide high refresh rates for fast light modulation in wavefront shaping. However, the binarized modulation of the optical field by the DMDs introduces a constant direct current (DC) optical field, which causes non-negligible background noise. This limits the modulation accuracy in DMD-based wavefront shaping. To address this, we propose a high-precision wavefront shaping method based on high-pass filtering, which eliminates the DC optical field and improves the anti-scattering focusing by filtering out the low-frequency spatial components. The experimental results show that this method significantly reduces the speckle similarity of optical fields modulated by different DMD patterns through the same scattering medium. Compared with the conventional method, this method can improve the anti-scattering focusing contrast by 1.7 times.

Harnessing Spectral Libraries From AVIRIS-NG Data for Precise PFT Classification: A Deep Learning Approach

The generation of spectral libraries using hyperspectral data allows for the capture of detailed spectral signatures, uncovering subtle variations in plant physiology, biochemistry, and growth stages, marking a significant advancement over traditional land cover classification methods. These spectral libraries enable improved forest classification accuracy and more precise differentiation of plant species and plant functional types (PFTs), thereby establishing hyperspectral sensing as a critical tool for PFT classification. This study aims to advance the classification and monitoring of PFTs in Shoolpaneshwar wildlife sanctuary, Gujarat, India using Airborne Visible/Infrared Imaging Spectrometer-Next Generation (AVIRIS-NG) and machine learning techniques. A comprehensive spectral library was developed, encompassing data from 130 plant species, with a focus on their spectral features to support precise PFT classification. The spectral data were collected using AVIRIS-NG hyperspectral imaging and ASD Handheld Spectroradiometer, capturing a wide range of wavelengths (400-1600 nm) to encompass the key physiological and biochemical traits of the plants. Plant species were grouped into five distinct PFTs using Fuzzy C-means clustering. Key spectral features, including band reflectance, vegetation indices, and derivative/continuum properties, were identified through a combination of ISODATA clustering and Jeffries-Matusita (JM) distance analysis, enabling effective feature selection for classification. To assess the utility of the spectral library, three advanced machine learning classifiers-Parzen Window (PW), Gradient Boosted Machine (GBM), and Stochastic Gradient Descent (SGD)-were rigorously evaluated. The GBM classifier achieved the highest accuracy, with an overall accuracy (OAA) of 0.94 and a Kappa coefficient of 0.93 across five PFTs.

Non-line-of-sight virtual modulated range migration imaging based on super-resolution histograms

High-resolution non-line-of-sight (NLOS) imaging under nanosecond time-resolution conditions is challenging in applications. We propose a novel NLOS imaging method consisting of deconvolution modified iterative back projection and virtual modulated range migration for low time-resolution system, obtaining super-resolution (SR) histogram signal and high-resolution NLOS images sequentially. The proposed method is applicable to both confocal and non-confocal configurations. Experimental results demonstrate that the method can obtain a 50 multiples SR histogram signal from 1 ns low resolution signals. Based on the SR histograms, our method can reconstruct higher resolution NLOS images with several times less data quantity than the related methods. These above advantages indicate that our method has significant application potential in NLOS systems with low time resolution.

Real-time non-line-of-sight computational imaging using spectrum filtering and motion compensation

Non-line-of-sight (NLOS) imaging aims at recovering the shape and albedo of hidden objects. Despite recent advances, real-time video of complex and dynamic scenes remains a major challenge owing to the weak signal of multiply scattered light. Here we propose and demonstrate a framework of spectrum filtering and motion compensation to realize high-quality NLOS video for room-sized scenes. Spectrum filtering leverages a wave-based model for denoising and deblurring in the frequency domain, enabling computational image reconstruction with a small number of sampling points. Motion compensation tailored with an interleaved scanning scheme can compute high-resolution live video during the acquisition of low-quality image sequences. Together, we demonstrate live NLOS videos at 4 fps for a variety of dynamic real-life scenes. The results mark a substantial stride toward real-time, large-scale and low-power NLOS imaging and sensing applications.

Efficient non-line-of-sight tracking with computational neuromorphic imaging

Non-line-of-sight (NLOS) sensing is an emerging technique that is capable of detecting objects hidden behind a wall, around corners, or behind other obstacles. However, NLOS tracking of moving objects is challenging due to signal redundancy and background interference. Here, we demonstrate computational neuromorphic imaging with an event camera for NLOS tracking, unaffected by the relay surface, which can efficiently obtain non-redundant information. We show how this sensor, which responds to changes in luminance within dynamic speckle fields, allows us to capture the most relevant events for direct motion estimation. The experimental results confirm that our method has superior performance in terms of efficiency, and accuracy, which greatly benefits from focusing on well-defined NLOS object tracking.

Memory-less scattering imaging with ultrafast convolutional optical neural networks

The optical memory effect in complex scattering media including turbid tissue and speckle layers has been a critical foundation for macroscopic and microscopic imaging methods. However, image reconstruction from strong scattering media without the optical memory effect has not been achieved. Here, we demonstrate image reconstruction through scattering layers where no optical memory effect exists, by developing a multistage convolutional optical neural network (ONN) integrated with multiple parallel kernels operating at the speed of light. Training this Fourier optics-based, parallel, one-step convolutional ONN with the strong scattering process for direct feature extraction, we achieve memory-less image reconstruction with a field of view enlarged by a factor up to 271. This device is dynamically reconfigurable for ultrafast multitask image reconstruction with a computational power of 1.57 peta-operations per second (POPS). Our achievement establishes an ultrafast and high energy-efficient optical machine learning platform for graphic processing.

Non-Line-of-Sight Imaging and Vibrometry Using a Comb-Calibrated Coherent Sensor

Non-line-of-sight (NLOS) imaging has the ability to reconstruct hidden objects, allowing a wide range of applications. Existing NLOS systems rely on pulsed lasers and time-resolved single-photon detectors to capture the information encoded in the time of flight of scattered photons. Despite remarkable advances, the pulsed time-of-flight LIDAR approach has limited temporal resolution and struggles to detect the frequency-associated information directly. Here, we propose and demonstrate the coherent scheme-frequency-modulated continuous wave calibrated by optical frequency comb-for high-resolution NLOS imaging, velocimetry, and vibrometry. Our comb-calibrated coherent sensor presents a system temporal resolution at subpicosecond and its superior signal-to-noise ratio permits NLOS imaging of complex scenes under strong ambient light. We show the capability of NLOS localization and 3D imaging at submillimeter scale and demonstrate NLOS vibrometry sensing at an accuracy of dozen Hertz. Our approach unlocks the coherent LIDAR techniques for widespread use in imaging science and optical sensing.

Enhancing the spatial resolution of time-of-flight based non-line-of-sight imaging via instrument response function deconvolution

Non-line-of-sight (NLOS) imaging retrieves the hidden scenes by utilizing the signals indirectly reflected by the relay wall. Benefiting from the picosecond-level timing accuracy, time-correlated single photon counting (TCSPC) based NLOS imaging can achieve theoretical spatial resolutions up to millimeter level. However, in practical applications, the total temporal resolution (also known as total time jitter, TTJ) of most current TCSPC systems exceeds hundreds of picoseconds due to the combined effects of multiple electronic devices, which restricts the underlying spatial resolution of NLOS imaging. In this paper, an instrument response function deconvolution (IRF-DC) method is proposed to overcome the constraints of a TCSPC system's TTJ on the spatial resolution of NLOS imaging. Specifically, we model the transient measurements as Poisson convolution process with the normalized IRF as convolution kernel, and solve the inverse problem with iterative deconvolution algorithm, which significantly improves the spatial resolution of NLOS imaging after reconstruction. Numerical simulations show that the IRF-DC facilitates light-cone transform and frequency-wavenumber migration solver to achieve successful reconstruction even when the system's TTJ reaches 1200 ps, which is equivalent to what was previously possible when TTJ was about 200 ps. In addition, the IRF-DC produces satisfactory reconstruction outcomes when the signal-to-noise ratio (SNR) is low. Furthermore, the effectiveness of the proposed method has also been experimentally verified. The proposed IRF-DC method is highly applicable and efficient, which may promote the development of high-resolution NLOS imaging.

Cohesive framework for non-line-of-sight imaging based on Dirac notation

The non-line-of-sight (NLOS) imaging field encompasses both experimental and computational frameworks that focus on imaging elements that are out of the direct line-of-sight, for example, imaging elements that are around a corner. Current NLOS imaging methods offer a compromise between accuracy and reconstruction time as experimental setups have become more reliable, faster, and more accurate. However, all these imaging methods implement different assumptions and light transport models that are only valid under particular circumstances. This paper lays down the foundation for a cohesive theoretical framework which provides insights about the limitations and virtues of existing approaches in a rigorous mathematical manner. In particular, we adopt Dirac notation and concepts borrowed from quantum mechanics to define a set of simple equations that enable: i) the derivation of other NLOS imaging methods from such single equation (we provide examples of the three most used frameworks in NLOS imaging: back-propagation, phasor fields, and f-k migration); ii) the demonstration that the Rayleigh-Sommerfeld diffraction operator is the propagation operator for wave-based imaging methods; and iii) the demonstration that back-propagation and wave-based imaging formulations are equivalent since, as we show, propagation operators are unitary. We expect that our proposed framework will deepen our understanding of the NLOS field and expand its utility in practical cases by providing a cohesive intuition on how to image complex NLOS scenes independently of the underlying reconstruction method.

Two-edge-resolved three-dimensional non-line-of-sight imaging with an ordinary camera

We introduce an approach for three-dimensional full-colour non-line-of-sight imaging with an ordinary camera that relies on a complementary combination of a new measurement acquisition strategy, scene representation model, and tailored reconstruction method. From an ordinary photograph of a matte line-of-sight surface illuminated by the hidden scene, our approach reconstructs a three-dimensional image of the scene hidden behind an occluding structure by exploiting two orthogonal edges of the structure for transverse resolution along azimuth and elevation angles and an information orthogonal scene representation for accurate range resolution. Prior demonstrations beyond two-dimensional reconstructions used expensive, specialized optical systems to gather information about the hidden scene. Here, we achieve accurate three-dimensional imaging using inexpensive, and ubiquitous hardware, without requiring a calibration image. Thus, our system may find use in indoor situations like reconnaissance and search-and-rescue.

Innovative OPA-based optical chip for enhanced digital holography

Digital holographic imaging has emerged as a transformative technology with significant implications for AR/VR devices. However, existing techniques often suffer from limitations such as restricted field of view (FOV), high power consumption, and contrast distortion. This paper introduces an innovative optical phased array (OPA)-based chip, integrating polarization, amplitude, and phase multiplexing for enhanced complex amplitude holographic imaging. A checkerboard-style staggered array is employed in the control strategy, substantially reducing power consumption and enabling the potential for large-scale array integration. To further enhance imaging quality, we introduce what we believe are two novel calibration strategies: one is to achieve super-resolution through block imaging methods, and the other is to image using sparse aperture methods. These advancements not only provide a robust foundation for high-quality holographic imaging, but also present a new paradigm for overcoming the inherent limitations of current active holographic imaging devices. Due to challenges in chip fabrication, the research is primarily simulation-based. Nevertheless, this work presents meaningful advancements in digital holographic imaging for AR/VR applications and provides a foundation for future experimental validations.

Imaging operator in indirect imaging correlography

Indirect imaging correlography (IIC) is a coherent imaging technique that provides access to the autocorrelation of the albedo of objects obscured from line-of-sight. This technique is used to recover sub-mm resolution images of obscured objects at large standoffs in non-line-of-sight (NLOS) imaging. However, predicting the exact resolving power of IIC in any given NLOS scene is complicated by the interplay between several factors, including object position and pose. This work puts forth a mathematical model for the imaging operator in IIC to accurately predict the images of objects in NLOS imaging scenes. Using the imaging operator, expressions for the spatial resolution as a function of scene parameters such as object position and pose are derived and validated experimentally. In addition, a self-supervised deep neural network framework to reconstruct images of objects from their autocorrelation is proposed. Using this framework, objects with ≈ 250 μ m features, located at 1 mt standoffs in an NLOS scene, are successfully reconstructed.

Toward Water-Immersion Programmable Meta-Display

Heading toward next-generation intelligent display, dynamic control capability for meta-devices is critical for real world applications. Beyond the conventional electrical/optical/mechanical/thermal tuning methods, liquid immersion recently has emerged as a facile tuning mechanism which is easily accessible (especially water) and practically implementable for large tuning area. However, due to the longstanding and critical drawback of lacking independent-encoding capability, the state-of-art immersion approach remains incapable of pixel-level programmable switching. Here a water-immersion tuning scheme with pixel-scale programmability for dynamic meta-displays is proposed. Tunable meta-pixels can be engineered to construct spectral selective patterns at prior-/post- immersion states, such that a metasurface enables pixel-level transforming animations for dynamic multifield meta-displays, including near-field dual-nanoprints and far-field dual-holographic displays. The proposed water-immersion programmable approach for meta-display, benefitting from its large tuning area, facile operation and strong repeatability, may find a revolutionary path toward next-generation intelligent display with practical applications in dynamic display/encryption, information anticounterfeit/storage, and optical sensors.

Light-driven single-cell rotational adhesion frequency assay

The interaction between cell surface receptors and extracellular ligands is highly related to many physiological processes in living systems. Many techniques have been developed to measure the ligand-receptor binding kinetics at the single-cell level. However, few techniques can measure the physiologically relevant shear binding affinity over a single cell in the clinical environment. Here, we develop a new optical technique, termed single-cell rotational adhesion frequency assay (scRAFA), that mimics in vivo cell adhesion to achieve label-free determination of both homogeneous and heterogeneous binding kinetics of targeted cells at the subcellular level. Moreover, the scRAFA is also applicable to analyze the binding affinities on a single cell in native human biofluids. With its superior performance and general applicability, scRAFA is expected to find applications in study of the spatial organization of cell surface receptors and diagnosis of infectious diseases.

SUPPLEMENTARY INFORMATION

The online version contains supplementary material available at 10.1186/s43593-022-00020-4.

Single-shot noninvasive imaging through scattering medium under white-light illumination

We experimentally investigate image reconstruction through a scattering medium under white-light illumination. To solve the inverse problem of noninvasive scattering imaging, a modified iterative algorithm is employed with an interpretable constraint on the optical transfer function (OTF). As a result, a sparse and real object can be retrieved whether it is illuminated with a narrowband or broadband light. Compared with the well-known speckle correlation technique (SCT), the proposed method requires no restrictions on the speckle autocorrelation and shows a potential advantage in scattering imaging.