Snapshot multidimensional photography through active optical mapping

In-sensor compressing via programmable optoelectronic sensors based on van der Waals heterostructures for intelligent machine vision

Efficiently capturing multidimensional signals containing spectral and temporal information is crucial for intelligent machine vision. Although in-sensor computing shows promise for efficient visual processing by reducing data transfer, its capability to compress temporal/spectral data is rarely reported. Here we demonstrate a programmable two-dimensional (2D) heterostructure-based optoelectronic sensor integrating sensing, memory, and computation for in-sensor data compression. Our 2D sensor captured and memorized/encoded optical signals, leading to in-device snapshot compression of dynamic videos and three-dimensional spectral data with a compression ratio of 8:1. The reconstruction quality, indicated by a peak signal-to-noise ratio value of 15.81 dB, is comparable to the 16.21 dB achieved through software. Meanwhile, the compressed action videos (in the form of 2D images) preserve all semantic information and can be accurately classified using in-sensor convolution without decompression, achieving accuracy on par with uncompressed videos (93.18% vs 83.43%). Our 2D optoelectronic sensors promote the development of efficient intelligent vision systems at the edge.

High-speed spectral imaging via multispectral pulse illumination and temporal-spectral decoupling

Spectral imaging (SI) faces inherent trade-offs among spatial, temporal, and spectral resolutions due to limited bandwidth. Current high-speed SI systems, relying on fast modulation or specific cameras, compromise spatial resolution or suffer high complexity and cost. This work presents a low-cost, high-speed SI scheme achieving up to 240 frames per second (fps) and even μs-resolved multispectral observation. This work reveals the inherent temporal-spectral redundancy in SI videos and reports an active temporal-spectral coupling (TSC) and decoupling (TSD) strategy for high-speed SI. We established a prototype system using a snapshot SI camera and a multispectral LED array. The LEDs sequentially illuminate the scene channel by channel in one exposure for TSC acquisition. Each channel of the measurement corresponds one-to-one to an illumination moment. A transformer-based network is applied to decouple textural and spectral information from the measurement and re-couple them to reconstruct images across all channels and time instances. Experiments validated that the reported framework can successfully record μs-resolved multispectral videos.

Compact single-shot multispectral polarization imager through joint spectral-polarization encoding

The technique of spectral polarization imaging (SPI) is a potent detection tool in various fields due to its ability to capture multi-dimensional information. However, existing SPI systems usually face challenges associated with architectural complexity and computational requirements, rendering them unsuitable for handheld, on-board, and real-time applications. Consequently, a compact single-shot multispectral polarization imager (CSMPI) is proposed, which employs a combined spectral-polarization encoding strategy to address the aforementioned issues. It incorporates a coded aperture for encoding multiple spectral channels together with linear polarization into a single measurement, enabling the simultaneous detection of up to nine light components with just one exposure. The resulting prototype consists solely of a color polarization detector and an imaging lens inserted with the small and easily fabricable coded aperture, which features compact dimensions of Φ5.5 cm × 21.5 cm and a light weight of approximately 670 g. This is particularly advantageous for application areas that require system miniaturization and rapid multi-dimensional detection.

Three-channel-switchable coded aperture snapshot multispectral polarization imaging

An ingenious and compact snapshot multispectral polarization imaging method is proposed based on a new, to the best of our knowledge, three-channel-switchable spectral polarization coded aperture. We utilize the coded aperture to simultaneously select three-channel light components and encode them with specific spectrum-polarization coefficients. It enables easy retrieval of each channel's light component from the mixed information via polarization measurements and linear decoding operations. Distinct three-channel light components can be detected simultaneously, thus achieving either three spectral images or linearly polarized ones per snapshot. The number of detectable light components is unlimited and triple that of snapshot times, showing its superior capability in measuring spectral polarization properties. The resulting prototype is miniaturized, featuring compact dimensions of Φ5.5 cm × 25 cm and a light weight of ∼800 g. This is attributed to its simplistic structure comprising a monochrome polarization detector and an imaging lens integrated with the coded aperture, making it suitable for portable and on-board applications. Furthermore, the absence of advanced or costly production technologies for manufacturing the prototype ensures an affordable price for its acquisition, facilitating widespread adoption and application of the proposed method.

SpectraTrack: megapixel, hundred-fps, and thousand-channel hyperspectral imaging

Hyperspectral imaging (HSI) finds broad applications in various fields due to its substantial optical signatures for the intrinsic identification of physical and chemical characteristics. However, it faces the inherent challenge of balancing spatial, temporal, and spectral resolution due to limited bandwidth. Here we present SpectraTrack, a computational HSI scheme that simultaneously achieves high spatial, temporal, and spectral resolution in the visible-to-near-infrared (VIS-NIR) spectral range. We deeply investigated the spatio-temporal-spectral multiplexing principle inherent in HSI videos. Based on this theoretical foundation, the SpectraTrack system uses two cameras including a line-scan imaging spectrometer for temporal-multiplexed hyperspectral data and an auxiliary RGB camera to capture motion flow. The motion flow guides hyperspectral reconstruction by reintegrating the scanned spectra into a 4D video. The SpectraTrack system can achieve around megapixel HSI at 100 fps with 1200 spectral channels, demonstrating its great application potential from drone-based anti-vibration video-rate HSI to high-throughput non-cooperative anti-spoofing.

Advancements in Hyperspectral Imaging and Computer-Aided Diagnostic Methods for the Enhanced Detection and Diagnosis of Head and Neck Cancer

Background/Objectives: Head and neck cancer (HNC), predominantly squamous cell carcinoma (SCC), presents a significant global health burden. Conventional diagnostic approaches often face challenges in terms of achieving early detection and accurate diagnosis. This review examines recent advancements in hyperspectral imaging (HSI), integrated with computer-aided diagnostic (CAD) techniques, to enhance HNC detection and diagnosis. Methods: A systematic review of seven rigorously selected studies was performed. We focused on CAD algorithms, such as convolutional neural networks (CNNs), support vector machines (SVMs), and linear discriminant analysis (LDA). These are applicable to the hyperspectral imaging of HNC tissues. Results: The meta-analysis findings indicate that LDA surpasses other algorithms, achieving an accuracy of 92%, sensitivity of 91%, and specificity of 93%. CNNs exhibit moderate performance, with an accuracy of 82%, sensitivity of 77%, and specificity of 86%. SVMs demonstrate the lowest performance, with an accuracy of 76% and sensitivity of 48%, but maintain a high specificity level at 89%. Additionally, in vivo studies demonstrate superior performance when compared to ex vivo studies, reporting higher accuracy (81%), sensitivity (83%), and specificity (79%). Conclusion: Despite these promising findings, challenges persist, such as HSI's sensitivity to external conditions, the need for high-resolution and high-speed imaging, and the lack of comprehensive spectral databases. Future research should emphasize dimensionality reduction techniques, the integration of multiple machine learning models, and the development of extensive spectral libraries to enhance HSI's clinical utility in HNC diagnostics. This review underscores the transformative potential of HSI and CAD techniques in revolutionizing HNC diagnostics, facilitating more accurate and earlier detection, and improving patient outcomes.

Hyperspectral retinal imaging in Alzheimer's disease and age-related macular degeneration: a review

While Alzheimer's disease and other neurodegenerative diseases have traditionally been viewed as brain disorders, there is growing evidence indicating their manifestation in the eyes as well. The retina, being a developmental extension of the brain, represents the only part of the central nervous system that can be noninvasively imaged at a high spatial resolution. The discovery of the specific pathological hallmarks of Alzheimer's disease in the retina of patients holds great promise for disease diagnosis and monitoring, particularly in the early stages where disease progression can potentially be slowed. Among various retinal imaging methods, hyperspectral imaging has garnered significant attention in this field. It offers a label-free approach to detect disease biomarkers, making it especially valuable for large-scale population screening efforts. In this review, we discuss recent advances in the field and outline the current bottlenecks and enabling technologies that could propel this field toward clinical translation.

Real-time machine learning-enhanced hyperspectro-polarimetric imaging via an encoding metasurface

Light fields carry a wealth of information, including intensity, spectrum, and polarization. However, standard cameras capture only the intensity, disregarding other valuable information. While hyperspectral and polarimetric imaging systems capture spectral and polarization information, respectively, in addition to intensity, they are often bulky, slow, and costly. Here, we have developed an encoding metasurface paired with a neural network enabling a normal camera to acquire hyperspectro-polarimetric images from a single snapshot. Our experimental results demonstrate that this metasurface-enhanced camera can accurately resolve full-Stokes polarization across a broad spectral range (700 to 1150 nanometer) from a single snapshot, achieving a spectral sensitivity as high as 0.23 nanometer. In addition, our system captures full-Stokes hyperspectro-polarimetric video in real time at a rate of 28 frames per second, primarily limited by the camera's readout rate. Our encoding metasurface offers a compact, fast, and cost-effective solution for multidimensional imaging that effectively uses information within light fields.

Meta-Attention Network Based Spectral Reconstruction with Snapshot Near-Infrared Metasurface

Near-infrared (NIR) spectral information is important for detecting and analyzing material compositions. However, snapshot NIR spectral imaging systems still pose significant challenges owing to the lack of high-performance NIR filters and bulky setups, preventing effective encoding and integration with mobile devices. This study introduces a snapshot spectral imaging system that employs a compact NIR metasurface featuring 25 distinct C4 symmetry structures. Benefitting from the sufficient spectral variety and low correlation coefficient among these structures, center-wavelength accuracy of 0.05 nm and full width at half maximum accuracy of 0.13 nm are realized. The system maintains good performance within an incident angle of 1°. A novel meta-attention network prior iterative denoising reconstruction (MAN-IDR) algorithm is developed to achieve high-quality NIR spectral imaging. By leveraging the designed metasurface and MAN-IDR, the NIR spectral images, exhibiting precise textures, minimal artifacts in the spatial dimension, and little crosstalk between spectral channels, are reconstructed from a single grayscale recording image. The proposed NIR metasurface and MAN-IDR hold great promise for further integration with smartphones and drones, guaranteeing the adoption of NIR spectral imaging in real-world scenarios such as aerospace, health diagnostics, and machine vision.

Snapshot spectral imaging: from spatial-spectral mapping to metasurface-based imaging

Snapshot spectral imaging technology enables the capture of complete spectral information of objects in an extremely short period of time, offering wide-ranging applications in fields requiring dynamic observations such as environmental monitoring, medical diagnostics, and industrial inspection. In the past decades, snapshot spectral imaging has made remarkable breakthroughs with the emergence of new computational theories and optical components. From the early days of using various spatial-spectral data mapping methods, they have evolved to later attempts to encode various dimensions of light, such as amplitude, phase, and wavelength, and then computationally reconstruct them. This review focuses on a systematic presentation of the system architecture and mathematical modeling of these snapshot spectral imaging techniques. In addition, the introduction of metasurfaces expands the modulation of spatial-spectral data and brings advantages such as system size reduction, which has become a research hotspot in recent years and is regarded as the key to the next-generation snapshot spectral imaging techniques. This paper provides a systematic overview of the applications of metasurfaces in snapshot spectral imaging and provides an outlook on future directions and research priorities.

Tutorial on compressed ultrafast photography

Significance

Compressed ultrafast photography (CUP) is currently the world's fastest single-shot imaging technique. Through the integration of compressed sensing and streak imaging, CUP can capture a transient event in a single camera exposure with imaging speeds from thousands to trillions of frames per second, at micrometer-level spatial resolutions, and in broad sensing spectral ranges.

Aim

This tutorial aims to provide a comprehensive review of CUP in its fundamental methods, system implementations, biomedical applications, and prospect.

Approach

A step-by-step guideline to CUP's forward model and representative image reconstruction algorithms is presented with sample codes and illustrations in Matlab and Python. Then, CUP's hardware implementation is described with a focus on the representative techniques, advantages, and limitations of the three key components-the spatial encoder, the temporal shearing unit, and the two-dimensional sensor. Furthermore, four representative biomedical applications enabled by CUP are discussed, followed by the prospect of CUP's technical advancement.

Conclusions

CUP has emerged as a state-of-the-art ultrafast imaging technology. Its advanced imaging ability and versatility contribute to unprecedented observations and new applications in biomedicine. CUP holds great promise in improving technical specifications and facilitating the investigation of biomedical processes.

Tunable image-mapping optical coherence tomography

We present tunable image-mapping optical coherence tomography (TIM-OCT), which can provide optimized imaging performance for a given application by using a programmable phase-only spatial light modulator in a low-coherence full-field spectral-domain interferometer. The resultant system can provide either a high lateral resolution or a high axial resolution in a snapshot without moving parts. Alternatively, the system can achieve a high resolution along all dimensions through a multiple-shot acquisition. We evaluated TIM-OCT in imaging both standard targets and biological samples. Additionally, we demonstrated the integration of TIM-OCT with computational adaptive optics in correcting sample-induced optical aberrations.

Augmented light field tomography through parallel spectral encoding

Snapshot recording of transient dynamics in three dimensions (3-D) is highly demanded in both fundamental and applied sciences. Yet it remains challenging for conventional high-speed cameras to address this need due to limited electronic bandwidth and reliance on mechanical scanning. The emergence of light field tomography (LIFT) provides a new solution to these long-standing problems and enables 3-D imaging at an unprecedented frame rate. However, based on sparse-view computed tomography, LIFT can accommodate only a limited number of projections, degrading the resolution in the reconstructed image. To alleviate this problem, we herein present a spectral encoding scheme to significantly increase the number of allowable projections in LIFT while maintaining its snapshot advantage. The resultant system can record 3-D dynamics at a kilohertz volumetric frame rate. Moreover, by using a multichannel compressed sensing algorithm, we improve the image quality with an enhanced spatial resolution and suppressed aliasing artifacts.

Compact light field photography towards versatile three-dimensional vision

Inspired by natural living systems, modern cameras can attain three-dimensional vision via multi-view geometry like compound eyes in flies, or time-of-flight sensing like echolocation in bats. However, high-speed, accurate three-dimensional sensing capable of scaling over an extensive distance range and coping well with severe occlusions remains challenging. Here, we report compact light field photography for acquiring large-scale light fields with simple optics and a small number of sensors in arbitrary formats ranging from two-dimensional area to single-point detectors, culminating in a dense multi-view measurement with orders of magnitude lower dataload. We demonstrated compact light field photography for efficient multi-view acquisition of time-of-flight signals to enable snapshot three-dimensional imaging with an extended depth range and through severe scene occlusions. Moreover, we show how compact light field photography can exploit curved and disconnected surfaces for real-time non-line-of-sight 3D vision. Compact light field photography will broadly benefit high-speed 3D imaging and open up new avenues in various disciplines.

Light field imaging typically requires large format detectors and yet often compromises resolution or speed. Here, compact light field photography is presented to lift both restrictions to see through and around severe occlusions in 3D and real time.

Snapshot hyperspectral light field tomography

We present snapshot hyperspectral light field tomography (Hyper-LIFT), a highly efficient method in recording a 5D (x, y, spatial coordinates; θ, φ, angular coordinates; λ, wavelength) plenoptic function. Using a Dove prism array and a cylindrical lens array, we simultaneously acquire multi-angled 1D en face projections of the object like those in standard sparse-view computed tomography. We further disperse those projections and measure the spectra in parallel using a 2D image sensor. Within a single snapshot, the resultant system can capture a 5D data cube with 270 × 270 × 4 × 4 × 360 voxels. We demonstrated the performance of Hyper-LIFT in imaging spectral volumetric scenes.

Continuously streaming compressed high-speed photography using time delay integration

An imaging system capable of acquiring high-resolution data at a high speed is in demand. However, the amount of optical information captured by a modern camera is limited by the data transfer bandwidth of electronics, resulting in a reduced spatial and temporal resolution. To overcome this problem, we developed continuously streaming compressed high-speed photography, which can record a dynamic scene with an unprecedented space-bandwidth-time product. By performing compressed imaging in a time-delay-integration manner, we continuously recorded a 0.85 megapixel video at 200 kHz, corresponding to an information flux of 170 gigapixels per second.