Memory effects in propagation of optical waves through disordered media

Neurophotonics beyond the surface: unmasking the brain's complexity exploiting optical scattering

The intricate nature of the brain necessitates the application of advanced probing techniques to comprehensively study and understand its working mechanisms. Neurophotonics offers minimally invasive methods to probe the brain using optics at cellular and even molecular levels. However, multiple challenges persist, especially concerning imaging depth, field of view, speed, and biocompatibility. A major hindrance to solving these challenges in optics is the scattering nature of the brain. This perspective highlights the potential of complex media optics, a specialized area of study focused on light propagation in materials with intricate heterogeneous optical properties, in advancing and improving neuronal readouts for structural imaging and optical recordings of neuronal activity. Key strategies include wavefront shaping techniques and computational imaging and sensing techniques that exploit scattering properties for enhanced performance. We discuss the potential merger of the two fields as well as potential challenges and perspectives toward longer term in vivo applications.

Widefield functional speckle-correlation optical scattering mesoscopy toward hemodynamic imaging

Speckle-correlation optical scattering imaging (SCOSI) has shown the potential for non-invasive biomedical diagnostic applications, which directly utilizes the scattering patterns to reconstruct the deep and non-line-of-sight objects. However, the course of the translation of this technique to preclinical biomedical imaging applications has been postponed by the following two facts: 1) the field of view of SCOSI was significantly limited by the optical memory effect, and 2) the molecular-tagged functional imaging of the biological tissues remains largely unexplored. In this work, a proof-of-concept design of the first-generation widefield functional SCOSI (WF-SCOSI) system was presented for simultaneously achieving mesoscopic mapping of fluid morphology and flow rate, which was realized by implementing the concepts of scanning synthesis and fluorescence scattering flowmetry. The ex vivo imaging results of the fluorescence-labeled large-scale blood vessel network phantom underneath the strong scatters demonstrated the effectiveness of WF-SCOSI toward non-invasive hemodynamic imaging applications.

Single-pixel transmission matrix recovery via two-photon fluorescence

Imaging at depth in opaque materials has long been a challenge. Recently, wavefront shaping has enabled notable advance for deep imaging. Nevertheless, most noninvasive wavefront-shaping methods require cameras, lack the sensitivity for deep imaging under weak optical signals, or can only focus on a single "guidestar." Here, we retrieve the transmission matrix (TM) noninvasively using two-photon fluorescence exploiting a single-pixel detection combined with a computational framework, allowing to achieve single-target focus on multiple guidestars spread beyond the memory effect range. In addition, if we assume that memory effect correlations exist in the TM, we are able to substantially reduce the number of measurements needed.

Imaging objects hidden inside the strongly scattering media based on bidirectional ghost imaging

We demonstrate a novel, to the best of our knowledge, method for imaging objects hidden inside the strongly scattering media based on bidirectional ghost imaging (GI). In this method, GI is performed separately on both sides of the object, resulting in two GI results. Through an autocorrelation operation to the two GI results, the convolution between the autocorrelation of the object and the point spread function (PSF) of the strongly scattering media can be recovered. Therefore, the object can be recovered by obtaining the PSF of the strongly scattering media through noninvasive measurement or numerical calculation. Simulation and experimental results show that bidirectional ghost imaging (BGI) can reconstruct high-quality images, particularly when the thickness of the strongly scattering media greatly exceeds the scattering mean free path.

Wavefront shaping improves the transparency of the scattering media: a review

Significance

Wavefront shaping (WFS) can compensate for distortions by optimizing the wavefront of the input light or reversing the transmission matrix of the media. It is a promising field of research. A thorough understanding of principles and developments of WFS is important for optical research.

Aim

To provide insight into WFS for researchers who deal with scattering in biomedicine, imaging, and optical communication, our study summarizes the basic principles and methods of WFS and reviews recent progress.

Approach

The basic principles, methods of WFS, and the latest applications of WFS in focusing, imaging, and multimode fiber (MMF) endoscopy are described. The practical challenges and prospects of future development are also discussed.

Results

Data-driven learning-based methods are opening up new possibilities for WFS. High-resolution imaging through MMFs can support small-diameter endoscopy in the future.

Conclusion

The rapid development of WFS over the past decade has shown that the best solution is not to avoid scattering but to find ways to correct it or even use it. WFS with faster speed, more optical modes, and more modulation degrees of freedom will continue to drive exciting developments in various fields.

Fast wavefront shaping for two-photon brain imaging with multipatch correction

Nonlinear fluorescence microscopy promotes in-vivo optical imaging of cellular structure at diffraction-limited resolution deep inside scattering biological tissues. Active compensation of tissue-induced aberrations and light scattering through adaptive wavefront correction further extends the accessible depth by restoring high resolution at large depth. However, those corrections are only valid over a very limited field of view within the angular memory effect. To overcome this limitation, we introduce an acousto-optic light modulation technique for fluorescence imaging with simultaneous wavefront correction at pixel scan speed. Biaxial wavefront corrections are first learned by adaptive optimization at multiple locations in the image field. During image acquisition, the learned corrections are then switched on the fly according to the position of the excitation focus during the raster scan. The proposed microscope is applied to in vivo transcranial neuron imaging and demonstrates multi-patch correction of thinned skull-induced aberrations and scattering at 40-kHz data acquisition speed.

Wavefront shaping through a free-form scattering object

Wavefront shaping is a technique to study and control light transport inside scattering media. Wavefront shaping is considered to be applicable to any complex material, yet in most previous studies, the only sample geometries that are studied are slabs or wave-guides. In this paper, we study how macroscopic changes in the sample shape affect light scattering using the wavefront shaping technique. Using a flexible scattering material, we optimize the intensity of light in a focusing spot using wavefront shaping and record the optimized pattern, comparing the enhancement for different curvatures and beam radii. We validate our hypothesis that wavefront shaping has a similar enhancement regardless of the free-form shape of the sample and thus offers relevant potential for industrial applications. We propose a new figure of merit to evaluate the performance of wavefront shaping for different shapes. Surprisingly, based on this figure of merit, we observe that for this particular sample, wavefront shaping has a slightly better performance for a free-form shape than for a slab shape.

Overlapping speckle correlation algorithm for high-resolution imaging and tracking of objects in unknown scattering media

Optical imaging in scattering media is important to many fields but remains challenging. Recent methods have focused on imaging through thin scattering layers or thicker scattering media with prior knowledge of the sample, but this still limits practical applications. Here, we report an imaging method named 'speckle kinetography' that enables high-resolution imaging in unknown scattering media with thicknesses up to about 6 transport mean free paths. Speckle kinetography non-invasively records a series of incoherent speckle images accompanied by object motion and the inherently retained object information is extracted through an overlapping speckle correlation algorithm to construct the object's autocorrelation for imaging. Under single-colour light-emitting diode, white light, and fluorescence illumination, we experimentally demonstrate 1 μm resolution imaging and tracking of objects moving in scattering samples, while reducing the requirements for prior knowledge. We anticipate this method will enable imaging in currently inaccessible scenarios.

Integration of high-resolution imaging through scattering medium into a disposable micro-endoscope via projection of 2D spots-array

The objective of this research includes integration of high-resolution imaging through scattering medium, such as blood, into a disposable micro-endoscope. A fiber laser integrated into the micro-endoscope as part of its illumination channel, allows to project a tunable array of spots of light onto an object, that is located behind the scattering medium. We have a laser fiber as part of the illumination channel of a disposable micro-endoscope. By using proper optics, we convert the temporal modulation of the laser into spatial distribution. Thus, the result is generation of spatial spots when using a pulsed laser. The detection channel is a holographic recording of the collected back scattered light, that allows extraction of the electrical field. By time integrating the field we obtain the realization of the spatial array of illumination spots formed on top of the inspected object and behind the scattering medium. By changing the temporal modulation of the illumination laser (changing its temporal photonic signals), we can tune the positions of the spots in the illumination array. If the distance between the projected spots is larger than the imaging resolution, then by applying localization microscopy algorithms together with scanning of the position of the spots in the array, will yield a high-resolution reconstruction of the inspected object. We theoretically and experimentally demonstrate the discussed operation principle and show the potential of the proposed concept as a modality in medical endoscopic procedures.

Super-resolution imaging through scattering media based on improved triple correlation recursion and deterministic iterative estimation

Iterative phase retrieval algorithms are commonly used in computational techniques and optimization methods to obtain the reconstruction of objects hidden behind opaque scattering media. However, these methods are susceptible to converging to incorrect local minima, and the calculation results tend to be unstable. In this paper, a triple-correlation-based super-resolution imaging (TCSI) framework is proposed to achieve single-shot imaging of unknown objects hidden behind the scattering medium. The amplitude spectrum of the object is obtained by a speckle correlation (SC) method. Iterative relaxation recursion (IRR) sufficiently extracts object information from the triple correlation (TC) of the speckle patterns, serving as the prior initial guess for the iterative estimation algorithm (IE) to obtain a deterministic phase spectrum. Blur correction (BC) is then applied to the diffraction-limited image to achieve super-resolution imaging. Experimental results demonstrate that the flexible framework could effectively overcome the influence of speckle resolution and outperform traditional methods in terms of performance. Our approach provides a basis for non-invasively visualizing various samples behind scattering media.

Extrapolated speckle-correlation imaging with an untrained deep neural network

We present a method for speckle-correlation imaging with an extended field of view to observe spatially non-sparse objects. In speckle-correlation imaging, an object is recovered from a non-invasively captured image through a scattering medium by assuming shift-invariance of the optical process called the memory effect. The field of view of speckle-correlation imaging is limited by the size of the memory effect, and it can be extended by extrapolating the speckle correlation in the reconstruction process. However, spatially sparse objects are assumed in the inversion process because of its severe ill-posedness. To address this issue, we introduce a deep image prior, which regularizes the image statistics by using the structure of an untrained convolutional neural network, to speckle-correlation imaging. We experimentally demonstrated the proposed method and showed the possibility of extending the method to imaging through scattering media.

Tracing multiple scattering trajectories for deep optical imaging in scattering media

Multiple light scattering hampers imaging objects in complex scattering media. Approaches used in real practices mainly aim to filter out multiple scattering obscuring the ballistic waves that travel straight through the scattering medium. Here, we propose a method that makes the deterministic use of multiple scattering for microscopic imaging of an object embedded deep within scattering media. The proposed method finds a stack of multiple complex phase plates that generate similar light trajectories as the original scattering medium. By implementing the inverse scattering using the identified phase plates, our method rectifies multiple scattering and amplifies ballistic waves by almost 600 times. This leads to a significant increase in imaging depth-more than three times the scattering mean free path-as well as the correction of image distortions. Our study marks an important milestone in solving the long-standing high-order inverse scattering problems.

High-throughput imaging through dynamic scattering media based on speckle de-blurring

Effectively imaging through dynamic scattering media is of great importance and challenge. Some imaging methods based on physical or learning models have been designed for object reconstruction. However, with an increase in exposure time or more drastic changes in the scattering medium, the speckle pattern superimposed during camera integration time undergoes more significant changes, resulting in a modification of the collected speckle structure and increased blurring, which brings significant challenges to the reconstruction. Here, the clearer structural information of blurred speckles is unearthed with a presented speckle de-blurring algorithm, and a high-throughput imaging method through rapidly changing scattering media is proposed for reconstruction under long exposure. For the problem of varying blur degrees in different regions of the speckle, a block-based method is proposed to divide the speckle into distinct sub-speckles, which can realize the reconstruction of hidden objects. The imaging of hidden objects with different complexity through dynamic scattering media is demonstrated, and the reconstruction results are improved significantly for speckles with different blur degrees, which verifies the effectiveness of the method. This method is a high-throughput approach that enables non-invasive imaging solely through the collection of a single speckle. It directly operates on blurred speckles, making it suitable for traditional speckle-correlation methods and deep learning (DL) methods. This provides a new way of thinking about solving practical scattering imaging challenges.

Imaging through a scattering medium via model-driven deep learning

Imaging through a scattering medium is of great significance in many areas. Especially, speckle correlation imaging has been valued for its noninvasiveness. In this work, we report a deep learning solution that incorporates the physical model and an additional regularization for high-fidelity speckle correlation imaging. Without large-scale data to train, the physical model and regularization prior provide a correct direction for neural network to precisely reconstruct hidden objects from speckle under different scattering scenarios and noise levels. Experimental results demonstrate that the proposed method presents a significant advance in improving generalization and combating the invasion of noise.

Imaging through scattering media under strong ambient light interference via the lock-in process

Scattered light imaging techniques leveraging memory effects have been extensively investigated, yet most approaches are limited to operating in predominantly dark environments. The introduction of additional optical noise disrupts the fine structure of the original speckle pattern, undermining spatial correlation and resulting in imaging failure. In this study, we present a high-performance imaging method that integrates a lock-in process to overcome this limitation. Our experimental results demonstrate that the proposed technique enables successful imaging of targets in low signal-to-background ratio (SBR) environments, even at SBR levels as low as -28.0 dB. Furthermore, the method allows for the directional separation of targets with distinct modulation frequencies. This innovative approach has the potential to significantly expand the applicability of scattering imaging techniques by eliminating the constraints of dark field environments, thereby enhancing the convenience of in vivo microscopy and daytime astronomical observations.

Spatial intensity correlations of transmitted intensity patterns emerging from large particles

Propagation of a coherent light beam through a random medium generates speckle patterns, in which some information of media and object is hidden. Speckles produced by particles smaller than wavelength are studied thoroughly, yet it is also essential to investigate speckles produced by larger particles. In this paper, the spatial intensity correlations of transmitted speckle patterns generated by large particles are studied theoretically and experimentally. A semi-empirical expression of spatial intensity correlation function of speckle patterns is derived based on Bethe-Salpeter equation, taking particle size and concentration into account. After performing experiments with various particle sizes and concentrations, we fit the theoretical expression to experimental results and determine the introduced parameters. We analyze the variation of spatial intensity correlation function with particle size and concentration. Theoretical analyses and experimental results given in this paper have potential applications in coherent imaging through random and disordered media.

Characterization of the angular memory effect of dynamic turbid media

The optical angular memory effect (AME) is a basic feature of turbid media and defines the correlation of speckles when the incident light is tilted. AME based imaging through solid scattering media such as ground glass and biomedical tissue has been recently developed. However, in the case of liquid media such as turbid water or blood, the speckle pattern exhibits dynamic time-varying characteristics, which introduces several challenges. The AME of the thick volume dynamic media is particularly different from the layer scatterers. In practice, there are more parameters, e.g., scattering particle size, shape, density, or even the illuminating beam aperture that can influence the AME range. Experimental demonstration of AME phenomenon in liquid dynamic media and confirm the distinctions will contribution to complete the AME theory. In this paper, a dual-polarization speckle detection setup was developed to characterize the AME of dynamic turbid media, where two orthogonal polarized beams were employed for simultaneous detection by a single CCD. The AME of turbid water, milk and blood were measured. The influence of thickness, concentration, particle size and shape, and beam diameter were analyzed. The AME increasement of upon the decrease of beam diameter was tested and verified. The results demonstrate the feasibility of this method for investigating the AME phenomenon and provide guidance for AME based imaging through scattering media.

Complex amplitude field recovery of a scattering media obstructed object with multi-captured images

An iterative-based method for recovering the complex amplitude field behind scattering media is presented in this Letter. This method compensates the random phase modulation of scattering media by using multiple captured scattered light fields. Complex amplitude reconstruction with local iterative averaging of scattered light fields, and double weighted feedback is efficiently applied. Two feasible types of system setups, with varying detector positions and wavelength, are proposed. Simulations and proof-of-concept experiments are employed to demonstrate the effectiveness of the proposed method in reconstructing complex amplitude of a hidden target.

Enhanced secrecy in optical communication using speckle from multiple scattering layers

We study the secrecy of an optical communication system with two scattering layers, to hide both the sender and receiver, by measuring the correlation of the intermediate speckle generated between the two layers. The binary message is modulated as spatially shaped wavefronts, and the high number of transmission modes of the scattering layers allows for many uncorrelated incident wavefronts to send the same message, making it difficult for an attacker to intercept or decode the message and thus increasing secrecy. We collect 50,000 intermediate speckle patterns and analyze their correlation distribution using the Kolmogorov-Smirnov (K-S) test. We search for further correlations using the K-Means and Hierarchical unsupervised classification algorithms. We find no correlation between the intermediate speckle and the message, suggesting a person-in-the-middle attack is not possible. This method is compatible with any digital encryption method and is applicable for codifications in optical wireless communication (OWC).

Randomness assisted in-line holography with deep learning

We propose and demonstrate a holographic imaging scheme exploiting random illuminations for recording hologram and then applying numerical reconstruction and twin image removal. We use an in-line holographic geometry to record the hologram in terms of the second-order correlation and apply the numerical approach to reconstruct the recorded hologram. This strategy helps to reconstruct high-quality quantitative images in comparison to the conventional holography where the hologram is recorded in the intensity rather than the second-order intensity correlation. The twin image issue of the in-line holographic scheme is resolved by an unsupervised deep learning based method using an auto-encoder scheme. Proposed learning technique leverages the main characteristic of autoencoders to perform blind single-shot hologram reconstruction, and this does not require a dataset of samples with available ground truth for training and can reconstruct the hologram solely from the captured sample. Experimental results are presented for two objects, and a comparison of the reconstruction quality is given between the conventional inline holography and the one obtained with the proposed technique.

Scan-free time-of-flight-based three-dimensional imaging through a scattering layer

Reconstructing an object's three-dimensional shape behind a scattering layer with a single exposure is of great significance in real-life applications. However, due to the little information captured by a single exposure while strongly perturbed by the scattering layer and encoded by free-space propagation, existing methods cannot achieve scan-free three-dimensional reconstruction through the scattering layer in macroscopic scenarios using a short acquisition time of seconds. In this paper, we proposed a scan-free time-of-flight-based three-dimensional reconstruction method based on explicitly modeling and inverting the time-of-flight-based scattering light propagation in a non-confocal imaging system. The non-confocal time-of-flight-based scattering imaging model is developed to map the three-dimensional object shape information to the time-resolved measurements, by encoding the three-dimensional object shape into the free-space propagation result and then convolving with the scattering blur kernel derived from the diffusion equation. To solve the inverse problem, a three-dimensional shape reconstruction algorithm consisting of the deconvolution and diffractive wave propagation is developed to invert the effects caused by the scattering diffusion and the free-space propagation, which reshapes the temporal and spatial distribution of scattered signal photons and recovers the object shape information. Experiments on a real scattering imaging system are conducted to demonstrate the effectiveness of the proposed method. The single exposure used in the experiment only takes 3.5 s, which is more than 200 times faster than confocal scanning methods. Experimental results show that the proposed method outperforms existing methods in terms of three-dimensional reconstruction accuracy and imaging limit subjectively and objectively. Even though the signal photons captured by a single exposure are too highly scattered and attenuated to present any valid information in time gating, the proposed method can reconstruct three-dimensional objects located behind the scattering layer of 9.6 transport mean free paths (TMFPs), corresponding to the round-trip scattering length of 19.2 TMFPs.

Speckle-correlation imaging through a kaleidoscopic multimode fiber

Speckle-correlation imaging techniques are widely used for noninvasive imaging through complex scattering media. While light propagation through multimode fibers and scattering media share many analogies, reconstructing images through multimode fibers from speckle correlations remains an unsolved challenge. Here, we exploit a kaleidoscopic memory effect emerging in square-core multimode fibers and demonstrate fluorescence imaging with no prior knowledge on the fiber. Experimentally, our approach simply requires to translate random speckle patterns at the input of a square-core fiber and to measure the resulting fluorescence intensity with a bucket detector. The image of the fluorescent object is then reconstructed from the autocorrelation of the measured signal by solving an inverse problem. This strategy does not require the knowledge of the fragile deterministic relation between input and output fields, which makes it promising for the development of flexible minimally invasive endoscopes.

Imaging operator in indirect imaging correlography

Indirect imaging correlography (IIC) is a coherent imaging technique that provides access to the autocorrelation of the albedo of objects obscured from line-of-sight. This technique is used to recover sub-mm resolution images of obscured objects at large standoffs in non-line-of-sight (NLOS) imaging. However, predicting the exact resolving power of IIC in any given NLOS scene is complicated by the interplay between several factors, including object position and pose. This work puts forth a mathematical model for the imaging operator in IIC to accurately predict the images of objects in NLOS imaging scenes. Using the imaging operator, expressions for the spatial resolution as a function of scene parameters such as object position and pose are derived and validated experimentally. In addition, a self-supervised deep neural network framework to reconstruct images of objects from their autocorrelation is proposed. Using this framework, objects with ≈ 250 μ m features, located at 1 mt standoffs in an NLOS scene, are successfully reconstructed.

High-quality color image restoration from a disturbed graded-index imaging system by deep neural networks

Imaging transmission plays an important role in endoscopic clinical diagnosis involved in modern medical treatment. However, image distortion due to various reasons has been a major obstacle to state-of-art endoscopic development. Here, as a preliminary study we demonstrate ultra-efficient recovery of exemplary 2D color images transmitted by a disturbed graded-index (GRIN) imaging system through the deep neural networks (DNNs). Indeed, the GRIN imaging system can preserve analog images through the GRIN waveguides with high quality, while the DNNs serve as an efficient tool for imaging distortion correction. Combining GRIN imaging systems and DNNs can greatly reduce the training process and achieve ideal imaging transmission. We consider imaging distortion under different realistic conditions and use both pix2pix and U-net type DNNs to restore the images, indicating the suitable network in each condition. This method can automatically cleanse the distorted images with superior robustness and accuracy, which can potentially be used in minimally invasive medical applications.

Functional imaging through scattering medium via fluorescence speckle demixing and localization

Recently, fluorescence-based optical techniques have emerged as a powerful tool to probe information in the mammalian brain. However, tissue heterogeneities prevent clear imaging of deep neuron bodies due to light scattering. While several up-to-date approaches based on ballistic light allow to retrieve information at shallow depths inside the brain, non-invasive localization and functional imaging at depth still remains a challenge. It was recently shown that functional signals from time-varying fluorescent emitters located behind scattering samples could be retrieved by using a matrix factorization algorithm. Here we show that the seemingly information-less, low-contrast fluorescent speckle patterns recovered by the algorithm can be used to locate each individual emitter, even in the presence of background fluorescence. We test our approach by imaging the temporal activity of large groups of fluorescent sources behind different scattering phantoms mimicking biological tissues, and through a brain slice with a thickness of ∼200 µm.

Deep speckle reassignment: towards bootstrapped imaging in complex scattering states with limited speckle grains

Optical imaging through scattering media is a practical challenge with crucial applications in many fields. Many computational imaging methods have been designed for object reconstruction through opaque scattering layers, and remarkable recovery results have been demonstrated in the physical models or learning models. However, most of the imaging approaches are dependent on relatively ideal states with a sufficient number of speckle grains and adequate data volume. Here, the in-depth information with limited speckle grains has been unearthed with speckle reassignment and a bootstrapped imaging method is proposed for reconstruction in complex scattering states. Benefiting from the bootstrap priors-informed data augmentation strategy with a limited training dataset, the validity of the physics-aware learning method has been demonstrated and the high-fidelity reconstruction results through unknown diffusers are obtained. This bootstrapped imaging method with limited speckle grains broadens the way to highly scalable imaging in complex scattering scenes and gives a heuristic reference to practical imaging problems.

Central wavelength estimation in spectral imaging behind a diffuser via deep learning

Multispectral imaging through scattering media is an important practical issue in the field of sensing. The light from a scattering medium is expected to carry information about the spectral properties of the medium, as well as geometrical information. Because spatial and spectral information of the object is encoded in speckle images, the information about the structure and spectrum of the object behind the scattering medium can be estimated from those images. Here we propose a deep learning-based strategy that can estimate the central wavelength from speckle images captured with a monochrome camera. When objects behind scattering media are illuminated with narrowband light having different spectra with different spectral peaks, deep learning of speckle images acquired at different central wavelengths can extend the spectral region to reconstruct images and estimate the central wavelengths of the illumination light. The proposed method achieves central wavelength estimation in 1 nm steps for objects whose central wavelength varies in a range of 100 nm. Because our method can achieve image reconstruction and central wavelength estimation in a single shot using a monochrome camera, this technique will pave the way for multispectral imaging through scattering media.

Efficient framework of solving time-gated reflection matrix for imaging through turbid medium

Imaging through turbid medium is a long pursuit in many research fields, such as biomedicine, astronomy and automatic vehicle, in which the reflection matrix-based method is a promising solution. However, the epi-detection geometry suffers from round-trip distortion and it is challenging to isolate the input and output aberrations in non-ideal cases due to system imperfections and measurement noises. Here, we present an efficient framework based on single scattering accumulation together with phase unwrapping that can accurately separate input and output aberrations from the noise-affected reflection matrix. We propose to only correct the output aberration while suppressing the input aberration by incoherent averaging. The proposed method is faster in convergence and more robust against noise, avoiding precise and tedious system adjustments. In both simulations and experiments, we demonstrate the diffraction-limited resolution capability under optical thickness beyond 10 scattering mean free paths, showing the potential of applications in neuroscience and dermatology.

DeepSCI: scalable speckle correlation imaging using physics-enhanced deep learning

In this Letter we present a physics-enhanced deep learning approach for speckle correlation imaging (SCI), i.e., DeepSCI. DeepSCI incorporates the theoretical model of SCI into both the training and test stages of a neural network to achieve interpretable data preprocessing and model-driven fine-tuning, allowing the full use of data and physics priors. It can accurately reconstruct the image from the speckle pattern and is highly scalable to both medium perturbations and domain shifts. Our experimental results demonstrate the suitability and effectiveness of DeepSCI for solving the problem of limited generalization generally encountered in data-driven approaches.

Adaptive inverse mapping: a model-free semi-supervised learning approach towards robust imaging through dynamic scattering media

Imaging through scattering media is a useful and yet demanding task since it involves solving for an inverse mapping from speckle images to object images. It becomes even more challenging when the scattering medium undergoes dynamic changes. Various approaches have been proposed in recent years. However, none of them are able to preserve high image quality without either assuming a finite number of sources for dynamic changes, assuming a thin scattering medium, or requiring access to both ends of the medium. In this paper, we propose an adaptive inverse mapping (AIP) method, which requires no prior knowledge of the dynamic change and only needs output speckle images after initialization. We show that the inverse mapping can be corrected through unsupervised learning if the output speckle images are followed closely. We test the AIP method on two numerical simulations: a dynamic scattering system formulated as an evolving transmission matrix and a telescope with a changing random phase mask at a defocused plane. Then we experimentally apply the AIP method to a multimode-fiber-based imaging system with a changing fiber configuration. Increased robustness in imaging is observed in all three cases. AIP method's high imaging performance demonstrates great potential in imaging through dynamic scattering media.

Adaptive optics for optical microscopy [Invited]

Optical microscopy is widely used to visualize fine structures. When applied to bioimaging, its performance is often degraded by sample-induced aberrations. In recent years, adaptive optics (AO), originally developed to correct for atmosphere-associated aberrations, has been applied to a wide range of microscopy modalities, enabling high- or super-resolution imaging of biological structure and function in complex tissues. Here, we review classic and recently developed AO techniques and their applications in optical microscopy.

Using speckle correlations for single-shot 3D imaging

Recovery of a 3D object behind a scattering medium is an important problem in many fields, including biomedical and defense applications. Speckle correlation imaging can recover objects in a single shot but contains no depth information. To date, its extension to 3D recovery has relied on multiple measurements, multi-spectral light, or pre-calibration of the speckle with a reference object. Here, we show that the presence of a point source behind the scatterer enables single-shot reconstruction of multiple objects at multiple depths. The method relies on speckle scaling from the axial memory effect, in addition to the transverse one, and recovers objects directly, without the need for phase retrieval. We provide simulation and experimental results to show object reconstructions at different depths with a single-shot measurement. We also provide theoretical principles describing the region where speckle scales with axial distance and its effects on the depth of field. Our technique will be useful where a natural point source exists, such as fluorescence imaging or car headlights in fog.

Multi-target object scattering imaging with intensity correlation of structured illumination

Imaging through scattering layers based on the optical memory effect (OME) concept has been widely investigated in recent years. Among many scattering scenarios, it is very important to recover hidden targets with proper spatial distribution in the scene where multiple targets out of the OME range exist. In this Letter, we put forward a method for multi-target object scattering imaging. With the help of intensity correlation between the structured illumination patterns and recorded speckle images, the relative position of all hidden targets can be obtained and the movement of the targets within the OME range can be tracked. We experimentally implement scattering imaging with 16 targets and the motion tracking of them. Our results present a significant advance in a large field of view scattering imaging with multiple targets.

Path sampling and integration method to calculate speckle patterns

A stable speckle pattern is generated when a coherent beam illuminates a stationary scattering medium that contains numerous scatterers with fixed positions. To date, there has been no valid method to the best of our knowledge for calculating the speckle pattern of a macro medium with a large number of scatterers. Here, a new method based on possible path sampling with corresponding weights and coherent superposition is presented for the simulation of optical field propagation in a scattering medium and output speckle patterns. In this method, a photon is launched onto a medium with fixed scatterers. It propagates in one direction; upon collision with a scatterer, its direction is updated. The procedure is repeated until it exits the medium. A sampled path is obtained in this manner. By repeatedly launching photons, numerous independent optical paths can be sampled. A speckle pattern, corresponding to the probability density of the photon, is formed by the coherent superposition of sufficiently sampled path lengths ending on a receiving screen. This method can be used in sophisticated studies of the influences of medium parameters, motion of scatterers, sample distortions on speckle distributions, and morphological appearances. It can be used for micro-examination of optical fields in scattering media and may inspire new methods and techniques for non-invasive precision detection and diagnosis of scattering media.

Deep-learning-based cross-talk free and high-security compressive encryption with spatially incoherent illumination

Incoherent optical cryptosystem is promising for its immunity against coherent noise and insensitivity to misalignment, and compressive encryption is desirable considering the increasingly demand on the exchange of encrypted data via Internet. In this paper, we propose a novel optical compressive encryption approach with spatially incoherent illumination based on deep learning (DL) and space multiplexing. For encryption, the plaintexts are individually sent to the scattering-imaging-based encryption (SIBE) scheme where they are transformed to scattering images with noise appearances. Afterwards, these images are randomly sampled and then integrated into a single package (i.e., ciphertext) by space multiplexing. The decryption is basically the inverse of the encryption, while it involves an ill-posed problem (i.e., recovering the noise-like scattering image from its randomly sampled version). We demonstrated that such a problem can be well resolved by DL. The proposal is radically free from the cross-talk noise existing in many current multiple-image encryption schemes. Also, it gets rid of the linearity bothering the SIBE and is hence robust against the ciphertext-only attack based on phase retrieval algorithm. We present a series of experimental results to confirm the effectiveness and feasibility of the proposal.

Speckle autocorrelation separation for multi-target scattering imaging

Imaging through scattering media remains a big challenge in optics while the single-shot non-invasive speckle autocorrelation technique (SAT) is well-known as a promising way to handle it. However, it usually cannot recover a large-scale target or multiple isolated small ones due to the limited effective range of the optical memory effect (OME). In this paper, we propose a multi-target scattering imaging scheme by combining the traditional SA algorithm with a Deep Learning (DL) strategy. The basic idea is to extract each autocorrelation component of every target from the autocorrelation result of a mixed speckle using a suitable DL method. Once we get all the expected autocorrelation components, a typical phase retrieval algorithm (PRA) could be applied to reveal the shapes of all those corresponding small targets. In our experimental demonstration, up to five isolated targets are successfully recovered.

Deep learning for efficiently imaging through the localized speckle field of a multimode fiber

Due to the occurrence of redundant speckle, multimode fiber (MMF) imaging is extremely challenging. Our work studies the relationship between the effective feature distribution of the speckle field and the local spatial position and area, and proves that the information distribution of the speckle is highly redundant. The effective feature refers to the phase and amplitude information of the optical field carrying the image point information and the co-exciting very redundant information due to mode dispersion, interference, coupling, and entrained noise through transmission. The neural network Swin-Unet can well learn the association information between global and local features, greatly simplifies the fitting of the MMF end-to-end global mapping relationship, and achieves high-fidelity reconstruction from the local speckle field to the global image. This work will contribute to the realization of MMF real-time large-field endoscopic imaging.

Locating through dynamic scattering media based on speckle correlations

In complex imaging settings, optical scattering often prohibits the formation of a clear target image, and instead, only a speckle without the original spatial structure information is obtained. Scattering seriously interferes with the locating of targets; especially, when the scattering medium is dynamic, the dynamic nature leads to rapid decorrelation of optical information in time, and the challenge increases. Here, a locating method is proposed to detect the target hidden behind a dynamic scattering medium, which uses the a priori information of a known reference object in the neighborhood of the target. The research further designs an automatic calibration method to simplify the locating process, and analyzes the factors affecting positioning accuracy. The proposed method enables us to predict the position of a target from the autocorrelation of the captured speckle pattern; the angle and distance deviations of the target are all within 2.5%. This approach can locate a target using only a single-shot speckle pattern, and it is beneficial for target localization in dynamic scattering conditions.

Imaging through scattering media using differential intensity transmission matrices with different Hadamard orderings

A transmission matrix (TM) is a powerful tool for light focusing and imaging through scattering media. For measuring it, the normal way requires establishing a multiple-step phase-shifting interferometer, which makes the TM measurement not only complex and sensitive but also time-consuming. Imaging through scattering media using an intensity TM method can make the setup for TM measurement without the phase-shifting interferometer, thus it is much simple, more stable, and several times faster. Here, based upon a differential intensity TM method, we demonstrated it to do imaging through scattering media using different Hadamard orderings. To accelerate the TM measuring speed while degrading as little as possible of the imaging quality, a relatively reasonable strategy to plan Hadamard orderings for the TM measurement is designed since it can suggest us to preferentially measure the components in TM that are more important to the imaging quality. Thanks to the different Hadamard orderings, their influences on the imaging quality at different measuring ratios are investigated, thus an optimal measuring ordering for accelerating the TM measurement can be obtained, while only sacrificing as little as possible of the image fidelity. Simulations and experiments verify the effectiveness of the proposed method.

Thickness bound for nonlocal wide-field-of-view metalenses

Metalenses-flat lenses made with optical metasurfaces-promise to enable thinner, cheaper, and better imaging systems. Achieving a sufficient angular field of view (FOV) is crucial toward that goal and requires a tailored incident-angle-dependent response. Here, we show that there is an intrinsic trade-off between achieving a desired broad-angle response and reducing the thickness of the device. Like the memory effect in disordered media, this thickness bound originates from the Fourier transform duality between space and angle. One can write down the transmission matrix describing the desired angle-dependent response, convert it to the spatial basis where its degree of nonlocality can be quantified through a lateral spreading, and determine the minimal device thickness based on such a required lateral spreading. This approach is general. When applied to wide-FOV lenses, it predicts the minimal thickness as a function of the FOV, lens diameter, and numerical aperture. The bound is tight, as some inverse-designed multi-layer metasurfaces can approach the minimal thickness we found. This work offers guidance for the design of nonlocal metasurfaces, proposes a new framework for establishing bounds, and reveals the relation between angular diversity and spatial footprint in multi-channel systems.

Tracking moving objects through scattering media via speckle correlations

Scattering can rapidly degrade our ability to form an optical image, to the point where only speckle-like patterns can be measured. Truly non-invasive imaging through a strongly scattering obstacle is difficult, and usually reliant on a computationally intensive numerical reconstruction. In this work we show that, by combining the cross-correlations of the measured speckle pattern at different times, it is possible to track a moving object with minimal computational effort and over a large field of view.

Imaging though strongly scattering media is challenging and computationally intensive. Here, the authors show that tracking of moving objects can be achieved with minimal computational effort by combining cross-correlations of the measured speckle pattern at different times.

Non-invasive imaging through scattering medium and around corners beyond 3D memory effect

The three-dimensional (3D) memory effect (ME) has been shown to exist in a variety of scattering scenes. Limited by the scope of ME, speckle correlation technology only can be applied in a small imaging field of view (FOV) with a small depth of field (DOF). In this Letter, an untrained neural network is constructed and used as an optimization tool to restore the targets beyond the 3D ME range. The autocorrelation consistency relationship and the generative adversarial strategy are combined. Only single frame speckle and unaligned real targets are needed for online optimization; therefore, the neural network does not need to train in advance. Furthermore, the proposed method does not need to conduct additional modulation for the system. This method can reconstruct not only hidden targets behind the scattering medium, but also targets around corners. The combination strategy of the generative adversarial framework with physical priors used to decouple the aliasing information and reconstruct the target will provide inspiration for the field of computational imaging.

Physics-based neural network for non-invasive control of coherent light in scattering media

Optical imaging through complex media, such as biological tissues or fog, is challenging due to light scattering. In the multiple scattering regime, wavefront shaping provides an effective method to retrieve information; it relies on measuring how the propagation of different optical wavefronts are impacted by scattering. Based on this principle, several wavefront shaping techniques were successfully developed, but most of them are highly invasive and limited to proof-of-principle experiments. Here, we propose to use a neural network approach to non-invasively characterize and control light scattering inside the medium and also to retrieve information of hidden objects buried within it. Unlike most of the recently-proposed approaches, the architecture of our neural network with its layers, connected nodes and activation functions has a true physical meaning as it mimics the propagation of light in our optical system. It is trained with an experimentally-measured input/output dataset built from a series of incident light patterns and corresponding camera snapshots. We apply our physics-based neural network to a fluorescence microscope in epi-configuration and demonstrate its performance through numerical simulations and experiments. This flexible method can include physical priors and we show that it can be applied to other systems as, for example, non-linear or coherent contrast mechanisms.

Wavefront shaping: A versatile tool to conquer multiple scattering in multidisciplinary fields

Optical techniques offer a wide variety of applications as light-matter interactions provide extremely sensitive mechanisms to probe or treat target media. Most of these implementations rely on the usage of ballistic or quasi-ballistic photons to achieve high spatial resolution. However, the inherent scattering nature of light in biological tissues or tissue-like scattering media constitutes a critical obstacle that has restricted the penetration depth of non-scattered photons and hence limited the implementation of most optical techniques for wider applications. In addition, the components of an optical system are usually designed and manufactured for a fixed function or performance. Recent advances in wavefront shaping have demonstrated that scattering- or component-induced phase distortions can be compensated by optimizing the wavefront of the input light pattern through iteration or by conjugating the transmission matrix of the scattering medium. This offers unprecedented opportunities in many applications to achieve controllable optical delivery or detection at depths or dynamically configurable functionalities by using scattering media to substitute conventional optical components. In this article, the recent progress of wavefront shaping in multidisciplinary fields is reviewed, from optical focusing and imaging with scattering media, functionalized devices, modulation of mode coupling, and nonlinearity in multimode fiber to multimode fiber-based applications. Apart from insights into the underlying principles and recent advances in wavefront shaping implementations, practical limitations and roadmap for future development are discussed in depth. Looking back and looking forward, it is believed that wavefront shaping holds a bright future that will open new avenues for noninvasive or minimally invasive optical interactions and arbitrary control inside deep tissues. The high degree of freedom with multiple scattering will also provide unprecedented opportunities to develop novel optical devices based on a single scattering medium (generic or customized) that can outperform traditional optical components.

Weight of single and recurrent scattering in the reflection matrix of complex media

In a heterogeneous medium, the wave field can be decomposed as an infinite series known as the Born expansion. Each term of the Born expansion corresponds to a scattering order, it is thus theoretically possible to discriminate single and multiple scattering contribution to the field. Experimentally, what is actually measured is the total field in which all scattering orders interfere. Conventional imaging methods usually rely on the assumption that the multiple scattering contribution can be disregarded. In a back-scattering configuration, this assumption is valid for small depths, and begins to fail for depths larger than the scattering mean-free path ℓ_{s}. It is therefore a key issue to estimate the relative amount of single and multiple scattering in experimental data. To this end, a single-scattering estimator ρ[over ̂] computed from the reflection matrix has been introduced in order to assess the weight of single scattering in the backscattered wave field. In this paper, the meaning of this estimator is investigated and a particular attention is given to recurrent scattering. In a diffraction-limited experiment, a multiple scattering sequence is said to be recurrent if the first and last scattering events occur in the same resolution cell. Recurrent scattering is shown to be responsible for correlations between single scattering and higher scattering orders of the Born expansion, inducing a bias to the estimator ρ[over ̂] that should rather be termed confocal scattering ratio. Interestingly, a more robust estimator is built by projecting the reflection matrix in a focused basis. The argument is sustained by numerical simulations as well as ultrasonic data obtained around 1.5 MHz in a model medium made of nylon rods immersed in water. From a more general perspective, this work raises fundamental questions about the impact of recurrent scattering on wave imaging.

Remote laser-speckle sensing of heart sounds for health assessment and biometric identification

Assessment of heart sounds is the cornerstone of cardiac examination, but it requires a stethoscope, skills and experience, and a direct contact with the patient. We developed a contactless, machine-learning assisted method for heart-sound identification and quantification based on the remote measurement of the reflected laser speckle from the neck skin surface in healthy individuals. We compare the performance of this method to standard digital stethoscope recordings on an example task of heart-beat sound biometric identification. We show that our method outperforms the stethoscope even allowing identification on the test data taken on different days. This method might allow development of devices for remote monitoring of cardiovascular health in different settings.

A boundary migration model for imaging within volumetric scattering media

Effectively imaging within volumetric scattering media is of great importance and challenging especially in macroscopic applications. Recent works have demonstrated the ability to image through scattering media or within the weak volumetric scattering media using spatial distribution or temporal characteristics of the scattered field. Here, we focus on imaging Lambertian objects embedded in highly scattering media, where signal photons are dramatically attenuated during propagation and highly coupled with background photons. We address these challenges by providing a time-to-space boundary migration model (BMM) of the scattered field to convert the scattered measurements in spectral form to the scene information in the temporal domain using all of the optical signals. The experiments are conducted under two typical scattering scenarios: 2D and 3D Lambertian objects embedded in the polyethylene foam and the fog, which demonstrate the effectiveness of the proposed algorithm. It outperforms related works including time gating in terms of reconstruction precision and scattering strength. Even though the proportion of signal photons is only 0.75%, Lambertian objects located at more than 25 transport mean free paths (TMFPs), corresponding to the round-trip scattering length of more than 50 TMFPs, can be reconstructed. Also, the proposed method provides low reconstruction complexity and millisecond-scale runtime, which significantly benefits its application.

Imaging in scattering media is challenging due to signal attenuation and strong coupling of scattered and signal photons. The authors present a boundary migration model of the scattered field, converting scattered measurements in spectral form to scene information in temporal domain, and image Lambertian objects in highly scattering media.

Prior-free imaging unknown target through unknown scattering medium

Imaging through scattering medium based on deep learning has been extensively studied. However, existing methods mainly utilize paired data-prior and lack physical-process fusion, and it is difficult to reconstruct hidden targets without the trained networks. This paper proposes an unsupervised neural network that integrates the universal physical process. The reconstruction process of the network is irrelevant to the system and only requires one frame speckle pattern and unpaired targets. The proposed network enables online optimization by using physical process instead of fitting data. Thus, large-scale paired data no longer need to be obtained to train the network in advance, and the proposed method does not need prior information. The optimization of the network is a physical-based process rather than a data mapping process, and the proposed method also increases the insufficient generalization ability of the learning-based method in scattering medium and targets. The universal applicability of the proposed method to different optical systems increases the likelihood that the method will be used in practice.

Investigation of image plane for image reconstruction of objects through diffusers via deep learning

SIGNIFICANCE

The imaging of objects hidden in light-scattering media is a vital practical task in a wide range of applications, including biological imaging. Deep-learning-based methods have been used to reconstruct images behind scattering media under complex scattering conditions, but improvements in the quality of the reconstructed images are required.

AIM

To investigate the effect of image plane on the accuracy of reconstructed images.

APPROACH

Light reflected from an object passing through glass diffusers is captured by changing the image plane of an optical imaging system. Images are reconstructed by deep learning, and evaluated in terms of structural similarity index measure, classification accuracy of digital images, and training and testing error curves.

RESULTS

The reconstruction accuracy was improved for the case in which the diffuser was imaged, compared to the case where the object was imaged. The training and testing error curves show that the loss converged to lower values in fewer epochs when the diffuser was imaged.

CONCLUSIONS

The proposed approach demonstrates an improvement in the accuracy of the reconstruction of objects hidden through glass diffusers by imaging glass diffuser surfaces, and can be applied to objects at unknown locations in a scattering medium.

Explicit-restriction convolutional framework for lensless imaging

Mask-based lensless cameras break the constraints of traditional lens-based cameras, introducing highly flexible imaging systems. However, the inherent restrictions of imaging devices lead to low reconstruction quality. To overcome this challenge, we propose an explicit-restriction convolutional framework for lensless imaging, whose forward model effectively incorporates multiple restrictions by introducing the linear and noise-like nonlinear terms. As examples, numerical and experimental reconstructions based on the limitation of sensor size, pixel pitch, and bit depth are analyzed. By tailoring our framework for specific factors, better perceptual image quality or reconstructions with 4× pixel density can be achieved. This proposed framework can be extended to lensless imaging systems with different masks or structures.