Deep learning in holography and coherent imaging

Artificial intelligence in bacterial diagnostics and antimicrobial susceptibility testing: Current advances and future prospects

Recently, artificial intelligence (AI) has emerged as a transformative tool, enhancing the speed, accuracy, and scalability of bacterial diagnostics. This review explores the role of AI in revolutionizing bacterial detection and antimicrobial susceptibility testing (AST) by leveraging machine learning models, including Random Forest, Support Vector Machines (SVM), and deep learning architectures such as Convolutional Neural Networks (CNNs) and transformers. The integration of AI into these methods promises to address the current limitations of traditional techniques, offering a path toward more efficient, accessible, and reliable diagnostic solutions. In particular, AI-based approaches have demonstrated significant potential in resource-limited settings by enabling cost-effective and portable diagnostic solutions, reducing dependency on specialized infrastructure, and facilitating remote bacterial detection through smartphone-integrated platforms and telemedicine applications. This review highlights AI's transformative role in automating data analysis, minimizing human error, and delivering real-time diagnostic results, ultimately improving patient outcomes and optimizing healthcare efficiency. In addition, we not only examine the current advances in machine learning and deep learning but also review their applications in plate counting, mass spectrometry, morphology-based and motion-based microscopic detection, holographic microscopy, colorimetric and fluorescence detection, electrochemical sensors, Raman and Surface-Enhanced Raman Spectroscopy (SERS), and Atomic Force Microscopy (AFM) for bacterial diagnostics and AST. Finally, we discuss the future directions and potential advancements in AI-driven bacterial diagnostics.

Diatom Lensless Imaging Using Laser Scattering and Deep Learning

We present a novel approach for imaging diatoms using lensless imaging and deep learning. We used a laser beam to scatter off samples of diatomaceous earth (diatoms) and then recorded and transformed the scattered light into microscopy images of the diatoms. The predicted microscopy images gave an average SSIM of 0.98 and an average RMSE of 3.26 as compared to the experimental data. We also demonstrate the capability of determining the velocity and angle of movement of the diatoms from their scattering patterns as they were translated through the laser beam. This work shows the potential for imaging and identifying the movement of diatoms and other microsized organisms in situ within the marine environment. Implementing such a method for real-time image acquisition and analysis could enhance environmental management, including improving the early detection of harmful algal blooms.

Miniaturized High-Throughput and High-Resolution Platform for Continuous Live-Cell Monitoring via Lens-Free Imaging and Deep Learning

Monitoring the morphology and dynamics of both individual and collective cells is crucial for understanding the complexities of biological systems, investigating disease mechanisms, and advancing therapeutic strategies. However, traditional live-cell workstations that rely on microscopy often face inherent trade-offs between field of view (FOV) and resolution, making it difficult to achieve both high-throughput and high-resolution monitoring simultaneously. While existing lens-free imaging technologies enable high-throughput cell monitoring, they are often hindered by algorithmic complexity, long processing times that prevent real-time imaging, or insufficient resolution due to large sensor pixel sizes. To overcome these limitations, here an imaging platform is presented that integrates a custom-developed 500 nm pixel-size, 400-megapixel sensor with lens-free shadow imaging technology. This platform is capable of achieving imaging at a speed of up to 40s per frame, with a large FOV of 1 cm2 and an imaging signal-to-noise ratio (SNR) of 42 dB, enabling continuous tracking of individual and cell populations throughout their entire lifecycle. By leveraging deep learning algorithms, the system accurately analyzes cell movement trajectories, while the integration of a K-means unsupervised clustering algorithm ensures precise evaluation of cellular activity. This platform provides an effective solution for high-throughput live-cell morphology monitoring and dynamic analysis.

Lens-Free Holographic Imaging-Based Immunosensor Using Unpaired Data Set Signal-to-Noise Ratio-Enhanced Modal Transformation for Biosensing

Conventional microscopes have limited capacities to reconcile the trade-off between the lens and field of view (FOV). Thus, the imaging field and accuracy of immunosensors remain restricted. In this study, a holographic deep learning unpaired modal transformation-assisted immunosensor is presented, combining a portable lens-free holographic imaging device with a CuO2@SiO2 nanoparticle-based click reaction signal amplification strategy for accurate antibiotic detection. The immunosensor achieves both large FOV imaging (10.3-fold improvement over the microscope) and signal-to-noise ratio-enhanced holographic reconstruction (signal-to-noise ratio of 32.65 dB, structural similarity index of 0.83) by constructing a modal transformation model with unpaired data sets, thus resolving the complexity of one-to-one matching of data sets required by conventional methods. The immunosensor detects chloramphenicol with high sensitivity and a wide linear range (limit of detection = 3.54 pg/mL, dynamic range of 10 pg/mL to 50 ng/mL) within 40 min. As a portable detection device, it demonstrates potential as a sensitive and on-site detection platform for food safety inspection and clinical diagnosis.

Fourier-inspired single-pixel holography

Fourier-inspired single-pixel holography (FISH) is an effective digital holography (DH) approach that utilizes a single-pixel detector instead of a conventional camera to capture light field information. FISH combines the Fourier single-pixel imaging and off-axis holography technique, allowing one to acquire useful information directly, rather than recording the hologram in the spatial domain and filtering unwanted terms in the Fourier domain. Furthermore, we employ a deep learning technique to jointly optimize the sampling mask and the imaging enhancement model, to achieve high-quality results at a low sampling ratio. Both simulations and experimental results demonstrate the effectiveness of FISH in single-pixel phase imaging. FISH combines the strengths of single-pixel imaging (SPI) and DH, potentially expanding DH's applications to specialized spectral bands and low-light environments while equipping SPI with capabilities for phase detection and coherent gating.

Real-time holographic camera for obtaining real 3D scene hologram

As a frontier technology, holography has important research values in fields such as bio-micrographic imaging, light field modulation and data storage. However, the real-time acquisition of 3D scenes and high-fidelity reconstruction technology has not yet made a breakthrough, which has seriously hindered the development of holography. Here, a novel holographic camera is proposed to solve the above inherent problems completely. The proposed holographic camera consists of the acquisition end and the calculation end. At the acquisition end of the holographic camera, specially configured liquid materials and liquid lens structure based on voice-coil motor-driving are used to produce the liquid camera, so that the liquid camera can quickly capture the focus stack of the real 3D scene within 15 ms. At the calculation end, a new structured focus stack network (FS-Net) is designed for hologram calculation. After training the FS-Net with the focus stack renderer and learnable Zernike phase, it enables hologram calculation within 13 ms. As the first device to achieve real-time incoherent acquisition and high-fidelity holographic reconstruction of a real 3D scene, our proposed holographic camera breaks technical bottlenecks of difficulty in acquiring the real 3D scene, low quality of the holographic reconstructed image, and incorrect defocus blur. The experimental results demonstrate the effectiveness of our holographic camera in the acquisition of focal plane information and hologram calculation of the real 3D scene. The proposed holographic camera opens up a new way for the application of holography in fields such as 3D display, light field modulation, and 3D measurement.

Deep learning-enabled filter-free fluorescence microscope

Optical filtering is an indispensable part of fluorescence microscopy for selectively highlighting molecules labeled with a specific fluorophore and suppressing background noise. However, the utilization of optical filtering sets increases the complexity, size, and cost of microscopic systems, making them less suitable for multifluorescence channel, high-speed imaging. Here, we present filter-free fluorescence microscopic imaging enabled with deep learning-based digital spectral filtering. This approach allows for automatic fluorescence channel selection after image acquisition and accurate prediction of fluorescence by computing color changes due to spectral shifts with the presence of excitation scattering. Fluorescence prediction for cells and tissues labeled with various fluorophores was demonstrated under different magnification powers. The technique offers accurate identification of labeling with robust sensitivity and specificity, achieving consistent results with the reference standard. Beyond fluorescence microscopy, the deep learning-enabled spectral filtering strategy has the potential to drive the development of other biomedical applications, including cytometry and endoscopy.

Unsupervised Learning-Assisted Acoustic-Driven Nano-Lens Holography for the Ultrasensitive and Amplification-Free Detection of Viable Bacteria

Bacterial infection is a crucial factor resulting in public health issues worldwide, often triggering epidemics and even fatalities. The accurate, rapid, and convenient detection of viable bacteria is an effective method for reducing infections and illness outbreaks. Here, an unsupervised learning-assisted and surface acoustic wave-interdigital transducer-driven nano-lens holography biosensing platform is developed for the ultrasensitive and amplification-free detection of viable bacteria. The monitoring device integrated with the nano-lens effect can achieve the holographic imaging of polystyrene microsphere probes in an ultra-wide field of view (∽28.28 mm2), with a sensitivity limit of as low as 99 nm. A lightweight unsupervised learning hologram processing algorithm considerably reduces training time and computing hardware requirements, without requiring datasets with manual labels. By combining phage-mediated viable bacterial DNA extraction and enhanced CRISPR-Cas12a systems, this strategy successfully achieves the ultrasensitive detection of viable Salmonella in various real samples, demonstrating enhanced accuracy validated with the qPCR benchmark method. This approach has a low cost (∽$500) and is rapid (∽1 h) and highly sensitive (∽38 CFU mL-1), allowing for the amplification-free detection of viable bacteria and emerging as a powerful tool for food safety inspection and clinical diagnosis.

Generalizable deep learning approach for 3D particle imaging using holographic microscopy (HM)

Despite its potential for label-free particle diagnostics, holographic microscopy is limited by specialized processing methods that struggle to generalize across diverse settings. We introduce a deep learning architecture leveraging human perception of longitudinal variation of diffracted patterns of particles, which enables highly generalizable analysis of 3D particle information with orders of magnitude improvement in processing speed. Trained with minimal synthetic and real holograms of simple particles, our method demonstrates exceptional performance across various challenging cases, including high particle concentrations, significant noise, and a wide range of particle sizes, complex shapes, and optical properties, exceeding the diversity of training datasets.

Tilted-Mode All-Optical Diffractive Deep Neural Networks

Diffractive deep neural networks (D2NNs) typically adopt a densely cascaded arrangement of diffractive masks, leading to multiple reflections of diffracted light between adjacent masks, thereby affecting the network's inference capability. It is challenging to fully simulate this multiple-reflection phenomenon. To eliminate this phenomenon, we designed tilted-mode all-optical diffractive deep neural networks (T-D2NNs) and proposed a theoretical model for diffraction propagation in the tilted mode. Simulation results indicate that T-D2NNs address the performance degradation caused by interlayer reflections in D2NNs constructed with high-index diffractive masks. In classification tasks, T-D2NNs achieve better classification results compared to D2NNs that consider interlayer reflections.

Roadmap on computational methods in optical imaging and holography [invited]

Computational methods have been established as cornerstones in optical imaging and holography in recent years. Every year, the dependence of optical imaging and holography on computational methods is increasing significantly to the extent that optical methods and components are being completely and efficiently replaced with computational methods at low cost. This roadmap reviews the current scenario in four major areas namely incoherent digital holography, quantitative phase imaging, imaging through scattering layers, and super-resolution imaging. In addition to registering the perspectives of the modern-day architects of the above research areas, the roadmap also reports some of the latest studies on the topic. Computational codes and pseudocodes are presented for computational methods in a plug-and-play fashion for readers to not only read and understand but also practice the latest algorithms with their data. We believe that this roadmap will be a valuable tool for analyzing the current trends in computational methods to predict and prepare the future of computational methods in optical imaging and holography.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00340-024-08280-3.

Generating Multi-Depth 3D Holograms Using a Fully Convolutional Neural Network

Efficiently generating 3D holograms is one of the most challenging research topics in the field of holography. This work introduces a method for generating multi-depth phase-only holograms using a fully convolutional neural network (FCN). The method primarily involves a forward-backward-diffraction framework to compute multi-depth diffraction fields, along with a layer-by-layer replacement method (L2RM) to handle occlusion relationships. The diffraction fields computed by the former are fed into the carefully designed FCN, which leverages its powerful non-linear fitting capability to generate multi-depth holograms of 3D scenes. The latter can smooth the boundaries of different layers in scene reconstruction by complementing information of occluded objects, thus enhancing the reconstruction quality of holograms. The proposed method can generate a multi-depth 3D hologram with a PSNR of 31.8 dB in just 90 ms for a resolution of 2160 × 3840 on the NVIDIA Tesla A100 40G tensor core GPU. Additionally, numerical and experimental results indicate that the generated holograms accurately reconstruct clear 3D scenes with correct occlusion relationships and provide excellent depth focusing.

Quantitative phase imaging based on holography: trends and new perspectives

In 1948, Dennis Gabor proposed the concept of holography, providing a pioneering solution to a quantitative description of the optical wavefront. After 75 years of development, holographic imaging has become a powerful tool for optical wavefront measurement and quantitative phase imaging. The emergence of this technology has given fresh energy to physics, biology, and materials science. Digital holography (DH) possesses the quantitative advantages of wide-field, non-contact, precise, and dynamic measurement capability for complex-waves. DH has unique capabilities for the propagation of optical fields by measuring light scattering with phase information. It offers quantitative visualization of the refractive index and thickness distribution of weak absorption samples, which plays a vital role in the pathophysiology of various diseases and the characterization of various materials. It provides a possibility to bridge the gap between the imaging and scattering disciplines. The propagation of wavefront is described by the complex amplitude. The complex-value in the complex-domain is reconstructed from the intensity-value measurement by camera in the real-domain. Here, we regard the process of holographic recording and reconstruction as a transformation between complex-domain and real-domain, and discuss the mathematics and physical principles of reconstruction. We review the DH in underlying principles, technical approaches, and the breadth of applications. We conclude with emerging challenges and opportunities based on combining holographic imaging with other methodologies that expand the scope and utility of holographic imaging even further. The multidisciplinary nature brings technology and application experts together in label-free cell biology, analytical chemistry, clinical sciences, wavefront sensing, and semiconductor production.

Off-axis digital lensless holographic microscopy based on spatially multiplexed interferometry

Significance

Digital holographic microscopy (DHM) is a label-free microscopy technique that provides time-resolved quantitative phase imaging (QPI) by measuring the optical path delay of light induced by transparent biological samples. DHM has been utilized for various biomedical applications, such as cancer research and sperm cell assessment, as well as for in vitro drug or toxicity testing. Its lensless version, digital lensless holographic microscopy (DLHM), is an emerging technology that offers size-reduced, lightweight, and cost-effective imaging systems. These features make DLHM applicable, for example, in limited resource laboratories, remote areas, and point-of-care applications.

Aim

In addition to the abovementioned advantages, in-line arrangements for DLHM also include the limitation of the twin-image presence, which can restrict accurate QPI. We therefore propose a compact lensless common-path interferometric off-axis approach that is capable of quantitative imaging of fast-moving biological specimens, such as living cells in flow.

Approach

We suggest lensless spatially multiplexed interferometric microscopy (LESSMIM) as a lens-free variant of the previously reported spatially multiplexed interferometric microscopy (SMIM) concept. LESSMIM comprises a common-path interferometric architecture that is based on a single diffraction grating to achieve digital off-axis holography. From a series of single-shot off-axis holograms, twin-image free and time-resolved QPI is achieved by commonly used methods for Fourier filtering-based reconstruction, aberration compensation, and numerical propagation.

Results

Initially, the LESSMIM concept is experimentally demonstrated by results from a resolution test chart and investigations on temporal stability. Then, the accuracy of QPI and capabilities for imaging of living adherent cell cultures is characterized. Finally, utilizing a microfluidic channel, the cytometry of suspended cells in flow is evaluated.

Conclusions

LESSMIM overcomes several limitations of in-line DLHM and provides fast time-resolved QPI in a compact optical arrangement. In summary, LESSMIM represents a promising technique with potential biomedical applications for fast imaging such as in imaging flow cytometry or sperm cell analysis.

Quantitative phase imaging techniques for measuring scattering properties of cells and tissues: a review-part I

Significance

Quantitative phase imaging (QPI) techniques offer intrinsic information about the sample of interest in a label-free, noninvasive manner and have an enormous potential for wide biomedical applications with negligible perturbations to the natural state of the sample in vitro.

Aim

We aim to present an in-depth review of the scattering formulation of light-matter interactions as applied to biological samples such as cells and tissues, discuss the relevant quantitative phase measurement techniques, and present a summary of various reported applications.

Approach

We start with scattering theory and scattering properties of biological samples followed by an exploration of various microscopy configurations for 2D QPI for measurement of structure and dynamics.

Results

We reviewed 157 publications and presented a range of QPI techniques and discussed suitable applications for each. We also presented the theoretical frameworks for phase reconstruction associated with the discussed techniques and highlighted their domains of validity.

Conclusions

We provide detailed theoretical as well as system-level information for a wide range of QPI techniques. Our study can serve as a guideline for new researchers looking for an exhaustive literature review of QPI methods and relevant applications.

All-optical complex field imaging using diffractive processors

Complex field imaging, which captures both the amplitude and phase information of input optical fields or objects, can offer rich structural insights into samples, such as their absorption and refractive index distributions. However, conventional image sensors are intensity-based and inherently lack the capability to directly measure the phase distribution of a field. This limitation can be overcome using interferometric or holographic methods, often supplemented by iterative phase retrieval algorithms, leading to a considerable increase in hardware complexity and computational demand. Here, we present a complex field imager design that enables snapshot imaging of both the amplitude and quantitative phase information of input fields using an intensity-based sensor array without any digital processing. Our design utilizes successive deep learning-optimized diffractive surfaces that are structured to collectively modulate the input complex field, forming two independent imaging channels that perform amplitude-to-amplitude and phase-to-intensity transformations between the input and output planes within a compact optical design, axially spanning ~100 wavelengths. The intensity distributions of the output fields at these two channels on the sensor plane directly correspond to the amplitude and quantitative phase profiles of the input complex field, eliminating the need for any digital image reconstruction algorithms. We experimentally validated the efficacy of our complex field diffractive imager designs through 3D-printed prototypes operating at the terahertz spectrum, with the output amplitude and phase channel images closely aligning with our numerical simulations. We envision that this complex field imager will have various applications in security, biomedical imaging, sensing and material science, among others.

Label-Free Multiplex Profiling of Exosomal Proteins with a Deep Learning-Driven 3D Surround-Enhancing SERS Platform for Early Cancer Diagnosis

Identification of protein profiling on plasma exosomes by SERS can be a promising strategy for early cancer diagnosis. However, it is still challenging to detect multiple exosomal proteins simultaneously by SERS since the Raman signals of exosomes detected by conventional colloidal nanocrystals or two-dimensional SERS substrates are incomplete and complex. Herein, we develop a novel three-dimensional (3D) surround-enhancing SERS platform, named 3D se-SERS, for the multiplex detection of exosomal proteins. In this 3D se-SERS, proteins and exosomes are covered with "hotspots" generated by the gold nanoparticles, which surround the analytes densely and three-dimensionally, providing sensitive and comprehensive SERS signals. Combining this 3D se-SERS with a deep learning model, we successfully quantitatively profiled seven proteins including CD63, CD81, CD9, CD151, CD171, TSPAN8, and PD-L1 on the surface of plasma exosomes from patients, which can predict the occurrence and advancement of lung cancer. This 3D se-SERS integrating deep learning technique benefits from high sensitivity and significant multiplexing ability for comprehensive analysis of proteins and exosomes, demonstrating the potential of deep learning-driven 3D se-SERS technology for plasma exosome-based early cancer diagnosis.

A Holographic-Type Model in the Description of Polymer-Drug Delivery Processes

A unitary model of drug release dynamics is proposed, assuming that the polymer-drug system can be assimilated into a multifractal mathematical object. Then, we made a description of drug release dynamics that implies, via Scale Relativity Theory, the functionality of continuous and undifferentiable curves (fractal or multifractal curves), possibly leading to holographic-like behaviors. At such a conjuncture, the Schrödinger and Madelung multifractal scenarios become compatible: in the Schrödinger multifractal scenario, various modes of drug release can be "mimicked" (via period doubling, damped oscillations, modulated and "chaotic" regimes), while the Madelung multifractal scenario involves multifractal diffusion laws (Fickian and non-Fickian diffusions). In conclusion, we propose a unitary model for describing release dynamics in polymer-drug systems. In the model proposed, the polymer-drug dynamics can be described by employing the Scale Relativity Theory in the monofractal case or also in the multifractal one.

ICF-PR-Net: a deep phase retrieval neural network for X-ray phase contrast imaging of inertial confinement fusion capsules

X-ray phase contrast imaging (XPCI) has demonstrated capability to characterize inertial confinement fusion (ICF) capsules, and phase retrieval can reconstruct phase information from intensity images. This study introduces ICF-PR-Net, a novel deep learning-based phase retrieval method for ICF-XPCI. We numerically constructed datasets based on ICF capsule shape features, and proposed an object-image loss function to add image formation physics to network training. ICF-PR-Net outperformed traditional methods as it exhibited satisfactory robustness against strong noise and nonuniform background and was well-suited for ICF-XPCI's constrained experimental conditions and single exposure limit. Numerical and experimental results showed that ICF-PR-Net accurately retrieved the phase and absorption while maintaining retrieval quality in different situations. Overall, the ICF-PR-Net enables the diagnosis of the inner interface and electron density of capsules to address ignition-preventing problems, such as hydrodynamic instability growth.

EVATOM: an optical, label-free, machine learning assisted embryo health assessment tool

The combination of a good quality embryo and proper maternal health factors promise higher chances of a successful in vitro fertilization (IVF) procedure leading to clinical pregnancy and live birth. Of these two factors, selection of a good embryo is a controllable aspect. The current gold standard in clinical practice is visual assessment of an embryo based on its morphological appearance by trained embryologists. More recently, machine learning has been incorporated into embryo selection "packages". Here, we report EVATOM: a machine-learning assisted embryo health assessment tool utilizing an optical quantitative phase imaging technique called artificial confocal microscopy (ACM). We present a label-free nucleus detection method with, to the best of our knowledge, novel quantitative embryo health biomarkers. Two viability assessment models are presented for grading embryos into two classes: healthy/intermediate (H/I) or sick (S) class. The models achieve a weighted F1 score of 1.0 and 0.99 respectively on the in-distribution test set of 72 fixed embryos and a weighted F1 score of 0.9 and 0.95 respectively on the out-of-distribution test dataset of 19 time-instances from 8 live embryos.

Dual-constrained physics-enhanced untrained neural network for lensless imaging

An untrained neural network (UNN) paves a new way to realize lensless imaging from single-frame intensity data. Based on the physics engine, such methods utilize the smoothness property of a convolutional kernel and provide an iterative self-supervised learning framework to release the needs of an end-to-end training scheme with a large dataset. However, the intrinsic overfitting problem of UNN is a challenging issue for stable and robust reconstruction. To address it, we model the phase retrieval problem into a dual-constrained untrained network, in which a phase-amplitude alternating optimization framework is designed to split the intensity-to-phase problem into two tasks: phase and amplitude optimization. In the process of phase optimization, we combine a deep image prior with a total variation prior to retrain the loss function for the phase update. In the process of amplitude optimization, a total variation denoising-based Wirtinger gradient descent method is constructed to form an amplitude constraint. Alternative iterations of the two tasks result in high-performance wavefield reconstruction. Experimental results demonstrate the superiority of our method.

Physics-driven universal twin-image removal network for digital in-line holographic microscopy

Digital in-line holographic microscopy (DIHM) enables efficient and cost-effective computational quantitative phase imaging with a large field of view, making it valuable for studying cell motility, migration, and bio-microfluidics. However, the quality of DIHM reconstructions is compromised by twin-image noise, posing a significant challenge. Conventional methods for mitigating this noise involve complex hardware setups or time-consuming algorithms with often limited effectiveness. In this work, we propose UTIRnet, a deep learning solution for fast, robust, and universally applicable twin-image suppression, trained exclusively on numerically generated datasets. The availability of open-source UTIRnet codes facilitates its implementation in various DIHM systems without the need for extensive experimental training data. Notably, our network ensures the consistency of reconstruction results with input holograms, imparting a physics-based foundation and enhancing reliability compared to conventional deep learning approaches. Experimental verification was conducted among others on live neural glial cell culture migration sensing, which is crucial for neurodegenerative disease research.

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

Optical tomography in a single camera frame using fringe-encoded deep-learning full-field OCT

Optical coherence tomography is a valuable tool for in vivo examination thanks to its superior combination of axial resolution, field-of-view and working distance. OCT images are reconstructed from several phases that are obtained by modulation/multiplexing of light wavelength or optical path. This paper shows that only one phase (and one camera frame) is sufficient for en face tomography. The idea is to encode a high-frequency fringe patterns into the selected layer of the sample using low-coherence interferometry. These patterns can then be efficiently extracted with a high-pass filter enhanced via deep learning networks to create the tomographic full-field OCT view. This brings 10-fold improvement in imaging speed, considerably reducing the phase errors and incoherent light artifacts related to in vivo movements. Moreover, this work opens a path for low-cost tomography with slow consumer cameras. Optically, the device resembles the conventional time-domain full-field OCT without incurring additional costs or a field-of-view/resolution reduction. The approach is validated by imaging in vivo cornea in human subjects. Open-source and easy-to-follow codes for data generation/training/inference with U-Net/Pix2Pix networks are provided to be used in a variety of image-to-image translation tasks.

Six-pack holography for dynamic profiling of thick and extended objects by simultaneous three-wavelength phase unwrapping with doubled field of view

Dynamic holographic profiling of thick samples is limited due to the reduced field of view (FOV) of off-axis holography. We present an improved six-pack holography system for the simultaneous acquisition of six complex wavefronts in a single camera exposure from two fields of view (FOVs) and three wavelengths, for quantitative phase unwrapping of thick and extended transparent objects. By dynamically generating three synthetic wavelength quantitative phase maps for each of the two FOVs, with the longest wavelength being 6207 nm, hierarchical phase unwrapping can be used to reduce noise while maintaining the improvements in the 2π phase ambiguity due to the longer synthetic wavelength. The system was tested on a 7 μm tall PDMS microchannel and is shown to produce quantitative phase maps with 96% accuracy, while the hierarchical unwrapping reduces noise by 93%. A monolayer of live onion epidermal tissue was also successfully scanned, demonstrating the potential of the system to dynamically decrease scanning time of optically thick and extended samples.

High-fidelity and high-robustness free-space ghost transmission in complex media with coherent light source using physics-driven untrained neural network

It is well recognized that it is challenging to realize high-fidelity and high-robustness ghost transmission through complex media in free space using coherent light source. In this paper, we report a new method to realize high-fidelity and high-robustness ghost transmission through complex media by generating random amplitude-only patterns as 2D information carriers using physics-driven untrained neural network (UNN). The random patterns are generated to encode analog signals (i.e., ghost) without any training datasets and labeled data, and are used as information carriers in a free-space optical channel. Coherent light source modulated by the random patterns propagates through complex media, and a single-pixel detector is utilized to collect light intensities at the receiving end. A series of optical experiments have been conducted to verify the proposed approach. Experimental results demonstrate that the proposed method can realize high-fidelity and high-robustness analog-signal (ghost) transmission in complex environments, e.g., around a corner, or dynamic and turbid water. The proposed approach using the designed physics-driven UNN could open an avenue for high-fidelity free-space ghost transmission through complex media.

Machine learning assisted health viability assay for mouse embryos with artificial confocal microscopy (ACM)

The combination of a good quality embryo and proper maternal health factors promise higher chances of a successful in vitro fertilization (IVF) procedure leading to clinical pregnancy and live birth. Of these two factors, selection of a good embryo is a controllable aspect. The current gold standard in clinical practice is visual assessment of an embryo based on its morphological appearance by trained embryologists. More recently, machine learning has been incorporated into embryo selection "packages". Here, we report a machine-learning assisted embryo health assessment tool utilizing a quantitative phase imaging technique called artificial confocal microscopy (ACM). We present a label-free nucleus detection method with novel quantitative embryo health biomarkers. Two viability assessment models are presented for grading embryos into two classes: healthy/intermediate (H/I) or sick (S) class. The models achieve a weighted F1 score of 1.0 and 0.99 respectively on the in-distribution test set of 72 fixed embryos and a weighted F1 score of 0.9 and 0.95 respectively on the out-of-distribution test dataset of 19 time-instances from 8 live embryos.

Color multilayer holographic near-eye augmented reality display

This study demonstrates a full-color near-eye holographic display capable of superimposing color virtual scenes with 2D, 3D, and multiple objects with extended depth upon a real scene, which also has the ability to present different 3D information depending on the focus of the user's eyes using a single computer-generated hologram per color channel. Our setup makes use of a hologram generation method based on two-step propagation and the singular value decomposition of the Fresnel transform impulse response function to efficiently generate the holograms of the target scene. Then, we test our proposal by implementing a holographic display that makes use of a phase-only spatial light modulator and time-division multiplexing for color reproduction. We demonstrate the superior quality and computation speed of this approach compared with other hologram generation techniques with both numerical and experimental results.

On-chip digital holographic interferometry for measuring wavefront deformation in transparent samples

This paper describes on-chip digital holographic interferometry for measuring the wavefront deformation of transparent samples. The interferometer is based on a Mach-Zehnder arrangement with a waveguide in the reference arm, which allows for a compact on-chip arrangement. The method thus exploits the sensitivity of digital holographic interferometry and the advantages of the on-chip approach, which provides high spatial resolution over a large area, simplicity, and compactness of the system. The method's performance is demonstrated by measuring a model glass sample fabricated by depositing SiO₂ layers of different thicknesses on a planar glass substrate and visualizing the domain structure in periodically poled lithium niobate. Finally, the results of the measurement made with the on-chip digital holographic interferometer were compared with those made with a conventional Mach-Zehnder type digital holographic interferometer with lens and with a commercial white light interferometer. The comparison of the obtained results indicates that the on-chip digital holographic interferometer provides accuracy comparable to conventional methods while offering the benefits of a large field of view and simplicity.

JPEG Pleno holography presents the numerical reconstruction software for holograms: an excursion in holographic views

Digital reconstructions of numerical holograms enable data visualization and serve a multitude of purposes ranging from microscopy to holographic displays. Over the years, many pipelines have been developed for specific hologram types. Within the standardization effort of JPEG Pleno holography, an open-source MATLAB toolbox was developed that reflects the best current consensus. It can process Fresnel, angular spectrum, and Fourier-Fresnel holograms with one or more color channels; it also allows for diffraction-limited numerical reconstructions. The latter provides a way to reconstruct holograms at their intrinsic physical instead of an arbitrarily chosen numerical resolution. The Numerical Reconstruction Software for Holograms v10 supports all large public data sets featured by UBI, BCOM, ETRI, and ETRO, in their native and vertical off-axis binary forms. Through the release of this software, we hope to improve the reproducibility of research, thus enabling consistent comparison of data between research groups and the quality of specific numerical reconstructions.

Diffractive interconnects: all-optical permutation operation using diffractive networks

Permutation matrices form an important computational building block frequently used in various fields including, e.g., communications, information security, and data processing. Optical implementation of permutation operators with relatively large number of input-output interconnections based on power-efficient, fast, and compact platforms is highly desirable. Here, we present diffractive optical networks engineered through deep learning to all-optically perform permutation operations that can scale to hundreds of thousands of interconnections between an input and an output field-of-view using passive transmissive layers that are individually structured at the wavelength scale. Our findings indicate that the capacity of the diffractive optical network in approximating a given permutation operation increases proportional to the number of diffractive layers and trainable transmission elements in the system. Such deeper diffractive network designs can pose practical challenges in terms of physical alignment and output diffraction efficiency of the system. We addressed these challenges by designing misalignment tolerant diffractive designs that can all-optically perform arbitrarily selected permutation operations, and experimentally demonstrated, for the first time, a diffractive permutation network that operates at THz part of the spectrum. Diffractive permutation networks might find various applications in, e.g., security, image encryption, and data processing, along with telecommunications; especially with the carrier frequencies in wireless communications approaching THz-bands, the presented diffractive permutation networks can potentially serve as channel routing and interconnection panels in wireless networks.

Multi-Illumination Single-Holographic-Exposure Lensless Fresnel (MISHELF) Microscopy: Principles and Biomedical Applications

Lensless holographic microscopy (LHM) comes out as a promising label-free technique since it supplies high-quality imaging and adaptive magnification in a lens-free, compact and cost-effective way. Compact sizes and reduced prices of LHMs make them a perfect instrument for point-of-care diagnosis and increase their usability in limited-resource laboratories, remote areas, and poor countries. LHM can provide excellent intensity and phase imaging when the twin image is removed. In that sense, multi-illumination single-holographic-exposure lensless Fresnel (MISHELF) microscopy appears as a single-shot and phase-retrieved imaging technique employing multiple illumination/detection channels and a fast-iterative phase-retrieval algorithm. In this contribution, we review MISHELF microscopy through the description of the principles, the analysis of the performance, the presentation of the microscope prototypes and the inclusion of the main biomedical applications reported so far.

Portable, Automated and Deep-Learning-Enabled Microscopy for Smartphone-Tethered Optical Platform Towards Remote Homecare Diagnostics: A Review

Globally new pandemic diseases induce urgent demands for portable diagnostic systems to prevent and control infectious diseases. Smartphone-based portable diagnostic devices are significantly efficient tools to user-friendly connect personalized health conditions and collect valuable optical information for rapid diagnosis and biomedical research through at-home screening. Deep learning algorithms for portable microscopes also help to enhance diagnostic accuracy by reducing the imaging resolution gap between benchtop and portable microscopes. This review highlighted recent progress and continued efforts in a smartphone-tethered optical platform through portable, automated, and deep-learning-enabled microscopy for personalized diagnostics and remote monitoring. In detail, the optical platforms through smartphone-based microscopes and lens-free holographic microscopy are introduced, and deep learning-based portable microscopic imaging is explained to improve the image resolution and accuracy of diagnostics. The challenges and prospects of portable optical systems with microfluidic channels and a compact microscope to screen COVID-19 in the current pandemic are also discussed. It has been believed that this review offers a novel guide for rapid diagnosis, biomedical imaging, and digital healthcare with low cost and portability.

Smartphone-based holographic measurement of polydisperse suspended particulate matter with various mass concentration ratios

Real-time monitoring of suspended particulate matter (PM) has become essential in daily life due to the adverse effects of long-term exposure to PMs on human health and ecosystems. However, conventional techniques for measuring micro-scale particulates commonly require expensive instruments. In this study, a smartphone-based device is developed for real-time monitoring of suspended PMs by integrating a smartphone-based digital holographic microscopy (S-DHM) and deep learning algorithms. The proposed S-DHM-based PM monitoring device is composed of affordable commercial optical components and a smartphone. Overall procedures including digital image processing, deep learning training, and correction process are optimized to minimize the prediction error and computational cost. The proposed device can rapidly measure the mass concentrations of coarse and fine PMs from holographic speckle patterns of suspended polydisperse PMs in water with measurement errors of 22.8 ± 18.1% and 13.5 ± 9.8%, respectively. With further advances in data acquisition and deep learning training, this study would contribute to the development of hand-held devices for monitoring polydisperse non-spherical pollutants suspended in various media.

Virtual Impactor-Based Label-Free Pollen Detection using Holography and Deep Learning

Exposure to bio-aerosols such as pollen can lead to adverse health effects. There is a need for a portable and cost-effective device for long-term monitoring and quantification of various types of pollen. To address this need, we present a mobile and cost-effective label-free sensor that takes holographic images of flowing particulate matter (PM) concentrated by a virtual impactor, which selectively slows down and guides particles larger than 6 μm to fly through an imaging window. The flowing particles are illuminated by a pulsed laser diode, casting their inline holograms on a complementary metal-oxide semiconductor image sensor in a lens-free mobile imaging device. The illumination contains three short pulses with a negligible shift of the flowing particle within one pulse, and triplicate holograms of the same particle are recorded at a single frame before it exits the imaging field-of-view, revealing different perspectives of each particle. The particles within the virtual impactor are localized through a differential detection scheme, and a deep neural network classifies the pollen type in a label-free manner based on the acquired holographic images. We demonstrated the success of this mobile pollen detector with a virtual impactor using different types of pollen (i.e., bermuda, elm, oak, pine, sycamore, and wheat) and achieved a blind classification accuracy of 92.91%. This mobile and cost-effective device weighs ∼700 g and can be used for label-free sensing and quantification of various bio-aerosols over extended periods since it is based on a cartridge-free virtual impactor that does not capture or immobilize PM.

Experimental optimization of lensless digital holographic microscopy with rotating diffuser-based coherent noise reduction

Laser-based lensless digital holographic microscopy (LDHM) is often spoiled by considerable coherent noise factor. We propose a novel LDHM method with significantly limited coherent artifacts, e.g., speckle noise and parasitic interference fringes. It is achieved by incorporating a rotating diffuser, which introduces partial spatial coherence and preserves high temporal coherence of laser light, crucial for credible in-line hologram reconstruction. We present the first implementation of the classical rotating diffuser concept in LDHM, significantly increasing the signal-to-noise ratio while preserving the straightforwardness and compactness of the LDHM imaging device. Prior to the introduction of the rotating diffusor, we performed LDHM experimental hardware optimization employing 4 light sources, 4 cameras, and 3 different optical magnifications (camera-sample distances). It was guided by the quantitative assessment of numerical amplitude/phase reconstruction of test targets, conducted upon standard deviation calculation (noise factor quantification), and resolution evaluation (information throughput quantification). Optimized rotating diffuser LDHM (RD-LDHM) method was successfully corroborated in technical test target imaging and examination of challenging biomedical sample (60 µm thick mouse brain tissue slice). Physical minimization of coherent noise (up to 50%) was positively verified, while preserving optimal spatial resolution of phase and amplitude imaging. Coherent noise removal, ensured by proposed RD-LDHM method, is especially important in biomedical inference, as speckles can falsely imitate valid biological features. Combining this favorable outcome with large field-of-view imaging can promote the use of reported RD-LDHM technique in high-throughput stain-free biomedical screening.

Spectroscopy of individual Brownian nanoparticles in real-time using holographic localization

Individual nanoparticle spectroscopic characterization is fundamental, but challenging in liquids. While confocal selectivity is necessary to isolate a particle in a crowd, Brownian motion constantly offsets the particle from the light collection volume. Here, we present a system able to acquire holograms and reconstruct them to precisely determine the 3D position of a particle in real time. These coordinates drive an adaptive system comprising two galvanometric mirrors (x,y, transverse directions) and a tunable lens (z, longitudinal) which redirect light scattered from the corresponding region of space towards the confocal entrance of a spectrometer, thus allowing long spectral investigations on individual, freely-moving particles. A study of the movements and spectra of individual 100 nm Au nanoparticles undergoing two types of aggregations illustrates the possibilities of the method.

Computational ghost imaging based on a conditional generation countermeasure network under a low sampling rate

In this paper, an end-to-end depth neural network based on a conditional generative adversarial network for computational ghost imaging (CGANCGI) is proposed to restore clear object images with high quality at a sub-Nyquist sampling rate. The 2D light signal collected by a CMOS camera and the gray image of the original measured object are used as the input of the network model; then, the CGANCGI network is trained, and the measured object image is recovered directly from the 2D light signal. Experiments have verified that the proposed method only needs 1/10 of traditional deep learning samples to achieve fast image restoration with high-quality, and its peak signal-to-noise ratio and structural similarity are, respectively, four to six times and five to seven times higher than those of the original image, which prove that our method has practical application prospects in ghost imaging under low sampling rates.

Low-intensity illumination for lensless digital holographic microscopy with minimized sample interaction

Exposure to laser light alters cell culture examination via optical microscopic imaging techniques based on label-free coherent digital holography. To mitigate this detrimental feature, researchers tend to use a broader spectrum and lower intensity of illumination, which can decrease the quality of holographic imaging due to lower resolution and higher noise. We study the lensless digital holographic microscopy (LDHM) ability to operate in the low photon budget (LPB) regime to enable imaging of unimpaired live cells with minimized sample interaction. Low-cost off-the-shelf components are used, promoting the usability of such a straightforward approach. We show that recording data in the LPB regime (down to 7 µW of illumination power) does not limit the contrast or resolution of the hologram phase and amplitude reconstruction compared to regular illumination. The LPB generates hardware camera shot noise, however, to be effectively minimized via numerical denoising. The ability to obtain high-quality, high-resolution optical complex field reconstruction was confirmed using the USAF 1951 amplitude sample, phase resolution test target, and finally, live glial restricted progenitor cells (as a challenging strongly absorbing and scattering biomedical sample). The proposed approach based on severely limiting the photon budget in lensless holographic microscopy method can open new avenues in high-throughout (optimal resolution, large field-of-view, and high signal-to-noise-ratio single-hologram reconstruction) cell culture imaging with minimized sample interaction.

Lensless computational imaging with a hybrid framework of holographic propagation and deep learning

Lensless imaging has attracted attention as it avoids the bulky optical lens. Lensless holographic imaging is a type of a lensless imaging technique. Recently, deep learning has also shown tremendous potential in lensless holographic imaging. A labeled complex field including real and imaginary components of the samples is usually used as a training dataset. However, obtaining such a holographic dataset is challenging. In this Letter, we propose a lensless computational imaging technique with a hybrid framework of holographic propagation and deep learning. The proposed framework takes recorded holograms as input instead of complex fields, and compares the input and regenerated holograms. Compared to previous supervised learning schemes with a labeled complex field, our method does not require this supervision. Furthermore, we use the generative adversarial network to constrain the proposed framework and tackle the trivial solution. We demonstrate high-quality reconstruction with the proposed framework compared to previous deep learning methods.

Fourier Imager Network (FIN): A deep neural network for hologram reconstruction with superior external generalization

Deep learning-based image reconstruction methods have achieved remarkable success in phase recovery and holographic imaging. However, the generalization of their image reconstruction performance to new types of samples never seen by the network remains a challenge. Here we introduce a deep learning framework, termed Fourier Imager Network (FIN), that can perform end-to-end phase recovery and image reconstruction from raw holograms of new types of samples, exhibiting unprecedented success in external generalization. FIN architecture is based on spatial Fourier transform modules that process the spatial frequencies of its inputs using learnable filters and a global receptive field. Compared with existing convolutional deep neural networks used for hologram reconstruction, FIN exhibits superior generalization to new types of samples, while also being much faster in its image inference speed, completing the hologram reconstruction task in ~0.04 s per 1 mm² of the sample area. We experimentally validated the performance of FIN by training it using human lung tissue samples and blindly testing it on human prostate, salivary gland tissue and Pap smear samples, proving its superior external generalization and image reconstruction speed. Beyond holographic microscopy and quantitative phase imaging, FIN and the underlying neural network architecture might open up various new opportunities to design broadly generalizable deep learning models in computational imaging and machine vision fields.

Deep Learning-Based Label-Free Surface-Enhanced Raman Scattering Screening and Recognition of Small-Molecule Binding Sites in Proteins

Identification of small-molecule binding sites in proteins is of great significance in analysis of protein function and drug design. Modified sites can be recognized via proteolytic cleavage followed by liquid chromatography-mass spectrometry (LC-MS); however, this has always been impeded by the complexity of peptide mixtures and the elaborate synthetic design for tags. Here, we demonstrate a novel technique for identifying protein binding sites using a deep learning-based label-free surface-enhanced Raman scattering (SERS) screening (DLSS) strategy. In DLSS, the deep learning model that was trained with large SERS signals could detect signal features of small molecules with high accuracy (>99%). Without any secondary tag, the small molecules are directly complexed with proteins. After proteolysis and LC, SERS signals of all LC fractions are collected and input into the model, whereby the fractions containing the small-molecule-modified peptides can be recognized by the model and sent to MS/MS to identify the binding site(s). By using an automated DLSS system, we successfully identified the modification sites of fomepizole in alcohol dehydrogenase, which is coordinated with zinc along with three peptides. We also showed that the DLSS strategy works for identification of amino-acid residues that covalently bond with ibrutinib in Bruton tyrosine kinase. These results suggest that the DLSS strategy, which provides high molecular recognition capability to LC-MS analysis, has potential in drug discovery, proteomics, and metabolomics.

Real-time 3D imaging of ocean algae with crosstalk suppressed single-shot digital holographic microscopy

Digital holographic microscopy (DHM) has the potential to reconstruct the 3D shape of volumetric samples from a single-shot hologram in a label-free and noninvasive manner. However, the holographic reconstruction is significantly compromised by the out-of-focus image resulting from the crosstalk between refocused planes, leading to the low fidelity of the results. In this paper, we propose a crosstalk suppression algorithm-assisted 3D imaging method combined with a home built DHM system to achieve accurate 3D imaging of ocean algae using only a single hologram. As a key step in the algorithm, a hybrid edge detection strategy using gradient-based and deep learning-based methods is proposed to offer accurate boundary information for the downstream processing. With this information, the crosstalk of each refocused plane can be estimated with adjacent refocused planes. Empowered by this method, we demonstrated successful 3D imaging of six kinds of ocean algae that agree well with the ground truth; we further demonstrated that this method could achieve real-time 3D imaging of the quick swimming ocean algae in the water environment. To our knowledge, this is the first time single-shot DHM is reported in 3D imaging of ocean algae, paving the way for on-site monitoring of the ocean algae.

Accurate automatic object 4D tracking in digital in-line holographic microscopy based on computationally rendered dark fields

Building on Gabor seminal principle, digital in-line holographic microscopy provides efficient means for space-time investigations of large volumes of interest. Thus, it has a pivotal impact on particle tracking that is crucial in advancing various branches of science and technology, e.g., microfluidics and biophysical processes examination (cell motility, migration, interplay etc.). Well-established algorithms often rely on heavily regularized inverse problem modelling and encounter limitations in terms of tracking accuracy, hologram signal-to-noise ratio, accessible object volume, particle concentration and computational burden. This work demonstrates the DarkTrack algorithm-a new approach to versatile, fast, precise, and robust 4D holographic tracking based on deterministic computationally rendered high-contrast dark fields. Its unique capabilities are quantitatively corroborated employing a novel numerical engine for simulating Gabor holographic recording of time-variant volumes filled with predefined dynamic particles. Our solution accounts for multiple scattering and thus it is poised to secure an important gap in holographic particle tracking technology and allow for ground-truth-driven benchmarking and quantitative assessment of tracking algorithms. Proof-of-concept experimental evaluation of DarkTrack is presented via analyzing live spermatozoa. Software supporting both novel numerical holographic engine and DarkTrack algorithm is made open access, which opens new possibilities and sets the stage for democratization of robust holographic 4D particle examination.

Learning-based complex field recovery from digital hologram with various depth objects

In this paper, we investigate a learning-based complex field recovery technique of an object from its digital hologram. Most of the previous learning-based approaches first propagate the captured hologram to the object plane and then suppress the DC and conjugate noise in the reconstruction. To the contrary, the proposed technique utilizes a deep learning network to extract the object complex field in the hologram plane directly, making it robust to the object depth variations and well suited for three-dimensional objects. Unlike the previous approaches which concentrate on transparent biological samples having near-uniform amplitude, the proposed technique is applied to more general objects which have large amplitude variations. The proposed technique is verified by numerical simulations and optical experiments, demonstrating its feasibility.

Ultra-dense moving cascaded metasurface holography by using a physics-driven neural network

Metasurfaces are promising platforms for integrated compact optical systems. Traditional metasurface holography design algorithms are limited to information capacity due to finite spatial bandwidth production, which is insufficient for the growing demand for big data storage and encryption. Here, we propose and demonstrate deep learning empowered ultra-dense complex-amplitude holography using step-moving cascaded metasurfaces. Using deep learning artificial intelligence optimization strategy, the barriers of traditional algorithms can be conquered to meet diverse practical requirements. Two metasurfaces are cascaded to form the desired holography. One of them can move to switch the reconstruction images due to diffraction propagation accumulated during the cascaded path. The diffraction pattern from the first metasurface propagates at a different distance and meets with the second metasurface, reconstructing the target holographic reconstructions in the far-field. Such a technique can provide a new solution for multi-dimensional beam shaping, optical encryption, camouflage, integrated on-chip ultra-high-density storage, etc.

Rapid species identification of pathogenic bacteria from a minute quantity exploiting three-dimensional quantitative phase imaging and artificial neural network

The healthcare industry is in dire need of rapid microbial identification techniques for treating microbial infections. Microbial infections are a major healthcare issue worldwide, as these widespread diseases often develop into deadly symptoms. While studies have shown that an early appropriate antibiotic treatment significantly reduces the mortality of an infection, this effective treatment is difficult to practice. The main obstacle to early appropriate antibiotic treatments is the long turnaround time of the routine microbial identification, which includes time-consuming sample growth. Here, we propose a microscopy-based framework that identifies the pathogen from single to few cells. Our framework obtains and exploits the morphology of the limited sample by incorporating three-dimensional quantitative phase imaging and an artificial neural network. We demonstrate the identification of 19 bacterial species that cause bloodstream infections, achieving an accuracy of 82.5% from an individual bacterial cell or cluster. This performance, comparable to that of the gold standard mass spectroscopy under a sufficient amount of sample, underpins the effectiveness of our framework in clinical applications. Furthermore, our accuracy increases with multiple measurements, reaching 99.9% with seven different measurements of cells or clusters. We believe that our framework can serve as a beneficial advisory tool for clinicians during the initial treatment of infections.

cGAN-assisted imaging through stationary scattering media

Analyzing images taken through scattering media is challenging, owing to speckle decorrelations from perturbations in the media. For in-line imaging modalities, which are appealing because they are compact, require no moving parts, and are robust, negating the effects of such scattering becomes particularly challenging. Here we explore the use of conditional generative adversarial networks (cGANs) to mitigate the effects of the additional scatterers in in-line geometries, including digital holographic microscopy. Using light scattering simulations and experiments on objects of interest with and without additional scatterers, we find that cGANs can be quickly trained with minuscule datasets and can also efficiently learn the one-to-one statistical mapping between the cross-domain input-output image pairs. Importantly, the output images are faithful enough to enable quantitative feature extraction. We also show that with rapid training using only 20 image pairs, it is possible to negate this undesired scattering to accurately localize diffraction-limited impulses with high spatial accuracy, therefore transforming a shift variant system to a linear shift invariant (LSI) system.

LOEN: Lensless opto-electronic neural network empowered machine vision

Machine vision faces bottlenecks in computing power consumption and large amounts of data. Although opto-electronic hybrid neural networks can provide assistance, they usually have complex structures and are highly dependent on a coherent light source; therefore, they are not suitable for natural lighting environment applications. In this paper, we propose a novel lensless opto-electronic neural network architecture for machine vision applications. The architecture optimizes a passive optical mask by means of a task-oriented neural network design, performs the optical convolution calculation operation using the lensless architecture, and reduces the device size and amount of calculation required. We demonstrate the performance of handwritten digit classification tasks with a multiple-kernel mask in which accuracies of as much as 97.21% were achieved. Furthermore, we optimize a large-kernel mask to perform optical encryption for privacy-protecting face recognition, thereby obtaining the same recognition accuracy performance as no-encryption methods. Compared with the random MLS pattern, the recognition accuracy is improved by more than 6%.