High-resolution 3D refractive index microscopy of multiple-scattering samples from intensity images

Bidirectional in-silico clearing approach for deep refractive-index tomography using a sparsely sampled transmission matrix

Optical diffraction tomography (ODT) enables the label-free volumetric imaging of biological specimens by mapping their three-dimensional refractive index (RI) distribution. However, the depth of imaging achievable is restricted due to spatially inhomogeneous RI distributions that induce multiple scattering. In this study, we introduce a novel ODT technique named bidirectional in-silico clearing RI tomography. This method incorporates both forward and reversed in-silico clearing. For the reversed in-silico clearing, we have integrated an ODT reconstruction framework with a transmission matrix approach, which enables RI reconstruction and wave backpropagation from the illumination side without necessitating modifications to the conventional ODT setup. Furthermore, the framework employs a sparsely sampled transmission matrix, significantly reducing the requisite number of measurements and computational expenses. Employing this proposed technique, we successfully imaged a spheroid with a thickness of 263 µm, corresponding to 11.4 scattering mean free paths. This method was successfully applied to various biological specimens, including liver and colon spheroids, demonstrating consistent imaging performance across samples with varied morphologies.

Computational optical sectioning via near-field multi-slice ptychography

We introduce a method for the computational sectioning of optically thick samples based on a combination of near-field and multi-slice ptychography. The method enables a large field-of-view 3D phase imaging of samples that is an order of magnitude thicker than the depth of field of bright-field microscopy. An axial resolution for these thick samples is maintained in the presence of multiple scattering, revealing a complex structure beyond the depth of the field limit. In this Letter, we describe the new, to the best of our knowledge, approach and demonstrate its effectiveness using a range of samples with diverse thicknesses and optical properties.

Fourier ptychographic microscopy with adaptive resolution strategy

Fourier ptychographic microscopy (FPM) is a method capable of reconstructing a high-resolution, wide field-of-view (FOV) image, where dark-field images provide the high-frequency information required for the iterative process. Theoretically, using more dark-field images can lead to results with higher resolution. However, the resolution required to clearly detect samples with different microscales varies. For certain samples, the limit resolution of the imaging system may exceed the one required to resolve the details. This suggests that simply increasing the number of dark-field images will not improve the recognition capability for such samples and may instead significantly increase the computational cost. To address this issue, this Letter proposes an adaptive resolution strategy that automatically assigns the resolution required for the sample. Based on a Tenengrad approach, this strategy determines the number of images required for reconstruction by evaluating a series of differential images among the reconstructions for a certain subregion and then efficiently completes the full-FOV reconstruction according to the determined resolution. We conducted the full-FOV reconstruction utilizing feature-domain FPM for both the USAF resolution test chart and a human red blood cell sample. Employing the adaptive resolution strategy, the preservation of reconstruction resolution can be ensured while respectively economizing approximately 76% and 89% of the time.

Perspective on quantitative phase imaging to improve precision cancer medicine

Significance

Quantitative phase imaging (QPI) offers a label-free approach to non-invasively characterize cellular processes by exploiting their refractive index based intrinsic contrast. QPI captures this contrast by translating refractive index associated phase shifts into intensity-based quantifiable data with nanoscale sensitivity. It holds significant potential for advancing precision cancer medicine by providing quantitative characterization of the biophysical properties of cells and tissue in their natural states.

Aim

This perspective aims to discuss the potential of QPI to increase our understanding of cancer development and its response to therapeutics. It also explores new developments in QPI methods towards advancing personalized cancer therapy and early detection.

Approach

We begin by detailing the technical advancements of QPI, examining its implementations across transmission and reflection geometries and phase retrieval methods, both interferometric and non-interferometric. The focus then shifts to QPI's applications in cancer research, including dynamic cell mass imaging for drug response assessment, cancer risk stratification, and in-vivo tissue imaging.

Results

QPI has emerged as a crucial tool in precision cancer medicine, offering insights into tumor biology and treatment efficacy. Its sensitivity to detecting nanoscale changes holds promise for enhancing cancer diagnostics, risk assessment, and prognostication. The future of QPI is envisioned in its integration with artificial intelligence, morpho-dynamics, and spatial biology, broadening its impact in cancer research.

Conclusions

QPI presents significant potential in advancing precision cancer medicine and redefining our approach to cancer diagnosis, monitoring, and treatment. Future directions include harnessing high-throughput dynamic imaging, 3D QPI for realistic tumor models, and combining artificial intelligence with multi-omics data to extend QPI's capabilities. As a result, QPI stands at the forefront of cancer research and clinical application in cancer care.

Three-dimensional tomographic reconstruction using Voronoi weighting

Three-dimensional tomographic reconstruction requires careful selection of the illumination angles, often under certain measurement constraints. When the angular distribution must be nonuniform, appropriate selection of the reconstruction weights is necessary. We show that Voronoi weighting can significantly improve the fidelity of optical diffraction tomography.

Detectability of unresolved particles in off-axis digital holographic microscopy

Off-axis digital holographic microscopy (DHM) provides both amplitude and phase images, and so it may be used for label-free 3D tracking of micro- and nano-sized particles of different compositions, including biological cells, strongly absorbing particles, and strongly scattering particles. Contrast is provided by differences in either the real or imaginary parts of the refractive index (phase contrast and absorption) and/or by scattering. While numerous studies have focused on phase contrast and improving resolution in DHM, particularly axial resolution, absent have been studies quantifying the limits of detection for unresolved particles. This limit has important implications for microbial detection, including in life-detection missions for space flight. Here we examine the limits of detection of nanosized particles as a function of particle optical properties, microscope optics (including camera well depth and substrate), and data processing techniques and find that DHM provides contrast in both amplitude and phase for unresolved spheres, in rough agreement with Mie theory scattering cross-sections. Amplitude reconstructions are more useful than phase for low-index spheres and should not be neglected in DHM analysis.

Recent Advances and Current Trends in Transmission Tomographic Diffraction Microscopy

Optical microscopy techniques are among the most used methods in biomedical sample characterization. In their more advanced realization, optical microscopes demonstrate resolution down to the nanometric scale. These methods rely on the use of fluorescent sample labeling in order to break the diffraction limit. However, fluorescent molecules' phototoxicity or photobleaching is not always compatible with the investigated samples. To overcome this limitation, quantitative phase imaging techniques have been proposed. Among these, holographic imaging has demonstrated its ability to image living microscopic samples without staining. However, for a 3D assessment of samples, tomographic acquisitions are needed. Tomographic Diffraction Microscopy (TDM) combines holographic acquisitions with tomographic reconstructions. Relying on a 3D synthetic aperture process, TDM allows for 3D quantitative measurements of the complex refractive index of the investigated sample. Since its initial proposition by Emil Wolf in 1969, the concept of TDM has found a lot of applications and has become one of the hot topics in biomedical imaging. This review focuses on recent achievements in TDM development. Current trends and perspectives of the technique are also discussed.

Redundant information model for Fourier ptychographic microscopy

Fourier ptychographic microscopy (FPM) is a computational optical imaging technique that overcomes the traditional trade-off between resolution and field of view (FOV) by exploiting abundant redundant information in both spatial and frequency domains for high-quality image reconstruction. However, the redundant information in FPM remains ambiguous or abstract, which presents challenges to further enhance imaging capabilities and deepen our understanding of the FPM technique. Inspired by Shannon's information theory and extensive experimental experience in FPM, we defined the specimen complexity and reconstruction algorithm utilization rate and reported a model of redundant information for FPM to predict reconstruction results and guide the optimization of imaging parameters. The model has been validated through extensive simulations and experiments. In addition, it provides a useful tool to evaluate different algorithms, revealing a utilization rate of 24%±1% for the Gauss-Newton algorithm, LED Multiplexing, Wavelength Multiplexing, EPRY-FPM, and GS. In contrast, mPIE exhibits a lower utilization rate of 19%±1%.

Roadmap on Label-Free Super-Resolution Imaging

Label-free super-resolution (LFSR) imaging relies on light-scattering processes in nanoscale objects without a need for fluorescent (FL) staining required in super-resolved FL microscopy. The objectives of this Roadmap are to present a comprehensive vision of the developments, the state-of-the-art in this field, and to discuss the resolution boundaries and hurdles which need to be overcome to break the classical diffraction limit of the LFSR imaging. The scope of this Roadmap spans from the advanced interference detection techniques, where the diffraction-limited lateral resolution is combined with unsurpassed axial and temporal resolution, to techniques with true lateral super-resolution capability which are based on understanding resolution as an information science problem, on using novel structured illumination, near-field scanning, and nonlinear optics approaches, and on designing superlenses based on nanoplasmonics, metamaterials, transformation optics, and microsphere-assisted approaches. To this end, this Roadmap brings under the same umbrella researchers from the physics and biomedical optics communities in which such studies have often been developing separately. The ultimate intent of this paper is to create a vision for the current and future developments of LFSR imaging based on its physical mechanisms and to create a great opening for the series of articles in this field.

Ptychographic lens-less birefringence microscopy using a mask-modulated polarization image sensor

Birefringence, an inherent characteristic of optically anisotropic materials, is widely utilized in various imaging applications ranging from material characterizations to clinical diagnosis. Polarized light microscopy enables high-resolution, high-contrast imaging of optically anisotropic specimens, but it is associated with mechanical rotations of polarizer/analyzer and relatively complex optical designs. Here, we present a form of lens-less polarization-sensitive microscopy capable of complex and birefringence imaging of transparent objects without an optical lens and any moving parts. Our method exploits an optical mask-modulated polarization image sensor and single-input-state LED illumination design to obtain complex and birefringence images of the object via ptychographic phase retrieval. Using a camera with a pixel size of 3.45 μm, the method achieves birefringence imaging with a half-pitch resolution of 2.46 μm over a 59.74 mm2 field-of-view, which corresponds to a space-bandwidth product of 9.9 megapixels. We demonstrate the high-resolution, large-area, phase and birefringence imaging capability of our method by presenting the phase and birefringence images of various anisotropic objects, including a monosodium urate crystal, and excised mouse eye and heart tissues.

Refractive index tomography with a physics-based optical neural network

The non-interference three-dimensional refractive index (RI) tomography has attracted extensive attention in the life science field for its simple system implementation and robust imaging performance. However, the complexity inherent in the physical propagation process poses significant challenges when the sample under study deviates from the weak scattering approximation. Such conditions complicate the task of achieving global optimization with conventional algorithms, rendering the reconstruction process both time-consuming and potentially ineffective. To address such limitations, this paper proposes an untrained multi-slice neural network (MSNN) with an optical structure, in which each layer has a clear corresponding physical meaning according to the beam propagation model. The network does not require pre-training and performs good generalization and can be recovered through the optimization of a set of intensity images. Concurrently, MSNN can calibrate the intensity of different illumination by learnable parameters, and the multiple backscattering effects have also been taken into consideration by integrating a "scattering attenuation layer" between adjacent "RI" layers in the MSNN. Both simulations and experiments have been conducted carefully to demonstrate the effectiveness and feasibility of the proposed method. Experimental results reveal that MSNN can enhance clarity with increased efficiency in RI tomography. The implementation of MSNN introduces a novel paradigm for RI tomography.

Fast bidirectional vector wave propagation method showcased on targeted noise reduction in imaging fiber bundles using 3D-printed micro optics

In order to extend simulation capabilities for reflective and catadioptric 3D-printed micro optics, we present a fast bidirectional vector wave propagation method (BWPM). Contrary to established fast simulation methods like the wave propagation method (WPM), the BWPM allows for the additional consideration of reflected and backwards propagating electric fields. We study the convergence of the BWPM and investigate relevant simulation examples. Especially, the BWPM is used for evaluation of 3D-printed index matching caps (IMCs) in order to suppress back reflected light in imaging fibers, used for keyhole access endoscopy. Simulations studying the viability of IMCs are followed up with experimental investigations. We demonstrate that 3D-printed IMCs can be used to suppress noise caused by back reflected light, that otherwise would prohibit the use of imaging fibers in an epi-illumination configuration.

Design of Fourier ptychographic illuminator for single full-FOV reconstruction

Fourier ptychographic microscopy (FPM) is a spatial-temporal-modulation high-throughput imaging technique via a sequential angle-varied LED illumination. Therefore, the illuminator is one of the key components and the design of this illuminator is significant. However, because of the property of spherical wave, partial coherence, and aperture-induced vignetting, the acquired images must be processed in blocks first, and rely on parallel reconstruction via a graphics processing unit (GPU). The high cost makes it unappealing compared with commercial whole slide imaging system via a low-cost central processing unit (CPU). Especially, the vignetting severely destroys the space-invariant model and induces obvious artifacts in FPM, which is the most difficult problem. The conventional method is to divide the field of view (FOV) into many tiles and omit those imperfect images, which is crude and may discards low frequency information. In this paper, we reevaluated the conditions of vignetting in FPM. Through our analysis, the maximum side length of FOV is 0.759 mm for a single full-FOV reconstruction via a 4×/0.1 NA objective and a 4 mm spacing LED array in theory, while almost 1.0 mm can be achieved in practice due to the tolerance of algorithm. We found that FPM system can treat the vignetting coefficient Vf below 0.1 as brightfield images and Vf lager than 0.9 as darkfield images, respectively. We reported an optimized distribution for designing an illuminator without vignetting effect according to the off-the-shelf commercial products, which can reconstruct full FOV in one time via a CPU. By adjusting the distribution of LED units, the system could retrieve the object with the side length of FOV up to 3.8 mm for a single full-FOV reconstruction, which achieves the largest FOV that a typical 4×/0.1 NA objective with the field number of 22 mm can afford.

Three-dimensional refractive index estimation based on deep-inverse non-interferometric optical diffraction tomography (ODT-Deep)

Optical diffraction tomography (ODT) solves an inverse scattering problem to obtain label-free, 3D refractive index (RI) estimation of biological specimens. This work demonstrates 3D RI retrieval methods suitable for partially-coherent ODT systems supported by intensity-only measurements consisting of axial and angular illumination scanning. This framework allows for access to 3D quantitative RI contrast using a simplified non-interferometric technique. We consider a traditional iterative tomographic solver based on a multiple in-plane representation of the optical scattering process and gradient descent optimization adapted for focus-scanning systems, as well as an approach that relies solely on 3D convolutional neural networks (CNNs) to invert the scattering process. The approaches are validated using simulations of the 3D scattering potential for weak phase 3D biological samples.

Unlocking the secrets of the invisible world: incredible deep optical imaging through in-silico clearing

In-silico clearing enables deep optical imaging of biological samples by correcting image blur caused by scattering and aberration. This breakthrough method offers researchers unprecedented insights into three-dimensional biological systems, with enormous potential for advancing biology and medicine to better understand living organisms and human health.

Far-field super-resolution chemical microscopy

Far-field chemical microscopy providing molecular electronic or vibrational fingerprint information opens a new window for the study of three-dimensional biological, material, and chemical systems. Chemical microscopy provides a nondestructive way of chemical identification without exterior labels. However, the diffraction limit of optics hindered it from discovering more details under the resolution limit. Recent development of super-resolution techniques gives enlightenment to open this door behind far-field chemical microscopy. Here, we review recent advances that have pushed the boundary of far-field chemical microscopy in terms of spatial resolution. We further highlight applications in biomedical research, material characterization, environmental study, cultural heritage conservation, and integrated chip inspection.

Spatial- and Fourier-domain ptychography for high-throughput bio-imaging

First envisioned for determining crystalline structures, ptychography has become a useful imaging tool for microscopists. However, ptychography remains underused by biomedical researchers due to its limited resolution and throughput in the visible light regime. Recent developments of spatial- and Fourier-domain ptychography have successfully addressed these issues and now offer the potential for high-resolution, high-throughput optical imaging with minimal hardware modifications to existing microscopy setups, often providing an excellent trade-off between resolution and field of view inherent to conventional imaging systems, giving biomedical researchers the best of both worlds. Here, we provide extensive information to enable the implementation of ptychography by biomedical researchers in the visible light regime. We first discuss the intrinsic connections between spatial-domain coded ptychography and Fourier ptychography. A step-by-step guide then provides the user instructions for developing both systems with practical examples. In the spatial-domain implementation, we explain how a large-scale, high-performance blood-cell lens can be made at negligible expense. In the Fourier-domain implementation, we explain how adding a low-cost light source to a regular microscope can improve the resolution beyond the limit of the objective lens. The turnkey operation of these setups is suitable for use by professional research laboratories, as well as citizen scientists. Users with basic experience in optics and programming can build the setups within a week. The do-it-yourself nature of the setups also allows these procedures to be implemented in laboratory courses related to Fourier optics, biomedical instrumentation, digital image processing, robotics and capstone projects.

Efficient and accurate intensity diffraction tomography of multiple-scattering samples

Optical Diffraction Tomography (ODT) is a label-free method to quantitatively estimate the 3D refractive index (RI) distributions of microscopic samples. Recently, significant efforts were directed towards methods to model multiple-scattering objects. The fidelity of reconstructions rely on accurately modelling light-matter interactions, but the efficient simulation of light propagation through high-RI structures over a large range of illumination angles is still challenging. Here we present a solution dealing with these problems, proposing a method that allows one to efficiently model the tomographic image formation for strongly scattering objects illuminated over a wide range of angles. Instead of propagating tilted plane waves we apply rotations on the illuminated object and optical field and formulate a new and robust multi-slice model suitable for high-RI contrast structures. We test reconstructions made by our approach against simulations and experiments, using rigorous solutions to Maxwell's equations as ground truth. We find the proposed method to produce reconstructions of higher fidelity compared to conventional multi-slice methods, especially for the challenging case of strongly scattering samples where conventional reconstruction methods fail.

Polarization-sensitive intensity diffraction tomography

Optical anisotropy, which is an intrinsic property of many materials, originates from the structural arrangement of molecular structures, and to date, various polarization-sensitive imaging (PSI) methods have been developed to investigate the nature of anisotropic materials. In particular, the recently developed tomographic PSI technologies enable the investigation of anisotropic materials through volumetric mappings of the anisotropy distribution of these materials. However, these reported methods mostly operate on a single scattering model, and are thus not suitable for three-dimensional (3D) PSI imaging of multiple scattering samples. Here, we present a novel reference-free 3D polarization-sensitive computational imaging technique-polarization-sensitive intensity diffraction tomography (PS-IDT)-that enables the reconstruction of 3D anisotropy distribution of both weakly and multiple scattering specimens from multiple intensity-only measurements. A 3D anisotropic object is illuminated by circularly polarized plane waves at various illumination angles to encode the isotropic and anisotropic structural information into 2D intensity information. These information are then recorded separately through two orthogonal analyzer states, and a 3D Jones matrix is iteratively reconstructed based on the vectorial multi-slice beam propagation model and gradient descent method. We demonstrate the 3D anisotropy imaging capabilities of PS-IDT by presenting 3D anisotropy maps of various samples, including potato starch granules and tardigrade.

Influence of Yokukansan on the refractive index of neuroblastoma cells

Yokukansan (YKS) is a traditional Japanese herbal medicine that is increasingly being studied for its effects on neurodegenerative diseases. In our study, we presented a novel methodology for a multimodal analysis of the effects of YKS on nerve cells. The measurements of 3D refractive index distribution and its changes performed by holographic tomography were supported with an investigation by Raman micro-spectroscopy and fluorescence microscopy to gather complementary morphological and chemical information about cells and YKS influence. It was shown that at the concentrations tested, YKS inhibits proliferation, possibly involving reactive oxygen species. Also substantial changes in the cell RI after few hours of YKS exposure were detected, followed by longer-term changes in cell lipid composition and chromatin state.

Parallelized computational 3D video microscopy of freely moving organisms at multiple gigapixels per second

Wide field of view microscopy that can resolve 3D information at high speed and spatial resolution is highly desirable for studying the behaviour of freely moving model organisms. However, it is challenging to design an optical instrument that optimises all these properties simultaneously. Existing techniques typically require the acquisition of sequential image snapshots to observe large areas or measure 3D information, thus compromising on speed and throughput. Here, we present 3D-RAPID, a computational microscope based on a synchronized array of 54 cameras that can capture high-speed 3D topographic videos over an area of 135 cm2, achieving up to 230 frames per second at spatiotemporal throughputs exceeding 5 gigapixels per second. 3D-RAPID employs a 3D reconstruction algorithm that, for each synchronized snapshot, fuses all 54 images into a composite that includes a co-registered 3D height map. The self-supervised 3D reconstruction algorithm trains a neural network to map raw photometric images to 3D topography using stereo overlap redundancy and ray-propagation physics as the only supervision mechanism. The resulting reconstruction process is thus robust to generalization errors and scales to arbitrarily long videos from arbitrarily sized camera arrays. We demonstrate the broad applicability of 3D-RAPID with collections of several freely behaving organisms, including ants, fruit flies, and zebrafish larvae.

In-silico clearing approach for deep refractive index tomography by partial reconstruction and wave-backpropagation

Refractive index (RI) is considered to be a fundamental physical and biophysical parameter in biological imaging, as it governs light-matter interactions and light propagation while reflecting cellular properties. RI tomography enables volumetric visualization of RI distribution, allowing biologically relevant analysis of a sample. However, multiple scattering (MS) and sample-induced aberration (SIA) caused by the inhomogeneity in RI distribution of a thick sample make its visualization challenging. This paper proposes a deep RI tomographic approach to overcome MS and SIA and allow the enhanced reconstruction of thick samples compared to that enabled by conventional linear-model-based RI tomography. The proposed approach consists of partial RI reconstruction using multiple holograms acquired with angular diversity and their backpropagation using the reconstructed partial RI map, which unambiguously reconstructs the next partial volume. Repeating this operation efficiently reconstructs the entire RI tomogram while suppressing MS and SIA. We visualized a multicellular spheroid of diameter 140 µm within minutes of reconstruction, thereby demonstrating the enhanced deep visualization capability and computational efficiency of the proposed method compared to those of conventional RI tomography. Furthermore, we quantified the high-RI structures and morphological changes inside multicellular spheroids, indicating that the proposed method can retrieve biologically relevant information from the RI distribution. Benefitting from the excellent biological interpretability of RI distributions, the label-free deep visualization capability of the proposed method facilitates a noninvasive understanding of the architecture and time-course morphological changes of thick multicellular specimens.

Single-shot refractive index slice imaging using spectrally multiplexed optical transfer function reshaping

The refractive index (RI) of cells and tissues is crucial in pathophysiology as a noninvasive and quantitative imaging contrast. Although its measurements have been demonstrated using three-dimensional quantitative phase imaging methods, these methods often require bulky interferometric setups or multiple measurements, which limits the measurement sensitivity and speed. Here, we present a single-shot RI imaging method that visualizes the RI of the in-focus region of a sample. By exploiting spectral multiplexing and optical transfer function engineering, three color-coded intensity images of a sample with three optimized illuminations were simultaneously obtained in a single-shot measurement. The measured intensity images were then deconvoluted to obtain the RI image of the in-focus slice of the sample. As a proof of concept, a setup was built using Fresnel lenses and a liquid-crystal display. For validation purposes, we measured microspheres of known RI and cross-validated the results with simulated results. Various static and highly dynamic biological cells were imaged to demonstrate that the proposed method can conduct single-shot RI slice imaging of biological samples with subcellular resolution.

3D full-wave multi-scattering forward solver for coherent microscopes

A rigorous forward model solver for conventional coherent microscope is presented. The forward model is derived from Maxwell's equations and models the wave behaviour of light matter interaction. Vectorial waves and multiple-scattering effect are considered in this model. Scattered field can be calculated with given distribution of the refractive index of the biological sample. Bright field images can be obtained by combining the scattered field and reflected illumination, and experimental validation is included. Insights into the utility of the full-wave multi-scattering (FWMS) solver and comparison with the conventional Born approximation based solver are presented. The model is also generalizable to the other forms of label-free coherent microscopes, such as quantitative phase microscope and dark-field microscope.

Optical ptychography for biomedical imaging: recent progress and future directions [Invited]

Ptychography is an enabling microscopy technique for both fundamental and applied sciences. In the past decade, it has become an indispensable imaging tool in most X-ray synchrotrons and national laboratories worldwide. However, ptychography's limited resolution and throughput in the visible light regime have prevented its wide adoption in biomedical research. Recent developments in this technique have resolved these issues and offer turnkey solutions for high-throughput optical imaging with minimum hardware modifications. The demonstrated imaging throughput is now greater than that of a high-end whole slide scanner. In this review, we discuss the basic principle of ptychography and summarize the main milestones of its development. Different ptychographic implementations are categorized into four groups based on their lensless/lens-based configurations and coded-illumination/coded-detection operations. We also highlight the related biomedical applications, including digital pathology, drug screening, urinalysis, blood analysis, cytometric analysis, rare cell screening, cell culture monitoring, cell and tissue imaging in 2D and 3D, polarimetric analysis, among others. Ptychography for high-throughput optical imaging, currently in its early stages, will continue to improve in performance and expand in its applications. We conclude this review article by pointing out several directions for its future development.

Multiple-scattering simulator-trained neural network for intensity diffraction tomography

Recovering 3D phase features of complex biological samples traditionally sacrifices computational efficiency and processing time for physical model accuracy and reconstruction quality. Here, we overcome this challenge using an approximant-guided deep learning framework in a high-speed intensity diffraction tomography system. Applying a physics model simulator-based learning strategy trained entirely on natural image datasets, we show our network can robustly reconstruct complex 3D biological samples. To achieve highly efficient training and prediction, we implement a lightweight 2D network structure that utilizes a multi-channel input for encoding the axial information. We demonstrate this framework on experimental measurements of weakly scattering epithelial buccal cells and strongly scattering C. elegans worms. We benchmark the network's performance against a state-of-the-art multiple-scattering model-based iterative reconstruction algorithm. We highlight the network's robustness by reconstructing dynamic samples from a living worm video. We further emphasize the network's generalization capabilities by recovering algae samples imaged from different experimental setups. To assess the prediction quality, we develop a quantitative evaluation metric to show that our predictions are consistent with both multiple-scattering physics and experimental measurements.

High-fidelity optical diffraction tomography of live organisms using iodixanol refractive index matching

Optical diffraction tomography (ODT) enables the three-dimensional (3D) refractive index (RI) reconstruction. However, when the RI difference between a sample and a medium increases, the effects of light scattering become significant, preventing the acquisition of high-quality and accurate RI reconstructions. Herein, we present a method for high-fidelity ODT by introducing non-toxic RI matching media. Optimally reducing the RI contrast enhances the fidelity and accuracy of 3D RI reconstruction, enabling visualization of the morphology and intra-organization of live biological samples without producing toxic effects. We validate our method using various biological organisms, including C. albicans and C. elegans.

3D scattering microphantom sample to assess quantitative accuracy in tomographic phase microscopy techniques

In this paper we present a structurally-complex biomimetic scattering structure, fabricated with two-photon polymerization, and utilize this object in order to benchmark a computational imaging system. The phantom allows to tailor the scattering by modifying its degrees of freedom i.e. refractive index contrast and scattering layer dimensions and incorporates a 3D imaging quality test, representing a single cell within tissue. While the sample may be used with multiple 3D microscopy techniques, we demonstrate the impact of scattering on three tomographic phase microscopy (TPM) reconstruction methods. One of these methods assumes the sample to be weak-scattering, while the other two take multiple scattering into account. The study is performed at two wavelengths (visible and near-infrared), which serve as a scaling factor for the scattering phenomenon. We find that changing the wavelength from visible into near-infrared impacts the applicability of TPM reconstruction methods. As a result of reduced scattering in near-infrared region, the multiple-scattering-oriented techniques perform in fact worse than a method aimed for weak-scattering samples. This implies a necessity of selecting proper approach depending on sample's scattering characteristics even in case of subtle changes in the object-light interaction.

Three-dimensional bi-functional refractive index and fluorescence microscopy (BRIEF)

Fluorescence microscopy is a powerful tool for imaging biological samples with molecular specificity. In contrast, phase microscopy provides label-free measurement of the sample's refractive index (RI), which is an intrinsic optical property that quantitatively relates to cell morphology, mass, and stiffness. Conventional imaging techniques measure either the labeled fluorescence (functional) information or the label-free RI (structural) information, though it may be valuable to have both. For example, biological tissues have heterogeneous RI distributions, causing sample-induced scattering that degrades the fluorescence image quality. When both fluorescence and 3D RI are measured, one can use the RI information to digitally correct multiple-scattering effects in the fluorescence image. Here, we develop a new computational multi-modal imaging method based on epi-mode microscopy that reconstructs both 3D fluorescence and 3D RI from a single dataset. We acquire dozens of fluorescence images, each 'illuminated' by a single fluorophore, then solve an inverse problem with a multiple-scattering forward model. We experimentally demonstrate our method for epi-mode 3D RI imaging and digital correction of multiple-scattering effects in fluorescence images.

Numerical refractive index correction for the stitching procedure in tomographic quantitative phase imaging

Tomographic quantitative phase imaging (QPI) lacks an absolute refractive index value baseline, which poses a problem when large dense objects extending over multiple fields of view are measured volume by volume and stitched together. Some of the measurements lack the natural baseline value that is provided by the mounting medium with a known refractive index. In this work, we discuss the problem of the refractive index (RI) baseline of individual reconstructed volumes that are deprived of access to mounting medium due to the extent of the object. The solution of this problem is provided by establishing the RI offsets based on the overlapping regions. We have proven that the process of finding the offset RI values may be justifiably reduced to the analogous procedure in the 2D baseline correction (2D-BC). Finally, we proposed the enhancement of the state-of-the-art 2D-BC procedure previously introduced in the context of 2D QPI. The processing is validated at the examples of a synthetic dataset and a liver organoid.

Finding intracellular lipid droplets from the single-cell biolens' signature in a holographic flow-cytometry assay

In recent years, intracellular LDs have been discovered to play an important role in several pathologies. Therefore, detection of LDs would provide an in-demand diagnostic tool if coupled with flow-cytometry to give significant statistical analysis and especially if the diagnosis is made in full non-invasive mode. Here we combine the experimental results of in-flow tomographic phase microscopy with a suited numerical simulation to demonstrate that intracellular LDs can be easily detected through a label-free approach based on the direct analysis of the 2D quantitative phase maps recorded by a holographic flow cytometer. In fact, we demonstrate that the presence of LDs affects the optical focusing lensing features of the embracing cell, which can be considered a biological lens. The research was conducted on white blood cells (i.e., lymphocytes and monocytes) and ovarian cancer cells. Results show that the biolens properties of cells can be a rapid biomarker that aids in boosting the diagnosis of LDs-related pathologies by means of the holographic flow-cytometry assay for fast, non-destructive, and high-throughput screening of statistically significant number of cells.

Ptychographic reconstruction with object initialization

X-ray ptychography is a cutting edge imaging technique providing ultra-high spatial resolutions. In ptychography, phase retrieval, i.e., the recovery of a complex valued signal from intensity-only measurements, is enabled by exploiting a redundancy of information contained in diffraction patterns measured with overlapping illuminations. For samples that are considerably larger than the probe we show that during the iteration the bulk information has to propagate from the sample edges to the center. This constitutes an inherent limitation of reconstruction speed for algorithms that use a flat initialization. Here, we experimentally demonstrate that a considerable improvement of computational speed can be achieved by utilizing a low resolution sample wavefront retrieved from measured diffraction patterns as object initialization. In addition, we show that this approach avoids phase artifacts associated with large phase gradients and may alleviate the requirements on phase structure within the probe. Object initialization is computationally fast, potentially beneficial for bulky sample and compatible with flat samples. Therefore, the presented approach is readily adaptable with established ptychographic reconstruction algorithms implying a wide spread use.

High-fidelity intensity diffraction tomography with a non-paraxial multiple-scattering model

We propose a novel intensity diffraction tomography (IDT) reconstruction algorithm based on the split-step non-paraxial (SSNP) model for recovering the 3D refractive index (RI) distribution of multiple-scattering biological samples. High-quality IDT reconstruction requires high-angle illumination to encode both low- and high- spatial frequency information of the 3D biological sample. We show that our SSNP model can more accurately compute multiple scattering from high-angle illumination compared to paraxial approximation-based multiple-scattering models. We apply this SSNP model to both sequential and multiplexed IDT techniques. We develop a unified reconstruction algorithm for both IDT modalities that is highly computationally efficient and is implemented by a modular automatic differentiation framework. We demonstrate the capability of our reconstruction algorithm on both weakly scattering buccal epithelial cells and strongly scattering live C. elegans worms and live C. elegans embryos.

Use of Magnetic Modulation of Nitrogen-Vacancy Center Fluorescence in Nanodiamonds for Quantitative Analysis of Nanoparticles in Organisms

The fluorescence intensity emitted by nitrogen-vacancy (NV) centers in diamond nanoparticles can be readily modulated by the application of a magnetic field using a small electromagnet. By acquiring interleaved images acquired in the presence and absence of the magnetic field and performing digital subtraction, the fluorescence intensity of the NV nanodiamond can be isolated from scattering and autofluorescence even when these backgrounds are changing monotonically during the experiments. This approach has the potential to enable the robust identification of nanodiamonds in organisms and other complex environments. Yet, the practical application of magnetic modulation imaging to realistic systems requires the use of quantitative analysis methods based on signal-to-noise considerations. Here, we describe the use of magnetic modulation to analyze the uptake of diamond nanoparticles from an aqueous environment into Caenorhabditis elegans, used here as a model system for identification and quantification of nanodiamonds in complex matrices. Based on the observed signal-to-noise ratio of sets of digitally subtracted images, we show that nanodiamonds can be identified on an individual pixel basis with a >99.95% confidence. To determine whether surface functionalization of the nanodiamond significantly impacted uptake, we used this approach to analyze the presence of nanodiamonds in C. elegans that had been exposed to these functionalized nanodiamonds in the water column, with uptake likely occurring by ingestion. In each case, the images show a significant nanoparticle uptake. However, differences in uptake between the three ligands were not outside of the experimental error, indicating that additional factors beyond the surface charge are important factors controlling uptake. Analysis of the number of pixels above the threshold in individual C. elegans organisms revealed distributions that deviate significantly from a Poisson distribution, suggesting that uptake of nanoparticles may not be a statistically independent event. The results presented here demonstrate that magnetic modulation combined with quantitative analysis of the resulting images can be used to robustly characterize nanoparticle uptake into organisms.

Computational 3D microscopy with optical coherence refraction tomography

Optical coherence tomography (OCT) has seen widespread success as an in vivo clinical diagnostic 3D imaging modality, impacting areas including ophthalmology, cardiology, and gastroenterology. Despite its many advantages, such as high sensitivity, speed, and depth penetration, OCT suffers from several shortcomings that ultimately limit its utility as a 3D microscopy tool, such as its pervasive coherent speckle noise and poor lateral resolution required to maintain millimeter-scale imaging depths. Here, we present 3D optical coherence refraction tomography (OCRT), a computational extension of OCT which synthesizes an incoherent contrast mechanism by combining multiple OCT volumes, acquired across two rotation axes, to form a resolution-enhanced, speckle-reduced, refraction-corrected 3D reconstruction. Our label-free computational 3D microscope features a novel optical design incorporating a parabolic mirror to enable the capture of 5D plenoptic datasets, consisting of millimetric 3D fields of view over up to ±75° without moving the sample. We demonstrate that 3D OCRT reveals 3D features unobserved by conventional OCT in fruit fly, zebrafish, and mouse samples.

Optical diffraction tomography of 3D microstructures using a low coherence source

Optical diffraction tomography (ODT) is a label-free technique for three dimensional imaging of micron-sized objects. Coherence and limited sampling of 3D Fourier space are often responsible for the appearance of artifacts. Here we present an ODT microscope that uses low temporal coherence light and spatial light modulators to retrieve reliable 3D maps of the refractive index. A common-path interferometer, based on a spatial light modulator, measures the complex fields transmitted by a sample. Measured fields, acquired while scanning the illumination direction using a digital micro-mirror device, are fed into a Rytov reconstruction algorithm to obtain refractive index maps whose accuracy is directly evaluated on microfabricated 3D test objects. Even for challenging shapes such as pyramids, bridges, and dumbbells, we obtain volumetric reconstructions that compare very well with electron microscopy images.

Multi-slice ptychographic imaging with multistage coarse-to-fine reconstruction

The ability to image 3D samples with optical sectioning is essential for the study of tomographic morphology in material and biological sciences. However, it is often hampered by limitations of acquisition speed and equipment complexity when performing 3D volumetric imaging. Here, we propose, to the best of our knowledge, a new method for 3D reconstruction from a minimum of four intensity-only measurements. The complementary structured patterns provided by the digital micromirror device (DMD) irradiate the outermost layer of the sample to generate the corresponding diffraction intensities for recording, which enables rapid scanning of loaded patterns for fast acquisition. Our multistage reconstruction algorithm first extracts the overall coarse-grained information, and then iteratively optimizes the information of different layers to obtain fine features, thereby achieving high-resolution 3D tomography. The high-fidelity reconstruction in experiments on two-slice resolution targets, unstained Polyrhachis vicina Roger and freely moving C. elegans proves the robustness of the method. Compared with traditional 3D reconstruction methods such as interferometry-based methods or Fourier ptychographic tomography (FPT), our method increases the reconstruction speed by at least 10 times and is suitable for label-free dynamic imaging in multiple-scattering samples. Such 3D reconstruction suggests potential applications in a wide range of fields.

Transport of intensity diffraction tomography with non-interferometric synthetic aperture for three-dimensional label-free microscopy

We present a new label-free three-dimensional (3D) microscopy technique, termed transport of intensity diffraction tomography with non-interferometric synthetic aperture (TIDT-NSA). Without resorting to interferometric detection, TIDT-NSA retrieves the 3D refractive index (RI) distribution of biological specimens from 3D intensity-only measurements at various illumination angles, allowing incoherent-diffraction-limited quantitative 3D phase-contrast imaging. The unique combination of z-scanning the sample with illumination angle diversity in TIDT-NSA provides strong defocus phase contrast and better optical sectioning capabilities suitable for high-resolution tomography of thick biological samples. Based on an off-the-shelf bright-field microscope with a programmable light-emitting-diode (LED) illumination source, TIDT-NSA achieves an imaging resolution of 206 nm laterally and 520 nm axially with a high-NA oil immersion objective. We validate the 3D RI tomographic imaging performance on various unlabeled fixed and live samples, including human breast cancer cell lines MCF-7, human hepatocyte carcinoma cell lines HepG2, mouse macrophage cell lines RAW 264.7, Caenorhabditis elegans (C. elegans), and live Henrietta Lacks (HeLa) cells. These results establish TIDT-NSA as a new non-interferometric approach to optical diffraction tomography and 3D label-free microscopy, permitting quantitative characterization of cell morphology and time-dependent subcellular changes for widespread biological and medical applications.

3D time-lapse imaging of a mouse embryo using intensity diffraction tomography embedded inside a deep learning framework

We present a compact 3D diffractive microscope that can be inserted directly in a cell incubator for long-term observation of developing organisms. Our setup is particularly simple and robust, since it does not include any moving parts and is compatible with commercial cell culture containers. It has been designed to image large specimens (>100×100×100µm³) with subcellular resolution. The sample's optical properties [refractive index (RI) and absorption] are reconstructed in 3D from intensity-only images recorded with different illumination angles produced by an LED array. The reconstruction is performed using the beam propagation method embedded inside a deep-learning network where the layers encode the optical properties of the object. This deep neural network is trained for a given multiangle intensity acquisition. After training, the weights of the neural network deliver the 3D distribution of the optical properties of the sample. The effect of spherical aberrations due to the sample holder/air interfaces are taken into account in the forward model. Using this approach, we performed time-lapse 3D imaging of preimplantation mouse embryos over six days. Images of embryos from a single cell (low-scattering regime) to the blastocyst stage (highly scattering regime) were successfully reconstructed. Due to its subcellular resolution, our system can provide quantitative information on the embryos' development and viability. Hence, this technology opens what we believe to be novel opportunities for 3D label-free live-cell imaging of whole embryos or organoids over long observation times.

Quantitative imaging of Caenorhabditis elegans dauer larvae during cryptobiotic transition

Upon starvation or overcrowding, the nematode Caenorhabditis elegans enters diapause by forming a dauer larva, which can then further survive harsh desiccation in an anhydrobiotic state. We have previously identified the genetic and biochemical pathways essential for survival-but without detailed knowledge of their material properties, the mechanistic understanding of this intriguing phenomenon remains incomplete. Here we employed optical diffraction tomography (ODT) to quantitatively assess the internal mass density distribution of living larvae in the reproductive and diapause stages. ODT revealed that the properties of the dauer larvae undergo a dramatic transition upon harsh desiccation. Moreover, mutants that are sensitive to desiccation displayed structural abnormalities in the anhydrobiotic stage that could not be observed by conventional microscopy. Our advance opens a door to quantitatively assessing the transitions in material properties and structure necessary to fully understand an organism on the verge of life and death.

High-resolution 3D Fourier ptychographic reconstruction using a hemispherical illumination source with multiplexed-coded strategy

Fourier ptychography is a promising and flexible imaging technique that can achieve 2D quantitative reconstruction with higher resolution beyond the limitation of the system. Meanwhile, by using different imaging models, the same platform can be applied to achieve 3D refractive index reconstruction. To improve the illumination NA as much as possible while reducing the intensity attenuation problem caused by the LED board used in the traditional FP platform, we apply a hemispherical lighting structure and design a new LED arrangement according to 3D Fourier diffraction theory. Therefore, we could obtain the illumination of 0.98NA using 187 LEDs and achieve imaging half-pitch resolutions of ∼174 nm and ∼524 nm for the lateral and axial directions respectively, using a 40×/0.6NA objective lens. Furthermore, to reduce the number of captured images required and realize real-time data collection, we apply the multiplexed-coded illumination strategy and compare several coded patterns through simulation and experiment. Through comparison, we determined a radial-coded illumination pattern that could achieve more similar results as sequential scanning and increase the acquisition speed to above 1 Hz. Therefore, this paper provides the possibility of this technique in real-time 3D observation of in vitro live samples.

Towards machine learning for heterogeneous inverse scattering in 3D microscopy

Light propagating through a nonuniform medium scatters as it interacts with particles with different refractive properties such as cells in the tissue. In this work we aim to utilize this scattering process to learn a volumetric reconstruction of scattering parameters, in particular particle densities. We target microscopy applications where coherent speckle effects are an integral part of the imaging process. We argue that the key for successful learning is modeling realistic speckles in the training process. To this end, we build on the development of recent physically accurate speckle simulators. We also explore how to incorporate speckle statistics, such as the memory effect, in the learning framework. Overall, this paper contributes an analysis of multiple aspects of the network design including the learning architecture, the training data and the desired input features. We hope this study will pave the road for future design of learning based imaging systems in this challenging domain.

Self-calibrated 3D differential phase contrast microscopy with optimized illumination

3D phase imaging recovers an object's volumetric refractive index from intensity and/or holographic measurements. Partially coherent methods, such as illumination-based differential phase contrast (DPC), are particularly simple to implement in a commercial brightfield microscope. 3D DPC acquires images at multiple focus positions and with different illumination source patterns in order to reconstruct 3D refractive index. Here, we present a practical extension of the 3D DPC method that does not require a precise motion stage for scanning the focus and uses optimized illumination patterns for improved performance. The user scans the focus by hand, using the microscope's focus knob, and the algorithm self-calibrates the axial position to solve for the 3D refractive index of the sample through a computational inverse problem. We further show that the illumination patterns can be optimized by an end-to-end learning procedure. Combining these two, we demonstrate improved 3D DPC with a commercial microscope whose only hardware modification is LED array illumination.

Computational multi-wavelength phase synthesis using convolutional neural networks [Invited]

Multi-wavelength digital holographic microscopy (MWDHM) provides indirect measurements of the refractive index for non-dispersive samples. Successive-shot MWDHM is not appropriate for dynamic samples and single-shot MWDHM significantly increases the complexity of the optical setup due to the need for multiple lasers or a wavelength tunable source. Here we consider deep learning convolutional neural networks for computational phase synthesis to obtain high-speed simultaneous phase estimates on different wavelengths and thus single-shot estimates of the integral refractive index without increased experimental complexity. This novel, to the best of our knowledge, computational concept is validated using cell phantoms consisting of internal refractive index variations representing cytoplasm and membrane-bound organelles, respectively, and a simulation of a realistic holographic recording process. Specifically, in this work we employed data-driven computational techniques to perform accurate dual-wavelength hologram synthesis (hologram-to-hologram prediction), dual-wavelength phase synthesis (unwrapped phase-to-phase prediction), direct phase-to-index prediction using a single wavelength, hologram-to-phase prediction, and 2D phase unwrapping with sharp discontinuities (wrapped-to-unwrapped phase prediction).

Multiple-scattering suppressive refractive index tomography for the label-free quantitative assessment of multicellular spheroids

Refractive index (RI) tomography is a quantitative tomographic technique used to visualize the intrinsic contrast of unlabeled biological samples. Conventional RI reconstruction algorithms are based on weak-scattering approximation, such as the Born or Rytov approximation. Although these linear algorithms are computationally efficient, they are invalid when the fields are strongly distorted by multiple scattering (MS) of specimens. Herein, we propose an approach to reconstruct the RI distributions of MS objects even under weak-scattering approximation using an MS-suppressive operation. The operation converts the distorted fields into MS-suppressed fields, where weak-scattering approximation is applicable. Using this approach, we reconstructed a whole multicellular spheroid and successfully visualized its internal subcellular structures. Our work facilitates the realization of RI tomography of MS specimens and label-free quantitative analysis of 3D multicellular specimens.

Optical Diffraction Tomography Using Nearly In-Line Holography with a Broadband LED Source

We present optical tomography methods for a 3D refractive index reconstruction of weakly scattering objects using LED light sources. We are able to record holograms by minimizing the optical path difference between the signal and reference beams while separating the scattered field from its twin image. We recorded multiple holograms by illuminating the LEDs sequentially and reconstructed the 3D refractive index reconstruction of the sample. The reconstructions show high signal-to-noise ratio in which the effect of speckle artifacts is highly minimized due to the partially incoherent illumination of the LEDs. Results from combining different illumination wavelengths are also described demonstrating higher acquisition speed.

Correlative all-optical quantification of mass density and mechanics of subcellular compartments with fluorescence specificity

Quantitative measurements of physical parameters become increasingly important for understanding biological processes. Brillouin microscopy (BM) has recently emerged as one technique providing the 3D distribution of viscoelastic properties inside biological samples - so far relying on the implicit assumption that refractive index (RI) and density can be neglected. Here, we present a novel method (FOB microscopy) combining BM with optical diffraction tomography and epifluorescence imaging for explicitly measuring the Brillouin shift, RI, and absolute density with specificity to fluorescently labeled structures. We show that neglecting the RI and density might lead to erroneous conclusions. Investigating the nucleoplasm of wild-type HeLa cells, we find that it has lower density but higher longitudinal modulus than the cytoplasm. Thus, the longitudinal modulus is not merely sensitive to the water content of the sample - a postulate vividly discussed in the field. We demonstrate the further utility of FOB on various biological systems including adipocytes and intracellular membraneless compartments. FOB microscopy can provide unexpected scientific discoveries and shed quantitative light on processes such as phase separation and transition inside living cells.

Local Surface Chemistry Dynamically Monitored by Quantitative Phase Microscopy

Surface modification by photo grafting constitutes an interesting strategy to prepare functional surfaces. Precision applications, however, demand quantitative methods able to monitor and control the amount and distribution of surface modifications, which is hard to achieve, particularly in operando conditions. In this paper, a label-free, cost-effective, all-optical method based on wavefront sensing which is able to quantitatively track the evolution of grafted layers in real-time, is presented. By positioning a simple thin diffuser in the close vicinity of a camera, the thickness of grafted patterns is directly evaluated with sub-nanometric sensitivity and diffraction-limited lateral resolution. By performing an in-depth kinetic analysis of the local modification of an inert substrate (glass cover slips) through photografting of arydiazonium salts, different growth regimes are characterized and several parameters are estimated, such as the grafting efficiency, density and the apparent refractive index distribution of the resulting grafted layers. Both focused and widefield-grafting can be quantitatively monitored in real time, providing valuable guidelines to maximize functionalization efficiency. The association of a well-characterized versatile photografting reaction with the proposed flexible and sensitive monitoring strategy enables functional surfaces to be prepared, and puts surface micro- to submicro-structuration within the reach of most laboratories.