Machine learning holography for 3D particle field imaging

Generalizable deep learning approach for 3D particle imaging using holographic microscopy (HM)

Despite its potential for label-free particle diagnostics, holographic microscopy is limited by specialized processing methods that struggle to generalize across diverse settings. We introduce a deep learning architecture leveraging human perception of longitudinal variation of diffracted patterns of particles, which enables highly generalizable analysis of 3D particle information with orders of magnitude improvement in processing speed. Trained with minimal synthetic and real holograms of simple particles, our method demonstrates exceptional performance across various challenging cases, including high particle concentrations, significant noise, and a wide range of particle sizes, complex shapes, and optical properties, exceeding the diversity of training datasets.

Digital in-line holographic microscopy for label-free identification and tracking of biological cells

Digital in-line holographic microscopy (DIHM) is a non-invasive, real-time, label-free technique that captures three-dimensional (3D) positional, orientational, and morphological information from digital holographic images of living biological cells. Unlike conventional microscopies, the DIHM technique enables precise measurements of dynamic behaviors exhibited by living cells within a 3D volume. This review outlines the fundamental principles and comprehensive digital image processing procedures employed in DIHM-based cell tracking methods. In addition, recent applications of DIHM technique for label-free identification and digital tracking of various motile biological cells, including human blood cells, spermatozoa, diseased cells, and unicellular microorganisms, are thoroughly examined. Leveraging artificial intelligence has significantly enhanced both the speed and accuracy of digital image processing for cell tracking and identification. The quantitative data on cell morphology and dynamics captured by DIHM can effectively elucidate the underlying mechanisms governing various microbial behaviors and contribute to the accumulation of diagnostic databases and the development of clinical treatments.

Real-Time 3D Tracking of Multi-Particle in the Wide-Field Illumination Based on Deep Learning

In diverse realms of research, such as holographic optical tweezer mechanical measurements, colloidal particle motion state examinations, cell tracking, and drug delivery, the localization and analysis of particle motion command paramount significance. Algorithms ranging from conventional numerical methods to advanced deep-learning networks mark substantial strides in the sphere of particle orientation analysis. However, the need for datasets has hindered the application of deep learning in particle tracking. In this work, we elucidated an efficacious methodology pivoted toward generating synthetic datasets conducive to this domain that resonates with robustness and precision when applied to real-world data of tracking 3D particles. We developed a 3D real-time particle positioning network based on the CenterNet network. After conducting experiments, our network has achieved a horizontal positioning error of 0.0478 μm and a z-axis positioning error of 0.1990 μm. It shows the capability to handle real-time tracking of particles, diverse in dimensions, near the focal plane with high precision. In addition, we have rendered all datasets cultivated during this investigation accessible.

Identification method of raindrops and hailstones based on digital holographic interference

The identification of raindrops and hailstones is of great significance to the study of precipitation characteristics from the aspect of microphysics and can provide important data support for weather modification. In this paper, an identification method of raindrops and hailstones based on digital holographic interference is proposed. The grayscale gradient variance method is used to obtain the focus position of the particles. By means of binarization and morphological processing, digital holograms are processed to obtain clear profiles of the particles. Then the contour parameters of the particles are used to obtain the equivalent volume diameter and roundness. Finally, according to the equivalent volume diameter, roundness and lens-like effect of the particles, the phase states of the raindrop and hailstone are identified by the algorithm. Experiments show that the method proposed in this paper has a good identification effect on raindrops and hailstones. The research results can provide reference for the research of the identification method of raindrops and hailstones and the acquisition of accurate characteristic parameters.

Mimicking non-ideal instrument behavior for hologram processing using neural style translation

Holographic cloud probes provide unprecedented information on cloud particle density, size and position. Each laser shot captures particles within a large volume, where images can be computationally refocused to determine particle size and location. However, processing these holograms with standard methods or machine learning (ML) models requires considerable computational resources, time and occasional human intervention. ML models are trained on simulated holograms obtained from the physical model of the probe since real holograms have no absolute truth labels. Using another processing method to produce labels would be subject to errors that the ML model would subsequently inherit. Models perform well on real holograms only when image corruption is performed on the simulated images during training, thereby mimicking non-ideal conditions in the actual probe. Optimizing image corruption requires a cumbersome manual labeling effort. Here we demonstrate the application of the neural style translation approach to the simulated holograms. With a pre-trained convolutional neural network, the simulated holograms are "stylized" to resemble the real ones obtained from the probe, while at the same time preserving the simulated image "content" (e.g. the particle locations and sizes). With an ML model trained to predict particle locations and shapes on the stylized data sets, we observed comparable performance on both simulated and real holograms, obviating the need to perform manual labeling. The described approach is not specific to holograms and could be applied in other domains for capturing noise and imperfections in observational instruments to make simulated data more like real world observations.

Machine learning enables precise holographic characterization of colloidal materials in real time

Holographic particle characterization uses in-line holographic video microscopy to track and characterize individual colloidal particles dispersed in their native fluid media. Applications range from fundamental research in statistical physics to product development in biopharmaceuticals and medical diagnostic testing. The information encoded in a hologram can be extracted by fitting to a generative model based on the Lorenz-Mie theory of light scattering. Treating hologram analysis as a high-dimensional inverse problem has been exceptionally successful, with conventional optimization algorithms yielding nanometer precision for a typical particle's position and part-per-thousand precision for its size and index of refraction. Machine learning previously has been used to automate holographic particle characterization by detecting features of interest in multi-particle holograms and estimating the particles' positions and properties for subsequent refinement. This study presents an updated end-to-end neural-network solution called CATCH (Characterizing and Tracking Colloids Holographically) whose predictions are fast, precise, and accurate enough for many real-world high-throughput applications and can reliably bootstrap conventional optimization algorithms for the most demanding applications. The ability of CATCH to learn a representation of Lorenz-Mie theory that fits within a diminutive 200 kB hints at the possibility of developing a greatly simplified formulation of light scattering by small objects.

Ultraviolet digital holographic microscopy (DHM) of micron-scale particles from shocked Sn ejecta

A cloud of very fast, O(km/s), and very fine, O(µm), particles may be ejected when a strong shock impacts and possibly melts the free surface of a solid metal. To quantify these dynamics, this work develops an ultraviolet, long-working distance, two-pulse Digital Holographic Microscopy (DHM) configuration and is the first to replace film recording with digital sensors for this challenging application. A proposed multi-iteration DHM processing algorithm is demonstrated for automated measures of the sizes, velocities, and three-dimensional positions of non-spherical particles. Ejecta as small as 2 µm diameter are successfully tracked, while uncertainty simulations indicate that particle size distributions are accurately quantified for diameters ≥4 µm. These techniques are demonstrated on three explosively driven experiments. Measured ejecta size and velocity statistics are shown to be consistent with prior film-based recording, while also revealing spatial variations in velocities and 3D positions that have yet to be widely investigated. Having eliminated time-consuming analog film processing, the methodologies proposed here are expected to significantly accelerate future experimental investigation of ejecta physics.

Smartphone-based holographic measurement of polydisperse suspended particulate matter with various mass concentration ratios

Real-time monitoring of suspended particulate matter (PM) has become essential in daily life due to the adverse effects of long-term exposure to PMs on human health and ecosystems. However, conventional techniques for measuring micro-scale particulates commonly require expensive instruments. In this study, a smartphone-based device is developed for real-time monitoring of suspended PMs by integrating a smartphone-based digital holographic microscopy (S-DHM) and deep learning algorithms. The proposed S-DHM-based PM monitoring device is composed of affordable commercial optical components and a smartphone. Overall procedures including digital image processing, deep learning training, and correction process are optimized to minimize the prediction error and computational cost. The proposed device can rapidly measure the mass concentrations of coarse and fine PMs from holographic speckle patterns of suspended polydisperse PMs in water with measurement errors of 22.8 ± 18.1% and 13.5 ± 9.8%, respectively. With further advances in data acquisition and deep learning training, this study would contribute to the development of hand-held devices for monitoring polydisperse non-spherical pollutants suspended in various media.

Accurate automatic object 4D tracking in digital in-line holographic microscopy based on computationally rendered dark fields

Building on Gabor seminal principle, digital in-line holographic microscopy provides efficient means for space-time investigations of large volumes of interest. Thus, it has a pivotal impact on particle tracking that is crucial in advancing various branches of science and technology, e.g., microfluidics and biophysical processes examination (cell motility, migration, interplay etc.). Well-established algorithms often rely on heavily regularized inverse problem modelling and encounter limitations in terms of tracking accuracy, hologram signal-to-noise ratio, accessible object volume, particle concentration and computational burden. This work demonstrates the DarkTrack algorithm-a new approach to versatile, fast, precise, and robust 4D holographic tracking based on deterministic computationally rendered high-contrast dark fields. Its unique capabilities are quantitatively corroborated employing a novel numerical engine for simulating Gabor holographic recording of time-variant volumes filled with predefined dynamic particles. Our solution accounts for multiple scattering and thus it is poised to secure an important gap in holographic particle tracking technology and allow for ground-truth-driven benchmarking and quantitative assessment of tracking algorithms. Proof-of-concept experimental evaluation of DarkTrack is presented via analyzing live spermatozoa. Software supporting both novel numerical holographic engine and DarkTrack algorithm is made open access, which opens new possibilities and sets the stage for democratization of robust holographic 4D particle examination.

Learning-based complex field recovery from digital hologram with various depth objects

In this paper, we investigate a learning-based complex field recovery technique of an object from its digital hologram. Most of the previous learning-based approaches first propagate the captured hologram to the object plane and then suppress the DC and conjugate noise in the reconstruction. To the contrary, the proposed technique utilizes a deep learning network to extract the object complex field in the hologram plane directly, making it robust to the object depth variations and well suited for three-dimensional objects. Unlike the previous approaches which concentrate on transparent biological samples having near-uniform amplitude, the proposed technique is applied to more general objects which have large amplitude variations. The proposed technique is verified by numerical simulations and optical experiments, demonstrating its feasibility.

Investigation of cloud droplets velocity extraction based on depth expansion and self-fusion of reconstructed hologram

The velocity of cloud droplets has a significant effect on the investigation of the turbulence-cloud microphysics interaction mechanism. The paper proposes an in-line digital holographic interferometry (DHI) technique based on depth expansion and self-fusion algorithm to simultaneously extract particle velocity from eight holograms. In comparison to the two-frame exposure method, the extraction efficiency of velocity is raised by threefold, and the number of reference particles used for particle registration is increased to eight. The experimental results obtained in the cloud chamber show that the velocity of cloud droplets increases fourfold from the stabilization phase to the dissipation phase. The measurement deviations of two phases are 1.138 and 1.153 mm/s, respectively. Additionally, this method provides a rapid solution for three-dimensional particle velocimetry investigation of turbulent field stacking and cloud droplets collisions.

Holographic 3D particle reconstruction using a one-stage network

Volumetric reconstruction of a three-dimensional (3D) particle field with high resolution and low latency is an ambitious and valuable task. As a compact and high-throughput imaging system, digital holography (DH) encodes the 3D information of a particle volume into a two-dimensional (2D) interference pattern. In this work, we propose a one-stage network (OSNet) for 3D particle volumetric reconstruction. Specifically, by a single feed-forward process, OSNet can retrieve the 3D coordinates of the particles directly from the holograms without high-fidelity image reconstruction at each depth slice. Evaluation results from both synthetic and experimental data confirm the feasibility and robustness of our method under different particle concentrations and noise levels in terms of detection rate and position accuracy, with improved processing speed. The additional applications of 3D particle tracking are also investigated, facilitating the analysis of the dynamic displacements and motions for micro-objects or cells. It can be further extended to various types of computational imaging problems sharing similar traits.

Deep learning for digital holography: a review

Recent years have witnessed the unprecedented progress of deep learning applications in digital holography (DH). Nevertheless, there remain huge potentials in how deep learning can further improve performance and enable new functionalities for DH. Here, we survey recent developments in various DH applications powered by deep learning algorithms. This article starts with a brief introduction to digital holographic imaging, then summarizes the most relevant deep learning techniques for DH, with discussions on their benefits and challenges. We then present case studies covering a wide range of problems and applications in order to highlight research achievements to date. We provide an outlook of several promising directions to widen the use of deep learning in various DH applications.

Machine Learning Assisted Classification of Cell Lines and Cell States on Quantitative Phase Images

In this report, we present implementation and validation of machine-learning classifiers for distinguishing between cell types (HeLa, A549, 3T3 cell lines) and states (live, necrosis, apoptosis) based on the analysis of optical parameters derived from cell phase images. Validation of the developed classifier shows the accuracy for distinguishing between the three cell types of about 93% and between different cell states of the same cell line of about 89%. In the field test of the developed algorithm, we demonstrate successful evaluation of the temporal dynamics of relative amounts of live, apoptotic and necrotic cells after photodynamic treatment at different doses.

Volumetric monitoring of airborne particulate matter concentration using smartphone-based digital holographic microscopy and deep learning

Airborne particulate matter (PM) has become a global environmental issue. This PM has harmful effects on public health and precision industries. Conventional air-quality monitoring methods usually utilize expensive equipment, and they are cumbersome to handle for accurate and high throughput measurements. In addition, commercial particle counters have technical limitations in high-concentration measurement, and data fluctuations are induced during air sampling. In this study, a novel smartphone-based technique for monitoring airborne PM concentrations was developed using smartphone-based digital holographic microscopy (S-DHM) and deep learning network called Holo-SpeckleNet. Holographic speckle images of various PM concentrations were recorded by the S-DHM system. The recorded speckle images and the corresponding ground truth PM concentrations were used to train deep learning algorithms consisting of a deep autoencoder and regression layers. The performance of the proposed smartphone-based PM monitoring technique was validated through hyperparameter optimization. The developed S-DHM integrated with Holo-SpeckleNet can be smartly and effectively utilized for portable PM monitoring and safety alarm provision under perilous environmental conditions.

Accurate real-time monitoring of high particulate matter concentration based on holographic speckles and deep learning

Accurate real-time monitoring of particulate matter (PM) has emerged as a global issue due to the hazardous effects of PM on public health and industry. However, conventional PM monitoring techniques are usually cumbersome and require expensive equipments. In this study, Holo-SpeckleNet is proposed as a fast and accurate PM concentration measurement technique with high throughput using a deep learning based holographic speckle pattern analysis. Speckle pattern datasets of PMs for a wide range of PM concentrations were acquired by using a digital in-line holography microscopy system. Deep autoencoder and regression algorithms were trained with the captured speckle pattern datasets to directly measure PM concentration from speckle pattern images without any air intake device and time-consuming post image processing. The proposed technique was applied to predict various PM concentrations using the test datasets, optimize hyperparameters, and compare its performance with a convolutional neural network (CNN) algorithm. As a result, high PM concentration values can be measured over air quality index of 150, above which human exposure is unhealthy. In addition, the proposed technique exhibits higher measurement accuracy and less overfitting than the CNN with a relative error of 7.46 ± 3.92%. It can be applied to design a compact air quality monitoring device for highly accurate and real-time measurement of PM concentrations under hazardous environment, such as factories or construction sites.

Influence of sparse constraint functions on compressive holographic tomography

In this paper, we quantified and analyzed the impact of the l₁ norm and total variation (TV) norm sparse constraints on the reconstruction quality under different interlayer spacings, sampling rates, and signal-to-noise ratios. For high-quality holograms, the results of compressive-sensing reconstruction using l₁ norm achieved higher quality than those by the TV norm. In contrast, for low-quality holograms, the quality of TV-norm-based reconstruction results was relatively stable and better than that of l₁ norm. In addition, we explained why interlayer spacing cannot be smaller and recommend the use of axial resolution of the digital holography system as the interlayer spacing. The conclusions are valuable in the choice of sparse constraints in compressive holographic tomography.

DNN-FZA camera: a deep learning approach toward broadband FZA lensless imaging

In mask-based lensless imaging, iterative reconstruction methods based on the geometric optics model produce artifacts and are computationally expensive. We present a prototype of a lensless camera that uses a deep neural network (DNN) to realize rapid reconstruction for Fresnel zone aperture (FZA) imaging. A deep back-projection network (DBPN) is connected behind a U-Net providing an error feedback mechanism, which realizes the self-correction of features to recover the image detail. A diffraction model generates the training data under conditions of broadband incoherent imaging. In the reconstructed results, blur caused by diffraction is shown to have been ameliorated, while the computing time is 2 orders of magnitude faster than the traditional iterative image reconstruction algorithms. This strategy could drastically reduce the design and assembly costs of cameras, paving the way for integration of portable sensors and systems.

2D vs 3D tracking in bacterial motility analysis

<abstract> <p>Digital holographic microscopy provides the ability to observe throughout a large volume without refocusing. This capability enables simultaneous observations of large numbers of microorganisms swimming in an essentially unconstrained fashion. However, computational tools for tracking large 4D datasets remain lacking. In this paper, we examine the errors introduced by tracking bacterial motion as 2D projections vs. 3D volumes under different circumstances: bacteria free in liquid media and bacteria near a glass surface. We find that while XYZ speeds are generally equal to or larger than XY speeds, they are still within empirical uncertainties. Additionally, when studying dynamic surface behavior, the Z coordinate cannot be neglected.</p> </abstract>

Pattern detection in colloidal assembly: A mosaic of analysis techniques

Characterization of the morphology, identification of patterns and quantification of order encountered in colloidal assemblies is essential for several reasons. First of all, it is useful to compare different self-assembly methods and assess the influence of different process parameters on the final colloidal pattern. In addition, casting light on the structures formed by colloidal particles can help to get better insight into colloidal interactions and understand phase transitions. Finally, the growing interest in colloidal assemblies in materials science for practical applications going from optoelectronics to biosensing imposes a thorough characterization of the morphology of colloidal assemblies because of the intimate relationship between morphology and physical properties (e.g. optical and mechanical) of a material. Several image analysis techniques developed to investigate images (acquired via scanning electron microscopy, digital video microscopy and other imaging methods) provide variegated and complementary information on the colloidal structures under scrutiny. However, understanding how to use such image analysis tools to get information on the characteristics of the colloidal assemblies may represent a non-trivial task, because it requires the combination of approaches drawn from diverse disciplines such as image processing, computational geometry and computational topology and their application to a primarily physico-chemical process. Moreover, the lack of a systematic description of such analysis tools makes it difficult to select the ones more suitable for the features of the colloidal assembly under examination. In this review we provide a methodical and extensive description of real-space image analysis tools by explaining their principles and their application to the investigation of two-dimensional colloidal assemblies with different morphological characteristics.

Point spread function in interferometric scattering microscopy (iSCAT). Part I: aberrations in defocusing and axial localization

Interferometric scattering (iSCAT) microscopy is an emerging label-free technique optimized for the sensitive detection of nano-matter. Previous iSCAT studies have approximated the point spread function in iSCAT by a Gaussian intensity distribution. However, recent efforts to track the mobility of nanoparticles in challenging speckle environments and over extended axial ranges has necessitated a quantitative description of the interferometric point spread function (iPSF). We present a robust vectorial diffraction model for the iPSF in tandem with experimental measurements and rigorous FDTD simulations. We examine the iPSF under various imaging scenarios to understand how aberrations due to the experimental configuration encode information about the nanoparticle. We show that the lateral shape of the iPSF can be used to achieve nanometric three-dimensional localization over an extended axial range on the order of 10 µm either by means of a fit to an analytical model or calibration-free unsupervised machine learning. Our results have immediate implications for three-dimensional single particle tracking in complex scattering media.

Hybrid remapping particle field reconstruction method for synthetic aperture particle image velocimetry

The flow field velocity is an important parameter for completely characterizing the topologies of unsteady coherent flow structures. Synthetic aperture (SA)-based particle image velocimetry (SAPIV) has been used for three-dimensional flow measurements, owing to its wide range of acceptable tracer particle intensities and ability to view partially occluded fields. However, SAPIV typically suffers from poor reconstruction quality for nonuniformly illuminated particle volumes. In this paper, we propose a hybrid remapping particle field reconstruction method for SAPIV in a nonuniformly illuminated fluid flow. Both additive and minimum line-of-sight remapping are used to reconstruct the in-focus particles from the refocused image stacks. The structural similarity between the images projected by the reconstructed particle field and the images captured by the cameras are used to determine the reconstruction quality. This method was verified by both synthetic simulation and an experimental implementation. The performance of the proposed technique was compared with existing methods. The proposed method has the best reconstruction quality and computational speed among the considered methods.

Reducing data acquisition for light-sheet microscopy by extrapolation between imaged planes

Light-sheet fluorescence microscopy (LSFM) is a powerful technique that can provide high-resolution images of biological samples. Therefore, this technique offers significant improvement for three-dimensional (3D) imaging of living cells. However, producing high-resolution 3D images of a single cell or biological tissues, normally requires high acquisition rate of focal planes, which means a large amount of sample sections. Consequently, it consumes a vast amount of processing time and memory, especially when studying real-time processes inside living cells. We describe an approach to minimize data acquisition by interpolation between planes using a phase retrieval algorithm. We demonstrate this approach on LSFM data sets and show reconstruction of intermediate sections of the sparse samples. Since this method diminishes the required amount of acquisition focal planes, it also reduces acquisition time of samples as well. Our suggested method has proven to reconstruct unacquired intermediate planes from diluted data sets up to 10× fold. The reconstructed planes were found correlated to the original preacquired samples (control group) with correlation coefficient of up to 90%. Given the findings, this procedure appears to be a powerful method for inquiring and analyzing biological samples.