Accurate phase retrieval of complex 3D point spread functions with deep residual neural networks

Universal inverse modeling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single-molecule localization, aberration correction and deconvolution. Here we present universal inverse modeling of point spread functions (uiPSF), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single-molecule localization microscopy (SMLM). Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system- or sample-specific characteristics, for example, the bead size, field- and depth- dependent aberrations, and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single-molecule blinking data.

Using Diffraction Deep Neural Networks for Indirect Phase Recovery Based on Zernike Polynomials

The phase recovery module is dedicated to acquiring phase distribution information within imaging systems, enabling the monitoring and adjustment of a system's performance. Traditional phase inversion techniques exhibit limitations, such as the speed of the sensor and complexity of the system. Therefore, we propose an indirect phase retrieval approach based on a diffraction neural network. By utilizing non-source diffraction through multiple layers of diffraction units, this approach reconstructs coefficients based on Zernike polynomials from incident beams with distorted phases, thereby indirectly synthesizing interference phases. Through network training and simulation testing, we validate the effectiveness of this approach, showcasing the trained network's capacity for single-order phase recognition and multi-order composite phase inversion. We conduct an analysis of the network's generalization and evaluate the impact of the network depth on the restoration accuracy. The test results reveal an average root mean square error of 0.086λ for phase inversion. This research provides new insights and methodologies for the development of the phase recovery component in adaptive optics systems.

Aberration Estimation for Synthetic Aperture Digital Holographic Microscope Using Deep Neural Network

Digital holographic microscopy (DHM) is a valuable technique for investigating the optical properties of samples through the measurement of intensity and phase of diffracted beams. However, DHMs are constrained by Lagrange invariance, compromising the spatial bandwidth product (SBP) which relates resolution and field of view. Synthetic aperture DHM (SA-DHM) was introduced to overcome this limitation, but it faces significant challenges such as aberrations in synthesizing the optical information corresponding to the steering angle of incident wave. This paper proposes a novel approach utilizing deep neural networks (DNNs) for compensating aberrations in SA-DHM, extending the compensation scope beyond the numerical aperture (NA) of the objective lens. The method involves training a DNN from diffraction patterns and Zernike coefficients through a circular aperture, enabling effective aberration compensation in the illumination beam. This method makes it possible to estimate aberration coefficients from the only part of the diffracted beam cutoff by the circular aperture mask. With the proposed technique, the simulation results present improved resolution and quality of sample images. The integration of deep neural networks with SA-DHM holds promise for advancing microscopy capabilities and overcoming existing limitations.

Universal adaptive optics for microscopy through embedded neural network control

The resolution and contrast of microscope imaging is often affected by aberrations introduced by imperfect optical systems and inhomogeneous refractive structures in specimens. Adaptive optics (AO) compensates these aberrations and restores diffraction limited performance. A wide range of AO solutions have been introduced, often tailored to a specific microscope type or application. Until now, a universal AO solution - one that can be readily transferred between microscope modalities - has not been deployed. We propose versatile and fast aberration correction using a physics-based machine learning assisted wavefront-sensorless AO control (MLAO) method. Unlike previous ML methods, we used a specially constructed neural network (NN) architecture, designed using physical understanding of the general microscope image formation, that was embedded in the control loop of different microscope systems. The approach means that not only is the resulting NN orders of magnitude simpler than previous NN methods, but the concept is translatable across microscope modalities. We demonstrated the method on a two-photon, a three-photon and a widefield three-dimensional (3D) structured illumination microscope. Results showed that the method outperformed commonly-used modal-based sensorless AO methods. We also showed that our ML-based method was robust in a range of challenging imaging conditions, such as 3D sample structures, specimen motion, low signal to noise ratio and activity-induced fluorescence fluctuations. Moreover, as the bespoke architecture encapsulated physical understanding of the imaging process, the internal NN configuration was no-longer a "black box", but provided physical insights on internal workings, which could influence future designs.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

Phase retrieval for the generation of arbitrary intensity distributions using an optofluidic phase shifter

An optofluidic phase shifter can be used to generate virtually arbitrary intensity patterns, but only if the phase shift generated by the controllably deformed fluidic surface can be appropriately defined. To enable this functionality, we present two phase retrieval algorithms based on neural networks and least-squares optimization which are used to determine the necessary phase profile to generate a desired target intensity pattern with high accuracy. We demonstrate the utility of the algorithms by showing experimentally the ability of an optofluidic phase shifter to generate arbitrary complex intensity distributions.

Combining deep learning approaches and point spread function engineering for simultaneous 3D position and 3D orientation measurements of fluorescent single molecules

Point Spread Function (PSF) engineering is an effective method to increase the sensitivity of single-molecule fluorescence images to specific parameters. Classical phase mask optimization approaches have enabled the creation of new PSFs that can achieve, for example, localization precision of a few nanometers axially over a capture range of several microns with bright emitters. However, for complex high-dimensional optimization problems, classical approaches are difficult to implement and can be very time-consuming for computation. The advent of deep learning methods and their application to single-molecule imaging has provided a way to solve these problems. Here, we propose to combine PSF engineering and deep learning approaches to obtain both an optimized phase mask and a neural network structure to obtain the 3D position and 3D orientation of fixed fluorescent molecules. Our approach allows us to obtain an axial localization precision around 30 nanometers, as well as an orientation precision around 5 degrees for orientations and positions over a one micron depth range for a signal-to-noise ratio consistent with what is typical in single-molecule cellular imaging experiments.

Deep learning-based adaptive optics for light sheet fluorescence microscopy

Light sheet fluorescence microscopy (LSFM) is a high-speed imaging technique that is often used to image intact tissue-cleared specimens with cellular or subcellular resolution. Like other optical imaging systems, LSFM suffers from sample-induced optical aberrations that decrement imaging quality. Optical aberrations become more severe when imaging a few millimeters deep into tissue-cleared specimens, complicating subsequent analyses. Adaptive optics are commonly used to correct sample-induced aberrations using a deformable mirror. However, routinely used sensorless adaptive optics techniques are slow, as they require multiple images of the same region of interest to iteratively estimate the aberrations. In addition to the fading of fluorescent signal, this is a major limitation as thousands of images are required to image a single intact organ even without adaptive optics. Thus, a fast and accurate aberration estimation method is needed. Here, we used deep-learning techniques to estimate sample-induced aberrations from only two images of the same region of interest in cleared tissues. We show that the application of correction using a deformable mirror greatly improves image quality. We also introduce a sampling technique that requires a minimum number of images to train the network. Two conceptually different network architectures are compared; one that shares convolutional features and another that estimates each aberration independently. Overall, we have presented an efficient way to correct aberrations in LSFM and to improve image quality.

Fast High-Resolution Phase Diversity Wavefront Sensing with L-BFGS Algorithm

The presence of manufacture error in large mirrors introduces high-order aberrations, which can severely influence the intensity distribution of point spread function. Therefore, high-resolution phase diversity wavefront sensing is usually needed. However, high-resolution phase diversity wavefront sensing is restricted with the problem of low efficiency and stagnation. This paper proposes a fast high-resolution phase diversity method with limited memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm, which can accurately detect aberrations in the presence of high-order aberrations. An analytical gradient of the objective function for phase-diversity is integrated into the framework of the L-BFGS nonlinear optimization algorithm. L-BFGS algorithm is specifically suitable for high-resolution wavefront sensing where a large phase matrix is optimized. The performance of phase diversity with L-BFGS is compared to other iterative method through simulations and a real experiment. This work contributes to fast high-resolution image-based wavefront sensing with a high robustness.

Adaptive optics for optical microscopy [Invited]

Optical microscopy is widely used to visualize fine structures. When applied to bioimaging, its performance is often degraded by sample-induced aberrations. In recent years, adaptive optics (AO), originally developed to correct for atmosphere-associated aberrations, has been applied to a wide range of microscopy modalities, enabling high- or super-resolution imaging of biological structure and function in complex tissues. Here, we review classic and recently developed AO techniques and their applications in optical microscopy.

Machine-learning-powered extraction of molecular diffusivity from single-molecule images for super-resolution mapping

While critical to biological processes, molecular diffusion is difficult to quantify, and spatial mapping of local diffusivity is even more challenging. Here we report a machine-learning-enabled approach, pixels-to-diffusivity (Pix2D), to directly extract the diffusion coefficient D from single-molecule images, and consequently enable super-resolved D spatial mapping. Working with single-molecule images recorded at a fixed framerate under typical single-molecule localization microscopy (SMLM) conditions, Pix2D exploits the often undesired yet evident motion blur, i.e., the convolution of single-molecule motion trajectory during the frame recording time with the diffraction-limited point spread function (PSF) of the microscope. Whereas the stochastic nature of diffusion imprints diverse diffusion trajectories to different molecules diffusing at the same given D, we construct a convolutional neural network (CNN) model that takes a stack of single-molecule images as the input and evaluates a D-value as the output. We thus validate robust D evaluation and spatial mapping with simulated data, and with experimental data successfully characterize D differences for supported lipid bilayers of different compositions and resolve gel and fluidic phases at the nanoscale.

Direct retrieval of Zernike-based pupil functions using integrated diffractive deep neural networks

Retrieving the pupil phase of a beam path is a central problem for optical systems across scales, from telescopes, where the phase information allows for aberration correction, to the imaging of near-transparent biological samples in phase contrast microscopy. Current phase retrieval schemes rely on complex digital algorithms that process data acquired from precise wavefront sensors, reconstructing the optical phase information at great expense of computational resources. Here, we present a compact optical-electronic module based on multi-layered diffractive neural networks printed on imaging sensors, capable of directly retrieving Zernike-based pupil phase distributions from an incident point spread function. We demonstrate this concept numerically and experimentally, showing the direct pupil phase retrieval of superpositions of the first 14 Zernike polynomials. The integrability of the diffractive elements with CMOS sensors shows the potential for the direct extraction of the pupil phase information from a detector module without additional digital post-processing.

Deep learning in single-molecule imaging and analysis: recent advances and prospects

Single-molecule microscopy is advantageous in characterizing heterogeneous dynamics at the molecular level. However, there are several challenges that currently hinder the wide application of single molecule imaging in bio-chemical studies, including how to perform single-molecule measurements efficiently with minimal run-to-run variations, how to analyze weak single-molecule signals efficiently and accurately without the influence of human bias, and how to extract complete information about dynamics of interest from single-molecule data. As a new class of computer algorithms that simulate the human brain to extract data features, deep learning networks excel in task parallelism and model generalization, and are well-suited for handling nonlinear functions and extracting weak features, which provide a promising approach for single-molecule experiment automation and data processing. In this perspective, we will highlight recent advances in the application of deep learning to single-molecule studies, discuss how deep learning has been used to address the challenges in the field as well as the pitfalls of existing applications, and outline the directions for future development.

Jitter-Robust Phase Retrieval Wavefront Sensing Algorithms

Phase retrieval wavefront sensing methods are now of importance for imaging quality maintenance of space telescopes. However, their accuracy is susceptible to line-of-sight jitter due to the micro-vibration of the platform, which changes the intensity distribution of the image. The effect of the jitter shows some stochastic properties and it is hard to present an analytic solution to this problem. This paper establishes a framework for jitter-robust image-based wavefront sensing algorithm, which utilizes two-dimensional Gaussian convolution to describe the effect of jitter on an image. On this basis, two classes of jitter-robust phase retrieval algorithms are proposed, which can be categorized into iterative-transform algorithms and parametric algorithms, respectively. Further discussions are presented for the cases where the magnitude of jitter is unknown to us. Detailed simulations and a real experiment are performed to demonstrate the effectiveness and practicality of the proposed approaches. This work improves the accuracy and practicality of the phase retrieval wavefront sensing methods in the space condition with non-ignorable micro-vibration.

High-accuracy, direct aberration determination using self-attention-armed deep convolutional neural networks

Optical microscopes have long been essential for many scientific disciplines. However, the resolution and contrast of such microscopic images are dramatically affected by aberrations. In this study, compacted with adaptive optics, we propose a machine learning technique, called the "phase-retrieval deep convolutional neural networks (PRDCNNs)". This aberration determination architecture is direct and exhibits high accuracy and certain generalization ability. Notably, its performance surpasses those of similar, existing methods, with fewer fluctuations and greater robustness against noise. We anticipate future application of the proposed PRDCNNs to super-resolution microscopes. This article is protected by copyright. All rights reserved.

Light Sheet Illumination for 3D Single-Molecule Super-Resolution Imaging of Neuronal Synapses

The function of the neuronal synapse depends on the dynamics and interactions of individual molecules at the nanoscale. With the development of single-molecule super-resolution microscopy over the last decades, researchers now have a powerful and versatile imaging tool for mapping the molecular mechanisms behind the biological function. However, imaging of thicker samples, such as mammalian cells and tissue, in all three dimensions is still challenging due to increased fluorescence background and imaging volumes. The combination of single-molecule imaging with light sheet illumination is an emerging approach that allows for imaging of biological samples with reduced fluorescence background, photobleaching, and photodamage. In this review, we first present a brief overview of light sheet illumination and previous super-resolution techniques used for imaging of neurons and synapses. We then provide an in-depth technical review of the fundamental concepts and the current state of the art in the fields of three-dimensional single-molecule tracking and super-resolution imaging with light sheet illumination. We review how light sheet illumination can improve single-molecule tracking and super-resolution imaging in individual neurons and synapses, and we discuss emerging perspectives and new innovations that have the potential to enable and improve single-molecule imaging in brain tissue.

Deep learning wavefront sensing for fine phasing of segmented mirrors

Segmented primary mirror provides many crucial important advantages for the construction of extra-large space telescopes. The imaging quality of this class of telescope is susceptible to phasing error between primary mirror segments. Deep learning has been widely applied in the field of optical imaging and wavefront sensing, including phasing segmented mirrors. Compared to other image-based phasing techniques, such as phase retrieval and phase diversity, deep learning has the advantage of high efficiency and free of stagnation problem. However, at present deep learning methods are mainly applied to coarse phasing and used to estimate piston error between segments. In this paper, deep Bi-GRU neural work is introduced to fine phasing of segmented mirrors, which not only has a much simpler structure than CNN or LSTM network, but also can effectively solve the gradient vanishing problem in training due to long term dependencies. By incorporating phasing errors (piston and tip-tilt errors), some low-order aberrations as well as other practical considerations, Bi-GRU neural work can effectively be used for fine phasing of segmented mirrors. Simulations and real experiments are used to demonstrate the accuracy and effectiveness of the proposed methods.

Alignment of electron optical beam shaping elements using a convolutional neural network

A convolutional neural network is used to align an orbital angular momentum sorter in a transmission electron microscope. The method is demonstrated using simulations and experiments. As a result of its accuracy and speed, it offers the possibility of real-time tuning of other electron optical devices and electron beam shaping configurations.

Super-resolution Microscopy with Single Molecules in Biology and Beyond-Essentials, Current Trends, and Future Challenges

Single-molecule super-resolution microscopy has developed from a specialized technique into one of the most versatile and powerful imaging methods of the nanoscale over the past two decades. In this perspective, we provide a brief overview of the historical development of the field, the fundamental concepts, the methodology required to obtain maximum quantitative information, and the current state of the art. Then, we will discuss emerging perspectives and areas where innovation and further improvement are needed. Despite the tremendous progress, the full potential of single-molecule super-resolution microscopy is yet to be realized, which will be enabled by the research ahead of us.

Practical sensorless aberration estimation for 3D microscopy with deep learning

Estimation of optical aberrations from volumetric intensity images is a key step in sensorless adaptive optics for 3D microscopy. Recent approaches based on deep learning promise accurate results at fast processing speeds. However, collecting ground truth microscopy data for training the network is typically very difficult or even impossible thereby limiting this approach in practice. Here, we demonstrate that neural networks trained only on simulated data yield accurate predictions for real experimental images. We validate our approach on simulated and experimental datasets acquired with two different microscopy modalities and also compare the results to non-learned methods. Additionally, we study the predictability of individual aberrations with respect to their data requirements and find that the symmetry of the wavefront plays a crucial role. Finally, we make our implementation freely available as open source software in Python.

Correction of refractive index mismatch-induced aberrations under radially polarized illumination by deep learning

Radially polarized field under strong focusing has emerged as a powerful manner for fluorescence microscopy. However, the refractive index (RI) mismatch-induced aberrations seriously degrade imaging performance, especially under high numerical aperture (NA). Traditional adaptive optics (AO) method is limited by its tedious procedure. Here, we present a computational strategy that uses artificial neural networks to correct the aberrations induced by RI mismatch. There are no requirements for expensive hardware and complicated wavefront sensing in our framework when the deep network training is completed. The structural similarity index (SSIM) criteria and spatial frequency spectrum analysis demonstrate that our deep-learning-based method has a better performance compared to the widely used Richardson-Lucy (RL) deconvolution method at different imaging depth on simulation data. Additionally, the generalization of our trained network model is tested on new types of samples that are not present in the training procedure to further evaluate the utility of the network, and the performance is also superior to RL deconvolution.

Addressing systematic errors in axial distance measurements in single-emitter localization microscopy

Nanoscale localization of point emitters is critical to several methods in optical fluorescence microscopy, including single-molecule super-resolution imaging and tracking. While the precision of the localization procedure has been the topic of extensive study, localization accuracy has been less emphasized, in part due to the challenge of producing an experimental sample containing unperturbed point emitters at known three-dimensional positions in a relevant geometry. We report a new experimental system which reproduces a widely-adopted geometry in high-numerical aperture localization microscopy, in which molecules are situated in an aqueous medium above a glass coverslip imaged with an oil-immersion objective. We demonstrate a calibration procedure that enables measurement of the depth-dependent point spread function (PSF) for open aperture imaging as well as imaging with engineered PSFs with index mismatch. We reveal the complicated, depth-varying behavior of the focal plane position in this system and discuss the axial localization biases incurred by common approximations of this behavior. We compare our results to theoretical calculations.

Direct determination of aberration functions in microscopy by an artificial neural network

Adaptive optics relies on the fast and accurate determination of aberrations but is often hindered by wavefront sensor limitations or lengthy optimization algorithms. Deep learning by artificial neural networks has recently been shown to provide determination of aberration coefficients from various microscope metrics. Here we numerically investigate the direct determination of aberration functions in the pupil plane of a high numerical aperture microscope using an artificial neural network. We show that an aberration function can be determined from fluorescent guide stars and used to improve the Strehl ratio without the need for reconstruction from Zernike polynomial coefficients.

VIPR: vectorial implementation of phase retrieval for fast and accurate microscopic pixel-wise pupil estimation

In microscopy, proper modeling of the image formation has a substantial effect on the precision and accuracy in localization experiments and facilitates the correction of aberrations in adaptive optics experiments. The observed images are subject to polarization effects, refractive index variations, and system specific constraints. Previously reported techniques have addressed these challenges by using complicated calibration samples, computationally heavy numerical algorithms, and various mathematical simplifications. In this work, we present a phase retrieval approach based on an analytical derivation of the vectorial diffraction model. Our method produces an accurate estimate of the system's phase information, without any prior knowledge about the aberrations, in under a minute.

Deep learning in single-molecule microscopy: fundamentals, caveats, and recent developments [Invited]

Deep learning-based data analysis methods have gained considerable attention in all fields of science over the last decade. In recent years, this trend has reached the single-molecule community. In this review, we will survey significant contributions of the application of deep learning in single-molecule imaging experiments. Additionally, we will describe the historical events that led to the development of modern deep learning methods, summarize the fundamental concepts of deep learning, and highlight the importance of proper data composition for accurate, unbiased results.