Efficient image reconstruction of high-density molecules with augmented Lagrangian method in super-resolution microscopy

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Potential quality improvement of stochastic optical localization nanoscopy images obtained by frame by frame localization algorithms

A data movie of stochastic optical localization nanoscopy contains spatial and temporal correlations, both providing information of emitter locations. The majority of localization algorithms in the literature estimate emitter locations by frame-by-frame localization (FFL), which exploit only the spatial correlation and leave the temporal correlation into the FFL nanoscopy images. The temporal correlation contained in the FFL images, if exploited, can improve the localization accuracy and the image quality. In this paper, we analyze the properties of the FFL images in terms of root mean square minimum distance (RMSMD) and root mean square error (RMSE). It is shown that RMSMD and RMSE can be potentially reduced by a maximum fold equal to the square root of the average number of activations per emitter. Analyzed and revealed are also several statistical properties of RMSMD and RMSE and their relationship with respect to a large number of data frames, bias and variance of localization errors, small localization errors, sample drift, and the worst FFL image. Numerical examples are taken and the results confirm the prediction of analysis. The ideas about how to develop an algorithm to exploit the temporal correlation of FFL images are also briefly discussed. The results suggest development of two kinds of localization algorithms: the algorithms that can exploit the temporal correlation of FFL images and the unbiased localization algorithms.

Image reconstruction with a deep convolutional neural network in high-density super-resolution microscopy

An accurate and fast reconstruction algorithm is crucial for the improvement of temporal resolution in high-density super-resolution microscopy, particularly in view of the challenges associated with live-cell imaging. In this work, we design a deep network based on a convolutional neural network to take advantage of its enhanced ability in high-density molecule localization, and introduce a residual layer into the network to reduce noise. The proposed scheme also incorporates robustness against variations of both the full width at half maximum (FWHM) and the pixel size. We validate our algorithm on both simulated and experimental data by achieving performance improvement in terms of loss value and image quality, and demonstrate live-cell imaging with temporal resolution of 0.5 seconds by recovering mitochondria dynamics.