Sparse deconvolution of high-density super-resolution images

Super-Resolution Stimulated Raman Scattering Microscopy with Graphical User Interface-Supported A-PoD

Raman microscopy is a vibrational imaging technology that can detect molecular chemical bond vibrational signals. Since this signal is originated from almost every vibrational mode of molecules with different vibrational energy levels, it provides spatiotemporal distribution of various molecules in living organisms without the need for any labeling. The limitations of low signal strength in Raman microscopy have been effectively addressed by incorporating a stimulated emission process, leading to the development of stimulated Raman scattering (SRS) microscopy. Furthermore, the issue of low spatial resolution has been resolved through the application of computational techniques, specifically image deconvolution. In this article, we present a comprehensive guide to super-resolution SRS microscopy using an Adam-based pointillism deconvolution (A-PoD) algorithm, complemented by a user-friendly graphical user interface (GUI). We delve into the crucial parameters and conditions necessary for achieving super-resolved images through SRS imaging. Additionally, we provide a step-by-step walkthrough of the preprocessing steps and the use of GUI-supported A-PoD. This complete package offers a user-friendly platform for super-resolution SRS microscopy, enhancing the versatility and applicability of this advanced microscopy technique to reveal nanoscopic multimolecular nature. © 2024 The Authors. Current Protocols published by Wiley Periodicals LLC. Basic Protocol: Super-resolution stimulated Raman scattering microscopy with graphical user interface-supported A-PoD Support Protocol: Deuterium labeling on cells with heavy water for metabolic imaging.

Resolution enhancement with deblurring by pixel reassignment (DPR)

Improving the spatial resolution of a fluorescence microscope has been an ongoing challenge in the imaging community. To address this challenge, a variety of approaches have been taken, ranging from instrumentation development to image post-processing. An example of the latter is deconvolution, where images are numerically deblurred based on a knowledge of the microscope point spread function. However, deconvolution can easily lead to noise-amplification artifacts. Deblurring by post-processing can also lead to negativities or fail to conserve local linearity between sample and image. We describe here a simple image deblurring algorithm based on pixel reassignment that inherently avoids such artifacts and can be applied to general microscope modalities and fluorophore types. Our algorithm helps distinguish nearby fluorophores even when these are separated by distances smaller than the conventional resolution limit, helping facilitate, for example, the application of single-molecule localization microscopy in dense samples. We demonstrate the versatility and performance of our algorithm under a variety of imaging conditions.

Super-resolution SRS microscopy with A-PoD

Stimulated Raman scattering (SRS) offers the ability to image metabolic dynamics with high signal-to-noise ratio. However, its spatial resolution is limited by the numerical aperture of the imaging objective and the scattering cross-section of molecules. To achieve super-resolved SRS imaging, we developed a deconvolution algorithm, adaptive moment estimation (Adam) optimization-based pointillism deconvolution (A-PoD) and demonstrated a spatial resolution of lower than 59 nm on the membrane of a single lipid droplet (LD). We applied A-PoD to spatially correlated multiphoton fluorescence imaging and deuterium oxide (D2O)-probed SRS (DO-SRS) imaging from diverse samples to compare nanoscopic distributions of proteins and lipids in cells and subcellular organelles. We successfully differentiated newly synthesized lipids in LDs using A-PoD-coupled DO-SRS. The A-PoD-enhanced DO-SRS imaging method was also applied to reveal metabolic changes in brain samples from Drosophila on different diets. This new approach allows us to quantitatively measure the nanoscopic colocalization of biomolecules and metabolic dynamics in organelles.

State-of-the-Art Approaches for Image Deconvolution Problems, including Modern Deep Learning Architectures

In modern digital microscopy, deconvolution methods are widely used to eliminate a number of image defects and increase resolution. In this review, we have divided these methods into classical, deep learning-based, and optimization-based methods. The review describes the major architectures of neural networks, such as convolutional and generative adversarial networks, autoencoders, various forms of recurrent networks, and the attention mechanism used for the deconvolution problem. Special attention is paid to deep learning as the most powerful and flexible modern approach. The review describes the major architectures of neural networks used for the deconvolution problem. We describe the difficulties in their application, such as the discrepancy between the standard loss functions and the visual content and the heterogeneity of the images. Next, we examine how to deal with this by introducing new loss functions, multiscale learning, and prior knowledge of visual content. In conclusion, a review of promising directions and further development of deconvolution methods in microscopy is given.

3DM: deep decomposition and deconvolution microscopy for rapid neural activity imaging

We report the development of deep decomposition and deconvolution microscopy (3DM), a computational microscopy method for the volumetric imaging of neural activity. 3DM overcomes the major challenge of deconvolution microscopy, the ill-posed inverse problem. We take advantage of the temporal sparsity of neural activity to reformulate and solve the inverse problem using two neural networks which perform sparse decomposition and deconvolution. We demonstrate the capability of 3DM via in vivo imaging of the neural activity of a whole larval zebrafish brain with a field of view of 1040 µm × 400 µm × 235 µm and with estimated lateral and axial resolutions of 1.7 µm and 5.4 µm, respectively, at imaging rates of up to 4.2 volumes per second.

Parametric comparison between sparsity-based and deep learning-based image reconstruction of super-resolution fluorescence microscopy

Sparsity-based and deep learning-based image reconstruction algorithms are two promising approaches to accelerate the image acquisition process for localization-based super-resolution microscopy, by allowing a higher density of fluorescing emitters to be imaged in a single frame. Despite the surging popularity, a comprehensive parametric study guiding the practical applications of sparsity-based and deep learning-based image reconstruction algorithms is yet to be conducted. In this study, we examined the performance of sparsity- and deep learning-based algorithms in reconstructing super-resolution images using simulated fluorescent microscopy images. The simulated images were synthesized with varying levels of sparsity and connectivity. We found the deep learning-based VDSR recovers image faster, with a higher recall rate and localization accuracy. The sparsity-based SPIDER recovers more zero pixels truthfully. We also compared the two algorithms using images acquired from a real super-resolution experiment, yielding results agreeing with the results from the evaluation using simulated images. We concluded that VDSR is preferable when accurate emitter localization is needed while SPIDER is more suitable when evaluation of the number of emitters is critical.

Design of experiments for the optimization of SOFI super-resolution microscopy imaging

Super-resolution optical fluctuation imaging (SOFI) is a well-known super-resolution technique appreciated for its versatility and broad applicability. However, even though an extended theoretical description is available, it is still not fully understood how the interplay between different experimental parameters influences the quality of a SOFI image. We investigated the relationship between five experimental parameters (measurement time, on-time t _on, off-time t _off, probe brightness, and out of focus background) and the quality of the super-resolved images they yielded, expressed as Signal to Noise Ratio (SNR). Empirical relationships were modeled for second- and third-order SOFI using data simulated according to a D-Optimal design of experiments, which is an ad-hoc design built to reduce the experimental load when the total number of trials to be conducted becomes too high for practical applications. This approach proves to be more reliable and efficient for parameter optimization compared to the more classical parameter by parameter approach. Our results indicate that the best image quality is achieved for the fastest emitter blinking (lowest t _on and t _off), lowest background level, and the highest measurement duration, while the brightness variation does not affect the quality in a statistically significant way within the investigated range. However, when the ranges spanned by the parameters are constrained, a different set of optimal conditions may arise. For example, for second-order SOFI, we identified situations in which the increase of t _off can be beneficial to SNR, such as when the measurement duration is long enough. In general, optimal values of t _on and t _off have been found to be highly dependent from each other and from the measurement duration.

Time division ghost imaging

In the conventional ghost imaging, it requires to illuminate a large number of patterns in order to reconstruct a good quality image under a low signal-to-noise ratio. We propose a new method so called time division ghost imaging to improve the quality of the image in noisy environment. In this procedure, the total number of patterns in the calculation process of the correlation function are divided into the sub-units with fewer illuminated patterns. Then one calculates the correlation for each sub-unit, and synthesizes the intermediate images obtained at each sub-unit. The validation and effectiveness of this method are confirmed by simulation and experiment, showing the robustness to noise.

Smoothness correction for better SOFI imaging

Sub-diffraction or super-resolution fluorescence imaging allows the visualization of the cellular morphology and interactions at the nanoscale. Statistical analysis methods such as super-resolution optical fluctuation imaging (SOFI) obtain an improved spatial resolution by analyzing fluorophore blinking but can be perturbed by the presence of non-stationary processes such as photodestruction or fluctuations in the illumination. In this work, we propose to use Whittaker smoothing to remove these smooth signal trends and retain only the information associated to independent blinking of the emitters, thus enhancing the SOFI signals. We find that our method works well to correct photodestruction, especially when it occurs quickly. The resulting images show a much higher contrast, strongly suppressed background and a more detailed visualization of cellular structures. Our method is parameter-free and computationally efficient, and can be readily applied on both two-dimensional and three-dimensional data.

A comprehensive review of deep learning-based single image super-resolution

Image super-resolution (SR) is one of the vital image processing methods that improve the resolution of an image in the field of computer vision. In the last two decades, significant progress has been made in the field of super-resolution, especially by utilizing deep learning methods. This survey is an effort to provide a detailed survey of recent progress in single-image super-resolution in the perspective of deep learning while also informing about the initial classical methods used for image super-resolution. The survey classifies the image SR methods into four categories, i.e., classical methods, supervised learning-based methods, unsupervised learning-based methods, and domain-specific SR methods. We also introduce the problem of SR to provide intuition about image quality metrics, available reference datasets, and SR challenges. Deep learning-based approaches of SR are evaluated using a reference dataset. Some of the reviewed state-of-the-art image SR methods include the enhanced deep SR network (EDSR), cycle-in-cycle GAN (CinCGAN), multiscale residual network (MSRN), meta residual dense network (Meta-RDN), recurrent back-projection network (RBPN), second-order attention network (SAN), SR feedback network (SRFBN) and the wavelet-based residual attention network (WRAN). Finally, this survey is concluded with future directions and trends in SR and open problems in SR to be addressed by the researchers.

Recent advances in point spread function engineering and related computational microscopy approaches: from one viewpoint

This personal hybrid review piece, written in light of my recipience of the UIPAB 2020 young investigator award, contains a mixture of my scientific biography and work so far. This paper is not intended to be a comprehensive review, but only to highlight my contributions to computation-related aspects of super-resolution microscopy, as well as their origins and future directions.

New algorithm using L1 regularization for measuring electron energy spectra

Retrieving the spectrum of physical radiation from experimental measurements typically involves using a mathematical algorithm to deconvolve the instrument response function from the measured signal. However, in the field of signal processing known as "Source Separation" (SS), which refers to the process of computationally retrieving the separate source components that generate an overlapping signal on the detector, the deconvolution process can become an ill-posed problem and crosstalk complicates the separation of the individual sources. To overcome this problem, we have designed a magnetic spectrometer for inline electron energy spectrum diagnosis and developed an analysis algorithm using techniques applicable to the problem of SS. An unknown polychromatic electron spectrum is calculated by sparse coding using a Gaussian basis function and an L1 regularization algorithm with a sparsity constraint. This technique is verified by using a specially designed magnetic field electron spectrometer. We use Monte Carlo simulations of the detector response to Maxwellian input energy distributions with electron temperatures of 5.0 MeV, 10.0 MeV, and 15.0 MeV to show that the calculated sparse spectrum can reproduce the input spectrum with an optimum energy bin width automatically selected by the L1 regularization. The spectra are reproduced with a high accuracy of less than 4.0% error, without an initial value. The technique is then applied to experimental measurements of intense laser accelerated electron beams from solid targets. Our analysis concept of spectral retrieval and automatic optimization of energy bin width by sparse coding could form the basis of a novel diagnostic method for spectroscopy.

Sub-Micron Spatial Resolution in Far-Field Raman Imaging Using Positivity-Constrained Super-Resolution

Raman microscopy is a valuable tool for detecting physical and chemical properties of a sample material. When probing nanomaterials or nanocomposites the spatial resolution of Raman microscopy is not always adequate as it is limited by the optical diffraction limit. Numerical post-processing with super-resolution algorithms provides a means to enhance resolution and can be straightforwardly applied. The aim of this work is to present interior point least squares (IPLS) as a powerful tool for super-resolution in Raman imaging through constrained optimization. IPLS's potential for super-resolution is illustrated on numerically generated test images. Its resolving power is demonstrated on Raman spectroscopic data of a polymer nanowire sample. Comparison to atomic force microscopy data of the same sample substantiates that the presented method is a promising technique for analyzing nanomaterial samples.

Compressed Blind Deconvolution and Denoising for Complementary Beam Subtraction Light-Sheet Fluorescence Microscopy

OBJECTIVE

The side-lobes of a Bessel beam (BB) create a severe out-of-focus background in scanning light-sheet fluorescence microscopy, thereby extremely limiting the axial resolution. The complementary beam subtraction (CBS) method can significantly reduce the out-of-focus background by double scanning a BB and its complementary beam. However, the blurring and noise caused by the system instability during the double scanning and subtraction operations degrade the image quality significantly. Therefore, we propose a compressed blind deconvolution and denoising (CBDD) method that solves this problem.

METHODS

We use a unified formulation that comprehensively takes advantage of multiple compressed sensing reconstructions and blind sparse representation.

RESULTS

The simulations and experiments were performed using the microbeads and model organisms to verify the effectiveness of the proposed method. Compared with the CBS light-sheet method, the proposed CBDD algorithm achieved the gain improvement in the axial and lateral resolution of about 1.81 and 2.22 times, respectively, while the average signal-to-noise ratio (SNR) was increased by about 3 dB.

CONCLUSION

Accordingly, the proposed method can suppress the noise level, enhance the SNR, and recover the degraded resolution simultaneously.

SIGNIFICANCE

The obtained results demonstrate the proposed CBDD algorithm is well suited to improve the imaging performance of the CBS light-sheet fluorescence microscopy.

Computational multifocal microscopy

Despite recent advances, high performance single-shot 3D microscopy remains an elusive task. By introducing designed diffractive optical elements (DOEs), one is capable of converting a microscope into a 3D "kaleidoscope," in which case the snapshot image consists of an array of tiles and each tile focuses on different depths. However, the acquired multifocal microscopic (MFM) image suffers from multiple sources of degradation, which prevents MFM from further applications. We propose a unifying computational framework which simplifies the imaging system and achieves 3D reconstruction via computation. Our optical configuration omits optical elements for correcting chromatic aberrations and redesigns the multifocal grating to enlarge the tracking area. Our proposed setup features only one single grating in addition to a regular microscope. The aberration correction, along with Poisson and background denoising, are incorporated in our deconvolution-based fully-automated algorithm, which requires no empirical parameter-tuning. In experiments, we achieve spatial resolutions of 0.35um (lateral) and 0.5um (axial), which are comparable to the resolution that can be achieved with confocal deconvolution microscopy. We demonstrate a 3D video of moving bacteria recorded at 25 frames per second using our proposed computational multifocal microscopy technique.

EXACT SPIKE TRAIN INFERENCE VIA ℓ0 OPTIMIZATION

In recent years new technologies in neuroscience have made it possible to measure the activities of large numbers of neurons simultaneously in behaving animals. For each neuron a fluorescence trace is measured; this can be seen as a first-order approximation of the neuron's activity over time. Determining the exact time at which a neuron spikes on the basis of its fluorescence trace is an important open problem in the field of computational neuroscience. Recently, a convex optimization problem involving an ℓ1 penalty was proposed for this task. In this paper we slightly modify that recent proposal by replacing the ℓ1 penalty with an ℓ0 penalty. In stark contrast to the conventional wisdom that ℓ0 optimization problems are computationally intractable, we show that the resulting optimization problem can be efficiently solved for the global optimum using an extremely simple and efficient dynamic programming algorithm. Our R-language implementation of the proposed algorithm runs in a few minutes on fluorescence traces of 100,000 timesteps. Furthermore, our proposal leads to substantial improvements over the previous ℓ1 proposal, in simulations as well as on two calcium imaging datasets. R-language software for our proposal is available on CRAN in the package LZeroSpikeInference. Instructions for running this software in python can be found at https://github.com/jewellsean/LZeroSpikeInference.

Sparsity-based super-resolution microscopy from correlation information: erratum

Two references on sparse deconvolution of high-density fluorescence microscopy images are added to [Opt. Express26(14), 18238 (2018)].

3D super-resolution microscopy reflects mitochondrial cristae alternations and mtDNA nucleoid size and distribution

3D super-resolution microscopy based on the direct stochastic optical reconstruction microscopy (dSTORM) with primary Alexa-Fluor-647-conjugated antibodies is a powerful method for accessing changes of objects that could be normally resolved only by electron microscopy. Despite the fact that mitochondrial cristae yet to become resolved, we have indicated changes in cristae width and/or morphology by dSTORM of ATP-synthase F₁ subunit α (F₁α). Obtained 3D images were analyzed with the help of Ripley's K-function modeling spatial patterns or transferring them into distance distribution function. Resulting histograms of distances frequency distribution provide most frequent distances (MFD) between the localized single antibody molecules. In fasting state of model pancreatic β-cells, INS-1E, MFD between F₁α were ~80 nm at 0 and 3 mM glucose, whereas decreased to 61 nm and 57 nm upon glucose-stimulated insulin secretion (GSIS) at 11 mM and 20 mM glucose, respectively. Shorter F₁α interdistances reflected cristae width decrease upon GSIS, since such repositioning of F₁α correlated to average 20 nm and 15 nm cristae width at 0 and 3 mM glucose, and 9 nm or 8 nm after higher glucose simulating GSIS (11, 20 mM glucose, respectively). Also, submitochondrial entities such as nucleoids of mtDNA were resolved e.g. after bromo-deoxyuridine (BrDU) pretreatment using anti-BrDU dSTORM. MFD in distances distribution histograms reflected an average nucleoid diameter (<100 nm) and average distances between nucleoids (~1000 nm). Double channel PALM/dSTORM with Eos-lactamase-β plus anti-TFAM dSTORM confirmed the latter average inter-nucleoid distance. In conclusion, 3D single molecule (dSTORM) microscopy is a reasonable tool for studying mitochondrion.

Quantum correlation enhanced super-resolution localization microscopy enabled by a fibre bundle camera

Despite advances in low-light-level detection, single-photon methods such as photon correlation have rarely been used in the context of imaging. The few demonstrations, for example of subdiffraction-limited imaging utilizing quantum statistics of photons, have remained in the realm of proof-of-principle demonstrations. This is primarily due to a combination of low values of fill factors, quantum efficiencies, frame rates and signal-to-noise characteristic of most available single-photon sensitive imaging detectors. Here we describe an imaging device based on a fibre bundle coupled to single-photon avalanche detectors that combines a large fill factor, a high quantum efficiency, a low noise and scalable architecture. Our device enables localization-based super-resolution microscopy in a non-sparse non-stationary scene, utilizing information on the number of active emitters, as gathered from non-classical photon statistics.

Multiplexed high-content analysis of mitochondrial morphofunction using live-cell microscopy

Mitochondria have a central role in cellular (patho)physiology, and they display a highly variable morphology that is probably coupled to their functional state. Here we present a protocol that allows unbiased and automated quantification of mitochondrial 'morphofunction' (i.e., morphology and membrane potential), cellular parameters (size, confluence) and nuclear parameters (number, morphology) in intact living primary human skin fibroblasts (PHSFs). Cells are cultured in 96-well plates and stained with tetramethyl rhodamine methyl ester (TMRM), calcein-AM (acetoxy-methyl ester) and Hoechst 33258. Next, multispectral fluorescence images are acquired using automated microscopy and processed to extract 44 descriptors. Subsequently, the descriptor data are subjected to a quality control (QC) algorithm based upon principal component analysis (PCA) and interpreted using univariate, bivariate and multivariate analysis. The protocol requires a time investment of ∼4 h distributed over 2 d. Although it is specifically developed for PHSFs, which are widely used in preclinical research, the protocol is portable to other cell types and can be scaled up for implementation in high-content screening.