Bayesian estimation for optimized structured illumination microscopy

Approaching maximum resolution in structured illumination microscopy via accurate noise modeling

Biological images captured by microscopes are characterized by heterogeneous signal-to-noise ratios (SNRs) due to spatially varying photon emission across the field of view convoluted with camera noise. State-of-the-art unsupervised structured illumination microscopy (SIM) reconstruction methods, commonly implemented in the Fourier domain, often do not accurately model this noise. Such methods therefore suffer from high-frequency artifacts, user-dependent choices of smoothness constraints making assumptions on biological features, and unphysical negative values in the recovered fluorescence intensity map. On the other hand, supervised algorithms rely on large datasets for training, and often require retraining for new sample structures. Consequently, achieving high contrast near the maximum theoretical resolution in an unsupervised, physically principled manner remains an open problem. Here, we propose Bayesian-SIM (B-SIM), a Bayesian framework to quantitatively reconstruct SIM data, rectifying these shortcomings by accurately incorporating known noise sources in the spatial domain. To accelerate the reconstruction process, we use the finite extent of the point-spread-function to devise a parallelized Monte Carlo strategy involving chunking and restitching of the inferred fluorescence intensity. We benchmark our framework on both simulated and experimental images, and demonstrate improved contrast permitting feature recovery at up to 25% shorter length scales over state-of-the-art methods at both high- and low SNR. B-SIM enables unsupervised, quantitative, physically accurate reconstruction without the need for labeled training data, democratizing high-quality SIM reconstruction and expands the capabilities of live-cell SIM to lower SNR, potentially revealing biological features in previously inaccessible regimes.

Surpassing light inhomogeneities in structured-illumination microscopy with FlexSIM

Super-resolution structured-illumination microscopy (SIM) is a powerful technique that allows one to surpass the diffraction limit by up to a factor two. Yet, its practical use is hampered by its sensitivity to imaging conditions which makes it prone to reconstruction artefacts. In this work, we present FlexSIM, a flexible SIM reconstruction method capable to handle highly challenging data. Specifically, we demonstrate the ability of FlexSIM to deal with the distortion of patterns, the high level of noise encountered in live imaging, as well as out-of-focus fluorescence. Moreover, we show that FlexSIM achieves state-of-the-art performance over a variety of open SIM datasets.

Dynamics of mitochondrial membranes under photo-oxidative stress with high spatiotemporal resolution

In our study, we harnessed an original Enhanced Speed Structured Illumination Microscopy (Fast-SIM) imaging setup to explore the dynamics of mitochondrial and inner membrane ultrastructure under specific photo-oxidation stress induced by Chlorin-e6 and light irradiation. Notably, our Fast-SIM system allowed us to observe and quantify a distinct remodeling and shortening of the mitochondrial structure after 60-80 s of irradiation. These changes were accompanied by fusion events of adjacent inner membrane cristae and global swelling of the organelle. Preceding these alterations, a larger sequence was characterized by heightened dynamics within the mitochondrial network, featuring events such as mitochondrial fission, rapid formation of tubular prolongations, and fluctuations in cristae structure. Our findings provide compelling evidence that, among enhanced-resolution microscopy techniques, Fast-SIM emerges as the most suitable approach for non-invasive dynamic studies of mitochondrial structure in living cells. For the first time, this approach allows quantitative and qualitative characterization of successive steps in the photo-induced oxidation process with sufficient spatial and temporal resolution.

Improving optical sectioning with spinning disk structured illumination microscopy

A new fluorescence microscopy technique for optical sectioning was investigated. This technique combined Spinning Disk microscopy (SD) with Structured Illumination Microscopy (SIM), resulting in more background removal than either method. Spinning Disk Structured Illumination Microscopy (SD-SIM) resulted in higher signal-to-background ratios. The method detected and quantified a dendritic spine neck that was impossible to detect with either SIM or SD alone.

Super-Resolution Imaging of Neuronal Structures with Structured Illumination Microscopy

Super-resolution structured illumination microscopy (SR-SIM) is an optical fluorescence microscopy method which is suitable for imaging a wide variety of cells and tissues in biological and biomedical research. Typically, SIM methods use high spatial frequency illumination patterns generated by laser interference. This approach provides high resolution but is limited to thin samples such as cultured cells. Using a different strategy for processing raw data and coarser illumination patterns, we imaged through a 150-micrometer-thick coronal section of a mouse brain expressing GFP in a subset of neurons. The resolution reached 144 nm, an improvement of 1.7-fold beyond conventional widefield imaging.

Superresolution structured illumination microscopy reconstruction algorithms: a review

Structured illumination microscopy (SIM) has become the standard for next-generation wide-field microscopy, offering ultrahigh imaging speed, superresolution, a large field-of-view, and long-term imaging. Over the past decade, SIM hardware and software have flourished, leading to successful applications in various biological questions. However, unlocking the full potential of SIM system hardware requires the development of advanced reconstruction algorithms. Here, we introduce the basic theory of two SIM algorithms, namely, optical sectioning SIM (OS-SIM) and superresolution SIM (SR-SIM), and summarize their implementation modalities. We then provide a brief overview of existing OS-SIM processing algorithms and review the development of SR-SIM reconstruction algorithms, focusing primarily on 2D-SIM, 3D-SIM, and blind-SIM. To showcase the state-of-the-art development of SIM systems and assist users in selecting a commercial SIM system for a specific application, we compare the features of representative off-the-shelf SIM systems. Finally, we provide perspectives on the potential future developments of SIM.

Super-resolution imaging of neuronal structure with structured illumination microscopy

Super-resolution structured illumination microscopy (SR-SIM) is a method in optical fluorescence microscopy which is suitable for imaging a wide variety of cells and tissues in biological and biomedical research. Typically, SIM methods use high spatial frequency illumination patterns generated by laser interference. This approach provides high resolution but is limited to thin samples such as cultured cells. Using a different strategy for processing the raw data and coarser illumination patterns, we imaged through a 150 µm thick coronal section of a mouse brain expressing GFP in a subset of neurons. The resolution reached 144 nm, an improvement of 1.7 fold beyond conventional widefield imaging.

Rapid, artifact-reduced, image reconstruction for super-resolution structured illumination microscopy

Super-resolution structured illumination microscopy (SR-SIM) is finding increasing application in biomedical research due to its superior ability to visualize subcellular dynamics in living cells. However, during image reconstruction artifacts can be introduced and when coupled with time-consuming postprocessing procedures, limits this technique from becoming a routine imaging tool for biologists. To address these issues, an accelerated, artifact-reduced reconstruction algorithm termed joint space frequency reconstruction-based artifact reduction algorithm (JSFR-AR-SIM) was developed by integrating a high-speed reconstruction framework with a high-fidelity optimization approach designed to suppress the sidelobe artifact. Consequently, JSFR-AR-SIM produces high-quality, super-resolution images with minimal artifacts, and the reconstruction speed is increased. We anticipate this algorithm to facilitate SR-SIM becoming a routine tool in biomedical laboratories.

Flexible Multiplane Structured Illumination Microscope with a Four-Camera Detector

Fluorescence microscopy provides an unparalleled tool for imaging biological samples. However, producing high-quality volumetric images quickly and without excessive complexity remains a challenge. Here, we demonstrate a four-camera structured illumination microscope (SIM) capable of simultaneously imaging multiple focal planes, allowing for the capture of 3D fluorescent images without any axial movement of the sample. This setup allows for the acquisition of many different 3D imaging modes, including 3D time lapses, high-axial-resolution 3D images, and large 3D mosaics. We imaged mitochondrial motions in live cells, neuronal structure in Drosophila larvae, and imaged up to 130 µm deep into mouse brain tissue. After SIM processing, the resolution measured using one of the four cameras improved from 357 nm to 253 nm when using a 30×/1.05 NA objective.

Answering some questions about structured illumination microscopy

Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood 'super-resolution' method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

SIM reconstruction framework for high-speed multi-dimensional super-resolution imaging

Structured illumination microscopy (SIM) holds great promise for live cell imaging applications due to its potential to obtain multidimensional information such as intensity, spectrum and polarization (I, λ , p) at high spatial-temporal resolution, enabling the observation of more complex dynamic interactions between subcellular structures. However, the reconstruction results of polarized samples are prone to artifacts because all current SIM reconstruction frameworks use incomplete imaging models which neglect polarization modulation. Such polarization-related artifacts are especially prevalent for SIM reconstruction using a reduced number of raw images (RSIM) and severely undermine the ability of SIM to capture multi-dimensional information. Here, we report a new SIM reconstruction framework (PRSIM) that can recover multi-dimensional information (I, λ, p) using a reduced number of raw images. PRSIM adopts a complete imaging model that is versatile for normal and polarized samples and uses a frequency-domain iterative reconstruction algorithm for artifact-free super-resolution (SR) reconstruction. It can simultaneously obtain the SR spatial structure and polarization orientation of polarized samples using 6 raw SIM images and can perform SR reconstruction using 4 SIM images for normal samples. In addition, PRSIM has less spatial computational complexity and achieves reconstruction speeds tens of times higher than that of the state-of-the-art non-iterative RSIM, making it more suitable for large field-of-view imaging. Thus, PRSIM is expected to facilitate the development of SIM into an ultra-high-speed and multi-dimensional SR imaging tool.

Improved resolution in 3D structured illumination microscopy using 3D model-based restoration with positivity-constraint

The performance of structured illumination microscopy (SIM) systems depends on the computational method used to process the raw data. In this paper, we present a regularized three-dimensional (3D) model-based (MB) restoration method with positivity constraint (PC) for 3D processing of data from 3D-SIM (or 3-beam interference SIM), in which the structured illumination pattern varies laterally and axially. The proposed 3D-MBPC method introduces positivity in the solution through the reconstruction of an auxiliary function using a conjugate-gradient method that minimizes the mean squared error between the data and the 3D imaging model. The 3D-MBPC method provides axial super resolution, which is not the same as improved optical sectioning demonstrated with model-based approaches based on the 2D-SIM (or 2-beam interference SIM) imaging model, for either 2D or 3D processing of a single plane from a 3D-SIM dataset. Results obtained with our 3D-MBPC method show improved 3D resolution over what is achieved by the standard generalized Wiener filter method, the first known method that performs 3D processing of 3D-SIM data. Noisy simulation results quantify the achieved 3D resolution, which is shown to match theoretical predictions. Experimental verification of the 3D-MBPC method with biological data demonstrates successful application to data volumes of different sizes.

Method for assessing the spatiotemporal resolution of structured illumination microscopy (SIM)

A method is proposed for assessing the temporal resolution of structured illumination microscopy (SIM), by tracking the amplitude of different spatial frequency components over time, and comparing them to a temporally-oscillating ground-truth. This method is used to gain insight into the performance limits of SIM, along with alternative reconstruction techniques (termed 'rolling SIM') that claim to improve temporal resolution. Results show that the temporal resolution of SIM varies considerably between low and high spatial frequencies, and that, despite being used in several high profile papers and commercial microscope software, rolling SIM provides no increase in temporal resolution over conventional SIM.

Rapid Image Reconstruction of Structured Illumination Microscopy Directly in the Spatial Domain

Super-resolution structured illumination microscopy (SIM) routinely performs image reconstruction in the frequency domain using an approach termed frequency-domain reconstruction (FDR). Due to multiple Fourier transforms between the spatial and frequency domains, SIM suffers from low reconstruction speed, constraining its applications in real-time, dynamic imaging. To overcome this limitation, we developed a new method for SIM image reconstruction, termed spatial domain reconstruction (SDR). SDR is intrinsically simpler than FDR, does not require Fourier transforms and the theory predicts it to be a rapid image reconstruction method. Results show that SDR reconstructs a super-resolution image 7-fold faster than FDR, producing images that are equal to either FDR or the widely-used FairSIM. We provide a proof-of-principle using mobile fluorescent beads to demonstrate the utility of SDR in imaging moving objects. Consequently, replacement of the FDR approach with SDR significantly enhances SIM as the desired method for live-cell, instant super-resolution imaging. This means that SDR-SIM is a "What You See Is What You Get" approach to super-resolution imaging.

Unified joint reconstruction approach for random illumination microscopy

Random illumination microscopy (RIM) using uncontrolled speckle patterns has shown the capacity to surpass the Abbe's diffraction barrier, providing the possibility to design inexpensive and versatile structured illumination microscopy (SIM) devices. In this paper, I first present a review of the state-of-the-art joint reconstruction methods in RIM, and then propose a unified joint reconstruction approach in which the performance of various regularization terms can be evaluated under the same model. The model hyperparameter is easily tuned and robust in comparison to the previous methods and ℓ2,1 regularizer is proven to be a reasonable prior in most practical situations. Moreover, the degradation entailed by out-of-focus light in conventional SIM can be easily solved in RIM setup.

Artifact-free whole-slide imaging with structured illumination microscopy and Bayesian image reconstruction

BACKGROUND

Structured illumination microscopy (SIM) is a method that can be used to image biological samples and can achieve both optical sectioning and super-resolution effects. Optimization of the imaging set-up and data-processing methods results in high-quality images without artifacts due to mosaicking or due to the use of SIM methods. Reconstruction methods based on Bayesian estimation can be used to produce images with a resolution beyond that dictated by the optical system.

FINDINGS

Five complete datasets are presented including large panoramic SIM images of human tissues in pathophysiological conditions. Cancers of the prostate, skin, ovary, and breast, as well as tuberculosis of the lung, were imaged using SIM. The samples are available commercially and are standard histological preparations stained with hematoxylin-eosin.

CONCLUSION

The use of fluorescence microscopy is increasing in histopathology. There is a need for methods that reduce artifacts caused by the use of image-stitching methods or optical sectioning methods such as SIM. Stitched SIM images produce results that may be useful for intraoperative histology. Releasing high-quality, full-slide images and related data will aid researchers in furthering the field of fluorescent histopathology.

Jointly super-resolved and optically sectioned Bayesian reconstruction method for structured illumination microscopy

Structured Illumination Microscopy (SIM) is an imaging technique for achieving both super-resolution (SR) and optical sectioning (OS) in wide-field microscopy. It consists in illuminating the sample with periodic patterns at different orientations and positions. The resulting images are then processed to reconstruct the observed object with SR and/or OS. In this work, we present BOSSA-SIM, a general-purpose SIM reconstruction method, applicable to moving objects such as encountered in in vivo retinal imaging, that enables SR and OS jointly in a fully unsupervised Bayesian framework. By modeling a 2-layer object composed of an in-focus layer and a defocused layer, we show that BOSSA-SIM is able to jointly reconstruct them so as to get a super-resolved and optically sectioned in-focus layer. The achieved performance, assessed quantitatively by simulations for several noise levels, compares favorably with a state-of-the-art method. Finally, we validate our method on open-access experimental microscopy data.

Temporal super-resolution microscopy using a hue-encoded shutter

Limited time-resolution in microscopy is an obstacle to many biological studies. Despite recent advances in hardware, digital cameras have limited operation modes that constrain frame-rate, integration time, and color sensing patterns. In this paper, we propose an approach to extend the temporal resolution of a conventional digital color camera by leveraging a multi-color illumination source. Our method allows for the imaging of single-hue objects at an increased frame-rate by trading spectral for temporal information (while retaining the ability to measure base hue). It also allows rapid switching to standard RGB acquisition. We evaluated the feasibility and performance of our method via experiments with mobile resolution targets. We observed a time-resolution increase by a factor 2.8 with a three-fold increase in temporal sampling rate. We further illustrate the use of our method to image the beating heart of a zebrafish larva, allowing the display of color or fast grayscale images. Our method is particularly well-suited to extend the capabilities of imaging systems where the flexibility of rapidly switching between high frame rate and color imaging are necessary.

Near-field Fourier ptychography: super-resolution phase retrieval via speckle illumination

High spatial resolution is the goal of many imaging systems. While designing a high-resolution lens with diffraction-limited performance over a large field of view remains a difficult task, creating a complex speckle pattern with wavelength-limited spatial features is easily accomplished with a simple random diffuser. With this observation and the concept of near-field ptychography, we report a new imaging modality, termed near-field Fourier ptychography, to address high-resolution imaging challenges in both microscopic and macroscopic imaging settings. 'Near-field' refers to placing the object at a short defocus distance with a large Fresnel number. We project a speckle pattern with fine spatial features on the object instead of directly resolving the spatial features via a high-resolution lens. We then translate the object (or speckle) to different positions and acquire the corresponding images by using a low-resolution lens. A ptychographic phase retrieval process is used to recover the complex object, the unknown speckle pattern, and the coherent transfer function at the same time. In a microscopic imaging setup, we use a 0.12 numerical aperture (NA) lens to achieve an NA of 0.85 in the reconstruction process. In a macroscale photographic imaging setup, we achieve ~7-fold resolution gain by using a photographic lens. The collection optics do not determine the final achievable resolution; rather, the speckle pattern's feature size does. This is similar to our recent demonstration in fluorescence imaging settings (Guo et al., Biomed. Opt. Express, 9(1), 2018). The reported imaging modality can be employed in light, coherent X-ray, and transmission electron imaging systems to increase resolution and provide quantitative absorption and object phase contrast.

Structured illumination imaging without grating rotation based on mirror operation on 1D Fourier spectrum

Structured illumination microscopy (SIM) is a rapidly developing a super-resolution optical microscopy technique. With SIM, the grating is needed in order to rotate several angles for illuminating the sample in different directions. Multiple rotations reduce the imaging speed and grating rotation angle errors damage the image recovery quality. We introduce mirror transformation on one-dimension (1D) Fourier spectrum to SIM for resolving the problems of low imaging speed and severe impact on image reconstruction quality by grating rotation angle errors. When mirror operation and SIM are combined, the grating is placed at an orientation for obtaining three shadow images. The three shadow images are acquired by CCD at three different phase shift for a direction of grating. Thus, the SIM imaging speed is faster and the effect on image reconstruction quality by grating rotation angle errors is greatly reduced.

Super-resolution algorithm based on Richardson-Lucy deconvolution for three-dimensional structured illumination microscopy

Three-dimensional structured illumination microscopy (3D-SIM) is a wide-field super-resolution technique in fluorescent imaging that can double the resolution beyond its classical limit. We introduce, to the best of our knowledge, a new 3D reconstruction algorithm based on the Richardson-Lucy deconvolution. The 3D-SIM imaging principle and the reconstruction steps are demonstrated in detail. Microspheres and biological specimen are used to present the performance of this method. The background of the out-of-focus portion is effectively suppressed, and true optical sectioning and super-resolution can be achieved simultaneously. For the custom-built 3D-SIM and this reconstruction algorithm, the measured resolution was 99.5±5  nm laterally and 294±9  nm axially.

Imaging tissues and cells beyond the diffraction limit with structured illumination microscopy and Bayesian image reconstruction

Background

Structured illumination microscopy (SIM) is a family of methods in optical fluorescence microscopy that can achieve both optical sectioning and super-resolution effects. SIM is a valuable method for high-resolution imaging of fixed cells or tissues labeled with conventional fluorophores, as well as for imaging the dynamics of live cells expressing fluorescent protein constructs. In SIM, one acquires a set of images with shifting illumination patterns. This set of images is subsequently treated with image analysis algorithms to produce an image with reduced out-of-focus light (optical sectioning) and/or with improved resolution (super-resolution).

Findings

Five complete, freely available SIM datasets are presented including raw and analyzed data. We report methods for image acquisition and analysis using open-source software along with examples of the resulting images when processed with different methods. We processed the data using established optical sectioning SIM and super-resolution SIM methods and with newer Bayesian restoration approaches that we are developing.

Conclusions

Various methods for SIM data acquisition and processing are actively being developed, but complete raw data from SIM experiments are not typically published. Publically available, high-quality raw data with examples of processed results will aid researchers when developing new methods in SIM. Biologists will also find interest in the high-resolution images of animal tissues and cells we acquired. All of the data were processed with SIMToolbox, an open-source and freely available software solution for SIM.

In Vivo Superresolution Imaging of Neuronal Structure in the Mouse Brain

OBJECTIVE

this study proposes and evaluates a technique for in vivo deep-tissue superresolution imaging in the light-scattering mouse brain at up to a 3.5 Hz 2-D imaging rate with a 21×21 μm² field of view.

METHODS

we combine the deep-tissue penetration and high imaging speed of resonant laser scanning two-photon (2P) microscopy with the superresolution ability of patterned excitation microscopy. Using high-frequency intensity modulation of the scanned two-photon excitation beam, we generate patterned illumination at the imaging plane. Using the principles of structured illumination, the high-frequency components in the collected images are then used to reconstruct images with an approximate twofold increase in optical resolution.

RESULTS

using our technique, resonant 2P superresolution patterned excitation reconstruction microscopy, we demonstrate our ability to investigate nanoscopic neuronal architecture in the cerebral cortex of the mouse brain at a depth of 120 μm in vivo and 210 μm ex vivo with a resolution of 119 nm. This technique optimizes the combination of speed and depth for improved in vivo imaging in the rodent neocortex.

CONCLUSION

this study demonstrates a potentially useful technique for superresolution in vivo investigations in the rodent brain in deep tissue, creating a platform for investigating nanoscopic neuronal dynamics.

SIGNIFICANCE

this technique optimizes the combination of speed and depth for improved superresolution in vivo imaging in the rodent neocortex.

Tunable-frequency three-dimensional structured illumination microscopy with reduced data-acquisition

The performance of a tunable three-dimensional (3D) structured illumination microscope (SIM) system and its ability to provide simultaneously super-resolution (SR) and optical-sectioning (OS) capabilities are investigated. Numerical results show that the performance of our 3D-SIM system is comparable with the one provided by a three-wave interference SIM, while requiring 40% fewer images for the reconstruction and providing frequency tunability in a cost-effective implementation. The performance of the system has been validated experimentally with images from test samples, which were also imaged with a commercial SIM based on incoherent-grid projection for comparison. Restored images from data acquired from an axially-thin fluorescent layer show a 1.6× improvement in OS capability compared to the commercial instrument while results from a fluorescent tilted USAF target show the OS and SR capabilities achieved by our system.

Fast Optical Sectioning for Widefield Fluorescence Mesoscopy with the Mesolens based on HiLo Microscopy

We present here a fast optical sectioning method for mesoscopy based on HiLo microscopy, which makes possible imaging of specimens of up to 4.4 mm × 3 mm × 3 mm in volume in under 17 hours (estimated for a z-stack comprising 1000 images excluding computation time) with subcellular resolution throughout. Widefield epifluorescence imaging is performed with the Mesolens using a high pixel-number camera capable of sensor-shifting to generate a 259.5 Megapixel image, and we have developed custom software to perform HiLo processing of the very large datasets. Using this method, we obtain comparable sectioning strength to confocal laser scanning microscopy (CLSM), with sections as thin as 6.8 ± 0.2 μm and raw acquisition speed of 1 minute per slice which is up to 30 times faster than CLSM on the full field of view (FOV) of the Mesolens of 4.4 mm with lateral resolution of 0.7 μm and axial resolution of 7 μm. We have applied this HiLo mesoscopy method to image fixed and fluorescently stained hippocampal neuronal specimens and a 5-day old zebrafish larva.

A frequency domain SIM reconstruction algorithm using reduced number of images

Conventional two-dimensional structured illumination microscopy (SIM) requires 9 raw images to reconstruct a super-resolved image. In order to increase the frame rate of 2DSIM, attempts are being made to reduce the number of raw SIM images. However, all the proposed SIM reconstruction algorithms (SIM-RA) capable of reconstructing super-resolution (SR) image with a reduced number of raw SIM images operate in the spatial domain. Here, we present a frequency domain SIM-RA based on ordinary least squares technique, which enables reconstruction of SR image using 4 raw SIM images. Unlike the spatial domain RA, which produces the SR image through iterative convergence, the presented RA provides a single step solution. It also reveals the fundamental limitation of least number of raw images required for resolution doubling in SIM.

Nonsmooth Convex Optimization for Structured Illumination Microscopy Image Reconstruction

In this paper, we propose a new approach for structured illumination microscopy image reconstruction. We first introduce the principles of this imaging modality and describe the forward model. We then propose the minimization of nonsmooth convex objective functions for the recovery of the unknown image. In this context, we investigate two data-fitting terms for Poisson-Gaussian noise and introduce a new patch-based regularization method. This approach is tested against other regularization approaches on a realistic benchmark. Finally, we perform some test experiments on images acquired on two different microscopes.

Image restoration approach to address reduced modulation contrast in structured illumination microscopy

The performance of structured illumination microscopy (SIM) is hampered in many biological applications due to the inability to modulate the light when imaging deep into the sample. This is in part because sample-induced aberration reduces the modulation contrast of the structured pattern. In this paper, we present an image restoration approach suitable for processing raw incoherent-grid-projection SIM data with a low fringe contrast. Restoration results from simulated and experimental ApoTome SIM data show results with improved signal-to-noise ratio (SNR) and optical sectioning compared to the results obtained from existing methods, such as 2D demodulation and 3D SIM deconvolution. Our proposed method provides satisfactory results (quantified by the achieved SNR and normalized mean square error) even when the modulation contrast of the illumination pattern is as low as 7%.

Imaging neuronal structure dynamics using 2-photon super-resolution patterned excitation reconstruction microscopy

Visualizing fine neuronal structures deep inside strongly light-scattering brain tissue remains a challenge in neuroscience. Recent nanoscopy techniques have reached the necessary resolution but often suffer from limited imaging depth, long imaging time or high light fluence requirements. Here, we present two-photon super-resolution patterned excitation reconstruction (2P-SuPER) microscopy for 3-dimensional imaging of dendritic spine dynamics at a maximum demonstrated imaging depth of 130 μm in living brain tissue with approximately 100 nm spatial resolution. We confirmed 2P-SuPER resolution using fluorescence nanoparticle and quantum dot phantoms and imaged spiny neurons in acute brain slices. We induced hippocampal plasticity and showed that 2P-SuPER can resolve increases in dendritic spine head sizes on CA1 pyramidal neurons following theta-burst stimulation of Schaffer collateral axons. 2P-SuPER further revealed nanoscopic increases in dendritic spine neck widths, a feature of synaptic plasticity that has not been thoroughly investigated due to the combined limit of resolution and penetration depth in existing imaging technologies.

Structured illumination microscopy with interleaved reconstruction (SIMILR)

Structured illumination microscopy (SIM) is the commonly used super-resolution (SR) technique for imaging subcellular dynamics. However, due to its need for multiple illumination patterns, the frame rate is just a fraction of that of conventional microscopy and is thus too slow for fast dynamic studies. A new SR image reconstruction method that maximizes the use of each subframe of the acquisition series is proposed for improving the super-resolved frame rate by N times for N illumination directions. The method requires no changes in raw data and is appropriate for many versions of SIM setup, including those implementing fast illumination pattern generation mechanism based on spatial light modulator or digital micromirror device. The performance of the proposed method is demonstrated through imaging the highly dynamic endoplasmic reticulum where continuous rapid growths or shape changes of tiny structures are observed.

Speed limits of structured illumination microscopy

A theoretical framework for widefield structured illumination microscopy (SIM) reconstruction from fewer than the commonly used nine raw frame acquisitions is introduced and applied in silico and in vitro. The proposed scheme avoids the recording of redundant spatial frequency components, which was necessary in previous SIM algorithms. This allows for gentler superresolution imaging at faster speeds. A doubling of frame rates is possible solely via changes in the computational reconstruction procedure. Furthermore, we explore numerically the effect of the sample movement on the reconstruction quality and the number of raw frames recordable. Our results show that there exists a limit above which deconvolution microscopy becomes superior to SIM.

Resolution enhancement of the microscopic imaging by unknown sinusoidal structured illumination with iterative algorithm

Microscopy by sinusoidal structured illumination is a conventional method to improve resolution, which largely depends on accurate knowledge of the illumination pattern. Two steps are included in the reconstruction process of our proposed technique. The first step solves the parameters of the structured illumination in the spatial domain. Besides the phase-shifting amounts, the period, the modulation factor, and the background intensity of the pattern are extracted from three segmented raw images by the iterative algorithm. The second step is retrieval and synthesis of the low- and high-frequency information of the object in the Fourier space with obtained data. Since the unknown object information is not involved in the pattern parameters' solving process, it is possible to figure out the problem with higher precision and less requirements. We test the performance of this method in the experiments. The resolution is improved with the designed carrier frequency of the illumination pattern.

Joint Reconstruction Strategy for Structured Illumination Microscopy With Unknown Illuminations

The blind structured illumination microscopy strategy proposed by Mudry et al. is fully re-founded in this paper, unveiling the central role of the sparsity of the illumination patterns in the mechanism that drives super-resolution in the method. A numerical analysis shows that the resolving power of the method can be further enhanced with optimized one-photon or two-photon speckle illuminations. A much improved numerical implementation is provided for the reconstruction problem under the image positivity constraint. This algorithm rests on a new preconditioned proximal iteration faster than existing solutions, paving the way to 3D and real-time 2D reconstruction.

Navigating challenges in the application of superresolution microscopy

In 2014, the Nobel Prize in Chemistry was awarded to three scientists who have made groundbreaking contributions to the field of superresolution (SR) microscopy (SRM). The first commercial SR microscope came to market a decade earlier, and many other commercial options have followed. As commercialization has lowered the barrier to using SRM and the awarding of the Nobel Prize has drawn attention to these methods, biologists have begun adopting SRM to address a wide range of questions in many types of specimens. There is no shortage of reviews on the fundamental principles of SRM and the remarkable achievements made with these methods. We approach SRM from another direction: we focus on the current practical limitations and compromises that must be made when designing an SRM experiment. We provide information and resources to help biologists navigate through common pitfalls in SRM specimen preparation and optimization of image acquisition as well as errors and artifacts that may compromise the reproducibility of SRM data.

Virtual k-Space Modulation Optical Microscopy

We report a novel superresolution microscopy approach for imaging fluorescence samples. The reported approach, termed virtual k-space modulation optical microscopy (VIKMOM), is able to improve the lateral resolution by a factor of 2, reduce the background level, improve the optical sectioning effect and correct for unknown optical aberrations. In the acquisition process of VIKMOM, we used a scanning confocal microscope setup with a 2D detector array to capture sample information at each scanned x-y position. In the recovery process of VIKMOM, we first modulated the captured data by virtual k-space coding and then employed a ptychography-inspired procedure to recover the sample information and correct for unknown optical aberrations. We demonstrated the performance of the reported approach by imaging fluorescent beads, fixed bovine pulmonary artery endothelial (BPAE) cells, and living human astrocytes (HA). As the VIKMOM approach is fully compatible with conventional confocal microscope setups, it may provide a turn-key solution for imaging biological samples with ∼100  nm lateral resolution, in two or three dimensions, with improved optical sectioning capabilities and aberration correcting.

Deconvolution methods for structured illumination microscopy

We compare two recently developed multiple-frame deconvolution approaches for the reconstruction of structured illumination microscopy (SIM) data: the pattern-illuminated Fourier ptychography algorithm (piFP) and the joint Richardson-Lucy deconvolution (jRL). The quality of the images reconstructed by these methods is compared in terms of the achieved resolution improvement, noise enhancement, and inherent artifacts. Furthermore, we study the issue of object-dependent resolution improvement by considering the modulation transfer functions derived from different types of objects. The performance of the considered methods is tested in experiments and benchmarked with a commercial SIM microscope. We find that the piFP method resolves periodic and isolated structures equally well, whereas the jRL method provides significantly higher resolution for isolated objects compared to periodic ones. Images reconstructed by the piFP and jRL algorithms are comparable to the images reconstructed using the generalized Wiener filter applied in most commercial SIM microscopes. An advantage of the discussed algorithms is that they allow the reconstruction of SIM images acquired under different types of illumination, such as multi-spot or random illumination.

Resolution doubling with a reduced number of image acquisitions

Structured illumination technique enhances the lateral resolution by projecting non-uniform intensity patterns on a sample. In a typical implementation, three lateral phase shifts (0, 2π/3, 4π/3) are needed for each orientation of the sinusoidal pattern, and 3 different orientations are needed to double the bandwidth isotopically in the Fourier domain. To this end, 9 incoherent images are needed in the acquisition process. In this paper, we discuss an imaging strategy for the structured illumination technique and demonstrate the use of a modified incoherent Fourier ptychographic procedure for reducing the number of acquisitions. In the first implementation, we used complementary sinusoidal patterns for sample illumination. We show that, the number of lateral phase shifts can be reduced from 3 to 2 for each orientation of the sinusoidal pattern and the total number of image acquisitions can be reduced to 6 with 3 orientations. In the second implementation, we further reduce the number of image acquisitions to 4. We also show that, the resolution-doubled image can be recovered even with unknown phases of the sinusoidal patterns. We validate the proposed imaging procedure with non-fluorescence samples. The reported approach may shorten the acquisition time of super-resolution imaging and reduce phototoxicity of biological samples.

Optical Sectioning and High Resolution in Single-Slice Structured Illumination Microscopy by Thick Slice Blind-SIM Reconstruction

The microscope image of a thick fluorescent sample taken at a given focal plane is plagued by out-of-focus fluorescence and diffraction limited resolution. In this work, we show that a single slice of Structured Illumination Microscopy (two or three beam SIM) data can be processed to provide an image exhibiting tight sectioning and high transverse resolution. Our reconstruction algorithm is adapted from the blind-SIM technique which requires very little knowledge of the illumination patterns. It is thus able to deal with illumination distortions induced by the sample or illumination optics. We named this new algorithm thick slice blind-SIM because it models a three-dimensional sample even though only a single two-dimensional plane of focus was measured.

Forward Looking Radar Imaging by Truncated Singular Value Decomposition and Its Application for Adverse Weather Aircraft Landing

The forward looking radar imaging task is a practical and challenging problem for adverse weather aircraft landing industry. Deconvolution method can realize the forward looking imaging but it often leads to the noise amplification in the radar image. In this paper, a forward looking radar imaging based on deconvolution method is presented for adverse weather aircraft landing. We first present the theoretical background of forward looking radar imaging task and its application for aircraft landing. Then, we convert the forward looking radar imaging task into a corresponding deconvolution problem, which is solved in the framework of algebraic theory using truncated singular decomposition method. The key issue regarding the selecting of the truncated parameter is addressed using generalized cross validation approach. Simulation and experimental results demonstrate that the proposed method is effective in achieving angular resolution enhancement with suppressing the noise amplification in forward looking radar imaging.

Out-of-focus background subtraction for fast structured illumination super-resolution microscopy of optically thick samples

We propose a structured illumination microscopy method to combine super resolution and optical sectioning in three-dimensional (3D) samples that allows the use of two-dimensional (2D) data processing. Indeed, obtaining super-resolution images of thick samples is a difficult task if low spatial frequencies are present in the in-focus section of the sample, as these frequencies have to be distinguished from the out-of-focus background. A rigorous treatment would require a 3D reconstruction of the whole sample using a 3D point spread function and a 3D stack of structured illumination data. The number of raw images required, 15 per optical section in this case, limits the rate at which high-resolution images can be obtained. We show that by a succession of two different treatments of structured illumination data we can estimate the contrast of the illumination pattern and remove the out-of-focus content from the raw images. After this cleaning step, we can obtain super-resolution images of optical sections in thick samples using a two-beam harmonic illumination pattern and a limited number of raw images. This two-step processing makes it possible to obtain super resolved optical sections in thick samples as fast as if the sample was two-dimensional.

Bayesian deconvolution for angular super-resolution in forward-looking scanning radar

Scanning radar is of notable importance for ground surveillance, terrain mapping and disaster rescue. However, the angular resolution of a scanning radar image is poor compared to the achievable range resolution. This paper presents a deconvolution algorithm for angular super-resolution in scanning radar based on Bayesian theory, which states that the angular super-resolution can be realized by solving the corresponding deconvolution problem with the maximum a posteriori (MAP) criterion. The algorithm considers that the noise is composed of two mutually independent parts, i.e., a Gaussian signal-independent component and a Poisson signal-dependent component. In addition, the Laplace distribution is used to represent the prior information about the targets under the assumption that the radar image of interest can be represented by the dominant scatters in the scene. Experimental results demonstrate that the proposed deconvolution algorithm has higher precision for angular super-resolution compared with the conventional algorithms, such as the Tikhonov regularization algorithm, the Wiener filter and the Richardson–Lucy algorithm.

Fast high-resolution 3D total internal reflection fluorescence microscopy by incidence angle scanning and azimuthal averaging

Total internal reflection fluorescence microscopy (TIRFM) is the method of choice to visualize a variety of cellular processes in particular events localized near the plasma membrane of live adherent cells. This imaging technique not relying on particular fluorescent probes provides a high sectioning capability. It is, however, restricted to a single plane. We present here a method based on a versatile design enabling fast multiwavelength azimuthal averaging and incidence angles scanning to computationally reconstruct 3D images sequences. We achieve unprecedented 50-nm axial resolution over a range of 800 nm above the coverslip. We apply this imaging modality to obtain structural and dynamical information about 3D actin architectures. We also temporally decipher distinct Rab11a-dependent exocytosis events in 3D at a rate of seven stacks per second.

Three-dimensional super-resolution structured illumination microscopy with maximum a posteriori probability image estimation

We introduce and demonstrate a new high performance image reconstruction method for super-resolution structured illumination microscopy based on maximum a posteriori probability estimation (MAP-SIM). Imaging performance is demonstrated on a variety of fluorescent samples of different thickness, labeling density and noise levels. The method provides good suppression of out of focus light, improves spatial resolution, and allows reconstruction of both 2D and 3D images of cells even in the case of weak signals. The method can be used to process both optical sectioning and super-resolution structured illumination microscopy data to create high quality super-resolution images.

Full-field dual-color 100-nm super-resolution imaging reveals organization and dynamics of mitochondrial and ER networks

Most structured illumination microscopes use a physical or synthetic grating that is projected into the sample plane to generate a periodic illumination pattern. Albeit simple and cost-effective, this arrangement hampers fast or multi-color acquisition, which is a critical requirement for time-lapse imaging of cellular and sub-cellular dynamics. In this study, we designed and implemented an interferometric approach allowing large-field, fast, dual-color imaging at an isotropic 100-nm resolution based on a sub-diffraction fringe pattern generated by the interference of two colliding evanescent waves. Our all-mirror-based system generates illumination pat-terns of arbitrary orientation and period, limited only by the illumination aperture (NA = 1.45), the response time of a fast, piezo-driven tip-tilt mirror (10 ms) and the available fluorescence signal. At low µW laser powers suitable for long-period observation of life cells and with a camera exposure time of 20 ms, our system permits the acquisition of super-resolved 50 µm by 50 µm images at 3.3 Hz. The possibility it offers for rapidly adjusting the pattern between images is particularly advantageous for experiments that require multi-scale and multi-color information. We demonstrate the performance of our instrument by imaging mitochondrial dynamics in cultured cortical astrocytes. As an illustration of dual-color excitation dual-color detection, we also resolve interaction sites between near-membrane mitochondria and the endoplasmic reticulum. Our TIRF-SIM microscope provides a versatile, compact and cost-effective arrangement for super-resolution imaging, allowing the investigation of co-localization and dynamic interactions between organelles--important questions in both cell biology and neurophysiology.

Estimating the granularity coefficient of a Potts-Markov random field within a Markov chain Monte Carlo algorithm

This paper addresses the problem of estimating the Potts parameter β jointly with the unknown parameters of a Bayesian model within a Markov chain Monte Carlo (MCMC) algorithm. Standard MCMC methods cannot be applied to this problem because performing inference on β requires computing the intractable normalizing constant of the Potts model. In the proposed MCMC method, the estimation of β is conducted using a likelihood-free Metropolis-Hastings algorithm. Experimental results obtained for synthetic data show that estimating β jointly with the other unknown parameters leads to estimation results that are as good as those obtained with the actual value of β. On the other hand, choosing an incorrect value of β can degrade estimation performance significantly. To illustrate the interest of this method, the proposed algorithm is successfully applied to real bidimensional SAR and tridimensional ultrasound images.

Structured oblique illumination microscopy for enhanced resolution imaging of non-fluorescent, coherently scattering samples

Many biological structures of interest are beyond the diffraction limit of conventional microscopes and their visualization requires application of super-resolution techniques. Such techniques have found remarkable success in surpassing the diffraction limit to achieve sub-diffraction limited resolution; however, they are predominantly limited to fluorescent samples. Here, we introduce a non-fluorescent analogue to structured illumination microscopy, termed structured oblique illumination microscopy (SOIM), where we use simultaneous oblique illuminations of the sample to multiplex high spatial-frequency content into the frequency support of the system. We introduce a theoretical framework describing how to demodulate this multiplexed information to reconstruct an image with a spatial-frequency support exceeding that of the system's classical diffraction limit. This approach allows enhanced-resolution imaging of non-fluorescent samples. Experimental confirmation of the approach is obtained in a reflection test target with moderate numerical aperture.