Autonomous illumination control for localization microscopy

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

An integrated platform for high-throughput nanoscopy

Single-molecule localization microscopy enables three-dimensional fluorescence imaging at tens-of-nanometer resolution, but requires many camera frames to reconstruct a super-resolved image. This limits the typical throughput to tens of cells per day. While frame rates can now be increased by over an order of magnitude, the large data volumes become limiting in existing workflows. Here we present an integrated acquisition and analysis platform leveraging microscopy-specific data compression, distributed storage and distributed analysis to enable an acquisition and analysis throughput of 10,000 cells per day. The platform facilitates graphically reconfigurable analyses to be automatically initiated from the microscope during acquisition and remotely executed, and can even feed back and queue new acquisition tasks on the microscope. We demonstrate the utility of this framework by imaging hundreds of cells per well in multi-well sample formats. Our platform, implemented within the PYthon-Microscopy Environment (PYME), is easily configurable to control custom microscopes, and includes a plugin framework for user-defined extensions.

Light-sheets and smart microscopy, an exciting future is dawning

Light-sheet fluorescence microscopy has transformed our ability to visualize and quantitatively measure biological processes rapidly and over long time periods. In this review, we discuss current and future developments in light-sheet fluorescence microscopy that we expect to further expand its capabilities. This includes smart and adaptive imaging schemes to overcome traditional imaging trade-offs, i.e., spatiotemporal resolution, field of view and sample health. In smart microscopy, a microscope will autonomously decide where, when, what and how to image. We further assess how image restoration techniques provide avenues to overcome these tradeoffs and how "open top" light-sheet microscopes may enable multi-modal imaging with high throughput. As such, we predict that light-sheet microscopy will fulfill an important role in biomedical and clinical imaging in the future.

Dual-step reconstruction algorithm to improve microscopy resolution by deep learning

Deep learning plays an important role in the field of machine learning, which has been developed and used in a wide range of areas. Many deep-learning-based methods have been proposed to improve image resolution, most of which are based on image-to-image translation algorithms. The performance of neural networks used to achieve image translation always depends on the feature difference between input and output images. Therefore, these deep-learning-based methods sometimes do not have good performance when the feature differences between low-resolution and high-resolution images are too large. In this paper, we introduce a dual-step neural network algorithm to improve image resolution step by step. Compared with conventional deep-learning methods that use input and output images with huge differences for training, this algorithm learning from input and output images with fewer differences can improve the performance of neural networks. This method was used to reconstruct high-resolution images of fluorescence nanoparticles in cells.

Optofluidic imaging meets deep learning: from merging to emerging

Propelled by the striking advances in optical microscopy and deep learning (DL), the role of imaging in lab-on-a-chip has dramatically been transformed from a silo inspection tool to a quantitative "smart" engine. A suite of advanced optical microscopes now enables imaging over a range of spatial scales (from molecules to organisms) and temporal window (from microseconds to hours). On the other hand, the staggering diversity of DL algorithms has revolutionized image processing and analysis at the scale and complexity that were once inconceivable. Recognizing these exciting but overwhelming developments, we provide a timely review of their latest trends in the context of lab-on-a-chip imaging, or coined optofluidic imaging. More importantly, here we discuss the strengths and caveats of how to adopt, reinvent, and integrate these imaging techniques and DL algorithms in order to tailor different lab-on-a-chip applications. In particular, we highlight three areas where the latest advances in lab-on-a-chip imaging and DL can form unique synergisms: image formation, image analytics and intelligent image-guided autonomous lab-on-a-chip. Despite the on-going challenges, we anticipate that they will represent the next frontiers in lab-on-a-chip imaging that will spearhead new capabilities in advancing analytical chemistry research, accelerating biological discovery, and empowering new intelligent clinical applications.

Event-driven acquisition for content-enriched microscopy

A common goal of fluorescence microscopy is to collect data on specific biological events. Yet, the event-specific content that can be collected from a sample is limited, especially for rare or stochastic processes. This is due in part to photobleaching and phototoxicity, which constrain imaging speed and duration. We developed an event-driven acquisition framework, in which neural-network-based recognition of specific biological events triggers real-time control in an instant structured illumination microscope. Our setup adapts acquisitions on-the-fly by switching between a slow imaging rate while detecting the onset of events, and a fast imaging rate during their progression. Thus, we capture mitochondrial and bacterial divisions at imaging rates that match their dynamic timescales, while extending overall imaging durations. Because event-driven acquisition allows the microscope to respond specifically to complex biological events, it acquires data enriched in relevant content.

Real-time image resolution measurement for single molecule localization microscopy

Recent advancements in single molecule localization microscopy (SMLM) have demonstrated outstanding potential applications in high-throughput and high-content screening imaging. One major limitation to such applications is to find a way to optimize imaging throughput without scarifying image quality, especially the homogeneity in image resolution, during the imaging of hundreds of field-of-views (FOVs) in heterogeneous samples. Here we introduce a real-time image resolution measurement method for SMLM to solve this problem. This method is under the heuristic framework of overall image resolution that counts on localization precision and localization density. Rather than estimating the mean localization density after completing the entire SMLM process, this method uses the spatial Poisson process to model the random activation of molecules and thus determines the localization density in real-time. We demonstrate that the method is valid in real-time resolution measurement and is effective in guaranteeing homogeneous image resolution across multiple representative FOVs with optimized imaging throughput.

Smart imaging to empower brain-wide neuroscience at single-cell levels

A deep understanding of the neuronal connectivity and networks with detailed cell typing across brain regions is necessary to unravel the mechanisms behind the emotional and memorial functions as well as to find the treatment of brain impairment. Brain-wide imaging with single-cell resolution provides unique advantages to access morphological features of a neuron and to investigate the connectivity of neuron networks, which has led to exciting discoveries over the past years based on animal models, such as rodents. Nonetheless, high-throughput systems are in urgent demand to support studies of neural morphologies at larger scale and more detailed level, as well as to enable research on non-human primates (NHP) and human brains. The advances in artificial intelligence (AI) and computational resources bring great opportunity to 'smart' imaging systems, i.e., to automate, speed up, optimize and upgrade the imaging systems with AI and computational strategies. In this light, we review the important computational techniques that can support smart systems in brain-wide imaging at single-cell resolution.

Raw Data to Results: A Hands-On Introduction and Overview of Computational Analysis for Single-Molecule Localization Microscopy

Single-molecule localization microscopy (SMLM) is an advanced microscopy method that uses the blinking of fluorescent molecules to determine the position of these molecules with a resolution below the diffraction limit (∼5-40 nm). While SMLM imaging itself is becoming more popular, the computational analysis surrounding the technique is still a specialized area and often remains a "black box" for experimental researchers. Here, we provide an introduction to the required computational analysis of SMLM imaging, post-processing and typical data analysis. Importantly, user-friendly, ready-to-use and well-documented code in Python and MATLAB with exemplary data is provided as an interactive experience for the reader, as well as a starting point for further analysis. Our code is supplemented by descriptions of the computational problems and their implementation. We discuss the state of the art in computational methods and software suites used in SMLM imaging and data analysis. Finally, we give an outlook into further computational challenges in the field.

Technological advances in super-resolution microscopy to study cellular processes

Since its initial demonstration in 2000, far-field super-resolution light microscopy has undergone tremendous technological developments. In parallel, these developments have opened a new window into visualizing the inner life of cells at unprecedented levels of detail. Here, we review the technical details behind the most common implementations of super-resolution microscopy and highlight some of the recent, promising advances in this field.

Deep Learning in Biomedical Optics

This article reviews deep learning applications in biomedical optics with a particular emphasis on image formation. The review is organized by imaging domains within biomedical optics and includes microscopy, fluorescence lifetime imaging, in vivo microscopy, widefield endoscopy, optical coherence tomography, photoacoustic imaging, diffuse tomography, and functional optical brain imaging. For each of these domains, we summarize how deep learning has been applied and highlight methods by which deep learning can enable new capabilities for optics in medicine. Challenges and opportunities to improve translation and adoption of deep learning in biomedical optics are also summarized. Lasers Surg. Med. © 2021 Wiley Periodicals LLC.

Challenges facing quantitative large-scale optical super-resolution, and some simple solutions

Optical super-resolution microscopy (SRM) has enabled biologists to visualize cellular structures with near-molecular resolution, giving unprecedented access to details about the amounts, sizes, and spatial distributions of macromolecules in the cell. Precisely quantifying these molecular details requires large datasets of high-quality, reproducible SRM images. In this review, we discuss the unique set of challenges facing quantitative SRM, giving particular attention to the shortcomings of conventional specimen preparation techniques and the necessity for optimal labeling of molecular targets. We further discuss the obstacles to scaling SRM methods, such as lengthy image acquisition and complex SRM data analysis. For each of these challenges, we review the recent advances in the field that circumvent these pitfalls and provide practical advice to biologists for optimizing SRM experiments.

Analysing errors in single-molecule localisation microscopy

In single molecule localisation microscopy (SMLM) a super-resolution image of the distribution of fluorophores in the sample is built up from the localised positions of many individual molecules. It has become widely used due to its experimental simplicity and the high resolution that can be achieved. However, the factors which limit resolution in a reconstructed image, and the artefacts which can be present, are completely different to those present in standard fluorescent microscopy techniques. Artefacts may be difficult for users to identify, particularly as they can cause images to appear falsely sharp, an effect called artificial sharpening. Here we discuss the different sources of error and bias in SMLM, and the methods available for avoiding or detecting them.

Parameter-free image resolution estimation based on decorrelation analysis

Super-resolution microscopy opened diverse new avenues of research by overcoming the resolution limit imposed by diffraction. Exploitation of the fluorescent emission of individual fluorophores made it possible to reveal structures beyond the diffraction limit. To accurately determine the resolution achieved during imaging is challenging with existing metrics. Here, we propose a method for assessing the resolution of individual super-resolved images based on image partial phase autocorrelation. The algorithm is model-free and does not require any user-defined parameters. We demonstrate its performance on a wide variety of imaging modalities, including diffraction-limited techniques. Finally, we show how our method can be used to optimize image acquisition and post-processing in super-resolution microscopy.

Divide and conquer: real-time maximum likelihood fitting of multiple emitters for super-resolution localization microscopy

Multi-emitter localization has great potential for maximizing the imaging speed of super-resolution localization microscopy. However, the slow image analysis speed of reported multi-emitter localization algorithms limits their usage in mostly off-line image processing with small image size. Here we adopt the well-known divide and conquer strategy in computer science and present a fitting-based method called QC-STORM for fast multi-emitter localization. Using simulated and experimental data, we verify that QC-STORM is capable of providing real-time full image processing on raw images with 100 µm × 100 µm field of view and 10 ms exposure time, with comparable spatial resolution as the popular fitting-based ThunderSTORM and the up-to-date non-iterative WindSTORM. This study pushes the development and practical use of super-resolution localization microscopy in high-throughput or high-content imaging of cell-to-cell differences or discovering rare events in a large cell population.