Rapid image deconvolution and multiview fusion for optical microscopy

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

Zero-shot learning enables instant denoising and super-resolution in optical fluorescence microscopy

Computational super-resolution methods, including conventional analytical algorithms and deep learning models, have substantially improved optical microscopy. Among them, supervised deep neural networks have demonstrated outstanding performance, however, demanding abundant high-quality training data, which are laborious and even impractical to acquire due to the high dynamics of living cells. Here, we develop zero-shot deconvolution networks (ZS-DeconvNet) that instantly enhance the resolution of microscope images by more than 1.5-fold over the diffraction limit with 10-fold lower fluorescence than ordinary super-resolution imaging conditions, in an unsupervised manner without the need for either ground truths or additional data acquisition. We demonstrate the versatile applicability of ZS-DeconvNet on multiple imaging modalities, including total internal reflection fluorescence microscopy, three-dimensional wide-field microscopy, confocal microscopy, two-photon microscopy, lattice light-sheet microscopy, and multimodal structured illumination microscopy, which enables multi-color, long-term, super-resolution 2D/3D imaging of subcellular bioprocesses from mitotic single cells to multicellular embryos of mouse and C. elegans.

Live-cell single-molecule fluorescence microscopy for protruding organelles reveals regulatory mechanisms of MYO7A-driven cargo transport in stereocilia of inner ear hair cells

Stereocilia are unidirectional F-actin-based cylindrical protrusions on the apical surface of inner ear hair cells and function as biological mechanosensors of sound and acceleration. Development of functional stereocilia requires motor activities of unconventional myosins to transport proteins necessary for elongating the F-actin cores and to assemble the mechanoelectrical transduction (MET) channel complex. However, how each myosin localizes in stereocilia using the energy from ATP hydrolysis is only partially understood. In this study, we develop a methodology for live-cell single-molecule fluorescence microscopy of organelles protruding from the apical surface using a dual-view light-sheet microscope, diSPIM. We demonstrate that MYO7A, a component of the MET machinery, traffics as a dimer in stereocilia. Movements of MYO7A are restricted when scaffolded by the plasma membrane and F-actin as mediated by MYO7A's interacting partners. Here, we discuss the technical details of our methodology and its future applications including analyses of cargo transportation in various organelles.

Light-Sheet Imaging to Reveal Cardiac Structure in Rodent Hearts

Light-sheet microscopy (LSM) plays a pivotal role in comprehending the intricate three-dimensional (3D) structure of the heart, providing crucial insights into fundamental cardiac physiology and pathologic responses. We hereby delve into the development and implementation of the LSM technique to elucidate the micro-architecture of the heart in mouse models. The methodology integrates a customized LSM system with tissue clearing techniques, mitigating light scattering within cardiac tissues for volumetric imaging. The combination of conventional LSM with image stitching and multiview deconvolution approaches allows for the capture of the entire heart. To address the inherent trade-off between axial resolution and field of view (FOV), we further introduce an axially swept light-sheet microscopy (ASLM) method to minimize out-of-focus light and uniformly illuminate the heart across the propagation direction. In the meanwhile, tissue clearing methods such as iDISCO enhance light penetration, facilitating the visualization of deep structures and ensuring a comprehensive examination of the myocardium throughout the entire heart. The combination of the proposed LSM and tissue clearing methods presents a promising platform for researchers in resolving cardiac structures in rodent hearts, holding great potential for the understanding of cardiac morphogenesis and remodeling.

Transgenic Tg(Kcnj10-ZsGreen) fluorescent reporter mice allow visualization of intermediate cells in the stria vascularis

The stria vascularis (SV) is a stratified epithelium in the lateral wall of the mammalian cochlea, responsible for both endolymphatic ion homeostasis and generation of the endocochlear potential (EP) critical for normal hearing. The SV has three layers consisting predominantly of basal, intermediate, and marginal cells. Intermediate and marginal cells form an intricate interdigitated network of cell projections making discrimination of the cells challenging. To enable intermediate cell visualization, we engineered by BAC transgenesis, reporter mouse lines expressing ZsGreen fluorescent protein under the control of Kcnj10 promoter and regulatory sequences. Kcnj10 encodes KCNJ10 protein (also known as Kir4.1 or Kir1.2), an ATP-sensitive inwardly-rectifying potassium channel critical to EP generation, highly expressed in SV intermediate cells. In these transgenic mice, ZsGreen fluorescence mimics Kcnj10 endogenous expression in the cochlea and was detected in the intermediate cells of the SV, in the inner phalangeal cells, Hensen's, Deiters' and pillar cells, in a subset of spiral ganglion neurons, and in glial cells. We show that expression of the transgene in hemizygous mice does not alter auditory function, nor EP. These transgenic Tg(Kcnj10-ZsGreen) mice allow live and fixed tissue visualization of ZsGreen-expressing intermediate cells and will facilitate future studies of stria vascularis cell function.

Harnessing artificial intelligence to reduce phototoxicity in live imaging

Fluorescence microscopy is essential for studying living cells, tissues and organisms. However, the fluorescent light that switches on fluorescent molecules also harms the samples, jeopardizing the validity of results - particularly in techniques such as super-resolution microscopy, which demands extended illumination. Artificial intelligence (AI)-enabled software capable of denoising, image restoration, temporal interpolation or cross-modal style transfer has great potential to rescue live imaging data and limit photodamage. Yet we believe the focus should be on maintaining light-induced damage at levels that preserve natural cell behaviour. In this Opinion piece, we argue that a shift in role for AIs is needed - AI should be used to extract rich insights from gentle imaging rather than recover compromised data from harsh illumination. Although AI can enhance imaging, our ultimate goal should be to uncover biological truths, not just retrieve data. It is essential to prioritize minimizing photodamage over merely pushing technical limits. Our approach is aimed towards gentle acquisition and observation of undisturbed living systems, aligning with the essence of live-cell fluorescence microscopy.

Noise learning of instruments for high-contrast, high-resolution and fast hyperspectral microscopy and nanoscopy

The low scattering efficiency of Raman scattering makes it challenging to simultaneously achieve good signal-to-noise ratio (SNR), high imaging speed, and adequate spatial and spectral resolutions. Here, we report a noise learning (NL) approach that estimates the intrinsic noise distribution of each instrument by statistically learning the noise in the pixel-spatial frequency domain. The estimated noise is then removed from the noisy spectra. This enhances the SNR by ca. 10 folds, and suppresses the mean-square error by almost 150 folds. NL allows us to improve the positioning accuracy and spatial resolution and largely eliminates the impact of thermal drift on tip-enhanced Raman spectroscopic nanoimaging. NL is also applicable to enhance SNR in fluorescence and photoluminescence imaging. Our method manages the ground truth spectra and the instrumental noise simultaneously within the training dataset, which bypasses the tedious labelling of huge dataset required in conventional deep learning, potentially shifting deep learning from sample-dependent to instrument-dependent.

Pro-regenerative biomaterials recruit immunoregulatory dendritic cells after traumatic injury

During wound healing and surgical implantation, the body establishes a delicate balance between immune activation to fight off infection and clear debris and immune tolerance to control reactivity against self-tissue. Nonetheless, how such a balance is achieved is not well understood. Here we describe that pro-regenerative biomaterials for muscle injury treatment promote the proliferation of a BATF3-dependent CD103+XCR1+CD206+CD301b+ dendritic cell population associated with cross-presentation and self-tolerance. Upregulation of E-cadherin, the ligand for CD103, and XCL-1 in injured tissue suggests a mechanism for cell recruitment to trauma. Muscle injury recruited natural killer cells that produced Xcl1 when stimulated with fragmented extracellular matrix. Without cross-presenting cells, T-cell activation increases, pro-regenerative macrophage polarization decreases and there are alterations in myogenesis, adipogenesis, fibrosis and increased muscle calcification. These results, previously observed in cancer progression, suggest a fundamental mechanism of immune regulation in trauma and material implantation with implications for both short- and long-term injury recovery.

Enhancing Light-Sheet Fluorescence Microscopy Illumination Beams through Deep Design Optimization

Light sheet fluorescence microscopy (LSFM) provides the benefit of optical sectioning coupled with rapid acquisition times for imaging of tissue-cleared specimen. This allows for high-resolution 3D imaging of large tissue volumes. Inherently to LSFM, the quality of the imaging heavily relies on the characteristics of the illumination beam, with the notion that the illumination beam only illuminates a thin section that is being imaged. Therefore, substantial efforts are dedicated to identifying slender, non-diffracting beam profiles that can yield uniform and high-contrast images. An ongoing debate concerns the employment of the most optimal illumination beam; Gaussian, Bessel, Airy patterns and/or others. Comparisons among different beam profiles is challenging as their optimization objective is often different. Given that our large imaging datasets (~0.5TB images per sample) is already analyzed using deep learning models, we envisioned a different approach to this problem by hypothesizing that we can tailor the illumination beam to boost the deep learning models performance. We achieve this by integrating the physical LSFM illumination model after passing through a variable phase mask into the training of a cell detection network. Here we report that the joint optimization continuously updates the phase mask, improving the image quality for better cell detection. Our method's efficacy is demonstrated through both simulations and experiments, revealing substantial enhancements in imaging quality compared to traditional Gaussian light sheet. We offer valuable insights for designing microscopy systems through a computational approach that exhibits significant potential for advancing optics design that relies on deep learning models for analysis of imaging datasets.

Universal inverse modelling of point spread functions for SMLM localization and microscope characterization

The point spread function (PSF) of a microscope describes the image of a point emitter. Knowing the accurate PSF model is essential for various imaging tasks, including single molecule localization, aberration correction and deconvolution. Here we present uiPSF (universal inverse modelling of Point Spread Functions), a toolbox to infer accurate PSF models from microscopy data, using either image stacks of fluorescent beads or directly images of blinking fluorophores, the raw data in single molecule localization microscopy (SMLM). The resulting PSF model enables accurate 3D super-resolution imaging using SMLM. Additionally, uiPSF can be used to characterize and optimize a microscope system by quantifying the aberrations, including field-dependent aberrations, and resolutions. Our modular framework is applicable to a variety of microscope modalities and the PSF model incorporates system or sample specific characteristics, e.g., the bead size, depth dependent aberrations and transformations among channels. We demonstrate its application in single or multiple channels or large field-of-view SMLM systems, 4Pi-SMLM, and lattice light-sheet microscopes using either bead data or single molecule blinking data.

[Oral panorama reconstruction method based on pre-segmentation and Bezier function]

For patients with partial jaw defects, cysts and dental implants, doctors need to take panoramic X-ray films or manually draw dental arch lines to generate Panorama images in order to observe their complete dentition information during oral diagnosis. In order to solve the problems of additional burden for patients to take panoramic X-ray films and time-consuming issue for doctors to manually segment dental arch lines, this paper proposes an automatic panorama reconstruction method based on cone beam computerized tomography (CBCT). The V-network (VNet) is used to pre-segment the teeth and the background to generate the corresponding binary image, and then the Bezier curve is used to define the best dental arch curve to generate the oral panorama. In addition, this research also addressed the issues of mistakenly recognizing the teeth and jaws as dental arches, incomplete coverage of the dental arch area by the generated dental arch lines, and low robustness, providing intelligent methods for dental diagnosis and improve the work efficiency of doctors.

Deep learning-based aberration compensation improves contrast and resolution in fluorescence microscopy

Optical aberrations hinder fluorescence microscopy of thick samples, reducing image signal, contrast, and resolution. Here we introduce a deep learning-based strategy for aberration compensation, improving image quality without slowing image acquisition, applying additional dose, or introducing more optics into the imaging path. Our method (i) introduces synthetic aberrations to images acquired on the shallow side of image stacks, making them resemble those acquired deeper into the volume and (ii) trains neural networks to reverse the effect of these aberrations. We use simulations to show that applying the trained 'de-aberration' networks outperforms alternative methods, and subsequently apply the networks to diverse datasets captured with confocal, light-sheet, multi-photon, and super-resolution microscopy. In all cases, the improved quality of the restored data facilitates qualitative image inspection and improves downstream image quantitation, including orientational analysis of blood vessels in mouse tissue and improved membrane and nuclear segmentation in C. elegans embryos.

Transgenic Tg(Kcnj10-ZsGreen) Fluorescent Reporter Mice Allow Visualization of Intermediate Cells in the Stria Vascularis

The stria vascularis (SV) is a stratified epithelium in the lateral wall of the mammalian cochlea, responsible for both endolymphatic ion homeostasis and generation of the endocochlear potential (EP) critical for normal hearing. The SV has three layers consisting predominantly of basal, intermediate, and marginal cells. Intermediate and marginal cells form an intricate interdigitated network of cell projections making discrimination of the cells challenging. To enable intermediate cell visualization, we engineered by BAC transgenesis, reporter mouse lines expressing ZsGreen fluorescent protein under the control of Kcnj10 promoter and regulatory sequences. Kcnj10 encodes KCNJ10 protein (also known as Kir4.1 or Kir1.2), an ATP-sensitive inwardly-rectifying potassium channel critical to EP generation, highly expressed in SV intermediate cells. In these transgenic mice, ZsGreen fluorescence mimics Kcnj10 endogenous expression in the cochlea and was detected in the intermediate cells of the SV, in the inner phalangeal cells, Hensen's, Deiters' and pillar cells, in a subset of spiral ganglion neurons, and in glial cells. We show that expression of the transgene in hemizygous mice does not alter auditory function, nor EP These transgenic Tg(Kcnj10-ZsGreen) mice allow live and fixed tissue visualization of ZsGreen-expressing intermediate cells and will facilitate future studies of stria vascularis cell function.

Experimental and theoretical model for the origin of coiling of cellular protrusions around fibers

Protrusions at the leading-edge of a cell play an important role in sensing the extracellular cues during cellular spreading and motility. Recent studies provided indications that these protrusions wrap (coil) around the extracellular fibers. However, the physics of this coiling process, and the mechanisms that drive it, are not well understood. We present a combined theoretical and experimental study of the coiling of cellular protrusions on fibers of different geometry. Our theoretical model describes membrane protrusions that are produced by curved membrane proteins that recruit the protrusive forces of actin polymerization, and identifies the role of bending and adhesion energies in orienting the leading-edges of the protrusions along the azimuthal (coiling) direction. Our model predicts that the cell's leading-edge coils on fibers with circular cross-section (above some critical radius), but the coiling ceases for flattened fibers of highly elliptical cross-section. These predictions are verified by 3D visualization and quantitation of coiling on suspended fibers using Dual-View light-sheet microscopy (diSPIM). Overall, we provide a theoretical framework, supported by experiments, which explains the physical origin of the coiling phenomenon.

Three-dimensional structured illumination microscopy with enhanced axial resolution

The axial resolution of three-dimensional structured illumination microscopy (3D SIM) is limited to ∼300 nm. Here we present two distinct, complementary methods to improve axial resolution in 3D SIM with minimal or no modification to the optical system. We show that placing a mirror directly opposite the sample enables four-beam interference with higher spatial frequency content than 3D SIM illumination, offering near-isotropic imaging with ∼120-nm lateral and 160-nm axial resolution. We also developed a deep learning method achieving ∼120-nm isotropic resolution. This method can be combined with denoising to facilitate volumetric imaging spanning dozens of timepoints. We demonstrate the potential of these advances by imaging a variety of cellular samples, delineating the nanoscale distribution of vimentin and microtubule filaments, observing the relative positions of caveolar coat proteins and lysosomal markers and visualizing cytoskeletal dynamics within T cells in the early stages of immune synapse formation.

Optimal sparsity allows reliable system-aware restoration of fluorescence microscopy images

Fluorescence microscopy is one of the most indispensable and informative driving forces for biological research, but the extent of observable biological phenomena is essentially determined by the content and quality of the acquired images. To address the different noise sources that can degrade these images, we introduce an algorithm for multiscale image restoration through optimally sparse representation (MIRO). MIRO is a deterministic framework that models the acquisition process and uses pixelwise noise correction to improve image quality. Our study demonstrates that this approach yields a remarkable restoration of the fluorescence signal for a wide range of microscopy systems, regardless of the detector used (e.g., electron-multiplying charge-coupled device, scientific complementary metal-oxide semiconductor, or photomultiplier tube). MIRO improves current imaging capabilities, enabling fast, low-light optical microscopy, accurate image analysis, and robust machine intelligence when integrated with deep neural networks. This expands the range of biological knowledge that can be obtained from fluorescence microscopy.

Deep self-learning enables fast, high-fidelity isotropic resolution restoration for volumetric fluorescence microscopy

One intrinsic yet critical issue that troubles the field of fluorescence microscopy ever since its introduction is the unmatched resolution in the lateral and axial directions (i.e., resolution anisotropy), which severely deteriorates the quality, reconstruction, and analysis of 3D volume images. By leveraging the natural anisotropy, we present a deep self-learning method termed Self-Net that significantly improves the resolution of axial images by using the lateral images from the same raw dataset as rational targets. By incorporating unsupervised learning for realistic anisotropic degradation and supervised learning for high-fidelity isotropic recovery, our method can effectively suppress the hallucination with substantially enhanced image quality compared to previously reported methods. In the experiments, we show that Self-Net can reconstruct high-fidelity isotropic 3D images from organelle to tissue levels via raw images from various microscopy platforms, e.g., wide-field, laser-scanning, or super-resolution microscopy. For the first time, Self-Net enables isotropic whole-brain imaging at a voxel resolution of 0.2 × 0.2 × 0.2 μm3, which addresses the last-mile problem of data quality in single-neuron morphology visualization and reconstruction with minimal effort and cost. Overall, Self-Net is a promising approach to overcoming the inherent resolution anisotropy for all classes of 3D fluorescence microscopy.

Augmented Super-Resolution Radial Fluctuations (aSRRF) Pushing the Limits of Structured Illumination Microscopy

Structured illumination microscopy (SIM) is a versatile super-resolution technique known for its compatibility with a wide range of probes and fast implementation. While 3D SIM is capable of achieving a spatial resolution of ∼120 nm laterally and ∼300 nm axially, attempting to further enhance the resolution through methods such as nonlinear SIM or 4-beam SIM introduces complexities in optical configurations, increased phototoxicity, and reduced temporal resolution. Here, we have developed a novel method that combines SIM with augmented super-resolution radial fluctuations (aSRRF) utilizing a single image through image augmentation. By applying aSRRF reconstruction to SIM images, we can enhance the SIM resolution to ∼50 nm isotopically, without requiring any modifications to the optical system or sample acquisition process. Additionaly, we have incorporated the aSRRF approach into an ImageJ plugin and demonstrated its versatility across various fluorescence microscopy images, showcasing a remarkable two-fold resolution increase.

Expansion-assisted selective plane illumination microscopy for nanoscale imaging of centimeter-scale tissues

Recent advances in tissue processing, labeling, and fluorescence microscopy are providing unprecedented views of the structure of cells and tissues at sub-diffraction resolutions and near single molecule sensitivity, driving discoveries in diverse fields of biology, including neuroscience. Biological tissue is organized over scales of nanometers to centimeters. Harnessing molecular imaging across three-dimensional samples on this scale requires new types of microscopes with larger fields of view and working distance, as well as higher imaging throughput. We present a new expansion-assisted selective plane illumination microscope (ExA-SPIM) with diffraction-limited and aberration-free performance over a large field of view (85 mm 2 ) and working distance (35 mm). Combined with new tissue clearing and expansion methods, the microscope allows nanoscale imaging of centimeter-scale samples, including entire mouse brains, with diffraction-limited resolutions and high contrast without sectioning. We illustrate ExA-SPIM by reconstructing individual neurons across the mouse brain, imaging cortico-spinal neurons in the macaque motor cortex, and tracing axons in human white matter.

Airy beam light sheet microscopy boosted by deep learning deconvolution

Common light sheet microscopy comes with a trade-off between light sheet width defining the optical sectioning and the usable field of view arising from the divergence of the illuminating Gaussian beam. To overcome this, low-diverging Airy beams have been introduced. Airy beams, however, exhibit side lobes degrading image contrast. Here, we constructed an Airy beam light sheet microscope, and developed a deep learning image deconvolution to remove the effects of the side lobes without knowledge of the point spread function. Using a generative adversarial network and high-quality training data, we significantly enhanced image contrast and improved the performance of a bicubic upscaling. We evaluated the performance with fluorescently labeled neurons in mouse brain tissue samples. We found that deep learning-based deconvolution was about 20-fold faster than the standard approach. The combination of Airy beam light sheet microscopy and deep learning deconvolution allows imaging large volumes rapidly and with high quality.

Label-free cleared tissue microscopy and machine learning for 3D histopathology of biomaterial implants

Tissue clearing of whole intact organs has enhanced imaging by enabling the exploration of tissue structure at a subcellular level in three-dimensional space. Although clearing and imaging of the whole organ have been used to study tissue biology, the microenvironment in which cells evolve to adapt to biomaterial implants or allografts in the body is poorly understood. Obtaining high-resolution information from complex cell-biomaterial interactions with volumetric landscapes represents a key challenge in the fields of biomaterials and regenerative medicine. To provide a new approach to examine how tissue responds to biomaterial implants, we apply cleared tissue light-sheet microscopy and three-dimensional reconstruction to utilize the wealth of autofluorescence information for visualizing and contrasting anatomical structures. This study demonstrates the adaptability of the clearing and imaging technique to provide sub-cellular resolution (0.6 μm isotropic) 3D maps of various tissue types, using samples from fully intact peritoneal organs to volumetric muscle loss injury specimens. Specifically, in the volumetric muscle loss injury model, we provide 3D visualization of the implanted extracellular matrix biomaterial in the wound bed of the quadricep muscle groups and further apply computational-driven image classification to analyze the autofluorescence spectrum at multiple emission wavelengths to categorize tissue types at the injured site interacting with the biomaterial scaffolds.

Partially interpretable image deconvolution framework based on the Richardson-Lucy model

Fluorescence microscopy typically suffers from aberration induced by system and sample, which could be circumvented by image deconvolution. We proposed a novel, to the best of our knowledge, Richardson-Lucy (RL) model-driven deconvolution framework to improve reconstruction performance and speed. Two kinds of neural networks within this framework were devised, which are partially interpretable compared with previous deep learning methods. We first introduce RL into deep feature space, which has superior generalizability to the convolutional neural networks (CNN). We further accelerate it with an unmatched backprojector, providing a five times faster reconstruction speed than classic RL. Our deconvolution approaches outperform both CNN and traditional methods regarding image quality for blurred images caused by out-of-focus or imaging system aberration.

Efficient 3D light-sheet imaging of very large-scale optically cleared human brain and prostate tissue samples

The ability to image human tissue samples in 3D, with both cellular resolution and a large field of view (FOV), can improve fundamental and clinical investigations. Here, we demonstrate the feasibility of light-sheet imaging of ~5 cm³ sized formalin fixed human brain and up to ~7 cm³ sized formalin fixed paraffin embedded (FFPE) prostate cancer samples, processed with the FFPE-MASH protocol. We present a light-sheet microscopy prototype, the cleared-tissue dual view Selective Plane Illumination Microscope (ct-dSPIM), capable of fast 3D high-resolution acquisitions of cm³ scale cleared tissue. We used mosaic scans for fast 3D overviews of entire tissue samples or higher resolution overviews of large ROIs with various speeds: (a) Mosaic 16 (16.4 µm isotropic resolution, ~1.7 h/cm³), (b) Mosaic 4 (4.1 µm isotropic resolution, ~ 5 h/cm³) and (c) Mosaic 0.5 (0.5 µm near isotropic resolution, ~15.8 h/cm³). We could visualise cortical layers and neurons around the border of human brain areas V1&V2, and could demonstrate suitable imaging quality for Gleason score grading in thick prostate cancer samples. We show that ct-dSPIM imaging is an excellent technique to quantitatively assess entire MASH prepared large-scale human tissue samples in 3D, with considerable future clinical potential.

Optimised generalized polarisation analysis of C-laurdan reveals clear order differences between T cell membrane compartments

Heterogenous packing of plasma membrane lipids is important for cellular processes like signalling, adhesion and sorting of membrane components. Solvatochromic membrane fluorophores that respond to changes from liquid-ordered (l_o) phase to liquid-disordered (l_d) by red shifts in their emission spectra are often used to assess lipid packing. Their response can be quantified using generalized polarisation (GP) using fluorescence microscopy images from two emission ranges, preferably from a region of interest (ROI) limited to a specific membrane compartment. However, image quality is limited by Poisson noise and convolution by the point spread function of the imaging system. Examining GP-analysis of C-laurdan labelled T cells using the image restoration procedure deconvolution, we demonstrate that deconvolution substantially improves the image resolution by making the plasma membrane clearly discernible and facilitating plasma membrane ROI selection. We conclude that automatic ROI selection has advantages over manual ROI selection when it comes to reproducibility and speed, but reliable GP-measurements can also be obtained by manually demarcated ROIs. We find that deconvolution enhances the difference in GP-values between the plasma and intracellular membranes and demonstrate that moving an intensity defined plasma membrane ROI outwards from the cell further improves this differentiation. By systematically changing the key deconvolution regularization parameter signal to noise, we establish a protocol for deconvolution optimisation applicable to any solvatochromic dye and imaging system. The image processing and ROI selection protocol presented improves both the resolution and precision of GP-measurement and will enable detection of smaller changes in membrane order than is currently achievable.

SPITFIR(e): a supermaneuverable algorithm for fast denoising and deconvolution of 3D fluorescence microscopy images and videos

Modern fluorescent microscopy imaging is still limited by the optical aberrations and the photon budget available in the specimen. A direct consequence is the necessity to develop flexible and "off-road" algorithms in order to recover structural details and improve spatial resolution, which is critical when restraining the illumination to low levels in order to limit photo-damages. Here, we report SPITFIR(e) a flexible method designed to accurately and quickly restore 2D-3D fluorescence microscopy images and videos (4D images). We designed a generic sparse-promoting regularizer to subtract undesirable out-of-focus background and we developed a primal-dual algorithm for fast optimization. SPITFIR(e) is a "swiss-knife" method for practitioners as it adapts to any microscopy techniques, to various sources of signal degradation (noise, blur), to variable image contents, as well as to low signal-to-noise ratios. Our method outperforms existing state-of-the-art algorithms, and is more flexible than supervised deep-learning methods requiring ground truth datasets. The performance, the flexibility, and the ability to push the spatiotemporal resolution limit of sub-diffracted fluorescence microscopy techniques are demonstrated on experimental datasets acquired with various microscopy techniques from 3D spinning-disk confocal up to lattice light sheet microscopy.

Analysis of Spatial Molecular Data

Digital analysis of pathology whole-slide images has been recently gaining interest in the context of cancer diagnosis and treatment. In particular, deep learning methods have demonstrated significant potential in supporting pathology analysis, recently detecting molecular traits never before recognized in pathology H&E whole-slide images (WSIs). Alongside these advancements in the digital analysis of WSIs, it is becoming increasingly evident that both spatial and overall tumor heterogeneity may be significant determinants of cancer prognosis and treatment outcome. In this chapter, we describe methods that aim to support these two elements. We describe both an end-to-end deep learning pipeline for producing limited spatial transcriptomics from WSIs with associated bulk gene expression data, as well as an algorithm for quantifying spatial tumor heterogeneity based on the results of this pipeline.

Practical considerations for quantitative light sheet fluorescence microscopy

Fluorescence microscopy has evolved from a purely observational tool to a platform for quantitative, hypothesis-driven research. As such, the demand for faster and less phototoxic imaging modalities has spurred a rapid growth in light sheet fluorescence microscopy (LSFM). By restricting the excitation to a thin plane, LSFM reduces the overall light dose to a specimen while simultaneously improving image contrast. However, the defining characteristics of light sheet microscopes subsequently warrant unique considerations in their use for quantitative experiments. In this Perspective, we outline many of the pitfalls in LSFM that can compromise analysis and confound interpretation. Moreover, we offer guidance in addressing these caveats when possible. In doing so, we hope to provide a useful resource for life scientists seeking to adopt LSFM to quantitatively address complex biological hypotheses.

Experimentally unsupervised deconvolution for light-sheet microscopy with propagation-invariant beams

Deconvolution is a challenging inverse problem, particularly in techniques that employ complex engineered point-spread functions, such as microscopy with propagation-invariant beams. Here, we present a deep-learning method for deconvolution that, in lieu of end-to-end training with ground truths, is trained using known physics of the imaging system. Specifically, we train a generative adversarial network with images generated with the known point-spread function of the system, and combine this with unpaired experimental data that preserve perceptual content. Our method rapidly and robustly deconvolves and super-resolves microscopy images, demonstrating a two-fold improvement in image contrast to conventional deconvolution methods. In contrast to common end-to-end networks that often require 1000-10,000s paired images, our method is experimentally unsupervised and can be trained solely on a few hundred regions of interest. We demonstrate its performance on light-sheet microscopy with propagation-invariant Airy beams in oocytes, preimplantation embryos and excised brain tissue, as well as illustrate its utility for Bessel-beam LSM. This method aims to democratise learned methods for deconvolution, as it does not require data acquisition outwith the conventional imaging protocol.

Incorporating the image formation process into deep learning improves network performance

We present Richardson–Lucy network (RLN), a fast and lightweight deep learning method for three-dimensional fluorescence microscopy deconvolution. RLN combines the traditional Richardson–Lucy iteration with a fully convolutional network structure, establishing a connection to the image formation process and thereby improving network performance. Containing only roughly 16,000 parameters, RLN enables four- to 50-fold faster processing than purely data-driven networks with many more parameters. By visual and quantitative analysis, we show that RLN provides better deconvolution, better generalizability and fewer artifacts than other networks, especially along the axial dimension. RLN outperforms classic Richardson–Lucy deconvolution on volumes contaminated with severe out of focus fluorescence or noise and provides four- to sixfold faster reconstructions of large, cleared-tissue datasets than classic multi-view pipelines. We demonstrate RLN’s performance on cells, tissues and embryos imaged with widefield-, light-sheet-, confocal- and super-resolution microscopy.

Richardson–Lucy Network (RLN) combines the traditional Richardson–Lucy iteration with deep learning for improved deconvolution. RLN is more generalizable, offers fewer artifacts and requires less computing time than alternative approaches.

Multiscale fluorescence imaging of living samples

Fluorescence microscopy is a highly effective tool for interrogating biological structure and function, particularly when imaging across multiple spatiotemporal scales. Here we survey recent innovations and applications in the relatively understudied area of multiscale fluorescence imaging of living samples. We discuss fundamental challenges in live multiscale imaging and describe successful examples that highlight the power of this approach. We attempt to synthesize general strategies from these test cases, aiming to help accelerate progress in this exciting area.

Review on data analysis methods for mesoscale neural imaging in vivo

Significance: Mesoscale neural imaging in vivo has gained extreme popularity in neuroscience for its capacity of recording large-scale neurons in action. Optical imaging with single-cell resolution and millimeter-level field of view in vivo has been providing an accumulated database of neuron-behavior correspondence. Meanwhile, optical detection of neuron signals is easily contaminated by noises, background, crosstalk, and motion artifacts, while neural-level signal processing and network-level coordinate are extremely complicated, leading to laborious and challenging signal processing demands. The existing data analysis procedure remains unstandardized, which could be daunting to neophytes or neuroscientists without computational background. Aim: We hope to provide a general data analysis pipeline of mesoscale neural imaging shared between imaging modalities and systems. Approach: We divide the pipeline into two main stages. The first stage focuses on extracting high-fidelity neural responses at single-cell level from raw images, including motion registration, image denoising, neuron segmentation, and signal extraction. The second stage focuses on data mining, including neural functional mapping, clustering, and brain-wide network deduction. Results: Here, we introduce the general pipeline of processing the mesoscale neural images. We explain the principles of these procedures and compare different approaches and their application scopes with detailed discussions about the shortcomings and remaining challenges. Conclusions: There are great challenges and opportunities brought by the large-scale mesoscale data, such as the balance between fidelity and efficiency, increasing computational load, and neural network interpretability. We believe that global circuits on single-neuron level will be more extensively explored in the future.

Image Segmentation and Quantification of Droplet dPCR Based on Thermal Bubble Printing Technology

Thermal inkjet printing can generate more than 300,000 droplets of picoliter scale within one second stably, and the image analysis workflow is used to quantify the positive and negative values of the droplets. In this paper, the SimpleBlobDetector detection algorithm is used to identify and localize droplets with a volume of 24 pL in bright field images and suppress bright spots and scratches when performing droplet location identification. The polynomial surface fitting of the pixel grayscale value of the fluorescence channel image can effectively compensate and correct the image vignetting caused by the optical path, and the compensated fluorescence image can accurately classify positive and negative droplets by the k-means clustering algorithm. 20 µL of the sample solution in the result reading chip can produce more than 100,000 effective droplets. The effective droplet identification correct rate of 20 images of random statistical samples can reach more than 99% and the classification accuracy of positive and negative droplets can reach more than 98% on average. This paper overcomes the problem of effectively classifying positive and negative droplets caused by the poor image quality of photographed picolitre ddPCR droplets caused by optical hardware limitations.

Stereotyped behavioral maturation and rhythmic quiescence in C.elegans embryos

Systematic analysis of rich behavioral recordings is being used to uncover how circuits encode complex behaviors. Here, we apply this approach to embryos. What are the first embryonic behaviors and how do they evolve as early neurodevelopment ensues? To address these questions, we present a systematic description of behavioral maturation for Caenorhabditis elegans embryos. Posture libraries were built using a genetically encoded motion capture suit imaged with light-sheet microscopy and annotated using custom tracking software. Analysis of cell trajectories, postures, and behavioral motifs revealed a stereotyped developmental progression. Early movement is dominated by flipping between dorsal and ventral coiling, which gradually slows into a period of reduced motility. Late-stage embryos exhibit sinusoidal waves of dorsoventral bends, prolonged bouts of directed motion, and a rhythmic pattern of pausing, which we designate slow wave twitch (SWT). Synaptic transmission is required for late-stage motion but not for early flipping nor the intervening inactive phase. A high-throughput behavioral assay and calcium imaging revealed that SWT is elicited by the rhythmic activity of a quiescence-promoting neuron (RIS). Similar periodic quiescent states are seen prenatally in diverse animals and may play an important role in promoting normal developmental outcomes.

Image Reconstruction in Light-Sheet Microscopy: Spatially Varying Deconvolution and Mixed Noise

We study the problem of deconvolution for light-sheet microscopy, where the data is corrupted by spatially varying blur and a combination of Poisson and Gaussian noise. The spatial variation of the point spread function of a light-sheet microscope is determined by the interaction between the excitation sheet and the detection objective PSF. We introduce a model of the image formation process that incorporates this interaction and we formulate a variational model that accounts for the combination of Poisson and Gaussian noise through a data fidelity term consisting of the infimal convolution of the single noise fidelities, first introduced in L. Calatroni et al. (SIAM J Imaging Sci 10(3):1196-1233, 2017). We establish convergence rates and a discrepancy principle for the infimal convolution fidelity and the inverse problem is solved by applying the primal-dual hybrid gradient (PDHG) algorithm in a novel way. Numerical experiments performed on simulated and real data show superior reconstruction results in comparison with other methods.

FCE-Net: a fast image contrast enhancement method based on deep learning for biomedical optical images

Optical imaging is an important tool for exploring and understanding structures of biological tissues. However, due to the heterogeneity of biological tissues, the intensity distribution of the signal is not uniform and contrast is normally degraded in the raw image. It is difficult to be used for subsequent image analysis and information extraction directly. Here, we propose a fast image contrast enhancement method based on deep learning called Fast Contrast Enhancement Network (FCE-Net). We divided network into dual-path to simultaneously obtain spatial information and large receptive field. And we introduced the spatial attention mechanism to enhance the inter-spatial relationship. We showed that the cell counting task of mouse brain images processed by FCE-Net was with average precision rate of 97.6% ± 1.6%, and average recall rate of 98.4% ± 1.4%. After processing with FCE-Net, the images from vascular extraction (DRIVE) dataset could be segmented with spatial attention U-Net (SA-UNet) to achieve state-of-the-art performance. By comparing FCE-Net with previous methods, we demonstrated that FCE-Net could obtain higher accuracy while maintaining the processing speed. The images with size of 1024 × 1024 pixels could be processed by FCE-Net with 37fps based on our workstation. Our method has great potential for further image analysis and information extraction from large-scale or dynamic biomedical optical images.

Using Light-Sheet Microscopy to Study Spontaneous Activity in the Developing Lateral-Line System

Hair cells are the sensory receptors in the auditory and vestibular systems of all vertebrates, and in the lateral-line system of aquatic vertebrates. The purpose of this work is to explore the zebrafish lateral-line system as a model to study and understand spontaneous activity in vivo. Our work applies genetically encoded calcium indicators along with light-sheet fluorescence microscopy to visualize spontaneous calcium activity in the developing lateral-line system. Consistent with our previous work, we show that spontaneous calcium activity is present in developing lateral-line hair cells. We now show that supporting cells that surround hair cells, and cholinergic efferent terminals that directly contact hair cells are also spontaneously active. Using two-color functional imaging we demonstrate that spontaneous activity in hair cells does not correlate with activity in either supporting cells or cholinergic terminals. We find that during lateral-line development, hair cells autonomously generate spontaneous events. Using localized calcium indicators, we show that within hair cells, spontaneous calcium activity occurs in two distinct domains—the mechanosensory bundle and the presynapse. Further, spontaneous activity in the mechanosensory bundle ultimately drives spontaneous calcium influx at the presynapse. Comprehensively, our results indicate that in developing lateral-line hair cells, autonomously generated spontaneous activity originates with spontaneous mechanosensory events.

Answering some questions about structured illumination microscopy

Structured illumination microscopy (SIM) provides images of fluorescent objects at an enhanced resolution greater than that of conventional epifluorescence wide-field microscopy. Initially demonstrated in 1999 to enhance the lateral resolution twofold, it has since been extended to enhance axial resolution twofold (2008), applied to live-cell imaging (2009) and combined with myriad other techniques, including interferometric detection (2008), confocal microscopy (2010) and light sheet illumination (2012). Despite these impressive developments, SIM remains, perhaps, the most poorly understood 'super-resolution' method. In this article, we provide answers to the 13 questions regarding SIM proposed by Prakash et al. along with answers to a further three questions. After providing a general overview of the technique and its developments, we explain why SIM as normally used is still diffraction-limited. We then highlight the necessity for a non-polynomial, and not just nonlinear, response to the illuminating light in order to make SIM a true, diffraction-unlimited, super-resolution technique. In addition, we present a derivation of a real-space SIM reconstruction approach that can be used to process conventional SIM and image scanning microscopy (ISM) data and extended to process data with quasi-arbitrary illumination patterns. Finally, we provide a simple bibliometric analysis of SIM development over the past two decades and provide a short outlook on potential future work. This article is part of the Theo Murphy meeting issue 'Super-resolution structured illumination microscopy (part 2)'.

Simulation-driven learning: a deep learning approach for image scanning microscopy via physical imaging models

Image reconstruction based on deep learning has become an effective tool in fluorescence microscopy. Most deep learning reconstruction methods ignore the mechanism of the imaging process where a large number of datasets are required. In addition, a lot of time is spent solving the aliasing problem from multi-scaled image pairs for data pre-processing. Here we demonstrate an improved generative adversarial network for image scanning microscopy (ISM) that can be trained by simulation data and has good generalization. Based on physical imaging models, this method can generate matching image pairs from simulation images and uses them as datasets for network training, without capturing a large number of real ISM images and avoiding image alignment preprocessing. Simulation and experimental results show that this simulation data-driven method improves the imaging quality of conventional microscopic images and reduces the cost of experiments. This method provides inspiration for optimizing network generalizability of the deep learning network.

Advanced high resolution three-dimensional imaging to visualize the cerebral neurovascular network in stroke

As an important method to accurately and timely diagnose stroke and study physiological characteristics and pathological mechanism in it, imaging technology has gone through more than a century of iteration. The interaction of cells densely packed in the brain is three-dimensional (3D), but the flat images brought by traditional visualization methods show only a few cells and ignore connections outside the slices. The increased resolution allows for a more microscopic and underlying view. Today's intuitive 3D imagings of micron or even nanometer scale are showing its essentiality in stroke. In recent years, 3D imaging technology has gained rapid development. With the overhaul of imaging mediums and the innovation of imaging mode, the resolution has been significantly improved, endowing researchers with the capability of holistic observation of a large volume, real-time monitoring of tiny voxels, and quantitative measurement of spatial parameters. In this review, we will summarize the current methods of high-resolution 3D imaging applied in stroke.

DaXi-high-resolution, large imaging volume and multi-view single-objective light-sheet microscopy

The promise of single-objective light-sheet microscopy is to combine the convenience of standard single-objective microscopes with the speed, coverage, resolution and gentleness of light-sheet microscopes. We present DaXi, a single-objective light-sheet microscope design based on oblique plane illumination that achieves: (1) a wider field of view and high-resolution imaging via a custom remote focusing objective; (2) fast volumetric imaging over larger volumes without compromising image quality or necessitating tiled acquisition; (3) fuller image coverage for large samples via multi-view imaging and (4) higher throughput multi-well imaging via remote coverslip placement. Our instrument achieves a resolution of 450 nm laterally and 2 μm axially over an imaging volume of 3,000 × 800 × 300 μm. We demonstrate the speed, field of view, resolution and versatility of our instrument by imaging various systems, including Drosophila egg chamber development, zebrafish whole-brain activity and zebrafish embryonic development – up to nine embryos at a time.

The DaXi single-objective light-sheet microscope achieves fast, high-quality imaging of large volumes. DaXi’s design allows increased scanning range without sacrificing imaging speed or quality, multiview imaging and versatile sample mounting.

Light-Field Microscopy for Optical Imaging of Neuronal Activity: When Model-Based Methods Meet Data-Driven Approaches

Understanding how networks of neurons process information is one of the key challenges in modern neuroscience. A necessary step to achieve this goal is to be able to observe the dynamics of large populations of neurons over a large area of the brain. Light-field microscopy (LFM), a type of scanless microscope, is a particularly attractive candidate for high-speed three-dimensional (3D) imaging. It captures volumetric information in a single snapshot, allowing volumetric imaging at video frame-rates. Specific features of imaging neuronal activity using LFM call for the development of novel machine learning approaches that fully exploit priors embedded in physics and optics models. Signal processing theory and wave-optics theory could play a key role in filling this gap, and contribute to novel computational methods with enhanced interpretability and generalization by integrating model-driven and data-driven approaches. This paper is devoted to a comprehensive survey to state-of-the-art of computational methods for LFM, with a focus on model-based and data-driven approaches.

Blind deconvolution in autocorrelation inversion for multiview light-sheet microscopy

Combining the information coming from multiview acquisitions is a problem of great interest in light-sheet microscopy. Aligning the views and increasing the resolution of their fusion can be challenging, especially if the setup is not fully calibrated. Here, we tackle these issues by proposing a new reconstruction method based on autocorrelation inversion that avoids alignment procedures. On top of this, we add a blind deconvolution step to improve the resolution of the final reconstruction. Our method permits us to achieve inherently aligned, highly resolved reconstructions while, at the same time, estimating the unknown point-spread function of the system.

Two plus one is almost three: a fast approximation for multi-view deconvolution

Multi-view deconvolution is a powerful image-processing tool for light sheet fluorescence microscopy, providing isotropic resolution and enhancing the image content. However, performing these calculations on large datasets is computationally demanding and time-consuming even on high-end workstations. Especially in long-time measurements on developing animals, huge amounts of image data are acquired. To keep them manageable, redundancies should be removed right after image acquisition. To this end, we report a fast approximation to three-dimensional multi-view deconvolution, denoted 2D+1D multi-view deconvolution, which is able to keep up with the data flow. It first operates on the two dimensions perpendicular and subsequently on the one parallel to the rotation axis, exploiting the rotational symmetry of the point spread function along the rotation axis. We validated our algorithm and evaluated it quantitatively against two-dimensional and three-dimensional multi-view deconvolution using simulated and real image data. 2D+1D multi-view deconvolution takes similar computation time but performs markedly better than the two-dimensional approximation only. Therefore, it will be most useful for image processing in time-critical applications, where the full 3D multi-view deconvolution cannot keep up with the data flow.

A deep-learning-based workflow to deal with the defocusing problem in high-throughput experiments

The increasing throughput of experiments in biomaterials research makes automatic techniques more and more necessary. Among all the characterization methods, microscopy makes fundamental contributions to biomaterials science where precisely focused images are the basis of related research. Although automatic focusing has been widely applied in all kinds of microscopes, defocused images can still be acquired now and then due to factors including background noises of materials and mechanical errors. Herein, we present a deep-learning-based method for the automatic sorting and reconstruction of defocused cell images. First, the defocusing problem is illustrated on a high-throughput cell microarray. Then, a comprehensive dataset of phase-contrast images captured from varied conditions containing multiple cell types, magnifications, and substrate materials is prepared to establish and test our method. We obtain high accuracy of over 0.993 on the dataset using a simple network architecture that requires less than half of the training time compared with the classical ResNetV2 architecture. Moreover, the subcellular-level reconstruction of heavily defocused cell images is achieved with another architecture. The applicability of the established workflow in practice is finally demonstrated on the high-throughput cell microarray. The intelligent workflow does not require a priori knowledge of focusing algorithms, possessing widespread application value in cell experiments concerning high-throughput or time-lapse imaging.

•

An automatic workflow for the sorting and reconstruction of defocused images.

•

Representative data containing varied cells, substrate materials, and magnifications.

•

Effectiveness of the workflow in high-throughput experiments.

•

Publicly accessible image datasets and codes.

State-of-the-Art Approaches for Image Deconvolution Problems, including Modern Deep Learning Architectures

In modern digital microscopy, deconvolution methods are widely used to eliminate a number of image defects and increase resolution. In this review, we have divided these methods into classical, deep learning-based, and optimization-based methods. The review describes the major architectures of neural networks, such as convolutional and generative adversarial networks, autoencoders, various forms of recurrent networks, and the attention mechanism used for the deconvolution problem. Special attention is paid to deep learning as the most powerful and flexible modern approach. The review describes the major architectures of neural networks used for the deconvolution problem. We describe the difficulties in their application, such as the discrepancy between the standard loss functions and the visual content and the heterogeneity of the images. Next, we examine how to deal with this by introducing new loss functions, multiscale learning, and prior knowledge of visual content. In conclusion, a review of promising directions and further development of deconvolution methods in microscopy is given.

High Resolution Powder Electron Diffraction in Scanning Electron Microscopy

A modern scanning electron microscope equipped with a pixelated detector of transmitted electrons can record a four-dimensional (4D) dataset containing a two-dimensional (2D) array of 2D nanobeam electron diffraction patterns; this is known as a four-dimensional scanning transmission electron microscopy (4D-STEM). In this work, we introduce a new version of our method called 4D-STEM/PNBD (powder nanobeam diffraction), which yields high-resolution powder diffractograms, whose quality is fully comparable to standard TEM/SAED (selected-area electron diffraction) patterns. Our method converts a complex 4D-STEM dataset measured on a nanocrystalline material to a single 2D powder electron diffractogram, which is easy to process with standard software. The original version of 4D-STEM/PNBD method, which suffered from low resolution, was improved in three important areas: (i) an optimized data collection protocol enables the experimental determination of the point spread function (PSF) of the primary electron beam, (ii) an improved data processing combines an entropy-based filtering of the whole dataset with a PSF-deconvolution of the individual 2D diffractograms and (iii) completely re-written software automates all calculations and requires just a minimal user input. The new method was applied to Au, TbF₃ and TiO₂ nanocrystals and the resolution of the 4D-STEM/PNBD diffractograms was even slightly better than that of TEM/SAED.