Denoising of Microscopy Images: A Review of the State-of-the-Art, and a New Sparsity-Based Method

Optical label-free microscopy characterization of dielectric nanoparticles

In order to relate nanoparticle properties to function, fast and detailed particle characterization is needed. The ability to characterize nanoparticle samples using optical microscopy techniques has drastically improved over the past few decades; consequently, there are now numerous microscopy methods available for detailed characterization of particles with nanometric size. However, there is currently no "one size fits all" solution to the problem of nanoparticle characterization. Instead, since the available techniques have different detection limits and deliver related but different quantitative information, the measurement and analysis approaches need to be selected and adapted for the sample at hand. In this tutorial, we review the optical theory of single particle scattering and how it relates to the differences and similarities in the quantitative particle information obtained from commonly used label-free microscopy techniques, with an emphasis on nanometric (submicron) sized dielectric particles. Particular emphasis is placed on how the optical signal relates to mass, size, structure, and material properties of the detected particles and to its combination with diffusivity-based particle sizing. We also discuss emerging opportunities in the wake of new technology development, including examples of adaptable python notebooks for deep learning image analysis, with the ambition to guide the choice of measurement strategy based on various challenges related to different types of nanoparticle samples and associated analytical demands.

Compressive electron backscatter diffraction imaging

Electron backscatter diffraction (EBSD) has developed over the last few decades into a valuable crystallographic characterisation method for a wide range of sample types. Despite these advances, issues such as the complexity of sample preparation, relatively slow acquisition, and damage in beam-sensitive samples, still limit the quantity and quality of interpretable data that can be obtained. To mitigate these issues, here we propose a method based on the subsampling of probe positions and subsequent reconstruction of an incomplete data set. The missing probe locations (or pixels in the image) are recovered via an inpainting process using a dictionary-learning based method called beta-process factor analysis (BPFA). To investigate the robustness of both our inpainting method and Hough-based indexing, we simulate subsampled and noisy EBSD data sets from a real fully sampled Ni-superalloy data set for different subsampling ratios of probe positions using both Gaussian and Poisson noise models. We find that zero solution pixel detection (inpainting un-indexed pixels) enables higher-quality reconstructions to be obtained. Numerical tests confirm high-quality reconstruction of band contrast and inverse pole figure maps from only 10% of the probe positions, with the potential to reduce this to 5% if only inverse pole figure maps are needed. These results show the potential application of this method in EBSD, allowing for faster analysis and extending the use of this technique to beam sensitive materials.

Cellpose3: one-click image restoration for improved cellular segmentation

Generalist methods for cellular segmentation have good out-of-the-box performance on a variety of image types; however, existing methods struggle for images that are degraded by noise, blurring or undersampling, all of which are common in microscopy. We focused the development of Cellpose3 on addressing these cases and here we demonstrate substantial out-of-the-box gains in segmentation and image quality for noisy, blurry and undersampled images. Unlike previous approaches that train models to restore pixel values, we trained Cellpose3 to output images that are well segmented by a generalist segmentation model, while maintaining perceptual similarity to the target images. Furthermore, we trained the restoration models on a large, varied collection of datasets, thus ensuring good generalization to user images. We provide these tools as 'one-click' buttons inside the graphical interface of Cellpose as well as in the Cellpose API.

Deep Equilibrium Unfolding Learning for Noise Estimation and Removal in Optical Molecular Imaging

In clinical optical molecular imaging, the need for real-time high frame rates and low excitation doses to ensure patient safety inherently increases susceptibility to detection noise. Faced with the challenge of image degradation caused by severe noise, image denoising is essential for mitigating the trade-off between acquisition cost and image quality. However, prevailing deep learning methods exhibit uncontrollable and suboptimal performance with limited interpretability, primarily due to neglecting underlying physical model and frequency information. In this work, we introduce an end-to-end model-driven Deep Equilibrium Unfolding Mamba (DEQ-UMamba) that integrates proximal gradient descent technique and learnt spatial-frequency characteristics to decouple complex noise structures into statistical distributions, enabling effective noise estimation and suppression in fluorescent images. Moreover, to address the computational limitations of unfolding networks, DEQ-UMamba trains an implicit mapping by directly differentiating the equilibrium point of the convergent solution, thereby ensuring stability and avoiding non-convergent behavior. With each network module aligned to a corresponding operation in the iterative optimization process, the proposed method achieves clear structural interpretability and strong performance. Comprehensive experiments conducted on both clinical and in vivo datasets demonstrate that DEQ-UMamba outperforms current state-of-the-art alternatives while utilizing fewer parameters, facilitating the advancement of cost-effective and high-quality clinical molecular imaging.

Deep learning enhanced light sheet fluorescence microscopy for in vivo 4D imaging of zebrafish heart beating

Time-resolved volumetric fluorescence imaging over an extended duration with high spatial/temporal resolution is a key driving force in biomedical research for investigating spatial-temporal dynamics at organism-level systems, yet it remains a major challenge due to the trade-off among imaging speed, light exposure, illumination power, and image quality. Here, we present a deep-learning enhanced light sheet fluorescence microscopy (LSFM) approach that addresses the restoration of rapid volumetric time-lapse imaging with less than 0.03% light exposure and 3.3% acquisition time compared to a typical standard acquisition. We demonstrate that the convolutional neural network (CNN)-transformer network developed here, namely U-net integrated transformer (UI-Trans), successfully achieves the mitigation of complex noise-scattering-coupled degradation and outperforms state-of-the-art deep learning networks, due to its capability of faithfully learning fine details while comprehending complex global features. With the fast generation of appropriate training data via flexible switching between confocal line-scanning LSFM (LS-LSFM) and conventional LSFM, this method achieves a three- to five-fold signal-to-noise ratio (SNR) improvement and ~1.8 times contrast improvement in ex vivo zebrafish heart imaging and long-term in vivo 4D (3D morphology + time) imaging of heartbeat dynamics at different developmental stages with ultra-economical acquisitions in terms of light dosage and acquisition time.

Clinical confocal laser endomicroscopy for imaging of autofluorescence signals of human brain tumors and non-tumor brain

PURPOSE

Analysis of autofluorescence holds promise for brain tumor delineation and diagnosis. Therefore, we investigated the potential of a commercial confocal laser scanning endomicroscopy (CLE) system for clinical imaging of brain tumors.

METHODS

A clinical CLE system with fiber probe and 488 nm laser excitation was used to acquire images of tissue autofluorescence. Fresh samples were obtained from routine surgeries (glioblastoma n = 6, meningioma n = 6, brain metastases n = 10, pituitary adenoma n = 2, non-tumor from surgery for the treatment of pharmacoresistant epilepsy n = 2). Additionally, in situ intraoperative label-free CLE was performed in three cases. The autofluorescence images were visually inspected for feature identification and quantification. For reference, tissue cryosections were prepared and further analyzed by label-free multiphoton microscopy and HE histology.

RESULTS

Label-free CLE enabled the acquisition of autofluorescence images for all cases. Autofluorescent structures were assigned to the cytoplasmic compartment of cells, elastin fibers, psammoma bodies and blood vessels by comparison to references. Sparse punctuated autofluorescence was identified in most images across all cases, while dense punctuated autofluorescence was most frequent in glioblastomas. Autofluorescent cells were observed in higher abundancies in images of non-tumor samples. Diffuse autofluorescence, fibers and round fluorescent structures were predominantly found in tumor tissues.

CONCLUSION

Label-free CLE imaging through an approved clinical device was able to visualize the characteristic autofluorescence patterns of human brain tumors and non-tumor brain tissue ex vivo and in situ. Therefore, this approach offers the possibility to obtain intraoperative diagnostic information before resection, importantly independent of any kind of marker or label.

Opportunities and challenges for deep learning in cell dynamics research

The growth of artificial intelligence (AI) has led to an increase in the adoption of computer vision and deep learning (DL) techniques for the evaluation of microscopy images and movies. This adoption has not only addressed hurdles in quantitative analysis of dynamic cell biological processes but has also started to support advances in drug development, precision medicine, and genome-phenome mapping. We survey existing AI-based techniques and tools, as well as open-source datasets, with a specific focus on the computational tasks of segmentation, classification, and tracking of cellular and subcellular structures and dynamics. We summarise long-standing challenges in microscopy video analysis from a computational perspective and review emerging research frontiers and innovative applications for DL-guided automation in cell dynamics research.

Benchmarking robustness of deep neural networks in semantic segmentation of fluorescence microscopy images

BACKGROUND

Fluorescence microscopy (FM) is an important and widely adopted biological imaging technique. Segmentation is often the first step in quantitative analysis of FM images. Deep neural networks (DNNs) have become the state-of-the-art tools for image segmentation. However, their performance on natural images may collapse under certain image corruptions or adversarial attacks. This poses real risks to their deployment in real-world applications. Although the robustness of DNN models in segmenting natural images has been studied extensively, their robustness in segmenting FM images remains poorly understood RESULTS: To address this deficiency, we have developed an assay that benchmarks robustness of DNN segmentation models using datasets of realistic synthetic 2D FM images with precisely controlled corruptions or adversarial attacks. Using this assay, we have benchmarked robustness of ten representative models such as DeepLab and Vision Transformer. We find that models with good robustness on natural images may perform poorly on FM images. We also find new robustness properties of DNN models and new connections between their corruption robustness and adversarial robustness. To further assess the robustness of the selected models, we have also benchmarked them on real microscopy images of different modalities without using simulated degradation. The results are consistent with those obtained on the realistic synthetic images, confirming the fidelity and reliability of our image synthesis method as well as the effectiveness of our assay.

CONCLUSIONS

Based on comprehensive benchmarking experiments, we have found distinct robustness properties of deep neural networks in semantic segmentation of FM images. Based on the findings, we have made specific recommendations on selection and design of robust models for FM image segmentation.

DeepLeish: a deep learning based support system for the detection of Leishmaniasis parasite from Giemsa-stained microscope images

BACKGROUND

Leishmaniasis is a vector-born neglected parasitic disease belonging to the genus Leishmania. Out of the 30 Leishmania species, 21 species cause human infection that affect the skin and the internal organs. Around, 700,000 to 1,000,000 of the newly infected cases and 26,000 to 65,000 deaths are reported worldwide annually. The disease exhibits three clinical presentations, namely, the cutaneous, muco-cutaneous and visceral Leishmaniasis which affects the skin, mucosal membrane and the internal organs, respectively. The relapsing behavior of the disease limits its diagnosis and treatment efficiency. The common diagnostic approaches follow subjective, error-prone, repetitive processes. Despite, an ever pressing need for an accurate detection of Leishmaniasis, the research conducted so far is scarce. In this regard, the main aim of the current research is to develop an artificial intelligence based detection tool for the Leishmaniasis from the Geimsa-stained microscopic images using deep learning method.

METHODS

Stained microscopic images were acquired locally and labeled by experts. The images were augmented using different methods to prevent overfitting and improve the generalizability of the system. Fine-tuned Faster RCNN, SSD, and YOLOV5 models were used for object detection. Mean average precision (MAP), precision, and Recall were calculated to evaluate and compare the performance of the models.

RESULTS

The fine-tuned YOLOV5 outperformed the other models such as Faster RCNN and SSD, with the MAP scores, of 73%, 54% and 57%, respectively.

CONCLUSION

The currently developed YOLOV5 model can be tested in the clinics to assist the laboratorists in diagnosing Leishmaniasis from the microscopic images. Particularly, in low-resourced healthcare facilities, with fewer qualified medical professionals or hematologists, our AI support system can assist in reducing the diagnosing time, workload, and misdiagnosis. Furthermore, the dataset collected by us will be shared with other researchers who seek to improve upon the detection system of the parasite. The current model detects the parasites even in the presence of the monocyte cells, but sometimes, the accuracy decreases due to the differences in the sizes of the parasite cells alongside the blood cells. The incorporation of cascaded networks in future and the quantification of the parasite load, shall overcome the limitations of the currently developed system.

Connecto-informatics at the mesoscale: current advances in image processing and analysis for mapping the brain connectivity

Mapping neural connections within the brain has been a fundamental goal in neuroscience to understand better its functions and changes that follow aging and diseases. Developments in imaging technology, such as microscopy and labeling tools, have allowed researchers to visualize this connectivity through high-resolution brain-wide imaging. With this, image processing and analysis have become more crucial. However, despite the wealth of neural images generated, access to an integrated image processing and analysis pipeline to process these data is challenging due to scattered information on available tools and methods. To map the neural connections, registration to atlases and feature extraction through segmentation and signal detection are necessary. In this review, our goal is to provide an updated overview of recent advances in these image-processing methods, with a particular focus on fluorescent images of the mouse brain. Our goal is to outline a pathway toward an integrated image-processing pipeline tailored for connecto-informatics. An integrated workflow of these image processing will facilitate researchers' approach to mapping brain connectivity to better understand complex brain networks and their underlying brain functions. By highlighting the image-processing tools available for fluroscent imaging of the mouse brain, this review will contribute to a deeper grasp of connecto-informatics, paving the way for better comprehension of brain connectivity and its implications.

Exceeding the limit for microscopic image translation with a deep learning-based unified framework

Deep learning algorithms have been widely used in microscopic image translation. The corresponding data-driven models can be trained by supervised or unsupervised learning depending on the availability of paired data. However, general cases are where the data are only roughly paired such that supervised learning could be invalid due to data unalignment, and unsupervised learning would be less ideal as the roughly paired information is not utilized. In this work, we propose a unified framework (U-Frame) that unifies supervised and unsupervised learning by introducing a tolerance size that can be adjusted automatically according to the degree of data misalignment. Together with the implementation of a global sampling rule, we demonstrate that U-Frame consistently outperforms both supervised and unsupervised learning in all levels of data misalignments (even for perfectly aligned image pairs) in a myriad of image translation applications, including pseudo-optical sectioning, virtual histological staining (with clinical evaluations for cancer diagnosis), improvement of signal-to-noise ratio or resolution, and prediction of fluorescent labels, potentially serving as new standard for image translation.

Dynamic Mode Decomposition of Multiphoton and Stimulated Emission Depletion Microscopy Data for Analysis of Fluorescent Probes in Cellular Membranes

An analysis of the membrane organization and intracellular trafficking of lipids often relies on multiphoton (MP) and super-resolution microscopy of fluorescent lipid probes. A disadvantage of particularly intrinsically fluorescent lipid probes, such as the cholesterol and ergosterol analogue, dehydroergosterol (DHE), is their low MP absorption cross-section, resulting in a low signal-to-noise ratio (SNR) in live-cell imaging. Stimulated emission depletion (STED) microscopy of membrane probes like Nile Red enables one to resolve membrane features beyond the diffraction limit but exposes the sample to a lot of excitation light and suffers from a low SNR and photobleaching. Here, dynamic mode decomposition (DMD) and its variant, higher-order DMD (HoDMD), are applied to efficiently reconstruct and denoise the MP and STED microscopy data of lipid probes, allowing for an improved visualization of the membranes in cells. HoDMD also allows us to decompose and reconstruct two-photon polarimetry images of TopFluor-cholesterol in model and cellular membranes. Finally, DMD is shown to not only reconstruct and denoise 3D-STED image stacks of Nile Red-labeled cells but also to predict unseen image frames, thereby allowing for interpolation images along the optical axis. This important feature of DMD can be used to reduce the number of image acquisitions, thereby minimizing the light exposure of biological samples without compromising image quality. Thus, DMD as a computational tool enables gentler live-cell imaging of fluorescent probes in cellular membranes by MP and STED microscopy.

Facile Conversion and Optimization of Structured Illumination Image Reconstruction Code into the GPU Environment

Superresolution, structured illumination microscopy (SIM) is an ideal modality for imaging live cells due to its relatively high speed and low photon-induced damage to the cells. The rate-limiting step in observing a superresolution image in SIM is often the reconstruction speed of the algorithm used to form a single image from as many as nine raw images. Reconstruction algorithms impose a significant computing burden due to an intricate workflow and a large number of often complex calculations to produce the final image. Further adding to the computing burden is that the code, even within the MATLAB environment, can be inefficiently written by microscopists who are noncomputer science researchers. In addition, they do not take into consideration the processing power of the graphics processing unit (GPU) of the computer. To address these issues, we present simple but efficient approaches to first revise MATLAB code, followed by conversion to GPU-optimized code. When combined with cost-effective, high-performance GPU-enabled computers, a 4- to 500-fold improvement in algorithm execution speed is observed as shown for the image denoising Hessian-SIM algorithm. Importantly, the improved algorithm produces images identical in quality to the original.

Low-frequency background estimation and noise separation from high-frequency for background and noise subtraction

In fluorescence microscopy, background blur and noise are two main factors preventing the achievement of high-signal-to-noise ratio (SNR) imaging. Background blur primarily emanates from inherent factors including the spontaneous fluorescence of biological samples and out-of-focus backgrounds, while noise encompasses Gaussian and Poisson noise components. To achieve background blur subtraction and denoising simultaneously, a pioneering algorithm based on low-frequency background estimation and noise separation from high-frequency (LBNH-BNS) is presented, which effectively disentangles noise from the desired signal. Furthermore, it seamlessly integrates low-frequency features derived from background blur estimation, leading to the effective elimination of noise and background blur in wide-field fluorescence images. In comparisons with other state-of-the-art background removal algorithms, LBNH-BNS demonstrates significant advantages in key quantitative metrics such as peak signal-to-noise ratio (PSNR) and manifests substantial visual enhancements. LBNH-BNS holds immense potential for advancing the overall performance and quality of wide-field fluorescence imaging techniques.

Spatial redundancy transformer for self-supervised fluorescence image denoising

Fluorescence imaging with high signal-to-noise ratios has become the foundation of accurate visualization and analysis of biological phenomena. However, the inevitable noise poses a formidable challenge to imaging sensitivity. Here we provide the spatial redundancy denoising transformer (SRDTrans) to remove noise from fluorescence images in a self-supervised manner. First, a sampling strategy based on spatial redundancy is proposed to extract adjacent orthogonal training pairs, which eliminates the dependence on high imaging speed. Second, we designed a lightweight spatiotemporal transformer architecture to capture long-range dependencies and high-resolution features at low computational cost. SRDTrans can restore high-frequency information without producing oversmoothed structures and distorted fluorescence traces. Finally, we demonstrate the state-of-the-art denoising performance of SRDTrans on single-molecule localization microscopy and two-photon volumetric calcium imaging. SRDTrans does not contain any assumptions about the imaging process and the sample, thus can be easily extended to various imaging modalities and biological applications.

Edge effect of wide spectrum denoising in super-resolution microscopy

During the stochastic optical reconstruction microscope (STORM) raw image acquisition in super-resolution microscopy, noise is inevitable. Noise not only reduces the temporal and spatial resolution of the super-resolution image but also leads to the failure of super-resolution image reconstruction. Wide spectrum denoising (WSD) can effectively remove various random noises (such as Poisson noise and Gaussian noise) from the STORM raw image to improve the super-resolution image reconstruction. We found that there is an obvious edge effect in WSD, and its influence on STORM raw image denoising and super-resolution image reconstruction is studied. We then proposed the method of restraining edge effect. The simulation and real experiment results show that edge trimming can effectively suppress the edge effect, thus leading to better super-resolution image reconstruction.

Optimal sparsity allows reliable system-aware restoration of fluorescence microscopy images

Fluorescence microscopy is one of the most indispensable and informative driving forces for biological research, but the extent of observable biological phenomena is essentially determined by the content and quality of the acquired images. To address the different noise sources that can degrade these images, we introduce an algorithm for multiscale image restoration through optimally sparse representation (MIRO). MIRO is a deterministic framework that models the acquisition process and uses pixelwise noise correction to improve image quality. Our study demonstrates that this approach yields a remarkable restoration of the fluorescence signal for a wide range of microscopy systems, regardless of the detector used (e.g., electron-multiplying charge-coupled device, scientific complementary metal-oxide semiconductor, or photomultiplier tube). MIRO improves current imaging capabilities, enabling fast, low-light optical microscopy, accurate image analysis, and robust machine intelligence when integrated with deep neural networks. This expands the range of biological knowledge that can be obtained from fluorescence microscopy.

Serial intravital 2-photon microscopy and analysis of the kidney using upright microscopes

Serial intravital 2-photon microscopy of the kidney and other abdominal organs is a powerful technique to assess tissue function and structure simultaneously and over time. Thus, serial intravital microscopy can capture dynamic tissue changes during health and disease and holds great potential to characterize (patho-) physiological processes with subcellular resolution. However, successful image acquisition and analysis require significant expertise and impose multiple potential challenges. Abdominal organs are rhythmically displaced by breathing movements which hamper high-resolution imaging. Traditionally, kidney intravital imaging is performed on inverted microscopes where breathing movements are partly compensated by the weight of the animal pressing down. Here, we present a custom and easy-to-implement setup for intravital imaging of the kidney and other abdominal organs on upright microscopes. Furthermore, we provide image processing protocols and a new plugin for the free image analysis software FIJI to process multichannel fluorescence microscopy data. The proposed image processing pipelines cover multiple image denoising algorithms, sample drift correction using 2D registration, and alignment of serial imaging data collected over several weeks using landmark-based 3D registration. The provided tools aim to lower the barrier of entry to intravital microscopy of the kidney and are readily applicable by biomedical practitioners.

Real-time denoising enables high-sensitivity fluorescence time-lapse imaging beyond the shot-noise limit

A fundamental challenge in fluorescence microscopy is the photon shot noise arising from the inevitable stochasticity of photon detection. Noise increases measurement uncertainty and limits imaging resolution, speed and sensitivity. To achieve high-sensitivity fluorescence imaging beyond the shot-noise limit, we present DeepCAD-RT, a self-supervised deep learning method for real-time noise suppression. Based on our previous framework DeepCAD, we reduced the number of network parameters by 94%, memory consumption by 27-fold and processing time by a factor of 20, allowing real-time processing on a two-photon microscope. A high imaging signal-to-noise ratio can be acquired with tenfold fewer photons than in standard imaging approaches. We demonstrate the utility of DeepCAD-RT in a series of photon-limited experiments, including in vivo calcium imaging of mice, zebrafish larva and fruit flies, recording of three-dimensional (3D) migration of neutrophils after acute brain injury and imaging of 3D dynamics of cortical ATP release. DeepCAD-RT will facilitate the morphological and functional interrogation of biological dynamics with a minimal photon budget.

Enhancing Weak-Signal Extraction for Single-Molecule Localization Microscopy

Single-molecule localization microscopy (SMLM) has been widely used in biological imaging due to its ultrahigh spatial resolution. However, due to the strategy of reducing photodamage to living cells, the fluorescence signals of emitters are usually weak and the detector noises become non-negligible, which leads to localization misalignments and signal losses, thus deteriorating the imaging capability of SMLM. Here, we propose an active modulation method to control the fluorescence of the probe emitters. It actually marks the emitters with artificial blinking character, which directly distinguishes weak signals from multiple detector noises. We demonstrated from simulations and experiments that this method improves the signal-to-noise ratio by about 10 dB over the non-modulated method and boosts the sensitivity of single-molecule localization down to -4 dB, which significantly reduces localization misalignments and signal losses in SMLM. This signal-noise decoupling strategy is generally applicable to the super-resolution system with versatile labeled probes to improve their imaging capability. We also showed its application to the densely labeled sample, showing its flexibility in super-resolution nanoscopy.

Aerial Image Denoising Using a Best-So-Far ABC-based Adaptive Filter Method

Nowadays, digital images play an increasingly important role in helping to explain phenomena and to attract people’s attention through various types of media rather than the use of text. However, the quality of digital images may be degraded due to noise that has occurred either during their recording or their transmission via a network. Therefore, removal of image noise, which is known as “image denoising”, is one of the primary required tasks in digital image processing. Various methods in earlier studies have been developed and proposed to remove the noise found in images. For example, the use of metric filters to eliminate noise has received much attention from researchers in recent literature. However, the convergence speed when searching for the optimal filter coefficient of these proposed algorithms is quite low. Previous research in the past few years has found that biologically inspired approaches are among the more promising metaheuristic methods used to find optimal solutions. In this work, an image denoising approach based on the best-so-far (BSF) ABC algorithm combined with an adaptive filter is proposed to enhance the performance of searching for the optimal filter coefficient in the denoising process. Experimental results indicate that the denoising of images employing the proposed BSF ABC technique yields good quality and the ability to remove noise while preventing the features of the image from being lost in the denoising process. The denoised image quality obtained by the proposed method achieves a 20% increase compared with other recently developed techniques in the field of biologically inspired approaches.

Deep learning-based noise filtering toward millisecond order imaging by using scanning transmission electron microscopy

Application of scanning transmission electron microscopy (STEM) to in situ observation will be essential in the current and emerging data-driven materials science by taking STEM's high affinity with various analytical options into account. As is well known, STEM's image acquisition time needs to be further shortened to capture a targeted phenomenon in real-time as STEM's current temporal resolution is far below the conventional TEM's. However, rapid image acquisition in the millisecond per frame or faster generally causes image distortion, poor electron signals, and unidirectional blurring, which are obstacles for realizing video-rate STEM observation. Here we show an image correction framework integrating deep learning (DL)-based denoising and image distortion correction schemes optimized for STEM rapid image acquisition. By comparing a series of distortion corrected rapid scan images with corresponding regular scan speed images, the trained DL network is shown to remove not only the statistical noise but also the unidirectional blurring. This result demonstrates that rapid as well as high-quality image acquisition by STEM without hardware modification can be established by the DL. The DL-based noise filter could be applied to in-situ observation, such as dislocation activities under external stimuli, with high spatio-temporal resolution.

Computational Methods for Single-Cell Imaging and Omics Data Integration

Integrating single cell omics and single cell imaging allows for a more effective characterisation of the underlying mechanisms that drive a phenotype at the tissue level, creating a comprehensive profile at the cellular level. Although the use of imaging data is well established in biomedical research, its primary application has been to observe phenotypes at the tissue or organ level, often using medical imaging techniques such as MRI, CT, and PET. These imaging technologies complement omics-based data in biomedical research because they are helpful for identifying associations between genotype and phenotype, along with functional changes occurring at the tissue level. Single cell imaging can act as an intermediary between these levels. Meanwhile new technologies continue to arrive that can be used to interrogate the genome of single cells and its related omics datasets. As these two areas, single cell imaging and single cell omics, each advance independently with the development of novel techniques, the opportunity to integrate these data types becomes more and more attractive. This review outlines some of the technologies and methods currently available for generating, processing, and analysing single-cell omics- and imaging data, and how they could be integrated to further our understanding of complex biological phenomena like ageing. We include an emphasis on machine learning algorithms because of their ability to identify complex patterns in large multidimensional data.

Imaging in focus: An introduction to denoising bioimages in the era of deep learning

Fluorescence microscopy enables the direct observation of previously hidden dynamic processes of life, allowing profound insights into mechanisms of health and disease. However, imaging of live samples is fundamentally limited by the toxicity of the illuminating light and images are often acquired using low light conditions. As a consequence, images can become very noisy which severely complicates their interpretation. In recent years, deep learning (DL) has emerged as a very successful approach to remove this noise while retaining the useful signal. Unlike classical algorithms which use well-defined mathematical functions to remove noise, DL methods learn to denoise from example data, providing a powerful content-aware approach. In this review, we first describe the different types of noise that typically corrupt fluorescence microscopy images and introduce the denoising task. We then present the main DL-based denoising methods and their relative advantages and disadvantages. We aim to provide insights into how DL-based denoising methods operate and help users choose the most appropriate tools for their applications.

Live Imaging of the Lung

Live imaging is critical to determining the dynamics and spatial interactions of cells within the tissue environment. In the lung, this has proven to be difficult due to the motion brought about by ventilation and cardiac contractions. A previous version of this Current Protocols in Cytometry article reported protocols for imaging ex vivo live lung slices and the intact mouse lung. Here, we update those protocols by adding new methodologies, new approaches for quantitative image analysis, and new areas of potential application. © 2020 Wiley Periodicals LLC. Basic Protocol 1: Live imaging of lung slices Support Protocol 1: Staining lung sections with fluorescent antibodies Basic Protocol 2: Live imaging in the mouse lung Support Protocol 2: Intratracheal instillations Support Protocol 3: Intravascular instillations Support Protocol 4: Monitoring vital signs of the mouse during live lung imaging Support Protocol 5: Antibodies Support Protocol 6: Fluorescent reporter mice Basic Protocol 3: Quantification of neutrophil-platelet aggregation in pulmonary vasculature Basic Protocol 4: Quantification of platelet-dependent pulmonary thrombosis Basic Protocol 5: Quantification of pulmonary vascular permeability.

Blind Deconvolution Based on Compressed Sensing with bi-l0-l2-norm Regularization in Light Microscopy Image

Blind deconvolution of light microscopy images could improve the ability of distinguishing cell-level substances. In this study, we investigated the blind deconvolution framework for a light microscope image, which combines the benefits of bi-l₀-l₂-norm regularization with compressed sensing and conjugated gradient algorithms. Several existing regularization approaches were limited by staircase artifacts (or cartooned artifacts) and noise amplification. Thus, we implemented our strategy to overcome these problems using the bi-l₀-l₂-norm regularization proposed. It was investigated through simulations and experiments using optical microscopy images including the background noise. The sharpness was improved through the successful image restoration while minimizing the noise amplification. In addition, quantitative factors of the restored images, including the intensity profile, root-mean-square error (RMSE), edge preservation index (EPI), structural similarity index measure (SSIM), and normalized noise power spectrum, were improved compared to those of existing or comparative images. In particular, the results of using the proposed method showed RMSE, EPI, and SSIM values of approximately 0.12, 0.81, and 0.88 when compared with the reference. In addition, RMSE, EPI, and SSIM values in the restored image were proven to be improved by about 5.97, 1.26, and 1.61 times compared with the degraded image. Consequently, the proposed method is expected to be effective for image restoration and to reduce the cost of a high-performance light microscope.

A Low-Cost Automated Digital Microscopy Platform for Automatic Identification of Diatoms

Currently, microalgae (i.e., diatoms) constitute a generally accepted bioindicator of water quality and therefore provide an index of the status of biological ecosystems. Diatom detection for specimen counting and sample classification are two difficult time-consuming tasks for the few existing expert diatomists. To mitigate this challenge, in this work, we propose a fully operative low-cost automated microscope, integrating algorithms for: (1) stage and focus control, (2) image acquisition (slide scanning, stitching, contrast enhancement), and (3) diatom detection and a prospective specimen classification (among 80 taxa). Deep learning algorithms have been applied to overcome the difficult selection of image descriptors imposed by classical machine learning strategies. With respect to the mentioned strategies, the best results were obtained by deep neural networks with a maximum precision of 86% (with the YOLO network) for detection and 99.51% for classification, among 80 different species (with the AlexNet network). All the developed operational modules are integrated and controlled by the user from the developed graphical user interface running in the main controller. With the developed operative platform, it is noteworthy that this work provides a quite useful toolbox for phycologists in their daily challenging tasks to identify and classify diatoms.

A bird's-eye view of deep learning in bioimage analysis

Deep learning of artificial neural networks has become the de facto standard approach to solving data analysis problems in virtually all fields of science and engineering. Also in biology and medicine, deep learning technologies are fundamentally transforming how we acquire, process, analyze, and interpret data, with potentially far-reaching consequences for healthcare. In this mini-review, we take a bird's-eye view at the past, present, and future developments of deep learning, starting from science at large, to biomedical imaging, and bioimage analysis in particular.

HISTOBREAST, a collection of brightfield microscopy images of Haematoxylin and Eosin stained breast tissue

Modern histopathology workflows rely on the digitization of histology slides. The quality of the resulting digital representations, in the form of histology slide image mosaics, depends on various specific acquisition conditions and on the image processing steps that underlie the generation of the final mosaic, e.g. registration and blending of the contained image tiles. We introduce HISTOBREAST, an extensive collection of brightfield microscopy images that we collected in a principled manner under different acquisition conditions on Haematoxylin - Eosin (H&E) stained breast tissue. HISTOBREAST is comprised of neighbour image tiles and ensemble of mosaics composed from different combinations of the available image tiles, exhibiting progressively degraded quality levels. HISTOBREAST can be used to benchmark image processing and computer vision techniques with respect to their robustness to image modifications specific to brightfield microscopy of H&E stained tissues. Furthermore, HISTOBREAST can serve in the development of new image processing methods, with the purpose of ensuring robustness to typical image artefacts that raise interpretation problems for expert histopathologists and affect the results of computerized image analysis.

Mu-net: Multi-scale U-net for two-photon microscopy image denoising and restoration

Advances in two two-photon microscopy (2PM) have made three-dimensional (3D) neural imaging of deep cortical regions possible. However, 2PM often suffers from poor image quality because of various noise factors, including blur, white noise, and photo bleaching. In addition, the effectiveness of the existing image processing methods is limited because of the special features of 2PM images such as deeper tissue penetration but higher image noises owing to rapid laser scanning. To address the denoising problems in 2PM 3D images, we present a new algorithm based on deep convolutional neural networks (CNNs). The proposed model consists of multiple U-nets in which an individual U-net removes noises at different scales and then yields a performance improvement based on a coarse-to-fine strategy. Moreover, the constituent CNNs employ fully 3D convolution operations. Such an architecture enables the proposed model to facilitate end-to-end learning without any pre/post processing. Based on the experiments on 2PM image denoising, we observed that our new algorithm demonstrates substantial performance improvements over other baseline methods.

Fast and accurate sCMOS noise correction for fluorescence microscopy

The rapid development of scientific CMOS (sCMOS) technology has greatly advanced optical microscopy for biomedical research with superior sensitivity, resolution, field-of-view, and frame rates. However, for sCMOS sensors, the parallel charge-voltage conversion and different responsivity at each pixel induces extra readout and pattern noise compared to charge-coupled devices (CCD) and electron-multiplying CCD (EM-CCD) sensors. This can produce artifacts, deteriorate imaging capability, and hinder quantification of fluorescent signals, thereby compromising strategies to reduce photo-damage to live samples. Here, we propose a content-adaptive algorithm for the automatic correction of sCMOS-related noise (ACsN) for fluorescence microscopy. ACsN combines camera physics and layered sparse filtering to significantly reduce the most relevant noise sources in a sCMOS sensor while preserving the fine details of the signal. The method improves the camera performance, enabling fast, low-light and quantitative optical microscopy with video-rate denoising for a broad range of imaging conditions and modalities.

Scientific complementary metal-oxide semiconductor (sCMOS) cameras have advanced the imaging field, but they often suffer from additional noise compared to CCD sensors. Here the authors present a content-adaptive algorithm for the automatic correction of sCMOS-related noise for fluorescence microscopy.

Removal of computed tomography ring artifacts via radial basis function artificial neural networks

Ring artifacts in computed tomography (CT) images are caused by the undesirable response of detector pixels, which leads to the degradation of CT images. Accordingly, it affects the image interpretation, post-processing, and quantitative analysis. In this study, a radial basis function neural network (RBFNN) was used to remove ring artifacts. The proposed method employs polar coordinate transformation. First, ring artifacts were transformed into linear artifacts by polar coordinate transformation. Then, smoothing operators were applied to locate these artifacts exactly. Subsequently, RBFNN was operated on each linear artifact. The neuron numbers of the input, hidden, and output layers of the neural network were 8, 40, and 1, respectively. Neurons in the input layer were selected according to the characteristics of the artifact itself and its relationship with the surrounding normal pixels. For the training of the neural network, a hybrid of adaptive gradient descent algorithm (AGDA) and gravitational search algorithm (GSA) was adopted. After the corrected image was obtained using the updated neural network, the inverse coordinate transformation was implemented. The experimental data were divided into simulated ring artifacts and real ring artifacts, which were based on brain and abdomen CT images. Compared with current artifact removal methods, the proposed method removed ring artifacts more effectively and retained the maximum detail of normal tissues. In addition, for index analysis, the performance of proposed method was superior to that of the other methods.

DeepLSR: a deep learning approach for laser speckle reduction

Speckle artifacts degrade image quality in virtually all modalities that utilize coherent energy, including optical coherence tomography, reflectance confocal microscopy, ultrasound, and widefield imaging with laser illumination. We present an adversarial deep learning framework for laser speckle reduction, called DeepLSR (https://durr.jhu.edu/DeepLSR), that transforms images from a source domain of coherent illumination to a target domain of speckle-free, incoherent illumination. We apply this method to widefield images of objects and tissues illuminated with a multi-wavelength laser, using light emitting diode-illuminated images as ground truth. In images of gastrointestinal tissues, DeepLSR reduces laser speckle noise by 6.4 dB, compared to a 2.9 dB reduction from optimized non-local means processing, a 3.0 dB reduction from BM3D, and a 3.7 dB reduction from an optical speckle reducer utilizing an oscillating diffuser. Further, DeepLSR can be combined with optical speckle reduction to reduce speckle noise by 9.4 dB. This dramatic reduction in speckle noise may enable the use of coherent light sources in applications that require small illumination sources and high-quality imaging, including medical endoscopy.