Noise2Atom: unsupervised denoising for scanning transmission electron microscopy images

Perspective and prospects of in situ transmission/scanning transmission electron microscopy

In situ transmission/scanning transmission electron microscopy (TEM/STEM) measurements have taken a central stage for establishing structure-chemistry-property relationship over the past couple of decades. The challenges for realizing 'a lab-in-gap', i.e. gap between the objective lens pole pieces, or 'a lab-on-chip', to be used to carry out experiments are being met through continuous instrumental developments. Commercially available TEM columns and sample holder, that have been modified for in situ experimentation, have contributed to uncover structural and chemical changes occurring in the sample when subjected to external stimulus such as temperature, pressure, radiation (photon, ions and electrons), environment (gas, liquid and magnetic or electrical field) or a combination thereof. Whereas atomic resolution images and spectroscopy data are being collected routinely using TEM/STEM, temporal resolution is limited to millisecond. On the other hand, better than femtosecond temporal resolution can be achieved using an ultrafast electron microscopy or dynamic TEM, but the spatial resolution is limited to sub-nanometers. In either case, in situ experiments generate large datasets that need to be transferred, stored and analyzed. The advent of artificial intelligence, especially machine learning platforms, is proving crucial to deal with this big data problem. Further developments are still needed in order to fully exploit our capability to understand, measure and control chemical and/or physical processes. We present the current state of instrumental and computational capabilities and discuss future possibilities.

Quantifying the thickness of WTe2 using atomic-resolution STEM simulations and supervised machine learning

For two-dimensional (2D) materials, the exact thickness of the material often dictates its physical and chemical properties. The 2D quantum material WTe2 possesses properties that vary significantly from a single layer to multiple layers, yet it has a complicated crystal structure that makes it difficult to differentiate thicknesses in atomic-resolution images. Furthermore, its air sensitivity and susceptibility to electron beam-induced damage heighten the need for direct ways to determine the thickness and atomic structure without acquiring multiple measurements or transferring samples in ambient atmosphere. Here, we demonstrate a new method to identify the thickness up to ten van der Waals layers in Td-WTe2 using atomic-resolution high-angle annular dark-field scanning transmission electron microscopy image simulation. Our approach is based on analyzing the intensity line profiles of overlapping atomic columns and building a standard neural network model from the line profile features. We observe that it is possible to clearly distinguish between even and odd thicknesses (up to seven layers), without using machine learning, by comparing the deconvoluted peak intensity ratios or the area ratios. The standard neural network model trained on the line profile features allows thicknesses to be distinguished up to ten layers and exhibits an accuracy of up to 94% in the presence of Gaussian and Poisson noise. This method efficiently quantifies thicknesses in Td-WTe2, can be extended to related 2D materials, and provides a pathway to characterize precise atomic structures, including local thickness variations and atomic defects, for few-layer 2D materials with overlapping atomic column positions.

Quantitative Description of Metal Center Organization and Interactions in Single-Atom Catalysts

Ultra-high-density single-atom catalysts (UHD-SACs) present unique opportunities for harnessing cooperative effects between neighboring metal centers. However, the lack of tools to establish correlations between the density, types, and arrangements of isolated metal atoms and the support surface properties hinders efforts to engineer advanced material architectures. Here, this work precisely describes the metal center organization in various mono- and multimetallic UHD-SACs based on nitrogen-doped carbon (NC) supports by coupling transmission electron microscopy with tailored machine-learning methods (released as a user-friendly web app) and density functional theory simulations. This approach quantifies the non-negligible presence of multimers with increasing atom density, characterizes the size and shape of these low-nuclearity clusters, and identifies surface atom density criteria to ensure isolation. Further, it provides previously inaccessible experimental insights into coordination site arrangements in the NC host, uncovering a repulsive interaction that influences the disordered distribution of metal centers in UHD-SACs. This observation holds in multimetallic systems, where chemically-specific analysis quantifies the degree of intermixing. These fundamental insights into the materials chemistry of single-atom catalysts are crucial for designing catalytic systems with superior reactivity.

Chirality Analysis of Complex Microparticles using Deep Learning on Realistic Sets of Microscopy Images

Nanoscale chirality is an actively growing research field spurred by the giant chiroptical activity, enantioselective biological activity, and asymmetric catalytic activity of chiral nanostructures. Compared to chiral molecules, the handedness of chiral nano- and microstructures can be directly established via electron microscopy, which can be utilized for the automatic analysis of chiral nanostructures and prediction of their properties. However, chirality in complex materials may have multiple geometric forms and scales. Computational identification of chirality from electron microscopy images rather than optical measurements is convenient but is fundamentally challenging, too, because (1) image features differentiating left- and right-handed particles can be ambiguous and (2) three-dimensional structure essential for chirality is 'flattened' into two-dimensional projections. Here, we show that deep learning algorithms can identify twisted bowtie-shaped microparticles with nearly 100% accuracy and classify them as left- and right-handed with as high as 99% accuracy. Importantly, such accuracy was achieved with as few as 30 original electron microscopy images of bowties. Furthermore, after training on bowtie particles with complex nanostructured features, the model can recognize other chiral shapes with different geometries without retraining for their specific chiral geometry with 93% accuracy, indicating the true learning abilities of the employed neural networks. These findings indicate that our algorithm trained on a practically feasible set of experimental data enables automated analysis of microscopy data for the accelerated discovery of chiral particles and their complex systems for multiple applications.

Self-supervised machine learning pushes the sensitivity limit in label-free detection of single proteins below 10 kDa

Interferometric scattering (iSCAT) microscopy is a label-free optical method capable of detecting single proteins, localizing their binding positions with nanometer precision, and measuring their mass. In the ideal case, iSCAT is limited by shot noise such that collection of more photons should extend its detection sensitivity to biomolecules of arbitrarily low mass. However, a number of technical noise sources combined with speckle-like background fluctuations have restricted the detection limit in iSCAT. Here, we show that an unsupervised machine learning isolation forest algorithm for anomaly detection pushes the mass sensitivity limit by a factor of 4 to below 10 kDa. We implement this scheme both with a user-defined feature matrix and a self-supervised FastDVDNet and validate our results with correlative fluorescence images recorded in total internal reflection mode. Our work opens the door to optical investigations of small traces of biomolecules and disease markers such as α-synuclein, chemokines and cytokines.

Machine learning in electron microscopy for advanced nanocharacterization: current developments, available tools and future outlook

In the last few years, electron microscopy has experienced a new methodological paradigm aimed to fix the bottlenecks and overcome the challenges of its analytical workflow. Machine learning and artificial intelligence are answering this call providing powerful resources towards automation, exploration, and development. In this review, we evaluate the state-of-the-art of machine learning applied to electron microscopy (and obliquely, to materials and nano-sciences). We start from the traditional imaging techniques to reach the newest higher-dimensionality ones, also covering the recent advances in spectroscopy and tomography. Additionally, the present review provides a practical guide for microscopists, and in general for material scientists, but not necessarily advanced machine learning practitioners, to straightforwardly apply the offered set of tools to their own research. To conclude, we explore the state-of-the-art of other disciplines with a broader experience in applying artificial intelligence methods to their research (e.g., high-energy physics, astronomy, Earth sciences, and even robotics, videogames, or marketing and finances), in order to narrow down the incoming future of electron microscopy, its challenges and outlook.

SIM-STEM Lab: Incorporating Compressed Sensing Theory for Fast STEM Simulation

Recently it has been shown that precise dose control and an increase in the overall acquisition speed of atomic resolution scanning transmission electron microscope (STEM) images can be achieved by acquiring only a small fraction of the pixels in the image experimentally and then reconstructing the full image using an inpainting algorithm. In this paper, we apply the same inpainting approach (a form of compressed sensing) to simulated, sub-sampled atomic resolution STEM images. We find that it is possible to significantly sub-sample the area that is simulated, the number of g-vectors contributing the image, and the number of frozen phonon configurations contributing to the final image while still producing an acceptable fit to a fully sampled simulation. Here we discuss the parameters that we use and how the resulting simulations can be quantifiably compared to the full simulations. As with any Compressed Sensing methodology, care must be taken to ensure that isolated events are not excluded from the process, but the observed increase in simulation speed provides significant opportunities for real time simulations, image classification and analytics to be performed as a supplement to experiments on a microscope to be developed in the future.

Deep-learning microscopy image reconstruction with quality control reveals second-scale rearrangements in RNA polymerase II clusters

Fluorescence microscopy, a central tool of biological research, is subject to inherent trade-offs in experiment design. For instance, image acquisition speed can only be increased in exchange for a lowered signal quality, or for an increased rate of photo-damage to the specimen. Computational denoising can recover some loss of signal, extending the trade-off margin for high-speed imaging. Recently proposed denoising on the basis of neural networks shows exceptional performance but raises concerns of errors typical of neural networks. Here, we present a work-flow that supports an empirically optimized reduction of exposure times, as well as per-image quality control to exclude images with reconstruction errors. We implement this work-flow on the basis of the denoising tool Noise2Void and assess the molecular state and 3D shape of RNA polymerase II (Pol II) clusters in live zebrafish embryos. Image acquisition speed could be tripled, achieving 2-s time resolution and 350-nm lateral image resolution. The obtained data reveal stereotyped events of approximately 10 s duration: initially, the molecular mark for recruited Pol II increases, then the mark for active Pol II increases, and finally Pol II clusters take on a stretched and unfolded shape. An independent analysis based on fixed sample images reproduces this sequence of events, and suggests that they are related to the transient association of genes with Pol II clusters. Our work-flow consists of procedures that can be implemented on commercial fluorescence microscopes without any hardware or software modification, and should, therefore, be transferable to many other applications.

Automated Image Analysis for Single-Atom Detection in Catalytic Materials by Transmission Electron Microscopy

Single-atom catalytic sites may have existed in all supported transition metal catalysts since their first application. Yet, interest in the design of single-atom heterogeneous catalysts (SACs) only really grew when advances in transmission electron microscopy (TEM) permitted direct confirmation of metal site isolation. While atomic-resolution imaging remains a central characterization tool, poor statistical significance, reproducibility, and interoperability limit its scope for deriving robust characteristics about these frontier catalytic materials. Here, we introduce a customized deep-learning method for automated atom detection in image analysis, a rate-limiting step toward high-throughput TEM. Platinum atoms stabilized on a functionalized carbon support with a challenging irregular three-dimensional morphology serve as a practically relevant test system with promising scope in thermo- and electrochemical applications. The model detects over 20,000 atomic positions for the statistical analysis of important properties for establishing structure-performance relations over nanostructured catalysts, like the surface density, proximity, clustering extent, and dispersion uniformity of supported metal species. Good performance obtained on direct application of the model to an iron SAC based on carbon nitride demonstrates its generalizability for single-atom detection on carbon-related materials. The approach establishes a route to integrate artificial intelligence into routine TEM workflows. It accelerates image processing times by orders of magnitude and reduces human bias by providing an uncertainty analysis that is not readily quantifiable in manual atom identification, improving standardization and scalability.

Applications of deep learning in electron microscopy

We review the growing use of machine learning in electron microscopy (EM) driven in part by the availability of fast detectors operating at kiloHertz frame rates leading to large data sets that cannot be processed using manually implemented algorithms. We summarize the various network architectures and error metrics that have been applied to a range of EM-related problems including denoising and inpainting. We then provide a review of the application of these in both physical and life sciences, highlighting how conventional networks and training data have been specifically modified for EM.

Deep learning 2D and 3D optical sectioning microscopy using cross-modality Pix2Pix cGAN image translation

Structured illumination microscopy (SIM) reconstructs optically-sectioned images of a sample from multiple spatially-patterned wide-field images, but the traditional single non-patterned wide-field images are more inexpensively obtained since they do not require generation of specialized illumination patterns. In this work, we translated wide-field fluorescence microscopy images to optically-sectioned SIM images by a Pix2Pix conditional generative adversarial network (cGAN). Our model shows the capability of both 2D cross-modality image translation from wide-field images to optical sections, and further demonstrates potential to recover 3D optically-sectioned volumes from wide-field image stacks. The utility of the model was tested on a variety of samples including fluorescent beads and fresh human tissue samples.

A new unsupervised pseudo-siamese network with two filling strategies for image denoising and quality enhancement

Digital image noise may be introduced during acquisition, transmission, or processing and affects readability and image processing effectiveness. The accuracy of established image processing techniques, such as segmentation, recognition, and edge detection, is adversely impacted by noise. There exists an extensive body of work which focuses on circumventing such issues through digital image enhancement and noise reduction, but this work is limited by a number of constraints including the application of non-adaptive parameters, potential loss of edge detail information, and (with supervised approaches) a requirement for clean, labeled, training data. This paper, developed on the principle of Noise2Void, presents a new unsupervised learning approach incorporating a pseudo-siamese network. Our method enables image denoising without the need for clean images or paired noise images, instead requiring only noise images. Two independent branches of the network utilize different filling strategies, namely zero filling and adjacent pixel filling. Then, the network employs a loss function to improve the similarity of the results in the two branches. We also modify the Efficient Channel Attention module to extract more diverse features and improve performance on the basis of global average pooling. Experimental results show that compared with traditional methods, the pseudo-siamese network has a greater improvement on the ADNI dataset in terms of quantitative and qualitative evaluation. Our method therefore has practical utility in cases where clean images are difficult to obtain.

Five-second STEM dislocation tomography for 300 nm thick specimen assisted by deep-learning-based noise filtering

Scanning transmission electron microscopy (STEM) is suitable for visualizing the inside of a relatively thick specimen than the conventional transmission electron microscopy, whose resolution is limited by the chromatic aberration of image forming lenses, and thus, the STEM mode has been employed frequently for computed electron tomography based three-dimensional (3D) structural characterization and combined with analytical methods such as annular dark field imaging or spectroscopies. However, the image quality of STEM is severely suffered by noise or artifacts especially when rapid imaging, in the order of millisecond per frame or faster, is pursued. Here we demonstrate a deep-learning-assisted rapid STEM tomography, which visualizes 3D dislocation arrangement only within five-second acquisition of all the tilt-series images even in a 300 nm thick steel specimen. The developed method offers a new platform for various in situ or operando 3D microanalyses in which dealing with relatively thick specimens or covering media like liquid cells are required.

RapidEELS: machine learning for denoising and classification in rapid acquisition electron energy loss spectroscopy

Recent advances in detectors for imaging and spectroscopy have afforded in situ, rapid acquisition of hyperspectral data. While electron energy loss spectroscopy (EELS) data acquisition speeds with electron counting are regularly reaching 400 frames per second with near-zero read noise, signal to noise ratio (SNR) remains a challenge owing to fundamental counting statistics. In order to advance understanding of transient materials phenomena during rapid acquisition EELS, trustworthy analysis of noisy spectra must be demonstrated. In this study, we applied machine learning techniques to denoise high frame rate spectra, benchmarking with slower frame rate "ground truths". The results provide a foundation for reliable use of low SNR data acquired in rapid, in-situ spectroscopy experiments. Such a tool-set is a first step toward both automation in microscopy as well as use of these methods to interrogate otherwise poorly understood transformations.

Defect Detection in Atomic Resolution Transmission Electron Microscopy Images Using Machine Learning

Point defects play a fundamental role in the discovery of new materials due to their strong influence on material properties and behavior. At present, imaging techniques based on transmission electron microscopy (TEM) are widely employed for characterizing point defects in materials. However, current methods for defect detection predominantly involve visual inspection of TEM images, which is laborious and poses difficulties in materials where defect related contrast is weak or ambiguous. Recent efforts to develop machine learning methods for the detection of point defects in TEM images have focused on supervised methods that require labeled training data that is generated via simulation. Motivated by a desire for machine learning methods that can be trained on experimental data, we propose two self-supervised machine learning algorithms that are trained solely on images that are defect-free. Our proposed methods use principal components analysis (PCA) and convolutional neural networks (CNN) to analyze a TEM image and predict the location of a defect. Using simulated TEM images, we show that PCA can be used to accurately locate point defects in the case where there is no imaging noise. In the case where there is imaging noise, we show that incorporating a CNN dramatically improves model performance. Our models rely on a novel approach that uses the residual between a TEM image and its PCA reconstruction.