A data reduction and compression description for high throughput time-resolved electron microscopy

Perspective and prospects of in situ transmission/scanning transmission electron microscopy

In situ transmission/scanning transmission electron microscopy (TEM/STEM) measurements have taken a central stage for establishing structure-chemistry-property relationship over the past couple of decades. The challenges for realizing 'a lab-in-gap', i.e. gap between the objective lens pole pieces, or 'a lab-on-chip', to be used to carry out experiments are being met through continuous instrumental developments. Commercially available TEM columns and sample holder, that have been modified for in situ experimentation, have contributed to uncover structural and chemical changes occurring in the sample when subjected to external stimulus such as temperature, pressure, radiation (photon, ions and electrons), environment (gas, liquid and magnetic or electrical field) or a combination thereof. Whereas atomic resolution images and spectroscopy data are being collected routinely using TEM/STEM, temporal resolution is limited to millisecond. On the other hand, better than femtosecond temporal resolution can be achieved using an ultrafast electron microscopy or dynamic TEM, but the spatial resolution is limited to sub-nanometers. In either case, in situ experiments generate large datasets that need to be transferred, stored and analyzed. The advent of artificial intelligence, especially machine learning platforms, is proving crucial to deal with this big data problem. Further developments are still needed in order to fully exploit our capability to understand, measure and control chemical and/or physical processes. We present the current state of instrumental and computational capabilities and discuss future possibilities.

Deep Learning Approach for High-accuracy Electron Counting of Monolithic Active Pixel Sensor-type Direct Electron Detectors at Increased Electron Dose

Electron counting can be performed algorithmically for monolithic active pixel sensor direct electron detectors to eliminate readout noise and Landau noise arising from the variability in the amount of deposited energy for each electron. Errors in existing counting algorithms include mistakenly counting a multielectron strike as a single electron event, and inaccurately locating the incident position of the electron due to lateral spread of deposited energy and dark noise. Here, we report a supervised deep learning (DL) approach based on Faster region-based convolutional neural network (R-CNN) to recognize single electron events at varying electron doses and voltages. The DL approach shows high accuracy according to the near-ideal modulation transfer function (MTF) and detector quantum efficiency for sparse images. It predicts, on average, 0.47 pixel deviation from the incident positions for 200 kV electrons versus 0.59 pixel using the conventional counting method. The DL approach also shows better robustness against coincidence loss as the electron dose increases, maintaining the MTF at half Nyquist frequency above 0.83 as the electron density increases to 0.06 e-/pixel. Thus, the DL model extends the advantages of counting analysis to higher dose rates than conventional methods.

Operando Electron Microscopy of Catalysts: The Missing Cornerstone in Heterogeneous Catalysis Research?

Heterogeneous catalysis in thermal gas-phase and electrochemical liquid-phase chemical conversion plays an important role in our modern energy landscape. However, many of the structural features that drive efficient chemical energy conversion are still unknown. These features are, in general, highly distinct on the local scale and lack translational symmetry, and thus, they are difficult to capture without the required spatial and temporal resolution. Correlating these structures to their function will, conversely, allow us to disentangle irrelevant and relevant features, explore the entanglement of different local structures, and provide us with the necessary understanding to tailor novel catalyst systems with improved productivity. This critical review provides a summary of the still immature field of operando electron microscopy for thermal gas-phase and electrochemical liquid-phase reactions. It focuses on the complexity of investigating catalytic reactions and catalysts, progress in the field, and analysis. The forthcoming advances are discussed in view of correlative techniques, artificial intelligence in analysis, and novel reactor designs.

Time-resolved transmission electron microscopy for nanoscale chemical dynamics

The ability of transmission electron microscopy (TEM) to image a structure ranging from millimetres to Ångströms has made it an indispensable component of the toolkit of modern chemists. TEM has enabled unprecedented understanding of the atomic structures of materials and how structure relates to properties and functions. Recent developments in TEM have advanced the technique beyond static material characterization to probing structural evolution on the nanoscale in real time. Accompanying advances in data collection have pushed the temporal resolution into the microsecond regime with the use of direct-electron detectors and down to the femtosecond regime with pump-probe microscopy. Consequently, studies have deftly applied TEM for understanding nanoscale dynamics, often in operando. In this Review, time-resolved in situ TEM techniques and their applications for probing chemical and physical processes are discussed, along with emerging directions in the TEM field.

A Comparison of Lossless Compression Methods in Microscopy Data Storage Applications

Modern high-throughput microscopy methods such as light-sheet imaging and electron microscopy are capable of producing petabytes of data inside of a single experiment. Storage of these large images, however, is challenging because of the difficulty of moving, storing, and analyzing such vast amounts of data, which is often collected at very high data rates (>1GBps). In this report, we provide a comparison of the performance of several compression algorithms using a collection of published and unpublished datasets including confocal, fMOST, and pathology images. We also use simulated data to demonstrate the efficiency of each algorithm as image content or entropy increases. As a result of this work, we recommend the use of the BLOSC algorithm combined with ZSTD for various microscopy applications, as it produces the best compression ratio over a collection of conditions.

Event driven 4D STEM acquisition with a Timepix3 detector: Microsecond dwell time and faster scans for high precision and low dose applications

Four dimensional scanning transmission electron microscopy (4D STEM) records the scattering of electrons in a material in great detail. The benefits offered by 4D STEM are substantial, with the wealth of data it provides facilitating for instance high precision, high electron dose efficiency phase imaging via centre of mass or ptychography based analysis. However the requirement for a 2D image of the scattering to be recorded at each probe position has long placed a severe bottleneck on the speed at which 4D STEM can be performed. Recent advances in camera technology have greatly reduced this bottleneck, with the detection efficiency of direct electron detectors being especially well suited to the technique. However even the fastest frame driven pixelated detectors still significantly limit the scan speed which can be used in 4D STEM, making the resulting data susceptible to drift and hampering its use for low dose beam sensitive applications. Here we report the development of the use of an event driven Timepix3 direct electron camera that allows us to overcome this bottleneck and achieve 4D STEM dwell times down to 100 ns; orders of magnitude faster than what has been possible with frame based readout. We characterize the detector for different acceleration voltages and show that the method is especially well suited for low dose imaging and promises rich datasets without compromising dwell time when compared to conventional STEM imaging.