Deep-Learning Electron Diffractive Imaging

On the use of deep learning for phase recovery

Phase recovery (PR) refers to calculating the phase of the light field from its intensity measurements. As exemplified from quantitative phase imaging and coherent diffraction imaging to adaptive optics, PR is essential for reconstructing the refractive index distribution or topography of an object and correcting the aberration of an imaging system. In recent years, deep learning (DL), often implemented through deep neural networks, has provided unprecedented support for computational imaging, leading to more efficient solutions for various PR problems. In this review, we first briefly introduce conventional methods for PR. Then, we review how DL provides support for PR from the following three stages, namely, pre-processing, in-processing, and post-processing. We also review how DL is used in phase image processing. Finally, we summarize the work in DL for PR and provide an outlook on how to better use DL to improve the reliability and efficiency of PR. Furthermore, we present a live-updating resource ( https://github.com/kqwang/phase-recovery ) for readers to learn more about PR.

Physics constrained unsupervised deep learning for rapid, high resolution scanning coherent diffraction reconstruction

By circumventing the resolution limitations of optics, coherent diffractive imaging (CDI) and ptychography are making their way into scientific fields ranging from X-ray imaging to astronomy. Yet, the need for time consuming iterative phase recovery hampers real-time imaging. While supervised deep learning strategies have increased reconstruction speed, they sacrifice image quality. Furthermore, these methods' demand for extensive labeled training data is experimentally burdensome. Here, we propose an unsupervised physics-informed neural network reconstruction method, PtychoPINN, that retains the factor of 100-to-1000 speedup of deep learning-based reconstruction while improving reconstruction quality by combining the diffraction forward map with real-space constraints from overlapping measurements. In particular, PtychoPINN gains a factor of 4 in linear resolution and an 8 dB improvement in PSNR while also accruing improvements in generalizability and robustness. This blend of performance and computational efficiency offers exciting prospects for high-resolution real-time imaging in high-throughput environments such as X-ray free electron lasers (XFELs) and diffraction-limited light sources.

Solving complex nanostructures with ptychographic atomic electron tomography

Transmission electron microscopy (TEM) is essential for determining atomic scale structures in structural biology and materials science. In structural biology, three-dimensional structures of proteins are routinely determined from thousands of identical particles using phase-contrast TEM. In materials science, three-dimensional atomic structures of complex nanomaterials have been determined using atomic electron tomography (AET). However, neither of these methods can determine the three-dimensional atomic structure of heterogeneous nanomaterials containing light elements. Here, we perform ptychographic electron tomography from 34.5 million diffraction patterns to reconstruct an atomic resolution tilt series of a double wall-carbon nanotube (DW-CNT) encapsulating a complex ZrTe sandwich structure. Class averaging the resulting tilt series images and subpixel localization of the atomic peaks reveals a Zr11Te50 structure containing a previously unobserved ZrTe2 phase in the core. The experimental realization of atomic resolution ptychographic electron tomography will allow for the structural determination of a wide range of beam-sensitive nanomaterials containing light elements.

Deep learning at the edge enables real-time streaming ptychographic imaging

Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the oversampling constraints, allowing low-dose imaging using orders of magnitude less data than required by traditional methods.

Using machine learning to improve multi-qubit state discrimination of trapped ions from uncertain EMCCD measurements

This paper proposes a residual network (ResNet)-based convolutional neural network (CNN) model to improve multi-qubit state measurements using an electron-multiplying charge-coupled device (EMCCD). The CNN model is developed to simultaneously use the intensity of pixel values and the shape of ion images in determining the quantum states of ions. In contrast, conventional methods use only the intensity values. In our experiments, the proposed model achieved a 99.53±0.14% mean individual measurement fidelity (MIMF) of 4 trapped ions, reducing the error by 46% when compared to the MIMF of maximum likelihood estimation method of 99.13±0.08%. In addition, it is experimentally shown that the model is also robust against the ion image drift, which was tested by intentionally shifting the ion images.

Applications of Transmission Electron Microscopy in Phase Engineering of Nanomaterials

Phase engineering of nanomaterials (PEN) is an emerging field that aims to tailor the physicochemical properties of nanomaterials by precisely manipulating their crystal phases. To advance PEN effectively, it is vital to possess the capability of characterizing the structures and compositions of nanomaterials with precision. Transmission electron microscopy (TEM) is a versatile tool that combines reciprocal-space diffraction, real-space imaging, and spectroscopic techniques, allowing for comprehensive characterization with exceptional resolution in the domains of time, space, momentum, and, increasingly, even energy. In this Review, we first introduce the fundamental mechanisms behind various TEM-related techniques, along with their respective application scopes and limitations. Subsequently, we review notable applications of TEM in PEN research, including applications in fields such as metallic nanostructures, carbon allotropes, low-dimensional materials, and nanoporous materials. Specifically, we underscore its efficacy in phase identification, composition and chemical state analysis, in situ observations of phase evolution, as well as the challenges encountered when dealing with beam-sensitive materials. Furthermore, we discuss the potential generation of artifacts during TEM imaging, particularly in scanning modes, and propose methods to minimize their occurrence. Finally, we offer our insights into the present state and future trends of this field, discussing emerging technologies including four-dimensional scanning TEM, three-dimensional atomic-resolution imaging, and electron microscopy automation while highlighting the significance and feasibility of these advancements.

Machine learning assisted interpretation of 2D solid-state nuclear magnetic resonance spectra

A machine learning methodology using deep neural network (DNN) for interpreting multidimensional solid-state nuclear magnetic resonance (SSNMR) of various synthetic and natural polymers is presented. The separated local field (SLF) SSNMR which correlates local well-defined heteronuclear dipolar with the tensor orientation of the chemical shift anisotropy (CSA) of spin in the solid state can provide valuable structure and molecular dynamics information of synthetic and biopolymers. Compared with the traditional linear least-square fitting, the proposed DNN-based methodology can efficiently and accurately determine the tensor orientation of CSA of both 13C and 15N in all four samples. The method achieves prediction precisions of the Euler angles with < ±5° and is characterized by low training costs and high efficiency (<1 s). The feasibility and robustness of the DNN-based analysis methodology are confirmed by comparison to reported-literature values. This strategy is expected to aid in the interpretation of complex multidimensional NMR spectra of complicated polymer system.