Deep neural networks for classifying complex features in diffraction images

Robust image descriptor for machine learning based data reduction in serial crystallography

Serial crystallography experiments at synchrotron and X-ray free-electron laser (XFEL) sources are producing crystallographic data sets of ever-increasing volume. While these experiments have large data sets and high-frame-rate detectors (around 3520 frames per second), only a small percentage of the data are useful for downstream analysis. Thus, an efficient and real-time data classification pipeline is essential to differentiate reliably between useful and non-useful images, typically known as 'hit' and 'miss', respectively, and keep only hit images on disk for further analysis such as peak finding and indexing. While feature-point extraction is a key component of modern approaches to image classification, existing approaches require computationally expensive patch preprocessing to handle perspective distortion. This paper proposes a pipeline to categorize the data, consisting of a real-time feature extraction algorithm called modified and parallelized FAST (MP-FAST), an image descriptor and a machine learning classifier. For parallelizing the primary operations of the proposed pipeline, central processing units, graphics processing units and field-programmable gate arrays are implemented and their performances compared. Finally, MP-FAST-based image classification is evaluated using a multi-layer perceptron on various data sets, including both synthetic and experimental data. This approach demonstrates superior performance compared with other feature extractors and classifiers.

Weakly supervised learning for pattern classification in serial femtosecond crystallography

Serial femtosecond crystallography at X-ray free electron laser facilities opens a new era for the determination of crystal structure. However, the data processing of those experiments is facing unprecedented challenge, because the total number of diffraction patterns needed to determinate a high-resolution structure is huge. Machine learning methods are very likely to play important roles in dealing with such a large volume of data. Convolutional neural networks have made a great success in the field of pattern classification, however, training of the networks need very large datasets with labels. This heavy dependence on labeled datasets will seriously restrict the application of networks, because it is very costly to annotate a large number of diffraction patterns. In this article we present our job on the classification of diffraction pattern by weakly supervised algorithms, with the aim of reducing as much as possible the size of the labeled dataset required for training. Our result shows that weakly supervised methods can significantly reduce the need for the number of labeled patterns while achieving comparable accuracy to fully supervised methods.

Application of self-supervised approaches to the classification of X-ray diffraction spectra during phase transitions

Spectroscopy and X-ray diffraction techniques encode ample information on investigated samples. The ability of rapidly and accurately extracting these enhances the means to steer the experiment, as well as the understanding of the underlying processes governing the experiment. It improves the efficiency of the experiment, and maximizes the scientific outcome. To address this, we introduce and validate three frameworks based on self-supervised learning which are capable of classifying 1D spectral curves using data transformations preserving the scientific content and only a small amount of data labeled by domain experts. In particular, in this work we focus on the identification of phase transitions in samples investigated by x-ray powder diffraction. We demonstrate that the three frameworks, based either on relational reasoning, contrastive learning, or a combination of the two, are capable of accurately identifying phase transitions. Furthermore, we discuss in detail the selection of data augmentation techniques, crucial to ensure that scientifically meaningful information is retained.

Finding the semantic similarity in single-particle diffraction images using self-supervised contrastive projection learning

Single-shot coherent diffraction imaging of isolated nanosized particles has seen remarkable success in recent years, yielding in-situ measurements with ultra-high spatial and temporal resolution. The progress of high-repetition-rate sources for intense X-ray pulses has further enabled recording datasets containing millions of diffraction images, which are needed for the structure determination of specimens with greater structural variety and dynamic experiments. The size of the datasets, however, represents a monumental problem for their analysis. Here, we present an automatized approach for finding semantic similarities in coherent diffraction images without relying on human expert labeling. By introducing the concept of projection learning, we extend self-supervised contrastive learning to the context of coherent diffraction imaging and achieve a dimensionality reduction producing semantically meaningful embeddings that align with physical intuition. The method yields substantial improvements compared to previous approaches, paving the way toward real-time and large-scale analysis of coherent diffraction experiments at X-ray free-electron lasers.

Coherent correlation imaging for resolving fluctuating states of matter

Fluctuations and stochastic transitions are ubiquitous in nanometre-scale systems, especially in the presence of disorder. However, their direct observation has so far been impeded by a seemingly fundamental, signal-limited compromise between spatial and temporal resolution. Here we develop coherent correlation imaging (CCI) to overcome this dilemma. Our method begins by classifying recorded camera frames in Fourier space. Contrast and spatial resolution emerge by averaging selectively over same-state frames. Temporal resolution down to the acquisition time of a single frame arises independently from an exceptionally low misclassification rate, which we achieve by combining a correlation-based similarity metric^1,2 with a modified, iterative hierarchical clustering algorithm^3,4. We apply CCI to study previously inaccessible magnetic fluctuations in a highly degenerate magnetic stripe domain state with nanometre-scale resolution. We uncover an intricate network of transitions between more than 30 discrete states. Our spatiotemporal data enable us to reconstruct the pinning energy landscape and to thereby explain the dynamics observed on a microscopic level. CCI massively expands the potential of emerging high-coherence X-ray sources and paves the way for addressing large fundamental questions such as the contribution of pinning^5-8 and topology^9-12 in phase transitions and the role of spin and charge order fluctuations in high-temperature superconductivity^13,14.

The Scatman: an approximate method for fast wide-angle scattering simulations

Single-shot coherent diffraction imaging (CDI) is a powerful approach to characterize the structure and dynamics of isolated nanoscale objects such as single viruses, aerosols, nanocrystals and droplets. Using X-ray wavelengths, the diffraction images in CDI experiments usually cover only small scattering angles of a few degrees. These small-angle patterns represent the magnitude of the Fourier transform of the 2D projection of the sample's electron density, which can be reconstructed efficiently but lacks any depth information. In cases where the diffracted signal can be measured up to scattering angles exceeding ∼10°, i.e. in the wide-angle regime, some 3D morphological information of the target is contained in a single-shot diffraction pattern. However, the extraction of the 3D structural information is no longer straightforward and defines the key challenge in wide-angle CDI. So far, the most convenient approach relies on iterative forward fitting of the scattering pattern using scattering simulations. Here the Scatman is presented, an approximate and fast numerical tool for the simulation and iterative fitting of wide-angle scattering images of isolated samples. Furthermore, the open-source software implementation of the Scatman algorithm, PyScatman, is published and described in detail. The Scatman approach, which has already been applied in previous work for forward-fitting-based shape retrieval, adopts the multi-slice Fourier transform method. The effects of optical properties are partially included, yielding quantitative results for small, isolated and weakly interacting samples. PyScatman is capable of computing wide-angle scattering patterns in a few milliseconds even on consumer-level computing hardware, potentially enabling new data analysis schemes for wide-angle coherent diffraction experiments.

Classification of diffraction patterns using a convolutional neural network in single-particle-imaging experiments performed at X-ray free-electron lasers

Single particle imaging (SPI) at X-ray free-electron lasers is particularly well suited to determining the 3D structure of particles at room temperature. For a successful reconstruction, diffraction patterns originating from a single hit must be isolated from a large number of acquired patterns. It is proposed that this task could be formulated as an image-classification problem and solved using convolutional neural network (CNN) architectures. Two CNN configurations are developed: one that maximizes the F1 score and one that emphasizes high recall. The CNNs are also combined with expectation-maximization (EM) selection as well as size filtering. It is observed that the CNN selections have lower contrast in power spectral density functions relative to the EM selection used in previous work. However, the reconstruction of the CNN-based selections gives similar results. Introducing CNNs into SPI experiments allows the reconstruction pipeline to be streamlined, enables researchers to classify patterns on the fly, and, as a consequence, enables them to tightly control the duration of their experiments. Incorporating non-standard artificial-intelligence-based solutions into an existing SPI analysis workflow may be beneficial for the future development of SPI experiments.

Unsupervised learning approaches to characterizing heterogeneous samples using X-ray single-particle imaging

One of the outstanding analytical problems in X-ray single-particle imaging (SPI) is the classification of structural heterogeneity, which is especially difficult given the low signal-to-noise ratios of individual patterns and the fact that even identical objects can yield patterns that vary greatly when orientation is taken into consideration. Proposed here are two methods which explicitly account for this orientation-induced variation and can robustly determine the structural landscape of a sample ensemble. The first, termed common-line principal component analysis (PCA), provides a rough classification which is essentially parameter free and can be run automatically on any SPI dataset. The second method, utilizing variation auto-encoders (VAEs), can generate 3D structures of the objects at any point in the structural landscape. Both these methods are implemented in combination with the noise-tolerant expand-maximize-compress (EMC) algorithm and its utility is demonstrated by applying it to an experimental dataset from gold nanoparticles with only a few thousand photons per pattern. Both discrete structural classes and continuous deformations are recovered. These developments diverge from previous approaches of extracting reproducible subsets of patterns from a dataset and open up the possibility of moving beyond the study of homogeneous sample sets to addressing open questions on topics such as nanocrystal growth and dynamics, as well as phase transitions which have not been externally triggered.

Noise reduction and mask removal neural network for X-ray single-particle imaging

Free-electron lasers could enable X-ray imaging of single biological macromolecules and the study of protein dynamics, paving the way for a powerful new imaging tool in structural biology, but a low signal-to-noise ratio and missing regions in the detectors, colloquially termed 'masks', affect data collection and hamper real-time evaluation of experimental data. In this article, the challenges posed by noise and masks are tackled by introducing a neural network pipeline that aims to restore diffraction intensities. For training and testing of the model, a data set of diffraction patterns was simulated from 10 900 different proteins with molecular weights within the range of 10-100 kDa and collected at a photon energy of 8 keV. The method is compared with a simple low-pass filtering algorithm based on autocorrelation constraints. The results show an improvement in the mean-squared error of roughly two orders of magnitude in the presence of masks compared with the noisy data. The algorithm was also tested at increasing mask width, leading to the conclusion that demasking can achieve good results when the mask is smaller than half of the central speckle of the pattern. The results highlight the competitiveness of this model for data processing and the feasibility of restoring diffraction intensities from unknown structures in real time using deep learning methods. Finally, an example is shown of this preprocessing making orientation recovery more reliable, especially for data sets containing very few patterns, using the expansion-maximization-compression algorithm.

Reconstruction of nanoscale particles from single-shot wide-angle free-electron-laser diffraction patterns with physics-informed neural networks

Single-shot wide-angle diffraction imaging is a widely used method to investigate the structure of noncrystallizing objects such as nanoclusters, large proteins, or even viruses. Its main advantage is that information about the three-dimensional structure of the object is already contained in a single image. This makes it useful for the reconstruction of fragile and nonreproducible particles without the need for tomographic measurements. However, currently there is no efficient numerical inversion algorithm available that is capable of determining the object's structure in real time. Neural networks, on the other hand, excel in image processing tasks suited for such purpose. Here we show how a physics-informed deep neural network can be used to reconstruct complete three-dimensional object models of uniform, convex particles on a voxel grid from single two-dimensional wide-angle scattering patterns. We demonstrate its universal reconstruction capabilities for silver nanoclusters, where the network uncovers novel geometric structures that reproduce the experimental scattering data with very high precision.

Classification of diffraction patterns in single particle imaging experiments performed at x-ray free-electron lasers using a convolutional neural network

Single particle imaging (SPI) is a promising method of native structure determination, which has undergone fast progress with the development of x-ray free-electron lasers. Large amounts of data are collected during SPI experiments, driving the need for automated data analysis. The necessary data analysis pipeline has a number of steps including binary object classification (single versus non-single hits). Classification and object detection are areas where deep neural networks currently outperform other approaches. In this work, we use the fast object detector networks YOLOv2 and YOLOv3. By exploiting transfer learning, a moderate amount of data is sufficient to train the neural network. We demonstrate here that a convolutional neural network can be successfully used to classify data from SPI experiments. We compare the results of classification for the two different networks, with different depth and architecture, by applying them to the same SPI data with different data representation. The best results are obtained for diffracted intensity represented by color images on a linear scale using YOLOv2 for classification. It shows an accuracy of about 95% with precision and recall of about 50% and 60%, respectively, in comparison to manual data classification.

Selecting XFEL single-particle snapshots by geometric machine learning

A promising new route for structural biology is single-particle imaging with an X-ray Free-Electron Laser (XFEL). This method has the advantage that the samples do not require crystallization and can be examined at room temperature. However, high-resolution structures can only be obtained from a sufficiently large number of diffraction patterns of individual molecules, so-called single particles. Here, we present a method that allows for efficient identification of single particles in very large XFEL datasets, operates at low signal levels, and is tolerant to background. This method uses supervised Geometric Machine Learning (GML) to extract low-dimensional feature vectors from a training dataset, fuse test datasets into the feature space of training datasets, and separate the data into binary distributions of "single particles" and "non-single particles." As a proof of principle, we tested simulated and experimental datasets of the Coliphage PR772 virus. We created a training dataset and classified three types of test datasets: First, a noise-free simulated test dataset, which gave near perfect separation. Second, simulated test datasets that were modified to reflect different levels of photon counts and background noise. These modified datasets were used to quantify the predictive limits of our approach. Third, an experimental dataset collected at the Stanford Linear Accelerator Center. The single-particle identification for this experimental dataset was compared with previously published results and it was found that GML covers a wide photon-count range, outperforming other single-particle identification methods. Moreover, a major advantage of GML is its ability to retrieve single particles in the presence of structural variability.

Fast reconstruction of single-shot wide-angle diffraction images through deep learning

Single-shot x-ray imaging of short-lived nanostructures such as clusters and nanoparticles near a phase transition or non-crystalizing objects such as large proteins and viruses is currently the most elegant method for characterizing their structure. Using hard x-ray radiation provides scattering images that encode two-dimensional projections, which can be combined to identify the full three-dimensional object structure from multiple identical samples. Wide-angle scattering using XUV or soft x-rays, despite yielding lower resolution, provides three-dimensional structural information in a single shot and has opened routes towards the characterization of non-reproducible objects in the gas phase. The retrieval of the structural information contained in wide-angle scattering images is highly non-trivial, and currently no efficient rigorous algorithm is known. Here we show that deep learning networks, trained with simulated scattering data, allow for fast and accurate reconstruction of shape and orientation of nanoparticles from experimental images. The gain in speed compared to conventional retrieval techniques opens the route for automated structure reconstruction algorithms capable of real-time discrimination and pre-identification of nanostructures in scattering experiments with high repetition rate—thus representing the enabling technology for fast femtosecond nanocrystallography.

Super-resolution of real-world rock microcomputed tomography images using cycle-consistent generative adversarial networks

Digital rock imaging plays an important role in studying the microstructure and macroscopic properties of rocks, where microcomputed tomography (MCT) is widely used. Due to the inherent limitations of MCT, a balance should be made between the field of view (FOV) and resolution of rock MCT images-a large FOV at low resolution (LR) or a small FOV at high resolution (HR). However, large FOV and HR are both expected for reliable analysis results in practice. Super-resolution (SR) is an effective solution to break through the mutual restriction between the FOV and resolution of rock MCT images, for it can reconstruct an HR image from a LR observation. Most of the existing SR methods cannot produce satisfactory HR results on real-world rock MCT images. One of the main reasons for this is that paired images are usually needed to learn the relationship between LR and HR rock images. However, it is challenging to collect such a dataset in a real scenario. Meanwhile, the simulated datasets may be unable to accurately reflect the model in actual applications. To address these problems, we propose a cycle-consistent generative adversarial network (CycleGAN)-based SR approach for real-world rock MCT images, namely, SRCycleGAN. In the off-line training phase, a set of unpaired rock MCT images is used to train the proposed SRCycleGAN, which can model the mapping between rock MCT images at different resolutions. In the on-line testing phase, the resolution of the LR input is enhanced via the learned mapping by SRCycleGAN. Experimental results show that the proposed SRCycleGAN can greatly improve the quality of simulated and real-world rock MCT images. The HR images reconstructed by SRCycleGAN show good agreement with the targets in terms of both the visual quality and the statistical parameters, including the porosity, the local porosity distribution, the two-point correlation function, the lineal-path function, the two-point cluster function, the chord-length distribution function, and the pore size distribution. Large FOV and HR rock MCT images can be obtained with the help of SRCycleGAN. Hence, this work makes it possible to generate HR rock MCT images that exceed the limitations of imaging systems on FOV and resolution.