Unsupervised classification of single-particle X-ray diffraction snapshots by spectral clustering

Characterization of Biological Samples Using Ultra-Short and Ultra-Bright XFEL Pulses

The advent of X-ray Free Electron Lasers (XFELs) has ushered in a transformative era in the field of structural biology, materials science, and ultrafast physics. These state-of-the-art facilities generate ultra-bright, femtosecond-long X-ray pulses, allowing researchers to delve into the structure and dynamics of molecular systems with unprecedented temporal and spatial resolutions. The unique properties of XFEL pulses have opened new avenues for scientific exploration that were previously considered unattainable. One of the most notable applications of XFELs is in structural biology. Traditional X-ray crystallography, while instrumental in determining the structures of countless biomolecules, often requires large, high-quality crystals and may not capture highly transient states of proteins. XFELs, with their ability to produce diffraction patterns from nanocrystals or even single particles, have provided solutions to these challenges. XFEL has expanded the toolbox of structural biologists by enabling structural determination approaches such as Single Particle Imaging (SPI) and Serial X-ray Crystallography (SFX). Despite their remarkable capabilities, the journey of XFELs is still in its nascent stages, with ongoing advancements aimed at improving their coherence, pulse duration, and wavelength tunability.

SpeckleNN: a unified embedding for real-time speckle pattern classification in X-ray single-particle imaging with limited labeled examples

With X-ray free-electron lasers (XFELs), it is possible to determine the three-dimensional structure of noncrystalline nanoscale particles using X-ray single-particle imaging (SPI) techniques at room temperature. Classifying SPI scattering patterns, or `speckles', to extract single-hits that are needed for real-time vetoing and three-dimensional reconstruction poses a challenge for high-data-rate facilities like the European XFEL and LCLS-II-HE. Here, we introduce SpeckleNN, a unified embedding model for real-time speckle pattern classification with limited labeled examples that can scale linearly with dataset size. Trained with twin neural networks, SpeckleNN maps speckle patterns to a unified embedding vector space, where similarity is measured by Euclidean distance. We highlight its few-shot classification capability on new never-seen samples and its robust performance despite having only tens of labels per classification category even in the presence of substantial missing detector areas. Without the need for excessive manual labeling or even a full detector image, our classification method offers a great solution for real-time high-throughput SPI experiments.

Finding the semantic similarity in single-particle diffraction images using self-supervised contrastive projection learning

Single-shot coherent diffraction imaging of isolated nanosized particles has seen remarkable success in recent years, yielding in-situ measurements with ultra-high spatial and temporal resolution. The progress of high-repetition-rate sources for intense X-ray pulses has further enabled recording datasets containing millions of diffraction images, which are needed for the structure determination of specimens with greater structural variety and dynamic experiments. The size of the datasets, however, represents a monumental problem for their analysis. Here, we present an automatized approach for finding semantic similarities in coherent diffraction images without relying on human expert labeling. By introducing the concept of projection learning, we extend self-supervised contrastive learning to the context of coherent diffraction imaging and achieve a dimensionality reduction producing semantically meaningful embeddings that align with physical intuition. The method yields substantial improvements compared to previous approaches, paving the way toward real-time and large-scale analysis of coherent diffraction experiments at X-ray free-electron lasers.

Skopi: a simulation package for diffractive imaging of noncrystalline biomolecules

X-ray free-electron lasers (XFELs) have the ability to produce ultra-bright femtosecond X-ray pulses for coherent diffraction imaging of biomolecules. While the development of methods and algorithms for macromolecular crystallography is now mature, XFEL experiments involving aerosolized or solvated biomolecular samples offer new challenges in terms of both experimental design and data processing. Skopi is a simulation package that can generate single-hit diffraction images for reconstruction algorithms, multi-hit diffraction images of aggregated particles for training machine learning classifiers using labeled data, diffraction images of randomly distributed particles for fluctuation X-ray scattering algorithms, and diffraction images of reference and target particles for holographic reconstruction algorithms. Skopi is a resource to aid feasibility studies and advance the development of algorithms for noncrystalline experiments at XFEL facilities.

Unsupervised learning approaches to characterizing heterogeneous samples using X-ray single-particle imaging

One of the outstanding analytical problems in X-ray single-particle imaging (SPI) is the classification of structural heterogeneity, which is especially difficult given the low signal-to-noise ratios of individual patterns and the fact that even identical objects can yield patterns that vary greatly when orientation is taken into consideration. Proposed here are two methods which explicitly account for this orientation-induced variation and can robustly determine the structural landscape of a sample ensemble. The first, termed common-line principal component analysis (PCA), provides a rough classification which is essentially parameter free and can be run automatically on any SPI dataset. The second method, utilizing variation auto-encoders (VAEs), can generate 3D structures of the objects at any point in the structural landscape. Both these methods are implemented in combination with the noise-tolerant expand-maximize-compress (EMC) algorithm and its utility is demonstrated by applying it to an experimental dataset from gold nanoparticles with only a few thousand photons per pattern. Both discrete structural classes and continuous deformations are recovered. These developments diverge from previous approaches of extracting reproducible subsets of patterns from a dataset and open up the possibility of moving beyond the study of homogeneous sample sets to addressing open questions on topics such as nanocrystal growth and dynamics, as well as phase transitions which have not been externally triggered.

X-ray fan beam coded aperture transmission and diffraction imaging for fast material analysis

X-ray transmission imaging has been used in a variety of applications for high-resolution measurements based on shape and density. Similarly, X-ray diffraction (XRD) imaging has been used widely for molecular structure-based identification of materials. Combining these X-ray methods has the potential to provide high-resolution material identification, exceeding the capabilities of either modality alone. However, XRD imaging methods have been limited in application by their long measurement times and poor spatial resolution, which has generally precluded combined, rapid measurements of X-ray transmission and diffraction. In this work, we present a novel X-ray fan beam coded aperture transmission and diffraction imaging system, developed using commercially available components, for rapid and accurate non-destructive imaging of industrial and biomedical specimens. The imaging system uses a 160 kV Bremsstrahlung X-ray source while achieving a spatial resolution of ≈ 1 × 1 mm² and a spectral accuracy of > 95% with only 15 s exposures per 150 mm fan beam slice. Applications of this technology are reported in geological imaging, pharmaceutical inspection, and medical diagnosis. The performance of the imaging system indicates improved material differentiation relative to transmission imaging alone at scan times suitable for a variety of industrial and biomedical applications.

An encryption-decryption framework to validating single-particle imaging

We propose an encryption-decryption framework for validating diffraction intensity volumes reconstructed using single-particle imaging (SPI) with X-ray free-electron lasers (XFELs) when the ground truth volume is absent. This conceptual framework exploits each reconstructed volumes' ability to decipher latent variables (e.g. orientations) of unseen sentinel diffraction patterns. Using this framework, we quantify novel measures of orientation disconcurrence, inconsistency, and disagreement between the decryptions by two independently reconstructed volumes. We also study how these measures can be used to define data sufficiency and its relation to spatial resolution, and the practical consequences of focusing XFEL pulses to smaller foci. This conceptual framework overcomes critical ambiguities in using Fourier Shell Correlation (FSC) as a validation measure for SPI. Finally, we show how this encryption-decryption framework naturally leads to an information-theoretic reformulation of the resolving power of XFEL-SPI, which we hope will lead to principled frameworks for experiment and instrument design.

Selecting XFEL single-particle snapshots by geometric machine learning

A promising new route for structural biology is single-particle imaging with an X-ray Free-Electron Laser (XFEL). This method has the advantage that the samples do not require crystallization and can be examined at room temperature. However, high-resolution structures can only be obtained from a sufficiently large number of diffraction patterns of individual molecules, so-called single particles. Here, we present a method that allows for efficient identification of single particles in very large XFEL datasets, operates at low signal levels, and is tolerant to background. This method uses supervised Geometric Machine Learning (GML) to extract low-dimensional feature vectors from a training dataset, fuse test datasets into the feature space of training datasets, and separate the data into binary distributions of "single particles" and "non-single particles." As a proof of principle, we tested simulated and experimental datasets of the Coliphage PR772 virus. We created a training dataset and classified three types of test datasets: First, a noise-free simulated test dataset, which gave near perfect separation. Second, simulated test datasets that were modified to reflect different levels of photon counts and background noise. These modified datasets were used to quantify the predictive limits of our approach. Third, an experimental dataset collected at the Stanford Linear Accelerator Center. The single-particle identification for this experimental dataset was compared with previously published results and it was found that GML covers a wide photon-count range, outperforming other single-particle identification methods. Moreover, a major advantage of GML is its ability to retrieve single particles in the presence of structural variability.

Flash X-ray diffraction imaging in 3D: a proposed analysis pipeline

Modern Flash X-ray diffraction Imaging (FXI) acquires diffraction signals from single biomolecules at a high repetition rate from X-ray Free Electron Lasers (XFELs), easily obtaining millions of 2D diffraction patterns from a single experiment. Due to the stochastic nature of FXI experiments and the massive volumes of data, retrieving 3D electron densities from raw 2D diffraction patterns is a challenging and time-consuming task. We propose a semi-automatic data analysis pipeline for FXI experiments, which includes four steps: hit-finding and preliminary filtering, pattern classification, 3D Fourier reconstruction, and post-analysis. We also include a recently developed bootstrap methodology in the post-analysis step for uncertainty analysis and quality control. To achieve the best possible resolution, we further suggest using background subtraction, signal windowing, and convex optimization techniques when retrieving the Fourier phases in the post-analysis step. As an application example, we quantified the 3D electron structure of the PR772 virus using the proposed data analysis pipeline. The retrieved structure was above the detector edge resolution and clearly showed the pseudo-icosahedral capsid of the PR772.

Rapid Identification of X-ray Diffraction Patterns Based on Very Limited Data by Interpretable Convolutional Neural Networks

Large volumes of data from material characterizations call for rapid and automatic data analysis to accelerate materials discovery. Herein, we report a convolutional neural network (CNN) that was trained based on theoretical data and very limited experimental data for fast identification of experimental X-ray diffraction (XRD) patterns of metal-organic frameworks (MOFs). To augment the data for training the model, noise was extracted from experimental data and shuffled; then it was merged with the main peaks that were extracted from theoretical spectra to synthesize new spectra. For the first time, one-to-one material identification was achieved. Theoretical MOFs patterns (1012) were augmented to a whole data set of 72 864 samples. It was then randomly shuffled and split into training (58 292 samples) and validation (14 572 samples) data sets at a ratio of 4:1. For the task of discriminating, the optimized model showed the highest identification accuracy of 96.7% for the top 5 ranking on a test data set of 30 hold-out samples. Neighborhood component analysis (NCA) on the experimental XRD samples shows that the samples from the same material are clustered in groups in the NCA map. Analysis on the class activation maps of the last CNN layer further discloses the mechanism by which the CNN model successfully identifies individual MOFs from the XRD patterns. This CNN model trained by the data augmentation technique would not only open numerous potential applications for identifying XRD patterns for different materials, but also pave avenues to autonomously analyze data by other characterization tools such as FTIR, Raman, and NMR spectroscopies.

Methods and application of coherent X-ray diffraction imaging of noncrystalline particles

Microscopic imaging techniques have been developed to visualize events occurring in biological cells. Coherent X-ray diffraction imaging is one of the techniques applicable to structural analyses of cells and organelles, which have never been crystallized. In the experiment, a single noncrystalline particle is illuminated by an X-ray beam with almost complete spatial coherence. The structure of the particle projected along the direction of the beam is, in principle, retrieved from a finely recorded diffraction pattern alone by using iterative phase-retrieval algorithms. Here, we describe fundamental theory and experimental methods of coherent X-ray diffraction imaging and the recent application in structural studies of noncrystalline specimens by using X-rays available at Super Photon Ring of 8-Gev and SPring-8 Angstrom Compact Free Electron Laser in Japan.

Low-signal limit of X-ray single particle diffractive imaging

An outstanding question in X-ray single particle imaging experiments has been the feasibility of imaging sub 10-nm-sized biomolecules under realistic experimental conditions where very few photons are expected to be measured in a single snapshot and instrument background may be significant relative to particle scattering. While analyses of simulated data have shown that the determination of an average image should be feasible using Bayesian methods such as the EMC algorithm, this has yet to be demonstrated using experimental data containing realistic non-isotropic instrument background, sample variability and other experimental factors. In this work, we show that the orientation and phase retrieval steps work at photon counts diluted to the signal levels one expects from smaller molecules or with weaker pulses, using data from experimental measurements of 60-nm PR772 viruses. Even when the signal is reduced to a fraction as little as 1/256, the virus electron density determined using ab initio phasing is of almost the same quality as the high-signal data. However, we are still limited by the total number of patterns collected, which may soon be mitigated by the advent of high repetition-rate sources like the European XFEL and LCLS-II.

Deep neural networks for classifying complex features in diffraction images

Intense short-wavelength pulses from free-electron lasers and high-harmonic-generation sources enable diffractive imaging of individual nanosized objects with a single x-ray laser shot. The enormous data sets with up to several million diffraction patterns present a severe problem for data analysis because of the high dimensionality of imaging data. Feature recognition and selection is a crucial step to reduce the dimensionality. Usually, custom-made algorithms are developed at a considerable effort to approximate the particular features connected to an individual specimen, but because they face different experimental conditions, these approaches do not generalize well. On the other hand, deep neural networks are the principal instrument for today's revolution in automated image recognition, a development that has not been adapted to its full potential for data analysis in science. We recently published [Langbehn et al., Phys. Rev. Lett. 121, 255301 (2018)PRLTAO0031-900710.1103/PhysRevLett.121.255301] the application of a deep neural network as a feature extractor for wide-angle diffraction images of helium nanodroplets. Here we present the setup, our modifications, and the training process of the deep neural network for diffraction image classification and its systematic bench marking. We find that deep neural networks significantly outperform previous attempts for sorting and classifying complex diffraction patterns and are a significant improvement for the much-needed assistance during postprocessing of large amounts of experimental coherent diffraction imaging data.

Evaluation of the performance of classification algorithms for XFEL single-particle imaging data

Using X-ray free-electron lasers (XFELs), it is possible to determine three-dimensional structures of nanoscale particles using single-particle imaging methods. Classification algorithms are needed to sort out the single-particle diffraction patterns from the large amount of XFEL experimental data. However, different methods often yield inconsistent results. This study compared the performance of three classification algorithms: convolutional neural network, graph cut and diffusion map manifold embedding methods. The identified single-particle diffraction data of the PR772 virus particles were assembled in the three-dimensional Fourier space for real-space model reconstruction. The comparison showed that these three classification methods lead to different datasets and subsequently result in different electron density maps of the reconstructed models. Interestingly, the common dataset selected by these three methods improved the quality of the merged diffraction volume, as well as the resolutions of the reconstructed maps.

Supervised classification methods for flash X-ray single particle diffraction imaging

Current Flash X-ray single-particle diffraction Imaging (FXI) experiments, which operate on modern X-ray Free Electron Lasers (XFELs), can record millions of interpretable diffraction patterns from individual biomolecules per day. Due to the practical limitations with the FXI technology, those patterns will to a varying degree include scatterings from contaminated samples. Also, the heterogeneity of the sample biomolecules is unavoidable and complicates data processing. Reducing the data volumes and selecting high-quality single-molecule patterns are therefore critical steps in the experimental setup. In this paper, we present two supervised template-based learning methods for classifying FXI patterns. Our Eigen-Image and Log-Likelihood classifier can find the best-matched template for a single-molecule pattern within a few milliseconds. It is also straightforward to parallelize them so as to match the XFEL repetition rate fully, thereby enabling processing at site. The methods perform in a stable way on various kinds of synthetic data. As a practical example we tested our methods on a real mimivirus dataset, obtaining a convincing classification accuracy of 0.9.

Ab initio structure determination from experimental fluctuation X-ray scattering data

Fluctuation X-ray scattering is a biophysical structural characterization technique that overcomes low data-to-parameter ratios encountered in traditional X-ray methods used for studying noncrystalline samples. By collecting a series of ultrashort X-ray exposures on an ensemble of particles at a free-electron laser, information-dense experimental data can be extracted that ultimately result in structures with a greater level of detail than can be obtained using traditional X-ray scattering methods. In this article we demonstrate the practical feasibility of this technique by introducing data-processing techniques and advanced noise-filtering methods that reduce the required data collection time to less than a few minutes. This will ultimately allow one to visualize details of structural dynamics that may be inaccessible through traditional methods.

Fluctuation X-ray scattering (FXS) is an emerging experimental technique in which X-ray solution scattering data are collected from particles in solution using ultrashort X-ray exposures generated by a free-electron laser (FEL). FXS experiments overcome the low data-to-parameter ratios associated with traditional solution scattering measurements by providing several orders of magnitude more information in the final processed data. Here we demonstrate the practical feasibility of FEL-based FXS on a biological multiple-particle system and describe data-processing techniques required to extract robust FXS data and significantly reduce the required number of snapshots needed by introducing an iterative noise-filtering technique. We showcase a successful ab initio electron density reconstruction from such an experiment, studying the Paramecium bursaria Chlorella virus (PBCV-1).

Correlations in Scattered X-Ray Laser Pulses Reveal Nanoscale Structural Features of Viruses

We use extremely bright and ultrashort pulses from an x-ray free-electron laser (XFEL) to measure correlations in x rays scattered from individual bioparticles. This allows us to go beyond the traditional crystallography and single-particle imaging approaches for structure investigations. We employ angular correlations to recover the three-dimensional (3D) structure of nanoscale viruses from x-ray diffraction data measured at the Linac Coherent Light Source. Correlations provide us with a comprehensive structural fingerprint of a 3D virus, which we use both for model-based and ab initio structure recovery. The analyses reveal a clear indication that the structure of the viruses deviates from the expected perfect icosahedral symmetry. Our results anticipate exciting opportunities for XFEL studies of the structure and dynamics of nanoscale objects by means of angular correlations.

XFELs for structure and dynamics in biology

The development and application of the free-electron X-ray laser (XFEL) to structure and dynamics in biology since its inception in 2009 are reviewed. The research opportunities which result from the ability to outrun most radiation-damage effects are outlined, and some grand challenges are suggested. By avoiding the need to cool samples to minimize damage, the XFEL has permitted atomic resolution imaging of molecular processes on the 100 fs timescale under near-physiological conditions and in the correct thermal bath in which molecular machines operate. Radiation damage, comparisons of XFEL and synchrotron work, single-particle diffraction, fast solution scattering, pump-probe studies on photosensitive proteins, mix-and-inject experiments, caged molecules, pH jump and other reaction-initiation methods, and the study of molecular machines are all discussed. Sample-delivery methods and data-analysis algorithms for the various modes, from serial femtosecond crystallo-graphy to fast solution scattering, fluctuation X-ray scattering, mixing jet experiments and single-particle diffraction, are also reviewed.

Healing X-ray scattering images

X-ray scattering images contain numerous gaps and defects arising from detector limitations and experimental configuration. We present a method to heal X-ray scattering images, filling gaps in the data and removing defects in a physically meaningful manner. Unlike generic inpainting methods, this method is closely tuned to the expected structure of reciprocal-space data. In particular, we exploit statistical tests and symmetry analysis to identify the structure of an image; we then copy, average and interpolate measured data into gaps in a way that respects the identified structure and symmetry. Importantly, the underlying analysis methods provide useful characterization of structures present in the image, including the identification of diffuse versus sharp features, anisotropy and symmetry. The presented method leverages known characteristics of reciprocal space, enabling physically reasonable reconstruction even with large image gaps. The method will correspondingly fail for images that violate these underlying assumptions. The method assumes point symmetry and is thus applicable to small-angle X-ray scattering (SAXS) data, but only to a subset of wide-angle data. Our method succeeds in filling gaps and healing defects in experimental images, including extending data beyond the original detector borders.

Reconstruction from limited single-particle diffraction data via simultaneous determination of state, orientation, intensity, and phase

Free-electron lasers now have the ability to collect X-ray diffraction patterns from individual molecules; however, each sample is delivered at unknown orientation and may be in one of several conformational states, each with a different molecular structure. Hit rates are often low, typically around 0.1%, limiting the number of useful images that can be collected. Determining accurate structural information requires classifying and orienting each image, accurately assembling them into a 3D diffraction intensity function, and determining missing phase information. Additionally, single particles typically scatter very few photons, leading to high image noise levels. We develop a multitiered iterative phasing algorithm to reconstruct structural information from single-particle diffraction data by simultaneously determining the states, orientations, intensities, phases, and underlying structure in a single iterative procedure. We leverage real-space constraints on the structure to help guide optimization and reconstruct underlying structure from very few images with excellent global convergence properties. We show that this approach can determine structural resolution beyond what is suggested by standard Shannon sampling arguments for ideal images and is also robust to noise.

Single-shot diffraction data from the Mimivirus particle using an X-ray free-electron laser

Free-electron lasers (FEL) hold the potential to revolutionize structural biology by producing X-ray pules short enough to outrun radiation damage, thus allowing imaging of biological samples without the limitation from radiation damage. Thus, a major part of the scientific case for the first FELs was three-dimensional (3D) reconstruction of non-crystalline biological objects. In a recent publication we demonstrated the first 3D reconstruction of a biological object from an X-ray FEL using this technique. The sample was the giant Mimivirus, which is one of the largest known viruses with a diameter of 450 nm. Here we present the dataset used for this successful reconstruction. Data-analysis methods for single-particle imaging at FELs are undergoing heavy development but data collection relies on very limited time available through a highly competitive proposal process. This dataset provides experimental data to the entire community and could boost algorithm development and provide a benchmark dataset for new algorithms.

CFEL-ASG Software Suite (CASS): usage for free-electron laser experiments with biological focus

CASS [Foucar et al. (2012). Comput. Phys. Commun.183, 2207-2213] is a well established software suite for experiments performed at any sort of light source. It is based on a modular design and can easily be adapted for use at free-electron laser (FEL) experiments that have a biological focus. This article will list all the additional functionality and enhancements of CASS for use with FEL experiments that have been introduced since the first publication. The article will also highlight some advanced experiments with biological aspects that have been performed.

Sorting algorithms for single-particle imaging experiments at X-ray free-electron lasers

Modern X-ray free-electron lasers (XFELs) operating at high repetition rates produce a tremendous amount of data. It is a great challenge to classify this information and reduce the initial data set to a manageable size for further analysis. Here an approach for classification of diffraction patterns measured in prototypical diffract-and-destroy single-particle imaging experiments at XFELs is presented. It is proposed that the data are classified on the basis of a set of parameters that take into account the underlying diffraction physics and specific relations between the real-space structure of a particle and its reciprocal-space intensity distribution. The approach is demonstrated by applying principal component analysis and support vector machine algorithms to the simulated and measured X-ray data sets.

Classification of projection images of proteins with structural polymorphism by manifold: a simulation study for x-ray free-electron laser diffraction imaging

Coherent x-ray diffraction imaging (CXDI) enables us to visualize noncrystalline sample particles with micrometer to submicrometer dimensions. Using x-ray free-electron laser (XFEL) sources, two-dimensional diffraction patterns are collected from fresh samples supplied to the irradiation area in the "diffraction-before-destruction" scheme. A recent significant increase in the intensity of the XFEL pulse is promising and will allow us to visualize the three-dimensional structures of proteins using XFEL-CXDI in the future. For the protocol proposed for molecular structure determination using future XFEL-CXDI [T. Oroguchi and M. Nakasako, Phys. Rev. E 87, 022712 (2013)10.1103/PhysRevE.87.022712], we require an algorithm that can classify the data in accordance with the structural polymorphism of proteins arising from their conformational dynamics. However, most of the algorithms proposed primarily require the numbers of conformational classes, and then the results are biased by the numbers. To improve this point, here we examine whether a method based on the manifold concept can classify simulated XFEL-CXDI data with respect to the structural polymorphism of a protein that predominantly adopts two states. After random sampling of the conformations of the two states and in-between states from the trajectories of molecular dynamics simulations, a diffraction pattern is calculated from each conformation. Classification was performed by using our custom-made program suite named enma, in which the diffusion map (DM) method developed based on the manifold concept was implemented. We successfully classify most of the projection electron density maps phase retrieved from diffraction patterns into each of the two states and in-between conformations without the knowledge of the number of conformational classes. We also examined the classification of the projection electron density maps of each of the three states with respect to the Euler angle. The present results suggest that the DM method is imperative for future applications of XFEL-CXDI experiments for proteins, and clarify issues to be taken care of in the future application.

Data acquisition system for X-ray free-electron laser experiments at SACLA

A data acquisition system for X-ray free-electron laser experiments at SACLA has been developed. The system has been designed for reliable shot-to-shot data storage with a high data stream greater than 4 Gbps and massive data analysis. Configuration of the system and examples of prompt data analysis during experiments are presented. Upgrade plans for the system to extend flexibility are described.

High-resolution structure of viruses from random diffraction snapshots

The advent of the X-ray free-electron laser (XFEL) has made it possible to record diffraction snapshots of biological entities injected into the X-ray beam before the onset of radiation damage. Algorithmic means must then be used to determine the snapshot orientations and thence the three-dimensional structure of the object. Existing Bayesian approaches are limited in reconstruction resolution typically to 1/10 of the object diameter, with the computational expense increasing as the eighth power of the ratio of diameter to resolution. We present an approach capable of exploiting object symmetries to recover three-dimensional structure to high resolution, and thus reconstruct the structure of the satellite tobacco necrosis virus to atomic level. Our approach offers the highest reconstruction resolution for XFEL snapshots to date and provides a potentially powerful alternative route for analysis of data from crystalline and nano-crystalline objects.

Conformations of macromolecules and their complexes from heterogeneous datasets

We describe a new generation of algorithms capable of mapping the structure and conformations of macromolecules and their complexes from large ensembles of heterogeneous snapshots, and demonstrate the feasibility of determining both discrete and continuous macromolecular conformational spectra. These algorithms naturally incorporate conformational heterogeneity without resort to sorting and classification, or prior knowledge of the type of heterogeneity present. They are applicable to single-particle diffraction and image datasets produced by X-ray lasers and cryo-electron microscopy, respectively, and particularly suitable for systems not easily amenable to purification or crystallization.

Cheetah: software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data

The emerging technique of serial X-ray diffraction, in which diffraction data are collected from samples flowing across a pulsed X-ray source at repetition rates of 100 Hz or higher, has necessitated the development of new software in order to handle the large data volumes produced. Sorting of data according to different criteria and rapid filtering of events to retain only diffraction patterns of interest results in significant reductions in data volume, thereby simplifying subsequent data analysis and management tasks. Meanwhile the generation of reduced data in the form of virtual powder patterns, radial stacks, histograms and other meta data creates data set summaries for analysis and overall experiment evaluation. Rapid data reduction early in the analysis pipeline is proving to be an essential first step in serial imaging experiments, prompting the authors to make the tool described in this article available to the general community. Originally developed for experiments at X-ray free-electron lasers, the software is based on a modular facility-independent library to promote portability between different experiments and is available under version 3 or later of the GNU General Public License.

Data processing software suite SITENNO for coherent X-ray diffraction imaging using the X-ray free-electron laser SACLA

Coherent X-ray diffraction imaging is a promising technique for visualizing the structures of non-crystalline particles with dimensions of micrometers to sub-micrometers. Recently, X-ray free-electron laser sources have enabled efficient experiments in the `diffraction before destruction' scheme. Diffraction experiments have been conducted at SPring-8 Angstrom Compact free-electron LAser (SACLA) using the custom-made diffraction apparatus KOTOBUKI-1 and two multiport CCD detectors. In the experiments, ten thousands of single-shot diffraction patterns can be collected within several hours. Then, diffraction patterns with significant levels of intensity suitable for structural analysis must be found, direct-beam positions in diffraction patterns determined, diffraction patterns from the two CCD detectors merged, and phase-retrieval calculations for structural analyses performed. A software suite named SITENNO has been developed to semi-automatically apply the four-step processing to a huge number of diffraction data. Here, details of the algorithm used in the suite are described and the performance for approximately 9000 diffraction patterns collected from cuboid-shaped copper oxide particles reported. Using the SITENNO suite, it is possible to conduct experiments with data processing immediately after the data collection, and to characterize the size distribution and internal structures of the non-crystalline particles.

Conformation sequence recovery of a non-periodic object from a diffraction-before-destruction experiment

Knowledge of the sequence of different conformational states of a protein molecule is key to better understanding its biological function. A diffraction pattern from a single conformational state can be captured with an ultrafast X-ray Free-Electron Laser (XFEL) before the target is completely annihilated by the radiation. In this paper, we report the first experimental demonstration of conformation sequence recovery using diffraction patterns from randomly ordered conformations of a non-periodic object using the dimensional reduction technique Isomap and coherent diffraction imaging.

Automated identification and classification of single particle serial femtosecond X-ray diffraction data

The first hard X-ray laser, the Linac Coherent Light Source (LCLS), produces 120 shots per second. Particles injected into the X-ray beam are hit randomly and in unknown orientations by the extremely intense X-ray pulses, where the femtosecond-duration X-ray pulses diffract from the sample before the particle structure is significantly changed even though the sample is ultimately destroyed by the deposited X-ray energy. Single particle X-ray diffraction experiments generate data at the FEL repetition rate, resulting in more than 400,000 detector readouts in an hour, the data stream during an experiment contains blank frames mixed with hits on single particles, clusters and contaminants. The diffraction signal is generally weak and it is superimposed on a low but continually fluctuating background signal, originating from photon noise in the beam line and electronic noise from the detector. Meanwhile, explosion of the sample creates fragments with a characteristic signature. Here, we describe methods based on rapid image analysis combined with ion Time-of-Flight (ToF) spectroscopy of the fragments to achieve an efficient, automated and unsupervised sorting of diffraction data. The studies described here form a basis for the development of real-time frame rejection methods, e.g. for the European XFEL, which is expected to produce 100 million pulses per hour.

Toward unsupervised single-shot diffractive imaging of heterogeneous particles using X-ray free-electron lasers

Single shot diffraction imaging experiments via X-ray free-electron lasers can generate as many as hundreds of thousands of diffraction patterns of scattering objects. Recovering the real space contrast of a scattering object from these patterns currently requires a reconstruction process with user guidance in a number of steps, introducing severe bottlenecks in data processing. We present a series of measures that replace user guidance with algorithms that reconstruct contrasts in an unsupervised fashion. We demonstrate the feasibility of automating the reconstruction process by generating hundreds of contrasts obtained from soot particle diffraction experiments.

Prospect of Retrieving Vibrational Wave Function by Single-Object Scattering Sampling

The exact shape of wave functions has never been directly measured because an ensemble measurement is often overwhelmed by the contributions of highly populated configurations. In this work, we explore the possibility of directly obtaining vibrational wave functions by single-object scattering sampling (SOSS) using intense, ultrashort X-ray pulses provided by X-ray free electron lasers. Previously, single-molecule diffraction experiments using femtosecond X-ray pulses have been proposed with the prospect of determining three-dimensional structure of macromolecules without the need of single-crystal samples. In contrast to the previous proposals, SOSS is designed for obtaining the structural variations of constantly fluctuating molecules by sampling many single-shot, single-object scattering patterns. From the simulations on iodine molecules adopting various pulse characteristics and molecular parameters, we were able to reconstruct vibrational wave functions of molecular iodine and found that SOSS is feasible under appropriate experimental conditions.

Optimal mapping of x-ray laser diffraction patterns into three dimensions using routing algorithms

Coherent diffractive imaging with x-ray free-electron lasers (XFEL) promises high-resolution structure determination of noncrystalline objects. Randomly oriented particles are exposed to XFEL pulses for acquisition of two-dimensional (2D) diffraction snapshots. The knowledge of their orientations enables 3D imaging by multiview reconstruction, combining 2D diffraction snapshots in different orientations. Here we introduce a globally optimal algorithm that can infer these orientations. We apply it to experimental XFEL data of nanoparticles and so determine their 3D electron density.

Emerging opportunities in structural biology with X-ray free-electron lasers

X-ray free-electron lasers (X-FELs) produce X-ray pulses with extremely brilliant peak intensity and ultrashort pulse duration. It has been proposed that radiation damage can be 'outrun' by using an ultra intense and short X-FEL pulse that passes a biological sample before the onset of significant radiation damage. The concept of 'diffraction-before-destruction' has been demonstrated recently at the Linac Coherent Light Source, the first operational hard X-ray FEL, for protein nanocrystals and giant virus particles. The continuous diffraction patterns from single particles allow solving the classical 'phase problem' by the oversampling method with iterative algorithms. If enough data are collected from many identical copies of a (biological) particle, its three-dimensional structure can be reconstructed. We review the current status and future prospects of serial femtosecond crystallography (SFX) and single-particle coherent diffraction imaging (CDI) with X-FELs.

Femtosecond free-electron laser x-ray diffraction data sets for algorithm development

We describe femtosecond X-ray diffraction data sets of viruses and nanoparticles collected at the Linac Coherent Light Source. The data establish the first large benchmark data sets for coherent diffraction methods freely available to the public, to bolster the development of algorithms that are essential for developing this novel approach as a useful imaging technique. Applications are 2D reconstructions, orientation classification and finally 3D imaging by assembling 2D patterns into a 3D diffraction volume.