IOTA: integration optimization, triage and analysis tool for the processing of XFEL diffraction images

Processing of Multicrystal Diffraction Patterns in Macromolecular Crystallography Using Serial Crystallography Programs

Cryocrystallography is a widely used method for determining the crystal structure of macromolecules. This technique uses a cryoenvironment, which significantly reduces the radiation damage to the crystals and has the advantage of requiring only one crystal for structural determination. In standard cryocrystallography, a single crystal is used for collecting diffraction data, which include single-crystal diffraction patterns. However, the X-ray data recorded often may contain diffraction patterns from several crystals. The indexing of multicrystal diffraction patterns in cryocrystallography requires more precise data processing techniques and is therefore time consuming. Here, an approach for processing multicrystal diffraction data using a serial crystallography program is introduced that allows for the integration of multicrystal diffraction patterns from a single image. Multicrystal diffraction data were collected from lysozyme crystals and processed using the serial crystallography program CrystFEL. From 360 images containing multicrystal diffraction patterns, 1138 and 691 crystal lattices could be obtained using the XGANDALF and MOSFLM indexing algorithms, respectively. Using this indexed multi-lattice information, the crystal structure of the lysozyme could be determined successfully at a resolution of 1.9 Å. Therefore, the proposed approach, which is based on serial crystallography, is suitable for processing multicrystal diffraction data in cryocrystallography.

DatView: a graphical user interface for visualizing and querying large data sets in serial femtosecond crystallography

DatView is a new graphical user interface (GUI) for plotting parameters to explore correlations, identify outliers and export subsets of data. It was designed to simplify and expedite analysis of very large unmerged serial femtosecond crystallography (SFX) data sets composed of indexing results from hundreds of thousands of microcrystal diffraction patterns. However, DatView works with any tabulated data, offering its functionality to many applications outside serial crystallography. In DatView's user-friendly GUI, selections are drawn onto plots and synchronized across all other plots, so correlations between multiple parameters in large multi-parameter data sets can be rapidly identified. It also includes an item viewer for displaying images in the current selection alongside the associated metadata. For serial crystallography data processed by indexamajig from CrystFEL [White, Kirian, Martin, Aquila, Nass, Barty & Chapman (2012 ▸). J. Appl. Cryst. 45, 335-341], DatView generates a table of parameters and metadata from stream files and, optionally, the associated HDF5 files. By combining the functionality of several commonly needed tools for SFX in a single GUI that operates on tabulated data, the time needed to load and calculate statistics from large data sets is reduced. This paper describes how DatView facilitates (i) efficient feedback during data collection by examining trends in time, sample position or any parameter, (ii) determination of optimal indexing and integration parameters via the comparison mode, (iii) identification of systematic errors in unmerged SFX data sets, and (iv) sorting and highly flexible data filtering (plot selections, Boolean filters and more), including direct export of subset CrystFEL stream files for further processing.

XGANDALF - extended gradient descent algorithm for lattice finding

Serial crystallography records still diffraction patterns from single, randomly oriented crystals, then merges data from hundreds or thousands of them to form a complete data set. To process the data, the diffraction patterns must first be indexed, equivalent to determining the orientation of each crystal. A novel automatic indexing algorithm is presented, which in tests usually gives significantly higher indexing rates than alternative programs currently available for this task. The algorithm does not require prior knowledge of the lattice parameters but can make use of that information if provided, and also allows indexing of diffraction patterns generated by several crystals in the beam. Cases with a small number of Bragg spots per pattern appear to particularly benefit from the new approach. The algorithm has been implemented and optimized for fast execution, making it suitable for real-time feedback during serial crystallography experiments. It is implemented in an open-source C++ library and distributed under the LGPLv3 licence. An interface to it has been added to the CrystFEL software suite.

Data-driven challenges and opportunities in crystallography

Structural biology is in the midst of a revolution fueled by faster and more powerful instruments capable of delivering orders of magnitude more data than their predecessors. This increased pace in data gathering introduces new experimental and computational challenges, frustrating real-time processing and interpretation of data and requiring long-term solutions for data archival and retrieval. This combination of challenges and opportunities is driving the exploration of new areas of structural biology, including studies of macromolecular dynamics and the investigation of molecular ensembles in search of a better understanding of conformational landscapes. The next generation of instruments promises to yield even greater data rates, requiring a concerted effort by institutions, centers and individuals to extract meaning from every bit and make data accessible to the community at large, facilitating data mining efforts by individuals or groups as analysis tools improve.

ClickX: a visualization-based program for preprocessing of serial crystallography data

A Python-based program for serial crystallography experimental data preprocessing is developed for both online and offline analysis. Enhanced features include a graphical user interface, batch job execution and fast parameter optimizations.

Serial crystallography is a powerful technique in structure determination using many small crystals at X-ray free-electron laser or synchrotron radiation facilities. The large diffraction data volumes require high-throughput software to preprocess the raw images for subsequent analysis. ClickX is a program designated for serial crystallography data preprocessing, capable of rapid data sorting for online feedback and peak-finding refinement by parameter optimization. The graphical user interface (GUI) provides convenient access to various operations such as pattern visualization, statistics plotting and parameter tuning. A batch job module is implemented to facilitate large-data-volume processing. A two-step geometry calibration for single-panel detectors is also integrated into the GUI, where the beam center and detector tilting angles are optimized using an ellipse center shifting method first, then all six parameters, including the photon energy and detector distance, are refined together using a residual minimization method. Implemented in Python, ClickX has good portability and extensibility, so that it can be installed, configured and used on any computing platform that provides a Python interface or common data file format. ClickX has been tested in online analysis at the Pohang Accelerator Laboratory X-ray Free-Electron Laser, Korea, and the Linac Coherent Light Source, USA. It has also been applied in post-experimental data analysis. The source code is available via https://github.com/LiuLab-CSRC/ClickX under a GNU General Public License.

Fixed-target serial oscillation crystallography at room temperature

A fixed-target approach to high-throughput room-temperature serial synchrotron crystallography with oscillation is described. Patterned silicon chips with microwells provide high crystal-loading density with an extremely high hit rate. The microfocus, undulator-fed beamline at CHESS, which has compound refractive optics and a fast-framing detector, was built and optimized for this experiment. The high-throughput oscillation method described here collects 1-5° of data per crystal at room temperature with fast (10° s-1) oscillation rates and translation times, giving a crystal-data collection rate of 2.5 Hz. Partial datasets collected by the oscillation method at a storage-ring source provide more complete data per crystal than still images, dramatically lowering the total number of crystals needed for a complete dataset suitable for structure solution and refinement - up to two orders of magnitude fewer being required. Thus, this method is particularly well suited to instances where crystal quantities are low. It is demonstrated, through comparison of first and last oscillation images of two systems, that dose and the effects of radiation damage can be minimized through fast rotation and low angular sweeps for each crystal.

Resolving indexing ambiguities in X-ray free-electron laser diffraction patterns

Processing X-ray free-electron laser (XFEL) diffraction images poses challenges, as an XFEL pulse is powerful enough to destroy or damage the diffracting volume and thereby yields only one diffraction image per volume. Moreover, the crystal is stationary during the femtosecond pulse, so reflections are generally only partially recorded. Therefore, each XFEL diffraction image must be scaled individually and, ideally, corrected for partiality prior to merging. An additional complication may arise owing to indexing ambiguities when the symmetry of the Bravais lattice is higher than that of the space group, or when the unit-cell dimensions are similar to each other. Here, an automated method is presented that diagnoses these indexing ambiguities based on the Brehm-Diederichs algorithm [Brehm & Diederichs (2014), Acta Cryst. D70, 101-109] and produces a consistent indexing choice for the large majority of diffraction images. This method was applied to an XFEL diffraction data set measured from crystals of the neuronal SNARE-complexin-1-synaptotagmin-1 complex. After correcting the indexing ambiguities, substantial improvements were observed in the merging statistics and the atomic model refinement R values. This method should be a useful addition to the arsenal of tools for the processing of XFEL diffraction data sets.

A convolutional neural network-based screening tool for X-ray serial crystallography

A new tool is introduced for screening macromolecular X-ray crystallography diffraction images produced at an X-ray free-electron laser light source. Based on a data-driven deep learning approach, the proposed tool executes a convolutional neural network to detect Bragg spots. Automatic image processing algorithms described can enable the classification of large data sets, acquired under realistic conditions consisting of noisy data with experimental artifacts. Outcomes are compared for different data regimes, including samples from multiple instruments and differing amounts of training data for neural network optimization.

Structural characterization of a highly-potent V3-glycan broadly neutralizing antibody bound to natively-glycosylated HIV-1 envelope

Broadly neutralizing antibodies (bNAbs) isolated from HIV-1-infected individuals inform HIV-1 vaccine design efforts. Developing bNAbs with increased efficacy requires understanding how antibodies interact with the native oligomannose and complex-type N-glycan shield that hides most protein epitopes on HIV-1 envelope (Env). Here we present crystal structures, including a 3.8-Å X-ray free electron laser dataset, of natively glycosylated Env trimers complexed with BG18, the most potent V3/N332gp120 glycan-targeting bNAb reported to date. Our structures show conserved contacts mediated by common D gene-encoded residues with the N332gp120 glycan and the gp120 GDIR peptide motif, but a distinct Env-binding orientation relative to PGT121/10-1074 bNAbs. BG18's binding orientation provides additional contacts with N392gp120 and N386gp120 glycans near the V3-loop base and engages protein components of the V1-loop. The BG18-natively-glycosylated Env structures facilitate understanding of bNAb-glycan interactions critical for using V3/N332gp120 bNAbs therapeutically and targeting their epitope for immunogen design.

Protein crystals IR laser ablated from aqueous solution at high speed retain their diffractive properties: applications in high-speed serial crystallography

In order to utilize the high repetition rates now available at X-ray free-electron laser sources for serial crystallography, methods must be developed to softly deliver large numbers of individual microcrystals at high repetition rates and high speeds. Picosecond infrared laser (PIRL) pulses, operating under desorption by impulsive vibrational excitation (DIVE) conditions, selectively excite the OH vibrational stretch of water to directly propel the excited volume at high speed with minimized heating effects, nucleation formation or cavitation-induced shock waves, leaving the analytes intact and undamaged. The soft nature and laser-based sampling flexibility provided by the technique make the PIRL system an interesting crystal delivery approach for serial crystallography. This paper demonstrates that protein crystals extracted directly from aqueous buffer solutionviaPIRL-DIVE ablation retain their diffractive properties and can be usefully exploited for structure determination at synchrotron sources. The remaining steps to implement the technology for high-speed serial femtosecond crystallography, such as single-crystal localization, high-speed sampling and synchronization, are described. This proof-of-principle experiment demonstrates the viability of a new laser-based high-speed crystal delivery system without the need for liquid-jet injectors or fixed-target mounting solutions.

Data systems for the Linac coherent light source

The data systems for X-ray free-electron laser (FEL) experiments at the Linac coherent light source (LCLS) are described. These systems are designed to acquire and to reliably transport shot-by-shot data at a peak throughput of 5 GB/s to the offline data storage where experimental data and the relevant metadata are archived and made available for user analysis. The analysis and monitoring implementation (AMI) and Photon Science ANAlysis (psana) software packages are described. Psana is open source and freely available.

Advances in X-ray free electron laser (XFEL) diffraction data processing applied to the crystal structure of the synaptotagmin-1 / SNARE complex

X-ray free electron lasers (XFELs) reduce the effects of radiation damage on macromolecular diffraction data and thereby extend the limiting resolution. Previously, we adapted classical post-refinement techniques to XFEL diffraction data to produce accurate diffraction data sets from a limited number of diffraction images (Uervirojnangkoorn et al., 2015), and went on to use these techniques to obtain a complete data set from crystals of the synaptotagmin-1 / SNARE complex and to determine the structure at 3.5 Å resolution (Zhou et al., 2015). Here, we describe new advances in our methods and present a reprocessed XFEL data set of the synaptotagmin-1 / SNARE complex. The reprocessing produced small improvements in electron density maps and the refined atomic model. The maps also contained more information than those of a lower resolution (4.1 Å) synchrotron data set. Processing a set of simulated XFEL diffraction images revealed that our methods yield accurate data and atomic models.

DOI:http://dx.doi.org/10.7554/eLife.18740.001

Acceleratingk-nearest-neighbor searches

The search for whichkpoints are closest to a given probe point in a space ofNknown points, the `k-nearest-neighbor' or `KNN' problem, is a computationally challenging problem of importance in many disciplines, such as the design of numerical databases, analysis of multi-dimensional experimental data sets, multi-particle simulations and data mining. A standard approach is to preprocess the data into a tree and make use of the triangle inequality to prune the search time to the order of the logarithm ofNfor a single nearest point in a well balanced tree. All known approaches suffer from the `curse of dimensionality', which causes the search to explore many more branches of the tree than one might wish as the dimensionality of the problem increases, driving search times closer to the order ofN. Looking forknearest points can sometimes be done in approximately the time needed to search for one nearest point, but more often it requiresksearches because the results are distributed widely. The result is very long search times, especially when the search radius is large andkis large, and individual distance calculations are very expensive, because the same probe-to-data-point distance calculations need to be executed repeatedly as the top of the tree is re-explored. Combining two acceleration techniques was found to improve the search time dramatically: (i) organizing the search into nested searches in non-overlapping annuli of increasing radii, using an estimation of the Hausdorff dimension applicable to this data instance from the results of earlier annuli to help set the radius of the next annulus; and (ii) caching all distance calculations involving the probe point to reduce the cost of repeated use of the same distances. The result of this acceleration in a search of the combined macromolecular and small-molecule data in a combined six-dimensional database of nearly 900 000 entries has been an improvement in the overall time of the searches by one to two orders of magnitude.

CCP-FEL: a collection of computer programs for free-electron laser research

The latest virtual special issue ofJournal of Applied Crystallography(http://journals.iucr.org/special_issues/2016/ccpfel) collects software for free-electron laser research and presents tools for a range of topics such as simulation of experiments, online monitoring of data collection, selection of hits, diagnostics of data quality, data management, data analysis and structure determination for both nanocrystallography and single-particle diffractive imaging. This article provides an introduction to the special issue.

Data systems for the Linac Coherent Light Source

The data acquisition and data management systems for X-ray free-electron laser experiments at the Linac Coherent Light Source are described. These systems are designed to acquire and to reliably transport shot-by-shot data at a peak throughput of 5 GB s−1to the offline data storage, where experimental data and the relevant metadata are archived and made available for user analysis. A case study of a serial femtosecond crystallography pipeline is presented.