JUNGFRAU detector for brighter x-ray sources: Solutions for IT and data science challenges in macromolecular crystallography

Quantifying bunch-mode influence on photon-counting detectors at SPring-8

Count-loss characteristics of photon-counting 2D detectors are demonstrated for eight bunch-modes at SPring-8 through Monte Carlo simulations. As an indicator, the effective maximum count rate was introduced to signify the X-ray intensity that the detector can count with a linearity of 1% or better after applying a count-loss correction in each bunch-mode. The effective maximum count rate is revealed to vary depending on the bunch-mode and the intrinsic dead time of the detectors, ranging from 0.012 to 0.916 Mcps (megacounts per second) for a 120 ns dead time, 0.009 to 0.807 Mcps for a 0.5 µs dead time and 0.020 to 0.273 Mcps for a 3 µs intrinsic detector dead time. Even with equal-interval bunch-modes at SPring-8, the effective maximum count rate does not exceed 1 Mcps pixel-1. In other words, to obtain data with a linearity better than 1%, the maximum intensity of X-rays entering the detector should be reduced to 1 Mcps pixel-1 or less, and, in some cases, even lower, depending on the bunch-mode. When applying count-loss correction using optimized dead times tailored to each bunch-mode, the effective maximum count rate exceeds the values above. However, differences in the effective maximum count rate due to bunch-modes persist. Users of photon-counting 2D detectors are encouraged to familiarize themselves with the count-loss characteristics dependent on bunch-mode, and to conduct experiments accordingly. In addition, when designing the time structure of bunch-modes at synchrotron radiation facilities, it is essential to take into account the impact on experiments using photon-counting 2D detectors.

Data reduction in protein serial crystallography

Serial crystallography (SX) has become an established technique for protein structure determination, especially when dealing with small or radiation-sensitive crystals and investigating fast or irreversible protein dynamics. The advent of newly developed multi-megapixel X-ray area detectors, capable of capturing over 1000 images per second, has brought about substantial benefits. However, this advancement also entails a notable increase in the volume of collected data. Today, up to 2 PB of data per experiment could be easily obtained under efficient operating conditions. The combined costs associated with storing data from multiple experiments provide a compelling incentive to develop strategies that effectively reduce the amount of data stored on disk while maintaining the quality of scientific outcomes. Lossless data-compression methods are designed to preserve the information content of the data but often struggle to achieve a high compression ratio when applied to experimental data that contain noise. Conversely, lossy compression methods offer the potential to greatly reduce the data volume. Nonetheless, it is vital to thoroughly assess the impact of data quality and scientific outcomes when employing lossy compression, as it inherently involves discarding information. The evaluation of lossy compression effects on data requires proper data quality metrics. In our research, we assess various approaches for both lossless and lossy compression techniques applied to SX data, and equally importantly, we describe metrics suitable for evaluating SX data quality.

A setup for hard x-ray time-resolved resonant inelastic x-ray scattering at SwissFEL

We present a new setup for resonant inelastic hard x-ray scattering at the Bernina beamline of SwissFEL with energy, momentum, and temporal resolution. The compact R = 0.5 m Johann-type spectrometer can be equipped with up to three crystal analyzers and allows efficient collection of RIXS spectra. Optical pumping for time-resolved studies can be realized with a broad span of optical wavelengths. We demonstrate the performance of the setup at an overall ∼180 meV resolution in a study of ground-state and photoexcited (at 400 nm) honeycomb 5d iridate α-Li2IrO3. Steady-state RIXS spectra at the iridium L3-edge (11.214 keV) have been collected and are in very good agreement with data collected at synchrotrons. The time-resolved RIXS transients exhibit changes in the energy loss region <2 eV, whose features mostly result from the hopping nature of 5d electrons in the honeycomb lattice. These changes are ascribed to modulations of the Ir-to-Ir inter-site transition scattering efficiency, which we associate to a transient screening of the on-site Coulomb interaction.

Deep learning at the edge enables real-time streaming ptychographic imaging

Coherent imaging techniques provide an unparalleled multi-scale view of materials across scientific and technological fields, from structural materials to quantum devices, from integrated circuits to biological cells. Driven by the construction of brighter sources and high-rate detectors, coherent imaging methods like ptychography are poised to revolutionize nanoscale materials characterization. However, these advancements are accompanied by significant increase in data and compute needs, which precludes real-time imaging, feedback and decision-making capabilities with conventional approaches. Here, we demonstrate a workflow that leverages artificial intelligence at the edge and high-performance computing to enable real-time inversion on X-ray ptychography data streamed directly from a detector at up to 2 kHz. The proposed AI-enabled workflow eliminates the oversampling constraints, allowing low-dose imaging using orders of magnitude less data than required by traditional methods.

Kilohertz serial crystallography with the JUNGFRAU detector at a fourth-generation synchrotron source

Serial and time-resolved macromolecular crystallography are on the rise. However, beam time at X-ray free-electron lasers is limited and most third-generation synchrotron-based macromolecular crystallography beamlines do not offer the necessary infrastructure yet. Here, a new setup is demonstrated, based on the JUNGFRAU detector and Jungfraujoch data-acquisition system, that enables collection of kilohertz serial crystallography data at fourth-generation synchrotrons. More importantly, it is shown that this setup is capable of collecting multiple-time-point time-resolved protein dynamics at kilohertz rates, allowing the probing of microsecond to second dynamics at synchrotrons in a fraction of the time needed previously. A high-quality complete X-ray dataset was obtained within 1 min from lysozyme microcrystals, and the dynamics of the light-driven sodium-pump membrane protein KR2 with a time resolution of 1 ms could be demonstrated. To make the setup more accessible for researchers, downstream data handling and analysis will be automated to allow on-the-fly spot finding and indexing, as well as data processing.

Anaerobic fixed-target serial crystallography using sandwiched silicon nitride membranes

In recent years, the emergence of serial crystallography, initially pioneered at X-ray free-electron lasers (XFELs), has sparked a growing interest in collecting macromolecular crystallographic data at room temperature. Various fixed-target serial crystallography techniques have been developed, ranging from commercially available chips to in-house designs implemented at different synchrotron facilities. Nevertheless, there is currently no commercially available chip (known to the authors) specifically designed for the direct handling of oxygen-sensitive samples. This study presents a methodology employing silicon nitride chips arranged in a `sandwich' configuration, enabling reliable room-temperature data collection from oxygen-sensitive samples. The method involves the utilization of a custom-made 3D-printed assembling tool and a MX sample holder. To validate the effectiveness of the proposed method, deoxyhemoglobin and methemoglobin samples were investigated using the BioMAX X-ray macromolecular crystallography beamline, the Balder X-ray absorption spectroscopy beamline and UV-Vis absorption spectroscopy.

RNA structures and dynamics with Å resolution revealed by x-ray free-electron lasers

RNA macromolecules, like proteins, fold to assume shapes that are intimately connected to their broadly recognized biological functions; however, because of their high charge and dynamic nature, RNA structures are far more challenging to determine. We introduce an approach that exploits the high brilliance of x-ray free-electron laser sources to reveal the formation and ready identification of angstrom-scale features in structured and unstructured RNAs. Previously unrecognized structural signatures of RNA secondary and tertiary structures are identified through wide-angle solution scattering experiments. With millisecond time resolution, we observe an RNA fold from a dynamically varying single strand through a base-paired intermediate to assume a triple-helix conformation. While the backbone orchestrates the folding, the final structure is locked in by base stacking. This method may help to rapidly characterize and identify structural elements in nucleic acids in both equilibrium and time-resolved experiments.

RNA structures and dynamics with Å resolution revealed by x-ray free electron lasers

RNA macromolecules, like proteins, fold to assume shapes that are intimately connected to their broadly recognized biological functions; however, because of their high charge and dynamic nature, RNA structures are far more challenging to determine. We introduce an approach that exploits the high brilliance of x-ray free electron laser sources to reveal the formation and ready identification of Å scale features in structured and unstructured RNAs. New structural signatures of RNA secondary and tertiary structures are identified through wide angle solution scattering experiments. With millisecond time resolution, we observe an RNA fold from a dynamically varying single strand through a base paired intermediate to assume a triple helix conformation. While the backbone orchestrates the folding, the final structure is locked in by base stacking. In addition to understanding how RNA triplexes form and thereby function as dynamic signaling elements, this new method can vastly increase the rate of structure determination for these biologically essential, but mostly uncharacterized macromolecules.

Data reduction for X-ray serial crystallography using machine learning

Serial crystallography experiments produce massive amounts of experimental data. Yet in spite of these large-scale data sets, only a small percentage of the data are useful for downstream analysis. Thus, it is essential to differentiate reliably between acceptable data (hits) and unacceptable data (misses). To this end, a novel pipeline is proposed to categorize the data, which extracts features from the images, summarizes these features with the 'bag of visual words' method and then classifies the images using machine learning. In addition, a novel study of various feature extractors and machine learning classifiers is presented, with the aim of finding the best feature extractor and machine learning classifier for serial crystallography data. The study reveals that the oriented FAST and rotated BRIEF (ORB) feature extractor with a multilayer perceptron classifier gives the best results. Finally, the ORB feature extractor with multilayer perceptron is evaluated on various data sets including both synthetic and experimental data, demonstrating superior performance compared with other feature extractors and classifiers.

Jungfraujoch: hardware-accelerated data-acquisition system for kilohertz pixel-array X-ray detectors

The JUNGFRAU 4-megapixel (4M) charge-integrating pixel-array detector, when operated at a full 2 kHz frame rate, streams data at a rate of 17 GB s^-1. To operate this detector for macromolecular crystallography beamlines, a data-acquisition system called Jungfraujoch was developed. The system, running on a single server with field-programmable gate arrays and general-purpose graphics processing units, is capable of handling data produced by the JUNGFRAU 4M detector, including conversion of raw pixel readout to photon counts, compression and on-the-fly spot finding. It was also demonstrated that 30 GB s^-1 can be handled in performance tests, indicating that the operation of even larger and faster detectors will be achievable in the future. The source code is available from a public repository.

Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography

X-ray crystallography has witnessed a massive development over the past decade, driven by large increases in the intensity and brightness of X-ray sources and enabled by employing high-frame-rate X-ray detectors. The analysis of large data sets is done via automatic algorithms that are vulnerable to imperfections in the detector and noise inherent with the detection process. By improving the model of the behaviour of the detector, data can be analysed more reliably and data storage costs can be significantly reduced. One major requirement is a software mask that identifies defective pixels in diffraction frames. This paper introduces a methodology and program based upon concepts of machine learning, called robust mask maker (RMM), for the generation of bad-pixel masks for large-area X-ray pixel detectors based on modern robust statistics. It is proposed to discriminate normally behaving pixels from abnormal pixels by analysing routine measurements made with and without X-ray illumination. Analysis software typically uses a Bragg peak finder to detect Bragg peaks and an indexing method to detect crystal lattices among those peaks. Without proper masking of the bad pixels, peak finding methods often confuse the abnormal values of bad pixels in a pattern with true Bragg peaks and flag such patterns as useful regardless, leading to storage of enormous uninformative data sets. Also, it is computationally very expensive for indexing methods to search for crystal lattices among false peaks and the solution may be biased. This paper shows how RMM vastly improves peak finders and prevents them from labelling bad pixels as Bragg peaks, by demonstrating its effectiveness on several serial crystallography data sets.

Evaluation of HPC Acceleration and Interconnect Technologies for High-Throughput Data Acquisition

Efficient data movement in multi-node systems is a crucial issue at the crossroads of scientific computing, big data, and high-performance computing, impacting demanding data acquisition applications from high-energy physics to astronomy, where dedicated accelerators such as FPGA devices play a key role coupled with high-performance interconnect technologies. Building on the outcome of the RECIPE Horizon 2020 research project, this work evaluates the use of high-bandwidth interconnect standards, namely InfiniBand EDR and HDR, along with remote direct memory access functions for direct exposure of FPGA accelerator memory across a multi-node system. The prototype we present aims at avoiding dedicated network interfaces built in the FPGA accelerator itself, leaving most of the resources for user acceleration and supporting state-of-the-art interconnect technologies. We present the detail of the proposed system and a quantitative evaluation in terms of end-to-end bandwidth as concretely measured with a real-world FPGA-based multi-node HPC workload.

Dynamic Structural Biology Experiments at XFEL or Synchrotron Sources

Macromolecular crystallography (MX) leverages the methods of physics and the language of chemistry to reveal fundamental insights into biology. Often beautifully artistic images present MX results to support profound functional hypotheses that are vital to entire life science research community. Over the past several decades, synchrotrons around the world have been the workhorses for X-ray diffraction data collection at many highly automated beamlines. The newest tools include X-ray-free electron lasers (XFELs) located at facilities in the USA, Japan, Korea, Switzerland, and Germany that deliver about nine orders of magnitude higher brightness in discrete femtosecond long pulses. At each of these facilities, new serial femtosecond crystallography (SFX) strategies exploit slurries of micron-size crystals by rapidly delivering individual crystals into the XFEL X-ray interaction region, from which one diffraction pattern is collected per crystal before it is destroyed by the intense X-ray pulse. Relatively simple adaptions to SFX methods produce time-resolved data collection strategies wherein reactions are triggered by visible light illumination or by chemical diffusion/mixing. Thus, XFELs provide new opportunities for high temporal and spatial resolution studies of systems engaged in function at physiological temperature. In this chapter, we summarize various issues related to microcrystal slurry preparation, sample delivery into the X-ray interaction region, and some emerging strategies for time-resolved SFX data collection.

RDMA data transfer and GPU acceleration methods for high-throughput online processing of serial crystallography images

The continual evolution of photon sources and high-performance detectors drives cutting-edge experiments that can produce very high throughput data streams and generate large data volumes that are challenging to manage and store. In these cases, efficient data transfer and processing architectures that allow online image correction, data reduction or compression become fundamental. This work investigates different technical options and methods for data placement from the detector head to the processing computing infrastructure, taking into account the particularities of modern modular high-performance detectors. In order to compare realistic figures, the future ESRF beamline dedicated to macromolecular X-ray crystallography, EBSL8, is taken as an example, which will use a PSI JUNGFRAU 4M detector generating up to 16 GB of data per second, operating continuously during several minutes. Although such an experiment seems possible at the target speed with the 100 Gb s^-1 network cards that are currently available, the simulations generated highlight some potential bottlenecks when using a traditional software stack. An evaluation of solutions is presented that implements remote direct memory access (RDMA) over converged ethernet techniques. A synchronization mechanism is proposed between a RDMA network interface card (RNIC) and a graphics processing unit (GPU) accelerator in charge of the online data processing. The placement of the detector images onto the GPU is made to overlap with the computation carried out, potentially hiding the transfer latencies. As a proof of concept, a detector simulator and a backend GPU receiver with a rejection and compression algorithm suitable for a synchrotron serial crystallography (SSX) experiment are developed. It is concluded that the available transfer throughput from the RNIC to the GPU accelerator is at present the major bottleneck in online processing for SSX experiments.

Gold Standard for macromolecular crystallography diffraction data

Macromolecular crystallography (MX) is the dominant means of determining the three-dimensional structures of biological macromolecules. Over the last few decades, most MX data have been collected at synchrotron beamlines using a large number of different detectors produced by various manufacturers and taking advantage of various protocols and goniometries. These data came in their own formats: sometimes proprietary, sometimes open. The associated metadata rarely reached the degree of completeness required for data management according to Findability, Accessibility, Interoperability and Reusability (FAIR) principles. Efforts to reuse old data by other investigators or even by the original investigators some time later were often frustrated. In the culmination of an effort dating back more than two decades, a large portion of the research community concerned with high data-rate macromolecular crystallography (HDRMX) has now agreed to an updated specification of data and metadata for diffraction images produced at synchrotron light sources and X-ray free-electron lasers (XFELs). This 'Gold Standard' will facilitate the processing of data sets independent of the facility at which they were collected and enable data archiving according to FAIR principles, with a particular focus on interoperability and reusability. This agreed standard builds on the NeXus/HDF5 NXmx application definition and the International Union of Crystallo-graphy (IUCr) imgCIF/CBF dictionary, and it is compatible with major data-processing programs and pipelines. Just as with the IUCr CBF/imgCIF standard from which it arose and to which it is tied, the NeXus/HDF5 NXmx Gold Standard application definition is intended to be applicable to all detectors used for crystallography, and all hardware and software developers in the field are encouraged to adopt and contribute to the standard.