Low cost, high performance processing of single particle cryo-electron microscopy data in the cloud

GoToCloud optimization of cloud computing environment for accelerating cryo-EM structure-based drug design

Cryogenic electron microscopy (Cryo-EM) is a widely used technique for visualizing the 3D structures of many drug design targets, including membrane proteins, at atomic resolution. However, the necessary throughput for structure-based drug design (SBDD) is not yet achieved. Currently, data analysis is a major bottleneck due to the rapid advancements in detector technology and image acquisition methods. Here we show "GoToCloud", a cloud-computing-based platform for advanced data analysis and data management in Cryo-EM. With GoToCloud, it is possible to optimize computing resources and reduce costs by selecting the most appropriate parallel processing settings for each processing step. Our benchmark tests on GoToCloud demonstrate that parallel computing settings, including the choice of computational hardware, as well as a required target resolution have significant impacts on the processing time and cost performance. Through this optimization of a cloud computing environment, GoToCloud emerges as a promising platform for the acceleration of Cryo-EM SBDD.

Big data in contemporary electron microscopy: challenges and opportunities in data transfer, compute and management

The second decade of the twenty-first century witnessed a new challenge in the handling of microscopy data. Big data, data deluge, large data, data compliance, data analytics, data integrity, data interoperability, data retention and data lifecycle are terms that have introduced themselves to the electron microscopy sciences. This is largely attributed to the booming development of new microscopy hardware tools. As a result, large digital image files with an average size of one terabyte within one single acquisition session is not uncommon nowadays, especially in the field of cryogenic electron microscopy. This brings along numerous challenges in data transfer, compute and management. In this review, we will discuss in detail the current state of international knowledge on big data in contemporary electron microscopy and how big data can be transferred, computed and managed efficiently and sustainably. Workflows, solutions, approaches and suggestions will be provided, with the example of the latest experiences in Australia. Finally, important principles such as data integrity, data lifetime and the FAIR and CARE principles will be considered.

Scipion3: A workflow engine for cryo-electron microscopy image processing and structural biology

Image-processing pipelines require the design of complex workflows combining many different steps that bring the raw acquired data to a final result with biological meaning. In the image-processing domain of cryo-electron microscopy single-particle analysis (cryo-EM SPA), hundreds of steps must be performed to obtain the three-dimensional structure of a biological macromolecule by integrating data spread over thousands of micrographs containing millions of copies of allegedly the same macromolecule. The execution of such complicated workflows demands a specific tool to keep track of all these steps performed. Additionally, due to the extremely low signal-to-noise ratio (SNR), the estimation of any image parameter is heavily affected by noise resulting in a significant fraction of incorrect estimates. Although low SNR and processing millions of images by hundreds of sequential steps requiring substantial computational resources are specific to cryo-EM, these characteristics may be shared by other biological imaging domains. Here, we present Scipion, a Python generic open-source workflow engine specifically adapted for image processing. Its main characteristics are: (a) interoperability, (b) smart object model, (c) gluing operations, (d) comparison operations, (e) wide set of domain-specific operations, (f) execution in streaming, (g) smooth integration in high-performance computing environments, (h) execution with and without graphical capabilities, (i) flexible visualization, (j) user authentication and private access to private data, (k) scripting capabilities, (l) high performance, (m) traceability, (n) reproducibility, (o) self-reporting, (p) reusability, (q) extensibility, (r) software updates, and (s) non-restrictive software licensing.

Machine Learning for Structure Determination in Single-Particle Cryo-Electron Microscopy: A Systematic Review

Recently, single-particle cryo-electron microscopy (cryo-EM) has become an indispensable method for determining macromolecular structures at high resolution to deeply explore the relevant molecular mechanism. Its recent breakthrough is mainly because of the rapid advances in hardware and image processing algorithms, especially machine learning. As an essential support of single-particle cryo-EM, machine learning has powered many aspects of structure determination and greatly promoted its development. In this article, we provide a systematic review of the applications of machine learning in this field. Our review begins with a brief introduction of single-particle cryo-EM, followed by the specific tasks and challenges of its image processing. Then, focusing on the workflow of structure determination, we describe relevant machine learning algorithms and applications at different steps, including particle picking, 2-D clustering, 3-D reconstruction, and other steps. As different tasks exhibit distinct characteristics, we introduce the evaluation metrics for each task and summarize their dynamics of technology development. Finally, we discuss the open issues and potential trends in this promising field.

Cloud computing platforms to support cryo-EM structure determination

Leveraging the power of single-particle cryo-electron microscopy (cryo-EM) requires robust and accessible computational infrastructure. Here, we summarize the cloud computing landscape and picture the outlook of a hybrid cryo-EM computing workflow, and make suggestions to the community to facilitate a future for cryo-EM that integrates into cloud computing infrastructure.

High-Throughput Cryo-EM Enabled by User-Free Preprocessing Routines

Single-particle cryoelectron microscopy (cryo-EM) continues to grow into a mainstream structural biology technique. Recent developments in data collection strategies alongside new sample preparation devices herald a future where users will collect multiple datasets per microscope session. To make cryo-EM data processing more automatic and user-friendly, we have developed an automatic pipeline for cryo-EM data preprocessing and assessment using a combination of deep-learning and image-analysis tools. We have verified the performance of this pipeline on a number of datasets and extended its scope to include sample screening by the user-free assessment of the qualities of a series of datasets under different conditions. We propose that our workflow provides a decision-free solution for cryo-EM, making data preprocessing more generalized and robust in the high-throughput era as well as more convenient for users from a range of backgrounds.

Separating distinct structures of multiple macromolecular assemblies from cryo-EM projections

Single particle analysis for structure determination in cryo-electron microscopy is traditionally applied to samples purified to near homogeneity as current reconstruction algorithms are not designed to handle heterogeneous mixtures of structures from many distinct macromolecular complexes. We extend on long established methods and demonstrate that relating two-dimensional projection images by their common lines in a graphical framework is sufficient for partitioning distinct protein and multiprotein complexes within the same data set. The feasibility of this approach is first demonstrated on a large set of synthetic reprojections from 35 unique macromolecular structures spanning a mass range of hundreds to thousands of kilodaltons. We then apply our algorithm on cryo-EM data collected from a mixture of five protein complexes and use existing methods to solve multiple three-dimensional structures ab initio. Incorporating methods to sort single particle cryo-EM data from extremely heterogeneous mixtures will alleviate the need for stringent purification and pave the way toward investigation of samples containing many unique structures.

Direct visualization of degradation microcompartments at the ER membrane

Endoplasmic reticulum-associated degradation (ERAD) is an essential process that removes misfolded proteins from the ER, preventing cellular dysfunction and disease. While most of the key components of ERAD are known, their specific localization remains a mystery. This study uses in situ cryo-electron tomography to directly visualize the ERAD machinery within the native cellular environment. Proteasomes and Cdc48, the complexes that extract and degrade ER proteins, cluster together in non–membrane-bound cytosolic microcompartments that contact ribosome-free patches on the ER membrane. This discrete molecular organization may facilitate efficient ERAD. Structural analysis reveals that proteasomes directly engage ER-localized substrates, providing evidence for a noncanonical “direct ERAD” pathway. In addition, live-cell fluorescence microscopy suggests that these ER-associated proteasome clusters form by liquid–liquid phase separation.

To promote the biochemical reactions of life, cells can compartmentalize molecular interaction partners together within separated non–membrane-bound regions. It is unknown whether this strategy is used to facilitate protein degradation at specific locations within the cell. Leveraging in situ cryo-electron tomography to image the native molecular landscape of the unicellular alga Chlamydomonas reinhardtii, we discovered that the cytosolic protein degradation machinery is concentrated within ∼200-nm foci that contact specialized patches of endoplasmic reticulum (ER) membrane away from the ER–Golgi interface. These non–membrane-bound microcompartments exclude ribosomes and consist of a core of densely clustered 26S proteasomes surrounded by a loose cloud of Cdc48. Active proteasomes in the microcompartments directly engage with putative substrate at the ER membrane, a function canonically assigned to Cdc48. Live-cell fluorescence microscopy revealed that the proteasome clusters are dynamic, with frequent assembly and fusion events. We propose that the microcompartments perform ER-associated degradation, colocalizing the degradation machinery at specific ER hot spots to enable efficient protein quality control.

3.1 Å structure of yeast RNA polymerase II elongation complex stalled at a cyclobutane pyrimidine dimer lesion solved using streptavidin affinity grids

Despite significant advances in all aspects of single particle cryo-electron microscopy (cryo-EM), specimen preparation still remains a challenge. During sample preparation, macromolecules interact with the air-water interface, which often leads to detrimental effects such as denaturation or adoption of preferred orientations, ultimately hindering structure determination. Randomly biotinylating the protein of interest (for example, at its primary amines) and then tethering it to a cryo-EM grid coated with two-dimensional crystals of streptavidin (acting as an affinity surface) can prevent the protein from interacting with the air-water interface. Recently, this approach was successfully used to solve a high-resolution structure of a test sample, a bacterial ribosome. However, whether this method can be used for samples where interaction with the air-water interface has been shown to be problematic remains to be determined. Here we report a 3.1 Å structure of an RNA polymerase II elongation complex stalled at a cyclobutane pyrimidine dimer lesion (Pol II EC(CPD)) solved using streptavidin grids. Our previous attempt to solve this structure using conventional sample preparation methods resulted in a poor quality cryo-EM map due to Pol II EC(CPD)'s adopting a strong preferred orientation. Imaging the same sample on streptavidin grids improved the angular distribution of its view, resulting in a high-resolution structure. This work shows that streptavidin affinity grids can be used to address known challenges posed by the interaction with the air-water interface.

The Architecture of Traveling Actin Waves Revealed by Cryo-Electron Tomography

Actin waves are dynamic supramolecular structures involved in cell migration, cytokinesis, adhesion, and neurogenesis. Although wave-like propagation of actin networks is a widespread phenomenon, the actin architecture underlying wave propagation remained unknown. In situ cryo-electron tomography of Dictyostelium cells unveils the wave architecture and provides evidence for wave progression by de novo actin nucleation. Subtomogram averaging reveals the structure of Arp2/3 complex-mediated branch junctions in their native state, and enables quantitative analysis of the 3D organization of branching within the waves. We find an excess of branches directed toward the substrate-attached membrane, and tent-like structures at sites of branch clustering. Fluorescence imaging shows that Arp2/3 clusters follow accumulation of the elongation factor VASP. We propose that filament growth toward the membrane lifts up the actin network as the wave propagates, until depolymerization of oblique filaments at the back causes the collapse of horizontal filaments into a compact layer.

Electron microscopy snapshots of single particles from single cells

Cryo-electron microscopy (cryo-EM) has become an indispensable tool for structural studies of biological macromolecules. Two additional predominant methods are available for studying the architectures of multiprotein complexes: 1) single-particle analysis of purified samples and 2) tomography of whole cells or cell sections. The former can produce high-resolution structures but is limited to highly purified samples, whereas the latter can capture proteins in their native state but has a low signal-to-noise ratio and yields lower-resolution structures. Here, we present a simple, adaptable method combining microfluidic single-cell extraction with single-particle analysis by EM to characterize protein complexes from individual Caenorhabditis elegans embryos. Using this approach, we uncover 3D structures of ribosomes directly from single embryo extracts. Moreover, we investigated structural dynamics during development by counting the number of ribosomes per polysome in early and late embryos. This approach has significant potential applications for counting protein complexes and studying protein architectures from single cells in developmental, evolutionary, and disease contexts.

cryoem-cloud-tools: A software platform to deploy and manage cryo-EM jobs in the cloud

Access to streamlined computational resources remains a significant bottleneck for new users of cryo-electron microscopy (cryo-EM). To address this, we have developed tools that will submit cryo-EM analysis routines and atomic model building jobs directly to Amazon Web Services (AWS) from a local computer or laptop. These new software tools ("cryoem-cloud-tools") have incorporated optimal data movement, security, and cost-saving strategies, giving novice users access to complex cryo-EM data processing pipelines. Integrating these tools into the RELION processing pipeline and graphical user interface we determined a 2.2 Å structure of ß-galactosidase in ∼55 h on AWS. We implemented a similar strategy to submit Rosetta atomic model building and refinement to AWS. These software tools dramatically reduce the barrier for entry of new users to cloud computing for cryo-EM and are freely available at cryoem-tools.cloud.

Structural basis for the initiation of eukaryotic transcription-coupled DNA repair

Eukaryotic transcription-coupled repair (TCR), or transcription-coupled nucleotide excision repair (TC-NER), is an important and well-conserved sub-pathway of nucleotide excision repair (NER) that preferentially removes DNA lesions from the template strand blocking RNA polymerase II (Pol II) translocation^1,2. Cockayne syndrome group B protein in humans (CSB, or ERCC6), or its yeast orthologs (Rad26 in Saccharomyces cerevisiae and Rhp26 in Schizosaccharomyces pombe), is among the first proteins to be recruited to the lesion-arrested Pol II during initiation of eukaryotic TCR^1,3–10. Mutations in CSB are associated with Cockayne syndrome, an autosomal-recessive neurologic disorder characterized by progeriod features, growth failure, and photosensitivity¹. The molecular mechanism of eukaryotic TCR initiation remains elusive, with several long-standing questions unanswered: How do cells distinguish DNA lesion-arrested Pol II from other forms of arrested Pol II? How does CSB interact with the arrested Pol II complex? What is the role of CSB in TCR initiation? The lack of structures of CSB or the Pol II-CSB complex have hindered our ability to answer those questions. Here we report the first structure of S. cerevisiae Pol II-Rad26 complex solved by cryo-electron microscopy (cryo-EM). The structure reveals that Rad26 binds to the DNA upstream of Pol II where it dramatically alters its path. Our structural and functional data suggest that the conserved Swi2/Snf2-family core ATPase domain promotes forward movement of Pol II and elucidate key roles for Rad26/CSB in both TCR and transcription elongation.

Big data in cryoEM: automated collection, processing and accessibility of EM data

The scope and complexity of cryogenic electron microscopy (cryoEM) data has greatly increased, and will continue to do so, due to recent and ongoing technical breakthroughs that have led to much improved resolutions for macromolecular structures solved using this method. This big data explosion includes single particle data as well as tomographic tilt series, both generally acquired as direct detector movies of ∼10-100 frames per image or per tilt-series. We provide a brief survey of the developments leading to the current status, and describe existing cryoEM pipelines, with an emphasis on the scope of data acquisition, methods for automation, and use of cloud storage and computing.

Lis1 Has Two Opposing Modes of Regulating Cytoplasmic Dynein

Regulation is central to the functional versatility of cytoplasmic dynein, a motor involved in intracellular transport, cell division, and neurodevelopment. Previous work established that Lis1, a conserved regulator of dynein, binds to its motor domain and induces a tight microtubule-binding state in dynein. The work we present here-a combination of biochemistry, single-molecule assays, and cryoelectron microscopy-led to the surprising discovery that Lis1 has two opposing modes of regulating dynein, being capable of inducing both low and high affinity for the microtubule. We show that these opposing modes depend on the stoichiometry of Lis1 binding to dynein and that this stoichiometry is regulated by the nucleotide state of dynein's AAA3 domain. The low-affinity state requires Lis1 to also bind to dynein at a novel conserved site, mutation of which disrupts Lis1's function in vivo. We propose a new model for the regulation of dynein by Lis1.

Best practices for managing large CryoEM facilities

This paper provides an overview of the discussion and presentations from the Workshop on the Management of Large CryoEM Facilities held at the New York Structural Biology Center, New York, NY on February 6-7, 2017. A major objective of the workshop was to discuss best practices for managing cryoEM facilities. The discussions were largely focused on supporting single-particle methods for cryoEM and topics included: user access, assessing projects, workflow, sample handling, microscopy, data management and processing, and user training.

TomoMiner and TomoMinerCloud: A Software Platform for Large-Scale Subtomogram Structural Analysis

Cryo-electron tomography (cryo-ET) captures the 3D electron density distribution of macromolecular complexes in close to native state. With the rapid advance of cryo-ET acquisition technologies, it is possible to generate large numbers (>100,000) of subtomograms, each containing a macromolecular complex. Often, these subtomograms represent a heterogeneous sample due to variations in the structure and composition of a complex in situ form or because particles are a mixture of different complexes. In this case subtomograms must be classified. However, classification of large numbers of subtomograms is a time-intensive task and often a limiting bottleneck. This paper introduces an open source software platform, TomoMiner, for large-scale subtomogram classification, template matching, subtomogram averaging, and alignment. Its scalable and robust parallel processing allows efficient classification of tens to hundreds of thousands of subtomograms. In addition, TomoMiner provides a pre-configured TomoMinerCloud computing service permitting users without sufficient computing resources instant access to TomoMiners high-performance features.

Progress and Potential of Electron Cryotomography as Illustrated by Its Application to Bacterial Chemoreceptor Arrays

Electron cryotomography (ECT) can produce three-dimensional images of biological samples such as intact cells in a near-native, frozen-hydrated state to macromolecular resolution (∼4 nm). Because one of its first and most common applications has been to bacterial chemoreceptor arrays, ECT's contributions to this field illustrate well its past, present, and future. While X-ray crystallography and nuclear magnetic resonance spectroscopy have revealed the structures of nearly all the individual components of chemoreceptor arrays, ECT has revealed the mesoscale information about how the components are arranged within cells. Receptors assemble into a universally conserved 12-nm hexagonal lattice linked by CheA/CheW rings. Membrane-bound arrays are single layered; cytoplasmic arrays are double layered. Images of in vitro reconstitutions have led to a model of how arrays assemble, and images of native arrays in different states have shown that the conformational changes associated with signal transduction are subtle, constraining models of activation and system cooperativity. Phase plates, better detectors, and more stable stages promise even higher resolution and broader application in the near future.

cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination

Single-particle electron cryomicroscopy (cryo-EM) is a powerful method for determining the structures of biological macromolecules. With automated microscopes, cryo-EM data can often be obtained in a few days. However, processing cryo-EM image data to reveal heterogeneity in the protein structure and to refine 3D maps to high resolution frequently becomes a severe bottleneck, requiring expert intervention, prior structural knowledge, and weeks of calculations on expensive computer clusters. Here we show that stochastic gradient descent (SGD) and branch-and-bound maximum likelihood optimization algorithms permit the major steps in cryo-EM structure determination to be performed in hours or minutes on an inexpensive desktop computer. Furthermore, SGD with Bayesian marginalization allows ab initio 3D classification, enabling automated analysis and discovery of unexpected structures without bias from a reference map. These algorithms are combined in a user-friendly computer program named cryoSPARC (http://www.cryosparc.com).

Structural Study of Heterogeneous Biological Samples by Cryoelectron Microscopy and Image Processing

In living organisms, biological macromolecules are intrinsically flexible and naturally exist in multiple conformations. Modern electron microscopy, especially at liquid nitrogen temperatures (cryo-EM), is able to visualise biocomplexes in nearly native conditions and in multiple conformational states. The advances made during the last decade in electronic technology and software development have led to the revelation of structural variations in complexes and also improved the resolution of EM structures. Nowadays, structural studies based on single particle analysis (SPA) suggests several approaches for the separation of different conformational states and therefore disclosure of the mechanisms for functioning of complexes. The task of resolving different states requires the examination of large datasets, sophisticated programs, and significant computing power. Some methods are based on analysis of two-dimensional images, while others are based on three-dimensional studies. In this review, we describe the basic principles implemented in the various techniques that are currently used in the analysis of structural conformations and provide some examples of successful applications of these methods in structural studies of biologically significant complexes.

Unravelling biological macromolecules with cryo-electron microscopy

Knowledge of the three-dimensional structures of proteins and other biological macromolecules often aids understanding of how they perform complicated tasks in the cell. Because many such tasks involve the cleavage or formation of chemical bonds, structural characterization at the atomic level is most useful. Developments in the electron microscopy of frozen hydrated samples (cryo-electron microscopy) are providing unprecedented opportunities for the structural characterization of biological macromolecules. This is resulting in a wave of information about processes in the cell that were impossible to characterize with existing techniques in structural biology.

Molecular dynamics-based refinement and validation for sub-5 Å cryo-electron microscopy maps

Two structure determination methods, based on the molecular dynamics flexible fitting (MDFF) paradigm, are presented that resolve sub-5 Å cryo-electron microscopy (EM) maps with either single structures or ensembles of such structures. The methods, denoted cascade MDFF and resolution exchange MDFF, sequentially re-refine a search model against a series of maps of progressively higher resolutions, which ends with the original experimental resolution. Application of sequential re-refinement enables MDFF to achieve a radius of convergence of ~25 Å demonstrated with the accurate modeling of β-galactosidase and TRPV1 proteins at 3.2 Å and 3.4 Å resolution, respectively. The MDFF refinements uniquely offer map-model validation and B-factor determination criteria based on the inherent dynamics of the macromolecules studied, captured by means of local root mean square fluctuations. The MDFF tools described are available to researchers through an easy-to-use and cost-effective cloud computing resource on Amazon Web Services.

DOI:http://dx.doi.org/10.7554/eLife.16105.001

To understand the roles that proteins and other large molecules play inside cells, it is important to determine their structures. One of the techniques that researchers can use to do this is called cryo-electron microscopy (cryo-EM), which rapidly freezes molecules to fix them in position before imaging them in fine detail.

The cryo-EM images are like maps that show the approximate position of atoms. These images must then be processed in order to build a three-dimensional model of the protein that shows how its atoms are arranged relative to each other. One computational approach called Molecular Dynamics Flexible Fitting (MDFF) works by flexibly fitting possible atomic structures into cryo-EM maps. Although this approach works well with relatively undetailed (or ‘low resolution’) cryo-EM images, it struggles to handle the high-resolution cryo-EM maps now being generated.

Singharoy, Teo, McGreevy et al. have now developed two MDFF methods – called cascade MDFF and resolution exchange MDFF – that help to resolve atomic models of biological molecules from cryo-EM images. Each method can refine poorly guessed models into ones that are consistent with the high-resolution experimental images. The refinement is achieved by interpreting a range of images that starts with a ‘fuzzy’ image. The contrast of the image is then progressively improved until an image is produced that has a resolution that is good enough to almost distinguish individual atoms.

The method works because each cryo-EM image shows not just one, but a collection of atomic structures that the molecule can take on, with the fuzzier parts of the image representing the more flexible parts of the molecule. By taking into account this flexibility, the large-scale features of the protein structure can be determined first from the fuzzier images, and increasing the contrast of the images allows smaller-scale refinements to be made to the structure.

The MDFF tools have been designed to be easy to use and are available to researchers at low cost through cloud computing platforms. They can now be used to unravel the structure of many different proteins and protein complexes including those involved in photosynthesis, respiration and protein synthesis.

DOI:http://dx.doi.org/10.7554/eLife.16105.002

The changing landscape of membrane protein structural biology through developments in electron microscopy

Membrane proteins are ubiquitous in biology and are key targets for therapeutic development. Despite this, our structural understanding has lagged behind that of their soluble counterparts. This review provides an overview of this important field, focusing in particular on the recent resurgence of electron microscopy (EM) and the increasing role it has to play in the structural studies of membrane proteins, and illustrating this through several case studies. In addition, we examine some of the challenges remaining in structural determination, and what steps are underway to enhance our knowledge of these enigmatic proteins.

An introduction to sample preparation and imaging by cryo-electron microscopy for structural biology

Transmission electron microscopy (EM) is a versatile technique that can be used to image biological specimens ranging from intact eukaryotic cells to individual proteins >150 kDa. There are several strategies for preparing samples for imaging by EM, including negative staining and cryogenic freezing. In the last few years, cryo-EM has undergone a ‘resolution revolution’, owing to both advances in imaging hardware, image processing software, and improvements in sample preparation, leading to growing number of researchers using cryo-EM as a research tool. However, cryo-EM is still a rapidly growing field, with unique challenges. Here, we summarise considerations for imaging of a range of specimens from macromolecular complexes to cells using EM.