Deep learning improves macromolecule identification in 3D cellular cryo-electron tomograms

What shapes template-matching performance in cryogenic electron tomography in situ?

The detection of specific biological macromolecules in cryogenic electron tomography data is frequently approached by applying cross-correlation-based 3D template matching. To reduce computational cost and noise, high binning is used to aggregate voxels before template matching. This remains a prevalent practice in both practical applications and methods development. Here, the relation between template size, shape and angular sampling is systematically evaluated to identify ribosomes in a ground-truth annotated data set. It is shown that at the commonly used binning, a detailed subtomogram average, a sphere and a heart emoji result in near-identical performance. These findings indicate that with current template-matching practices macromolecules can only be detected with high precision if their shape and size are sufficiently different from the background. Using theoretical considerations, the experimental results are rationalized and it is discussed why primarily low-frequency information remains at high binning and that template matching fails to be accurate because similarly shaped and sized macromolecules have similar low-frequency spectra. These challenges are discussed and potential enhancements for future template-matching methodologies are proposed.

ColabSeg: An interactive tool for editing, processing, and visualizing membrane segmentations from cryo-ET data

Cellular cryo-electron tomography (cryo-ET) has emerged as a key method to unravel the spatial and structural complexity of cells in their near-native state at unprecedented molecular resolution. To enable quantitative analysis of the complex shapes and morphologies of lipid membranes, the noisy three-dimensional (3D) volumes must be segmented. Despite recent advances, this task often requires considerable user intervention to curate the resulting segmentations. Here, we present ColabSeg, a Python-based tool for processing, visualizing, editing, and fitting membrane segmentations from cryo-ET data for downstream analysis. ColabSeg makes many well-established algorithms for point-cloud processing easily available to the broad community of structural biologists for applications in cryo-ET through its graphical user interface (GUI). We demonstrate the usefulness of the tool with a range of use cases and biological examples. Finally, for a large Mycoplasma pneumoniae dataset of 50 tomograms, we show how ColabSeg enables high-throughput membrane segmentation, which can be used as valuable training data for fully automated convolutional neural network (CNN)-based segmentation.

The advent of preventive high-resolution structural histopathology by artificial-intelligence-powered cryogenic electron tomography

Advances in cryogenic electron microscopy (cryoEM) single particle analysis have revolutionized structural biology by facilitating the in vitro determination of atomic- and near-atomic-resolution structures for fully hydrated macromolecular complexes exhibiting compositional and conformational heterogeneity across a wide range of sizes. Cryogenic electron tomography (cryoET) and subtomogram averaging are rapidly progressing toward delivering similar insights for macromolecular complexes in situ, without requiring tags or harsh biochemical purification. Furthermore, cryoET enables the visualization of cellular and tissue phenotypes directly at molecular, nanometric resolution without chemical fixation or staining artifacts. This forward-looking review covers recent developments in cryoEM/ET and related technologies such as cryogenic focused ion beam milling scanning electron microscopy and correlative light microscopy, increasingly enhanced and supported by artificial intelligence algorithms. Their potential application to emerging concepts is discussed, primarily the prospect of complementing medical histopathology analysis. Machine learning solutions are poised to address current challenges posed by "big data" in cryoET of tissues, cells, and macromolecules, offering the promise of enabling novel, quantitative insights into disease processes, which may translate into the clinic and lead to improved diagnostics and targeted therapeutics.

High-confidence 3D template matching for cryo-electron tomography

Visual proteomics attempts to build atlases of the molecular content of cells but the automated annotation of cryo electron tomograms remains challenging. Template matching (TM) and methods based on machine learning detect structural signatures of macromolecules. However, their applicability remains limited in terms of both the abundance and size of the molecular targets. Here we show that the performance of TM is greatly improved by using template-specific search parameter optimization and by including higher-resolution information. We establish a TM pipeline with systematically tuned parameters for the automated, objective and comprehensive identification of structures with confidence 10 to 100-fold above the noise level. We demonstrate high-fidelity and high-confidence localizations of nuclear pore complexes, vaults, ribosomes, proteasomes, fatty acid synthases, lipid membranes and microtubules, and individual subunits inside crowded eukaryotic cells. We provide software tools for the generic implementation of our method that is broadly applicable towards realizing visual proteomics.

TomoNet: A streamlined cryogenic electron tomography software pipeline with automatic particle picking on flexible lattices

Cryogenic electron tomography (cryoET) is capable of determining in situ biological structures of molecular complexes at near-atomic resolution by averaging half a million subtomograms. While abundant complexes/particles are often clustered in arrays, precisely locating and seamlessly averaging such particles across many tomograms present major challenges. Here, we developed TomoNet, a software package with a modern graphical user interface to carry out the entire pipeline of cryoET and subtomogram averaging to achieve high resolution. TomoNet features built-in automatic particle picking and three-dimensional (3D) classification functions and integrates commonly used packages to streamline high-resolution subtomogram averaging for structures in 1D, 2D, or 3D arrays. Automatic particle picking is accomplished in two complementary ways: one based on template matching and the other using deep learning. TomoNet's hierarchical file organization and visual display facilitate efficient data management as required for large cryoET datasets. Applications of TomoNet to three types of datasets demonstrate its capability of efficient and accurate particle picking on flexible and imperfect lattices to obtain high-resolution 3D biological structures: virus-like particles, bacterial surface layers within cellular lamellae, and membranes decorated with nuclear egress protein complexes. These results demonstrate TomoNet's potential for broad applications to various cryoET projects targeting high-resolution in situ structures.

Bridging structural and cell biology with cryo-electron microscopy

Most life scientists would agree that understanding how cellular processes work requires structural knowledge about the macromolecules involved. For example, deciphering the double-helical nature of DNA revealed essential aspects of how genetic information is stored, copied and repaired. Yet, being reductionist in nature, structural biology requires the purification of large amounts of macromolecules, often trimmed off larger functional units. The advent of cryogenic electron microscopy (cryo-EM) greatly facilitated the study of large, functional complexes and generally of samples that are hard to express, purify and/or crystallize. Nevertheless, cryo-EM still requires purification and thus visualization outside of the natural context in which macromolecules operate and coexist. Conversely, cell biologists have been imaging cells using a number of fast-evolving techniques that keep expanding their spatial and temporal reach, but always far from the resolution at which chemistry can be understood. Thus, structural and cell biology provide complementary, yet unconnected visions of the inner workings of cells. Here we discuss how the interplay between cryo-EM and cryo-electron tomography, as a connecting bridge to visualize macromolecules in situ, holds great promise to create comprehensive structural depictions of macromolecules as they interact in complex mixtures or, ultimately, inside the cell itself.

Rapid structural analysis of bacterial ribosomes in situ

Rapid structural analysis of purified proteins and their complexes has become increasingly common thanks to key methodological advances in cryo-electron microscopy (cryo-EM) and associated data processing software packages. In contrast, analogous structural analysis in cells via cryo-electron tomography (cryo-ET) remains challenging due to critical technical bottlenecks, including low-throughput sample preparation and imaging, and laborious data processing methods. Here, we describe the development of a rapid in situ cryo-ET sample preparation and data analysis workflow that results in the routine determination of sub-nm resolution ribosomal structures. We apply this workflow to E. coli, producing a 5.8 Å structure of the 70S ribosome from cells in less than 10 days, and we expect this workflow will be widely applicable to related bacterial samples.

Learning structural heterogeneity from cryo-electron sub-tomograms with tomoDRGN

Cryo-electron tomography (cryo-ET) enables observation of macromolecular complexes in their native, spatially contextualized cellular environment. Cryo-ET processing software to visualize such complexes at nanometer resolution via iterative alignment and averaging are well developed but rely upon assumptions of structural homogeneity among the complexes of interest. Recently developed tools allow for some assessment of structural diversity but have limited capacity to represent highly heterogeneous structures, including those undergoing continuous conformational changes. Here we extend the highly expressive cryoDRGN (Deep Reconstructing Generative Networks) deep learning architecture, originally created for single-particle cryo-electron microscopy analysis, to cryo-ET. Our new tool, tomoDRGN, learns a continuous low-dimensional representation of structural heterogeneity in cryo-ET datasets while also learning to reconstruct heterogeneous structural ensembles supported by the underlying data. Using simulated and experimental data, we describe and benchmark architectural choices within tomoDRGN that are uniquely necessitated and enabled by cryo-ET. We additionally illustrate tomoDRGN's efficacy in analyzing diverse datasets, using it to reveal high-level organization of human immunodeficiency virus (HIV) capsid complexes assembled in virus-like particles and to resolve extensive structural heterogeneity among ribosomes imaged in situ.

DeepETPicker: Fast and accurate 3D particle picking for cryo-electron tomography using weakly supervised deep learning

To solve three-dimensional structures of biological macromolecules in situ, large numbers of particles often need to be picked from cryo-electron tomograms. However, adoption of automated particle-picking methods remains limited because of their technical limitations. To overcome the limitations, we develop DeepETPicker, a deep learning model for fast and accurate picking of particles from cryo-electron tomograms. Training of DeepETPicker requires only weak supervision with low numbers of simplified labels, reducing the burden of manual annotation. The simplified labels combined with the customized and lightweight model architecture of DeepETPicker and accelerated pooling enable substantial performance improvement. When tested on simulated and real tomograms, DeepETPicker outperforms the competing state-of-the-art methods by achieving the highest overall accuracy and speed, which translate into higher authenticity and coordinates accuracy of picked particles and higher resolutions of final reconstruction maps. DeepETPicker is provided in open source with a user-friendly interface to support cryo-electron tomography in situ.

DUAL: deep unsupervised simultaneous simulation and denoising for cryo-electron tomography

Recent biotechnological developments in cryo-electron tomography allow direct visualization of native sub-cellular structures with unprecedented details and provide essential information on protein functions/dysfunctions. Denoising can enhance the visualization of protein structures and distributions. Automatic annotation via data simulation can ameliorate the time-consuming manual labeling of large-scale datasets. Here, we combine the two major cryo-ET tasks together in DUAL, by a specific cyclic generative adversarial network with novel noise disentanglement. This enables end-to-end unsupervised learning that requires no labeled data for training. The denoising branch outperforms existing works and substantially improves downstream particle picking accuracy on benchmark datasets. The simulation branch provides learning-based cryo-ET simulation for the first time and generates synthetic tomograms indistinguishable from experimental ones. Through comprehensive evaluations, we showcase the effectiveness of DUAL in detecting macromolecular complexes across a wide range of molecular weights in experimental datasets. The versatility of DUAL is expected to empower cryo-ET researchers by improving visual interpretability, enhancing structural detection accuracy, expediting annotation processes, facilitating cross-domain model adaptability, and compensating for missing wedge artifacts. Our work represents a significant advancement in the unsupervised mining of protein structures in cryo-ET, offering a multifaceted tool that facilitates cryo-ET research.

Integrating cellular electron microscopy with multimodal data to explore biology across space and time

Biology spans a continuum of length and time scales. Individual experimental methods only glimpse discrete pieces of this spectrum but can be combined to construct a more holistic view. In this Review, we detail the latest advancements in volume electron microscopy (vEM) and cryo-electron tomography (cryo-ET), which together can visualize biological complexity across scales from the organization of cells in large tissues to the molecular details inside native cellular environments. In addition, we discuss emerging methodologies for integrating three-dimensional electron microscopy (3DEM) imaging with multimodal data, including fluorescence microscopy, mass spectrometry, single-particle analysis, and AI-based structure prediction. This multifaceted approach fills gaps in the biological continuum, providing functional context, spatial organization, molecular identity, and native interactions. We conclude with a perspective on incorporating diverse data into computational simulations that further bridge and extend length scales while integrating the dimension of time.

Revealing the 3D Structure of Block Copolymers with Electron Microscopy: Current Status and Future Directions

Block copolymers (BCPs) are considered model systems for understanding and utilizing self-assembly in soft matter. Their tunable nanometric structure and composition enable comprehensive studies of self-assembly processes as well as make them relevant materials in diverse applications. A key step in developing and controlling BCP nanostructures is a full understanding of their three-dimensional (3D) structure and how this structure is affected by the BCP chemistry, confinement, boundary conditions, and the self-assembly evolution and dynamics. Electron microscopy (EM) is a leading method in BCP 3D characterization owing to its high resolution in imaging nanosized structures. Here we discuss the two main 3D EM methods: namely, transmission EM tomography and slice and view scanning EM tomography. We present each method's principles, examine their strengths and weaknesses, and discuss ways researchers have devised to overcome some of the challenges in BCP 3D characterization with EM- from specimen preparation to imaging radiation-sensitive materials. Importantly, we review current and new cutting-edge EM methods such as direct electron detectors, energy dispersive X-ray spectroscopy of soft matter, high temporal rate imaging, and single-particle analysis that have great potential for expanding the BCP understanding through EM in the future.

Genetically encoded multimeric tags for subcellular protein localization in cryo-EM

Cryo-electron tomography (cryo-ET) allows for label-free high-resolution imaging of macromolecular assemblies in their native cellular context. However, the localization of macromolecules of interest in tomographic volumes can be challenging. Here we present a ligand-inducible labeling strategy for intracellular proteins based on fluorescent, 25-nm-sized, genetically encoded multimeric particles (GEMs). The particles exhibit recognizable structural signatures, enabling their automated detection in cryo-ET data by convolutional neural networks. The coupling of GEMs to green fluorescent protein-tagged macromolecules of interest is triggered by addition of a small-molecule ligand, allowing for time-controlled labeling to minimize disturbance to native protein function. We demonstrate the applicability of GEMs for subcellular-level localization of endogenous and overexpressed proteins across different organelles in human cells using cryo-correlative fluorescence and cryo-ET imaging. We describe means for quantifying labeling specificity and efficiency, and for systematic optimization for rare and abundant protein targets, with emphasis on assessing the potential effects of labeling on protein function.

nextPYP: a comprehensive and scalable platform for characterizing protein variability in situ using single-particle cryo-electron tomography

Single-particle cryo-electron tomography is an emerging technique capable of determining the structure of proteins imaged within the native context of cells at molecular resolution. While high-throughput techniques for sample preparation and tilt-series acquisition are beginning to provide sufficient data to allow structural studies of proteins at physiological concentrations, the complex data analysis pipeline and the demanding storage and computational requirements pose major barriers for the development and broader adoption of this technology. Here, we present a scalable, end-to-end framework for single-particle cryo-electron tomography data analysis from on-the-fly pre-processing of tilt series to high-resolution refinement and classification, which allows efficient analysis and visualization of datasets with hundreds of tilt series and hundreds of thousands of particles. We validate our approach using in vitro and cellular datasets, demonstrating its effectiveness at achieving high-resolution and revealing conformational heterogeneity in situ. The framework is made available through an intuitive and easy-to-use computer application, nextPYP ( http://nextpyp.app ).

Charting the molecular landscape of the cell

Biological function of macromolecules is closely tied to their cellular location, as well as to interactions with other molecules within the native environment of the cell. Therefore, to obtain detailed mechanistic insights into macromolecular functionality, one of the outstanding targets for structural biology is to produce an atomic-level understanding of the cell. One structural biology technique that has already been used to directly derive atomic models of macromolecules from cells, without any additional external information, is electron cryotomography (cryoET). In this perspective article, we discuss possible routes to chart the molecular landscape of the cell by advancing cryoET imaging as well as by embedding cryoET into correlative imaging workflows.

Cryo-electron tomography reveals the structural diversity of cardiac proteins in their cellular context

Cardiovascular diseases are a leading cause of death worldwide, but our understanding of the underlying mechanisms is limited, in part because of the complexity of the cellular machinery that controls the heart muscle contraction cycle. Cryogenic electron tomography (cryo-ET) provides a way to visualize diverse cellular machinery while preserving contextual information like subcellular localization and transient complex formation, but this approach has not been widely applied to the study of heart muscle cells (cardiomyocytes). Here, we deploy a platform for studying cardiovascular disease by combining cryo-ET with human induced pluripotent stem cell-derived cardiomyocytes (hiPSC-CMs). After developing a cryo-ET workflow for visualizing macromolecules in hiPSC-CMs, we reconstructed sub-nanometer resolution structures of the human thin filament, a central component of the contractile machinery. We also visualized a previously unobserved organization of a regulatory complex that connects muscle contraction to calcium signaling (the troponin complex), highlighting the value of our approach for interrogating the structures of cardiac proteins in their cellular context.

Streamlined structure determination by cryo-electron tomography and subtomogram averaging using TomoBEAR

Structures of macromolecules in their native state provide unique unambiguous insights into their functions. Cryo-electron tomography combined with subtomogram averaging demonstrated the power to solve such structures in situ at resolutions in the range of 3 Angstrom for some macromolecules. In order to be applicable to the structural determination of the majority of macromolecules observable in cells in limited amounts, processing of tomographic data has to be performed in a high-throughput manner. Here we present TomoBEAR-a modular configurable workflow engine for streamlined processing of cryo-electron tomographic data for subtomogram averaging. TomoBEAR combines commonly used cryo-EM packages with reasonable presets to provide a transparent ("white box") approach for data management and processing. We demonstrate applications of TomoBEAR to two data sets of purified macromolecular targets, to an ion channel RyR1 in a membrane, and the tomograms of plasma FIB-milled lamellae and demonstrate the ability to produce high-resolution structures. TomoBEAR speeds up data processing, minimizes human interventions, and will help accelerate the adoption of in situ structural biology by cryo-ET. The source code and the documentation are freely available.

Computational methods for in situ structural studies with cryogenic electron tomography

Cryo-electron tomography (cryo-ET) plays a critical role in imaging microorganisms in situ in terms of further analyzing the working mechanisms of viruses and drug exploitation, among others. A data processing workflow for cryo-ET has been developed to reconstruct three-dimensional density maps and further build atomic models from a tilt series of two-dimensional projections. Low signal-to-noise ratio (SNR) and missing wedge are two major factors that make the reconstruction procedure challenging. Because only few near-atomic resolution structures have been reconstructed in cryo-ET, there is still much room to design new approaches to improve universal reconstruction resolutions. This review summarizes classical mathematical models and deep learning methods among general reconstruction steps. Moreover, we also discuss current limitations and prospects. This review can provide software and methods for each step of the entire procedure from tilt series by cryo-ET to 3D atomic structures. In addition, it can also help more experts in various fields comprehend a recent research trend in cryo-ET. Furthermore, we hope that more researchers can collaborate in developing computational methods and mathematical models for high-resolution three-dimensional structures from cryo-ET datasets.

Finding Nano-Ötzi: Cryo-Electron Tomography Visualization Guided by Learned Segmentation

Cryo-electron tomography (cryo-ET) is a new 3D imaging technique with unprecedented potential for resolving submicron structural details. Existing volume visualization methods, however, are not able to reveal details of interest due to low signal-to-noise ratio. In order to design more powerful transfer functions, we propose leveraging soft segmentation as an explicit component of visualization for noisy volumes. Our technical realization is based on semi-supervised learning, where we combine the advantages of two segmentation algorithms. First, the weak segmentation algorithm provides good results for propagating sparse user-provided labels to other voxels in the same volume and is used to generate dense pseudo-labels. Second, the powerful deep-learning-based segmentation algorithm learns from these pseudo-labels to generalize the segmentation to other unseen volumes, a task that the weak segmentation algorithm fails at completely. The proposed volume visualization uses deep-learning-based segmentation as a component for segmentation-aware transfer function design. Appropriate ramp parameters can be suggested automatically through frequency distribution analysis. Furthermore, our visualization uses gradient-free ambient occlusion shading to further suppress the visual presence of noise, and to give structural detail the desired prominence. The cryo-ET data studied in our technical experiments are based on the highest-quality tilted series of intact SARS-CoV-2 virions. Our technique shows the high impact in target sciences for visual data analysis of very noisy volumes that cannot be visualized with existing techniques.

Machine learning for cross-scale microscopy of viruses

Despite advances in virological sciences and antiviral research, viruses continue to emerge, circulate, and threaten public health. We still lack a comprehensive understanding of how cells and individuals remain susceptible to infectious agents. This deficiency is in part due to the complexity of viruses, including the cell states controlling virus-host interactions. Microscopy samples distinct cellular infection stages in a multi-parametric, time-resolved manner at molecular resolution and is increasingly enhanced by machine learning and deep learning. Here we discuss how state-of-the-art artificial intelligence (AI) augments light and electron microscopy and advances virological research of cells. We describe current procedures for image denoising, object segmentation, tracking, classification, and super-resolution and showcase examples of how AI has improved the acquisition and analyses of microscopy data. The power of AI-enhanced microscopy will continue to help unravel virus infection mechanisms, develop antiviral agents, and improve viral vectors.

PickYOLO: Fast deep learning particle detector for annotation of cryo electron tomograms

Particle localization (picking) in digital tomograms is a laborious and time-intensive step in cryogenic electron tomography (cryoET) analysis often requiring considerable user involvement, thus becoming a bottleneck for automated cryoET subtomogram averaging (STA) pipelines. In this paper, we introduce a deep learning framework called PickYOLO to tackle this problem. PickYOLO is a super-fast, universal particle detector based on the deep-learning real-time object recognition system YOLO (You Only Look Once), and tested on single particles, filamentous structures, and membrane-embedded particles. After training with the centre coordinates of a few hundred representative particles, the network automatically detects additional particles with high yield and reliability at a rate of 0.24-3.75 s per tomogram. PickYOLO can automatically detect number of particles comparable to those manually selected by experienced microscopists. This makes PickYOLO a valuable tool to substantially reduce the time and manual effort needed to analyse cryoET data for STA, greatly aiding in high-resolution cryoET structure determination.

Big data in contemporary electron microscopy: challenges and opportunities in data transfer, compute and management

The second decade of the twenty-first century witnessed a new challenge in the handling of microscopy data. Big data, data deluge, large data, data compliance, data analytics, data integrity, data interoperability, data retention and data lifecycle are terms that have introduced themselves to the electron microscopy sciences. This is largely attributed to the booming development of new microscopy hardware tools. As a result, large digital image files with an average size of one terabyte within one single acquisition session is not uncommon nowadays, especially in the field of cryogenic electron microscopy. This brings along numerous challenges in data transfer, compute and management. In this review, we will discuss in detail the current state of international knowledge on big data in contemporary electron microscopy and how big data can be transferred, computed and managed efficiently and sustainably. Workflows, solutions, approaches and suggestions will be provided, with the example of the latest experiences in Australia. Finally, important principles such as data integrity, data lifetime and the FAIR and CARE principles will be considered.

A practical multicellular sample preparation pipeline broadens the application of in situ cryo-electron tomography

The structural studies of macromolecules in their physiological context, particularly in tissue, is constrained by the bottleneck of sample preparation. In this study, we present a practical pipeline for preparing multicellular samples for cryo-electron tomography. The pipeline comprises sample isolation, vitrification, and lift-out-based lamella preparation using commercially available instruments. We demonstrate the efficacy of our pipeline by visualizing pancreatic β cells from mouse islets at the molecular level. This pipeline enables the determination of the properties of insulin crystals in situ for the first time, using unperturbed samples.

Extensive Angular Sampling Enables the Sensitive Localization of Macromolecules in Electron Tomograms

Cryo-electron tomography provides 3D images of macromolecules in their cellular context. To detect macromolecules in tomograms, template matching (TM) is often used, which uses 3D models that are often reliable for substantial parts of the macromolecules. However, the extent of rotational searches in particle detection has not been investigated due to computational limitations. Here, we provide a GPU implementation of TM as part of the PyTOM software package, which drastically speeds up the orientational search and allows for sampling beyond the Crowther criterion within a feasible timeframe. We quantify the improvements in sensitivity and false-discovery rate for the examples of ribosome identification and detection. Sampling at the Crowther criterion, which was effectively impossible with CPU implementations due to the extensive computation times, allows for automated extraction with high sensitivity. Consequently, we also show that an extensive angular sample renders 3D TM sensitive to the local alignment of tilt series and damage induced by focused ion beam milling. With this new release of PyTOM, we focused on integration with other software packages that support more refined subtomogram-averaging workflows. The automated classification of ribosomes by TM with appropriate angular sampling on locally corrected tomograms has a sufficiently low false-discovery rate, allowing for it to be directly used for high-resolution averaging and adequate sensitivity to reveal polysome organization.

Heterogeneous non-canonical nucleosomes predominate in yeast cells in situ

Nuclear processes depend on the organization of chromatin, whose basic units are cylinder-shaped complexes called nucleosomes. A subset of mammalian nucleosomes in situ (inside cells) resembles the canonical structure determined in vitro 25 years ago. Nucleosome structure in situ is otherwise poorly understood. Using cryo-electron tomography (cryo-ET) and 3D classification analysis of budding yeast cells, here we find that canonical nucleosomes account for less than 10% of total nucleosomes expected in situ. In a strain in which H2A-GFP is the sole source of histone H2A, class averages that resemble canonical nucleosomes both with and without GFP densities are found ex vivo (in nuclear lysates), but not in situ. These data suggest that the budding yeast intranuclear environment favors multiple non-canonical nucleosome conformations. Using the structural observations here and the results of previous genomics and biochemical studies, we propose a model in which the average budding yeast nucleosome's DNA is partially detached in situ.

A method for restoring signals and revealing individual macromolecule states in cryo-ET, REST

Cryo-electron tomography (cryo-ET) is widely used to explore the 3D density of biomacromolecules. However, the heavy noise and missing wedge effect prevent directly visualizing and analyzing the 3D reconstructions. Here, we introduced REST, a deep learning strategy-based method to establish the relationship between low-quality and high-quality density and transfer the knowledge to restore signals in cryo-ET. Test results on the simulated and real cryo-ET datasets show that REST performs well in denoising and compensating the missing wedge information. The application in dynamic nucleosomes, presenting either in the form of individual particles or in the context of cryo-FIB nuclei section, indicates that REST has the capability to reveal different conformations of target macromolecules without subtomogram averaging. Moreover, REST noticeably improves the reliability of particle picking. These advantages enable REST to be a powerful tool for the straightforward interpretation of target macromolecules by visual inspection of the density and of a broad range of other applications in cryo-ET, such as segmentation, particle picking, and subtomogram averaging.

TomoTwin: generalized 3D localization of macromolecules in cryo-electron tomograms with structural data mining

Cryogenic-electron tomography enables the visualization of cellular environments in extreme detail, however, tools to analyze the full amount of information contained within these densely packed volumes are still needed. Detailed analysis of macromolecules through subtomogram averaging requires particles to first be localized within the tomogram volume, a task complicated by several factors including a low signal to noise ratio and crowding of the cellular space. Available methods for this task suffer either from being error prone or requiring manual annotation of training data. To assist in this crucial particle picking step, we present TomoTwin: an open source general picking model for cryogenic-electron tomograms based on deep metric learning. By embedding tomograms in an information-rich, high-dimensional space that separates macromolecules according to their three-dimensional structure, TomoTwin allows users to identify proteins in tomograms de novo without manually creating training data or retraining the network to locate new proteins.

Bringing Structure to Cell Biology with Cryo-Electron Tomography

Recent advances in cryo-electron microscopy have marked only the beginning of the potential of this technique. To bring structure into cell biology, the modality of cryo-electron tomography has fast developed into a bona fide in situ structural biology technique where structures are determined in their native environment, the cell. Nearly every step of the cryo-focused ion beam-assisted electron tomography (cryo-FIB-ET) workflow has been improved upon in the past decade, since the first windows were carved into cells, unveiling macromolecular networks in near-native conditions. By bridging structural and cell biology, cryo-FIB-ET is advancing our understanding of structure-function relationships in their native environment and becoming a tool for discovering new biology.

Computational Methods Toward Unbiased Pattern Mining and Structure Determination in Cryo-Electron Tomography Data

Cryo-electron tomography can uniquely probe the native cellular environment for macromolecular structures. Tomograms feature complex data with densities of diverse, densely crowded macromolecular complexes, low signal-to-noise, and artifacts such as the missing wedge effect. Post-processing of this data generally involves isolating regions or particles of interest from tomograms, organizing them into related groups, and rendering final structures through subtomogram averaging. Template-matching and reference-based structure determination are popular analysis methods but are vulnerable to biases and can often require significant user input. Most importantly, these approaches cannot identify novel complexes that reside within the imaged cellular environment. To reliably extract and resolve structures of interest, efficient and unbiased approaches are therefore of great value. This review highlights notable computational software and discusses how they contribute to making automated structural pattern discovery a possibility. Perspectives emphasizing the importance of features for user-friendliness and accessibility are also presented.

Rapid Synthesis of Cryo-ET Data for Training Deep Learning Models

Deep learning excels at cryo-tomographic image restoration and segmentation tasks but is hindered by a lack of training data. Here we introduce cryo-TomoSim (CTS), a MATLAB-based software package that builds coarse-grained models of macromolecular complexes embedded in vitreous ice and then simulates transmitted electron tilt series for tomographic reconstruction. We then demonstrate the effectiveness of these simulated datasets in training different deep learning models for use on real cryotomographic reconstructions. Computer-generated ground truth datasets provide the means for training models with voxel-level precision, allowing for unprecedented denoising and precise molecular segmentation of datasets. By modeling phenomena such as a three-dimensional contrast transfer function, probabilistic detection events, and radiation-induced damage, the simulated cryo-electron tomograms can cover a large range of imaging content and conditions to optimize training sets. When paired with small amounts of training data from real tomograms, networks become incredibly accurate at segmenting in situ macromolecular assemblies across a wide range of biological contexts.

High-throughput cryo-ET structural pattern mining by unsupervised deep iterative subtomogram clustering

Cryoelectron tomography directly visualizes heterogeneous macromolecular structures in their native and complex cellular environments. However, existing computer-assisted structure sorting approaches are low throughput or inherently limited due to their dependency on available templates and manual labels. Here, we introduce a high-throughput template-and-label-free deep learning approach, Deep Iterative Subtomogram Clustering Approach (DISCA), that automatically detects subsets of homogeneous structures by learning and modeling 3D structural features and their distributions. Evaluation on five experimental cryo-ET datasets shows that an unsupervised deep learning based method can detect diverse structures with a wide range of molecular sizes. This unsupervised detection paves the way for systematic unbiased recognition of macromolecular complexes in situ.

Cryo-electron tomography on focused ion beam lamellae transforms structural cell biology

Cryogenic electron microscopy and data processing enable the determination of structures of isolated macromolecules to near-atomic resolution. However, these data do not provide structural information in the cellular environment where macromolecules perform their native functions, and vital molecular interactions can be lost during the isolation process. Cryogenic focused ion beam (FIB) fabrication generates thin lamellae of cellular samples and tissues, enabling structural studies on the near-native cellular interior and its surroundings by cryogenic electron tomography (cryo-ET). Cellular cryo-ET benefits from the technological developments in electron microscopes, detectors and data processing, and more in situ structures are being obtained and at increasingly higher resolution. In this Review, we discuss recent studies employing cryo-ET on FIB-generated lamellae and the technological developments in ultrarapid sample freezing, FIB fabrication of lamellae, tomography, data processing and correlative light and electron microscopy that have enabled these studies. Finally, we explore the future of cryo-ET in terms of both methods development and biological application.

Convolutional networks for supervised mining of molecular patterns within cellular context

Cryo-electron tomograms capture a wealth of structural information on the molecular constituents of cells and tissues. We present DeePiCt (deep picker in context), an open-source deep-learning framework for supervised segmentation and macromolecular complex localization in cryo-electron tomography. To train and benchmark DeePiCt on experimental data, we comprehensively annotated 20 tomograms of Schizosaccharomyces pombe for ribosomes, fatty acid synthases, membranes, nuclear pore complexes, organelles, and cytosol. By comparing DeePiCt to state-of-the-art approaches on this dataset, we show its unique ability to identify low-abundance and low-density complexes. We use DeePiCt to study compositionally distinct subpopulations of cellular ribosomes, with emphasis on their contextual association with mitochondria and the endoplasmic reticulum. Finally, applying pre-trained networks to a HeLa cell tomogram demonstrates that DeePiCt achieves high-quality predictions in unseen datasets from different biological species in a matter of minutes. The comprehensively annotated experimental data and pre-trained networks are provided for immediate use by the community.

Studying membrane modulation mechanisms by electron cryo-tomography

Membrane modulation is a key part of cellular life. Critical to processes like energy production, cell division, trafficking, migration and even pathogen entry, defects in membrane modulation are often associated with diseases. Studying the molecular mechanisms of membrane modulation is challenging due to the highly dynamic nature of the oligomeric assemblies involved, which adopt multiple conformations depending on the precise event they are participating in. With the development of electron cryo-tomography and subtomogram averaging, many of these challenges are being resolved as it is now possible to observe complex macromolecular assemblies inside a cell at nanometre to sub-nanometre resolutions. Here, we review the different ways electron cryo-tomography is being used to help uncover the molecular mechanisms used by cells to shape their membranes.

Electron microscopy holdings of the Protein Data Bank: the impact of the resolution revolution, new validation tools, and implications for the future

As a discipline, structural biology has been transformed by the three-dimensional electron microscopy (3DEM) "Resolution Revolution" made possible by convergence of robust cryo-preservation of vitrified biological materials, sample handling systems, and measurement stages operating a liquid nitrogen temperature, improvements in electron optics that preserve phase information at the atomic level, direct electron detectors (DEDs), high-speed computing with graphics processing units, and rapid advances in data acquisition and processing software. 3DEM structure information (atomic coordinates and related metadata) are archived in the open-access Protein Data Bank (PDB), which currently holds more than 11,000 3DEM structures of proteins and nucleic acids, and their complexes with one another and small-molecule ligands (~ 6% of the archive). Underlying experimental data (3DEM density maps and related metadata) are stored in the Electron Microscopy Data Bank (EMDB), which currently holds more than 21,000 3DEM density maps. After describing the history of the PDB and the Worldwide Protein Data Bank (wwPDB) partnership, which jointly manages both the PDB and EMDB archives, this review examines the origins of the resolution revolution and analyzes its impact on structural biology viewed through the lens of PDB holdings. Six areas of focus exemplifying the impact of 3DEM across the biosciences are discussed in detail (icosahedral viruses, ribosomes, integral membrane proteins, SARS-CoV-2 spike proteins, cryogenic electron tomography, and integrative structure determination combining 3DEM with complementary biophysical measurement techniques), followed by a review of 3DEM structure validation by the wwPDB that underscores the importance of community engagement.

Cryomicroscopy in situ: what is the smallest molecule that can be directly identified without labels in a cell?

Electron cryomicroscopy (cryoEM) has made great strides in the last decade, such that the atomic structure of most biological macromolecules can, at least in principle, be determined. Major technological advances - in electron imaging hardware, data analysis software, and cryogenic specimen preparation technology - continue at pace and contribute to the exponential growth in the number of atomic structures determined by cryoEM. It is now conceivable that within the next decade we will have structures for hundreds of thousands of unique protein and nucleic acid molecular complexes. But the answers to many important questions in biology would become obvious if we could identify these structures precisely inside cells with quantifiable error. In the context of an abundance of known structures, it is appropriate to consider the current state of electron cryomicroscopy for frozen specimens prepared directly from cells, and try to answer to the question of the title, both now and in the foreseeable future.

Strategies for picking membrane-associated particles within subtomogram averaging workflows

Cryo-electron tomography (cryo-ET) with subtomogram averaging (STA) has emerged as a key tool for determining macromolecular structure(s) in vitro and in situ. However, processing cryo-ET data with STA currently requires significant user expertise. Recent efforts have streamlined several steps in STA workflows; however, particle picking remains a time-consuming bottleneck for many projects and requires considerable user input. Here, we present several strategies for the time-efficient and accurate picking of membrane-associated particles using the COPII inner coat as a case study. We also discuss a range of particle cleaning solutions to remove both poor quality and false-positive particles from STA datasets. We provide a step-by-step guide and the necessary scripts for users to independently carry out the particle picking and cleaning strategies discussed.

Isotropic reconstruction for electron tomography with deep learning

Cryogenic electron tomography (cryoET) allows visualization of cellular structures in situ. However, anisotropic resolution arising from the intrinsic "missing-wedge" problem has presented major challenges in visualization and interpretation of tomograms. Here, we have developed IsoNet, a deep learning-based software package that iteratively reconstructs the missing-wedge information and increases signal-to-noise ratio, using the knowledge learned from raw tomograms. Without the need for sub-tomogram averaging, IsoNet generates tomograms with significantly reduced resolution anisotropy. Applications of IsoNet to three representative types of cryoET data demonstrate greatly improved structural interpretability: resolving lattice defects in immature HIV particles, establishing architecture of the paraflagellar rod in Eukaryotic flagella, and identifying heptagon-containing clathrin cages inside a neuronal synapse of cultured cells. Therefore, by overcoming two fundamental limitations of cryoET, IsoNet enables functional interpretation of cellular tomograms without sub-tomogram averaging. Its application to high-resolution cellular tomograms should also help identify differently oriented complexes of the same kind for sub-tomogram averaging.

Towards unsupervised classification of macromolecular complexes in cryo electron tomography: Challenges and opportunities

BACKGROUND AND OBJECTIVES

Cryo electron tomography visualizes native cells at nanometer resolution, but analysis is challenged by noise and artifacts. Recently, supervised deep learning methods have been applied to decipher the 3D spatial distribution of macromolecules. However, in order to discover unknown objects, unsupervised classification techniques are necessary. In this paper, we provide an overview of unsupervised deep learning techniques, discuss the challenges to analyze cryo-ET data, and provide a proof-of-concept on real data.

METHODS

We propose a weakly supervised subtomogram classification method based on transfer learning. We use a deep neural network to learn a clustering friendly representation able to capture 3D shapes in the presence of noise and artifacts. This representation is learned here from a synthetic data set.

RESULTS

We show that when applying k-means clustering given a learning-based representation, it becomes possible to satisfyingly classify real subtomograms according to structural similarity. It is worth noting that no manual annotation is used for performing classification.

CONCLUSIONS

We describe the advantages and limitations of our proof-of-concept and raise several perspectives for improving classification performance.

MemBrain: A deep learning-aided pipeline for detection of membrane proteins in Cryo-electron tomograms

BACKGROUND AND OBJECTIVE

Cryo-electron tomography (cryo-ET) is an imaging technique that enables 3D visualization of the native cellular environment at sub-nanometer resolution, providing unpreceded insights into the molecular organization of cells. However, cryo-electron tomograms suffer from low signal-to-noise ratios and anisotropic resolution, which makes subsequent image analysis challenging. In particular, the efficient detection of membrane-embedded proteins is a problem still lacking satisfactory solutions.

METHODS

We present MemBrain - a new deep learning-aided pipeline that automatically detects membrane-bound protein complexes in cryo-electron tomograms. After subvolumes are sampled along a segmented membrane, each subvolume is assigned a score using a convolutional neural network (CNN), and protein positions are extracted by a clustering algorithm. Incorporating rotational subvolume normalization and using a tiny receptive field simplify the task of protein detection and thus facilitate the network training.

RESULTS

MemBrain requires only a small quantity of training labels and achieves excellent performance with only a single annotated membrane (F1 score: 0.88). A detailed evaluation shows that our fully trained pipeline outperforms existing classical computer vision-based and CNN-based approaches by a large margin (F1 score: 0.92 vs. max. 0.63). Furthermore, in addition to protein center positions, MemBrain can determine protein orientations, which has not been implemented by any existing CNN-based method to date. We also show that a pre-trained MemBrain program generalizes to tomograms acquired using different cryo-ET methods and depicting different types of cells.

CONCLUSIONS

MemBrain is a powerful and annotation-efficient tool for the detection of membrane protein complexes in cryo-ET data, with the potential to be used in a wide range of biological studies. It is generalizable to various kinds of tomograms, making it possible to use pretrained models for different tasks. Its efficiency in terms of required annotations also allows rapid training and fine-tuning of models. The corresponding code, pretrained models, and instructions for operating the MemBrain program can be found at: https://github.com/CellArchLab/MemBrain.

Volumetric macromolecule identification in cryo-electron tomograms using capsule networks

Background

Despite recent advances in cellular cryo-electron tomography (CET), developing automated tools for macromolecule identification in submolecular resolution remains challenging due to the lack of annotated data and high structural complexities. To date, the extent of the deep learning methods constructed for this problem is limited to conventional Convolutional Neural Networks (CNNs). Identifying macromolecules of different types and sizes is a tedious and time-consuming task. In this paper, we employ a capsule-based architecture to automate the task of macromolecule identification, that we refer to as 3D-UCaps. In particular, the architecture is composed of three components: feature extractor, capsule encoder, and CNN decoder. The feature extractor converts voxel intensities of input sub-tomograms to activities of local features. The encoder is a 3D Capsule Network (CapsNet) that takes local features to generate a low-dimensional representation of the input. Then, a 3D CNN decoder reconstructs the sub-tomograms from the given representation by upsampling.

Results

We performed binary and multi-class localization and identification tasks on synthetic and experimental data. We observed that the 3D-UNet and the 3D-UCaps had an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1-$$\end{document}F1-score mostly above 60% and 70%, respectively, on the test data. In both network architectures, we observed degradation of at least 40% in the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score when identifying very small particles (PDB entry 3GL1) compared to a large particle (PDB entry 4D8Q). In the multi-class identification task of experimental data, 3D-UCaps had an \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score of 91% on the test data in contrast to 64% of the 3D-UNet. The better \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score of 3D-UCaps compared to 3D-UNet is obtained by a higher precision score. We speculate this to be due to the capsule network employed in the encoder. To study the effect of the CapsNet-based encoder architecture further, we performed an ablation study and perceived that the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F_1$$\end{document}F1-score is boosted as network depth is increased which is in contrast to the previously reported results for the 3D-UNet. To present a reproducible work, source code, trained models, data as well as visualization results are made publicly available.

Conclusion

Quantitative and qualitative results show that 3D-UCaps successfully perform various downstream tasks including identification and localization of macromolecules and can at least compete with CNN architectures for this task. Given that the capsule layers extract both the existence probability and the orientation of the molecules, this architecture has the potential to lead to representations of the data that are better interpretable than those of 3D-UNet.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12859-022-04901-w.

In situ single particle classification reveals distinct 60S maturation intermediates in cells

Previously we showed that high-resolution template matching can localize ribosomes in two-dimensional electron cryo-microscopy (cryo-EM) images of untilted Mycoplasma pneumoniae cells with high precision (Lucas et al., 2021). Here we show that comparing the signal-to-noise ratio (SNR) observed with 2DTM using different templates relative to the same cellular target can correct for local variation in noise and differentiate related complexes in focused ion beam (FIB)-milled cell sections. We use a maximum likelihood approach to define the probability of each particle belonging to each class, thereby establishing a statistic to describe the confidence of our classification. We apply this method in two contexts to locate and classify related intermediate states of 60S ribosome biogenesis in the Saccharomyces cerevisiae cell nucleus. In the first, we separate the nuclear pre-60S population from the cytoplasmic mature 60S population, using the subcellular localization to validate assignment. In the second, we show that relative 2DTM SNRs can be used to separate mixed populations of nuclear pre-60S that are not visually separable. 2DTM can distinguish related molecular populations without the need to generate 3D reconstructions from the data to be classified, permitting classification even when only a few target particles exist in a cell.

Structure Detection in Three-Dimensional Cellular Cryoelectron Tomograms by Reconstructing Two-Dimensional Annotated Tilt Series

The revolutionary technique cryoelectron tomography (cryo-ET) enables imaging of cellular structure and organization in a near-native environment at submolecular resolution, which is vital to subsequent data analysis and modeling. The conventional structure detection process first reconstructs the three-dimensional (3D) tomogram from a series of two-dimensional (2D) projections and then directly detects subcellular components found within the tomogram. However, this process is challenging due to potential structural information loss during the tomographic reconstruction and the limited scope of existing methods since most major state-of-the-art object detection methods are designed for 2D rather than 3D images. Therefore, in this article, as an alternative approach to complement the conventional process, we propose a novel 2D-to-3D framework that detects structures within 2D projection images before reconstructing the results back to 3D. We implemented the proposed framework as three specific algorithms for three individual tasks: semantic segmentation, edge detection, and object localization. As experimental validation of the 2D-to-3D framework for cryo-ET data, we applied the algorithms to the segmentation of mitochondrial calcium phosphate granules, detection of spherical edges, and localization of mitochondria. Quantitative and qualitative results show better performance for prediction tasks of segmentation on the 2D projections and promising performance on object localization and edge detection, paving the way for future studies in the exploration of cryo-ET for in situ structural biology.

Gold nanomaterials and their potential use as cryo-electron tomography labels

Rapid advances in cryo-electron tomography (cryo-ET) are driving a revolution in cellular structural biology. However, unambiguous identification of specific biomolecules within cellular tomograms remains challenging. Overcoming this obstacle and reliably identifying targets in the crowded cellular environment is of major importance for the understanding of cellular function and is a pre-requisite for high-resolution structural analysis. The use of highly-specific, readily visualised and adjustable labels would help mitigate this issue, improving both data quality and sample throughput. While progress has been made in cryo-CLEM and in the development of cloneable high-density tags, technical issues persist and a robust 'cryo-GFP' remains elusive. Readily-synthesized gold nanomaterials conjugated to small 'affinity modules' may represent a solution. The synthesis of materials including gold nanoclusters (AuNCs) and gold nanoparticles (AuNPs) is increasingly well understood and is now within the capabilities of non-specialist laboratories. The remarkable chemical and photophysical properties of <3nm diameter nanomaterials and their emergence as tools with widespread biomedical application presents significant opportunities to the cryo-microscopy community. In this review, we will outline developments in the synthesis, functionalisation and labelling uses of both AuNPs and AuNCs in cryo-ET, while discussing their potential as multi-modal probes for cryo-CLEM.

FSCC: Few-Shot Learning for Macromolecule Classification Based on Contrastive Learning and Distribution Calibration in Cryo-Electron Tomography

Cryo-electron tomography (Cryo-ET) is an emerging technology for three-dimensional (3D) visualization of macromolecular structures in the near-native state. To recover structures of macromolecules, millions of diverse macromolecules captured in tomograms should be accurately classified into structurally homogeneous subsets. Although existing supervised deep learning–based methods have improved classification accuracy, such trained models have limited ability to classify novel macromolecules that are unseen in the training stage. To adapt the trained model to the macromolecule classification of a novel class, massive labeled macromolecules of the novel class are needed. However, data labeling is very time-consuming and labor-intensive. In this work, we propose a novel few-shot learning method for the classification of novel macromolecules (named FSCC). A two-stage training strategy is designed in FSCC to enhance the generalization ability of the model to novel macromolecules. First, FSCC uses contrastive learning to pre-train the model on a sufficient number of labeled macromolecules. Second, FSCC uses distribution calibration to re-train the classifier, enabling the model to classify macromolecules of novel classes (unseen class in the pre-training). Distribution calibration transfers learned knowledge in the pre-training stage to novel macromolecules with limited labeled macromolecules of novel class. Experiments were performed on both synthetic and real datasets. On the synthetic datasets, compared with the state-of-the-art (SOTA) method based on supervised deep learning, FSCC achieves competitive performance. To achieve such performance, FSCC only needs five labeled macromolecules per novel class. However, the SOTA method needs 1100 ∼ 1500 labeled macromolecules per novel class. On the real datasets, FSCC improves the accuracy by 5% ∼ 16% when compared to the baseline model. These demonstrate good generalization ability of contrastive learning and calibration distribution to classify novel macromolecules with very few labeled macromolecules.