Quantifying Variability of Manual Annotation in Cryo-Electron Tomograms

Modified Dice Coefficients for Evaluation of Tumor Segmentation from PET Images: A Proof-of-Concept Study

The Sørensen-Dice similarity coefficient (DSC) is the most common evaluation metric used for image segmentation but it is not always ideal. Namely, the DSC values only depend on the number of misplaced elements instead of their location with respect to the correct segments. Because of this, the DSC is ill-suited for such tasks where the correct location of the borders of an object is difficult to define in an objective way, as is the case in tumor segmentation in positron emission tomography (PET) images. To avoid this issue, we introduce two different modifications of the DSC, one with weights and one with an additional loss term, which also evaluate the distance between the real and the predicted segments. We computed the values of DSC and our new coefficient from 191 predicted tumor segmentation masks created by using PET images of 89 head and neck squamous cell carcinoma patients. We compared the values of all three coefficients with the scores given to these masks by human evaluators. According to our results, the weighted modification of DSC had a higher correlation with the scores given by the human evaluators than the original DSC, and it also produced significantly less variation within the two highest score classes (p-value ≤ 0.018). The new weighted coefficient introduced here has much potential in the evaluation of segmentation results from medical imaging.

3D whole body preclinical micro-CT database of subcutaneous tumors in mice with annotations from 3 annotators

A pivotal animal model for development of anticancer molecules is mice with subcutaneous tumors, grown by injection of xenografted tumor cells, where micro-Computed Tomography (µCT) of the mice is used to analyze the efficacy of the anticancer molecule. Manual delineation of the tumor region is necessary for the analysis, which is time-consuming and inconsistent, highlighting the need for automatic segmentation (AS) tools. This study introduces a preclinical µCT database, comprising 452 whole-body scans from 223 individual mice with subcutaneous tumors, spanning ten diverse µCT datasets conducted between 2014 and 2020 on a preclinical PET/CT scanner, making it the hitherto largest dataset of its kind. Each tumor is annotated manually by three expert annotators, allowing for robust model development. Inter-annotator agreement was analyzed, and we report an overall annotation agreement of 0.903 ± 0.046 (mean ± std) Fleiss' Kappa and a mean deviation in volume estimation of 0.015 ± 0.010 cm3 (6.9% ± 4.7), which establishes a human baseline accuracy for delineation of subcutaneous tumors, while showing good inter-annotator agreement.

The advent of preventive high-resolution structural histopathology by artificial-intelligence-powered cryogenic electron tomography

Advances in cryogenic electron microscopy (cryoEM) single particle analysis have revolutionized structural biology by facilitating the in vitro determination of atomic- and near-atomic-resolution structures for fully hydrated macromolecular complexes exhibiting compositional and conformational heterogeneity across a wide range of sizes. Cryogenic electron tomography (cryoET) and subtomogram averaging are rapidly progressing toward delivering similar insights for macromolecular complexes in situ, without requiring tags or harsh biochemical purification. Furthermore, cryoET enables the visualization of cellular and tissue phenotypes directly at molecular, nanometric resolution without chemical fixation or staining artifacts. This forward-looking review covers recent developments in cryoEM/ET and related technologies such as cryogenic focused ion beam milling scanning electron microscopy and correlative light microscopy, increasingly enhanced and supported by artificial intelligence algorithms. Their potential application to emerging concepts is discussed, primarily the prospect of complementing medical histopathology analysis. Machine learning solutions are poised to address current challenges posed by "big data" in cryoET of tissues, cells, and macromolecules, offering the promise of enabling novel, quantitative insights into disease processes, which may translate into the clinic and lead to improved diagnostics and targeted therapeutics.

Missing Wedge Completion via Unsupervised Learning with Coordinate Networks

Cryogenic electron tomography (cryoET) is a powerful tool in structural biology, enabling detailed 3D imaging of biological specimens at a resolution of nanometers. Despite its potential, cryoET faces challenges such as the missing wedge problem, which limits reconstruction quality due to incomplete data collection angles. Recently, supervised deep learning methods leveraging convolutional neural networks (CNNs) have considerably addressed this issue; however, their pretraining requirements render them susceptible to inaccuracies and artifacts, particularly when representative training data is scarce. To overcome these limitations, we introduce a proof-of-concept unsupervised learning approach using coordinate networks (CNs) that optimizes network weights directly against input projections. This eliminates the need for pretraining, reducing reconstruction runtime by 3-20× compared to supervised methods. Our in silico results show improved shape completion and reduction of missing wedge artifacts, assessed through several voxel-based image quality metrics in real space and a novel directional Fourier Shell Correlation (FSC) metric. Our study illuminates benefits and considerations of both supervised and unsupervised approaches, guiding the development of improved reconstruction strategies.

High-confidence 3D template matching for cryo-electron tomography

Visual proteomics attempts to build atlases of the molecular content of cells but the automated annotation of cryo electron tomograms remains challenging. Template matching (TM) and methods based on machine learning detect structural signatures of macromolecules. However, their applicability remains limited in terms of both the abundance and size of the molecular targets. Here we show that the performance of TM is greatly improved by using template-specific search parameter optimization and by including higher-resolution information. We establish a TM pipeline with systematically tuned parameters for the automated, objective and comprehensive identification of structures with confidence 10 to 100-fold above the noise level. We demonstrate high-fidelity and high-confidence localizations of nuclear pore complexes, vaults, ribosomes, proteasomes, fatty acid synthases, lipid membranes and microtubules, and individual subunits inside crowded eukaryotic cells. We provide software tools for the generic implementation of our method that is broadly applicable towards realizing visual proteomics.

Missing Wedge Completion via Unsupervised Learning with Coordinate Networks

Cryogenic electron tomography (cryoET) is a powerful tool in structural biology, enabling detailed 3D imaging of biological specimens at a resolution of nanometers. Despite its potential, cryoET faces challenges such as the missing wedge problem, which limits reconstruction quality due to incomplete data collection angles. Recently, supervised deep learning methods leveraging convolutional neural networks (CNNs) have considerably addressed this issue; however, their pretraining requirements render them susceptible to inaccuracies and artifacts, particularly when representative training data is scarce. To overcome these limitations, we introduce a proof-of-concept unsupervised learning approach using coordinate networks (CNs) that optimizes network weights directly against input projections. This eliminates the need for pretraining, reducing reconstruction runtime by 3 - 20× compared to supervised methods. Our in silico results show improved shape completion and reduction of missing wedge artifacts, assessed through several voxel-based image quality metrics in real space and a novel directional Fourier Shell Correlation (FSC) metric. Our study illuminates benefits and considerations of both supervised and unsupervised approaches, guiding the development of improved reconstruction strategies.

Extracting nanoscale membrane morphology from single-molecule localizations

Membrane surface reconstruction at the nanometer scale is required for understanding mechanisms of subcellular shape change. This historically has been the domain of electron microscopy, but extraction of surfaces from specific labels is a difficult task in this imaging modality. Existing methods for extracting surfaces from fluorescence microscopy have poor resolution or require high-quality super-resolution data that are manually cleaned and curated. Here, we present NanoWrap, a new method for extracting surfaces from generalized single-molecule localization microscopy data. This makes it possible to study the shape of specifically labeled membranous structures inside cells. We validate NanoWrap using simulations and demonstrate its reconstruction capabilities on single-molecule localization microscopy data of the endoplasmic reticulum and mitochondria. NanoWrap is implemented in the open-source Python Microscopy Environment.

Extracting nanoscale membrane morphology from single-molecule localizations

Membrane surface reconstruction at the nanometer scale is required for understanding mechanisms of subcellular shape change. This historically has been the domain of electron microscopy, but extraction of surfaces from specific labels is a difficult task in this imaging modality. Existing methods for extracting surfaces from fluorescence microscopy have poor resolution or require high-quality super-resolution data that is manually cleaned and curated. Here we present a new method for extracting surfaces from generalized single-molecule localization microscopy (SMLM) data. This makes it possible to study the shape of specifically-labelled membraneous structures inside of cells. We validate our method using simulations and demonstrate its reconstruction capabilities on SMLM data of the endoplasmic reticulum and mitochondria. Our method is implemented in the open-source Python Microscopy Environment.

SIGNIFICANCE

We introduce a novel tool for reconstruction of subcellular membrane surfaces from single-molecule localization microscopy data and use it to visualize and quantify local shape and membrane-membrane interactions. We benchmark its performance on simulated data and demonstrate its fidelity to experimental data.

CryoET reveals organelle phenotypes in huntington disease patient iPSC-derived and mouse primary neurons

Huntington's disease (HD) is caused by an expanded CAG repeat in the huntingtin gene, yielding a Huntingtin protein with an expanded polyglutamine tract. While experiments with patient-derived induced pluripotent stem cells (iPSCs) can help understand disease, defining pathological biomarkers remains challenging. Here, we used cryogenic electron tomography to visualize neurites in HD patient iPSC-derived neurons with varying CAG repeats, and primary cortical neurons from BACHD, deltaN17-BACHD, and wild-type mice. In HD models, we discovered sheet aggregates in double membrane-bound organelles, and mitochondria with distorted cristae and enlarged granules, likely mitochondrial RNA granules. We used artificial intelligence to quantify mitochondrial granules, and proteomics experiments reveal differential protein content in isolated HD mitochondria. Knockdown of Protein Inhibitor of Activated STAT1 ameliorated aberrant phenotypes in iPSC- and BACHD neurons. We show that integrated ultrastructural and proteomic approaches may uncover early HD phenotypes to accelerate diagnostics and the development of targeted therapeutics for HD.

Efficient manual annotation of cryogenic electron tomograms using IMOD

Annotation highlights and segmentation isolates features in cryogenic electron tomograms to improve visualization and quantification of features (for example, their size and abundance, and spatial relationships with other features), facilitating phenotypic structural analyses of cellular tomograms. Here, we present a manual annotation protocol using the open-source software IMOD and describe segmentation of three types of common cellular features: membranes, large globules, and filaments. IMOD's interpolation function can improve the speed of manual annotation up to an order of magnitude.

Cryo-electron tomography provides topological insights into mutant huntingtin exon 1 and polyQ aggregates

Huntington disease (HD) is a neurodegenerative trinucleotide repeat disorder caused by an expanded poly-glutamine (polyQ) tract in the mutant huntingtin (mHTT) protein. The formation and topology of filamentous mHTT inclusions in the brain (hallmarks of HD implicated in neurotoxicity) remain elusive. Using cryo-electron tomography and subtomogram averaging, here we show that mHTT exon 1 and polyQ-only aggregates in vitro are structurally heterogenous and filamentous, similar to prior observations with other methods. Yet, we find filaments in both types of aggregates under ~2 nm in width, thinner than previously reported, and regions forming large sheets. In addition, our data show a prevalent subpopulation of filaments exhibiting a lumpy slab morphology in both aggregates, supportive of the polyQ core model. This provides a basis for future cryoET studies of various aggregated mHTT and polyQ constructs to improve their structure-based modeling as well as their identification in cells without fusion tags.

A massively multi-scale approach to characterizing tissue architecture by synchrotron micro-CT applied to the human placenta

Multi-scale structural assessment of biological soft tissue is challenging but essential to gain insight into structure-function relationships of tissue/organ. Using the human placenta as an example, this study brings together sophisticated sample preparation protocols, advanced imaging and robust, validated machine-learning segmentation techniques to provide the first massively multi-scale and multi-domain information that enables detailed morphological and functional analyses of both maternal and fetal placental domains. Finally, we quantify the scale-dependent error in morphological metrics of heterogeneous placental tissue, estimating the minimal tissue scale needed in extracting meaningful biological data. The developed protocol is beneficial for high-throughput investigation of structure-function relationships in both normal and diseased placentas, allowing us to optimize therapeutic approaches for pathological pregnancies. In addition, the methodology presented is applicable in the characterization of tissue architecture and physiological behaviours of other complex organs with similarity to the placenta, where an exchange barrier possesses circulating vascular and avascular fluid spaces.

Cryogenic Super-Resolution Fluorescence and Electron Microscopy Correlated at the Nanoscale

We review the emerging method of super-resolved cryogenic correlative light and electron microscopy (srCryoCLEM). Super-resolution (SR) fluorescence microscopy and cryogenic electron tomography (CET) are both powerful techniques for observing subcellular organization, but each approach has unique limitations. The combination of the two brings the single-molecule sensitivity and specificity of SR to the detailed cellular context and molecular scale resolution of CET. The resulting correlative data is more informative than the sum of its parts. The correlative images can be used to pinpoint the positions of fluorescently labeled proteins in the high-resolution context of CET with nanometer-scale precision and/or to identify proteins in electron-dense structures. The execution of srCryoCLEM is challenging and the approach is best described as a method that is still in its infancy with numerous technical challenges. In this review, we describe state-of-the-art srCryoCLEM experiments, discuss the most pressing challenges, and give a brief outlook on future applications.

Visualizing Individual RuBisCO and Its Assembly into Carboxysomes in Marine Cyanobacteria by Cryo-Electron Tomography

Cyanobacteria are photosynthetic organisms responsible for ~25% of the organic carbon fixation on earth. A key step in carbon fixation is catalyzed by ribulose bisphosphate carboxylase/oxygenase (RuBisCO), the most abundant enzyme in the biosphere. Applying Zernike phase-contrast electron cryo-tomography and automated annotation, we identified individual RuBisCO molecules and their assembly intermediates leading to the formation of carboxysomes inside Syn5 cyanophage infected cyanobacteria Synechococcus sp. WH8109 cells. Surprisingly, more RuBisCO molecules were found to be present as cytosolic free-standing complexes or clusters than as packaged assemblies inside carboxysomes. Cytosolic RuBisCO clusters and partially assembled carboxysomes identified in the cell tomograms support a concurrent assembly model involving both the protein shell and the enclosed RuBisCO. In mature carboxysomes, RuBisCO is neither randomly nor strictly icosahedrally packed within protein shells of variable sizes. A time-averaged molecular dynamics simulation showed a semi-liquid probability distribution of the RuBisCO in carboxysomes and correlated well with carboxysome subtomogram averages. Our structural observations reveal the various stages of RuBisCO assemblies, which could be important for understanding cellular function.

Cryo-electron tomography for the structural study of mitochondrial translation

Cryo-electron tomography (cryo-ET) enables the three-dimensional (3D) structural characterization of macromolecular complexes in their physiological environment. Thus, cryo-ET is uniquely suited to study the structural basis of biomolecular processes that are extremely difficult or even impossible to reconstitute using purified components. Translation of mitochondrial genes, which occurs in the secluded interior of mitochondria, falls into this category. Here, we describe the principles of cryo-ET in the context of mitochondrial translation and outline recent developments and challenges of the method. The 3D image of a frozen-hydrated biological sample is computed from its 2D projections, which are acquired using a transmission electron microscope. In conjunction with automated detection of different copies of the molecule of interest and averaging of the corresponding subtomograms, cryo-ET enables macromolecular structure determination in the native environment (i.e. in situ) at sub-nanometer resolution. The preservation of the native environment furthermore allows the extraction of contextual information about the molecules, including the location of specific molecules with respect to membranes, their relative positioning and the spatial organization with respect to other types of macromolecules. Recent preparative developments extend the field of application of cryo-ET from isolated organelles to cultured eukaryotic cells and even tissue, making the traditional borders between molecular and cellular structural biology disappear.

Visualizing Individual RuBisCO and Its Assembly into Carboxysomes in Marine Cyanobacteria by Cryo-Electron Tomography

Cyanobacteria are photosynthetic organisms responsible for ~25% of the organic carbon fixation on earth. A key step in carbon fixation is catalyzed by ribulose bisphosphate carboxylase/oxygenase (RuBisCO), the most abundant enzyme in the biosphere. Applying Zernike phase-contrast electron cryo-tomography and automated annotation, we identified individual RuBisCO molecules and their assembly intermediates leading to the formation of carboxysomes inside Syn5 cyanophage infected cyanobacteria Synechococcus sp. WH8109 cells. Surprisingly, more RuBisCO molecules were found to be present as cytosolic free-standing complexes or clusters than as packaged assemblies inside carboxysomes. Cytosolic RuBisCO clusters and partially assembled carboxysomes identified in the cell tomograms support a concurrent assembly model involving both the protein shell and the enclosed RuBisCO. In mature carboxysomes, RuBisCO is neither randomly nor strictly icosahedrally packed within protein shells of variable sizes. A time-averaged molecular dynamics simulation showed a semi-liquid probability distribution of the RuBisCO in carboxysomes and correlated well with carboxysome subtomogram averages. Our structural observations reveal the various stages of RuBisCO assemblies, which could be important for understanding cellular function.

Cryo-soft X-ray tomography: using soft X-rays to explore the ultrastructure of whole cells

Cryo-soft X-ray tomography is an imaging technique that addresses the need for mesoscale imaging of cellular ultrastructure of relatively thick samples without the need for staining or chemical modification. It allows the imaging of cellular ultrastructure to a resolution of 25–40 nm and can be used in correlation with other imaging modalities, such as electron tomography and fluorescence microscopy, to further enhance the information content derived from biological samples. An overview of the technique, discussion of sample suitability and information about sample preparation, data collection and data analysis is presented here. Recent developments and future outlook are also discussed.

Convolutional neural networks for automated annotation of cellular cryo-electron tomograms

Cellular Electron Cryotomography (CryoET) offers the ability to look inside cells and observe macromolecules frozen in action. A primary challenge for this technique is identifying and extracting the molecular components within the crowded cellular environment. We introduce a method using neural networks to dramatically reduce the time and human effort required for subcellular annotation and feature extraction. Subsequent subtomogram classification and averaging yields in-situ structures of molecular components of interest.

Volume Segmentation and Analysis of Biological Materials Using SuRVoS (Super-region Volume Segmentation) Workbench

Segmentation is the process of isolating specific regions or objects within an imaged volume, so that further study can be undertaken on these areas of interest. When considering the analysis of complex biological systems, the segmentation of three-dimensional image data is a time consuming and labor intensive step. With the increased availability of many imaging modalities and with automated data collection schemes, this poses an increased challenge for the modern experimental biologist to move from data to knowledge. This publication describes the use of SuRVoS Workbench, a program designed to address these issues by providing methods to semi-automatically segment complex biological volumetric data. Three datasets of differing magnification and imaging modalities are presented here, each highlighting different strategies of segmenting with SuRVoS. Phase contrast X-ray tomography (microCT) of the fruiting body of a plant is used to demonstrate segmentation using model training, cryo electron tomography (cryoET) of human platelets is used to demonstrate segmentation using super- and megavoxels, and cryo soft X-ray tomography (cryoSXT) of a mammalian cell line is used to demonstrate the label splitting tools. Strategies and parameters for each datatype are also presented. By blending a selection of semi-automatic processes into a single interactive tool, SuRVoS provides several benefits. Overall time to segment volumetric data is reduced by a factor of five when compared to manual segmentation, a mainstay in many image processing fields. This is a significant savings when full manual segmentation can take weeks of effort. Additionally, subjectivity is addressed through the use of computationally identified boundaries, and splitting complex collections of objects by their calculated properties rather than on a case-by-case basis.

The advent of structural biology in situ by single particle cryo-electron tomography

Single particle tomography (SPT), also known as subtomogram averaging, is a powerful technique uniquely poised to address questions in structural biology that are not amenable to more traditional approaches like X-ray crystallography, nuclear magnetic resonance, and conventional cryoEM single particle analysis. Owing to its potential for in situ structural biology at subnanometer resolution, SPT has been gaining enormous momentum in the last five years and is becoming a prominent, widely used technique. This method can be applied to unambiguously determine the structures of macromolecular complexes that exhibit compositional and conformational heterogeneity, both in vitro and in situ. Here we review the development of SPT, highlighting its applications and identifying areas of ongoing development.

SuRVoS: Super-Region Volume Segmentation workbench

Segmentation of biological volumes is a crucial step needed to fully analyse their scientific content. Not having access to convenient tools with which to segment or annotate the data means many biological volumes remain under-utilised. Automatic segmentation of biological volumes is still a very challenging research field, and current methods usually require a large amount of manually-produced training data to deliver a high-quality segmentation. However, the complex appearance of cellular features and the high variance from one sample to another, along with the time-consuming work of manually labelling complete volumes, makes the required training data very scarce or non-existent. Thus, fully automatic approaches are often infeasible for many practical applications. With the aim of unifying the segmentation power of automatic approaches with the user expertise and ability to manually annotate biological samples, we present a new workbench named SuRVoS (Super-Region Volume Segmentation). Within this software, a volume to be segmented is first partitioned into hierarchical segmentation layers (named Super-Regions) and is then interactively segmented with the user's knowledge input in the form of training annotations. SuRVoS first learns from and then extends user inputs to the rest of the volume, while using Super-Regions for quicker and easier segmentation than when using a voxel grid. These benefits are especially noticeable on noisy, low-dose, biological datasets.