An automatic method of detecting and tracking fiducial markers for alignment in electron tomography

Robust projection parameter calibration in cryo-ET with L1-norm optimization

Fiducial marker-based alignment in cryo-electron tomography (cryo-ET) has been extensively studied over a long period. The calibration of projection parameters using nonlinear least squares technique methodologies stands as the ultimate and pivotal stage in the alignment procedure. The efficacy of calibration is substantially impacted by noise and outliers in the marker data obtained from previous steps. Several robust fitting methods have been explored and implemented to address this issue by improving marker data or assigning weights to markers. However, these methods have their own limitations and often assume general Gaussian noise assumption, which may not accurately represent the distribution of noise and outliers in the marker data. In this work, we propose a robust projection parameter calibration model based on L1-norm optimization under Laplace noise assumption in order to overcome the limitations of existing methods. To efficiently solve the problem, we also design a faster and stabler first-order non-sparse method based on smooth approximation strategy. Additionally, we introduce subgradient and subdifferential for mathematical analysis. The accuracy, robustness, and efficacy of our approach are demonstrated through both simulated and real-world experiments.

MarkerDetector: A method for robust fiducial marker detection in electron micrographs using wavelet-based template

Fiducial marker detection in electron micrographs becomes an important and challenging task with the development of large-field electron microscopy. The fiducial marker detection plays an important role in several steps during the process of electron micrographs, such as the alignment and parameter calibrations. However, limited by the conditions of low signal-to-noise ratio (SNR) in the electron micrographs, the performance of fiducial marker detection is severely affected. In this work, we propose the MarkerDetector, a novel algorithm for detecting fiducial markers in electron micrographs. The proposed MarkerDetector is built upon the following contributions: Firstly, a wavelet-based template generation algorithm is devised in MarkerDetector. By adopting a shape-based criterion, a high-quality template can be obtained. Secondly, a robust marker determination strategy is devised by utilizing statistic-based filtering, which can guarantee the correctness of the detected fiducial markers. The average running time of our algorithm is 1.67 seconds with promising accuracy, indicating its practical feasibility for applications in electron micrographs.

A general image misalignment correction method for tomography experiments

Tomography experiments generate three-dimensional (3D) reconstructed slices from a series of two-dimensional (2D) projection images. However, the mechanical system generates joint offsets that result in unaligned 2D projections. This misalignment affects the reconstructed images and reduces their actual spatial resolution. In this study, we present a novel method called outer contour-based misalignment correction (OCMC) for correcting image misalignments in tomography. We use the sample's outer contour structure as auxiliary information to estimate the extent of misalignment in each image. This method is generic and can be used with various tomography imaging techniques. We validated our method with five datasets collected from different samples and across various tomography techniques. The OCMC method demonstrated significant advantages in terms alignment accuracy and time efficiency. As an end-to-end correction method, OCMC can be easily integrated into an online tomography data processing pipeline and facilitate feedback control in future synchrotron tomography experiments.

Cones and spirals: Multi-axis acquisition for scalar and vector electron tomography

Electron tomography (ET) has become an important tool for understanding the 3D nature of nanomaterials, with recent developments enabling not only scalar reconstructions of electron density, but also vector reconstructions of magnetic fields. However, whilst new signals have been incorporated into the ET toolkit, the acquisition schemes have largely kept to conventional single-axis tilt series for scalar ET, and dual-axis schemes for magnetic vector ET. In this work, we explore the potential of using multi-axis tilt schemes including conical and spiral tilt schemes to improve reconstruction fidelity in scalar and magnetic vector ET. Through a combination of systematic simulations and a proof-of-concept experiment, we show that spiral and conical tilt schemes have the potential to produce substantially improved reconstructions, laying the foundations of a new approach to electron tomography acquisition and reconstruction.

Numerical methods for CT reconstruction with unknown geometry parameters

Computed tomography (CT) techniques are well known for their ability to produce high-quality images needed for medical diagnostic purposes. Unfortunately, standard CT machines are extremely large, heavy, require careful and regular calibration, and are expensive, which can limit their availability in point-of-care situations. An alternative approach is to use portable machines, but parameters related to the geometry of these devices (e.g., distance between source and detector, orientation of source to detector) cannot always be precisely calibrated, and these parameters may change slightly when the machine is adjusted during the image acquisition process. In this work, we describe the non-linear inverse problem that models this situation, and discuss algorithms that can jointly estimate the geometry parameters and compute a reconstructed image. In particular, we propose a hybrid machine learning and block coordinate descent (ML-BCD) approach that uses an ML model to calibrate geometry parameters, and uses BCD to refine the predicted parameters and reconstruct the imaged object simultaneously. We show using numerical experiments that our new method can efficiently improve the accuracy of both the image and geometry parameters.

Feature detection network-based correction method for accurate nano-tomography reconstruction

Driven by the development of advanced x-ray optics such as Fresnel zone plates, nano-resolution full-field transmission x-ray microscopy (Nano-CT) has become a powerful technique for the non-destructive volumetric inspection of objects and has long been developed at different synchrotron radiation facilities. However, Nano-CT data are often associated with random sample jitter because of the drift or radial/axial error motion of the rotation stage during measurement. Without a proper sample jitter correction process prior to reconstruction, the use of Nano-CT in providing accurate 3D structure information for samples is almost impossible. In this paper, to realize accurate 3D reconstruction for Nano-CT, a correction method based on a feature detection neural network, which can automatically extract target features from a projective image and precisely correct sample jitter errors, is proposed, thereby resulting in high-quality nanoscale 3D reconstruction. Compared with other feature detection methods, even if the target feature is overlapped by other high-density materials or impurities, the proposed Nano-CT correction method still acquires sub-pixel accuracy in geometrical correction and is more suitable for Nano-CT reconstruction because of its universal and faster correction speed. The simulated and experimental datasets demonstrated the reliability and validity of the proposed Nano-CT correction method.

Joint iterative reconstruction and 3D rigid alignment for X-ray tomography

X-ray tomography is widely used for three-dimensional structure determination in many areas of science, from the millimeter to the nanometer scale. The resolution and quality of the 3D reconstruction is limited by the availability of alignment parameters that correct for the mechanical shifts of the sample or sample stage for the images that constitute a scan. In this paper we describe an algorithm for marker-free, fully automated and accurately aligned and reconstructed X-ray tomography data. Our approach solves the tomographic reconstruction jointly with projection data alignment based on a rigid-body deformation model. We demonstrate the robustness of our method on both synthetic phantom and experimental data and show that our method is highly efficient in recovering relatively large alignment errors without prior knowledge of a low resolution approximation of the 3D structure or a reasonable estimate of alignment parameters.

Optimization-based simultaneous alignment and reconstruction in multi-element tomography

As x-ray microscopy is pushed into the nanoscale with the advent of more bright and coherent x-ray sources, associated improvement in spatial resolution becomes highly vulnerable to geometrical errors and uncertainties during data collection. We address a form of error in tomography experiments, namely, the drift between projections during the tomographic scan. Our proposed method can simultaneously recover the drift, while tomographically reconstructing the specimen based on a joint iterative optimization scheme. This approach utilizes the correlation provided from different view angles and different signals. While generally applicable, we demonstrate our method on x-ray fluorescence tomography from a tissue specimen and compare the reconstruction quality with conventional methods.

A fast fiducial marker tracking model for fully automatic alignment in electron tomography

Motivation

Automatic alignment, especially fiducial marker-based alignment, has become increasingly important due to the high demand of subtomogram averaging and the rapid development of large-field electron microscopy. Among the alignment steps, fiducial marker tracking is a crucial one that determines the quality of the final alignment. Yet, it is still a challenging problem to track the fiducial markers accurately and effectively in a fully automatic manner.

Results

In this paper, we propose a robust and efficient scheme for fiducial marker tracking. Firstly, we theoretically prove the upper bound of the transformation deviation of aligning the positions of fiducial markers on two micrographs by affine transformation. Secondly, we design an automatic algorithm based on the Gaussian mixture model to accelerate the procedure of fiducial marker tracking. Thirdly, we propose a divide-and-conquer strategy against lens distortions to ensure the reliability of our scheme. To our knowledge, this is the first attempt that theoretically relates the projection model with the tracking model. The real-world experimental results further support our theoretical bound and demonstrate the effectiveness of our algorithm. This work facilitates the fully automatic tracking for datasets with a massive number of fiducial markers.

Availability and implementation

The C/C ++ source code that implements the fast fiducial marker tracking is available at https://github.com/icthrm/gmm-marker-tracking. Markerauto 1.6 version or later (also integrated in the AuTom platform at http://ear.ict.ac.cn/) offers a complete implementation for fast alignment, in which fast fiducial marker tracking is available by the '-t' option.

Contact

xin.gao@kaust.edu.sa.

Supplementary information

Supplementary data are available at Bioinformatics online.

High-Throughput Electron Cryo-tomography of Protein Complexes and Their Assembly

Electron cryo-tomography and subtomogram averaging enable visualization of protein complexes in situ, in three dimensions, in a near-native frozen-hydrated state to nanometer resolutions. To achieve this, intact cells are vitrified and imaged over a range of tilts within an electron microscope. These images can subsequently be reconstructed into a three-dimensional volume representation of the sample cell. Because complexes are visualized in situ, crucial insights into their mechanism, assembly process, and dynamic interactions with other proteins become possible. To illustrate the electron cryo-tomography workflow for visualizing protein complexes in situ, we describe our workflow of preparing samples, imaging, and image processing using Leginon for data collection, IMOD for image reconstruction, and PEET for subtomogram averaging.

Rapid alignment of nanotomography data using joint iterative reconstruction and reprojection

As x-ray and electron tomography is pushed further into the nanoscale, the limitations of rotation stages become more apparent, leading to challenges in the alignment of the acquired projection images. Here we present an approach for rapid post-acquisition alignment of these projections to obtain high quality three-dimensional images. Our approach is based on a joint estimation of alignment errors, and the object, using an iterative refinement procedure. With simulated data where we know the alignment error of each projection image, our approach shows a residual alignment error that is a factor of a thousand smaller, and it reaches the same error level in the reconstructed image in less than half the number of iterations. We then show its application to experimental data in x-ray and electron nanotomography.

3D reconstruction of biological structures: automated procedures for alignment and reconstruction of multiple tilt series in electron tomography

Transmission electron microscopy allows the collection of multiple views of specimens and their computerized three-dimensional reconstruction and analysis with electron tomography. Here we describe development of methods for automated multi-tilt data acquisition, tilt-series processing, and alignment which allow assembly of electron tomographic data from a greater number of tilt series, yielding enhanced data quality and increasing contrast associated with weakly stained structures. This scheme facilitates visualization of nanometer scale details of fine structure in volumes taken from plastic-embedded samples of biological specimens in all dimensions. As heavy metal-contrasted plastic-embedded samples are less sensitive to the overall dose rather than the electron dose rate, an optimal resampling of the reconstruction space can be achieved by accumulating lower dose electron micrographs of the same area over a wider range of specimen orientations. The computerized multiple tilt series collection scheme is implemented together with automated advanced procedures making collection, image alignment, and processing of multi-tilt tomography data a seamless process. We demonstrate high-quality reconstructions from samples of well-described biological structures. These include the giant Mimivirus and clathrin-coated vesicles, imaged in situ in their normal intracellular contexts. Examples are provided from samples of cultured cells prepared by high-pressure freezing and freeze-substitution as well as by chemical fixation before epoxy resin embedding.

Marker Detection in Electron Tomography: A Comparative Study

We conducted a comparative study of three widely used algorithms for the detection of fiducial markers in electron microscopy images. The algorithms were applied to four datasets from different sources. For the purpose of obtaining comparable results, we introduced figures of merit and implemented all three algorithms in a unified code base to exclude software-specific differences. The application of the algorithms revealed that none of the three algorithms is superior to the others in all cases. This leads to the conclusion that the choice of a marker detection algorithm highly depends on the properties of the dataset to be analyzed, even within the narrowed domain of electron tomography.

A new field-of-view autotracking method for online tomography reconstruction based on back-projected ray image cross-correlation

We devised a new field-of-view autotracking method for online tomography reconstruction based on a cross-correlation between a pair of neighbours, called back-projected ray images, among a specimen tilt sequence. One ray image is calculated through normal filtered back-projection only in the cross-sectional plane from each projection image. This ray-image matching can reliably track the field-of-view because a pair of neighbouring ray images mostly cross-correlates at the existing three-dimensional object position. Online experiments using real specimens resulted in successful autotracking performance with high accuracy, and online tomograms were obtained immediately after the final tracking at the last tilting angle.

Alternative automatic alignment method for specimen tilt-series images based on back-projected volume data cross-correlations

We devised a new automatic image alignment method for a specimen tilt series; this method is based on the volume data cross-correlation among 3-D cross-sections reconstructed from different sets of projection images (including a single image) for tilt-series alignment or tilt-axis search purposes. This method requires neither markers nor image feature points traceable through the tilt series, and it was examined through simulations and applied to biological thin sections. The method automatically aligned tilt series centred at the correctly detected tilt axis with a precision sufficient for practical applications.

Rapid method for electron tomographic reconstruction and three-dimensional modeling of the murine synapse using an automated fiducial marker-free system

Electron tomography (ET) has recently afforded new insights into neuronal architecture. However, the tedious process of sample preparation, image acquisition, alignment, back projection, and additional segmentation process of ET repels beginners. We have tried Hitachi's commercial packages integrated with a Hitachi H-7650 TEM to examine the potential of using an automated fiducial-less approach for our own neuroscience research. Semi-thick sections (200-300 nm) were cut from blocks of fixed mouse (C57BL) cerebellum and prepared for ET. Sets of images were collected automatically as each section was tilted by 2° increments (±60°). "Virtual" image volumes were computationally reconstructed in three dimension (3D) with the EMIP software using either the commonly used "weighted back-projection" (WBP) method or "topography-based reconstruction" (TBR) algorithm for comparison. Computed tomograms using the TBR were more precisely reconstructed compared with the WBP method. Following reconstruction, the image volumes were imported into the 3D editing software A-View and segmented according to synaptic organization. The detailed synaptic components were revealed by very thin virtual image slices; 3D models of synapse structure could be constructed efficiently. Overall, this simplified system provided us with a graspable tool for pursuing ET studies in neuroscience.

High-precision alignment of electron tomography tilt series using markers formed in helium-ion microscope

Tungsten nanodots formed in a helium-ion microscope (HIM) provide a practical means of aligning markers of electron tomography tilt series with a high degree of precision. The nanodots were formed using a HIM equipped with a W(CO)6 gas injection system, enabling the precise placement of the nanodots at desired locations of a sample. Template matching was applied to the markers formed in the HIM to detect the positions automatically. The relation between the positions of the markers and the accuracy of the alignment was also determined in order to achieve precise alignment. The method was applied to the markers in order to reconstruct three-dimensional (3D) images of a rod-shaped specimen that contained a 65-nm-diameter via structure in a Cu/Low-k interconnect.