Detection of herpesvirus capsids in transmission electron microscopy images using transfer learning

Machine learning for cross-scale microscopy of viruses

Despite advances in virological sciences and antiviral research, viruses continue to emerge, circulate, and threaten public health. We still lack a comprehensive understanding of how cells and individuals remain susceptible to infectious agents. This deficiency is in part due to the complexity of viruses, including the cell states controlling virus-host interactions. Microscopy samples distinct cellular infection stages in a multi-parametric, time-resolved manner at molecular resolution and is increasingly enhanced by machine learning and deep learning. Here we discuss how state-of-the-art artificial intelligence (AI) augments light and electron microscopy and advances virological research of cells. We describe current procedures for image denoising, object segmentation, tracking, classification, and super-resolution and showcase examples of how AI has improved the acquisition and analyses of microscopy data. The power of AI-enhanced microscopy will continue to help unravel virus infection mechanisms, develop antiviral agents, and improve viral vectors.

VISN: virus instance segmentation network for TEM images using deep attention transformer

The identification of viruses from negative staining transmission electron microscopy (TEM) images has mainly depended on experienced experts. Recent advances in artificial intelligence have enabled virus recognition using deep learning techniques. However, most of the existing methods only perform virus classification or semantic segmentation, and few studies have addressed the challenge of virus instance segmentation in TEM images. In this paper, we focus on the instance segmentation of severe acute respiratory syndrome coronavirus type 2 (SARS-CoV-2) and other respiratory viruses and provide experts with more effective information about viruses. We propose an effective virus instance segmentation network based on the You Only Look At CoefficienTs backbone, which integrates the Swin Transformer, dense connections and the coordinate-spatial attention mechanism, to identify SARS-CoV-2, H1N1 influenza virus, respiratory syncytial virus, Herpes simplex virus-1, Human adenovirus type 5 and Vaccinia virus. We also provide a public TEM virus dataset and conduct extensive comparative experiments. Our method achieves a mean average precision score of 83.8 and F1 score of 0.920, outperforming other state-of-the-art instance segmentation algorithms. The proposed automated method provides virologists with an effective approach for recognizing and identifying SARS-CoV-2 and assisting in the diagnosis of viruses. Our dataset and code are accessible at https://github.com/xiaochiHNU/Virus-Instance-Segmentation-Transformer-Network.

Applications of artificial neural networks in microorganism image analysis: a comprehensive review from conventional multilayer perceptron to popular convolutional neural network and potential visual transformer

Microorganisms are widely distributed in the human daily living environment. They play an essential role in environmental pollution control, disease prevention and treatment, and food and drug production. The analysis of microorganisms is essential for making full use of different microorganisms. The conventional analysis methods are laborious and time-consuming. Therefore, the automatic image analysis based on artificial neural networks is introduced to optimize it. However, the automatic microorganism image analysis faces many challenges, such as the requirement of a robust algorithm caused by various application occasions, insignificant features and easy under-segmentation caused by the image characteristic, and various analysis tasks. Therefore, we conduct this review to comprehensively discuss the characteristics of microorganism image analysis based on artificial neural networks. In this review, the background and motivation are introduced first. Then, the development of artificial neural networks and representative networks are presented. After that, the papers related to microorganism image analysis based on classical and deep neural networks are reviewed from the perspectives of different tasks. In the end, the methodology analysis and potential direction are discussed.

Weighted average ensemble-based semantic segmentation in biological electron microscopy images

Semantic segmentation of electron microscopy images using deep learning methods is a valuable tool for the detailed analysis of organelles and cell structures. However, these methods require a large amount of labeled ground truth data that is often unavailable. To address this limitation, we present a weighted average ensemble model that can automatically segment biological structures in electron microscopy images when trained with only a small dataset. Thus, we exploit the fact that a combination of diverse base-learners is able to outperform one single segmentation model. Our experiments with seven different biological electron microscopy datasets demonstrate quantitative and qualitative improvements. We show that the Grad-CAM method can be used to interpret and verify the prediction of our model. Compared with a standard U-Net, the performance of our method is superior for all tested datasets. Furthermore, our model leverages a limited number of labeled training data to segment the electron microscopy images and therefore has a high potential for automated biological applications.

TAIM: Tool for Analyzing Root Images to Calculate the Infection Rate of Arbuscular Mycorrhizal Fungi

Arbuscular mycorrhizal fungi (AMF) infect plant roots and are hypothesized to improve plant growth. Recently, AMF is now available for axenic culture. Therefore, AMF is expected to be used as a microbial fertilizer. To evaluate the usefulness of AMF as a microbial fertilizer, we need to investigate the relationship between the degree of root colonization of AMF and plant growth. The method popularly used for calculation of the degree of root colonization, termed the magnified intersections method, is performed manually and is too labor-intensive to enable an extensive survey to be undertaken. Therefore, we automated the magnified intersections method by developing an application named "Tool for Analyzing root images to calculate the Infection rate of arbuscular Mycorrhizal fungi: TAIM." TAIM is a web-based application that calculates the degree of AMF colonization from images using automated computer vision and pattern recognition techniques. Experimental results showed that TAIM correctly detected sampling areas for calculation of the degree of infection and classified the sampling areas with 87.4% accuracy. TAIM is publicly accessible at http://taim.imlab.jp/.

Applications of deep learning in electron microscopy

We review the growing use of machine learning in electron microscopy (EM) driven in part by the availability of fast detectors operating at kiloHertz frame rates leading to large data sets that cannot be processed using manually implemented algorithms. We summarize the various network architectures and error metrics that have been applied to a range of EM-related problems including denoising and inpainting. We then provide a review of the application of these in both physical and life sciences, highlighting how conventional networks and training data have been specifically modified for EM.

Correlative Light and Electron Microscopy of poly(ʟ-lactic acid) Spherulites for Fast Morphological Measurements using a Convolutional Neural Network

Polarized optical microscopy (POM) and transmission electron microscopy (TEM) are widely used for imaging polymer spherulite structures. TEM provides nanometer resolution to image small spherulites of sub-micrometer in diameter, while POM is more suitable for observing large spherulites. However, high-resolution images with a large field of view are challenging to achieve due to the deformations of ultrathin sectioned samples used for TEM observations. In this study, we demonstrated that correlative light and electron microscopy (CLEM) combining POM and TEM could effectively characterize the spherulite structures in a wide range from nanometer to several hundred micrometers that neither TEM nor POM alone could cover. Furthermore, the deformations of the TEM ultrathin sections were corrected by referencing to the POM images at the same position of the sample, and large-area TEM images without deformations were successfully produced. The spherulite structures of poly(ʟ-lactic acid) were successfully analyzed using CLEM and TEM in a wide range of spatial scales at the same field of view. The large-area TEM image (250 µm × 250 µm), consisting of 702 TEM images stitched together, was subjected to machine learning to extract the essential structural information of spherulites. Analysis using the convolutional neural network, a well-known algorithm You Only Look Once (YOLO), demonstrated that it was practical to accurately obtain the diameter distribution and the space-filling factor (relative crystallinity) of the spherulites. This study presents a new approach for acquiring high-resolution images with a large field of view and processing the images at a fast speed.

CEM500K, a large-scale heterogeneous unlabeled cellular electron microscopy image dataset for deep learning

Automated segmentation of cellular electron microscopy (EM) datasets remains a challenge. Supervised deep learning (DL) methods that rely on region-of-interest (ROI) annotations yield models that fail to generalize to unrelated datasets. Newer unsupervised DL algorithms require relevant pre-training images, however, pre-training on currently available EM datasets is computationally expensive and shows little value for unseen biological contexts, as these datasets are large and homogeneous. To address this issue, we present CEM500K, a nimble 25 GB dataset of 0.5 × 106 unique 2D cellular EM images curated from nearly 600 three-dimensional (3D) and 10,000 two-dimensional (2D) images from >100 unrelated imaging projects. We show that models pre-trained on CEM500K learn features that are biologically relevant and resilient to meaningful image augmentations. Critically, we evaluate transfer learning from these pre-trained models on six publicly available and one newly derived benchmark segmentation task and report state-of-the-art results on each. We release the CEM500K dataset, pre-trained models and curation pipeline for model building and further expansion by the EM community. Data and code are available at https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10592/ and https://git.io/JLLTz.

Segmentation of overlapping chromosome images using U-Net with improved dilated convolutions

Chromosome visualization has been used in human chromosome analysis and is a crucial step in clinical diagnosis and drug development. An important step in chromosome visualization is the extraction of chromosomes from chromosome images obtained by light microscopy. Chromosomes often overlap in a complex and variable manner, resulting in significant challenges in chromosome segmentation. The process of chromosome visualization requires manual intervention and is tedious. A method based on a neural network is proposed for the automatic segmentation of overlapping chromosome images to speed up the workflow of visualizing chromosomes. Three improved dilated convolutions are used in the chromosome image segmentation models based on U-Net. The proposed models successfully segment overlapping chromosomes in two publicly available overlapping chromosome data sets. Our models have better performance than existing overlapping chromosome segmentation methods based on U-Net. In summary, it is demonstrated that the improved dilated convolutions can be used for the automatic segmentation of overlapping chromosome images. The proposed improved dilated convolutions have a stable performance improvement, can be easily extended to the segmentation of multiple overlapping chromosomes, and are suitable as general neural network operations to replace standard convolutions in any network.

Deep Learning for Imaging and Detection of Microorganisms

Despite tremendous recent interest, the application of deep learning in microbiology has still not reached its full potential. To tackle the challenges faced by human-operated microscopy, deep-learning-based methods have been proposed for microscopic image analysis of a wide range of microorganisms, including viruses, bacteria, fungi, and parasites. We believe that deep-learning technology-based systems will be on the front line of monitoring and investigation of microorganisms.

Virus identification in electron microscopy images by residual mixed attention network

BACKGROUND AND OBJECTIVE

Virus identification in electron microscopy (EM) images is considered as one of the front-line method in pathogen diagnosis and re-emerging infectious agents. However, the existing methods either focused on the detection of a single virus or required large amounts of manual labeling work to segment virus. In this work, we focus on the task of virus classification and propose an effective and simple method to identify different viruses.

METHODS

We put forward a residual mixed attention network (RMAN) for virus classification. The proposed network uses channel attention, bottom-up and top-down attention, and incorporates a residual architecture in an end-to-end training manner, which is suitable for dealing with EM virus images and reducing the burden of manual annotation.

RESULTS

We validate the proposed network through extensive experiments on a transmission electron microscopy virus image dataset. The top-1 error rate of our RMAN on 12 virus classes is 4.285%, which surpasses that of state-of-the-art networks and even human experts. In addition, the ablation study and the visualization of class activation mapping (CAM) further demonstrate the effectiveness of our method.

CONCLUSIONS

The proposed automated method contributes to the development of medical virology, which provides virologists with a high-accuracy approach to recognize viruses as well as assist in the diagnosis of viruses.

Quantitative Electron Microscopy to Study HCMV Morphogenesis

The generation and release of mature virions from human cytomegalovirus (HCMV) infected cells is a multistep process, involving a profound reorganization of cellular structures and various stages of virus particle morphogenesis in different cellular compartments. Although the general steps of HCMV morphogenesis are known, it has become clear that the detailed molecular mechanisms are complex and dependent on various viral factors and cellular pathways. The lack of a full understanding of HCMV virion morphogenesis emphasizes the need of imaging techniques to visualize the different stages of virion assembly, such as electron microscopy. Here, we describe various electron microscopy techniques and the methodology of high-pressure freezing and freeze substitution for sample preparation to visualize HCMV morphogenesis. These methods are used in our laboratory in combination with a thorough quantification to characterize phenotypic alterations and to identify the function of viral and cellular proteins for the various morphogenesis stages.

Improved automatic detection of herpesvirus secondary envelopment stages in electron microscopy by augmenting training data with synthetic labelled images generated by a generative adversarial network

Detailed analysis of secondary envelopment of the herpesvirus human cytomegalovirus (HCMV) by transmission electron microscopy (TEM) is crucial for understanding the formation of infectious virions. Here, we present a convolutional neural network (CNN) that automatically recognises cytoplasmic capsids and distinguishes between three HCMV capsid envelopment stages in TEM images. 315 TEM images containing 2,610 expert-labelled capsids of the three classes were available for CNN training. To overcome the limitation of small training datasets and thus poor CNN performance, we used a deep learning method, the generative adversarial network (GAN), to automatically increase our labelled training dataset with 500 synthetic images and thus to 9,192 labelled capsids. The synthetic TEM images were added to the ground truth dataset to train the Faster R-CNN deep learning-based object detector. Training with 315 ground truth images yielded an average precision (AP) of 53.81% for detection, whereas the addition of 500 synthetic training images increased the AP to 76.48%. This shows that generation and additional use of synthetic labelled images for detector training is an inexpensive way to improve detector performance. This work combines the gold standard of secondary envelopment research with state-of-the-art deep learning technology to speed up automatic image analysis even when large labelled training datasets are not available.