In search of best automated model: Explaining nanoparticle TEM image segmentation

Exploiting Temporal Features in Calculating Automated Morphological Properties of Spiky Nanoparticles Using Deep Learning

Object segmentation in images is typically spatial and focuses on the spatial coherence of pixels. Nanoparticles in electron microscopy images are also segmented frame by frame, with subsequent morphological analysis. However, morphological analysis is inherently sequential, and a temporal regularity is evident in the process. In this study, we extend the spatially focused morphological analysis by incorporating a fusion of hard and soft inductive bias from sequential machine learning techniques to account for temporal relationships. Previously, spiky Au nanoparticles (Au-SNPs) in electron microscopy images were analyzed, and their morphological properties were automatically generated using a hourglass convolutional neural network architecture. In this study, recurrent layers are integrated to capture the natural, sequential growth of the particles. The network is trained with a spike-focused loss function. Continuous segmentation of the images explores the regressive relationships among natural growth features, generating morphological statistics of the nanoparticles. This study comprehensively evaluates the proposed approach by comparing the results of segmentation and morphological properties analysis, demonstrating its superiority over earlier methods.

An approach of separating the overlapped cells or nuclei based on the outer Canny edges and morphological erosion

In biomedicine, the automatic processing of medical microscope images plays a key role in the subsequent analysis and diagnosis. Cell or nucleus segmentation is one of the most challenging tasks for microscope image processing. Due to the frequently occurred overlapping, few segmentation methods can achieve satisfactory segmentation accuracy yet. In this paper, we propose an approach to separate the overlapped cells or nuclei based on the outer Canny edges and morphological erosion. The threshold selection is first used to segment the foreground and background of cell or nucleus images. For each binary connected domain in the segmentation image, an intersection based edge selection method is proposed to choose the outer Canny edges of the overlapped cells or nuclei. The outer Canny edges are used to generate a binary cell or nucleus image that is then used to compute the cell or nucleus seeds by the proposed morphological erosion method. The nuclei of the Human U2OS cells, the mouse NIH3T3 cells and the synthetic cells are used for evaluating our proposed approach. The quantitative quantification accuracy is computed by the Dice score and 95.53% is achieved by the proposed approach. Both the quantitative and the qualitative comparisons show that the accuracy of the proposed approach is better than those of the area constrained morphological erosion (ACME) method, the iterative erosion (IE) method, the morphology and watershed (MW) method, the Generalized Laplacian of Gaussian filters (GLGF) method and ellipse fitting (EF) method in separating the cells or nuclei in three publicly available datasets.

Segmentability evaluation of back-scattered SEM images of multiphase materials

Segmentation methods are very useful tools in the Electron Microscopy inspection of materials, enabling the extraction of quantitative results from microscopy images. Back-Scattered Electron (BSE) images carry information of the mean atomic number in the interaction volume and hence can be used to quantify the phase composition in multiphase materials. Since phase composition and proportion affects the material properties and hence its applications, the segmentation accuracy of such images rendered of critical importance for material science. In this work, the notion of segmentability for BSE images is proposed to define the ability of an image to be segmented accurately. This notion can be used to guide the image acquisition process so that segmentability is maximized and segmentation accuracy is ensured. An index is devised to quantify segmentability based on a combination of the modified Fisher Discrimination Ratio and of the second Minkowski functional capturing intensity and spatial aspects of BSE images respectively. The suggested Segmentability Index (SI) is validated in synthetic BSE images which are generated with a novel algorithm allowing the independent control of spatial distribution of phases and their grayscale intensity histograms. Additionally, SI is applied in real-synthetic BSE images, where the real greyscale distributions of Ordinary Portland Cement (OPC) clinker crystallographic phases are used, to demonstrate the ability of SI to indicate the optimum choice of critical image acquisition settings leading to the more accurate segmentation output.

Quantifying noise limitations of neural network segmentations in high-resolution transmission electron microscopy

Motivated by the need for low electron dose transmission electron microscopy imaging, we report the optimal frame dose (i.e.e-/Å2) range for object detection and segmentation tasks with neural networks. The MSD-net architecture shows promising abilities over the industry standard U-net architecture in generalising to frame doses below the range included in the training set, for both simulated and experimental images. It also presents a heightened ability to learn from lower dose images. The MSD-net displays mild visibility of a Au nanoparticle at 20-30 e-/Å2, and converges at 200 e-/Å2 where a full segmentation of the nanoparticle is achieved. Between 30 and 200 e-/Å2 object detection applications are still possible. This work also highlights the importance of modelling the modulation transfer function when training with simulated images for applications on images acquired with scintillator based detectors such as the Gatan Oneview camera. A parametric form of the modulation transfer function is applied with varying ranges of parameters, and the effects on low electron dose segmentation is presented.

Deep Learning-Based TEM Image Analysis for Fully Automated Detection of Gold Nanoparticles Internalized Within Tumor Cell

Transmission electron microscopy (TEM) imaging can be used for detection/localization of gold nanoparticles (GNPs) within tumor cells. However, quantitative analysis of GNP-containing cellular TEM images typically relies on conventional/thresholding-based methods, which are manual, time-consuming, and prone to human errors. In this study, therefore, deep learning (DL)-based methods were developed for fully automated detection of GNPs from cellular TEM images. Several models of "you only look once (YOLO)" v5 were implemented, with a few adjustments to enhance the model's performance by applying the transfer learning approach, adjusting the size of the input image, and choosing the best optimization algorithm. Seventy-eight original (12,040 augmented) TEM images of GNP-laden tumor cells were used for model implementation and validation. A maximum F1 score (harmonic mean of the precision and recall) of 0.982 was achieved by the best-trained models, while mean average precision was 0.989 and 0.843 at 0.50 and 0.50-0.95 intersection over union threshold, respectively. These results suggested the developed DL-based approach was capable of precisely estimating the number/position of internalized GNPs from cellular TEM images. A novel DL-based TEM image analysis tool from this study will benefit research/development efforts on GNP-based cancer therapeutics, for example, by enabling the modeling of GNP-laden tumor cells using nanometer-resolution TEM images.

Segmentation and Morphology Computation of a Spiky Nanoparticle Using the Hourglass Neural Network

Morphological measurements of nanoparticles in electron microscopy images are tedious, laborious, and often succumb to human errors. Deep learning methods in artificial intelligence (AI) paved the way for automated image understanding. This work proposes a deep neural network (DNN) for the automated segmentation of a Au spiky nanoparticle (SNP) in electron microscopic images, and the network is trained with a spike-focused loss function. The segmented images are used for the growth measurement of the Au SNP. The auxiliary loss function captures the spikes of the nanoparticle, which prioritizes the detection of spikes in the border regions. The growth of the particles measured by the proposed DNN is as good as the measurement in manually segmented images of the particles. The proposed DNN composition with the training methodology meticulously segments the particle and consequently provides accurate morphological analysis. Furthermore, the proposed network is tested on an embedded system for integration with the microscope hardware for real-time morphological analysis.

Deep learning for automated size and shape analysis of nanoparticles in scanning electron microscopy

The automated analysis of nanoparticles, imaged by scanning electron microscopy, was implemented by a deep-learning (artificial intelligence) procedure based on convolutional neural networks (CNNs). It is possible to extract quantitative information on particle size distributions and particle shapes from pseudo-three-dimensional secondary electron micrographs (SE) as well as from two-dimensional scanning transmission electron micrographs (STEM). After separation of particles from the background (segmentation), the particles were cut out from the image to be classified by their shape (e.g. sphere or cube). The segmentation ability of STEM images was considerably enhanced by introducing distance- and intensity-based pixel weight loss maps. This forced the neural network to put emphasis on areas which separate adjacent particles. Partially covered particles were recognized by training and excluded from the analysis. The separation of overlapping particles, quality control procedures to exclude agglomerates, and the computation of quantitative particle size distribution data (equivalent particle diameter, Feret diameter, circularity) were included into the routine.

A deep learning-based framework for automatic analysis of the nanoparticle morphology in SEM/TEM images

Scanning electron microscopy (SEM) and transmission electron microscopy (TEM) are important tools for characterizing nanomaterial morphology. Automatic analysis of the nanomaterial morphology in SEM/TEM images plays a crucial role in accelerating research on nanomaterials science. However, achieving a high-throughput automated online statistical analysis of the nanomaterial morphology in various complex SEM/TEM images is still a challenging task. In this paper, we propose a universal framework based on deep learning to perform a fast and accurate online statistical analysis of the nanoparticle morphology in complex SEM/TEM images. The proposed framework consists of three stages that are nanoparticle segmentation using a powerful light-weight deep learning network (NSNet), nanoparticle shape extraction, and statistical analysis. The experimental results show that NSNet in the proposed framework has achieved an accuracy of 86.2% and can process 11 SEM/TEM images per second on an embedded processor. Compared with other semantic segmentation models, NSNet is an optimal choice to ensure that the proposed framework still achieves accurate and fast segmentation even in SEM/TEM images with high background interference, extremely small nanoparticles and dense nanoparticles. Meanwhile, the equivalent diameter and Blaschke shape coefficient of the nanoparticle obtained by the proposed framework are 17.14 ± 5.9 and 0.18 ± 0.04, which are well consistent with those of manual statistical analysis. In short, the proposed framework has a promising future in driving the development of automatic and intelligent analysis technology for nanomaterial morphology.