Segmentation, Inference and Classification of Partially Overlapping Nanoparticles

MCount: An automated colony counting tool for high-throughput microbiology

Accurate colony counting is crucial for assessing microbial growth in high-throughput workflows. However, existing automated counting solutions struggle with the issue of merged colonies, a common occurrence in high-throughput plating. To overcome this limitation, we propose MCount, the only known solution that incorporates both contour information and regional algorithms for colony counting. By optimizing the pairing of contours with regional candidate circles, MCount can accurately infer the number of merged colonies. We evaluate MCount on a precisely labeled Escherichia coli dataset of 960 images (15,847 segments) and achieve an average error rate of 3.99%, significantly outperforming existing published solutions such as NICE (16.54%), AutoCellSeg (33.54%), and OpenCFU (50.31%). MCount is user-friendly as it only requires two hyperparameters. To further facilitate deployment in scenarios with limited labeled data, we propose statistical methods for selecting the hyperparameters using few labeled or even unlabeled data points, all of which guarantee consistently low error rates. MCount presents a promising solution for accurate and efficient colony counting in application workflows requiring high throughput, particularly in cases with merged colonies.

Automated analysis of transmission electron micrographs of metallic nanoparticles by machine learning

Metallic nanoparticles were analysed with respect to size and shape by a machine learning approach. This involved a separation of particles from the background (segmentation), a separation of overlapping particles, and the identification of individual particles. An algorithm to separate overlapping particles, based on ultimate erosion of convex shapes (UECS), was implemented. Finally, particle properties like size, circularity, equivalent diameter, and Feret diameter were computed for each particle of the whole particle population. Thus, particle size distributions can be easily created based on the various parameters. However, strongly overlapping particles are difficult and sometimes impossible to separate because of an a priori unknown shape of a particle that is partially lying in the shadow of another particle. The program is able to extract information from a sequence of images of the same sample, thereby increasing the number of analysed nanoparticles to several thousands. The machine learning approach is well-suited to identify particles at only limited particle-to-background contrast as is demonstrated for ultrasmall gold nanoparticles (2 nm).

Creating ground truth for nanocrystal morphology: a fully automated pipeline for unbiased transmission electron microscopy analysis

Control over colloidal nanocrystal morphology (size, size distribution, and shape) is important for tailoring the functionality of individual nanocrystals and their ensemble behavior. Despite this, traditional methods to quantify nanocrystal morphology are laborious. New developments in automated morphology classification will accelerate these analyses but the assessment of machine learning models is limited by human accuracy for ground truth, causing even unsupervised machine learning models to have inherent bias. Herein, we introduce synthetic image rendering to solve the ground truth problem of nanocrystal morphology classification. By simulating 2D images of nanocrystal shapes via a function of high-dimensional parameter space, we trained a convolutional neural network to link unique morphologies to their simulated parameters, defining nanocrystal morphology quantitatively rather than qualitatively. An automated pipeline then processes, quantitatively defines, and classifies nanocrystal morphology from experimental transmission electron microscopy (TEM) images. Using improved computer vision techniques, 42 650 nanocrystals were identified, assessed, and labeled with quantitative parameters, offering a 600-fold improvement in efficiency over best-practice manual measurements. A classification algorithm was trained with a prediction accuracy of 99.5%, which can successfully analyze a range of concave, convex, and irregular nanocrystal shapes. The resulting pipeline was applied to differentiating two syntheses of nominally cuboidal CsPbBr₃ nanocrystals and uniquely classifying binary nickel sulfide nanocrystal phase based on morphology. This pipeline provides a simple, efficient, and unbiased method to quantify nanocrystal morphology and represents a practical route to construct large datasets with an absolute ground truth for training unbiased morphology-based machine learning algorithms.

Trainable Segmentation for Transmission Electron Microscope Images of Inorganic Nanoparticles

We present a trainable segmentation method implemented within the python package ParticleSpy. The method takes user labelled pixels, which are used to train a classifier and segment images of inorganic nanoparticles from transmission electron microscope images. This implementation is based on the trainable Waikato Environment for Knowledge Analysis (WEKA) segmentation, but is written in python, allowing a large degree of flexibility and meaning it can be easily expanded using other python packages. We find that trainable segmentation offers better accuracy than global or local thresholding methods and requires as few as 100 user-labelled pixels to produce an accurate segmentation. Trainable segmentation presents a balance of accuracy and training time between global/local thresholding and neural networks, when used on transmission electron microscope images of nanoparticles. We also quantitatively investigate the effectiveness of the components of trainable segmentation, its filter kernels and classifiers, in order to demonstrate the use cases for the different filter kernels in ParticleSpy and the most accurate classifiers for different data types. A set of filter kernels is identified that are effective in distinguishing particles from background but that retain dissimilar features. In terms of classifiers, we find that different classifiers perform optimally for different image contrast; specifically, a Random Forest classifier performs best for high-contrast ADF images, but that QDA and Gaussian Naïve Bayes classifiers perform better for low-contrast TEM images. This article is protected by copyright. All rights reserved.

In search of best automated model: Explaining nanoparticle TEM image segmentation

Over the years, computer scientists are working on building models that aid the scientific community in many ways by cutting laboratory expenses or by saving time. Such models find useful applications in microscopy images as well. Determining the morphology of nanoparticles from Transmission Electron Microscopy images is a manual and cumbersome task. In the past years, scientists have tried to build models to automate the process of nanoparticle segmentation in microscopy images. This study focuses on finding the best segmentation model, which achieves high metrics and is robust to microscopy parameters. For this purpose, eight different models have been compared. The training dataset consists of 150 BF-TEM Platinum nanoparticle images containing 3629 nanoparticles of all kinds. Further, we examine the generalizability of the models on E-TEM Gold nanoparticle images. We also describe essential considerations while choosing a network for segmenting nanoparticle images that generalize well across the Platinum BF-TEM and Gold E-TEM nanoparticles dataset. The layer gradients are visualized to further explain the black-box nature of neural networks.

Statistically Representative Metrology of Nanoparticles via Unsupervised Machine Learning of TEM Images

The morphology of nanoparticles governs their properties for a range of important applications. Thus, the ability to statistically correlate this key particle performance parameter is paramount in achieving accurate control of nanoparticle properties. Among several effective techniques for morphological characterization of nanoparticles, transmission electron microscopy (TEM) can provide a direct, accurate characterization of the details of nanoparticle structures and morphology at atomic resolution. However, manually analyzing a large number of TEM images is laborious. In this work, we demonstrate an efficient, robust and highly automated unsupervised machine learning method for the metrology of nanoparticle systems based on TEM images. Our method not only can achieve statistically significant analysis, but it is also robust against variable image quality, imaging modalities, and particle dispersions. The ability to efficiently gain statistically significant particle metrology is critical in advancing precise particle synthesis and accurate property control.

Recognition of overlapping elliptical objects in a binary image

Recognition of overlapping objects is required in many applications in the field of computer vision. Examples include cell segmentation, bubble detection and bloodstain pattern analysis. This paper presents a method to identify overlapping objects by approximating them with ellipses. The method is intended to be applied to complex-shaped regions which are believed to be composed of one or more overlapping objects. The method has two primary steps. First, a pool of candidate ellipses are generated by applying the Euclidean distance transform on a compressed image and the pool is filtered by an overlaying method. Second, the concave points on the contour of the region of interest are extracted by polygon approximation to divide the contour into segments. Then, the optimal ellipses are selected from among the candidates by choosing a minimal subset that best fits the identified segments. We propose the use of the adjusted Rand index, commonly applied in clustering, to compare the fitting result with ground truth. Through a set of computational and optimization efficiencies, we are able to apply our approach in complex images comprised of a number of overlapped regions. Experimental results on a synthetic data set, two types of cell images and bloodstain patterns show superior accuracy and flexibility of our method in ellipse recognition, relative to other methods.

A Close Look at Molecular Self-Assembly with the Transmission Electron Microscope

Molecular self-assembly is pervasive in the formation of living and synthetic materials. Knowledge gained from research into the principles of molecular self-assembly drives innovation in the biological, chemical, and materials sciences. Self-assembly processes span a wide range of temporal and spatial domains and are often unintuitive and complex. Studying such complex processes requires an arsenal of analytical and computational tools. Within this arsenal, the transmission electron microscope stands out for its unique ability to visualize and quantify self-assembly structures and processes. This review describes the contribution that the transmission electron microscope has made to the field of molecular self-assembly. An emphasis is placed on which TEM methods are applicable to different structures and processes and how TEM can be used in combination with other experimental or computational methods. Finally, we provide an outlook on the current challenges to, and opportunities for, increasing the impact that the transmission electron microscope can have on molecular self-assembly.

Synthetic Image Rendering Solves Annotation Problem in Deep Learning Nanoparticle Segmentation

Nanoparticles occur in various environments as a consequence of man-made processes, which raises concerns about their impact on the environment and human health. To allow for proper risk assessment, a precise and statistically relevant analysis of particle characteristics (such as size, shape, and composition) is required that would greatly benefit from automated image analysis procedures. While deep learning shows impressive results in object detection tasks, its applicability is limited by the amount of representative, experimentally collected and manually annotated training data. Here, an elegant, flexible, and versatile method to bypass this costly and tedious data acquisition process is presented. It shows that using a rendering software allows to generate realistic, synthetic training data to train a state-of-the art deep neural network. Using this approach, a segmentation accuracy can be derived that is comparable to man-made annotations for toxicologically relevant metal-oxide nanoparticle ensembles which were chosen as examples. The presented study paves the way toward the use of deep learning for automated, high-throughput particle detection in a variety of imaging techniques such as in microscopies and spectroscopies, for a wide range of applications, including the detection of micro- and nanoplastic particles in water and tissue samples.

Robust Segmentation-Free Algorithm for Homogeneity Quantification in Images

OBJECTIVE

Homogeneity is a notion used to describe images in various fields and is often linked to critical aspects of those fields. However, this term is rarely defined in the literature and no gold standard exists for its quantification. A few quantification algorithms have been proposed, but they lack both simplicity and robustness. As a result, the scientific community uses the notion of homogeneity in subjective analysis, preventing objective comparison of a large number of data or of different studies. The main objectives of this manuscript are to propose a definition of homogeneity and an algorithm for its quantification.

METHOD

This algorithm, called MASQH, rely on a multi-scale, statistical and segmentation-free approach and outputs a single homogeneity index, which makes it robust and easy to use.

RESULTS

The performance and reliability of the method are demonstrated through three case studies: Firstly, on synthetic images to study the behavior and assess the relevance of the algorithm in diverse situations and hence, in various potential fields. Secondly, on histological images derived from experimental chitosan-platelet-rich-plasma hybrid biomaterial, where the quantitative results are compared to a qualitative classification provided by an expert in the field. Thirdly, on experimental nanocomposites images for which results are compared to two other homogeneity quantification algorithms from the field of nanocomposites.

CONCLUSION AND SIGNIFICANCE

By quantifying homogeneity, the MASQH method may help to compare disparate studies in the literature and quantitatively demonstrate the impact of homogeneity in various fields. The MASQH method is freely available online.

Deep Learning Based Instance Segmentation of Titanium Dioxide Particles in the Form of Agglomerates in Scanning Electron Microscopy

The size characterization of particles present in the form of agglomerates in images measured by scanning electron microscopy (SEM) requires a powerful image segmentation tool in order to properly define the boundaries of each particle. In this work, we propose to use an algorithm from the deep statistical learning community, the Mask-RCNN, coupled with transfer learning to overcome the problem of generalization of the commonly used image processing methods such as watershed or active contour. Indeed, the adjustment of the parameters of these algorithms is almost systematically necessary and slows down the automation of the processing chain. The Mask-RCNN is adapted here to the case study and we present results obtained on titanium dioxide samples (non-spherical particles) with a level of performance evaluated by different metrics such as the DICE coefficient, which reaches an average value of 0.95 on the test images.

AutoDetect-mNP: An Unsupervised Machine Learning Algorithm for Automated Analysis of Transmission Electron Microscope Images of Metal Nanoparticles

The synthesis quality of artificial inorganic nanocrystals is most often assessed by transmission electron microscopy (TEM) for which high-throughput advances have dramatically increased both the quantity and information richness of metal nanoparticle (mNP) characterization. Existing automated data analysis algorithms of TEM mNP images generally adopt a supervised approach, requiring a significant effort in human preparation of labeled data that reduces objectivity, efficiency, and generalizability. We have developed an unsupervised algorithm AutoDetect-mNP for automated analysis of TEM images that objectively extracts morphological information on convex mNPs from TEM images based on their shape attributes, requiring little to no human input in the process. The performance of AutoDetect-mNP is tested on two data sets of bright field TEM images of Au nanoparticles with different shapes and further extended to palladium nanocubes and cadmium selenide quantum dots, demonstrating that the algorithm is quantitatively reliable and can thus serve as a generalizable measure of the morphology distributions of any mNP synthesis. The AutoDetect-mNP algorithm will aid in future developments of high-throughput characterization of mNPs and the future advent of time-resolved TEM studies that can investigate reaction mechanisms of mNP synthesis and reactivity.

Critical Review of Processing and Classification Techniques for Images and Spectra in Microplastic Research

Microplastic research is a rapidly developing field, with urgent needs for high throughput and automated analysis techniques. We conducted a review covering image analysis from optical microscopy, scanning electron microscopy, fluorescence microscopy, and spectral analysis from Fourier transform infrared (FT-IR) spectroscopy, Raman spectroscopy, pyrolysis gas-chromatography mass-spectrometry, and energy dispersive X-ray spectroscopy. These techniques were commonly used to collect, process, and interpret data from microplastic samples. This review outlined and critiques current approaches for analysis steps in image processing (color, thresholding, particle quantification), spectral processing (background and baseline subtraction, smoothing and noise reduction, data transformation), image classification (reference libraries, morphology, color, and fluorescence intensity), and spectral classification (reference libraries, matching procedures, and best practices for developing in-house reference tools). We highlighted opportunities to advance microplastic data analysis and interpretation by (i) quantifying colors, shapes, sizes, and surface topologies with image analysis software, (ii) identifying threshold values of particle characteristics in images that distinguish plastic particles from other particles, (iii) advancing spectral processing and classification routines, (iv) creating and sharing robust spectral libraries, (v) conducting double blind and negative controls, (vi) sharing raw data and analysis code, and (vii) leveraging readily available data to develop machine learning classification models. We identified analytical needs that we could fill and developed supplementary information for a reference library of plastic images and spectra, a tutorial for basic image analysis, and a code to download images from peer reviewed literature. Our major findings were that research on microplastics was progressing toward the use of multiple analytical methods and increasingly incorporating chemical classification. We suggest that new and repurposed methods need to be developed for high throughput screening using a diversity of approaches and highlight machine learning as one potential avenue toward this capability.

Segmentation of Overlapping Cytoplasm in Cervical Smear Images via Adaptive Shape Priors Extracted From Contour Fragments

We present a novel approach for segmenting overlapping cytoplasm of cells in cervical smear images by leveraging the adaptive shape priors extracted from cytoplasm's contour fragments and shape statistics. The main challenge of this task is that many occluded boundaries in cytoplasm clumps are extremely difficult to be identified and, sometimes, even visually indistinguishable. Given a clump where multiple cytoplasms overlap, our method starts by cutting its contour into a set of contour fragments. We then locate the corresponding contour fragments of each cytoplasm by a grouping process. For each cytoplasm, according to the grouped fragments and a set of known shape references, we construct its shape and, then, connect the fragments to form a closed contour as the segmentation result, which is explicitly constrained by the constructed shape. We further integrate the intensity and curvature information, which is complementary to the shape priors extracted from contour fragments, into our framework to improve the segmentation accuracy. We propose to iteratively conduct fragments grouping, shape constructing, and fragments connecting for progressively refining the shape priors and improving the segmentation results. We extensively evaluate the effectiveness of our method on two typical cervical smear datasets. The experimental results demonstrate that our approach is highly effective and consistently outperforms the state-of-the-art approaches. The proposed method is general enough to be applied to other similar microscopic image segmentation tasks, where heavily overlapped objects exist.

Feature-Based Representation Improves Color Decomposition and Nuclear Detection Using a Convolutional Neural Network

Detection of nuclei is an important step in phenotypic profiling of 1) histology sections imaged in bright field; and 2) colony formation of the 3-D cell culture models that are imaged using confocal microscopy. It is shown that feature-based representation of the original image improves color decomposition (CD) and subsequent nuclear detection using convolutional neural networks independent of the imaging modality. The feature-based representation utilizes the Laplacian of Gaussian (LoG) filter, which accentuates blob-shape objects. Moreover, in the case of samples imaged in bright field, the LoG response also provides the necessary initial statistics for CD using nonnegative matrix factorization. Several permutations of input data representations and network architectures are evaluated to show that by coupling improved CD and the LoG response of this representation, detection of nuclei is advanced. In particular, the frequencies of detection of nuclei with the vesicular or necrotic phenotypes, or poor staining, are improved. The overall system has been evaluated against manually annotated images, and the F-scores for alternative representations and architectures are reported.

Mesenchymal stem cell-based therapy for autoimmune diseases: emerging roles of extracellular vesicles

In autoimmune disease body's own immune system knows healthy cells as undesired and foreign cells. Over 80 types of autoimmune diseases have been recognized. Currently, at clinical practice, treatment strategies for autoimmune disorders are based on relieving symptoms and preventing difficulties. In other words, there is no effective and useful therapy up to now. It has been well-known that mesenchymal stem cells (MSCs) possess immunomodulatory effects. This strongly suggests that MSCs might be as a novel modality for treatment of autoimmune diseases. Supporting this notion a few preclinical and clinical studies indicate that MSCs ameliorate autoimmune disorders. Interestingly, it has been found that the beneficial effects of MSCs in autoimmune disorders are not relying only on direct cell-to-cell communication but on their capability to produce a broad range of paracrine factors including growth factors, cytokines and extracellular vehicles (EVs). EVs are multi-signal messengers that play a serious role in intercellular signaling through carrying cargo such as mRNA, miRNA, and proteins. Numerous studies have shown that MSC-derived EVs are able to mimic the effects of the cell of origin on immune cells. In this review, we discuss the current studies dealing with MSC-based therapies in autoimmune diseases and provide a vision and highlight in order to introduce MSC-derived EVs as an alternative and emerging modality for autoimmune disorders.

Contour-Seed Pairs Learning-Based Framework for Simultaneously Detecting and Segmenting Various Overlapping Cells/Nuclei in Microscopy Images

In this paper, we propose a novel contour-seed pairs learning-based framework for robust and automated cell/nucleus segmentation. Automated granular object segmentation in microscopy images has significant clinical importance for pathology grading of the cell carcinoma and gene expression. The focus of the past literature is dominated by either segmenting a certain type of cells/nuclei or simply splitting the clustered objects without contours inference of them. Our method addresses these issues by formulating the detection and segmentation tasks in terms of a unified regression problem, where a cascade sparse regression chain model is trained and then applied to return object locations and entire boundaries of clustered objects. In particular, we first learn a set of online convolutional features in each layer. Then, in the proposed cascade sparse regression chain, with the input from the learned features, we iteratively update the locations and clustered object boundaries until convergence. In this way, the boundary evidences of each individual object can be easily delineated and be further fed to a complete contour inference procedure optimized by the minimum description length principle. For any probe image, our method enables to analyze free-lying and overlapping cells with complex shapes. Experimental results show that the proposed method is very generic and performs well on contour inferences of various cell/nucleus types. Compared with the current segmentation techniques, our approach achieves state-of-the-art performances on four challenging datasets, i.e., the kidney renal cell carcinoma histopathology dataset, Drosophila Kc167 cellular dataset, differential interference contrast red blood cell dataset, and cervical cytology dataset.

Gaining Control over Radiolytic Synthesis of Uniform Sub-3-nanometer Palladium Nanoparticles: Use of Aromatic Liquids in the Electron Microscope

Synthesizing nanomaterials of uniform shape and size is of critical importance to access and manipulate the novel structure-property relationships arising at the nanoscale, such as catalytic activity. In this work, we synthesize Pd nanoparticles with well-controlled size in the sub-3 nm range using scanning transmission electron microscopy (STEM) in combination with an in situ liquid stage. We use an aromatic hydrocarbon (toluene) as a solvent that is very resistant to high-energy electron irradiation, which creates a net reducing environment without the need for additives to scavenge oxidizing radicals. The primary reducing species is molecular hydrogen, which is a widely used reductant in the synthesis of supported metal catalysts. We propose a mechanism of particle formation based on the effect of tri-n-octylphosphine (TOP) on size stabilization, relatively low production of radicals, and autocatalytic reduction of Pd(II) compounds. We combine in situ STEM results with insights from in situ small-angle X-ray scattering (SAXS) from alcohol-based synthesis, having similar reduction potential, in a customized microfluidic device as well as ex situ bulk experiments. This has allowed us to develop a fundamental growth model for the synthesis of size-stabilized Pd nanoparticles and demonstrate the utility of correlating different in situ and ex situ characterization techniques to understand, and ultimately control, metal nanostructure synthesis.

Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review

Digital pathology and microscopy image analysis is widely used for comprehensive studies of cell morphology or tissue structure. Manual assessment is labor intensive and prone to interobserver variations. Computer-aided methods, which can significantly improve the objectivity and reproducibility, have attracted a great deal of interest in recent literature. Among the pipeline of building a computer-aided diagnosis system, nucleus or cell detection and segmentation play a very important role to describe the molecular morphological information. In the past few decades, many efforts have been devoted to automated nucleus/cell detection and segmentation. In this review, we provide a comprehensive summary of the recent state-of-the-art nucleus/cell segmentation approaches on different types of microscopy images including bright-field, phase-contrast, differential interference contrast, fluorescence, and electron microscopies. In addition, we discuss the challenges for the current methods and the potential future work of nucleus/cell detection and segmentation.

High-throughput histopathological image analysis via robust cell segmentation and hashing

Computer-aided diagnosis of histopathological images usually requires to examine all cells for accurate diagnosis. Traditional computational methods may have efficiency issues when performing cell-level analysis. In this paper, we propose a robust and scalable solution to enable such analysis in a real-time fashion. Specifically, a robust segmentation method is developed to delineate cells accurately using Gaussian-based hierarchical voting and repulsive balloon model. A large-scale image retrieval approach is also designed to examine and classify each cell of a testing image by comparing it with a massive database, e.g., half-million cells extracted from the training dataset. We evaluate this proposed framework on a challenging and important clinical use case, i.e., differentiation of two types of lung cancers (the adenocarcinoma and squamous carcinoma), using thousands of lung microscopic tissue images extracted from hundreds of patients. Our method has achieved promising accuracy and running time by searching among half-million cells .