Image-based cell phenotyping with deep learning

Evaluating feature extraction in ovarian cancer cell line co-cultures using deep neural networks

Single-cell image analysis is crucial for studying drug effects on cellular morphology and phenotypic changes. Most studies focus on single cell types, overlooking the complexity of cellular interactions. Here, we establish an analysis pipeline to extract phenotypic features of cancer cells cultured with fibroblasts. Using high-content imaging, we analyze an oncology drug library across five cancer and fibroblast cell line co-culture combinations, generating 61,440 images and ∼170 million single-cell objects. Traditional phenotyping with CellProfiler achieves an average enrichment score of 62.6% for mechanisms of action, while pre-trained neural networks (EfficientNetB0 and MobileNetV2) reach 61.0% and 62.0%, respectively. Variability in enrichment scores may reflect the use of multiple drug concentrations since not all induce significant morphological changes, as well as the cellular and genetic context of the treatment. Our study highlights nuanced drug-induced phenotypic variations and underscores the morphological heterogeneity of ovarian cancer cell lines and their response to complex co-culture environments.

Quantitative analysis of human umbilical vein endothelial cell morphology and tubulogenesis

Primary human umbilical vein endothelial cells can grow as both a monolayer in culture and also as a capillary-like network making them an ideal model system in order to study vascular remodelling. Image-based analysis can allow assessment of cell morphology and motility but is dependent on accurate cell segmentation which requires high-contrast images not normally achievable without fluorescent markers. Here, ptychography is employed as a label-free image-based modality in order to extract quantitative metrics of morphology and tubulogenesis from cultured HUVECs over time in an automated multiwell assay. Phase-specific parameters of dry mass, optical thickness and sphericity were extracted and assessed alongside other metrics of cell number and shape. Tubulogenesis could be captured dynamically without any imaging artefacts from use of a basement membrane matrix and metrics of tube number, growth and branching exported alongside morphology metrics at early time-points. Utilising ptychography-based image analysis, all VEGF165a isoforms studied, elicited a concentration-dependent effect on cell elongation and survival within a HUVEC monolayer. Pharmacologically relevant parameters of potency (EC50) and efficacy were derived, exemplifying this label-free approach for the multiparameter and multiwell quantitative study of vascular remodelling in physiologically relevant cells at 37°C.

Systematic data analysis pipeline for quantitative morphological cell phenotyping

Quantitative morphological phenotyping (QMP) is an image-based method used to capture morphological features at both the cellular and population level. Its interdisciplinary nature, spanning from data collection to result analysis and interpretation, can lead to uncertainties, particularly among those new to this actively growing field. High analytical specificity for a typical QMP is achieved through sophisticated approaches that can leverage subtle cellular morphological changes. Here, we outline a systematic workflow to refine the QMP methodology. For a practical review, we describe the main steps of a typical QMP; in each step, we discuss the available methods, their applications, advantages, and disadvantages, along with the R functions and packages for easy implementation. This review does not cover theoretical backgrounds, but provides several references for interested researchers. It aims to broaden the horizons for future phenome studies and demonstrate how to exploit years of endeavors to achieve more with less.

Automated High-Throughput Atomic Force Microscopy Single-Cell Nanomechanical Assay Enabled by Deep Learning-Based Optical Image Recognition

Mechanical forces are essential for life activities, and the mechanical phenotypes of single cells are increasingly gaining attention. Atomic force microscopy (AFM) has been a standard method for single-cell nanomechanical assays, but its efficiency is limited due to its reliance on manual operation. Here, we present a study of deep learning image recognition-assisted AFM that enables automated high-throughput single-cell nanomechanical measurements. On the basis of the label-free identification of the cell structures and the AFM probe in optical bright-field images as well as the consequent automated movement of the sample stage and AFM probe, the AFM probe tip could be accurately and sequentially moved onto the specific parts of individual living cells to perform a single-cell indentation assay or single-cell force spectroscopy in a time-efficient manner. The study illustrates a promising method based on deep learning for achieving operator-independent high-throughput AFM single-cell nanomechanics, which will benefit the application of AFM in mechanobiology.

High-Throughput Image-Based Assay for Identifying In Vitro Hepatocyte Microtubule Disruption

Disruption of microtubule stability in mammalian cells may lead to genotoxicity and carcinogenesis. The ability to screen for microtubule destabilization or stabilization is therefore a useful and efficient approach to aid in the design of molecules that are safe for human health. In this study, we developed a high-throughput 384-well assay combining immunocytochemistry with high-content imaging to assess microtubule disruption in the metabolically competent human liver cell line: HepaRG. To enhance analysis throughput, we implemented a supervised machine learning approach using a curated training library of 180 compounds. A majority voting ensemble of eight machine learning classifiers was employed for predicting microtubule disruptions. Our prediction model achieved over 99.0% accuracy and a 98.4% F1 score, which reflects the balance between precision and recall for in-sample validation and 93.5% accuracy and a 94.3% F1 score for out-of-sample validation. This automated image-based testing can provide a simple, high-throughput screening method for early stage discovery compounds to reduce the potential risk of genotoxicity for crop protection product development.

Relational Persistent Homology for Multispecies Data with Application to the Tumor Microenvironment

Topological data analysis (TDA) is an active field of mathematics for quantifying shape in complex data. Standard methods in TDA such as persistent homology (PH) are typically focused on the analysis of data consisting of a single entity (e.g., cells or molecular species). However, state-of-the-art data collection techniques now generate exquisitely detailed multispecies data, prompting a need for methods that can examine and quantify the relations among them. Such heterogeneous data types arise in many contexts, ranging from biomedical imaging, geospatial analysis, to species ecology. Here, we propose two methods for encoding spatial relations among different data types that are based on Dowker complexes and Witness complexes. We apply the methods to synthetic multispecies data of a tumor microenvironment and analyze topological features that capture relations between different cell types, e.g., blood vessels, macrophages, tumor cells, and necrotic cells. We demonstrate that relational topological features can extract biological insight, including the dominant immune cell phenotype (an important predictor of patient prognosis) and the parameter regimes of a data-generating model. The methods provide a quantitative perspective on the relational analysis of multispecies spatial data, overcome the limits of traditional PH, and are readily computable.

High-quality AFM image acquisition of living cells by modified residual encoder-decoder network

Atomic force microscope enables ultra-precision imaging of living cells. However, atomic force microscope imaging is a complex and time-consuming process. The obtained images of living cells usually have low resolution and are easily influenced by noise leading to unsatisfactory imaging quality, obstructing the research and analysis based on cell images. Herein, an adaptive attention image reconstruction network based on residual encoder-decoder was proposed, through the combination of deep learning technology and atomic force microscope imaging supporting high-quality cell image acquisition. Compared with other learning-based methods, the proposed network showed higher peak signal-to-noise ratio, higher structural similarity and better image reconstruction performances. In addition, the cell images reconstructed by each method were used for cell recognition, and the cell images reconstructed by the proposed network had the highest cell recognition rate. The proposed network has brought insights into the atomic force microscope-based imaging of living cells and cell image reconstruction, which is of great significance in biological and medical research.

A deep learning method that identifies cellular heterogeneity using nanoscale nuclear features

Cellular phenotypic heterogeneity is an important hallmark of many biological processes and understanding its origins remains a substantial challenge. This heterogeneity often reflects variations in the chromatin structure, influenced by factors such as viral infections and cancer, which dramatically reshape the cellular landscape. To address the challenge of identifying distinct cell states, we developed artificial intelligence of the nucleus (AINU), a deep learning method that can identify specific nuclear signatures at the nanoscale resolution. AINU can distinguish different cell states based on the spatial arrangement of core histone H3, RNA polymerase II or DNA from super-resolution microscopy images. With only a small number of images as the training data, AINU correctly identifies human somatic cells, human-induced pluripotent stem cells, very early stage infected cells transduced with DNA herpes simplex virus type 1 and even cancer cells after appropriate retraining. Finally, using AI interpretability methods, we find that the RNA polymerase II localizations in the nucleoli aid in distinguishing human-induced pluripotent stem cells from their somatic cells. Overall, AINU coupled with super-resolution microscopy of nuclear structures provides a robust tool for the precise detection of cellular heterogeneity, with considerable potential for advancing diagnostics and therapies in regenerative medicine, virology and cancer biology.

Cell-vision fusion: A Swin transformer-based approach for predicting kinase inhibitor mechanism of action from Cell Painting data

Image-based profiling of the cellular response to drug compounds has proven effective at characterizing the morphological changes resulting from perturbation experiments. As data availability increases, however, there are growing demands for novel deep-learning methods. We applied the SwinV2 computer vision architecture to predict the mechanism of action of 10 kinase inhibitor compounds directly from Cell Painting images. This method outperforms the standard approach of using image-based profiles (IBP)-multidimensional feature set representations generated by bioimaging software. Furthermore, our fusion approach-cell-vision fusion, combining three different data modalities, images, IBPs, and chemical structures-achieved 69.79% accuracy and 70.56% F1 score, 4.20% and 5.49% higher, respectively, than the best-performing IBP method. We provide three techniques, specific to Cell Painting images, which enable deep-learning architectures to train effectively and demonstrate approaches to combat the significant batch effects present in large Cell Painting datasets.

GammaGateR: semi-automated marker gating for single-cell multiplexed imaging

MOTIVATION

Multiplexed immunofluorescence (mIF) is an emerging assay for multichannel protein imaging that can decipher cell-level spatial features in tissues. However, existing automated cell phenotyping methods, such as clustering, face challenges in achieving consistency across experiments and often require subjective evaluation. As a result, mIF analyses often revert to marker gating based on manual thresholding of raw imaging data.

RESULTS

To address the need for an evaluable semi-automated algorithm, we developed GammaGateR, an R package for interactive marker gating designed specifically for segmented cell-level data from mIF images. Based on a novel closed-form gamma mixture model, GammaGateR provides estimates of marker-positive cell proportions and soft clustering of marker-positive cells. The model incorporates user-specified constraints that provide a consistent but slide-specific model fit. We compared GammaGateR against the newest unsupervised approach for annotating mIF data, employing two colon datasets and one ovarian cancer dataset for the evaluation. We showed that GammaGateR produces highly similar results to a silver standard established through manual annotation. Furthermore, we demonstrated its effectiveness in identifying biological signals, achieved by mapping known spatial interactions between CD68 and MUC5AC cells in the colon and by accurately predicting survival in ovarian cancer patients using the phenotype probabilities as input for machine learning methods. GammaGateR is a highly efficient tool that can improve the replicability of marker gating results, while reducing the time of manual segmentation.

AVAILABILITY AND IMPLEMENTATION

The R package is available at https://github.com/JiangmeiRubyXiong/GammaGateR.

Three million images and morphological profiles of cells treated with matched chemical and genetic perturbations

The identification of genetic and chemical perturbations with similar impacts on cell morphology can elucidate compounds' mechanisms of action or novel regulators of genetic pathways. Research on methods for identifying such similarities has lagged due to a lack of carefully designed and well-annotated image sets of cells treated with chemical and genetic perturbations. Here we create such a Resource dataset, CPJUMP1, in which each perturbed gene's product is a known target of at least two chemical compounds in the dataset. We systematically explore the directionality of correlations among perturbations that target the same protein encoded by a given gene, and we find that identifying matches between chemical and genetic perturbations is a challenging task. Our dataset and baseline analyses provide a benchmark for evaluating methods that measure perturbation similarities and impact, and more generally, learn effective representations of cellular state from microscopy images. Such advancements would accelerate the applications of image-based profiling of cellular states, such as uncovering drug mode of action or probing functional genomics.

Morphological profiling for drug discovery in the era of deep learning

Morphological profiling is a valuable tool in phenotypic drug discovery. The advent of high-throughput automated imaging has enabled the capturing of a wide range of morphological features of cells or organisms in response to perturbations at the single-cell resolution. Concurrently, significant advances in machine learning and deep learning, especially in computer vision, have led to substantial improvements in analyzing large-scale high-content images at high throughput. These efforts have facilitated understanding of compound mechanism of action, drug repurposing, characterization of cell morphodynamics under perturbation, and ultimately contributing to the development of novel therapeutics. In this review, we provide a comprehensive overview of the recent advances in the field of morphological profiling. We summarize the image profiling analysis workflow, survey a broad spectrum of analysis strategies encompassing feature engineering- and deep learning-based approaches, and introduce publicly available benchmark datasets. We place a particular emphasis on the application of deep learning in this pipeline, covering cell segmentation, image representation learning, and multimodal learning. Additionally, we illuminate the application of morphological profiling in phenotypic drug discovery and highlight potential challenges and opportunities in this field.

Machine learning aided single cell image analysis improves understanding of morphometric heterogeneity of human mesenchymal stem cells

The multipotent stem cells of our body have been largely harnessed in biotherapeutics. However, as they are derived from multiple anatomical sources, from different tissues, human mesenchymal stem cells (hMSCs) are a heterogeneous population showing ambiguity in their in vitro behavior. Intra-clonal population heterogeneity has also been identified and pre-clinical mechanistic studies suggest that these cumulatively depreciate the therapeutic effects of hMSC transplantation. Although various biomarkers identify these specific stem cell populations, recent artificial intelligence-based methods have capitalized on the cellular morphologies of hMSCs, opening a new approach to understand their attributes. A robust and rapid platform is required to accommodate and eliminate the heterogeneity observed in the cell population, to standardize the quality of hMSC therapeutics globally. Here, we report our primary findings of morphological heterogeneity observed within and across two sources of hMSCs namely, stem cells from human exfoliated deciduous teeth (SHEDs) and human Wharton jelly mesenchymal stem cells (hWJ MSCs), using real-time single-cell images generated on immunophenotyping by imaging flow cytometry (IFC). We used the ImageJ software for identification and comparison between the two types of hMSCs using statistically significant morphometric descriptors that are biologically relevant. To expand on these insights, we have further applied deep learning methods and successfully report the development of a Convolutional Neural Network-based image classifier. In our research, we introduced a machine learning methodology to streamline the entire procedure, utilizing convolutional neural networks and transfer learning for binary classification, achieving an accuracy rate of 97.54%. We have also critically discussed the challenges, comparisons between solutions and future directions of machine learning in hMSC classification in biotherapeutics.

Harnessing atomic force microscopy-based single-cell analysis to advance physical oncology

Single-cell analysis is an emerging and promising frontier in the field of life sciences, which is expected to facilitate the exploration of fundamental laws of physiological and pathological processes. Single-cell analysis allows experimental access to cell-to-cell heterogeneity to reveal the distinctive behaviors of individual cells, offering novel opportunities to dissect the complexity of severe human diseases such as cancers. Among the single-cell analysis tools, atomic force microscopy (AFM) is a powerful and versatile one which is able to nondestructively image the fine topographies and quantitatively measure multiple mechanical properties of single living cancer cells in their native states under aqueous conditions with unprecedented spatiotemporal resolution. Over the past few decades, AFM has been widely utilized to detect the structural and mechanical behaviors of individual cancer cells during the process of tumor formation, invasion, and metastasis, yielding numerous unique insights into tumor pathogenesis from the biomechanical perspective and contributing much to the field of cancer mechanobiology. Here, the achievements of AFM-based analysis of single cancer cells to advance physical oncology are comprehensively summarized, and challenges and future perspectives are also discussed. RESEARCH HIGHLIGHTS: Achievements of AFM in characterizing the structural and mechanical behaviors of single cancer cells are summarized, and future directions are discussed. AFM is not only capable of visualizing cellular fine structures, but can also measure multiple cellular mechanical properties as well as cell-generated mechanical forces. There is still plenty of room for harnessing AFM-based single-cell analysis to advance physical oncology.

Plant microphenotype: from innovative imaging to computational analysis

The microphenotype plays a key role in bridging the gap between the genotype and the complex macro phenotype. In this article, we review the advances in data acquisition and the intelligent analysis of plant microphenotyping and present applications of microphenotyping in plant science over the past two decades. We then point out several challenges in this field and suggest that cross-scale image acquisition strategies, powerful artificial intelligence algorithms, advanced genetic analysis, and computational phenotyping need to be established and performed to better understand interactions among genotype, environment, and management. Microphenotyping has entered the era of Microphenotyping 3.0 and will largely advance functional genomics and plant science.

Deep Learning Image Recognition-Assisted Atomic Force Microscopy for Single-Cell Efficient Mechanics in Co-culture Environments

Atomic force microscopy (AFM)-based force spectroscopy assay has become an important method for characterizing the mechanical properties of single living cells under aqueous conditions, but a disadvantage is its reliance on manual operation and experience as well as the resulting low throughput. Particularly, providing a capacity to accurately identify the type of the cell grown in co-culture environments without the need of fluorescent labeling will further facilitate the applications of AFM in life sciences. Here, we present a study of deep learning image recognition-assisted AFM, which not only enables fluorescence-independent recognition of the identity of single co-cultured cells but also allows efficient downstream AFM force measurements of the identified cells. With the use of the deep learning-based image recognition model, the viability and type of individual cells grown in co-culture environments were identified directly from the optical bright-field images, which were confirmed by the following cell growth and fluorescent labeling results. Based on the image recognition results, the positional relationship between the AFM probe and the targeted cell was automatically determined, allowing the precise movement of the AFM probe to the target cell to perform force measurements. The experimental results show that the presented method was applicable not only to the conventional (microsphere-modified) AFM probe used in AFM indentation assay for measuring the Young's modulus of single co-cultured cells but also to the single-cell probe used in AFM-based single-cell force spectroscopy (SCFS) assay for measuring the adhesion forces of single co-cultured cells. The study illustrates deep learning imaging recognition-assisted AFM as a promising approach for label-free and high-throughput detection of single-cell mechanics under co-culture conditions, which will facilitate unraveling the mechanical cues involved in cell-cell interactions in their native states at the single-cell level and will benefit the field of mechanobiology.

Microsnoop: A generalist tool for microscopy image representation

Accurate profiling of microscopy images from small scale to high throughput is an essential procedure in basic and applied biological research. Here, we present Microsnoop, a novel deep learning-based representation tool trained on large-scale microscopy images using masked self-supervised learning. Microsnoop can process various complex and heterogeneous images, and we classified images into three categories: single-cell, full-field, and batch-experiment images. Our benchmark study on 10 high-quality evaluation datasets, containing over 2,230,000 images, demonstrated Microsnoop's robust and state-of-the-art microscopy image representation ability, surpassing existing generalist and even several custom algorithms. Microsnoop can be integrated with other pipelines to perform tasks such as superresolution histopathology image and multimodal analysis. Furthermore, Microsnoop can be adapted to various hardware and can be easily deployed on local or cloud computing platforms. We will regularly retrain and reevaluate the model using community-contributed data to consistently improve Microsnoop.

VistoSeg: Processing utilities for high-resolution images for spatially resolved transcriptomics data

Spatially resolved transcriptomics (SRT) is a growing field that links gene expression to anatomical context. SRT approaches that use next-generation sequencing (NGS) combine RNA sequencing with histological or fluorescent imaging to generate spatial maps of gene expression in intact tissue sections. These technologies directly couple gene expression measurements with high-resolution histological or immunofluorescent images that contain rich morphological information about the tissue under study. While broad access to NGS-based spatial transcriptomic technology is now commercially available through the Visium platform from the vendor 10× Genomics, computational tools for extracting image-derived metrics for integration with gene expression data remain limited. We developed VistoSeg as a MATLAB pipeline to process, analyze and interactively visualize the high-resolution images generated in the Visium platform. VistoSeg outputs can be easily integrated with accompanying transcriptomic data to facilitate downstream analyses in common programing languages including R and Python. VistoSeg provides user-friendly tools for integrating image-derived metrics from histological and immunofluorescent images with spatially resolved gene expression data. Integration of this data enhances the ability to understand the transcriptional landscape within tissue architecture. VistoSeg is freely available at http://research.libd.org/VistoSeg/.

High throughput microscopy and single cell phenotypic image-based analysis in toxicology and drug discovery

Measuring single cell responses to the universe of chemicals (drugs, natural products, environmental toxicants etc.) is of paramount importance to human health as phenotypic variability in sensing stimuli is a hallmark of biology that is considered during high throughput screening. One of the ways to approach this problem is via high throughput, microscopy-based assays coupled with multi-dimensional single cell analysis methods. Here, we will summarize some of the efforts in this vast and growing field, focusing on phenotypic screens (e.g., Cell Painting), single cell analytics and quality control, with particular attention to environmental toxicology and drug screening. We will discuss advantages and limitations of high throughput assays with various end points and levels of complexity.

Evolution and impact of high content imaging

The field of high content imaging has steadily evolved and expanded substantially across many industry and academic research institutions since it was first described in the early 1990's. High content imaging refers to the automated acquisition and analysis of microscopic images from a variety of biological sample types. Integration of high content imaging microscopes with multiwell plate handling robotics enables high content imaging to be performed at scale and support medium- to high-throughput screening of pharmacological, genetic and diverse environmental perturbations upon complex biological systems ranging from 2D cell cultures to 3D tissue organoids to small model organisms. In this perspective article the authors provide a collective view on the following key discussion points relevant to the evolution of high content imaging: • Evolution and impact of high content imaging: An academic perspective • Evolution and impact of high content imaging: An industry perspective • Evolution of high content image analysis • Evolution of high content data analysis pipelines towards multiparametric and phenotypic profiling applications • The role of data integration and multiomics • The role and evolution of image data repositories and sharing standards • Future perspective of high content imaging hardware and software.

GammaGateR: semi-automated marker gating for single-cell multiplexed imaging

Motivation

Multiplexed immunofluorescence (mIF) is an emerging assay for multichannel protein imaging that can decipher cell-level spatial features in tissues. However, existing automated cell phenotyping methods, such as clustering, face challenges in achieving consistency across experiments and often require subjective evaluation. As a result, mIF analyses often revert to marker gating based on manual thresholding of raw imaging data.

Results

To address the need for an evaluable semi-automated algorithm, we developed GammaGateR, an R package for interactive marker gating designed specifically for segmented cell-level data from mIF images. Based on a novel closed-form gamma mixture model, GammaGateR provides estimates of marker-positive cell proportions and soft clustering of marker-positive cells. The model incorporates user-specified constraints that provide a consistent but slide-specific model fit. We compared GammaGateR against the newest unsupervised approach for annotating mIF data, employing two colon datasets and one ovarian cancer dataset for the evaluation. We showed that GammaGateR produces highly similar results to a silver standard established through manual annotation. Furthermore, we demonstrated its effectiveness in identifying biological signals, achieved by mapping known spatial interactions between CD68 and MUC5AC cells in the colon and by accurately predicting survival in ovarian cancer patients using the phenotype probabilities as input for machine learning methods. GammaGateR is a highly efficient tool that can improve the replicability of marker gating results, while reducing the time of manual segmentation.

Availability and Implementation

The R package is available at https://github.com/JiangmeiRubyXiong/GammaGateR.

Integrated Multimodal Omics and Dietary Approaches for the Management of Neurodegeneration

Neurodegenerative diseases, such as Alzheimer's disease and Parkinson's disease, are caused by a combination of multiple events that damage neuronal function. A well-characterized biomarker of neurodegeneration is the accumulation of proteinaceous aggregates in the brain. However, the gradually worsening symptoms of neurodegenerative diseases are unlikely to be solely due to the result of a mutation in a single gene, but rather a multi-step process involving epigenetic changes. Recently, it has been suggested that a fraction of epigenetic alternations may be correlated to neurodegeneration in the brain. Unlike DNA mutations, epigenetic alterations are reversible, and therefore raise the possibilities for therapeutic intervention, including dietary modifications. Additionally, reactive oxygen species may contribute to the pathogenesis of Alzheimer's disease and Parkinson's disease through epigenetic alternation. Given that the antioxidant properties of plant-derived phytochemicals are likely to exhibit pleiotropic effects against ROS-mediated epigenetic alternation, dietary intervention may be promising for the management of neurodegeneration in these diseases. In this review, the state-of-the-art applications using single-cell multimodal omics approaches, including epigenetics, and dietary approaches for the identification of novel biomarkers and therapeutic approaches for the treatment of neurodegenerative diseases are discussed.

Cell recognition based on atomic force microscopy and modified residual neural network

Cell recognition methods are in high demand in cell biology and medicine, and the method based on atomic force microscopy (AFM) shows a great value in application. The difference in mechanical properties or morphology of cells has been frequently used to detect whether cells are cancerous, but this detection method cannot be a general means for cancer cell detection, and the traditional artificial feature extraction method also has its limitations. In this work, we proposed an analytic method based on the physical properties of cells and deep learning method for recognizing cell types. The residual neural network used for recognition was modified by multi-scale convolutional fusion, attention mechanism and depthwise separable convolution, so as to optimize feature extraction and reduce operation costs. In the method, the collected cells were imaged by AFM, and the processed images were analyzed by the optimized convolutional neural network. The recognition results of two groups of cells (HL-7702 and SMMC-7721, SGC-7901 and GES-1) by this method show that the recognition rate of dataset with the combination of cell surface morphology, adhesion and Young's modulus is higher, and the recognition rate of the dataset with optimal resolution is higher. Our study indicated that the recognition of physical properties of cells using deep learning technology can serve as a universal and effective method for the automated analysis of cell information.

Optimizing the Cell Painting assay for image-based profiling

In image-based profiling, software extracts thousands of morphological features of cells from multi-channel fluorescence microscopy images, yielding single-cell profiles that can be used for basic research and drug discovery. Powerful applications have been proven, including clustering chemical and genetic perturbations on the basis of their similar morphological impact, identifying disease phenotypes by observing differences in profiles between healthy and diseased cells and predicting assay outcomes by using machine learning, among many others. Here, we provide an updated protocol for the most popular assay for image-based profiling, Cell Painting. Introduced in 2013, it uses six stains imaged in five channels and labels eight diverse components of the cell: DNA, cytoplasmic RNA, nucleoli, actin, Golgi apparatus, plasma membrane, endoplasmic reticulum and mitochondria. The original protocol was updated in 2016 on the basis of several years' experience running it at two sites, after optimizing it by visual stain quality. Here, we describe the work of the Joint Undertaking for Morphological Profiling Cell Painting Consortium, to improve upon the assay via quantitative optimization by measuring the assay's ability to detect morphological phenotypes and group similar perturbations together. The assay gives very robust outputs despite various changes to the protocol, and two vendors' dyes work equivalently well. We present Cell Painting version 3, in which some steps are simplified and several stain concentrations can be reduced, saving costs. Cell culture and image acquisition take 1-2 weeks for typically sized batches of ≤20 plates; feature extraction and data analysis take an additional 1-2 weeks.This protocol is an update to Nat. Protoc. 11, 1757-1774 (2016): https://doi.org/10.1038/nprot.2016.105.

Predicting compound activity from phenotypic profiles and chemical structures

Predicting assay results for compounds virtually using chemical structures and phenotypic profiles has the potential to reduce the time and resources of screens for drug discovery. Here, we evaluate the relative strength of three high-throughput data sources-chemical structures, imaging (Cell Painting), and gene-expression profiles (L1000)-to predict compound bioactivity using a historical collection of 16,170 compounds tested in 270 assays for a total of 585,439 readouts. All three data modalities can predict compound activity for 6-10% of assays, and in combination they predict 21% of assays with high accuracy, which is a 2 to 3 times higher success rate than using a single modality alone. In practice, the accuracy of predictors could be lower and still be useful, increasing the assays that can be predicted from 37% with chemical structures alone up to 64% when combined with phenotypic data. Our study shows that unbiased phenotypic profiling can be leveraged to enhance compound bioactivity prediction to accelerate the early stages of the drug-discovery process.

Galactosidase-catalyzed fluorescence amplification method (GAFAM): sensitive fluorescent immunohistochemistry using novel fluorogenic β-galactosidase substrates and its application in multiplex immunostaining

Multiplex immunohistochemistry/multiplex immunofluorescence (mIHC/mIF) enables the simultaneous detection of multiple markers in a single tissue section by visualizing the markers in different colors. Currently, tyramide signal amplification (TSA) is the most commonly used method because it is heat resistant to multiplexing. SPiDER-βGal (6'-(diethylamino)-4'-(fluoromethyl)spiro[isobenzofuran-1(3H),9'-[9H]xanthen]-3'-yl β-D-galactopyranoside), a novel fluorogenic substrate of β-galactosidase (β-gal) was reported recently. Its properties are favorable for application in sensitive mIF based on quinone methide chemistry. Combining SPiDER-βGal with its related substrates, a novel, sensitive fluorescent IHC method for formalin-fixed paraffin-embedded (FFPE) sections was developed, named the galactosidase-catalyzed fluorescence amplification method (GAFAM). Evaluation of GAFAM indicated the following characteristics: (1) the entire GAFAM procedure was complete within a few hours; (2) the optimal working concentration of the substrates was 20 μM; (3) the fluorescent product was heat resistant; (4) the GAFAM exhibited sensitivity comparable with that of TSA, which was higher than that of conventional IF; and (5) the GAFAM was applicable to mIF and multispectral imaging. GAFAM is expected to be applicable to IF (or mIF in combination with TSA), and is a promising tool for facilitating morphological research in various fields of life science.

Small-molecule fluorogenic probes for mitochondrial nanoscale imaging

Mitochondria are inextricably linked to the development of diseases and cell metabolism disorders. Super-resolution imaging (SRI) is crucial in enhancing our understanding of mitochondrial ultrafine structures and functions. In addition to high-precision instruments, super-resolution microscopy relies heavily on fluorescent materials with unique photophysical properties. Small-molecule fluorogenic probes (SMFPs) have excellent properties that make them ideal for mitochondrial SRI. This paper summarizes recent advances in the field of SMFPs, with a focus on the chemical and spectroscopic properties required for mitochondrial SRI. Finally, we discuss future challenges in this field, including the design principles of SMFPs and nanoscopic techniques.

DisGUVery: A Versatile Open-Source Software for High-Throughput Image Analysis of Giant Unilamellar Vesicles

Giant unilamellar vesicles (GUVs) are cell-sized aqueous compartments enclosed by a phospholipid bilayer. Due to their cell-mimicking properties, GUVs have become a widespread experimental tool in synthetic biology to study membrane properties and cellular processes. In stark contrast to the experimental progress, quantitative analysis of GUV microscopy images has received much less attention. Currently, most analysis is performed either manually or with custom-made scripts, which makes analysis time-consuming and results difficult to compare across studies. To make quantitative GUV analysis accessible and fast, we present DisGUVery, an open-source, versatile software that encapsulates multiple algorithms for automated detection and analysis of GUVs in microscopy images. With a performance analysis, we demonstrate that DisGUVery's three vesicle detection modules successfully identify GUVs in images obtained with a wide range of imaging sources, in various typical GUV experiments. Multiple predefined analysis modules allow the user to extract properties such as membrane fluorescence, vesicle shape, and internal fluorescence from large populations. A new membrane segmentation algorithm facilitates spatial fluorescence analysis of nonspherical vesicles. Altogether, DisGUVery provides an accessible tool to enable high-throughput automated analysis of GUVs, and thereby to promote quantitative data analysis in synthetic cell research.

Deep learning in image-based phenotypic drug discovery

Modern drug discovery approaches often use high-content imaging to systematically study the effect on cells of large libraries of chemical compounds. By automatically screening thousands or millions of images to identify specific drug-induced cellular phenotypes, for example, altered cellular morphology, these approaches can reveal 'hit' compounds offering therapeutic promise. In the past few years, artificial intelligence (AI) methods based on deep learning (DL) [a family of machine learning (ML) techniques] have disrupted virtually all image analysis tasks, from image classification to segmentation. These powerful methods also promise to impact drug discovery by accelerating the identification of effective drugs and their modes of action. In this review, we highlight applications and adaptations of ML, especially DL methods for cell-based phenotypic drug discovery (PDD).

Statistical Analysis of Multiplex Immunofluorescence and Immunohistochemistry Imaging Data

Advances in multiplexed single-cell immunofluorescence (mIF) and multiplex immunohistochemistry (mIHC) imaging technologies have enabled the analysis of cell-to-cell spatial relationships that promise to revolutionize our understanding of tissue-based diseases and autoimmune disorders. Multiplex images are collected as multichannel TIFF files; then denoised, segmented to identify cells and nuclei, normalized across slides with protein markers to correct for batch effects, and phenotyped; and then tissue composition and spatial context at the cellular level are analyzed. This chapter discusses methods and software infrastructure for image processing and statistical analysis of mIF/mIHC data.

Fisheye transformation enhances deep-learning-based single-cell phenotyping by including cellular microenvironment

Incorporating information about the surroundings can have a significant impact on successfully determining the class of an object. This is of particular interest when determining the phenotypes of cells, for example, in the context of high-throughput screens. We hypothesized that an ideal approach would consider the fully featured view of the cell of interest, include its neighboring microenvironment, and give lesser weight to cells that are far from the cell of interest. To satisfy these criteria, we present an approach with a transformation similar to those characteristic of fisheye cameras. Using this transformation with proper settings, we could significantly increase the accuracy of single-cell phenotyping, both in the case of cell culture and tissue-based microscopy images, and we present improved results on a dataset containing images of wild animals.

Characterization of drug effects on cell cultures from phase-contrast microscopy images

In this work, we classify chemotherapeutic agents (topoisomerase inhibitors) based on their effect on U-2 OS cells. We use phase-contrast microscopy images, which are faster and easier to obtain than fluorescence images and support live cell imaging. We use a convolutional neural network (CNN) trained end-to-end directly on the input images without requiring for manual segmentations or any other auxiliary data. Our method can distinguish between tested cytotoxic drugs with an accuracy of 98%, provided that their mechanism of action differs, outperforming previous work. The results are even better when substance-specific concentrations are used. We show the benefit of sharing the extracted features over all classes (drugs). Finally, a 2D visualization of these features reveals clusters, which correspond well to known class labels, suggesting the possible use of our methodology for drug discovery application in analyzing new, unseen drugs.

Morphology and gene expression profiling provide complementary information for mapping cell state

Morphological and gene expression profiling can cost-effectively capture thousands of features in thousands of samples across perturbations by disease, mutation, or drug treatments, but it is unclear to what extent the two modalities capture overlapping versus complementary information. Here, using both the L1000 and Cell Painting assays to profile gene expression and cell morphology, respectively, we perturb human A549 lung cancer cells with 1,327 small molecules from the Drug Repurposing Hub across six doses, providing a data resource including dose-response data from both assays. The two assays capture both shared and complementary information for mapping cell state. Cell Painting profiles from compound perturbations are more reproducible and show more diversity but measure fewer distinct groups of features. Applying unsupervised and supervised methods to predict compound mechanisms of action (MOAs) and gene targets, we find that the two assays not only provide a partially shared but also a complementary view of drug mechanisms. Given the numerous applications of profiling in biology, our analyses provide guidance for planning experiments that profile cells for detecting distinct cell types, disease phenotypes, and response to chemical or genetic perturbations.

Automated detection of patterned single-cells within hydrogel using deep learning

Single-cell analysis has been widely used in various biomedical engineering applications, ranging from cancer diagnostics, and immune response monitoring to drug screening. Single-cell isolation is fundamental for observing single-cell activities and an automatic finding method of accurate and reliable cell detection with few possible human errors is also essential. This paper reports trapping single cells into photo patternable hydrogel microwell arrays and isolating them. Additionally, we present an object detection-based DL algorithm that detects single cells in microwell arrays and predicts the presence of cells in resource-limited environments at the highest possible mAP (mean average precision) of 0.989 with an average inference time of 0.06 s. This algorithm leads to the enhancement of the high-throughput single-cell analysis, establishing high detection precision and reduced experimentation time.

In-cell NMR: Why and how?

NMR spectroscopy has been applied to cells and tissues analysis since its beginnings, as early as 1950. We have attempted to gather here in a didactic fashion the broad diversity of data and ideas that emerged from NMR investigations on living cells. Covering a large proportion of the periodic table, NMR spectroscopy permits scrutiny of a great variety of atomic nuclei in all living organisms non-invasively. It has thus provided quantitative information on cellular atoms and their chemical environment, dynamics, or interactions. We will show that NMR studies have generated valuable knowledge on a vast array of cellular molecules and events, from water, salts, metabolites, cell walls, proteins, nucleic acids, drugs and drug targets, to pH, redox equilibria and chemical reactions. The characterization of such a multitude of objects at the atomic scale has thus shaped our mental representation of cellular life at multiple levels, together with major techniques like mass-spectrometry or microscopies. NMR studies on cells has accompanied the developments of MRI and metabolomics, and various subfields have flourished, coined with appealing names: fluxomics, foodomics, MRI and MRS (i.e. imaging and localized spectroscopy of living tissues, respectively), whole-cell NMR, on-cell ligand-based NMR, systems NMR, cellular structural biology, in-cell NMR… All these have not grown separately, but rather by reinforcing each other like a braided trunk. Hence, we try here to provide an analytical account of a large ensemble of intricately linked approaches, whose integration has been and will be key to their success. We present extensive overviews, firstly on the various types of information provided by NMR in a cellular environment (the "why", oriented towards a broad readership), and secondly on the employed NMR techniques and setups (the "how", where we discuss the past, current and future methods). Each subsection is constructed as a historical anthology, showing how the intrinsic properties of NMR spectroscopy and its developments structured the accessible knowledge on cellular phenomena. Using this systematic approach, we sought i) to make this review accessible to the broadest audience and ii) to highlight some early techniques that may find renewed interest. Finally, we present a brief discussion on what may be potential and desirable developments in the context of integrative studies in biology.

Virtual screening for small-molecule pathway regulators by image-profile matching

Identifying the chemical regulators of biological pathways is a time-consuming bottleneck in developing therapeutics and research compounds. Typically, thousands to millions of candidate small molecules are tested in target-based biochemical screens or phenotypic cell-based screens, both expensive experiments customized to each disease. Here, our uncustomized, virtual, profile-based screening approach instead identifies compounds that match to pathways based on the phenotypic information in public cell image data, created using the Cell Painting assay. Our straightforward correlation-based computational strategy retrospectively uncovered the expected, known small-molecule regulators for 32% of positive-control gene queries. In prospective, discovery mode, we efficiently identified new compounds related to three query genes and validated them in subsequent gene-relevant assays, including compounds that phenocopy or pheno-oppose YAP1 overexpression and kill a Yap1-dependent sarcoma cell line. This image-profile-based approach could replace many customized labor- and resource-intensive screens and accelerate the discovery of biologically and therapeutically useful compounds.

Comparing machine learning and deep learning regression frameworks for accurate prediction of dielectrophoretic force

An intelligent sensing framework using Machine Learning (ML) and Deep Learning (DL) architectures to precisely quantify dielectrophoretic force invoked on microparticles in a textile electrode-based DEP sensing device is reported. The prediction accuracy and generalization ability of the framework was validated using experimental results. Images of pearl chain alignment at varying input voltages were used to build deep regression models using modified ML and CNN architectures that can correlate pearl chain alignment patterns of Saccharomyces cerevisiae(yeast) cells and polystyrene microbeads to DEP force. Various ML models such as K-Nearest Neighbor, Support Vector Machine, Random Forest, Neural Networks, and Linear Regression along with DL models such as Convolutional Neural Network (CNN) architectures of AlexNet, ResNet-50, MobileNetV2, and GoogLeNet have been analyzed in order to build an effective regression framework to estimate the force induced on yeast cells and microbeads. The efficiencies of the models were evaluated using Mean Absolute Error, Mean Absolute Relative, Mean Squared Error, R-squared, and Root Mean Square Error (RMSE) as evaluation metrics. ResNet-50 with RMSPROP gave the best performance, with a validation RMSE of 0.0918 on yeast cells while AlexNet with ADAM optimizer gave the best performance, with a validation RMSE of 0.1745 on microbeads. This provides a baseline for further studies in the application of deep learning in DEP aided Lab-on-Chip devices.

Multiplex Immunofluorescence and the Digital Image Analysis Workflow for Evaluation of the Tumor Immune Environment in Translational Research

A robust understanding of the tumor immune environment has important implications for cancer diagnosis, prognosis, research, and immunotherapy. Traditionally, immunohistochemistry (IHC) has been regarded as the standard method for detecting proteins in situ, but this technique allows for the evaluation of only one cell marker per tissue sample at a time. However, multiplexed imaging technologies enable the multiparametric analysis of a tissue section at the same time. Also, through the curation of specific antibody panels, these technologies enable researchers to study the cell subpopulations within a single immunological cell group. Thus, multiplexed imaging gives investigators the opportunity to better understand tumor cells, immune cells, and the interactions between them. In the multiplexed imaging technology workflow, once the protocol for a tumor immune micro environment study has been defined, histological slides are digitized to produce high-resolution images in which regions of interest are selected for the interrogation of simultaneously expressed immunomarkers (including those co-expressed by the same cell) by using an image analysis software and algorithm. Most currently available image analysis software packages use similar machine learning approaches in which tissue segmentation first defines the different components that make up the regions of interest and cell segmentation, then defines the different parameters, such as the nucleus and cytoplasm, that the software must utilize to segment single cells. Image analysis tools have driven dramatic evolution in the field of digital pathology over the past several decades and provided the data necessary for translational research and the discovery of new therapeutic targets. The next step in the growth of digital pathology is optimization and standardization of the different tasks in cancer research, including image analysis algorithm creation, to increase the amount of data generated and their accuracy in a short time as described herein. The aim of this review is to describe this process, including an image analysis algorithm creation for multiplex immunofluorescence analysis, as an essential part of the optimization and standardization of the different processes in cancer research, to increase the amount of data generated and their accuracy in a short time.

Self-supervised learning of cell type specificity from immunohistochemical images

MOTIVATION

Advances in bioimaging now permit in situ proteomic characterization of cell-cell interactions in complex tissues, with important applications across a spectrum of biological problems from development to disease. These methods depend on selection of antibodies targeting proteins that are expressed specifically in particular cell types. Candidate marker proteins are often identified from single-cell transcriptomic data, with variable rates of success, in part due to divergence between expression levels of proteins and the genes that encode them. In principle, marker identification could be improved by using existing databases of immunohistochemistry for thousands of antibodies in human tissue, such as the Human Protein Atlas. However, these data lack detailed annotations of the types of cells in each image.

RESULTS

We develop a method to predict cell type specificity of protein markers from unlabeled images. We train a convolutional neural network with a self-supervised objective to generate embeddings of the images. Using non-linear dimensionality reduction, we observe that the model clusters images according to cell types and anatomical regions for which the stained proteins are specific. We then use estimates of cell type specificity derived from an independent single-cell transcriptomics dataset to train an image classifier, without requiring any human labelling of images. Our scheme demonstrates superior classification of known proteomic markers in kidney compared to selection via single-cell transcriptomics.

AVAILABILITY AND IMPLEMENTATION

Code and trained model are available at www.github.com/murphy17/HPA-SimCLR.

SUPPLEMENTARY INFORMATION

Supplementary data are available at Bioinformatics online.

Cell Painting predicts impact of lung cancer variants

Most variants in most genes across most organisms have an unknown impact on the function of the corresponding gene. This gap in knowledge is especially acute in cancer, where clinical sequencing of tumors now routinely reveals patient-specific variants whose functional impact on the corresponding genes is unknown, impeding clinical utility. Transcriptional profiling was able to systematically distinguish these variants of unknown significance as impactful vs. neutral in an approach called expression-based variant-impact phenotyping. We profiled a set of lung adenocarcinoma-associated somatic variants using Cell Painting, a morphological profiling assay that captures features of cells based on microscopy using six stains of cell and organelle components. Using deep-learning-extracted features from each cell's image, we found that cell morphological profiling (cmVIP) can predict variants' functional impact and, particularly at the single-cell level, reveals biological insights into variants that can be explored at our public online portal. Given its low cost, convenient implementation, and single-cell resolution, cmVIP profiling therefore seems promising as an avenue for using non-gene specific assays to systematically assess the impact of variants, including disease-associated alleles, on gene function.

Temporal and Locational Values of Images Affecting the Deep Learning of Cancer Stem Cell Morphology

Deep learning is being increasingly applied for obtaining digital microscopy image data of cells. Well-defined annotated cell images have contributed to the development of the technology. Cell morphology is an inherent characteristic of each cell type. Moreover, the morphology of a cell changes during its lifetime because of cellular activity. Artificial intelligence (AI) capable of recognizing a mouse-induced pluripotent stem (miPS) cell cultured in a medium containing Lewis lung cancer (LLC) cell culture-conditioned medium (cm), miPS-LLCcm cell, which is a cancer stem cell (CSC) derived from miPS cell, would be suitable for basic and applied science. This study aims to clarify the limitation of AI models constructed using different datasets and the versatility improvement of AI models. The trained AI was used to segment CSC in phase-contrast images using conditional generative adversarial networks (CGAN). The dataset included blank cell images that were used for training the AI but they did not affect the quality of predicting CSC in phase contrast images compared with the dataset without the blank cell images. AI models trained using images of 1-day culture could predict CSC in images of 2-day culture; however, the quality of the CSC prediction was reduced. Convolutional neural network (CNN) classification indicated that miPS-LLCcm cell image classification was done based on cultivation day. By using a dataset that included images of each cell culture day, the prediction of CSC remains to be improved. This is useful because cells do not change the characteristics of stem cells owing to stem cell marker expression, even if the cell morphology changes during culture.

In-Cell Structural Biology by NMR: The Benefits of the Atomic Scale

In-cell structural biology aims at extracting structural information about proteins or nucleic acids in their native, cellular environment. This emerging field holds great promise and is already providing new facts and outlooks of interest at both fundamental and applied levels. NMR spectroscopy has important contributions on this stage: It brings information on a broad variety of nuclei at the atomic scale, which ensures its great versatility and uniqueness. Here, we detail the methods, the fundamental knowledge, and the applications in biomedical engineering related to in-cell structural biology by NMR. We finally propose a brief overview of the main other techniques in the field (EPR, smFRET, cryo-ET, etc.) to draw some advisable developments for in-cell NMR. In the era of large-scale screenings and deep learning, both accurate and qualitative experimental evidence are as essential as ever to understand the interior life of cells. In-cell structural biology by NMR spectroscopy can generate such a knowledge, and it does so at the atomic scale. This review is meant to deliver comprehensive but accessible information, with advanced technical details and reflections on the methods, the nature of the results, and the future of the field.

Interfacing Live Cells with Surfaces: A Concurrent Control Technique for Quantifying Surface Ligand Activity

Surface ligand activity is a key design parameter for successfully interfacing surfaces with cells─whether in the context of in vitro investigations for understanding cellular signaling pathways or more applied applications in drug delivery and medical implants. Unlike other crucial surface parameters, such as stiffness and roughness, surface ligand activity is typically based on a set of assumptions rather than directly measured, giving rise to interpretations of cell adhesion that can vary with the assumptions made. To fill this void, we have developed a concurrent control technique for directly characterizing in vitro ligand surface activity. Pairs of gold-coated glass chips were biofunctionalized with RGD ligand in a parallel workflow: one chip for in vitro applications and the other for surface plasmon resonance (SPR)-based RGD activity characterization. Recombinant α_Vβ₃ integrins were injected over the SPR chip surface as mimics of the cellular-membrane-bound receptors and the resulting binding kinetics parameterized to quantify surface ligand activity. These activity measurements were correlated with cell morphological features, measured by interfacing MDA-MB-231 cells with the in vitro chip surfaces on the live cell microscope. We demonstrate how the interpretation of a cell phenotype based on direct activity measurements can vary markedly from interpretations based on assumed activity. The SPR concurrent control approach has multiple advantages due to the fact that SPR is a standardized technique and has the sensitivity to measure ligand activity across the most relevant range of extracellular surface densities, while the in vitro chip design can be used with all commonly used light microscopy modalities (e.g., phase contrast, DIC, and fluorescence) so that a wide range of phenotypic and molecular markers can be correlated to the ligand surface activity.

A cell-cell atlas approach for understanding symbiotic interactions between microbes

Natural environments are composed of a huge diversity of microorganisms interacting with each other to form complex functional networks. Our understanding of the operative nature of host-symbiont associations is limited because propagating such associations in a laboratory is challenging. The advent of single-cell technologies applied to, for example, animal cells and apicomplexan parasites has revolutionized our understanding of development and disease. Such cell atlas approaches generate maps of cell-specific processes and variations within cellular populations. These methods can now be combined with cellular-imaging so that interaction stage versus transcriptome state can be quantized for microbe-microbe interactions. We predict that the combination of these methods applied to the study of symbioses will transform our understanding of many ecological interactions, including those sampled directly from natural environments.

Deep learning for bioimage analysis in developmental biology

Deep learning has transformed the way large and complex image datasets can be processed, reshaping what is possible in bioimage analysis. As the complexity and size of bioimage data continues to grow, this new analysis paradigm is becoming increasingly ubiquitous. In this Review, we begin by introducing the concepts needed for beginners to understand deep learning. We then review how deep learning has impacted bioimage analysis and explore the open-source resources available to integrate it into a research project. Finally, we discuss the future of deep learning applied to cell and developmental biology. We analyze how state-of-the-art methodologies have the potential to transform our understanding of biological systems through new image-based analysis and modelling that integrate multimodal inputs in space and time.

A Crop Image Segmentation and Extraction Algorithm Based on Mask RCNN

The wide variety of crops in the image of agricultural products and the confusion with the surrounding environment information makes it difficult for traditional methods to extract crops accurately and efficiently. In this paper, an automatic extraction algorithm is proposed for crop images based on Mask RCNN. First, the Fruits 360 Dataset label is set with Labelme. Then, the Fruits 360 Dataset is preprocessed. Next, the data are divided into a training set and a test set. Additionally, an improved Mask RCNN network model structure is established using the PyTorch 1.8.1 deep learning framework, and path aggregation and features are added to the network design enhanced functions, optimized region extraction network, and feature pyramid network. The spatial information of the feature map is saved by the bilinear interpolation method in ROIAlign. Finally, the edge accuracy of the segmentation mask is further improved by adding a micro-fully connected layer to the mask branch of the ROI output, employing the Sobel operator to predict the target edge, and adding the edge loss to the loss function. Compared with FCN and Mask RCNN and other image extraction algorithms, the experimental results demonstrate that the improved Mask RCNN algorithm proposed in this paper is better in the precision, Recall, Average precision, Mean Average Precision, and F1 scores of crop image extraction results.