The multimodality cell segmentation challenge: toward universal solutions

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multimodality cell segmentation benchmark, comprising more than 1,500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.

A Foundation Model for Cell Segmentation

Cells are a fundamental unit of biological organization, and identifying them in imaging data - cell segmentation - is a critical task for various cellular imaging experiments. While deep learning methods have led to substantial progress on this problem, most models in use are specialist models that work well for specific domains. Methods that have learned the general notion of "what is a cell" and can identify them across different domains of cellular imaging data have proven elusive. In this work, we present CellSAM, a foundation model for cell segmentation that generalizes across diverse cellular imaging data. CellSAM builds on top of the Segment Anything Model (SAM) by developing a prompt engineering approach for mask generation. We train an object detector, CellFinder, to automatically detect cells and prompt SAM to generate segmentations. We show that this approach allows a single model to achieve human-level performance for segmenting images of mammalian cells (in tissues and cell culture), yeast, and bacteria collected across various imaging modalities. We show that CellSAM has strong zero-shot performance and can be improved with a few examples via few-shot learning. We also show that CellSAM can unify bioimaging analysis workflows such as spatial transcriptomics and cell tracking. A deployed version of CellSAM is available at https://cellsam.deepcell.org/.

Accurate single-molecule spot detection for image-based spatial transcriptomics with weakly supervised deep learning

Image-based spatial transcriptomics methods enable transcriptome-scale gene expression measurements with spatial information but require complex, manually-tuned analysis pipelines. We present Polaris, an analysis pipeline for image-based spatial transcriptomics that combines deep learning models for cell segmentation and spot detection with a probabilistic gene decoder to quantify single-cell gene expression accurately. Polaris offers a unifying, turnkey solution for analyzing spatial transcriptomics data from MERFSIH, seqFISH, or ISS experiments. Polaris is available through the DeepCell software library (https://github.com/vanvalenlab/deepcell-spots) and https://www.deepcell.org.

A microwell platform for high-throughput longitudinal phenotyping and selective retrieval of organoids

Organoids are powerful experimental models for studying the ontogeny and progression of various diseases including cancer. Organoids are conventionally cultured in bulk using an extracellular matrix mimic. However, bulk-cultured organoids physically overlap, making it impossible to track the growth of individual organoids over time in high throughput. Moreover, local spatial variations in bulk matrix properties make it difficult to assess whether observed phenotypic heterogeneity between organoids results from intrinsic cell differences or differences in the microenvironment. Here, we developed a microwell-based method that enables high-throughput quantification of image-based parameters for organoids grown from single cells, which can further be retrieved from their microwells for molecular profiling. Coupled with a deep learning image-processing pipeline, we characterized phenotypic traits including growth rates, cellular movement, and apical-basal polarity in two CRISPR-engineered human gastric organoid models, identifying genomic changes associated with increased growth rate and changes in accessibility and expression correlated with apical-basal polarity. A record of this paper's transparent peer review process is included in the supplemental information.

A spatially resolved timeline of the human maternal-fetal interface

Beginning in the first trimester, fetally derived extravillous trophoblasts (EVTs) invade the uterus and remodel its spiral arteries, transforming them into large, dilated blood vessels. Several mechanisms have been proposed to explain how EVTs coordinate with the maternal decidua to promote a tissue microenvironment conducive to spiral artery remodelling (SAR)1-3. However, it remains a matter of debate regarding which immune and stromal cells participate in these interactions and how this evolves with respect to gestational age. Here we used a multiomics approach, combining the strengths of spatial proteomics and transcriptomics, to construct a spatiotemporal atlas of the human maternal-fetal interface in the first half of pregnancy. We used multiplexed ion beam imaging by time-of-flight and a 37-plex antibody panel to analyse around 500,000 cells and 588 arteries within intact decidua from 66 individuals between 6 and 20 weeks of gestation, integrating this dataset with co-registered transcriptomics profiles. Gestational age substantially influenced the frequency of maternal immune and stromal cells, with tolerogenic subsets expressing CD206, CD163, TIM-3, galectin-9 and IDO-1 becoming increasingly enriched and colocalized at later time points. By contrast, SAR progression preferentially correlated with EVT invasion and was transcriptionally defined by 78 gene ontology pathways exhibiting distinct monotonic and biphasic trends. Last, we developed an integrated model of SAR whereby invasion is accompanied by the upregulation of pro-angiogenic, immunoregulatory EVT programmes that promote interactions with the vascular endothelium while avoiding the activation of maternal immune cells.

Single-cell metabolic profiling of human cytotoxic T cells

Cellular metabolism regulates immune cell activation, differentiation and effector functions, but current metabolic approaches lack single-cell resolution and simultaneous characterization of cellular phenotype. In this study, we developed an approach to characterize the metabolic regulome of single cells together with their phenotypic identity. The method, termed single-cell metabolic regulome profiling (scMEP), quantifies proteins that regulate metabolic pathway activity using high-dimensional antibody-based technologies. We employed mass cytometry (cytometry by time of flight, CyTOF) to benchmark scMEP against bulk metabolic assays by reconstructing the metabolic remodeling of in vitro-activated naive and memory CD8+ T cells. We applied the approach to clinical samples and identified tissue-restricted, metabolically repressed cytotoxic T cells in human colorectal carcinoma. Combining our method with multiplexed ion beam imaging by time of flight (MIBI-TOF), we uncovered the spatial organization of metabolic programs in human tissues, which indicated exclusion of metabolically repressed immune cells from the tumor-immune boundary. Overall, our approach enables robust approximation of metabolic and functional states in individual cells.

Deep learning for cellular image analysis

Recent advances in computer vision and machine learning underpin a collection of algorithms with an impressive ability to decipher the content of images. These deep learning algorithms are being applied to biological images and are transforming the analysis and interpretation of imaging data. These advances are positioned to render difficult analyses routine and to enable researchers to carry out new, previously impossible experiments. Here we review the intersection between deep learning and cellular image analysis and provide an overview of both the mathematical mechanics and the programming frameworks of deep learning that are pertinent to life scientists. We survey the field's progress in four key applications: image classification, image segmentation, object tracking, and augmented microscopy. Last, we relay our labs' experience with three key aspects of implementing deep learning in the laboratory: annotating training data, selecting and training a range of neural network architectures, and deploying solutions. We also highlight existing datasets and implementations for each surveyed application.

A Structured Tumor-Immune Microenvironment in Triple Negative Breast Cancer Revealed by Multiplexed Ion Beam Imaging

The immune system is critical in modulating cancer progression, but knowledge of immune composition, phenotype, and interactions with tumor is limited. We used multiplexed ion beam imaging by time-of-flight (MIBI-TOF) to simultaneously quantify in situ expression of 36 proteins covering identity, function, and immune regulation at sub-cellular resolution in 41 triple-negative breast cancer patients. Multi-step processing, including deep-learning-based segmentation, revealed variability in the composition of tumor-immune populations across individuals, reconciled by overall immune infiltration and enriched co-occurrence of immune subpopulations and checkpoint expression. Spatial enrichment analysis showed immune mixed and compartmentalized tumors, coinciding with expression of PD1, PD-L1, and IDO in a cell-type- and location-specific manner. Ordered immune structures along the tumor-immune border were associated with compartmentalization and linked to survival. These data demonstrate organization in the tumor-immune microenvironment that is structured in cellular composition, spatial arrangement, and regulatory-protein expression and provide a framework to apply multiplexed imaging to immune oncology.

Measuring Signaling and RNA-Seq in the Same Cell Links Gene Expression to Dynamic Patterns of NF-κB Activation

Signaling proteins display remarkable cell-to-cell heterogeneity in their dynamic responses to stimuli, but the consequences of this heterogeneity remain largely unknown. For instance, the contribution of the dynamics of the innate immune transcription factor nuclear factor κB (NF-κB) to gene expression output is disputed. Here we explore these questions by integrating live-cell imaging approaches with single-cell sequencing technologies. We used this approach to measure both the dynamics of lipopolysaccharide-induced NF-κB activation and the global transcriptional response in the same individual cell. Our results identify multiple, distinct cytokine expression patterns that are correlated with NF-κB activation dynamics, establishing a functional role for NF-κB dynamics in determining cellular phenotypes. Applications of this approach to other model systems and single-cell sequencing technologies have significant potential for discovery, as it is now possible to trace cellular behavior from the initial stimulus, through the signaling pathways, down to genome-wide changes in gene expression, all inside of a single cell.

A single-molecule Hershey-Chase experiment

Ever since Hershey and Chase used phages to establish DNA as the carrier of genetic information in 1952, the precise mechanisms of phage DNA translocation have been a mystery. Although bulk measurements have set a timescale for in vivo DNA translocation during bacteriophage infection, measurements of DNA ejection by single bacteriophages have only been made in vitro. Here, we present direct visualization of single bacteriophages infecting individual Escherichia coli cells. For bacteriophage λ, we establish a mean ejection time of roughly 5 min with significant cell-to-cell variability, including pausing events. In contrast, corresponding in vitro single-molecule ejections are more uniform and finish within 10 s. Our data reveal that when plotted against the amount of DNA ejected, the velocity of ejection for two different genome lengths collapses onto a single curve. This suggests that in vivo ejections are controlled by the amount of DNA ejected. In contrast, in vitro DNA ejections are governed by the amount of DNA left inside the capsid. This analysis provides evidence against a purely intrastrand repulsion-based mechanism and suggests that cell-internal processes dominate. This provides a picture of the early stages of phage infection and sheds light on the problem of polymer translocation.

Ion-dependent dynamics of DNA ejections for bacteriophage lambda

We studied the control parameters that govern the dynamics of in vitro DNA ejection in bacteriophage lambda. Previous work demonstrated that bacteriophage DNA is highly pressurized, and this pressure has been hypothesized to help drive DNA ejection. Ions influence this process by screening charges on DNA; however, a systematic variation of salt concentrations to explore these effects has not been undertaken. To study the nature of the forces driving DNA ejection, we performed in vitro measurements of DNA ejection in bulk and at the single-phage level. We present measurements on the dynamics of ejection and on the self-repulsion force driving ejection. We examine the role of ion concentration and identity in both measurements, and show that the charge of counterions is an important control parameter. These measurements show that the mobility of ejecting DNA is independent of ionic concentrations for a given amount of DNA in the capsid. We also present evidence that phage DNA forms loops during ejection, and confirm that this effect occurs using optical tweezers.

Biochemistry on a leash: the roles of tether length and geometry in signal integration proteins

We use statistical mechanics and simple ideas from polymer physics to develop a quantitative model of proteins whose activity is controlled by flexibly tethered ligands and receptors. We predict how the properties of tethers influence the function of these proteins and demonstrate how their tether length dependence can be exploited to construct proteins whose integration of multiple signals can be tuned. One case study to which we apply these ideas is that of the Wiskott-Aldrich Syndrome Proteins as activators of actin polymerization. More generally, tethered ligands competing with those free in solution are common phenomena in biology, making this an important specific example of a widespread biological idea.