An objective comparison of cell-tracking algorithms

Enhancing cell instance segmentation in scanning electron microscopy images via a deep contour closing operator

Accurately segmenting and individualizing cells in scanning electron microscopy (SEM) images is a highly promising technique for elucidating tissue architecture in oncology. While current artificial intelligence (AI)-based methods are effective, errors persist, necessitating time-consuming manual corrections, particularly in areas where the quality of cell contours in the image is poor and requires gap filling. This study presents a novel AI-driven approach for refining cell boundary delineation to improve instance-based cell segmentation in SEM images, also reducing the necessity for residual manual correction. A convolutional neural network (CNN) Closing Operator (COp-Net) is introduced to address gaps in cell contours, effectively filling in regions with deficient or absent information. The network takes as input cell contour probability maps with potentially inadequate or missing information and outputs corrected cell contour delineations. The lack of training data was addressed by generating low integrity probability maps using a tailored partial differential equation (PDE). To ensure reproducibility, COp-Net weights and the source code for solving the PDE are publicly available at https://github.com/Florian-40/CellSegm. We showcase the efficacy of our approach in augmenting cell boundary precision using both private SEM images from patient-derived xenograft (PDX) hepatoblastoma tissues and publicly accessible images datasets. The proposed cell contour closing operator exhibits a notable improvement in tested datasets, achieving respectively close to 50% (private data) and 10% (public data) increase in the accurately-delineated cell proportion compared to state-of-the-art methods. Additionally, the need for manual corrections was significantly reduced, therefore facilitating the overall digitalization process. Our results demonstrate a notable enhancement in the accuracy of cell instance segmentation, particularly in highly challenging regions where image quality compromises the integrity of cell boundaries, necessitating gap filling. Therefore, our work should ultimately facilitate the study of tumour tissue bioarchitecture in onconanotomy field.

Scale selection and machine learning based cell segmentation and tracking in time lapse microscopy

Monitoring and tracking of cell motion is a key component for understanding disease mechanisms and evaluating the effects of treatments. Time-lapse optical microscopy has been commonly employed for studying cell cycle phases. However, usual manual cell tracking is very time consuming and has poor reproducibility. Automated cell tracking techniques are challenged by variability of cell region intensity distributions and resolution limitations. In this work, we introduce a comprehensive cell segmentation and tracking methodology. A key contribution of this work is that it employs multi-scale space-time interest point detection and characterization for automatic scale selection and cell segmentation. Another contribution is the use of a neural network with class prototype balancing for detection of cell regions. This work also offers a structured mathematical framework that uses graphs for track generation and cell event detection. We evaluated cell segmentation, detection, and tracking performance of our method on time-lapse sequences of the Cell Tracking Challenge (CTC). We also compared our technique to top performing techniques from CTC. Performance evaluation results indicate that the proposed methodology is competitive with these techniques, and that it generalizes very well to diverse cell types and sizes, and multiple imaging techniques. The code of our method is publicly available on https://github.com/smakrogi/CSTQ_Pub/ , (release v.3.2).

Supra-second tracking and live-cell karyotyping reveal principles of mitotic chromosome dynamics

Mitotic chromosome dynamics are essential for the three-dimensional organization of the genome during the cell cycle, but the spatiotemporal characteristics of this process remain unclear due to methodological challenges. While Hi-C methods capture interchromosomal contacts, they lack single-cell temporal dynamics, whereas microscopy struggles with bleaching and phototoxicity. Here, to overcome these limitations, we introduce Facilitated Segmentation and Tracking of Chromosomes in Mitosis Pipeline (FAST CHIMP), pairing time-lapse super-resolution microscopy with deep learning. FAST CHIMP tracked all human chromosomes with 8-s resolution from prophase to telophase, identified 15 out of 23 homologue pairs in single cells and compared chromosomal positioning between mother and daughter cells. It revealed a centrosome-motion-dependent flow that governs the mapping between chromosome locations at prophase and their metaphase plate position. In addition, FAST CHIMP measured supra-second dynamics of intra- and interchromosomal contacts. This tool adds a dynamic dimension to the study of chromatin behaviour in live cells, promising advances beyond the scope of existing methods.

Topology-preserving contourwise shape fusion

The preservation of morphological features, such as protrusions and concavities, and of the topology of input shapes is important when establishing reference data for benchmarking segmentation algorithms or when constructing a mean or median shape. We present a contourwise topology-preserving fusion method, called shape-aware topology-preserving means (SATM), for merging complex simply connected shapes. The method is based on key point matching and piecewise contour averaging. Unlike existing pixelwise and contourwise fusion methods, SATM preserves topology and does not smooth morphological features. We also present a detailed comparison of SATM with state-of-the-art fusion techniques for the purpose of benchmarking and median shape construction. Our experiments show that SATM outperforms these techniques in terms of shape-related measures that reflect shape complexity, manifesting itself as a reliable method for both establishing a consensus of segmentation annotations and for computing mean shapes.

Generative frame interpolation enhances tracking of biological objects in time-lapse microscopy

Object tracking in microscopy videos is crucial for understanding biological processes. While existing methods often require fine-tuning tracking algorithms to fit the image dataset, here we explored an alternative paradigm: augmenting the image time-lapse dataset to fit the tracking algorithm. To test this approach, we evaluated whether generative video frame interpolation can augment the temporal resolution of time-lapse microscopy and facilitate object tracking in multiple biological contexts. We systematically compared the capacity of Latent Diffusion Model for Video Frame Interpolation (LDMVFI), Real-time Intermediate Flow Estimation (RIFE), Compression-Driven Frame Interpolation (CDFI), and Frame Interpolation for Large Motion (FILM) to generate synthetic microscopy images derived from interpolating real images. Our testing image time series ranged from fluorescently labeled nuclei to bacteria, yeast, cancer cells, and organoids. We showed that the off-the-shelf frame interpolation algorithms produced bio-realistic image interpolation even without dataset-specific retraining, as judged by high structural image similarity and the capacity to produce segmentations that closely resemble results from real images. Using a simple tracking algorithm based on mask overlap, we confirmed that frame interpolation significantly improved tracking across several datasets without requiring extensive parameter tuning and capturing complex trajectories that were difficult to resolve in the original image time series. Taken together, our findings highlight the potential of generative frame interpolation to improve tracking in time-lapse microscopy across diverse scenarios, suggesting that a generalist tracking algorithm for microscopy could be developed by combining deep learning segmentation models with generative frame interpolation.

Universal consensus 3D segmentation of cells from 2D segmented stacks

Cell segmentation is the foundation of a wide range of microscopy-based biological studies. Deep learning has revolutionized 2D cell segmentation, enabling generalized solutions across cell types and imaging modalities. This has been driven by the ease of scaling up image acquisition, annotation, and computation. However, 3D cell segmentation, requiring dense annotation of 2D slices still poses significant challenges. Manual labeling of 3D cells to train broadly applicable segmentation models is prohibitive. Even in high-contrast images annotation is ambiguous and time-consuming. Here we develop a theory and toolbox, u-Segment3D, for 2D-to-3D segmentation, compatible with any 2D method generating pixel-based instance cell masks. u-Segment3D translates and enhances 2D instance segmentations to a 3D consensus instance segmentation without training data, as demonstrated on 11 real-life datasets, >70,000 cells, spanning single cells, cell aggregates, and tissue. Moreover, u-Segment3D is competitive with native 3D segmentation, even exceeding when cells are crowded and have complex morphologies.

CeDaD-a novel assay for simultaneous tracking of cell death and division in a single population

The cell division cycle and the various forms of programmed cell death are interconnected. A prominent example is the tumor suppressor p53, which not only induces apoptosis but also plays an important role in the arrest of the cell cycle. Consequently, simultaneous analysis of cell division and cell death is frequently of significant interest in cell biology research. Traditionally, these processes require distinct assays, making concurrent analysis challenging. To address this, we present a novel combined assay, called CeDaD assay-Cell Death and Division assay-which allows for the simultaneous quantification of cell division and cell death within a single-cell population. This assay utilizes a straightforward flow cytometric approach, combining a staining based on carboxyfluorescein succinimidyl ester (CFSE) to monitor cell division with an annexin V-derived staining to assess the extent of cell death.

Segment Anything for Microscopy

Accurate segmentation of objects in microscopy images remains a bottleneck for many researchers despite the number of tools developed for this purpose. Here, we present Segment Anything for Microscopy (μSAM), a tool for segmentation and tracking in multidimensional microscopy data. It is based on Segment Anything, a vision foundation model for image segmentation. We extend it by fine-tuning generalist models for light and electron microscopy that clearly improve segmentation quality for a wide range of imaging conditions. We also implement interactive and automatic segmentation in a napari plugin that can speed up diverse segmentation tasks and provides a unified solution for microscopy annotation across different microscopy modalities. Our work constitutes the application of vision foundation models in microscopy, laying the groundwork for solving image analysis tasks in this domain with a small set of powerful deep learning models.

DeepKymoTracker: A tool for accurate construction of cell lineage trees for highly motile cells

Time-lapse microscopy has long been used to record cell lineage trees. Successful construction of a lineage tree requires tracking and preserving the identity of multiple cells across many images. If a single cell is misidentified the identity of all its progeny will be corrupted and inferences about heritability may be incorrect. Successfully avoiding such identity errors is challenging, however, when studying highly-motile cells such as T lymphocytes which readily change shape from one image to the next. To address this problem, we developed DeepKymoTracker, a pipeline for combined tracking and segmentation. Central to DeepKymoTracker is the use of a seed, a marker for each cell which transmits information about cell position and identity between sets of images during tracking, as well as between tracking and segmentation steps. The seed allows a 3D convolutional neural network (CNN) to detect and associate cells across several consecutive images in an integrated way, reducing the risk of a single poor image corrupting cell identity. DeepKymoTracker was trained extensively on synthetic and experimental T lymphocyte images. It was benchmarked against five publicly available, automatic analysis tools and outperformed them in almost all respects. The software is written in pure Python and is freely available. We suggest this tool is particularly suited to the tracking of cells in suspension, whose fast motion makes lineage assembly particularly difficult.

GobletNet: Wavelet-Based High-Frequency Fusion Network for Semantic Segmentation of Electron Microscopy Images

Semantic segmentation of electron microscopy (EM) images is crucial for nanoscale analysis. With the development of deep neural networks (DNNs), semantic segmentation of EM images has achieved remarkable success. However, current EM image segmentation models are usually extensions or adaptations of natural or biomedical models. They lack the full exploration and utilization of the intrinsic characteristics of EM images. Furthermore, they are often designed only for several specific segmentation objects and lack versatility. In this study, we quantitatively analyze the characteristics of EM images compared with those of natural and other biomedical images via the wavelet transform. To better utilize these characteristics, we design a high-frequency (HF) fusion network, GobletNet, which outperforms state-of-the-art models by a large margin in the semantic segmentation of EM images. We use the wavelet transform to generate HF images as extra inputs and use an extra encoding branch to extract HF information. Furthermore, we introduce a fusion-attention module (FAM) into GobletNet to facilitate better absorption and fusion of information from raw images and HF images. Extensive benchmarking on seven public EM datasets (EPFL, CREMI, SNEMI3D, UroCell, MitoEM, Nanowire and BetaSeg) demonstrates the effectiveness of our model. The code is available at https://github.com/Yanfeng-Zhou/GobletNet.

Improving 3D deep learning segmentation with biophysically motivated cell synthesis

Biomedical research increasingly relies on three-dimensional (3D) cell culture models and artificial-intelligence-based analysis can potentially facilitate a detailed and accurate feature extraction on a single-cell level. However, this requires for a precise segmentation of 3D cell datasets, which in turn demands high-quality ground truth for training. Manual annotation, the gold standard for ground truth data, is too time-consuming and thus not feasible for the generation of large 3D training datasets. To address this, we present a framework for generating 3D training data, which integrates biophysical modeling for realistic cell shape and alignment. Our approach allows the in silico generation of coherent membrane and nuclei signals, that enable the training of segmentation models utilizing both channels for improved performance. Furthermore, we present a generative adversarial network (GAN) training scheme that generates not only image data but also matching labels. Quantitative evaluation shows superior performance of biophysical motivated synthetic training data, even outperforming manual annotation and pretrained models. This underscores the potential of incorporating biophysical modeling for enhancing synthetic training data quality.

A data-driven approach to establishing cell motility patterns as predictors of macrophage subtypes and their relation to cell morphology

The motility of macrophages in response to microenvironment stimuli is a hallmark of innate immunity, where macrophages play pro-inflammatory or pro-reparatory roles depending on their activation status during wound healing. Cell size and shape have been informative in defining macrophage subtypes. Studies show pro and anti-inflammatory macrophages exhibit distinct migratory behaviors, in vitro, in 3D and in vivo but this link has not been rigorously studied. We apply both morphology and motility-based image processing approaches to analyze live cell images consisting of macrophage phenotypes. Macrophage subtypes are differentiated from primary murine bone marrow derived macrophages using a potent lipopolysaccharide (LPS) or cytokine interleukin-4 (IL-4). We show that morphology is tightly linked to motility, which leads to our hypothesis that motility analysis could be used alone or in conjunction with morphological features for improved prediction of macrophage subtypes. We train a support vector machine (SVM) classifier to predict macrophage subtypes based on morphology alone, motility alone, and both morphology and motility combined. We show that motility has comparable predictive capabilities as morphology. However, using both measures can enhance predictive capabilities. While motility and morphological features can be individually ambiguous identifiers, together they provide significantly improved prediction accuracies (75%) from a training dataset of 1000 cells tracked over time using only phase contrast time-lapse microscopy. Thus, the approach combining cell motility and cell morphology information can lead to methods that accurately assess functionally diverse macrophage phenotypes quickly and efficiently. This can support the development of cost efficient and high through-put methods for screening biochemicals targeting macrophage polarization.

A spheroid whole mount drug testing pipeline with machine-learning based image analysis identifies cell-type specific differences in drug efficacy on a single-cell level

BACKGROUND

The growth and drug response of tumors are influenced by their stromal composition, both in vivo and 3D-cell culture models. Cell-type inherent features as well as mutual relationships between the different cell types in a tumor might affect drug susceptibility of the tumor as a whole and/or of its cell populations. However, a lack of single-cell procedures with sufficient detail has hampered the automated observation of cell-type-specific effects in three-dimensional stroma-tumor cell co-cultures.

METHODS

Here, we developed a high-content pipeline ranging from the setup of novel tumor-fibroblast spheroid co-cultures over optical tissue clearing, whole mount staining, and 3D confocal microscopy to optimized 3D-image segmentation and a 3D-deep-learning model to automate the analysis of a range of cell-type-specific processes, such as cell proliferation, apoptosis, necrosis, drug susceptibility, nuclear morphology, and cell density.

RESULTS

This demonstrated that co-cultures of KP-4 tumor cells with CCD-1137Sk fibroblasts exhibited a growth advantage compared to tumor cell mono-cultures, resulting in higher cell counts following cytostatic treatments with paclitaxel and doxorubicin. However, cell-type-specific single-cell analysis revealed that this apparent benefit of co-cultures was due to a higher resilience of fibroblasts against the drugs and did not indicate a higher drug resistance of the KP-4 cancer cells during co-culture. Conversely, cancer cells were partially even more susceptible in the presence of fibroblasts than in mono-cultures.

CONCLUSION

In summary, this underlines that a novel cell-type-specific single-cell analysis method can reveal critical insights regarding the mechanism of action of drug substances in three-dimensional cell culture models.

Harnessing the power of artificial intelligence for human living organoid research

As a powerful paradigm, artificial intelligence (AI) is rapidly impacting every aspect of our day-to-day life and scientific research through interdisciplinary transformations. Living human organoids (LOs) have a great potential for in vitro reshaping many aspects of in vivo true human organs, including organ development, disease occurrence, and drug responses. To date, AI has driven the revolutionary advances of human organoids in life science, precision medicine and pharmaceutical science in an unprecedented way. Herein, we provide a forward-looking review, the frontiers of LOs, covering the engineered construction strategies and multidisciplinary technologies for developing LOs, highlighting the cutting-edge achievements and the prospective applications of AI in LOs, particularly in biological study, disease occurrence, disease diagnosis and prediction and drug screening in preclinical assay. Moreover, we shed light on the new research trends harnessing the power of AI for LO research in the context of multidisciplinary technologies. The aim of this paper is to motivate researchers to explore organ function throughout the human life cycle, narrow the gap between in vitro microphysiological models and the real human body, accurately predict human-related responses to external stimuli (cues and drugs), accelerate the preclinical-to-clinical transformation, and ultimately enhance the health and well-being of patients.

Cellular behavior analysis from live-cell imaging of TCR T cell-cancer cell interactions

T cell therapies, such as chimeric antigen receptor (CAR) T cells and T cell receptor (TCR) T cells, are a growing class of anti-cancer treatments. However, expansion to novel indications and beyond last-line treatment requires engineering cells' dynamic population behaviors. Here we develop the tools for cellular behavior analysis of T cells from live-cell imaging, a common and inexpensive experimental setup used to evaluate engineered T cells. We first develop a state-of-the-art segmentation and tracking pipeline, Caliban, based on human-in-the-loop deep learning. We then build the Occident pipeline to collect a catalog of phenotypes that characterize cell populations, morphology, movement, and interactions in co-cultures of modified T cells and antigen-presenting tumor cells. We use Caliban and Occident to interrogate how interactions between T cells and cancer cells differ when beneficial knock-outs of RASA2 and CUL5 are introduced into TCR T cells. We apply spatiotemporal models to quantify T cell recruitment and proliferation after interactions with cancer cells. We discover that, compared to a safe harbor knockout control, RASA2 knockout T cells have longer interaction times with cancer cells leading to greater T cell activation and killing efficacy, while CUL5 knockout T cells have increased proliferation rates leading to greater numbers of T cells for hunting. Together, segmentation and tracking from Caliban and phenotype quantification from Occident enable cellular behavior analysis to better engineer T cell therapies for improved cancer treatment.

Scale Selection and Machine Learning-based Cell Segmentation and Tracking in Time Lapse Microscopy

Monitoring and tracking of cell motion is a key component for understanding disease mechanisms and evaluating the effects of treatments. Time-lapse optical microscopy has been commonly employed for studying cell cycle phases. However, usual manual cell tracking is very time consuming and has poor reproducibility. Automated cell tracking techniques are challenged by variability of cell region intensity distributions and resolution limitations. In this work, we introduce a comprehensive cell segmentation and tracking methodology. A key contribution of this work is that it employs multi-scale space-time interest point detection and characterization for automatic scale selection and cell segmentation. Another contribution is the use of a neural network with class prototype balancing for detection of cell regions. This work also offers a structured mathematical framework that uses graphs for track generation and cell event detection. We evaluated cell segmentation, detection, and tracking performance of our method on time-lapse sequences of the Cell Tracking Challenge (CTC). We also compared our technique to top performing techniques from CTC. Performance evaluation results indicate that the proposed methodology is competitive with these techniques, and that it generalizes very well to diverse cell types and sizes, and multiple imaging techniques.

The crucial role of bioimage analysts in scientific research and publication

Bioimage analysis (BIA), a crucial discipline in biological research, overcomes the limitations of subjective analysis in microscopy through the creation and application of quantitative and reproducible methods. The establishment of dedicated BIA support within academic institutions is vital to improving research quality and efficiency and can significantly advance scientific discovery. However, a lack of training resources, limited career paths and insufficient recognition of the contributions made by bioimage analysts prevent the full realization of this potential. This Perspective - the result of the recent The Company of Biologists Workshop 'Effectively Communicating Bioimage Analysis', which aimed to summarize the global BIA landscape, categorize obstacles and offer possible solutions - proposes strategies to bring about a cultural shift towards recognizing the value of BIA by standardizing tools, improving training and encouraging formal credit for contributions. We also advocate for increased funding, standardized practices and enhanced collaboration, and we conclude with a call to action for all stakeholders to join efforts in advancing BIA.

Label-free spatiotemporal decoding of single-cell fate via acoustic driven 3D tomography

Label-free three-dimensional imaging plays a crucial role in unraveling the complexities of cellular functions and interactions in biomedical research. Conventional single-cell optical tomography techniques offer affordability and the convenience of bypassing laborious cell labelling protocols. However, these methods are encumbered by restricted illumination scanning ranges on abaxial plane, resulting in the loss of intricate cellular imaging details. The ability to fully control cellular rotation across all angles has emerged as an optimal solution for capturing comprehensive structural details of cells. Here, we introduce a label-free, cost-effective, and readily fabricated contactless acoustic-induced vibration system, specifically designed to enable multi-degree-of-freedom rotation of cells, ultimately attaining stable in-situ rotation. Furthermore, by integrating this system with advanced deep learning technologies, we perform 3D reconstruction and morphological analysis on diverse cell types, thus validating groups of high-precision cell identification. Notably, long-term observation of cells reveals distinct features associated with drug-induced apoptosis in both cancerous and normal cells populations. This methodology, based on deep learning-enabled cell 3D reconstruction, charts a novel trajectory for groups of real-time cellular visualization, offering promising advancements in the realms of drug screening and post-single-cell analysis, thereby addressing potential clinical requisites.

CMTT-JTracker: a fully test-time adaptive framework serving automated cell lineage construction

Cell tracking is an essential function needed in automated cellular activity monitoring. In practice, processing methods striking a balance between computational efficiency and accuracy as well as demonstrating robust generalizability across diverse cell datasets are highly desired. This paper develops a central-metric fully test-time adaptive framework for cell tracking (CMTT-JTracker). Firstly, a CMTT mechanism is designed for the pre-segmentation of cell images, which enables extracting target information at different resolutions without additional training. Next, a multi-task learning network with the spatial attention scheme is developed to simultaneously realize detection and re-identification tasks based on features extracted by CMTT. Experimental results demonstrate that the CMTT-JTracker exhibits remarkable biological and tracking performance compared with benchmarking tracking methods. It achieves a multiple object tracking accuracy (MOTA) of $0.894$ on Fluo-N2DH-SIM+ and a MOTA of $0.850$ on PhC-C2DL-PSC. Experimental results further confirm that the CMTT applied solely as a segmentation unit outperforms the SOTA segmentation benchmarks on various datasets, particularly excelling in scenarios with dense cells. The Dice coefficients of the CMTT range from a high of $0.928$ to a low of $0.758$ across different datasets.

Ultrack: pushing the limits of cell tracking across biological scales

Tracking live cells across 2D, 3D, and multi-channel time-lapse recordings is crucial for understanding tissue-scale biological processes. Despite advancements in imaging technology, achieving accurate cell tracking remains challenging, particularly in complex and crowded tissues where cell segmentation is often ambiguous. We present Ultrack, a versatile and scalable cell-tracking method that tackles this challenge by considering candidate segmentations derived from multiple algorithms and parameter sets. Ultrack employs temporal consistency to select optimal segments, ensuring robust performance even under segmentation uncertainty. We validate our method on diverse datasets, including terabyte-scale developmental time-lapses of zebrafish, fruit fly, and nematode embryos, as well as multi-color and label-free cellular imaging. We show that Ultrack achieves state-of-the-art performance on the Cell Tracking Challenge and demonstrates superior accuracy in tracking densely packed embryonic cells over extended periods. Moreover, we propose an approach to tracking validation via dual-channel sparse labeling that enables high-fidelity ground truth generation, pushing the boundaries of long-term cell tracking assessment. Our method is freely available as a Python package with Fiji and napari plugins and can be deployed in a high-performance computing environment, facilitating widespread adoption by the research community.

Monitoring nucleolar-nucleoplasmic protein shuttling in living cells by high-content microscopy and automated image analysis

The nucleolus has core functions in ribosome biosynthesis, but also acts as a regulatory hub in a plethora of non-canonical processes, including cellular stress. Upon DNA damage, several DNA repair factors shuttle between the nucleolus and the nucleoplasm. Yet, the molecular mechanisms underlying such spatio-temporal protein dynamics remain to be deciphered. Here, we present a novel imaging platform to investigate nucleolar-nucleoplasmic protein shuttling in living cells. For image acquisition, we used a commercially available automated fluorescence microscope and for image analysis, we developed a KNIME workflow with implementation of machine learning-based tools. We validated the method with different nucleolar proteins, i.e., PARP1, TARG1 and APE1, by monitoring their shuttling dynamics upon oxidative stress. As a paradigm, we analyzed PARP1 shuttling upon H2O2 treatment in combination with a range of pharmacological inhibitors in a novel reporter cell line. These experiments revealed that inhibition of SIRT7 results in a loss of nucleolar PARP1 localization. Finally, we unraveled specific differences in PARP1 shuttling dynamics after co-treatment with H2O2 and different clinical PARP inhibitors. Collectively, this work delineates a highly sensitive and versatile bioimaging platform to investigate swift nucleolar-nucleoplasmic protein shuttling in living cells, which can be employed for pharmacological screening and in-depth mechanistic analyses.

A systematic evaluation of computational methods for cell segmentation

Cell segmentation is a fundamental task in analyzing biomedical images. Many computational methods have been developed for cell segmentation and instance segmentation, but their performances are not well understood in various scenarios. We systematically evaluated the performance of 18 segmentation methods to perform cell nuclei and whole cell segmentation using light microscopy and fluorescence staining images. We found that general-purpose methods incorporating the attention mechanism exhibit the best overall performance. We identified various factors influencing segmentation performances, including image channels, choice of training data, and cell morphology, and evaluated the generalizability of methods across image modalities. We also provide guidelines for choosing the optimal segmentation methods in various real application scenarios. We developed Seggal, an online resource for downloading segmentation models already pre-trained with various tissue and cell types, substantially reducing the time and effort for training cell segmentation models.

A multiparametric analysis including single-cell and subcellular feature assessment reveals differential behavior of spheroid cultures on distinct ultra-low attachment plate types

Spheroids have become principal three-dimensional models to study cancer, developmental processes, and drug efficacy. Single-cell analysis techniques have emerged as ideal tools to gauge the complexity of cellular responses in these models. However, the single-cell quantitative assessment based on 3D-microscopic data of the subcellular distribution of fluorescence markers, such as the nuclear/cytoplasm ratio of transcription factors, has largely remained elusive. For spheroid generation, ultra-low attachment plates are noteworthy due to their simplicity, compatibility with automation, and experimental and commercial accessibility. However, it is unknown whether and to what degree the plate type impacts spheroid formation and biology. This study developed a novel AI-based pipeline for the analysis of 3D-confocal data of optically cleared large spheroids at the wholemount, single-cell, and sub-cellular levels. To identify relevant samples for the pipeline, automated brightfield microscopy was employed to systematically compare the size and eccentricity of spheroids formed in six different plate types using four distinct human cell lines. This showed that all plate types exhibited similar spheroid-forming capabilities and the gross patterns of growth or shrinkage during 4 days after seeding were comparable. Yet, size and eccentricity varied systematically among specific cell lines and plate types. Based on this prescreen, spheroids of HaCaT keratinocytes and HT-29 cancer cells were further assessed. In HaCaT spheroids, the in-depth analysis revealed a correlation between spheroid size, cell proliferation, and the nuclear/cytoplasm ratio of the transcriptional coactivator, YAP1, as well as an inverse correlation with respect to cell differentiation. These findings, yielded with a spheroid model and at a single-cell level, corroborate earlier concepts of the role of YAP1 in cell proliferation and differentiation of keratinocytes in human skin. Further, the results show that the plate type may influence the outcome of experimental campaigns and that it is advisable to scan different plate types for the optimal configuration during a specific investigation.

A systematic evaluation of computational methods for cell segmentation

Cell segmentation is a fundamental task in analyzing biomedical images. Many computational methods have been developed for cell segmentation and instance segmentation, but their performances are not well understood in various scenarios. We systematically evaluated the performance of 18 segmentation methods to perform cell nuclei and whole cell segmentation using light microscopy and fluorescence staining images. We found that general-purpose methods incorporating the attention mechanism exhibit the best overall performance. We identified various factors influencing segmentation performances, including image channels, choice of training data, and cell morphology, and evaluated the generalizability of methods across image modalities. We also provide guidelines for choosing the optimal segmentation methods in various real application scenarios. We developed Seggal, an online resource for downloading segmentation models already pre-trained with various tissue and cell types, substantially reducing the time and effort for training cell segmentation models.

Deep 3D histology powered by tissue clearing, omics and AI

To comprehensively understand tissue and organism physiology and pathophysiology, it is essential to create complete three-dimensional (3D) cellular maps. These maps require structural data, such as the 3D configuration and positioning of tissues and cells, and molecular data on the constitution of each cell, spanning from the DNA sequence to protein expression. While single-cell transcriptomics is illuminating the cellular and molecular diversity across species and tissues, the 3D spatial context of these molecular data is often overlooked. Here, I discuss emerging 3D tissue histology techniques that add the missing third spatial dimension to biomedical research. Through innovations in tissue-clearing chemistry, labeling and volumetric imaging that enhance 3D reconstructions and their synergy with molecular techniques, these technologies will provide detailed blueprints of entire organs or organisms at the cellular level. Machine learning, especially deep learning, will be essential for extracting meaningful insights from the vast data. Further development of integrated structural, molecular and computational methods will unlock the full potential of next-generation 3D histology.

Isolated Random Forest Assisted Spatio-Temporal Ant Colony Evolutionary Algorithm for Cell Tracking in Time-Lapse Sequences

Multi-Object tracking in real world environments is a tough problem, especially for cell morphogenesis with division. Most cell tracking methods are hard to achieve reliable mitosis detection, efficient inter-frame matching, and accurate state estimation simultaneously within a unified tracking framework. In this paper, we propose a novel unified framework that leverages a spatio-temporal ant colony evolutionary algorithm to track cells amidst mitosis under measurement uncertainty. Each Bernoulli ant colony representing a migrating cell is able to capture the occurrence of mitosis through the proposed Isolation Random Forest (IRF)-assisted temporal mitosis detection algorithm with the assumption that mitotic cells exhibit unique spatio-temporal features different from non-mitotic ones. Guided by prediction of a division event, multiple ant colonies evolve between consecutive frames according to an augmented assignment matrix solved by the extended Hungarian method. To handle dense cell populations, an efficient group partition between cells and measurements is exploited, which enables multiple assignment tasks to be executed in parallel with a reduction in matrix dimension. After inter-frame traversing, the ant colony transitions to a foraging stage in which it begins approximating the Bernoulli parameter to estimate cell state by iteratively updating its pheromone field. Experiments on multi-cell tracking in the presence of cell mitosis and morphological changes are conducted, and the results demonstrate that the proposed method outperforms state-of-the-art approaches, striking a balance between accuracy and computational efficiency.

Bellybutton: accessible and customizable deep-learning image segmentation

The conversion of raw images into quantifiable data can be a major hurdle and time-sink in experimental research, and typically involves identifying region(s) of interest, a process known as segmentation. Machine learning tools for image segmentation are often specific to a set of tasks, such as tracking cells, or require substantial compute or coding knowledge to train and use. Here we introduce an easy-to-use (no coding required), image segmentation method, using a 15-layer convolutional neural network that can be trained on a laptop: Bellybutton. The algorithm trains on user-provided segmentation of example images, but, as we show, just one or even a sub-selection of one training image can be sufficient in some cases. We detail the machine learning method and give three use cases where Bellybutton correctly segments images despite substantial lighting, shape, size, focus, and/or structure variation across the regions(s) of interest. Instructions for easy download and use, with further details and the datasets used in this paper are available at pypi.org/project/Bellybuttonseg .

Automatic classification of normal and abnormal cell division using deep learning

In recent years, there has been a surge in the development of methods for cell segmentation and tracking, with initiatives like the Cell Tracking Challenge driving progress in the field. Most studies focus on regular cell population videos in which cells are segmented and followed, and parental relationships annotated. However, DNA damage induced by genotoxic drugs or ionizing radiation produces additional abnormal events since it leads to behaviors like abnormal cell divisions (resulting in a number of daughters different from two) and cell death. With this in mind, we developed an automatic mitosis classifier to categorize small mitosis image sequences centered around one cell as "Normal" or "Abnormal." These mitosis sequences were extracted from videos of cell populations exposed to varying levels of radiation that affect the cell cycle's development. We explored several deep-learning architectures and found that a network with a ResNet50 backbone and including a Long Short-Term Memory (LSTM) layer produced the best results (mean F1-score: 0.93 ± 0.06). In the future, we plan to integrate this classifier with cell segmentation and tracking to build phylogenetic trees of the population after genomic stress.

Optical microscopic imaging, manipulation, and analysis methods for morphogenesis research

Morphogenesis is a developmental process of organisms being shaped through complex and cooperative cellular movements. To understand the interplay between genetic programs and the resulting multicellular morphogenesis, it is essential to characterize the morphologies and dynamics at the single-cell level and to understand how physical forces serve as both signaling components and driving forces of tissue deformations. In recent years, advances in microscopy techniques have led to improvements in imaging speed, resolution and depth. Concurrently, the development of various software packages has supported large-scale, analyses of challenging images at the single-cell resolution. While these tools have enhanced our ability to examine dynamics of cells and mechanical processes during morphogenesis, their effective integration requires specialized expertise. With this background, this review provides a practical overview of those techniques. First, we introduce microscopic techniques for multicellular imaging and image analysis software tools with a focus on cell segmentation and tracking. Second, we provide an overview of cutting-edge techniques for mechanical manipulation of cells and tissues. Finally, we introduce recent findings on morphogenetic mechanisms and mechanosensations that have been achieved by effectively combining microscopy, image analysis tools and mechanical manipulation techniques.

Live-cell imaging powered by computation

The proliferation of microscopy methods for live-cell imaging offers many new possibilities for users but can also be challenging to navigate. The prevailing challenge in live-cell fluorescence microscopy is capturing intra-cellular dynamics while preserving cell viability. Computational methods can help to address this challenge and are now shifting the boundaries of what is possible to capture in living systems. In this Review, we discuss these computational methods focusing on artificial intelligence-based approaches that can be layered on top of commonly used existing microscopies as well as hybrid methods that integrate computation and microscope hardware. We specifically discuss how computational approaches can improve the signal-to-noise ratio, spatial resolution, temporal resolution and multi-colour capacity of live-cell imaging.

The multimodality cell segmentation challenge: toward universal solutions

Cell segmentation is a critical step for quantitative single-cell analysis in microscopy images. Existing cell segmentation methods are often tailored to specific modalities or require manual interventions to specify hyper-parameters in different experimental settings. Here, we present a multimodality cell segmentation benchmark, comprising more than 1,500 labeled images derived from more than 50 diverse biological experiments. The top participants developed a Transformer-based deep-learning algorithm that not only exceeds existing methods but can also be applied to diverse microscopy images across imaging platforms and tissue types without manual parameter adjustments. This benchmark and the improved algorithm offer promising avenues for more accurate and versatile cell analysis in microscopy imaging.

HFM-Tracker: a cell tracking algorithm based on hybrid feature matching

Cell migration is known to be a fundamental biological process, playing an essential role in development, homeostasis, and diseases. This paper introduces a cell tracking algorithm named HFM-Tracker (Hybrid Feature Matching Tracker) that automatically identifies cell migration behaviours in consecutive images. It combines Contour Attention (CA) and Adaptive Confusion Matrix (ACM) modules to accurately capture cell contours in each image and track the dynamic behaviors of migrating cells in the field of view. Cells are firstly located and identified via the CA module-based cell detection network, and then associated and tracked via a cell tracking algorithm employing a hybrid feature-matching strategy. This proposed HFM-Tracker exhibits superiorities in cell detection and tracking, achieving 75% in MOTA (Multiple Object Tracking Accuracy) and 65% in IDF1 (ID F1 score). It provides quantitative analysis of the cell morphology and migration features, which could further help in understanding the complicated and diverse cell migration processes.

Single-cell tracking as a tool for studying EMT-phenotypes

This article demonstrates that label-free single-cell video tracking is a useful approach for in vitro studies of Epithelial-Mesenchymal Transition (EMT). EMT is a highly heterogeneous process, involved in wound healing, embryogenesis and cancer. The process promotes metastasis, and increased understanding can aid development of novel therapeutic strategies. The role of EMT-associated biomarkers depends on biological context, making it challenging to compare and interpret data from different studies. We demonstrate single-cell video tracking for comprehensive phenotype analysis. In this study we performed single-cell video tracking on 72-h long recordings. We quantified several behaviours at a single-cell level during induced EMT in MDA-MB-468 cells. This revealed notable variations in migration speed, with different dose-response patterns and varying distributions of speed. By registering cell morphologies during the recording, we determined preferred paths of morphological transitions. We also found a clear association between migration speed and cell morphology. We found elevated rates of cell death, diminished proliferation, and an increase in mitotic failures followed by re-fusion of sister-cells. The method allows tracking of phenotypes in cell lineages, which can be particularly useful in epigenetic studies. Sister-cells were found to have significant similarities in their speeds and morphologies, illustrating the heritability of these traits.

Automatic detection of cell-cycle stages using recurrent neural networks

Mitosis is the process by which eukaryotic cells divide to produce two similar daughter cells with identical genetic material. Research into the process of mitosis is therefore of critical importance both for the basic understanding of cell biology and for the clinical approach to manifold pathologies resulting from its malfunctioning, including cancer. In this paper, we propose an approach to study mitotic progression automatically using deep learning. We used neural networks to predict different mitosis stages. We extracted video sequences of cells undergoing division and trained a Recurrent Neural Network (RNN) to extract image features. The use of RNN enabled better extraction of features. The RNN-based approach gave better performance compared to classifier based feature extraction methods which do not use time information. Evaluation of precision, recall, and F-score indicates the superiority of the proposed model compared to the baseline. To study the loss in performance due to confusion between adjacent classes, we plotted the confusion matrix as well. In addition, we visualized the feature space to understand why RNNs are better at classifying the mitosis stages than other classifier models, which indicated the formation of strong clusters for the different classes, clearly confirming the advantage of the proposed RNN-based approach.

Software Tools for 2D Cell Segmentation

Cell segmentation is an important task in the field of image processing, widely used in the life sciences and medical fields. Traditional methods are mainly based on pixel intensity and spatial relationships, but have limitations. In recent years, machine learning and deep learning methods have been widely used, providing more-accurate and efficient solutions for cell segmentation. The effort to develop efficient and accurate segmentation software tools has been one of the major focal points in the field of cell segmentation for years. However, each software tool has unique characteristics and adaptations, and no universal cell-segmentation software can achieve perfect results. In this review, we used three publicly available datasets containing multiple 2D cell-imaging modalities. Common segmentation metrics were used to evaluate the performance of eight segmentation tools to compare their generality and, thus, find the best-performing tool.

Denoising diffusion probabilistic models for generation of realistic fully-annotated microscopy image datasets

Recent advances in computer vision have led to significant progress in the generation of realistic image data, with denoising diffusion probabilistic models proving to be a particularly effective method. In this study, we demonstrate that diffusion models can effectively generate fully-annotated microscopy image data sets through an unsupervised and intuitive approach, using rough sketches of desired structures as the starting point. The proposed pipeline helps to reduce the reliance on manual annotations when training deep learning-based segmentation approaches and enables the segmentation of diverse datasets without the need for human annotations. We demonstrate that segmentation models trained with a small set of synthetic image data reach accuracy levels comparable to those of generalist models trained with a large and diverse collection of manually annotated image data, thereby offering a streamlined and specialized application of segmentation models.

Content-aware frame interpolation (CAFI): deep learning-based temporal super-resolution for fast bioimaging

The development of high-resolution microscopes has made it possible to investigate cellular processes in 3D and over time. However, observing fast cellular dynamics remains challenging because of photobleaching and phototoxicity. Here we report the implementation of two content-aware frame interpolation (CAFI) deep learning networks, Zooming SlowMo and Depth-Aware Video Frame Interpolation, that are highly suited for accurately predicting images in between image pairs, therefore improving the temporal resolution of image series post-acquisition. We show that CAFI is capable of understanding the motion context of biological structures and can perform better than standard interpolation methods. We benchmark CAFI's performance on 12 different datasets, obtained from four different microscopy modalities, and demonstrate its capabilities for single-particle tracking and nuclear segmentation. CAFI potentially allows for reduced light exposure and phototoxicity on the sample for improved long-term live-cell imaging. The models and the training and testing data are available via the ZeroCostDL4Mic platform.

Metrics reloaded: recommendations for image analysis validation

Increasing evidence shows that flaws in machine learning (ML) algorithm validation are an underestimated global problem. In biomedical image analysis, chosen performance metrics often do not reflect the domain interest, and thus fail to adequately measure scientific progress and hinder translation of ML techniques into practice. To overcome this, we created Metrics Reloaded, a comprehensive framework guiding researchers in the problem-aware selection of metrics. Developed by a large international consortium in a multistage Delphi process, it is based on the novel concept of a problem fingerprint-a structured representation of the given problem that captures all aspects that are relevant for metric selection, from the domain interest to the properties of the target structure(s), dataset and algorithm output. On the basis of the problem fingerprint, users are guided through the process of choosing and applying appropriate validation metrics while being made aware of potential pitfalls. Metrics Reloaded targets image analysis problems that can be interpreted as classification tasks at image, object or pixel level, namely image-level classification, object detection, semantic segmentation and instance segmentation tasks. To improve the user experience, we implemented the framework in the Metrics Reloaded online tool. Following the convergence of ML methodology across application domains, Metrics Reloaded fosters the convergence of validation methodology. Its applicability is demonstrated for various biomedical use cases.

CellBoost: A pipeline for machine assisted annotation in Neuroanatomy

One of the important yet labor intensive tasks in neuroanatomy is the identification of select populations of cells. Current high-throughput techniques enable marking cells with histochemical fluorescent molecules as well as through the genetic expression of fluorescent proteins. Modern scanning microscopes allow high resolution multi-channel imaging of the mechanically or optically sectioned brain with thousands of marked cells per square millimeter. Manual identification of all marked cells is prohibitively time consuming. At the same time, simple segmentation algorithms suffer from high error rates and sensitivity to variation in fluorescent intensity and spatial distribution. We present a methodology that combines human judgement and machine learning that serves to significantly reduce the labor of the anatomist while improving the consistency of the annotation. As a demonstration, we analyzed murine brains with marked premotor neurons in the brainstem. We compared the error rate of our method to the disagreement rate among human anatomists. This comparison shows that our method can reduce the time to annotate by as much as ten-fold without significantly increasing the rate of errors. We show that our method achieves significant reduction in labor while achieving an accuracy that is similar to the level of agreement between different anatomists.

Deep Learning-Based Cell Tracking in Deforming Organs and Moving Animals

Cell tracking is an essential step in extracting cellular signals from moving cells, which is vital for understanding the mechanisms underlying various biological functions and processes, particularly in organs such as the brain and heart. However, cells in living organisms often exhibit extensive and complex movements caused by organ deformation and whole-body motion. These movements pose a challenge in obtaining high-quality time-lapse cell images and tracking the intricate cell movements in the captured images. Recent advances in deep learning techniques provide powerful tools for detecting cells in low-quality images with densely packed cell populations, as well as estimating cell positions for cells undergoing large nonrigid movements. This chapter introduces the challenges of cell tracking in deforming organs and moving animals, outlines the solutions to these challenges, and presents a detailed protocol for data preparation, as well as for performing cell segmentation and tracking using the latest version of 3DeeCellTracker. This protocol is expected to enable researchers to gain deeper insights into organ dynamics and biological processes.

Advances in medical image analysis with vision Transformers: A comprehensive review

The remarkable performance of the Transformer architecture in natural language processing has recently also triggered broad interest in Computer Vision. Among other merits, Transformers are witnessed as capable of learning long-range dependencies and spatial correlations, which is a clear advantage over convolutional neural networks (CNNs), which have been the de facto standard in Computer Vision problems so far. Thus, Transformers have become an integral part of modern medical image analysis. In this review, we provide an encyclopedic review of the applications of Transformers in medical imaging. Specifically, we present a systematic and thorough review of relevant recent Transformer literature for different medical image analysis tasks, including classification, segmentation, detection, registration, synthesis, and clinical report generation. For each of these applications, we investigate the novelty, strengths and weaknesses of the different proposed strategies and develop taxonomies highlighting key properties and contributions. Further, if applicable, we outline current benchmarks on different datasets. Finally, we summarize key challenges and discuss different future research directions. In addition, we have provided cited papers with their corresponding implementations in https://github.com/mindflow-institue/Awesome-Transformer.

Methods and Measures for Investigating Microscale Motility

Motility is an essential factor for an organism's survival and diversification. With the advent of novel single-cell technologies, analytical frameworks, and theoretical methods, we can begin to probe the complex lives of microscopic motile organisms and answer the intertwining biological and physical questions of how these diverse lifeforms navigate their surroundings. Herein, we summarize the main mechanisms of microscale motility and give an overview of different experimental, analytical, and mathematical methods used to study them across different scales encompassing the molecular-, individual-, to population-level. We identify transferable techniques, pressing challenges, and future directions in the field. This review can serve as a starting point for researchers who are interested in exploring and quantifying the movements of organisms in the microscale world.

Live-cell imaging in the deep learning era

Live imaging is a powerful tool, enabling scientists to observe living organisms in real time. In particular, when combined with fluorescence microscopy, live imaging allows the monitoring of cellular components with high sensitivity and specificity. Yet, due to critical challenges (i.e., drift, phototoxicity, dataset size), implementing live imaging and analyzing the resulting datasets is rarely straightforward. Over the past years, the development of bioimage analysis tools, including deep learning, is changing how we perform live imaging. Here we briefly cover important computational methods aiding live imaging and carrying out key tasks such as drift correction, denoising, super-resolution imaging, artificial labeling, tracking, and time series analysis. We also cover recent advances in self-driving microscopy.

A Weakly Supervised Learning Method for Cell Detection and Tracking Using Incomplete Initial Annotations

The automatic detection of cells in microscopy image sequences is a significant task in biomedical research. However, routine microscopy images with cells, which are taken during the process whereby constant division and differentiation occur, are notoriously difficult to detect due to changes in their appearance and number. Recently, convolutional neural network (CNN)-based methods have made significant progress in cell detection and tracking. However, these approaches require many manually annotated data for fully supervised training, which is time-consuming and often requires professional researchers. To alleviate such tiresome and labor-intensive costs, we propose a novel weakly supervised learning cell detection and tracking framework that trains the deep neural network using incomplete initial labels. Our approach uses incomplete cell markers obtained from fluorescent images for initial training on the Induced Pluripotent Stem (iPS) cell dataset, which is rarely studied for cell detection and tracking. During training, the incomplete initial labels were updated iteratively by combining detection and tracking results to obtain a model with better robustness. Our method was evaluated using two fields of the iPS cell dataset, along with the cell detection accuracy (DET) evaluation metric from the Cell Tracking Challenge (CTC) initiative, and it achieved 0.862 and 0.924 DET, respectively. The transferability of the developed model was tested using the public dataset FluoN2DH-GOWT1, which was taken from CTC; this contains two datasets with reference annotations. We randomly removed parts of the annotations in each labeled data to simulate the initial annotations on the public dataset. After training the model on the two datasets, with labels that comprise 10% cell markers, the DET improved from 0.130 to 0.903 and 0.116 to 0.877. When trained with labels that comprise 60% cell markers, the performance was better than the model trained using the supervised learning method. This outcome indicates that the model's performance improved as the quality of the labels used for training increased.

Polarization differential interference contrast microscopy with physics-inspired plug-and-play denoiser for single-shot high-performance quantitative phase imaging

We present single-shot high-performance quantitative phase imaging with a physics-inspired plug-and-play denoiser for polarization differential interference contrast (PDIC) microscopy. The quantitative phase is recovered by the alternating direction method of multipliers (ADMM), balancing total variance regularization and a pre-trained dense residual U-net (DRUNet) denoiser. The custom DRUNet uses the Tanh activation function to guarantee the symmetry requirement for phase retrieval. In addition, we introduce an adaptive strategy accelerating convergence and explicitly incorporating measurement noise. After validating this deep denoiser-enhanced PDIC microscopy on simulated data and phantom experiments, we demonstrated high-performance phase imaging of histological tissue sections. The phase retrieval by the denoiser-enhanced PDIC microscopy achieves significantly higher quality and accuracy than the solution based on Fourier transforms or the iterative solution with total variance regularization alone.

Bayesian inference on the Allee effect in cancer cell line populations using time-lapse microscopy images

The Allee effect describes the phenomenon that the per capita reproduction rate increases along with the population density at low densities. Allee effects have been observed at all scales, including in microscopic environments where individual cells are taken into account. This is great interest to cancer research, as understanding critical tumour density thresholds can inform treatment plans for patients. In this paper, we introduce a simple model for cell division in the case where the cancer cell population is modelled as an interacting particle system. The rate of the cell division is dependent on the local cell density, introducing an Allee effect. We perform parameter inference of the key model parameters through Markov Chain Monte Carlo, and apply our procedure to two image sequences from a cervical cancer cell line. The inference method is verified on in silico data to accurately identify the key parameters, and results on the in vitro data strongly suggest an Allee effect.

ALFI: Cell cycle phenotype annotations of label-free time-lapse imaging data from cultured human cells

Detecting and tracking multiple moving objects in a video is a challenging task. For living cells, the task becomes even more arduous as cells change their morphology over time, can partially overlap, and mitosis leads to new cells. Differently from fluorescence microscopy, label-free techniques can be easily applied to almost all cell lines, reducing sample preparation complexity and phototoxicity. In this study, we present ALFI, a dataset of images and annotations for label-free microscopy, made publicly available to the scientific community, that notably extends the current panorama of expertly labeled data for detection and tracking of cultured living nontransformed and cancer human cells. It consists of 29 time-lapse image sequences from HeLa, U2OS, and hTERT RPE-1 cells under different experimental conditions, acquired by differential interference contrast microscopy, for a total of 237.9 hours. It contains various annotations (pixel-wise segmentation masks, object-wise bounding boxes, tracking information). The dataset is useful for testing and comparing methods for identifying interphase and mitotic events and reconstructing their lineage, and for discriminating different cellular phenotypes.

Addressing annotation and data scarcity when designing machine learning strategies for neurophotonics

Machine learning has revolutionized the way data are processed, allowing information to be extracted in a fraction of the time it would take an expert. In the field of neurophotonics, machine learning approaches are used to automatically detect and classify features of interest in complex images. One of the key challenges in applying machine learning methods to the field of neurophotonics is the scarcity of available data and the complexity associated with labeling them, which can limit the performance of data-driven algorithms. We present an overview of various strategies, such as weakly supervised learning, active learning, and domain adaptation that can be used to address the problem of labeled data scarcity in neurophotonics. We provide a comprehensive overview of the strengths and limitations of each approach and discuss their potential applications to bioimaging datasets. In addition, we highlight how different strategies can be combined to increase model performance on those datasets. The approaches we describe can help to improve the accessibility of machine learning-based analysis with limited number of annotated images for training and can enable researchers to extract more meaningful insights from small datasets.

Machine learning for cross-scale microscopy of viruses

Despite advances in virological sciences and antiviral research, viruses continue to emerge, circulate, and threaten public health. We still lack a comprehensive understanding of how cells and individuals remain susceptible to infectious agents. This deficiency is in part due to the complexity of viruses, including the cell states controlling virus-host interactions. Microscopy samples distinct cellular infection stages in a multi-parametric, time-resolved manner at molecular resolution and is increasingly enhanced by machine learning and deep learning. Here we discuss how state-of-the-art artificial intelligence (AI) augments light and electron microscopy and advances virological research of cells. We describe current procedures for image denoising, object segmentation, tracking, classification, and super-resolution and showcase examples of how AI has improved the acquisition and analyses of microscopy data. The power of AI-enhanced microscopy will continue to help unravel virus infection mechanisms, develop antiviral agents, and improve viral vectors.

Critical Assessment of Condensate Boundaries in Dual-Color Single Particle Tracking

Biomolecular condensates are membraneless cellular compartments generated by phase separation that regulate a broad variety of cellular functions by enriching some biomolecules while excluding others. Live-cell single particle tracking of individual fluorophore-labeled condensate components has provided insights into a condensate's mesoscopic organization and biological functions, such as revealing the recruitment, translation, and decay of RNAs within ribonucleoprotein (RNP) granules. Specifically, during dual-color tracking, one imaging channel provides a time series of individual biomolecule locations, while the other channel monitors the location of the condensate relative to these molecules. Therefore, an accurate assessment of a condensate's boundary is critical for combined live-cell single particle-condensate tracking. Despite its importance, a quantitative benchmarking and objective comparison of the various available boundary detection methods is missing due to the lack of an absolute ground truth for condensate images. Here, we use synthetic data of defined ground truth to generate noise-overlaid images of condensates with realistic phase separation parameters to benchmark the most commonly used methods for condensate boundary detection, including an emerging machine-learning method. We find that it is critical to carefully choose an optimal boundary detection method for a given dataset to obtain accurate measurements of single particle-condensate interactions. The criteria proposed in this study to guide the selection of an optimal boundary detection method can be broadly applied to imaging-based studies of condensates.