Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy

Advancements and Practical Considerations for Biophysical Research: Navigating the Challenges and Future of Super-resolution Microscopy

The introduction of super-resolution microscopy (SRM) has significantly advanced our understanding of cellular and molecular dynamics, offering a detailed view previously beyond our reach. Implementing SRM in biophysical research, however, presents numerous challenges. This review addresses the crucial aspects of utilizing SRM effectively, from selecting appropriate fluorophores and preparing samples to analyzing complex data sets. We explore recent technological advancements and methodological improvements that enhance the capabilities of SRM. Emphasizing the integration of SRM with other analytical methods, we aim to overcome inherent limitations and expand the scope of biological insights achievable. By providing a comprehensive guide for choosing the most suitable SRM methods based on specific research objectives, we aim to empower researchers to explore complex biological processes with enhanced precision and clarity, thereby advancing the frontiers of biophysical research.

Accessible high-speed image-activated cell sorting

Over the past six decades, fluorescence-activated cell sorting (FACS) has become an essential technology for basic and clinical research by enabling the isolation of cells of interest in high throughput. Recent technological advancements have started a new era of flow cytometry. By combining the spatial resolution of microscopy with high-speed cell sorting, new instruments allow cell sorting based on simple image-derived parameters or sophisticated image analysis algorithms, thereby greatly expanding the scope of applications. In this review, we discuss the systems that are commercially available or have been described in enough methodological and engineering detail to allow their replication. We summarize their strengths and limitations and highlight applications that have the potential to transform various fields in basic life science research and clinical settings.

A deep learning-based approach for efficient detection and classification of local Ca²⁺ release events in Full-Frame confocal imaging

The release of Ca2+ ions from intracellular stores plays a crucial role in many cellular processes, acting as a secondary messenger in various cell types, including cardiomyocytes, smooth muscle cells, hepatocytes, and many others. Detecting and classifying associated local Ca2+ release events is particularly important, as these events provide insight into the mechanisms, interplay, and interdependencies of local Ca2+release events underlying global intracellular Ca2+signaling. However, time-consuming and labor-intensive procedures often complicate analysis, especially with low signal-to-noise ratio imaging data. Here, we present an innovative deep learning-based approach for automatically detecting and classifying local Ca2+ release events. This approach is exemplified with rapid full-frame confocal imaging data recorded in isolated cardiomyocytes. To demonstrate the robustness and accuracy of our method, we first use conventional evaluation methods by comparing the intersection between manual annotations and the segmentation of Ca2+ release events provided by the deep learning method, as well as the annotated and recognized instances of individual events. In addition to these methods, we compare the performance of the proposed model with the annotation of six experts in the field. Our model can recognize more than 75 % of the annotated Ca2+ release events and correctly classify more than 75 %. A key result was that there were no significant differences between the annotations produced by human experts and the result of the proposed deep learning model. We conclude that the proposed approach is a robust and time-saving alternative to conventional full-frame confocal imaging analysis of local intracellular Ca2+ events.

Label-Free Techniques for Probing Biomolecular Condensates

Biomolecular condensates play important roles in a wide array of fundamental biological processes, such as cellular compartmentalization, cellular regulation, and other biochemical reactions. Since their discovery and first observations, an extensive and expansive library of tools has been developed to investigate various aspects and properties, encompassing structural and compositional information, material properties, and their evolution throughout the life cycle from formation to eventual dissolution. This Review presents an overview of the expanded set of tools and methods that researchers use to probe the properties of biomolecular condensates across diverse scales of length, concentration, stiffness, and time. In particular, we review recent years' exciting development of label-free techniques and methodologies. We broadly organize the set of tools into 3 categories: (1) imaging-based techniques, such as transmitted-light microscopy (TLM) and Brillouin microscopy (BM), (2) force spectroscopy techniques, such as atomic force microscopy (AFM) and the optical tweezer (OT), and (3) microfluidic platforms and emerging technologies. We point out the tools' key opportunities, challenges, and future perspectives and analyze their correlative potential as well as compatibility with other techniques. Additionally, we review emerging techniques, namely, differential dynamic microscopy (DDM) and interferometric scattering microscopy (iSCAT), that have huge potential for future applications in studying biomolecular condensates. Finally, we highlight how some of these techniques can be translated for diagnostics and therapy purposes. We hope this Review serves as a useful guide for new researchers in this field and aids in advancing the development of new biophysical tools to study biomolecular condensates.

Learning dynamical models of single and collective cell migration: a review

Single and collective cell migration are fundamental processes critical for physiological phenomena ranging from embryonic development and immune response to wound healing and cancer metastasis. To understand cell migration from a physical perspective, a broad variety of models for the underlying physical mechanisms that govern cell motility have been developed. A key challenge in the development of such models is how to connect them to experimental observations, which often exhibit complex stochastic behaviours. In this review, we discuss recent advances in data-driven theoretical approaches that directly connect with experimental data to infer dynamical models of stochastic cell migration. Leveraging advances in nanofabrication, image analysis, and tracking technology, experimental studies now provide unprecedented large datasets on cellular dynamics. In parallel, theoretical efforts have been directed towards integrating such datasets into physical models from the single cell to the tissue scale with the aim of conceptualising the emergent behaviour of cells. We first review how this inference problem has been addressed in both freely migrating and confined cells. Next, we discuss why these dynamics typically take the form of underdamped stochastic equations of motion, and how such equations can be inferred from data. We then review applications of data-driven inference and machine learning approaches to heterogeneity in cell behaviour, subcellular degrees of freedom, and to the collective dynamics of multicellular systems. Across these applications, we emphasise how data-driven methods can be integrated with physical active matter models of migrating cells, and help reveal how underlying molecular mechanisms control cell behaviour. Together, these data-driven approaches are a promising avenue for building physical models of cell migration directly from experimental data, and for providing conceptual links between different length-scales of description.

Efficient Supervised Pretraining of Swin-Transformer for Virtual Staining of Microscopy Images

Fluorescence staining is an important technique in life science for labeling cellular constituents. However, it also suffers from being time-consuming, having difficulty in simultaneous labeling, etc. Thus, virtual staining, which does not rely on chemical labeling, has been introduced. Recently, deep learning models such as transformers have been applied to virtual staining tasks. However, their performance relies on large-scale pretraining, hindering their development in the field. To reduce the reliance on large amounts of computation and data, we construct a Swin-transformer model and propose an efficient supervised pretraining method based on the masked autoencoder (MAE). Specifically, we adopt downsampling and grid sampling to mask 75% of pixels and reduce the number of tokens. The pretraining time of our method is only 1/16 compared with the original MAE. We also design a supervised proxy task to predict stained images with multiple styles instead of masked pixels. Additionally, most virtual staining approaches are based on private datasets and evaluated by different metrics, making a fair comparison difficult. Therefore, we develop a standard benchmark based on three public datasets and build a baseline for the convenience of future researchers. We conduct extensive experiments on three benchmark datasets, and the experimental results show the proposed method achieves the best performance both quantitatively and qualitatively. In addition, ablation studies are conducted, and experimental results illustrate the effectiveness of the proposed pretraining method. The benchmark and code are available at https://github.com/birkhoffkiki/CAS-Transformer.

Nondestructive, quantitative viability analysis of 3D tissue cultures using machine learning image segmentation

Ascertaining the collective viability of cells in different cell culture conditions has typically relied on averaging colorimetric indicators and is often reported out in simple binary readouts. Recent research has combined viability assessment techniques with image-based deep-learning models to automate the characterization of cellular properties. However, further development of viability measurements to assess the continuity of possible cellular states and responses to perturbation across cell culture conditions is needed. In this work, we demonstrate an image processing algorithm for quantifying features associated with cellular viability in 3D cultures without the need for assay-based indicators. We show that our algorithm performs similarly to a pair of human experts in whole-well images over a range of days and culture matrix compositions. To demonstrate potential utility, we perform a longitudinal study investigating the impact of a known therapeutic on pancreatic cancer spheroids. Using images taken with a high content imaging system, the algorithm successfully tracks viability at the individual spheroid and whole-well level. The method we propose reduces analysis time by 97% in comparison with the experts. Because the method is independent of the microscope or imaging system used, this approach lays the foundation for accelerating progress in and for improving the robustness and reproducibility of 3D culture analysis across biological and clinical research.

Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for the reliable tracking of scientific progress and for bridging the current chasm between artificial intelligence (AI) research and its translation into practice. However, increasing evidence shows that particularly in image analysis, metrics are often chosen inadequately in relation to the underlying research problem. This could be attributed to a lack of accessibility of metric-related knowledge: While taking into account the individual strengths, weaknesses, and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multi-stage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides the first reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Focusing on biomedical image analysis but with the potential of transfer to other fields, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. To facilitate comprehension, illustrations and specific examples accompany each pitfall. As a structured body of information accessible to researchers of all levels of expertise, this work enhances global comprehension of a key topic in image analysis validation.

Reversible, tunable epigenetic silencing of TCF1 generates flexibility in the T cell memory decision

The immune system encodes information about the severity of a pathogenic threat in the quantity and type of memory cells it forms. This encoding emerges from lymphocyte decisions to maintain or lose self-renewal and memory potential during a challenge. By tracking CD8+ T cells at the single-cell and clonal lineage level using time-resolved transcriptomics, quantitative live imaging, and an acute infection model, we find that T cells will maintain or lose memory potential early after antigen recognition. However, following pathogen clearance, T cells may regain memory potential if initially lost. Mechanistically, this flexibility is implemented by a stochastic cis-epigenetic switch that tunably and reversibly silences the memory regulator, TCF1, in response to stimulation. Mathematical modeling shows how this flexibility allows memory T cell numbers to scale robustly with pathogen virulence and immune response magnitudes. We propose that flexibility and stochasticity in cellular decisions ensure optimal immune responses against diverse threats.

Deep learning assisted single particle tracking for automated correlation between diffusion and function

Sub-cellular diffusion in living systems reflects cellular processes and interactions. Recent advances in optical microscopy allow the tracking of this nanoscale diffusion of individual objects with an unprecedented level of precision. However, the agnostic and automated extraction of functional information from the diffusion of molecules and organelles within the sub-cellular environment, is labor-intensive and poses a significant challenge. Here we introduce DeepSPT, a deep learning framework to interpret the diffusional 2D or 3D temporal behavior of objects in a rapid and efficient manner, agnostically. Demonstrating its versatility, we have applied DeepSPT to automated mapping of the early events of viral infections, identifying distinct types of endosomal organelles, and clathrin-coated pits and vesicles with up to 95% accuracy and within seconds instead of weeks. The fact that DeepSPT effectively extracts biological information from diffusion alone illustrates that besides structure, motion encodes function at the molecular and subcellular level.

Harnessing artificial intelligence to reduce phototoxicity in live imaging

Fluorescence microscopy is essential for studying living cells, tissues and organisms. However, the fluorescent light that switches on fluorescent molecules also harms the samples, jeopardizing the validity of results - particularly in techniques such as super-resolution microscopy, which demands extended illumination. Artificial intelligence (AI)-enabled software capable of denoising, image restoration, temporal interpolation or cross-modal style transfer has great potential to rescue live imaging data and limit photodamage. Yet we believe the focus should be on maintaining light-induced damage at levels that preserve natural cell behaviour. In this Opinion piece, we argue that a shift in role for AIs is needed - AI should be used to extract rich insights from gentle imaging rather than recover compromised data from harsh illumination. Although AI can enhance imaging, our ultimate goal should be to uncover biological truths, not just retrieve data. It is essential to prioritize minimizing photodamage over merely pushing technical limits. Our approach is aimed towards gentle acquisition and observation of undisturbed living systems, aligning with the essence of live-cell fluorescence microscopy.

Understanding metric-related pitfalls in image analysis validation

Validation metrics are key for tracking scientific progress and bridging the current chasm between artificial intelligence research and its translation into practice. However, increasing evidence shows that, particularly in image analysis, metrics are often chosen inadequately. Although taking into account the individual strengths, weaknesses and limitations of validation metrics is a critical prerequisite to making educated choices, the relevant knowledge is currently scattered and poorly accessible to individual researchers. Based on a multistage Delphi process conducted by a multidisciplinary expert consortium as well as extensive community feedback, the present work provides a reliable and comprehensive common point of access to information on pitfalls related to validation metrics in image analysis. Although focused on biomedical image analysis, the addressed pitfalls generalize across application domains and are categorized according to a newly created, domain-agnostic taxonomy. The work serves to enhance global comprehension of a key topic in image analysis validation.

Automated human induced pluripotent stem cell culture and sample preparation for 3D live-cell microscopy

To produce abundant cell culture samples to generate large, standardized image datasets of human induced pluripotent stem (hiPS) cells, we developed an automated workflow on a Hamilton STAR liquid handler system. This was developed specifically for culturing hiPS cell lines expressing fluorescently tagged proteins, which we have used to study the principles by which cells establish and maintain robust dynamic localization of cellular structures. This protocol includes all details for the maintenance, passage and seeding of cells, as well as Matrigel coating of 6-well plastic plates and 96-well optical-grade, glass plates. We also developed an automated image-based hiPS cell colony segmentation and feature extraction pipeline to streamline the process of predicting cell count and selecting wells with consistent morphology for high-resolution three-dimensional (3D) microscopy. The imaging samples produced with this protocol have been used to study the integrated intracellular organization and cell-to-cell variability of hiPS cells to train and develop deep learning-based label-free predictions from transmitted-light microscopy images and to develop deep learning-based generative models of single-cell organization. This protocol requires some experience with robotic equipment. However, we provide details and source code to facilitate implementation by biologists less experienced with robotics. The protocol is completed in less than 10 h with minimal human interaction. Overall, automation of our cell culture procedures increased our imaging samples' standardization, reproducibility, scalability and consistency. It also reduced the need for stringent culturist training and eliminated culturist-to-culturist variability, both of which were previous pain points of our original manual pipeline workflow.

Expanding the coverage of spatial proteomics: a machine learning approach

MOTIVATION

Multiplexed protein imaging methods use a chosen set of markers and provide valuable information about complex tissue structure and cellular heterogeneity. However, the number of markers that can be measured in the same tissue sample is inherently limited.

RESULTS

In this paper, we present an efficient method to choose a minimal predictive subset of markers that for the first time allows the prediction of full images for a much larger set of markers. We demonstrate that our approach also outperforms previous methods for predicting cell-level protein composition. Most importantly, we demonstrate that our approach can be used to select a marker set that enables prediction of a much larger set than could be measured concurrently.

AVAILABILITY AND IMPLEMENTATION

All code and intermediate results are available in a Reproducible Research Archive at https://github.com/murphygroup/CODEXPanelOptimization.

Fluorescence microscopy and correlative brightfield videos of mitochondria and vesicles in H9c2 cardiomyoblasts

This paper presents data acquired to study the dynamics and interactions of mitochondria and subcellular vesicles in living cardiomyoblasts. The study was motivated by the importance of mitochondrial quality control and turnover in cardiovascular health. Although fluorescence microscopy is an invaluable tool, it presents several limitations. Correlative fluorescence and brightfield images (label-free) were therefore acquired with the purpose of achieving virtual labelling via machine learning. In comparison with the fluorescence images of mitochondria, the brightfield images show vesicles and subcellular components, providing additional insights about sub-cellular components. A large part of the data contains correlative fluorescence images of lysosomes and/or endosomes over a duration of up to 400 timepoints (>30 min). The data can be reused for biological inferences about mitochondrial and vesicular morphology, dynamics, and interactions. Furthermore, virtual labelling of mitochondria or subcellular vesicles can be achieved using these datasets. Finally, the data can inspire new imaging experiments for cellular investigations or computational developments. The data is available through two large, open datasets on DataverseNO.

Automated Cell Lineage Reconstruction using Label-Free 4D Microscopy

Here we describe embGAN, a deep learning pipeline that addresses the challenge of automated cell detection and tracking in label-free 3D time lapse imaging. embGAN requires no manual data annotation for training, learns robust detections that exhibits a high degree of scale invariance and generalizes well to images acquired in multiple labs on multiple instruments.

Single-cell morphodynamical trajectories enable prediction of gene expression accompanying cell state change

Extracellular signals induce changes to molecular programs that modulate multiple cellular phenotypes, including proliferation, motility, and differentiation status. The connection between dynamically adapting phenotypic states and the molecular programs that define them is not well understood. Here we develop data-driven models of single-cell phenotypic responses to extracellular stimuli by linking gene transcription levels to "morphodynamics" - changes in cell morphology and motility observable in time-lapse image data. We adopt a dynamics-first view of cell state by grouping single-cell trajectories into states with shared morphodynamic responses. The single-cell trajectories enable development of a first-of-its-kind computational approach to map live-cell dynamics to snapshot gene transcript levels, which we term MMIST, Molecular and Morphodynamics-Integrated Single-cell Trajectories. The key conceptual advance of MMIST is that cell behavior can be quantified based on dynamically defined states and that extracellular signals alter the overall distribution of cell states by altering rates of switching between states. We find a cell state landscape that is bound by epithelial and mesenchymal endpoints, with distinct sequences of epithelial to mesenchymal transition (EMT) and mesenchymal to epithelial transition (MET) intermediates. The analysis yields predictions for gene expression changes consistent with curated EMT gene sets and provides a prediction of thousands of RNA transcripts through extracellular signal-induced EMT and MET with near-continuous time resolution. The MMIST framework leverages true single-cell dynamical behavior to generate molecular-level omics inferences and is broadly applicable to other biological domains, time-lapse imaging approaches and molecular snapshot data.

Regulated Behavior in Living Cells with Highly Aligned Configurations on Nanowrinkled Graphene Oxide Substrates: Deep Learning Based on Interplay of Cellular Contact Guidance

Micro-/nanotopographical cues have emerged as a practical and promising strategy for controlling cell fate and reprogramming, which play a key role as biophysical regulators in diverse cellular processes and behaviors. Extracellular biophysical factors can trigger intracellular physiological signaling via mechanotransduction and promote cellular responses such as cell adhesion, migration, proliferation, gene/protein expression, and differentiation. Here, we engineered a highly ordered nanowrinkled graphene oxide (GO) surface via the mechanical deformation of an ultrathin GO film on an elastomeric substrate to observe specific cellular responses based on surface-mediated topographical cues. The ultrathin GO film on the uniaxially prestrained elastomeric substrate through self-assembly and subsequent compressive force produced GO nanowrinkles with periodic amplitude. To examine the acute cellular behaviors on the GO-based cell interface with nanostructured arrays of wrinkles, we cultured L929 fibroblasts and HT22 hippocampal neuronal cells. As a result, our developed cell-culture substrate obviously provided a directional guidance effect. In addition, based on the observed results, we adapted a deep learning (DL)-based data processing technique to precisely interpret the cell behaviors on the nanowrinkled GO surfaces. According to the learning/transfer learning protocol of the DL network, we detected cell boundaries, elongation, and orientation and quantitatively evaluated cell velocity, traveling distance, displacement, and orientation. The presented experimental results have intriguing implications such that the nanotopographical microenvironment could engineer the living cells' morphological polarization to assemble them into useful tissue chips consisting of multiple cell types.

Prediction of single-cell RNA expression profiles in live cells by Raman microscopy with Raman2RNA

Single-cell RNA sequencing and other profiling assays have helped interrogate cells at unprecedented resolution and scale, but are inherently destructive. Raman microscopy reports on the vibrational energy levels of proteins and metabolites in a label-free and nondestructive manner at subcellular spatial resolution, but it lacks genetic and molecular interpretability. Here we present Raman2RNA (R2R), a method to infer single-cell expression profiles in live cells through label-free hyperspectral Raman microscopy images and domain translation. We predict single-cell RNA sequencing profiles nondestructively from Raman images using either anchor-based integration with single molecule fluorescence in situ hybridization, or anchor-free generation with adversarial autoencoders. R2R outperformed inference from brightfield images (cosine similarities: R2R >0.85 and brightfield <0.15). In reprogramming of mouse fibroblasts into induced pluripotent stem cells, R2R inferred the expression profiles of various cell states. With live-cell tracking of mouse embryonic stem cell differentiation, R2R traced the early emergence of lineage divergence and differentiation trajectories, overcoming discontinuities in expression space. R2R lays a foundation for future exploration of live genomic dynamics.

MMV_Im2Im: an open-source microscopy machine vision toolbox for image-to-image transformation

Over the past decade, deep learning (DL) research in computer vision has been growing rapidly, with many advances in DL-based image analysis methods for biomedical problems. In this work, we introduce MMV_Im2Im, a new open-source Python package for image-to-image transformation in bioimaging applications. MMV_Im2Im is designed with a generic image-to-image transformation framework that can be used for a wide range of tasks, including semantic segmentation, instance segmentation, image restoration, image generation, and so on. Our implementation takes advantage of state-of-the-art machine learning engineering techniques, allowing researchers to focus on their research without worrying about engineering details. We demonstrate the effectiveness of MMV_Im2Im on more than 10 different biomedical problems, showcasing its general potentials and applicabilities. For computational biomedical researchers, MMV_Im2Im provides a starting point for developing new biomedical image analysis or machine learning algorithms, where they can either reuse the code in this package or fork and extend this package to facilitate the development of new methods. Experimental biomedical researchers can benefit from this work by gaining a comprehensive view of the image-to-image transformation concept through diversified examples and use cases. We hope this work can give the community inspirations on how DL-based image-to-image transformation can be integrated into the assay development process, enabling new biomedical studies that cannot be done only with traditional experimental assays. To help researchers get started, we have provided source code, documentation, and tutorials for MMV_Im2Im at [https://github.com/MMV-Lab/mmv_im2im] under MIT license.

Organelle-specific phase contrast microscopy (OS-PCM) enables facile correlation study of organelles and proteins

Current methods for studying organelle and protein interactions and correlations depend on multiplex fluorescent labeling, which is experimentally complex and harmful to cells. Here we propose to solve this challenge via OS-PCM, where organelles are imaged and segmented without labels, and combined with standard fluorescence microscopy of protein distributions. In this work, we develop new neural networks to obtain unlabeled organelle, nucleus and membrane predictions from a single 2D image. Automated analysis is also implemented to obtain quantitative information regarding the spatial distribution and co-localization of both protein and organelle, as well as their relationship to the landmark structures of nucleus and membrane. Using mitochondria and DRP1 protein as a proof-of-concept, we conducted a correlation study where only DRP1 is labeled, with results consistent with prior reports utilizing multiplex labeling. Thus our work demonstrates that OS-PCM simplifies the correlation study of organelles and proteins.

Mantis: high-throughput 4D imaging and analysis of the molecular and physical architecture of cells

High-throughput dynamic imaging of cells and organelles is important for parsing complex cellular responses. We report a high-throughput 4D microscope, named Mantis, that combines two complementary, gentle, live-imaging technologies: remote-refocus label-free microscopy and oblique light-sheet fluorescence microscopy. We also report open-source software for automated acquisition, registration, and reconstruction, and virtual staining software for single-cell segmentation and phenotyping. Mantis enabled high-content correlative imaging of molecular components and the physical architecture of 20 cell lines every 15 minutes over 7.5 hours, and also detailed measurements of the impacts of viral infection on the architecture of host cells and host proteins. The Mantis platform can enable high-throughput profiling of intracellular dynamics, long-term imaging and analysis of cellular responses to stress, and live cell optical screens to dissect gene regulatory networks.

3DCNAS: A universal method for predicting the location of fluorescent organelles in living cells in three-dimensional space

Cellular biology research relies on microscopic imaging techniques for studying the complex structures and dynamic processes within cells. Fluorescence microscopy provides high sensitivity and subcellular resolution but has limitations such as photobleaching and sample preparation challenges. Transmission light microscopy offers a label-free alternative but lacks contrast for detailed interpretation. Deep learning methods have shown promise in analyzing cell images and extracting meaningful information. However, accurately learning and simulating diverse subcellular structures remain challenging. In this study, we propose a method named three-dimensional cell neural architecture search (3DCNAS) to predict subcellular structures of fluorescence using unlabeled transmitted light microscope images. By leveraging the automated search capability of differentiable neural architecture search (NAS), our method partially mitigates the issues of overfitting and underfitting caused by the distinct details of various subcellular structures. Furthermore, we apply our method to analyze cell dynamics in genome-edited human induced pluripotent stem cells during mitotic events. This allows us to study the functional roles of organelles and their involvement in cellular processes, contributing to a comprehensive understanding of cell biology and offering insights into disease pathogenesis.

Trustworthy in silico cell labeling via ensemble-based image translation

Artificial intelligence (AI) image translation has been a valuable tool for processing image data in biological and medical research. To apply such a tool in mission-critical applications, including drug screening, toxicity study, and clinical diagnostics, it is essential to ensure that the AI prediction is trustworthy. Here, we demonstrate that an ensemble learning method can quantify the uncertainty of AI image translation. We tested the uncertainty evaluation using experimentally acquired images of mesenchymal stromal cells. We find that the ensemble method reports a prediction standard deviation that correlates with the prediction error, estimating the prediction uncertainty. We show that this uncertainty is in agreement with the prediction error and Pearson correlation coefficient. We further show that the ensemble method can detect out-of-distribution input images by reporting increased uncertainty. Altogether, these results suggest that the ensemble-estimated uncertainty can be a useful indicator for identifying erroneous AI image translations.

CellVisioner: A Generalizable Cell Virtual Staining Toolbox based on Few-Shot Transfer Learning for Mechanobiological Analysis

Visualizing cellular structures especially the cytoskeleton and the nucleus is crucial for understanding mechanobiology, but traditional fluorescence staining has inherent limitations such as phototoxicity and photobleaching. Virtual staining techniques provide an alternative approach to addressing these issues but often require substantial amount of user training data. In this study, we develop a generalizable cell virtual staining toolbox (termed CellVisioner) based on few-shot transfer learning that requires substantially reduced user training data. CellVisioner can virtually stain F-actin and nuclei for various types of cells and extract single-cell parameters relevant to mechanobiology research. Taking the label-free single-cell images as input, CellVisioner can predict cell mechanobiological status (e.g., Yes-associated protein nuclear/cytoplasmic ratio) and perform long-term monitoring for living cells. We envision that CellVisioner would be a powerful tool to facilitate on-site mechanobiological research.

Live-cell imaging in the deep learning era

Live imaging is a powerful tool, enabling scientists to observe living organisms in real time. In particular, when combined with fluorescence microscopy, live imaging allows the monitoring of cellular components with high sensitivity and specificity. Yet, due to critical challenges (i.e., drift, phototoxicity, dataset size), implementing live imaging and analyzing the resulting datasets is rarely straightforward. Over the past years, the development of bioimage analysis tools, including deep learning, is changing how we perform live imaging. Here we briefly cover important computational methods aiding live imaging and carrying out key tasks such as drift correction, denoising, super-resolution imaging, artificial labeling, tracking, and time series analysis. We also cover recent advances in self-driving microscopy.

Large-scale pancreatic cancer detection via non-contrast CT and deep learning

Pancreatic ductal adenocarcinoma (PDAC), the most deadly solid malignancy, is typically detected late and at an inoperable stage. Early or incidental detection is associated with prolonged survival, but screening asymptomatic individuals for PDAC using a single test remains unfeasible due to the low prevalence and potential harms of false positives. Non-contrast computed tomography (CT), routinely performed for clinical indications, offers the potential for large-scale screening, however, identification of PDAC using non-contrast CT has long been considered impossible. Here, we develop a deep learning approach, pancreatic cancer detection with artificial intelligence (PANDA), that can detect and classify pancreatic lesions with high accuracy via non-contrast CT. PANDA is trained on a dataset of 3,208 patients from a single center. PANDA achieves an area under the receiver operating characteristic curve (AUC) of 0.986-0.996 for lesion detection in a multicenter validation involving 6,239 patients across 10 centers, outperforms the mean radiologist performance by 34.1% in sensitivity and 6.3% in specificity for PDAC identification, and achieves a sensitivity of 92.9% and specificity of 99.9% for lesion detection in a real-world multi-scenario validation consisting of 20,530 consecutive patients. Notably, PANDA utilized with non-contrast CT shows non-inferiority to radiology reports (using contrast-enhanced CT) in the differentiation of common pancreatic lesion subtypes. PANDA could potentially serve as a new tool for large-scale pancreatic cancer screening.

Label-free imaging of nuclear membrane for analysis of nuclear import of viral complexes

HIV-1 enters the nucleus of non-dividing cells through the nuclear pore complex where it integrates into the host genome. The mechanism of HIV-1 nuclear import remains poorly understood. A powerful means to investigate the docking of HIV-1 at the nuclear pore and nuclear import of viral complexes is through single virus tracking in live cells. This approach necessitates fluorescence labeling of HIV-1 particles and the nuclear envelope, which may be challenging, especially in the context of primary cells. Here, we leveraged a deep neural network model for label-free visualization of the nuclear envelope using transmitted light microscopy. A training image set of cells with fluorescently labeled nuclear Lamin B1 (ground truth), along with the corresponding transmitted light images, was acquired and used to train our model to predict the morphology of the nuclear envelope in fixed cells. This protocol yielded accurate predictions of the nuclear membrane and was used in conjunction with virus infection to examine the nuclear entry of fluorescently labeled HIV-1 complexes. Analyses of HIV-1 nuclear import as a function of virus input yielded identical numbers of fluorescent viral complexes per nucleus using the ground truth and predicted nuclear membrane images. We also demonstrate the utility of predicting the nuclear envelope based on transmitted light images for multicolor fluorescence microscopy of infected cells. Importantly, we show that our model can be adapted to predict the nuclear membrane of live cells imaged at 37 °C, making this approach compatible with single virus tracking. Collectively, these findings demonstrate the utility of deep learning approaches for label-free imaging of cellular structures during early stages of virus infection.

Label-free identification of protein aggregates using deep learning

Protein misfolding and aggregation play central roles in the pathogenesis of various neurodegenerative diseases (NDDs), including Huntington's disease, which is caused by a genetic mutation in exon 1 of the Huntingtin protein (Httex1). The fluorescent labels commonly used to visualize and monitor the dynamics of protein expression have been shown to alter the biophysical properties of proteins and the final ultrastructure, composition, and toxic properties of the formed aggregates. To overcome this limitation, we present a method for label-free identification of NDD-associated aggregates (LINA). Our approach utilizes deep learning to detect unlabeled and unaltered Httex1 aggregates in living cells from transmitted-light images, without the need for fluorescent labeling. Our models are robust across imaging conditions and on aggregates formed by different constructs of Httex1. LINA enables the dynamic identification of label-free aggregates and measurement of their dry mass and area changes during their growth process, offering high speed, specificity, and simplicity to analyze protein aggregation dynamics and obtain high-fidelity information.

Virtual staining for pixel-wise and quantitative analysis of single cell images

Immunocytochemical staining of microorganisms and cells has long been a popular method for examining their specific subcellular structures in greater detail. Recently, generative networks have emerged as an alternative to traditional immunostaining techniques. These networks infer fluorescence signatures from various imaging modalities and then virtually apply staining to the images in a digital environment. In numerous studies, virtual staining models have been trained on histopathology slides or intricate subcellular structures to enhance their accuracy and applicability. Despite the advancements in virtual staining technology, utilizing this method for quantitative analysis of microscopic images still poses a significant challenge. To address this issue, we propose a straightforward and automated approach for pixel-wise image-to-image translation. Our primary objective in this research is to leverage advanced virtual staining techniques to accurately measure the DNA fragmentation index in unstained sperm images. This not only offers a non-invasive approach to gauging sperm quality, but also paves the way for streamlined and efficient analyses without the constraints and potential biases introduced by traditional staining processes. This novel approach takes into account the limitations of conventional techniques and incorporates improvements to bolster the reliability of the virtual staining process. To further refine the results, we discuss various denoising techniques that can be employed to reduce the impact of background noise on the digital images. Additionally, we present a pixel-wise image matching algorithm designed to minimize the error caused by background noise and to prevent the introduction of bias into the analysis. By combining these approaches, we aim to develop a more effective and reliable method for quantitative analysis of virtually stained microscopic images, ultimately enhancing the study of microorganisms and cells at the subcellular level.

Facilitating cell segmentation with the projection-enhancement network

Contemporary approaches to instance segmentation in cell science use 2D or 3D convolutional networks depending on the experiment and data structures. However, limitations in microscopy systems or efforts to prevent phototoxicity commonly require recording sub-optimally sampled data that greatly reduces the utility of such 3D data, especially in crowded sample space with significant axial overlap between objects. In such regimes, 2D segmentations are both more reliable for cell morphology and easier to annotate. In this work, we propose the projection enhancement network (PEN), a novel convolutional module which processes the sub-sampled 3D data and produces a 2D RGB semantic compression, and is trained in conjunction with an instance segmentation network of choice to produce 2D segmentations. Our approach combines augmentation to increase cell density using a low-density cell image dataset to train PEN, and curated datasets to evaluate PEN. We show that with PEN, the learned semantic representation in CellPose encodes depth and greatly improves segmentation performance in comparison to maximum intensity projection images as input, but does not similarly aid segmentation in region-based networks like Mask-RCNN. Finally, we dissect the segmentation strength against cell density of PEN with CellPose on disseminated cells from side-by-side spheroids. We present PEN as a data-driven solution to form compressed representations of 3D data that improve 2D segmentations from instance segmentation networks.

Emerging open-channel droplet arrays for biosensing

Open-channel droplet arrays have attracted much attention in the fields of biochemical analysis, biofluid monitoring, biomarker recognition and cell interactions, as they have advantages with regard to miniaturization, parallelization, high-throughput, simplicity and accessibility. Such droplet arrays not only improve the sensitivity and accuracy of a biosensor, but also do not require sophisticated equipment or tedious processes, showing great potential in next-generation miniaturized sensing platforms. This review summarizes typical examples of open-channel microdroplet arrays and focuses on diversified biosensing integrated with multiple signal-output approaches (fluorescence, colorimetric, surface-enhanced Raman scattering (SERS), electrochemical, etc.). The limitations and development prospects of open-channel droplet arrays in biosensing are also discussed with regard to the increasing demand for biosensors.

DeepIFC: Virtual fluorescent labeling of blood cells in imaging flow cytometry data with deep learning

Imaging flow cytometry (IFC) combines flow cytometry with microscopy, allowing rapid characterization of cellular and molecular properties via high-throughput single-cell fluorescent imaging. However, fluorescent labeling is costly and time-consuming. We present a computational method called DeepIFC based on the Inception U-Net neural network architecture, able to generate fluorescent marker images and learn morphological features from IFC brightfield and darkfield images. Furthermore, the DeepIFC workflow identifies cell types from the generated fluorescent images and visualizes the single-cell features generated in a 2D space. We demonstrate that rarer cell types are predicted well when a balanced data set is used to train the model, and the model is able to recognize red blood cells not seen during model training as a distinct entity. In summary, DeepIFC allows accurate cell reconstruction, typing and recognition of unseen cell types from brightfield and darkfield images via virtual fluorescent labeling.

Building the next generation of virtual cells to understand cellular biology

Cell science has made significant progress by focusing on understanding individual cellular processes through reductionist approaches. However, the sheer volume of knowledge collected presents challenges in integrating this information across different scales of space and time to comprehend cellular behaviors, as well as making the data and methods more accessible for the community to tackle complex biological questions. This perspective proposes the creation of next-generation virtual cells, which are dynamic 3D models that integrate information from diverse sources, including simulations, biophysical models, image-based models, and evidence-based knowledge graphs. These virtual cells would provide statistically accurate and holistic views of real cells, bridging the gap between theoretical concepts and experimental data, and facilitating productive new collaborations among researchers across related fields.

insideOutside: an accessible algorithm for classifying interior and exterior points, with applications in embryology

A crucial aspect of embryology is relating the position of individual cells to the broader geometry of the embryo. A classic example of this is the first cell-fate decision of the mouse embryo, where interior cells become inner cell mass and exterior cells become trophectoderm. Fluorescent labelling, imaging, and quantification of tissue-specific proteins have advanced our understanding of this dynamic process. However, instances arise where these markers are either not available, or not reliable, and we are left only with the cells' spatial locations. Therefore, a simple, robust method for classifying interior and exterior cells of an embryo using spatial information is required. Here, we describe a simple mathematical framework and an unsupervised machine learning approach, termed insideOutside, for classifying interior and exterior points of a three-dimensional point-cloud, a common output from imaged cells within the early mouse embryo. We benchmark our method against other published methods to demonstrate that it yields greater accuracy in classification of nuclei from the pre-implantation mouse embryos and greater accuracy when challenged with local surface concavities. We have made MATLAB and Python implementations of the method freely available. This method should prove useful for embryology, with broader applications to similar data arising in the life sciences.

AutoUnmix: an autoencoder-based spectral unmixing method for multi-color fluorescence microscopy imaging

Multiplexed fluorescence microscopy imaging is widely used in biomedical applications. However, simultaneous imaging of multiple fluorophores can result in spectral leaks and overlapping, which greatly degrades image quality and subsequent analysis. Existing popular spectral unmixing methods are mainly based on computational intensive linear models, and the performance is heavily dependent on the reference spectra, which may greatly preclude its further applications. In this paper, we propose a deep learning-based blindly spectral unmixing method, termed AutoUnmix, to imitate the physical spectral mixing process. A transfer learning framework is further devised to allow our AutoUnmix to adapt to a variety of imaging systems without retraining the network. Our proposed method has demonstrated real-time unmixing capabilities, surpassing existing methods by up to 100-fold in terms of unmixing speed. We further validate the reconstruction performance on both synthetic datasets and biological samples. The unmixing results of AutoUnmix achieve the highest SSIM of 0.99 in both three- and four-color imaging, with nearly up to 20% higher than other popular unmixing methods. For experiments where spectral profiles and morphology are akin to simulated data, our method realizes the quantitative performance demonstrated above. Due to the desirable property of data independency and superior blind unmixing performance, we believe AutoUnmix is a powerful tool for studying the interaction process of different organelles labeled by multiple fluorophores.

Imaging brain tissue architecture across millimeter to nanometer scales

Mapping the complex and dense arrangement of cells and their connectivity in brain tissue demands nanoscale spatial resolution imaging. Super-resolution optical microscopy excels at visualizing specific molecules and individual cells but fails to provide tissue context. Here we developed Comprehensive Analysis of Tissues across Scales (CATS), a technology to densely map brain tissue architecture from millimeter regional to nanometer synaptic scales in diverse chemically fixed brain preparations, including rodent and human. CATS uses fixation-compatible extracellular labeling and optical imaging, including stimulated emission depletion or expansion microscopy, to comprehensively delineate cellular structures. It enables three-dimensional reconstruction of single synapses and mapping of synaptic connectivity by identification and analysis of putative synaptic cleft regions. Applying CATS to the mouse hippocampal mossy fiber circuitry, we reconstructed and quantified the synaptic input and output structure of identified neurons. We furthermore demonstrate applicability to clinically derived human tissue samples, including formalin-fixed paraffin-embedded routine diagnostic specimens, for visualizing the cellular architecture of brain tissue in health and disease.

Digital staining facilitates biomedical microscopy

Traditional staining of biological specimens for microscopic imaging entails time-consuming, laborious, and costly procedures, in addition to producing inconsistent labeling and causing irreversible sample damage. In recent years, computational "virtual" staining using deep learning techniques has evolved into a robust and comprehensive application for streamlining the staining process without typical histochemical staining-related drawbacks. Such virtual staining techniques can also be combined with neural networks designed to correct various microscopy aberrations, such as out-of-focus or motion blur artifacts, and improve upon diffracted-limited resolution. Here, we highlight how such methods lead to a host of new opportunities that can significantly improve both sample preparation and imaging in biomedical microscopy.

Label-free imaging of 3D pluripotent stem cell differentiation dynamics on chip

Massive, parallelized 3D stem cell cultures for engineering in vitro human cell types require imaging methods with high time and spatial resolution to fully exploit technological advances in cell culture technologies. Here, we introduce a large-scale integrated microfluidic chip platform for automated 3D stem cell differentiation. To fully enable dynamic high-content imaging on the chip platform, we developed a label-free deep learning method called Bright2Nuc to predict in silico nuclear staining in 3D from confocal microscopy bright-field images. Bright2Nuc was trained and applied to hundreds of 3D human induced pluripotent stem cell cultures differentiating toward definitive endoderm on a microfluidic platform. Combined with existing image analysis tools, Bright2Nuc segmented individual nuclei from bright-field images, quantified their morphological properties, predicted stem cell differentiation state, and tracked the cells over time. Our methods are available in an open-source pipeline, enabling researchers to upscale image acquisition and phenotyping of 3D cell culture.

MIA is an open-source standalone deep learning application for microscopic image analysis

In recent years, the amount of data generated by imaging techniques has grown rapidly, along with increasing computational power and the development of deep learning algorithms. To address the need for powerful automated image analysis tools for a broad range of applications in the biomedical sciences, the Microscopic Image Analyzer (MIA) was developed. MIA combines a graphical user interface that obviates the need for programming skills with state-of-the-art deep-learning algorithms for segmentation, object detection, and classification. It runs as a standalone, platform-independent application and uses open data formats, which are compatible with commonly used open-source software packages. The software provides a unified interface for easy image labeling, model training, and inference. Furthermore, the software was evaluated in a public competition and performed among the top three for all tested datasets.

Bright-field to fluorescence microscopy image translation for cell nuclei health quantification

Microscopy is a widely used method in biological research to observe the morphology and structure of cells. Amongst the plethora of microscopy techniques, fluorescent labeling with dyes or antibodies is the most popular method for revealing specific cellular organelles. However, fluorescent labeling also introduces new challenges to cellular observation, as it increases the workload, and the process may result in nonspecific labeling. Recent advances in deep visual learning have shown that there are systematic relationships between fluorescent and bright-field images, thus facilitating image translation between the two. In this article, we propose the cross-attention conditional generative adversarial network (XAcGAN) model. It employs state-of-the-art GANs (GANs) to solve the image translation task. The model uses supervised learning and combines attention-based networks to explore spatial information during translation. In addition, we demonstrate the successful application of XAcGAN to infer the health state of translated nuclei from bright-field microscopy images. The results show that our approach achieves excellent performance both in terms of image translation and nuclei state inference.

A live-cell image-based machine learning strategy for reducing variability in PSC differentiation systems

The differentiation of pluripotent stem cells (PSCs) into diverse functional cell types provides a promising solution to support drug discovery, disease modeling, and regenerative medicine. However, functional cell differentiation is currently limited by the substantial line-to-line and batch-to-batch variabilities, which severely impede the progress of scientific research and the manufacturing of cell products. For instance, PSC-to-cardiomyocyte (CM) differentiation is vulnerable to inappropriate doses of CHIR99021 (CHIR) that are applied in the initial stage of mesoderm differentiation. Here, by harnessing live-cell bright-field imaging and machine learning (ML), we realize real-time cell recognition in the entire differentiation process, e.g., CMs, cardiac progenitor cells (CPCs), PSC clones, and even misdifferentiated cells. This enables non-invasive prediction of differentiation efficiency, purification of ML-recognized CMs and CPCs for reducing cell contamination, early assessment of the CHIR dose for correcting the misdifferentiation trajectory, and evaluation of initial PSC colonies for controlling the start point of differentiation, all of which provide a more invulnerable differentiation method with resistance to variability. Moreover, with the established ML models as a readout for the chemical screen, we identify a CDK8 inhibitor that can further improve the cell resistance to the overdose of CHIR. Together, this study indicates that artificial intelligence is able to guide and iteratively optimize PSC differentiation to achieve consistently high efficiency across cell lines and batches, providing a better understanding and rational modulation of the differentiation process for functional cell manufacturing in biomedical applications.

Deep learning-enhanced fluorescence microscopy via confocal physical imaging model

Confocal microscopy is one of the most widely used tools for high-resolution cellular, tissue imaging and industrial inspection. Micrograph reconstruction based on deep learning has become an effective tool for modern microscopy imaging techniques. While most deep learning methods neglect the imaging process mechanism, which requires a lot of work to solve the multi-scale image pairs aliasing problem. We show that these limitations can be mitigated via an image degradation model based on Richards-Wolf vectorial diffraction integral and confocal imaging theory. The low-resolution images required for network training are generated by model degradation from their high-resolution counterparts, thereby eliminating the need for accurate image alignment. The image degradation model ensures the generalization and fidelity of the confocal images. By combining the residual neural network with a lightweight feature attention module with degradation model of confocal microscopy ensures high fidelity and generalization. Experiments on different measured data report that compared with the two deconvolution algorithms, non-negative least squares algorithm and Richardson-Lucy algorithm, the structural similarity index between the network output image and the real image reaches a high level above 0.82, and the peak signal-to-noise ratio can be improved by more than 0.6 dB. It also shows good applicability in different deep learning networks.

Digital labeling for 3D histology: segmenting blood vessels without a vascular contrast agent using deep learning

Recent advances in optical tissue clearing and three-dimensional (3D) fluorescence microscopy have enabled high resolution in situ imaging of intact tissues. Using simply prepared samples, we demonstrate here "digital labeling," a method to segment blood vessels in 3D volumes solely based on the autofluorescence signal and a nuclei stain (DAPI). We trained a deep-learning neural network based on the U-net architecture using a regression loss instead of a commonly used segmentation loss to achieve better detection of small vessels. We achieved high vessel detection accuracy and obtained accurate vascular morphometrics such as vessel length density and orientation. In the future, such digital labeling approach could easily be transferred to other biological structures.

Quantitative Single-Molecule Localization Microscopy

Super-resolution fluorescence microscopy allows the investigation of cellular structures at nanoscale resolution using light. Current developments in super-resolution microscopy have focused on reliable quantification of the underlying biological data. In this review, we first describe the basic principles of super-resolution microscopy techniques such as stimulated emission depletion (STED) microscopy and single-molecule localization microscopy (SMLM), and then give a broad overview of methodological developments to quantify super-resolution data, particularly those geared toward SMLM data. We cover commonly used techniques such as spatial point pattern analysis, colocalization, and protein copy number quantification but also describe more advanced techniques such as structural modeling, single-particle tracking, and biosensing. Finally, we provide an outlook on exciting new research directions to which quantitative super-resolution microscopy might be applied.

Morphodynamical cell state description via live-cell imaging trajectory embedding

Time-lapse imaging is a powerful approach to gain insight into the dynamic responses of cells, but the quantitative analysis of morphological changes over time remains challenging. Here, we exploit the concept of "trajectory embedding" to analyze cellular behavior using morphological feature trajectory histories-that is, multiple time points simultaneously, rather than the more common practice of examining morphological feature time courses in single timepoint (snapshot) morphological features. We apply this approach to analyze live-cell images of MCF10A mammary epithelial cells after treatment with a panel of microenvironmental perturbagens that strongly modulate cell motility, morphology, and cell cycle behavior. Our morphodynamical trajectory embedding analysis constructs a shared cell state landscape revealing ligand-specific regulation of cell state transitions and enables quantitative and descriptive models of single-cell trajectories. Additionally, we show that incorporation of trajectories into single-cell morphological analysis enables (i) systematic characterization of cell state trajectories, (ii) better separation of phenotypes, and (iii) more descriptive models of ligand-induced differences as compared to snapshot-based analysis. This morphodynamical trajectory embedding is broadly applicable to the quantitative analysis of cell responses via live-cell imaging across many biological and biomedical applications.

Microsystem Advances through Integration with Artificial Intelligence

Microfluidics is a rapidly growing discipline that involves studying and manipulating fluids at reduced length scale and volume, typically on the scale of micro- or nanoliters. Under the reduced length scale and larger surface-to-volume ratio, advantages of low reagent consumption, faster reaction kinetics, and more compact systems are evident in microfluidics. However, miniaturization of microfluidic chips and systems introduces challenges of stricter tolerances in designing and controlling them for interdisciplinary applications. Recent advances in artificial intelligence (AI) have brought innovation to microfluidics from design, simulation, automation, and optimization to bioanalysis and data analytics. In microfluidics, the Navier-Stokes equations, which are partial differential equations describing viscous fluid motion that in complete form are known to not have a general analytical solution, can be simplified and have fair performance through numerical approximation due to low inertia and laminar flow. Approximation using neural networks trained by rules of physical knowledge introduces a new possibility to predict the physicochemical nature. The combination of microfluidics and automation can produce large amounts of data, where features and patterns that are difficult to discern by a human can be extracted by machine learning. Therefore, integration with AI introduces the potential to revolutionize the microfluidic workflow by enabling the precision control and automation of data analysis. Deployment of smart microfluidics may be tremendously beneficial in various applications in the future, including high-throughput drug discovery, rapid point-of-care-testing (POCT), and personalized medicine. In this review, we summarize key microfluidic advances integrated with AI and discuss the outlook and possibilities of combining AI and microfluidics.

Confocal interferometric scattering microscopy reveals 3D nanoscopic structure and dynamics in live cells

Bright-field light microscopy and related phase-sensitive techniques play an important role in life sciences because they provide facile and label-free insights into biological specimens. However, lack of three-dimensional imaging and low sensitivity to nanoscopic features hamper their application in many high-end quantitative studies. Here, we demonstrate that interferometric scattering (iSCAT) microscopy operated in the confocal mode provides unique label-free solutions for live-cell studies. We reveal the nanometric topography of the nuclear envelope, quantify the dynamics of the endoplasmic reticulum, detect single microtubules, and map nanoscopic diffusion of clathrin-coated pits undergoing endocytosis. Furthermore, we introduce the combination of confocal and wide-field iSCAT modalities for simultaneous imaging of cellular structures and high-speed tracking of nanoscopic entities such as single SARS-CoV-2 virions. We benchmark our findings against simultaneously acquired fluorescence images. Confocal iSCAT can be readily implemented as an additional contrast mechanism in existing laser scanning microscopes. The method is ideally suited for live studies on primary cells that face labeling challenges and for very long measurements beyond photobleaching times.