Efficient prediction of a spatial transcriptomics profile better characterizes breast cancer tissue sections without costly experimentation

Combining spatial transcriptomics with tissue morphology

Spatial transcriptomics has transformed our understanding of tissue architecture by preserving the spatial context of gene expression patterns. Simultaneously, advances in imaging AI have enabled extraction of morphological features describing the tissue. This review introduces a framework for categorizing methods that combine spatial transcriptomics with tissue morphology, focusing on either translating or integrating morphological features into spatial transcriptomics. Translation involves using morphology to predict gene expression, creating super-resolution maps or inferring genetic information from H&E-stained samples. Integration enriches spatial transcriptomics by identifying morphological features that complement gene expression. We also explore learning strategies and future directions for this emerging field.

Spatial Mapping of Gene Signatures in Hematoxylin and Eosin-Stained Images: A Proof of Concept for Interpretable Predictions Using Additive Multiple Instance Learning

The relative abundance of cancer-associated fibroblast (CAF) subtypes influences a tumor's response to treatment, especially immunotherapy. However, the gene expression signatures associated with these CAF subtypes have yet to realize their potential as clinical biomarkers. Here, we describe an interpretable machine learning approach, additive multiple instance learning (aMIL), to predict bulk gene expression signatures from hematoxylin and eosin-stained whole-slide images, focusing on an immunosuppressive LRRC15+ CAF-enriched TGFβ-CAF signature. aMIL models accurately predicted TGFβ-CAF across various cancer types. Tissue regions contributing most highly to slide-level predictions of TGFβ-CAF were evaluated by machine learning models characterizing spatial distributions of diverse cell and tissue types, stromal subtypes, and nuclear morphology. In breast cancer, regions contributing most to TGFβ-CAF-high predictions ("excitatory") were localized to cancer stroma with high fibroblast density and mature collagen fibers. Regions contributing most to TGFβ-CAF-low predictions ("inhibitory") were localized to cancer epithelium and densely inflamed stroma. Fibroblast and lymphocyte nuclear morphology also differed between excitatory and inhibitory regions. Thus, aMIL enables a data-driven link between histologic features and transcription, offering biological interpretability beyond typical black-box models.

Decoding the blueprints of embryo development with single-cell and spatial omics

Embryonic development is a complex and intricately regulated process that encompasses precise control over cell differentiation, morphogenesis, and the underlying gene expression changes. Recent years have witnessed a remarkable acceleration in the development of single-cell and spatial omic technologies, enabling high-throughput profiling of transcriptomic and other multi-omic information at the individual cell level. These innovations offer fresh and multifaceted perspectives for investigating the intricate cellular and molecular mechanisms that govern embryonic development. In this review, we provide an in-depth exploration of the latest technical advancements in single-cell and spatial multi-omic methodologies and compile a systematic catalog of their applications in the field of embryonic development. We deconstruct the research strategies employed by recent studies that leverage single-cell sequencing techniques and underscore the unique advantages of spatial transcriptomics. Furthermore, we delve into both the current applications, data analysis algorithms and the untapped potential of these technologies in advancing our understanding of embryonic development. With the continuous evolution of multi-omic technologies, we anticipate their widespread adoption and profound contributions to unraveling the intricate molecular foundations underpinning embryo development in the foreseeable future.

Microfluidics for morpholomics and spatial omics applications

Creative designs, precise fluidic manipulation, and automation have supported the development of microfluidics for single-cell applications. Together with the advancements in detection technologies and artificial intelligence (AI), microfluidic-assisted platforms have been increasingly used for new modalities of single-cell investigations and in spatial omics applications. This review explores the use of microfluidic technologies for morpholomics and spatial omics with a focus on single-cell and tissue characterization. We emphasize how various fluid dynamic principles and unique design integrations enable highly precise fluid manipulation, enhancing sample handling in morpholomics. Additionally, we examine the use of microfluidics-assisted spatial barcoding with micrometer resolutions for the spatial profiling of tissue specimens. Finally, we discuss how microfluidics can serve as a bridge for integrating multiple unique fields in omics research and outline key challenges that these technologies may face in practical translation.

Benchmarking the translational potential of spatial gene expression prediction from histology

Spatial transcriptomics has enabled the quantification of gene expression at spatial coordinates across a tissue, offering crucial insights into molecular underpinnings of diseases. In light of this, several methods predicting spatial gene expression from paired histology images have provided the opportunity to enhance the utility of obtainable and cost-effective haematoxylin-and-eosin-stained histology images. To this end, we conduct a comprehensive benchmarking study encompassing eleven methods for predicting spatial gene expression with histology images. These methods are reproduced and evaluated using five Spatially Resolved Transcriptomics datasets, followed by external validation using The Cancer Genome Atlas data. Our evaluation incorporates diverse metrics which capture the performance of predicted gene expression, model generalisability, translational potential, usability and computational efficiency of each method. Our findings demonstrate the capacity of the methods to predict spatial gene expression from histology and highlight areas that can be addressed to support the advancement of this emerging field.

Exploring the role of metabolic pathways in TNBC immunotherapy: insights from single-cell and spatial transcriptomics

The article provides an overview of the current understanding of the interplay between metabolic pathways and immune function in the context of triple-negative breast cancer (TNBC). It highlights recent advancements in single-cell and spatial transcriptomics technologies, which have revolutionized the analysis of tumor heterogeneity and the immune microenvironment in TNBC. The review emphasizes the crucial role of metabolic reprogramming in modulating immune cell function, discussing how specific metabolic pathways, such as glycolysis, lipid metabolism, and amino acid metabolism, can directly impact the activity and phenotypes of various immune cell populations within the TNBC tumor microenvironment. Furthermore, the article explores the implications of these metabolic-immune interactions for the efficacy of immune checkpoint inhibitor (ICI) therapies in TNBC, suggesting that strategies targeting metabolic pathways may enhance the responsiveness to ICI treatments. Finally, the review outlines future directions and the potential for combination therapies that integrate metabolic modulation with immunotherapeutic approaches, offering promising avenues for improving clinical outcomes for TNBC patients.

ImSpiRE: image feature-aided spatial resolution enhancement method

The resolution of most spatially resolved transcriptomic technologies usually cannot attain the single-cell level, limiting their applications in biological discoveries. Here, we introduce ImSpiRE, an image feature-aided spatial resolution enhancement method for in situ capturing spatial transcriptome. Taking the information stored in histological images, ImSpiRE solves an optimal transport problem to redistribute the expression profiles of spots to construct new transcriptional profiles with enhanced resolution, together with extending the gene expression profiles into unmeasured regions. Applications to multiple datasets confirm that ImSpiRE can enhance spatial resolution to the subspot level while contributing to the discovery of tissue domains, signaling communication patterns, and spatiotemporal characterization.

Linking transcriptome and morphology in bone cells at cellular resolution with generative AI

Recent advancements in deep learning (DL) have revolutionized the capability of artificial intelligence (AI) by enabling the analysis of large-scale, complex datasets that are difficult for humans to interpret. However, large amounts of high-quality data are required to train such generative AI models successfully. With the rapid commercialization of single-cell sequencing and spatial transcriptomics platforms, the field is increasingly producing large-scale datasets such as histological images, single-cell molecular data, and spatial transcriptomic data. These molecular and morphological datasets parallel the multimodal text and image data used to train highly successful generative AI models for natural language processing and computer vision. Thus, these emerging data types offer great potential to train generative AI models that uncover intricate biological processes of bone cells at a cellular level. In this Perspective, we summarize the progress and prospects of generative AI applied to these datasets and their potential applications to bone research. In particular, we highlight three AI applications: predicting cell differentiation dynamics, linking molecular and morphological features, and predicting cellular responses to perturbations. To make generative AI models beneficial for bone research, important issues, such as technical biases in bone single-cell datasets, lack of profiling of important bone cell types, and lack of spatial information, needs to be addressed. Realizing the potential of generative AI for bone biology will also likely require generating large-scale, high-quality cellular-resolution spatial transcriptomics datasets, improving the sensitivity of current spatial transcriptomics datasets, and thorough experimental validation of model predictions.

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

STASCAN deciphers fine-resolution cell distribution maps in spatial transcriptomics by deep learning

Spatial transcriptomics technologies have been widely applied to decode cellular distribution by resolving gene expression profiles in tissue. However, sequencing techniques still limit the ability to create a fine-resolved spatial cell-type map. To this end, we develop a novel deep-learning-based approach, STASCAN, to predict the spatial cellular distribution of captured or uncharted areas where only histology images are available by cell feature learning integrating gene expression profiles and histology images. STASCAN is successfully applied across diverse datasets from different spatial transcriptomics technologies and displays significant advantages in deciphering higher-resolution cellular distribution and resolving enhanced organizational structures.

An initial game-theoretic assessment of enhanced tissue preparation and imaging protocols for improved deep learning inference of spatial transcriptomics from tissue morphology

The application of deep learning to spatial transcriptomics (ST) can reveal relationships between gene expression and tissue architecture. Prior work has demonstrated that inferring gene expression from tissue histomorphology can discern these spatial molecular markers to enable population scale studies, reducing the fiscal barriers associated with large-scale spatial profiling. However, while most improvements in algorithmic performance have focused on improving model architectures, little is known about how the quality of tissue preparation and imaging can affect deep learning model training for spatial inference from morphology and its potential for widespread clinical adoption. Prior studies for ST inference from histology typically utilize manually stained frozen sections with imaging on non-clinical grade scanners. Training such models on ST cohorts is also costly. We hypothesize that adopting tissue processing and imaging practices that mirror standards for clinical implementation (permanent sections, automated tissue staining, and clinical grade scanning) can significantly improve model performance. An enhanced specimen processing and imaging protocol was developed for deep learning-based ST inference from morphology. This protocol featured the Visium CytAssist assay to permit automated hematoxylin and eosin staining (e.g. Leica Bond), 40×-resolution imaging, and joining of multiple patients' tissue sections per capture area prior to ST profiling. Using a cohort of 13 pathologic T Stage-III stage colorectal cancer patients, we compared the performance of models trained on slide prepared using enhanced versus traditional (i.e. manual staining and low-resolution imaging) protocols. Leveraging Inceptionv3 neural networks, we predicted gene expression across serial, histologically-matched tissue sections using whole slide images (WSI) from both protocols. The data Shapley was used to quantify and compare marginal performance gains on a patient-by-patient basis attributed to using the enhanced protocol versus the actual costs of spatial profiling. Findings indicate that training and validating on WSI acquired through the enhanced protocol as opposed to the traditional method resulted in improved performance at lower fiscal cost. In the realm of ST, the enhancement of deep learning architectures frequently captures the spotlight; however, the significance of specimen processing and imaging is often understated. This research, informed through a game-theoretic lens, underscores the substantial impact that specimen preparation/imaging can have on spatial transcriptomic inference from morphology. It is essential to integrate such optimized processing protocols to facilitate the identification of prognostic markers at a larger scale.

Predicting spatially resolved gene expression via tissue morphology using adaptive spatial GNNs

MOTIVATION

Spatial transcriptomics technologies, which generate a spatial map of gene activity, can deepen the understanding of tissue architecture and its molecular underpinnings in health and disease. However, the high cost makes these technologies difficult to use in practice. Histological images co-registered with targeted tissues are more affordable and routinely generated in many research and clinical studies. Hence, predicting spatial gene expression from the morphological clues embedded in tissue histological images provides a scalable alternative approach to decoding tissue complexity.

RESULTS

Here, we present a graph neural network based framework to predict the spatial expression of highly expressed genes from tissue histological images. Extensive experiments on two separate breast cancer data cohorts demonstrate that our method improves the prediction performance compared to the state-of-the-art, and that our model can be used to better delineate spatial domains of biological interest.

AVAILABILITY AND IMPLEMENTATION

https://github.com/song0309/asGNN/.

HIPI: Spatially resolved multiplexed protein expression inferred from H&E WSIs

Solid tumors are characterized by complex interactions between the tumor, the immune system and the microenvironment. These interactions and intra-tumor variations have both diagnostic and prognostic significance and implications. However, quantifying the underlying processes in patient samples requires expensive and complicated molecular experiments. In contrast, H&E staining is typically performed as part of the routine standard process, and is very cheap. Here we present HIPI (H&E Image Interpretation and Protein Expression Inference) for predicting cell marker expression from tumor H&E images. We process paired H&E and CyCIF images taken from serial sections of colorectal cancers to train our model. We show that our model accurately predicts the spatial distribution of several important cell markers, on both held-out tumor regions as well as new tumor samples taken from different patients. Moreover, using only the tissue image morphology, HIPI is able to colocalize the interactions between different cell types, further demonstrating its potential clinical significance.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

Radiology and multi-scale data integration for precision oncology

In this Perspective paper we explore the potential of integrating radiological imaging with other data types, a critical yet underdeveloped area in comparison to the fusion of other multi-omic data. Radiological images provide a comprehensive, three-dimensional view of cancer, capturing features that would be missed by biopsies or other data modalities. This paper explores the complexities and challenges of incorporating medical imaging into data integration models, in the context of precision oncology. We present the different categories of imaging-omics integration and discuss recent progress, highlighting the opportunities that arise from bringing together spatial data on different scales.

Current computational methods for spatial transcriptomics in cancer biology

Cells in multicellular organisms constitute a self-organizing society by interacting with their neighbors. Cancer originates from malfunction of cellular behavior in the context of such a self-organizing system. The identities or characteristics of individual tumor cells can be represented by the hallmark of gene expression or transcriptome, which can be addressed using single-cell dissociation followed by RNA sequencing. However, the dissociation process of single cells results in losing the cellular address in tissue or neighbor information of each tumor cell, which is critical to understanding the malfunctioning cellular behavior in the microenvironment. Spatial transcriptomics technology enables measuring the transcriptome which is tagged by the address within a tissue. However, to understand cellular behavior in a self-organizing society, we need to apply mathematical or statistical methods. Here, we provide a review on current computational methods for spatial transcriptomics in cancer biology.

Data enhancement in the age of spatial biology

Unveiling the intricate interplay of cells in their native environment lies at the heart of understanding fundamental biological processes and unraveling disease mechanisms, particularly in complex diseases like cancer. Spatial transcriptomics (ST) offers a revolutionary lens into the spatial organization of gene expression within tissues, empowering researchers to study both cell heterogeneity and microenvironments in health and disease. However, current ST technologies often face limitations in either resolution or the number of genes profiled simultaneously. Integrating ST data with complementary sources, such as single-cell transcriptomics and detailed tissue staining images, presents a powerful solution to overcome these limitations. This review delves into the computational approaches driving the integration of spatial transcriptomics with other data types. By illuminating the key challenges and outlining the current algorithmic solutions, we aim to highlight the immense potential of these methods to revolutionize our understanding of cancer biology.

Prediction of DNA methylation-based tumor types from histopathology in central nervous system tumors with deep learning

Precision in the diagnosis of diverse central nervous system (CNS) tumor types is crucial for optimal treatment. DNA methylation profiles, which capture the methylation status of thousands of individual CpG sites, are state-of-the-art data-driven means to enhance diagnostic accuracy but are also time consuming and not widely available. Here, to address these limitations, we developed Deep lEarning from histoPathoLOgy and methYlation (DEPLOY), a deep learning model that classifies CNS tumors to ten major categories from histopathology. DEPLOY integrates three distinct components: the first classifies CNS tumors directly from slide images ('direct model'), the second initially generates predictions for DNA methylation beta values, which are subsequently used for tumor classification ('indirect model'), and the third classifies tumor types directly from routinely available patient demographics. First, we find that DEPLOY accurately predicts beta values from histopathology images. Second, using a ten-class model trained on an internal dataset of 1,796 patients, we predict the tumor categories in three independent external test datasets including 2,156 patients, achieving an overall accuracy of 95% and balanced accuracy of 91% on samples that are predicted with high confidence. These results showcase the potential future use of DEPLOY to assist pathologists in diagnosing CNS tumors within a clinically relevant short time frame.

Inferring single-cell spatial gene expression with tissue morphology via explainable deep learning

The spatial arrangement of cells is vital in developmental processes and organogenesis in multicellular life forms. Deep learning models trained with spatial omics data uncover complex patterns and relationships among cells, genes, and proteins in a high-dimensional space, providing new insights into biological processes and diseases. State-of-the-art in silico spatial multi-cell gene expression methods using histological images of tissue stained with hematoxylin and eosin (H&E) to characterize cellular heterogeneity. These computational techniques offer the advantage of analyzing vast amounts of spatial data in a scalable and automated manner, thereby accelerating scientific discovery and enabling more precise medical diagnostics and treatments. In this work, we developed a vision transformer (ViT) framework to map histological signatures to spatial single-cell transcriptomic signatures, named SPiRiT ( S patial Omics P rediction and R eproducibility integrated T ransformer). Our framework was enhanced by integrating cross validation with model interpretation during hyper-parameter tuning. SPiRiT predicts single-cell spatial gene expression using the matched histopathological image tiles of human breast cancer and whole mouse pup, evaluated by Xenium (10x Genomics) datasets. Furthermore, ViT model interpretation reveals the high-resolution, high attention area (HAR) that the ViT model uses to predict the gene expression, including marker genes for invasive cancer cells ( FASN ), stromal cells ( POSTN ), and lymphocytes ( IL7R ). In an apple-to-apple comparison with the ST-Net Convolutional Neural Network algorithm, SPiRiT improved predictive accuracy by 40% using human breast cancer Visium (10x Genomics) dataset. Cancer biomarker gene prediction and expression level are highly consistent with the tumor region annotation. In summary, our work highlights the feasibility to infer spatial single-cell gene expression using tissue morphology in multiple-species, i.e., human and mouse, and multi-organs, i.e., mouse whole body morphology. Importantly, incorporating model interpretation and vision transformer is expected to serve as a general-purpose framework for spatial transcriptomics.

High-density generation of spatial transcriptomics with STAGE

Spatial transcriptome technologies have enabled the measurement of gene expression while maintaining spatial location information for deciphering the spatial heterogeneity of biological tissues. However, they were heavily limited by the sparse spatial resolution and low data quality. To this end, we develop a spatial location-supervised auto-encoder generator STAGE for generating high-density spatial transcriptomics (ST). STAGE takes advantage of the customized supervised auto-encoder to learn continuous patterns of gene expression in space and generate high-resolution expressions for given spatial coordinates. STAGE can improve the low quality of spatial transcriptome data and smooth the generated manifold of gene expression through the de-noising function on the latent codes of the auto-encoder. Applications to four ST datasets, STAGE has shown better recovery performance for down-sampled data than existing methods, revealed significant tissue structure specificity, and enabled robust identification of spatially informative genes and patterns. In addition, STAGE can be extended to three-dimensional (3D) stacked ST data for generating gene expression at any position between consecutive sections for shaping high-density 3D ST configuration.

Harnessing TME depicted by histological images to improve cancer prognosis through a deep learning system

Spatial transcriptomics (ST) provides insights into the tumor microenvironment (TME), which is closely associated with cancer prognosis, but ST has limited clinical availability. In this study, we provide a powerful deep learning system to augment TME information based on histological images for patients without ST data, thereby empowering precise cancer prognosis. The system provides two connections to bridge existing gaps. The first is the integrated graph and image deep learning (IGI-DL) model, which predicts ST expression based on histological images with a 0.171 increase in mean correlation across three cancer types compared with five existing methods. The second connection is the cancer prognosis prediction model, based on TME depicted by spatial gene expression. Our survival model, using graphs with predicted ST features, achieves superior accuracy with a concordance index of 0.747 and 0.725 for The Cancer Genome Atlas breast cancer and colorectal cancer cohorts, outperforming other survival models. For the external Molecular and Cellular Oncology colorectal cancer cohort, our survival model maintains a stable advantage.

Spatial transcriptomics data and analytical methods: An updated perspective

Spatial transcriptomics (ST) is a newly emerging field that integrates high-resolution imaging and transcriptomic data to enable the high-throughput analysis of the spatial localization of transcripts in diverse biological systems. The rapid progress in this field necessitates the development of innovative computational methods to effectively tackle the distinct challenges posed by the analysis of ST data. These platforms, integrating AI techniques, offer a promising avenue for understanding disease mechanisms and expediting drug discovery. Despite significant advances in the development of ST data analysis techniques, there is an ongoing need to enhance these models for increased biological relevance. In this review, we briefly discuss the ST-related databases and current deep-learning-based models for spatial transcriptome data analyses and highlight their roles and future perspectives in biomedical applications.

Integrating single-cell transcriptomics with cellular phenotypes: cell morphology, Ca2+ imaging and electrophysiology

I review recent technological advancements in coupling single-cell transcriptomics with cellular phenotypes including morphology, calcium signaling, and electrophysiology. Single-cell RNA sequencing (scRNAseq) has revolutionized cell type classifications by capturing the transcriptional diversity of cells. A new wave of methods to integrate scRNAseq and biophysical measurements is facilitating the linkage of transcriptomic data to cellular function, which provides physiological insight into cellular states. I briefly discuss critical factors of these phenotypical characterizations such as timescales, information content, and analytical tools. Dedicated sections focus on the integration with cell morphology, calcium imaging, and electrophysiology (patch-seq), emphasizing their complementary roles. I discuss their application in elucidating cellular states, refining cell type classifications, and uncovering functional differences in cell subtypes. To illustrate the practical applications and benefits of these methods, I highlight their use in tissues with excitable cell-types such as the brain, pancreatic islets, and the retina. The potential of combining functional phenotyping with spatial transcriptomics for a detailed mapping of cell phenotypes in situ is explored. Finally, I discuss open questions and future perspectives, emphasizing the need for a shift towards broader accessibility through increased throughput.

Deep learning in spatially resolved transcriptfomics: a comprehensive technical view

Spatially resolved transcriptomics (SRT) is a pioneering method for simultaneously studying morphological contexts and gene expression at single-cell precision. Data emerging from SRT are multifaceted, presenting researchers with intricate gene expression matrices, precise spatial details and comprehensive histology visuals. Such rich and intricate datasets, unfortunately, render many conventional methods like traditional machine learning and statistical models ineffective. The unique challenges posed by the specialized nature of SRT data have led the scientific community to explore more sophisticated analytical avenues. Recent trends indicate an increasing reliance on deep learning algorithms, especially in areas such as spatial clustering, identification of spatially variable genes and data alignment tasks. In this manuscript, we provide a rigorous critique of these advanced deep learning methodologies, probing into their merits, limitations and avenues for further refinement. Our in-depth analysis underscores that while the recent innovations in deep learning tailored for SRT have been promising, there remains a substantial potential for enhancement. A crucial area that demands attention is the development of models that can incorporate intricate biological nuances, such as phylogeny-aware processing or in-depth analysis of minuscule histology image segments. Furthermore, addressing challenges like the elimination of batch effects, perfecting data normalization techniques and countering the overdispersion and zero inflation patterns seen in gene expression is pivotal. To support the broader scientific community in their SRT endeavors, we have meticulously assembled a comprehensive directory of readily accessible SRT databases, hoping to serve as a foundation for future research initiatives.

Potential to Enhance Large Scale Molecular Assessments of Skin Photoaging through Virtual Inference of Spatial Transcriptomics from Routine Staining

The advent of spatial transcriptomics technologies has heralded a renaissance in research to advance our understanding of the spatial cellular and transcriptional heterogeneity within tissues. Spatial transcriptomics allows investigation of the interplay between cells, molecular pathways, and the surrounding tissue architecture and can help elucidate developmental trajectories, disease pathogenesis, and various niches in the tumor microenvironment. Photoaging is the histological and molecular skin damage resulting from chronic/acute sun exposure and is a major risk factor for skin cancer. Spatial transcriptomics technologies hold promise for improving the reliability of evaluating photoaging and developing new therapeutics. Challenges to current methods include limited focus on dermal elastosis variations and reliance on self-reported measures, which can introduce subjectivity and inconsistency. Spatial transcriptomics offers an opportunity to assess photoaging objectively and reproducibly in studies of carcinogenesis and discern the effectiveness of therapies that intervene in photoaging and preventing cancer. Evaluation of distinct histological architectures using highly-multiplexed spatial technologies can identify specific cell lineages that have been understudied due to their location beyond the depth of UV penetration. However, the cost and interpatient variability using state-of-the-art assays such as the 10x Genomics Spatial Transcriptomics assays limits the scope and scale of large-scale molecular epidemiologic studies. Here, we investigate the inference of spatial transcriptomics information from routine hematoxylin and eosin-stained (H&E) tissue slides. We employed the Visium CytAssist spatial transcriptomics assay to analyze over 18,000 genes at a 50-micron resolution for four patients from a cohort of 261 skin specimens collected adjacent to surgical resection sites for basal cell and squamous cell keratinocyte tumors. The spatial transcriptomics data was co-registered with 40x resolution whole slide imaging (WSI) information. We developed machine learning models that achieved a macro-averaged median AUC and F1 score of 0.80 and 0.61 and Spearman coefficient of 0.60 in inferring transcriptomic profiles across the slides, and accurately captured biological pathways across various tissue architectures.

Transformer with convolution and graph-node co-embedding: An accurate and interpretable vision backbone for predicting gene expressions from local histopathological image

Inferring gene expressions from histopathological images has long been a fascinating yet challenging task, primarily due to the substantial disparities between the two modality. Existing strategies using local or global features of histological images are suffering model complexity, GPU consumption, low interpretability, insufficient encoding of local features, and over-smooth prediction of gene expressions among neighboring sites. In this paper, we develop TCGN (Transformer with Convolution and Graph-Node co-embedding method) for gene expression estimation from H&E-stained pathological slide images. TCGN comprises a combination of convolutional layers, transformer encoders, and graph neural networks, and is the first to integrate these blocks in a general and interpretable computer vision backbone. Notably, TCGN uniquely operates with just a single spot image as input for histopathological image analysis, simplifying the process while maintaining interpretability. We validate TCGN on three publicly available spatial transcriptomic datasets. TCGN consistently exhibited the best performance (with median PCC 0.232). TCGN offers superior accuracy while keeping parameters to a minimum (just 86.241 million), and it consumes minimal memory, allowing it to run smoothly even on personal computers. Moreover, TCGN can be extended to handle bulk RNA-seq data while providing the interpretability. Enhancing the accuracy of omics information prediction from pathological images not only establishes a connection between genotype and phenotype, enabling the prediction of costly-to-measure biomarkers from affordable histopathological images, but also lays the groundwork for future multi-modal data modeling. Our results confirm that TCGN is a powerful tool for inferring gene expressions from histopathological images in precision health applications.

GNTD: reconstructing spatial transcriptomes with graph-guided neural tensor decomposition informed by spatial and functional relations

Spatially-resolved RNA profiling has now been widely used to understand cells' structural organizations and functional roles in tissues, yet it is challenging to reconstruct the whole spatial transcriptomes due to various inherent technical limitations in tissue section preparation and RNA capture and fixation in the application of the spatial RNA profiling technologies. Here, we introduce a graph-guided neural tensor decomposition (GNTD) model for reconstructing whole spatial transcriptomes in tissues. GNTD employs a hierarchical tensor structure and formulation to explicitly model the high-order spatial gene expression data with a hierarchical nonlinear decomposition in a three-layer neural network, enhanced by spatial relations among the capture spots and gene functional relations for accurate reconstruction from highly sparse spatial profiling data. Extensive experiments on 22 Visium spatial transcriptomics datasets and 3 high-resolution Stereo-seq datasets as well as simulation data demonstrate that GNTD consistently improves the imputation accuracy in cross-validations driven by nonlinear tensor decomposition and incorporation of spatial and functional information, and confirm that the imputed spatial transcriptomes provide a more complete gene expression landscape for downstream analyses of cell/spot clustering for tissue segmentation, and spatial gene expression clustering and visualizations.

Deep topographic proteomics of a human brain tumour

The spatial organisation of cellular protein expression profiles within tissue determines cellular function and is key to understanding disease pathology. To define molecular phenotypes in the spatial context of tissue, there is a need for unbiased, quantitative technology capable of mapping proteomes within tissue structures. Here, we present a workflow for spatially-resolved, quantitative proteomics of tissue that generates maps of protein abundance across tissue slices derived from a human atypical teratoid-rhabdoid tumour at three spatial resolutions, the highest being 40 µm, to reveal distinct abundance patterns of thousands of proteins. We employ spatially-aware algorithms that do not require prior knowledge of the fine tissue structure to detect proteins and pathways with spatial abundance patterns and correlate proteins in the context of tissue heterogeneity and cellular features such as extracellular matrix or proximity to blood vessels. We identify PYGL, ASPH and CD45 as spatial markers for tumour boundary and reveal immune response-driven, spatially-organised protein networks of the extracellular tumour matrix. Overall, we demonstrate spatially-aware deep proteo-phenotyping of tissue heterogeneity, to re-define understanding tissue biology and pathology at the molecular level.

Feasibility of Inferring Spatial Transcriptomics from Single-Cell Histological Patterns for Studying Colon Cancer Tumor Heterogeneity

Background

Spatial transcriptomics involves studying the spatial organization of gene expression within tissues, offering insights into the molecular diversity of tumors. While spatial gene expression is commonly amalgamated from 1-10 cells across 50-micron spots, recent methods have demonstrated the capability to disaggregate this information at subspot resolution by leveraging both expression and histological patterns. However, elucidating such information from histology alone presents a significant challenge but if solved can better permit spatial molecular analysis at cellular resolution for instances where Visium data is not available, reducing study costs. This study explores integrating single-cell histological and transcriptomic data to infer spatial mRNA expression patterns in whole slide images collected from a cohort of stage pT3 colorectal cancer patients. A cell graph neural network algorithm was developed to align histological information extracted from detected cells with single cell RNA patterns through optimal transport methods, facilitating the analysis of cellular groupings and gene relationships. This approach leveraged spot-level expression as an intermediary to co-map histological and transcriptomic information at the single-cell level.

Results

Our study demonstrated that single-cell transcriptional heterogeneity within a spot could be predicted from histological markers extracted from cells detected within a spot. Furthermore, our model exhibited proficiency in delineating overarching gene expression patterns across whole-slide images. This approach compared favorably to traditional patch-based computer vision methods as well as other methods which did not incorporate single cell expression during the model fitting procedures. Topological nuances of single-cell expression within a Visium spot were preserved using the developed methodology.

Conclusion

This innovative approach augments the resolution of spatial molecular assays utilizing histology as a sole input through synergistic co-mapping of histological and transcriptomic datasets at the single-cell level, anchored by spatial transcriptomics. While initial results are promising, they warrant rigorous validation. This includes collaborating with pathologists for precise spatial identification of distinct cell types and utilizing sophisticated assays, such as Xenium, to attain deeper subcellular insights.

Prediction of cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin (H&E)-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an approach for predicting response to multiple targeted and immunotherapies from H&E-slides. In difference from existing approaches that aim to predict treatment response directly from the slides, ENLIGHT-DeepPT is an indirect two-step approach consisting of (1) DeepPT, a new deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response based on the DeepPT inferred expression values. DeepPT successfully predicts transcriptomics in all 16 TCGA cohorts tested and generalizes well to two independent datasets. Our key contribution is showing that ENLIGHT-DeepPT successfully predicts true responders in five independent patients' cohorts involving four different treatments spanning six cancer types with an overall odds ratio of 2.44, increasing the baseline response rate by 43.47% among predicted responders, without the need for any treatment data for training. Furthermore, its prediction accuracy on these datasets is comparable to a supervised approach predicting the response directly from the images, which needs to be trained and tested on the same cohort. ENLIGHT-DeepPT future application could provide clinicians with rapid treatment recommendations to an array of different therapies and importantly, may contribute to advancing precision oncology in developing countries.

Deep spatial-omics analysis of Head & Neck carcinomas provides alternative therapeutic targets and rationale for treatment failure

Immune checkpoint inhibitor (ICI) therapy has had limited success (<30%) in treating metastatic recurrent Head and Neck Oropharyngeal Squamous Cell Carcinomas (OPSCCs). We postulate that spatial determinants in the tumor play a critical role in cancer therapy outcomes. Here, we describe the case of a male patient diagnosed with p16+ OPSCC and extensive lung metastatic disease who failed Nivolumab and Pembrolizumab/Lenvatinib therapies. Using advanced integrative spatial proteogenomic analysis on the patient's recurrent OPSCC tumors we demonstrate that: (i) unbiased tissue clustering based on spatial transcriptomics (ST) successfully detected tumor cells and enabled the investigation of phenotypic traits such as proliferation or drug-resistance genes in the tumor's leading-edge and core; (ii) spatial proteomic imagining used in conjunction with ST (SpiCi, Spatial Proteomics inferred Cell identification) can resolve the profiling of tumor infiltrating immune cells, (iii) ST data allows for the discovery and ranking of clinically relevant alternative medicines based on their interaction with their matching ligand-receptor. Importantly, when the spatial profiles of ICI pre- and post-failure OPSCC tumors were compared, they exhibited highly similar PD-1/PD-L1low and VEGFAhigh expression, suggesting that these new tumors were not the product of ICI resistance but rather of Lenvatinib dose reduction due to complications. Our work establishes a path for incorporating spatial-omics in clinical settings to facilitate treatment personalization.

Deep learning exploration of single-cell and spatially resolved cancer transcriptomics to unravel tumour heterogeneity

Tumour heterogeneity is one of the critical confounding aspects in decoding tumour growth. Malignant cells display variations in their gene transcription profiles and mutation spectra even when originating from a single progenitor cell. Single-cell and spatial transcriptomics sequencing have recently emerged as key technologies for unravelling tumour heterogeneity. Single-cell sequencing promotes individual cell-type identification through transcriptome-wide gene expression measurements of each cell. Spatial transcriptomics facilitates identification of cell-cell interactions and the structural organization of heterogeneous cells within a tumour tissue through associating spatial RNA abundance of cells at distinct spots in the tissue section. However, extracting features and analyzing single-cell and spatial transcriptomics data poses challenges. Single-cell transcriptome data is extremely noisy and its sparse nature and dropouts can lead to misinterpretation of gene expression and the misclassification of cell types. Deep learning predictive power can overcome data challenges, provide high-resolution analysis and enhance precision oncology applications that involve early cancer prognosis, diagnosis, patient survival estimation and anti-cancer therapy planning. In this paper, we provide a background to and review of the recent progress of deep learning frameworks to investigate tumour heterogeneity using both single-cell and spatial transcriptomics data types.

Breast cancer histopathology image-based gene expression prediction using spatial transcriptomics data and deep learning

Tumour heterogeneity in breast cancer poses challenges in predicting outcome and response to therapy. Spatial transcriptomics technologies may address these challenges, as they provide a wealth of information about gene expression at the cell level, but they are expensive, hindering their use in large-scale clinical oncology studies. Predicting gene expression from hematoxylin and eosin stained histology images provides a more affordable alternative for such studies. Here we present BrST-Net, a deep learning framework for predicting gene expression from histopathology images using spatial transcriptomics data. Using this framework, we trained and evaluated four distinct state-of-the-art deep learning architectures, which include ResNet101, Inception-v3, EfficientNet (with six different variants), and vision transformer (with two different variants), all without utilizing pretrained weights for the prediction of 250 genes. To enhance the generalisation performance of the main network, we introduce an auxiliary network into the framework. Our methodology outperforms previous studies, with 237 genes identified with positive correlation, including 24 genes with a median correlation coefficient greater than 0.50. This is a notable improvement over previous studies, which could predict only 102 genes with positive correlation, with the highest correlation values ranging from 0.29 to 0.34.

Potential to Enhance Large Scale Molecular Assessments of Skin Photoaging through Virtual Inference of Spatial Transcriptomics from Routine Staining

The advent of spatial transcriptomics technologies has heralded a renaissance in research to advance our understanding of the spatial cellular and transcriptional heterogeneity within tissues. Spatial transcriptomics allows investigation of the interplay between cells, molecular pathways and the surrounding tissue architecture and can help elucidate developmental trajectories, disease pathogenesis, and various niches in the tumor microenvironment. Photoaging is the histological and molecular skin damage resulting from chronic/acute sun exposure and is a major risk factor for skin cancer. Spatial transcriptomics technologies hold promise for improving the reliability of evaluating photoaging and developing new therapeutics. Current challenges, including limited focus on dermal elastosis variations and reliance on self-reported measures, can introduce subjectivity and inconsistency. Spatial transcriptomics offer an opportunity to assess photoaging objectively and reproducibly in studies of carcinogenesis and discern the effectiveness of therapies that intervene on photoaging and prevent cancer. Evaluation of distinct histological architectures using highly-multiplexed spatial technologies can identify specific cell lineages that have been understudied due to their location beyond the depth of UV penetration. However, the cost and inter-patient variability using state-of-the-art assays such as the 10x Genomics Spatial Transcriptomics assays limits the scope and scale of large-scale molecular epidemiologic studies. Here, we investigate the inference of spatial transcriptomics information from routine hematoxylin and eosin-stained (H&E) tissue slides. We employed the Visium CytAssist spatial transcriptomics assay to analyze over 18,000 genes at a 50-micron resolution for four patients from a cohort of 261 skin specimens collected adjacent to surgical resection sites for basal and squamous keratinocyte tumors. The spatial transcriptomics data was co-registered with 40x resolution whole slide imaging (WSI) information. We developed machine learning models that achieved a macro-averaged median AUC and F1 score of 0.80 and 0.61 and Spearman coefficient of 0.60 in inferring transcriptomic profiles across the slides, and accurately captured biological pathways across various tissue architectures.

Integrated bioinformatics and statistical approach to identify the common molecular mechanisms of obesity that are linked to the development of two psychiatric disorders: Schizophrenia and major depressive disorder

Obesity is a chronic multifactorial disease characterized by the accumulation of body fat and serves as a gateway to a number of metabolic-related diseases. Epidemiologic data indicate that Obesity is acting as a risk factor for neuro-psychiatric disorders such as schizophrenia, major depression disorder and vice versa. However, how obesity may biologically interact with neurodevelopmental or neurological psychiatric conditions influenced by hereditary, environmental, and other factors is entirely unknown. To address this issue, we have developed a pipeline that integrates bioinformatics and statistical approaches such as transcriptomic analysis to identify differentially expressed genes (DEGs) and molecular mechanisms in patients with psychiatric disorders that are also common in obese patients. Biomarker genes expressed in schizophrenia, major depression, and obesity have been used to demonstrate such relationships depending on the previous research studies. The highly expressed genes identify commonly altered signalling pathways, gene ontology pathways, and gene-disease associations across disorders. The proposed method identified 163 significant genes and 134 significant pathways shared between obesity and schizophrenia. Similarly, there are 247 significant genes and 65 significant pathways that are shared by obesity and major depressive disorder. These genes and pathways increase the likelihood that psychiatric disorders and obesity are pathogenic. Thus, this study may help in the development of a restorative approach that will ameliorate the bidirectional relation between obesity and psychiatric disorder. Finally, we also validated our findings using genome-wide association study (GWAS) and whole-genome sequence (WGS) data from SCZ, MDD, and OBE. We confirmed the likely involvement of four significant genes both in transcriptomic and GWAS/WGS data. Moreover, we have performed co-expression cluster analysis of the transcriptomic data and compared it with the results of transcriptomic differential expression analysis and GWAS/WGS.

SPIN-AI: A Deep Learning Model That Identifies Spatially Predictive Genes

Spatially resolved sequencing technologies help us dissect how cells are organized in space. Several available computational approaches focus on the identification of spatially variable genes (SVGs), genes whose expression patterns vary in space. The detection of SVGs is analogous to the identification of differentially expressed genes and permits us to understand how genes and associated molecular processes are spatially distributed within cellular niches. However, the expression activities of SVGs fail to encode all information inherent in the spatial distribution of cells. Here, we devised a deep learning model, Spatially Informed Artificial Intelligence (SPIN-AI), to identify spatially predictive genes (SPGs), whose expression can predict how cells are organized in space. We used SPIN-AI on spatial transcriptomic data from squamous cell carcinoma (SCC) as a proof of concept. Our results demonstrate that SPGs not only recapitulate the biology of SCC but also identify genes distinct from SVGs. Moreover, we found a substantial number of ribosomal genes that were SPGs but not SVGs. Since SPGs possess the capability to predict spatial cellular organization, we reason that SPGs capture more biologically relevant information for a given cellular niche than SVGs. Thus, SPIN-AI has broad applications for detecting SPGs and uncovering which biological processes play important roles in governing cellular organization.

The role of tumor microenvironment in drug resistance: emerging technologies to unravel breast cancer heterogeneity

Breast cancer is a highly heterogeneous disease, at both inter- and intra-tumor levels, and this heterogeneity is a crucial determinant of malignant progression and response to treatments. In addition to genetic diversity and plasticity of cancer cells, the tumor microenvironment contributes to tumor heterogeneity shaping the physical and biological surroundings of the tumor. The activity of certain types of immune, endothelial or mesenchymal cells in the microenvironment can change the effectiveness of cancer therapies via a plethora of different mechanisms. Therefore, deciphering the interactions between the distinct cell types, their spatial organization and their specific contribution to tumor growth and drug sensitivity is still a major challenge. Dissecting intra-tumor heterogeneity is currently an urgent need to better define breast cancer biology and to develop therapeutic strategies targeting the microenvironment as helpful tools for combined and personalized treatment. In this review, we analyze the mechanisms by which the tumor microenvironment affects the characteristics of tumor heterogeneity that ultimately result in drug resistance, and we outline state of the art preclinical models and emerging technologies that will be instrumental in unraveling the impact of the tumor microenvironment on resistance to therapies.

Spatial RNA sequencing methods show high resolution of single cell in cancer metastasis and the formation of tumor microenvironment

Cancer metastasis often leads to death and therapeutic resistance. This process involves the participation of a variety of cell components, especially cellular and intercellular communications in the tumor microenvironment (TME). Using genetic sequencing technology to comprehensively characterize the tumor and TME is therefore key to understanding metastasis and therapeutic resistance. The use of spatial transcriptome sequencing enables the localization of gene expressions and cell activities in tissue sections. By examining the localization change as well as gene expression of these cells, it is possible to characterize the progress of tumor metastasis and TME formation. With improvements of this technology, spatial transcriptome sequencing technology has been extended from local regions to whole tissues, and from single sequencing technology to multimodal analysis combined with a variety of datasets. This has enabled the detection of every single cell in tissue slides, with high resolution, to provide more accurate predictive information for tumor treatments. In this review, we summarize the results of recent studies dealing with new multimodal methods and spatial transcriptome sequencing methods in tumors to illustrate recent developments in the imaging resolution of micro-tissues.

Detection of abnormal extraocular muscles in small datasets of computed tomography images using a three-dimensional variational autoencoder

We sought to establish an unsupervised algorithm with a three-dimensional (3D) variational autoencoder model (VAE) for the detection of abnormal extraocular muscles in small datasets of orbital computed tomography (CT) images. 334 CT images of normal orbits and 96 of abnormal orbits diagnosed as thyroid eye disease were used for training and validation; 24 normal and 11 abnormal orbits were used for the test. A 3D VAE was developed and trained. All images were preprocessed to emphasize extraocular muscles and to suppress background noise (e.g., high signal intensity from bones). The optimal cut-off value was identified through receiver operating characteristic (ROC) curve analysis. The ability of the model to detect muscles of abnormal size was assessed by visualization. The model achieved a sensitivity of 79.2%, specificity of 72.7%, accuracy of 77.1%, F1-score of 0.667, and AUROC of 0.801. Abnormal CT images correctly identified by the model showed differences in the reconstruction of extraocular muscles. The proposed model showed potential to detect abnormalities in extraocular muscles using a small dataset, similar to the diagnostic approach used by physicians. Unsupervised learning could serve as an alternative detection method for medical imaging studies in which annotation is difficult or impossible to perform.

Space in cancer biology: its role and implications

Tumor cells present complex behaviors in their interactions with other cells. This intricate behavior is driving the need to develop new tools to understand these ecosystems. The surge of spatial technologies allows evaluation of the complexity of relationships between cells present in a tumor, giving insights about tumor heterogeneity and the tumor microenvironment while providing clinically relevant metrics for tumor classification. In this review, we describe key results obtained using spatial techniques, present recent advances in methods to uncover spatially relevant biological significance, and summarize their main characteristics. We expect spatial technologies to significantly broaden our understanding of tumor biology and to generate clinically relevant tools that will ultimately impact personalized medicine.

Understanding Breast Cancers through Spatial and High-Resolution Visualization Using Imaging Technologies

Breast cancer is the most common cancer affecting women worldwide. Although many analyses and treatments have traditionally targeted the breast cancer cells themselves, recent studies have focused on investigating entire cancer tissues, including breast cancer cells. To understand the structure of breast cancer tissues, including breast cancer cells, it is necessary to investigate the three-dimensional location of the cells and/or proteins comprising the tissues and to clarify the relationship between the three-dimensional structure and malignant transformation or metastasis of breast cancers. In this review, we aim to summarize the methods for analyzing the three-dimensional structure of breast cancer tissue, paying particular attention to the recent technological advances in the combination of the tissue-clearing method and optical three-dimensional imaging. We also aimed to identify the latest methods for exploring the relationship between the three-dimensional cell arrangement in breast cancer tissues and the gene expression of each cell. Finally, we aimed to describe the three-dimensional imaging features of breast cancer tissues using noninvasive photoacoustic imaging methods.

Emerging artificial intelligence applications in Spatial Transcriptomics analysis

Spatial transcriptomics (ST) has advanced significantly in the last few years. Such advancement comes with the urgent need for novel computational methods to handle the unique challenges of ST data analysis. Many artificial intelligence (AI) methods have been developed to utilize various machine learning and deep learning techniques for computational ST analysis. This review provides a comprehensive and up-to-date survey of current AI methods for ST analysis.