Topological Tumor Graphs: A Graph-Based Spatial Model to Infer Stromal Recruitment for Immunosuppression in Melanoma Histology

Higher density of compact B cell clusters in invasive front may contribute to better prognosis in pancreatic ductal adenocarcinoma

While the correlation between T cells and patient survival was widely investigated, the clinical significance of CD20+ B cells in pancreatic ductal adenocarcinoma (PDAC) is less clear. We hypothesized that the spatial pattern of B cells within tumor microenvironment (TME) are more informative, which may reveal the prognostic significance for PDAC patients. Therefore, we developed a computer-based workflow to analyze CD20+ B cells in whole slide images (WSI) from 45 cases of PDAC patients. Depending on this workflow, annotations of each case which were created by pathologists were subdivided for three regions, including invasive front (IF), cancer center (CT) and cancer island (CI) to explore the association between the spatial pattern of CD20+ B cells and patient prognosis outcomes. After that, occupancy rate (as area under curve, occupancy AUC), fractal dimension differences (ΔFD), cluster density and coverage ratio were used to quantify the spatial pattern of B cells in TME. We observed B cells were distributed across different regions, manifesting in both clustered and dispersed patterns. Compared to features of B cells spatial distribution in CT region, B cells in IF region exhibited higher occupancy AUC (p = 0.00004), cluster density (p = 0.000002) and coverage ratio (p = 0.000884). Patients with longer survivals had smaller ΔFD (p = 0.05), higher B-cell cluster density (p = 0.003) and lower coverage ratio (p = 0.02) in IF region. Our study indicated the spatial distribution of B cells in IF and CT was different and the higher density of compact B-cell clusters in IF region may be associated with better prognosis in PDAC.

Identification of biomarkers and target drugs for melanoma: a topological and deep learning approach

Introduction

Melanoma, a highly aggressive malignancy characterized by rapid metastasis and elevated mortality rates, predominantly originates in cutaneous tissues. While surgical interventions, immunotherapy, and targeted therapies have advanced, the prognosis for advanced-stage melanoma remains dismal. Globally, melanoma incidence continues to rise, with the United States alone reporting over 100,000 new cases and 7,000 deaths annually. Despite the exponential growth of tumor data facilitated by next-generation sequencing (NGS), current analytical approaches predominantly emphasize single-gene analyses, neglecting critical insights into complex gene interaction networks. This study aims to address this gap by systematically exploring immune gene regulatory dynamics in melanoma progression.

Methods

We developed a bidirectional, weighted, signed, and directed topological immune gene regulatory network to compare transcriptional landscapes between benign melanocytic nevi and cutaneous melanoma. Advanced network analysis tools were employed to identify structural disparities and functional module shifts. Key driver genes were validated through topological centrality metrics. Additionally, deep learning models were implemented to predict drug-target interactions, leveraging molecular features derived from network analyses.

Results

Significant topological divergences emerged between nevi and melanoma networks, with dominant functional modules transitioning from cell cycle regulation in benign lesions to DNA repair and cell migration pathways in malignant tumors. A group of genes, including AURKA, CCNE1, APEX2, and EXOC8, were identified as potential orchestrators of immune microenvironment remodeling during malignant transformation. The deep learning framework successfully predicted 23 clinically actionable drug candidates targeting these molecular drivers.

Discussion

The observed module shift from cell cycle to invasion-related pathways provides mechanistic insights into melanoma progression, suggesting early therapeutic targeting of DNA repair machinery might mitigate metastatic potential. The identified hub genes, particularly AURKA and DDX19B, represent novel candidates for immunomodulatory interventions. Our computational drug prediction strategy bridges molecular network analysis with clinical translation, offering a paradigm for precision oncology in melanoma. Future studies should validate these targets in preclinical models and explore network-based biomarkers for early detection.

Predicting the efficacy of neoadjuvant chemotherapy in breast cancer patients based on ultrasound longitudinal temporal depth network fusion model

OBJECTIVE

The aim of this study was to develop and validate a deep learning radiomics (DLR) model based on longitudinal ultrasound data and clinical features to predict pathologic complete response (pCR) after neoadjuvant chemotherapy (NAC) in breast cancer patients.

METHODS

Between January 2018 and June 2023, 312 patients with histologically confirmed breast cancer were enrolled and randomly assigned to a training cohort (n = 219) and a test cohort (n = 93) in a 7:3 ratio. Next, pre-NAC and post-treatment 2-cycle ultrasound images were collected, and radiomics and deep learning features were extracted from NAC pre-treatment (Pre), post-treatment 2 cycle (Post), and Delta (pre-NAC-NAC 2 cycle) images. In the training cohort, to filter features, the intraclass correlation coefficient test, the Boruta algorithm, and the least absolute shrinkage and selection operator (LASSO) logistic regression were used. Single-modality models (Pre, Post, and Delta) were constructed based on five machine-learning classifiers. Finally, based on the classifier with the optimal predictive performance, the DLR model was constructed by combining Pre, Post, and Delta ultrasound features and was subsequently combined with clinical features to develop a combined model (Integrated). The discriminative power, predictive performance, and clinical utility of the models were further evaluated in the test cohort. Furthermore, patients were assigned into three subgroups, including the HR+/HER2-, HER2+, and TNBC subgroups, according to molecular typing to validate the predictability of the model across the different subgroups.

RESULTS

After feature screening, 16, 13, and 10 features were selected to construct the Pre model, Post model, and Delta model based on the five machine learning classifiers, respectively. The three single-modality models based on the XGBoost classifier displayed optimal predictive performance. Meanwhile, the DLR model (AUC of 0.827) was superior to the single-modality model (Pre, Post, and Delta AUCs of 0.726, 0.776, and 0.710, respectively) in terms of prediction performance. Moreover, multivariate logistic regression analysis identified Her-2 status and histological grade as independent risk factors for NAC response in breast cancer. In both the training and test cohorts, the Integrated model, which included Pre, Post, and Delta ultrasound features and clinical features, exhibited the highest predictive ability, with AUC values of 0.924 and 0.875, respectively. Likewise, the Integrated model displayed the highest predictive performance across the different subgroups.

CONCLUSION

The Integrated model, which incorporated pre-NAC treatment and early treatment ultrasound data and clinical features, accurately predicted pCR after NAC in breast cancer patients and provided valuable insights for personalized treatment strategies, allowing for timely adjustment of chemotherapy regimens.

Feature-interactive Siamese graph encoder-based image analysis to predict STAS from histopathology images in lung cancer

Spread through air spaces (STAS) is a distinct invasion pattern in lung cancer, crucial for prognosis assessment and guiding surgical decisions. Histopathology is the gold standard for STAS detection, yet traditional methods are subjective, time-consuming, and prone to misdiagnosis, limiting large-scale applications. We present VERN, an image analysis model utilizing a feature-interactive Siamese graph encoder to predict STAS from lung cancer histopathological images. VERN captures spatial topological features with feature sharing and skip connections to enhance model training. Using 1,546 histopathology slides, we built a large single-cohort STAS lung cancer dataset. VERN achieved an AUC of 0.9215 in internal validation and AUCs of 0.8275 and 0.8829 in frozen and paraffin-embedded test sections, respectively, demonstrating clinical-grade performance. Validated on a single-cohort and three external datasets, VERN showed robust predictive performance and generalizability, providing an open platform ( http://plr.20210706.xyz:5000/ ) to enhance STAS diagnosis efficiency and accuracy.

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Integrating histopathology and transcriptomics for spatial tumor microenvironment profiling in a melanoma case study

Local structures formed by cells in the tumor microenvironment (TME) play an important role in tumor development and treatment response. This study introduces SPoTLIghT, a computational framework providing a quantitative description of the tumor architecture from hematoxylin and eosin (H&E) slides. We trained a weakly supervised machine learning model on melanoma patients linking tile-level imaging features extracted from H&E slides to sample-level cell type quantifications derived from RNA-sequencing data. Using this model, SPoTLIghT provides spatial cellular maps for any H&E image, and converts them in graphs to derive 96 interpretable features capturing TME cellular organization. We show how SPoTLIghT's spatial features can distinguish microenvironment subtypes and reveal nuanced immune infiltration structures not apparent in molecular data alone. Finally, we use SPoTLIghT to effectively predict patients' prognosis in an independent melanoma cohort. SPoTLIghT enhances computational histopathology providing a quantitative and interpretable characterization of the spatial contexture of tumors.

Proximogram-A multi-omics network-based framework to capture tissue heterogeneity integrating single-cell omics and spatial profiling

The increasing availability of patient-derived multimodal biological data for various diseases has opened up avenues for finding the optimal methods for jointly leveraging the information extracted in a customizable and scalable manner. Here, we propose the Proximogram, a graph-based representation that provides a joint construct for embedding independently obtained omics and spatial data. To evaluate the representation, we generated proximograms from 2 distinct biological sources, namely, multiplexed immunofluorescence images and single-cell RNA-seq data obtained from patients across two pancreatic diseases that include normal and chronic Pancreatitis (CP) and pancreatic ductal adenocarcinoma (PDAC). The generated proximograms were used as inputs to 2 distinct graph deep-learning models. The improved classification results over simpler spatial-data-based input graphs point to the increased discriminatory power obtained by integrating structural information from single-cell ligand-receptor signaling data and the spatial architecture of cells in each disease class, which can help point to markers of high diagnostic significance.

Spatial effects of infiltrating T cells on neighbouring cancer cells and prognosis in stage III CRC patients

Colorectal cancer (CRC) is one of the most frequently occurring cancers, but prognostic biomarkers identifying patients at risk of recurrence are still lacking. In this study, we aimed to investigate in more detail the spatial relationship between intratumoural T cells, cancer cells, and cancer cell hallmarks as prognostic biomarkers in stage III colorectal cancer patients. We conducted multiplexed imaging of 56 protein markers at single-cell resolution on resected fixed tissue from stage III CRC patients who received adjuvant 5-fluorouracil (5FU)-based chemotherapy. Images underwent segmentation for tumour, stroma, and immune cells, and cancer cell 'state' protein marker expression was quantified at a cellular level. We developed a Python package for estimation of spatial proximity, nearest neighbour analysis focusing on cancer cell-T-cell interactions at single-cell level. In our discovery cohort (Memorial Sloan Kettering samples), we processed 462 core samples (total number of cells: 1,669,228) from 221 adjuvant 5FU-treated stage III patients. The validation cohort (Huntsville Clearview Cancer Center samples) consisted of 272 samples (total number of cells: 853,398) from 98 stage III CRC patients. While there were trends for an association between the percentage of cytotoxic T cells (across the whole cancer core), it did not reach significance (discovery cohort: p = 0.07; validation cohort: p = 0.19). We next utilised our region-based nearest neighbour approach to determine the spatial relationships between cytotoxic T cells, helper T cells, and cancer cell clusters. In both cohorts, we found that shorter distance between cytotoxic T cells, T helper cells, and cancer cells was significantly associated with increased disease-free survival. An unsupervised trained model that clustered patients based on the median distance between immune cells and cancer cells, as well as protein expression profiles, successfully classified patients into low-risk and high-risk groups (discovery cohort: p = 0.01; validation cohort: p = 0.003). © 2024 The Author(s). The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Computational methods and biomarker discovery strategies for spatial proteomics: a review in immuno-oncology

Advancements in imaging technologies have revolutionized our ability to deeply profile pathological tissue architectures, generating large volumes of imaging data with unparalleled spatial resolution. This type of data collection, namely, spatial proteomics, offers invaluable insights into various human diseases. Simultaneously, computational algorithms have evolved to manage the increasing dimensionality of spatial proteomics inherent in this progress. Numerous imaging-based computational frameworks, such as computational pathology, have been proposed for research and clinical applications. However, the development of these fields demands diverse domain expertise, creating barriers to their integration and further application. This review seeks to bridge this divide by presenting a comprehensive guideline. We consolidate prevailing computational methods and outline a roadmap from image processing to data-driven, statistics-informed biomarker discovery. Additionally, we explore future perspectives as the field moves toward interfacing with other quantitative domains, holding significant promise for precision care in immuno-oncology.

Impact of Genomic Mutation on Melanoma Immune Microenvironment and IFN-1 Pathway-Driven Therapeutic Responses

The BRAFV600E mutation, found in approximately 50% of melanoma cases, plays a crucial role in the activation of the MAPK/ERK signaling pathway, which promotes tumor cell proliferation. This study aimed to evaluate its impact on the melanoma immune microenvironment and therapeutic responses, particularly focusing on immunogenic cell death (ICD), a pivotal cytotoxic process triggering anti-tumor immune responses. Through comprehensive in silico analysis of the Cancer Genome Atlas data, we explored the association between the BRAFV600E mutation, immune subtype dynamics, and tumor mutation burden (TMB). Our findings revealed that the mutation correlated with a lower TMB, indicating a reduced generation of immunogenic neoantigens. Investigation into immune subtypes reveals an exacerbation of immunosuppression mechanisms in BRAFV600E-mutated tumors. To assess the response to ICD inducers, including doxorubicin and Me-ALA-based photodynamic therapy (PDT), compared to the non-ICD inducer cisplatin, we used distinct melanoma cell lines with wild-type BRAF (SK-MEL-2) and BRAFV600E mutation (SK-MEL-28, A375). We demonstrated a differential response to PDT between the WT and BRAFV600E cell lines. Further transcriptomic analysis revealed upregulation of IFNAR1, IFNAR2, and CXCL10 genes associated with the BRAFV600E mutation, suggesting their involvement in ICD. Using a gene reporter assay, we showed that PDT robustly activated the IFN-1 pathway through cGAS-STING signaling. Collectively, our results underscore the complex interplay between the BRAFV600E mutation and immune responses, suggesting a putative correlation between tumors carrying the mutation and their responsiveness to therapies inducing the IFN-1 pathway, such as the ICD inducer PDT, possibly mediated by the elevated expression of IFNAR1/2 receptors.

Deep cell phenotyping and spatial analysis of multiplexed imaging with TRACERx-PHLEX

The growing scale and dimensionality of multiplexed imaging require reproducible and comprehensive yet user-friendly computational pipelines. TRACERx-PHLEX performs deep learning-based cell segmentation (deep-imcyto), automated cell-type annotation (TYPEx) and interpretable spatial analysis (Spatial-PHLEX) as three independent but interoperable modules. PHLEX generates single-cell identities, cell densities within tissue compartments, marker positivity calls and spatial metrics such as cellular barrier scores, along with summary graphs and spatial visualisations. PHLEX was developed using imaging mass cytometry (IMC) in the TRACERx study, validated using published Co-detection by indexing (CODEX), IMC and orthogonal data and benchmarked against state-of-the-art approaches. We evaluated its use on different tissue types, tissue fixation conditions, image sizes and antibody panels. As PHLEX is an automated and containerised Nextflow pipeline, manual assessment, programming skills or pathology expertise are not essential. PHLEX offers an end-to-end solution in a growing field of highly multiplexed data and provides clinically relevant insights.

Spatial Architecture of Myeloid and T Cells Orchestrates Immune Evasion and Clinical Outcome in Lung Cancer

Understanding the role of the tumor microenvironment (TME) in lung cancer is critical to improving patient outcomes. We identified four histology-independent archetype TMEs in treatment-naïve early-stage lung cancer using imaging mass cytometry in the TRACERx study (n = 81 patients/198 samples/2.3 million cells). In immune-hot adenocarcinomas, spatial niches of T cells and macrophages increased with clonal neoantigen burden, whereas such an increase was observed for niches of plasma and B cells in immune-excluded squamous cell carcinomas (LUSC). Immune-low TMEs were associated with fibroblast barriers to immune infiltration. The fourth archetype, characterized by sparse lymphocytes and high tumor-associated neutrophil (TAN) infiltration, had tumor cells spatially separated from vasculature and exhibited low spatial intratumor heterogeneity. TAN-high LUSC had frequent PIK3CA mutations. TAN-high tumors harbored recently expanded and metastasis-seeding subclones and had a shorter disease-free survival independent of stage. These findings delineate genomic, immune, and physical barriers to immune surveillance and implicate neutrophil-rich TMEs in metastasis.

SIGNIFICANCE

This study provides novel insights into the spatial organization of the lung cancer TME in the context of tumor immunogenicity, tumor heterogeneity, and cancer evolution. Pairing the tumor evolutionary history with the spatially resolved TME suggests mechanistic hypotheses for tumor progression and metastasis with implications for patient outcome and treatment. This article is featured in Selected Articles from This Issue, p. 897.

Advancing Precision Medicine: Algebraic Topology and Differential Geometry in Radiology and Computational Pathology

Precision medicine aims to provide personalized care based on individual patient characteristics, rather than guideline-directed therapies for groups of diseases or patient demographics. Images-both radiology- and pathology-derived-are a major source of information on presence, type, and status of disease. Exploring the mathematical relationship of pixels in medical imaging ("radiomics") and cellular-scale structures in digital pathology slides ("pathomics") offers powerful tools for extracting both qualitative and, increasingly, quantitative data. These analytical approaches, however, may be significantly enhanced by applying additional methods arising from fields of mathematics such as differential geometry and algebraic topology that remain underexplored in this context. Geometry's strength lies in its ability to provide precise local measurements, such as curvature, that can be crucial for identifying abnormalities at multiple spatial levels. These measurements can augment the quantitative features extracted in conventional radiomics, leading to more nuanced diagnostics. By contrast, topology serves as a robust shape descriptor, capturing essential features such as connected components and holes. The field of topological data analysis was initially founded to explore the shape of data, with functional network connectivity in the brain being a prominent example. Increasingly, its tools are now being used to explore organizational patterns of physical structures in medical images and digitized pathology slides. By leveraging tools from both differential geometry and algebraic topology, researchers and clinicians may be able to obtain a more comprehensive, multi-layered understanding of medical images and contribute to precision medicine's armamentarium.

Current status of artificial intelligence methods for skin cancer survival analysis: a scoping review

Skin cancer mortality rates continue to rise, and survival analysis is increasingly needed to understand who is at risk and what interventions improve outcomes. However, current statistical methods are limited by inability to synthesize multiple data types, such as patient genetics, clinical history, demographics, and pathology and reveal significant multimodal relationships through predictive algorithms. Advances in computing power and data science enabled the rise of artificial intelligence (AI), which synthesizes vast amounts of data and applies algorithms that enable personalized diagnostic approaches. Here, we analyze AI methods used in skin cancer survival analysis, focusing on supervised learning, unsupervised learning, deep learning, and natural language processing. We illustrate strengths and weaknesses of these approaches with examples. Our PubMed search yielded 14 publications meeting inclusion criteria for this scoping review. Most publications focused on melanoma, particularly histopathologic interpretation with deep learning. Such concentration on a single type of skin cancer amid increasing focus on deep learning highlight growing areas for innovation; however, it also demonstrates opportunity for additional analysis that addresses other types of cutaneous malignancies and expands the scope of prognostication to combine both genetic, histopathologic, and clinical data. Moreover, researchers may leverage multiple AI methods for enhanced benefit in analyses. Expanding AI to this arena may enable improved survival analysis, targeted treatments, and outcomes.

Cell Spatial Analysis in Crohn's Disease: Unveiling Local Cell Arrangement Pattern with Graph-based Signatures

Crohn's disease (CD) is a chronic and relapsing inflammatory condition that affects segments of the gastrointestinal tract. CD activity is determined by histological findings, particularly the density of neutrophils observed on Hematoxylin and Eosin stains (H&E) imaging. However, understanding the broader morphometry and local cell arrangement beyond cell counting and tissue morphology remains challenging. To address this, we characterize six distinct cell types from H&E images and develop a novel approach for the local spatial signature of each cell. Specifically, we create a 10-cell neighborhood matrix, representing neighboring cell arrangements for each individual cell. Utilizing t-SNE for non-linear spatial projection in scatter-plot and Kernel Density Estimation contour-plot formats, our study examines patterns of differences in the cellular environment associated with the odds ratio of spatial patterns between active CD and control groups. This analysis is based on data collected at the two research institutes. The findings reveal heterogeneous nearest-neighbor patterns, signifying distinct tendencies of cell clustering, with a particular focus on the rectum region. These variations underscore the impact of data heterogeneity on cell spatial arrangements in CD patients. Moreover, the spatial distribution disparities between the two research sites highlight the significance of collaborative efforts among healthcare organizations. All research analysis pipeline tools are available at https://github.com/MASILab/cellNN.

Built to Last? Reproducibility and Reusability of Deep Learning Algorithms in Computational Pathology

Recent progress in computational pathology has been driven by deep learning. While code and data availability are essential to reproduce findings from preceding publications, ensuring a deep learning model's reusability is more challenging. For that, the codebase should be well-documented and easy to integrate into existing workflows and models should be robust toward noise and generalizable toward data from different sources. Strikingly, only a few computational pathology algorithms have been reused by other researchers so far, let alone employed in a clinical setting. To assess the current state of reproducibility and reusability of computational pathology algorithms, we evaluated peer-reviewed articles available in PubMed, published between January 2019 and March 2021, in 5 use cases: stain normalization; tissue type segmentation; evaluation of cell-level features; genetic alteration prediction; and inference of grading, staging, and prognostic information. We compiled criteria for data and code availability and statistical result analysis and assessed them in 160 publications. We found that only one-quarter (41 of 160 publications) made code publicly available. Among these 41 studies, three-quarters (30 of 41) analyzed their results statistically, half of them (20 of 41) released their trained model weights, and approximately a third (16 of 41) used an independent cohort for evaluation. Our review is intended for both pathologists interested in deep learning and researchers applying algorithms to computational pathology challenges. We provide a detailed overview of publications with published code in the field, list reusable data handling tools, and provide criteria for reproducibility and reusability.

Single-cell morphological and topological atlas reveals the ecosystem diversity of human breast cancer

Digital pathology allows computerized analysis of tumor ecosystem using whole slide images (WSIs). Here, we present single-cell morphological and topological profiling (sc-MTOP) to characterize tumor ecosystem by extracting the features of nuclear morphology and intercellular spatial relationship for individual cells. We construct a single-cell atlas comprising 410 million cells from 637 breast cancer WSIs and dissect the phenotypic diversity within tumor, inflammatory and stroma cells respectively. Spatially-resolved analysis identifies recurrent micro-ecological modules representing locoregional multicellular structures and reveals four breast cancer ecotypes correlating with distinct molecular features and patient prognosis. Further analysis with multiomics data uncovers clinically relevant ecosystem features. High abundance of locally-aggregated inflammatory cells indicates immune-activated tumor microenvironment and favorable immunotherapy response in triple-negative breast cancers. Morphological intratumor heterogeneity of tumor nuclei correlates with cell cycle pathway activation and CDK inhibitors responsiveness in hormone receptor-positive cases. sc-MTOP enables using WSIs to characterize tumor ecosystems at the single-cell level.

3D multiplexed tissue imaging reconstruction and optimized region of interest (ROI) selection through deep learning model of channels embedding

Introduction: Tissue-based sampling and diagnosis are defined as the extraction of information from certain limited spaces and its diagnostic significance of a certain object. Pathologists deal with issues related to tumor heterogeneity since analyzing a single sample does not necessarily capture a representative depiction of cancer, and a tissue biopsy usually only presents a small fraction of the tumor. Many multiplex tissue imaging platforms (MTIs) make the assumption that tissue microarrays (TMAs) containing small core samples of 2-dimensional (2D) tissue sections are a good approximation of bulk tumors although tumors are not 2D. However, emerging whole slide imaging (WSI) or 3D tumor atlases that use MTIs like cyclic immunofluorescence (CyCIF) strongly challenge this assumption. In spite of the additional insight gathered by measuring the tumor microenvironment in WSI or 3D, it can be prohibitively expensive and time-consuming to process tens or hundreds of tissue sections with CyCIF. Even when resources are not limited, the criteria for region of interest (ROI) selection in tissues for downstream analysis remain largely qualitative and subjective as stratified sampling requires the knowledge of objects and evaluates their features. Despite the fact TMAs fail to adequately approximate whole tissue features, a theoretical subsampling of tissue exists that can best represent the tumor in the whole slide image. Methods: To address these challenges, we propose deep learning approaches to learn multi-modal image translation tasks from two aspects: 1) generative modeling approach to reconstruct 3D CyCIF representation and 2) co-embedding CyCIF image and Hematoxylin and Eosin (H&E) section to learn multi-modal mappings by a cross-domain translation for minimum representative ROI selection. Results and discussion: We demonstrate that generative modeling enables a 3D virtual CyCIF reconstruction of a colorectal cancer specimen given a small subset of the imaging data at training time. By co-embedding histology and MTI features, we propose a simple convex optimization for objective ROI selection. We demonstrate the potential application of ROI selection and the efficiency of its performance with respect to cellular heterogeneity.

Application of digital pathology-based advanced analytics of tumour microenvironment organisation to predict prognosis and therapeutic response

In recent years, the application of advanced analytics, especially artificial intelligence (AI), to digital H&E images, and other histological image types, has begun to radically change how histological images are used in the clinic. Alongside the recognition that the tumour microenvironment (TME) has a profound impact on tumour phenotype, the technical development of highly multiplexed immunofluorescence platforms has enhanced the biological complexity that can be captured in the TME with high precision. AI has an increasingly powerful role in the recognition and quantitation of image features and the association of such features with clinically important outcomes, as occurs in distinct stages in conventional machine learning. Deep-learning algorithms are able to elucidate TME patterns inherent in the input data with minimum levels of human intelligence and, hence, have the potential to achieve clinically relevant predictions and discovery of important TME features. Furthermore, the diverse repertoire of deep-learning algorithms able to interrogate TME patterns extends beyond convolutional neural networks to include attention-based models, graph neural networks, and multimodal models. To date, AI models have largely been evaluated retrospectively, outside the well-established rigour of prospective clinical trials, in part because traditional clinical trial methodology may not always be suitable for the assessment of AI technology. However, to enable digital pathology-based advanced analytics to meaningfully impact clinical care, specific measures of 'added benefit' to the current standard of care and validation in a prospective setting are important. This will need to be accompanied by adequate measures of explainability and interpretability. Despite such challenges, the combination of expanding datasets, increased computational power, and the possibility of integration of pre-clinical experimental insights into model development means there is exciting potential for the future progress of these AI applications. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

The tumor-stroma ratio and the immune microenvironment improve the prognostic prediction of pancreatic ductal adenocarcinoma

Tumor-infiltrating immune cells and fibroblasts are significant components of the tumor microenvironment (TME) of pancreatic ductal adenocarcinoma (PDAC), and they participate in tumor progression as closely as tumor cells. However, the relationship between the features of the TME and patient outcomes and the interactions among TME components are still unclear. In this study, we evaluated the PDAC TME in terms of the quantity and location of cluster of differentiation (CD)4⁺ T cells, CD8⁺ T cells, macrophages, stromal maturity, and tumor-stroma ratio (TSR), as evaluated by immunohistochemical staining of serial whole-tissue sections from 116 patients with PDAC. The density of T cells and macrophages (mainly activated macrophages) was significantly higher at the invasive margins (IMs) than at the tumor center (TC). CD4⁺ T cells were significantly association with all the other tumor-associated immune cells (TAIs) including CD8, CD68 and CD206 positive cells. Tumors of the non-mature (intermediate and immature) stroma type harbored significantly more CD8⁺ T cells at the IMs and more CD68⁺ macrophages at the IMs and the TC. The density of CD4⁺, CD8⁺, and CD206⁺ cells at the TC; CD206⁺ cells at the IMs; and tumor-node-metastasis (TNM) staging were independent risk factors for patient outcomes, and the c-index of the risk nomogram for predicting the survival probability based on the TME features and TNM staging was 0.772 (95% confidence interval: 0.713-0.832). PDAC harbored a significantly immunosuppressive TME, of which the IMs were the hot zones for TAIs, while cells at the TC were more predictive of prognosis. Our results indicated that the model based on the features of the TME and TNM staging could predict patient outcomes.

Developing a definition of immune exclusion in cancer: results of a modified Delphi workshop

Checkpoint inhibitors represent an effective treatment approach for a variety of cancers through their inhibition of immune regulatory pathways within the tumor microenvironment (TME). Unfortunately only a minority of patients with cancer achieve clinical benefit from immunotherapy, with the TME emerging as an important predictor of outcomes and sensitivity to therapy. The extent and pattern of T-cell infiltration can vary prominently within/across tumors and represents a biological continuum. Three immune profiles have been identified along this continuum: 'immune-desert' or 'T-cell cold' phenotype, 'immune-active', 'inflamed', or 'T-cell hot' phenotype, and 'immune excluded' phenotype. Of the three profiles, immune excluded remains the most ill-defined with no clear, universally accepted definition even though it is commonly associated with lack of response to immune checkpoint inhibitors and poor clinical outcomes. To address this, 16 multidisciplinary cancer experts from around the world were invited to participate in a symposium using a three-round modified Delphi approach. The first round was an open-ended questionnaire distributed via email and the second was an in-person discussion of the first round results that allowed for statements to be revised as necessary to achieve a maximum consensus (75% agreement) among the rating committee (RC). The final round questionnaire was distributed to the RC via email and had a 100% completion rate. The Delphi process resulted in moving us closer to a consensus definition for immune exclusion that is practical, clinically pertinent, and applicable across a wide range of cancer histologies. A general consensus of the role of immune exclusion in resistance to checkpoint therapy and five research priorities emerged from this process. Together, these tools could help efforts designed to address the underlying mechanisms of immune exclusion that span cancer types and, ultimately, aid in the development of treatments to target these mechanisms to improve patient outcomes.

Weakly supervised deep learning to predict recurrence in low-grade endometrial cancer from multiplexed immunofluorescence images

Predicting recurrence in low-grade, early-stage endometrial cancer (EC) is both challenging and clinically relevant. We present a weakly-supervised deep learning framework, NaroNet, that can learn, without manual expert annotation, the complex tumor-immune interrelations at three levels: local phenotypes, cellular neighborhoods, and tissue areas. It uses multiplexed immunofluorescence for the simultaneous visualization and quantification of CD68 + macrophages, CD8 + T cells, FOXP3 + regulatory T cells, PD-L1/PD-1 protein expression, and tumor cells. We used 489 tumor cores from 250 patients to train a multilevel deep-learning model to predict tumor recurrence. Using a tenfold cross-validation strategy, our model achieved an area under the curve of 0.90 with a 95% confidence interval of 0.83-0.95. Our model predictions resulted in concordance for 96,8% of cases (κ = 0.88). This method could accurately assess the risk of recurrence in EC, outperforming current prognostic factors, including molecular subtyping.

Quantifying heterogeneity to drug response in cancer-stroma kinetics

A crucial challenge in medicine is choosing which drug (or combination) will be the most advantageous for a particular patient. Usually, drug response rates differ substantially, and the reasons for this response unpredictability remain ambiguous. Consequently, it is central to classify features that contribute to the observed drug response variability. Pancreatic cancer is one of the deadliest cancers with limited therapeutic achievements due to the massive presence of stroma that generates an environment that enables tumor growth, metastasis, and drug resistance. To understand the cancer-stroma cross talk within the tumor microenvironment and to develop personalized adjuvant therapies, there is a necessity for effective approaches that offer measurable data to monitor the effect of drugs at the single-cell level. Here, we develop a computational approach, based on cell imaging, that quantifies the cellular cross talk between pancreatic tumor cells (L3.6pl or AsPC1) and pancreatic stellate cells (PSCs), coordinating their kinetics in presence of the chemotherapeutic agent gemcitabine. We report significant heterogeneity in the organization of cellular interactions in response to the drug. For L3.6pl cells, gemcitabine sensibly decreases stroma-stroma interactions but increases stroma-cancer interactions, overall enhancing motility and crowding. In the AsPC1 case, gemcitabine promotes the interactions among tumor cells, but it does not affect stroma-cancer interplay, possibly suggesting a milder effect of the drug on cell dynamics.

Quantification of spatial and phenotypic heterogeneity in an agent-based model of tumour-macrophage interactions

We introduce a new spatial statistic, the weighted pair correlation function (wPCF). The wPCF extends the existing pair correlation function (PCF) and cross-PCF to describe spatial relationships between points marked with combinations of discrete and continuous labels. We validate its use through application to a new agent-based model (ABM) which simulates interactions between macrophages and tumour cells. These interactions are influenced by the spatial positions of the cells and by macrophage phenotype, a continuous variable that ranges from anti-tumour to pro-tumour. By varying model parameters that regulate macrophage phenotype, we show that the ABM exhibits behaviours which resemble the 'three Es of cancer immunoediting': Equilibrium, Escape, and Elimination. We use the wPCF to analyse synthetic images generated by the ABM. We show that the wPCF generates a 'human readable' statistical summary of where macrophages with different phenotypes are located relative to both blood vessels and tumour cells. We also define a distinct 'PCF signature' that characterises each of the three Es of immunoediting, by combining wPCF measurements with the cross-PCF describing interactions between vessels and tumour cells. By applying dimension reduction techniques to this signature, we identify its key features and train a support vector machine classifier to distinguish between simulation outputs based on their PCF signature. This proof-of-concept study shows how multiple spatial statistics can be combined to analyse the complex spatial features that the ABM generates, and to partition them into interpretable groups. The intricate spatial features produced by the ABM are similar to those generated by state-of-the-art multiplex imaging techniques which distinguish the spatial distribution and intensity of multiple biomarkers in biological tissue regions. Applying methods such as the wPCF to multiplex imaging data would exploit the continuous variation in biomarker intensities and generate more detailed characterisation of the spatial and phenotypic heterogeneity in tissue samples.

Predicting melanoma survival and metastasis with interpretable histopathological features and machine learning models

Introduction

Melanoma is the fifth most common cancer in US, and the incidence is increasing 1.4% annually. The overall survival rate for early-stage disease is 99.4%. However, melanoma can recur years later (in the same region of the body or as distant metastasis), and results in a dramatically lower survival rate. Currently there is no reliable method to predict tumor recurrence and metastasis on early primary tumor histological images.

Methods

To identify rapid, accurate, and cost-effective predictors of metastasis and survival, in this work, we applied various interpretable machine learning approaches to analyze melanoma histopathological H&E images. The result is a set of image features that can help clinicians identify high-risk-of-metastasis patients for increased clinical follow-up and precision treatment. We use simple models (i.e., logarithmic classification and KNN) and "human-interpretable" measures of cell morphology and tissue architecture (e.g., cell size, staining intensity, and cell density) to predict the melanoma survival on public and local Stage I-III cohorts as well as the metastasis risk on a local cohort.

Results

We use penalized survival regression to limit features available to downstream classifiers and investigate the utility of convolutional neural networks in isolating tumor regions to focus morphology extraction on only the tumor region. This approach allows us to predict survival and metastasis with a maximum F1 score of 0.72 and 0.73, respectively, and to visualize several high-risk cell morphologies.

Discussion

This lays the foundation for future work, which will focus on using our interpretable pipeline to predict metastasis in Stage I & II melanoma.

Towards a consensus definition of immune exclusion in cancer

Background

The immune cell topography of solid tumors has been increasingly recognized as an important predictive factor for progression of disease and response to immunotherapy. The distribution pattern of immune cells in solid tumors is commonly classified into three categories - namely, "Immune inflamed", "Immune desert" and "Immune excluded" - which, to some degree, connect immune cell presence and positioning within the tumor microenvironment to anti-tumor activity.

Materials and methods

In this review, we look at the ways immune exclusion has been defined in published literature and identify opportunities to develop consistent, quantifiable definitions, which in turn, will allow better determination of the underlying mechanisms that span cancer types and, ultimately, aid in the development of treatments to target these mechanisms.

Results

The definitions of tumor immune phenotypes, especially immune exclusion, have largely been conceptual. The existing literature lacks in consistency when it comes to practically defining immune exclusion, and there is no consensus on a definition. Majority of the definitions use somewhat arbitrary cut-offs in an attempt to place each tumor into a distinct phenotypic category. Tumor heterogeneity is often not accounted for, which limits the practical application of a definition.

Conclusions

We have identified two key issues in existing definitions of immune exclusion, establishing clinically relevant cut-offs within the spectrum of immune cell infiltration as well as tumor heterogeneity. We propose an approach to overcome these limitations, by reporting the degree of immune cell infiltration, tying cut-offs to clinically meaningful outcome measures, maximizing the number of regions of a tumor that are analyzed and reporting the degree of heterogeneity. This will allow for a consensus practical definition for operationalizing this categorization into clinical trial and signal-seeking endpoints.

Therapy sculpts the complex interplay between cancer and the immune system during tumour evolution

Cancer development is an evolutionary process. A key selection pressure is exerted by therapy, one of the few players in cancer evolution that can be controlled. As such, an understanding of how treatment acts to sculpt the tumour and its microenvironment and how this influences a tumour's subsequent evolutionary trajectory is critical. In this review, we examine cancer evolution and intra-tumour heterogeneity in the context of therapy. We focus on how radiotherapy, chemotherapy and immunotherapy shape both tumour development and the environment in which tumours evolve and how resistance can develop or be selected for during treatment.

Quantitative Spatial Profiling of Immune Populations in Pancreatic Ductal Adenocarcinoma Reveals Tumor Microenvironment Heterogeneity and Prognostic Biomarkers

Pancreatic ductal adenocarcinoma (PDAC) is a highly aggressive disease with poor 5-year survival rates, necessitating identification of novel therapeutic targets. Elucidating the biology of the tumor immune microenvironment (TiME) can provide vital insights into mechanisms of tumor progression. In this study, we developed a quantitative image processing platform to analyze sequential multiplexed IHC data from archival PDAC tissue resection specimens. A 27-plex marker panel was employed to simultaneously phenotype cell populations and their functional states, followed by a computational workflow to interrogate the immune contextures of the TiME in search of potential biomarkers. The PDAC TiME reflected a low-immunogenic ecosystem with both high intratumoral and intertumoral heterogeneity. Spatial analysis revealed that the relative distance between IL10+ myelomonocytes, PD-1+ CD4+ T cells, and granzyme B+ CD8+ T cells correlated significantly with survival, from which a spatial proximity signature termed imRS was derived that correlated with PDAC patient survival. Furthermore, spatial enrichment of CD8+ T cells in lymphoid aggregates was also linked to improved survival. Altogether, these findings indicate that the PDAC TiME, generally considered immuno-dormant or immunosuppressive, is a spatially nuanced ecosystem orchestrated by ordered immune hierarchies. This new understanding of spatial complexity may guide novel treatment strategies for PDAC.

SIGNIFICANCE

Quantitative image analysis of PDAC specimens reveals intertumoral and intratumoral heterogeneity of immune populations and identifies spatial immune architectures that are significantly associated with disease prognosis.

Application of Artificial Intelligence in Pathology: Trends and Challenges

Given the recent success of artificial intelligence (AI) in computer vision applications, many pathologists anticipate that AI will be able to assist them in a variety of digital pathology tasks. Simultaneously, tremendous advancements in deep learning have enabled a synergy with artificial intelligence (AI), allowing for image-based diagnosis on the background of digital pathology. There are efforts for developing AI-based tools to save pathologists time and eliminate errors. Here, we describe the elements in the development of computational pathology (CPATH), its applicability to AI development, and the challenges it faces, such as algorithm validation and interpretability, computing systems, reimbursement, ethics, and regulations. Furthermore, we present an overview of novel AI-based approaches that could be integrated into pathology laboratory workflows.

Spatial Transcriptomic Analysis Reveals Associations between Genes and Cellular Topology in Breast and Prostate Cancers

BACKGROUND

Cancer is the leading cause of death worldwide with breast and prostate cancer the most common among women and men, respectively. Gene expression and image features are independently prognostic of patient survival; but until the advent of spatial transcriptomics (ST), it was not possible to determine how gene expression of cells was tied to their spatial relationships (i.e., topology).

METHODS

We identify topology-associated genes (TAGs) that correlate with 700 image topological features (ITFs) in breast and prostate cancer ST samples. Genes and image topological features are independently clustered and correlated with each other. Themes among genes correlated with ITFs are investigated by functional enrichment analysis.

RESULTS

Overall, topology-associated genes (TAG) corresponding to extracellular matrix (ECM) and Collagen Type I Trimer gene ontology terms are common to both prostate and breast cancer. In breast cancer specifically, we identify the ZAG-PIP Complex as a TAG. In prostate cancer, we identify distinct TAGs that are enriched for GI dysmotility and the IgA immunoglobulin complex. We identified TAGs in every ST slide regardless of cancer type.

CONCLUSIONS

These TAGs are enriched for ontology terms, illustrating the biological relevance to our image topology features and their potential utility in diagnostic and prognostic models.

Computational tumor stroma reaction evaluation led to novel prognosis-associated fibrosis and molecular signature discoveries in high-grade serous ovarian carcinoma

Background

As one of the key criteria to differentiate benign vs. malignant tumors in ovarian and other solid cancers, tumor-stroma reaction (TSR) is long observed by pathologists and has been found correlated with patient prognosis. However, paucity of study aims to overcome subjective bias or automate TSR evaluation for enabling association analysis to a large cohort.

Materials and methods

Serving as positive and negative sets of TSR studies, H&E slides of primary tumors of high-grade serous ovarian carcinoma (HGSOC) (n = 291) and serous borderline ovarian tumor (SBOT) (n = 15) were digitally scanned. Three pathologist-defined quantification criteria were used to characterize the extents of TSR. Scores for each criterion were annotated (0/1/2 as none-low/intermediate/high) in the training set consisting of 18,265 H&E patches. Serial of deep learning (DL) models were trained to identify tumor vs. stroma regions and predict TSR scores. After cross-validation and independent validations, the trained models were generalized to the entire HGSOC cohort and correlated with clinical characteristics. In a subset of cases tumor transcriptomes were available, gene- and pathway-level association studies were conducted with TSR scores.

Results

The trained models accurately identified the tumor stroma tissue regions and predicted TSR scores. Within tumor stroma interface region, TSR fibrosis scores were strongly associated with patient prognosis. Cancer signaling aberrations associated 14 KEGG pathways were also found positively correlated with TSR-fibrosis score.

Conclusion

With the aid of DL, TSR evaluation could be generalized to large cohort to enable prognostic association analysis and facilitate discovering novel gene and pathways associated with disease progress.

An Open-Source AI Framework for the Analysis of Single Cells in Whole-Slide Images with a Note on CD276 in Glioblastoma

Routine examination of entire histological slides at cellular resolution poses a significant if not insurmountable challenge to human observers. However, high-resolution data such as the cellular distribution of proteins in tissues, e.g., those obtained following immunochemical staining, are highly desirable. Our present study extends the applicability of the PathoFusion framework to the cellular level. We illustrate our approach using the detection of CD276 immunoreactive cells in glioblastoma as an example. Following automatic identification by means of PathoFusion's bifocal convolutional neural network (BCNN) model, individual cells are automatically profiled and counted. Only discriminable cells selected through data filtering and thresholding were segmented for cell-level analysis. Subsequently, we converted the detection signals into the corresponding heatmaps visualizing the distribution of the detected cells in entire whole-slide images of adjacent H&E-stained sections using the Discrete Wavelet Transform (DWT). Our results demonstrate that PathoFusion is capable of autonomously detecting and counting individual immunochemically labelled cells with a high prediction performance of 0.992 AUC and 97.7% accuracy. The data can be used for whole-slide cross-modality analyses, e.g., relationships between immunochemical signals and anaplastic histological features. PathoFusion has the potential to be applied to additional problems that seek to correlate heterogeneous data streams and to serve as a clinically applicable, weakly supervised system for histological image analyses in (neuro)pathology.

HistoML, a markup language for representation and exchange of histopathological features in pathology images

The study of histopathological phenotypes is vital for cancer research and medicine as it links molecular mechanisms to disease prognosis. It typically involves integration of heterogenous histopathological features in whole-slide images (WSI) to objectively characterize a histopathological phenotype. However, the large-scale implementation of phenotype characterization has been hindered by the fragmentation of histopathological features, resulting from the lack of a standardized format and a controlled vocabulary for structured and unambiguous representation of semantics in WSIs. To fill this gap, we propose the Histopathology Markup Language (HistoML), a representation language along with a controlled vocabulary (Histopathology Ontology) based on Semantic Web technologies. Multiscale features within a WSI, from single-cell features to mesoscopic features, could be represented using HistoML which is a crucial step towards the goal of making WSIs findable, accessible, interoperable and reusable (FAIR). We pilot HistoML in representing WSIs of kidney cancer as well as thyroid carcinoma and exemplify the uses of HistoML representations in semantic queries to demonstrate the potential of HistoML-powered applications for phenotype characterization.

Spatial heterogeneity and organization of tumor mutation burden with immune infiltrates within tumors based on whole slide images correlated with patient survival in bladder cancer

Background

High tumor mutation burden (TMB-H) could result in an increased number of neoepitopes from somatic mutations expressed by a patient's own tumor cell which can be recognized and targeted by neighboring tumor-infiltrating lymphocytes (TILs). Deeper understanding of spatial heterogeneity and organization of tumor cells and their neighboring immune infiltrates within tumors could provide new insights into tumor progression and treatment response.

Methods

Here we first developed computational approaches using whole slide images (WSIs) to predict bladder cancer patients' TMB status and TILs across tumor regions, and then investigate spatial heterogeneity and organization of regions harboring TMB-H tumor cells and TILs within tumors, as well as their prognostic utility. Results: In experiments using WSIs from The Cancer Genome Atlas (TCGA) bladder cancer (BLCA), our findings show that computational pathology can reliably predict patient-level TMB status and delineate spatial TMB heterogeneity and co-organization with TILs. TMB-H patients with low spatial heterogeneity enriched with high TILs show improved overall survival.

Conclusions

Computational approaches using WSIs have the potential to provide rapid and cost-effective TMB testing and TILs detection. Survival analysis illuminates potential clinical utility of spatial heterogeneity and co-organization of TMB and TILs as a prognostic biomarker in BLCA which warrants further validation in future studies.

An integrated deep learning model for the prediction of pathological complete response to neoadjuvant chemotherapy with serial ultrasonography in breast cancer patients: a multicentre, retrospective study

BACKGROUND

The biological phenotype of tumours evolves during neoadjuvant chemotherapy (NAC). Accurate prediction of pathological complete response (pCR) to NAC in the early-stage or posttreatment can optimize treatment strategies or improve the breast-conserving rate. This study aimed to develop and validate an autosegmentation-based serial ultrasonography assessment system (SUAS) that incorporated serial ultrasonographic features throughout the NAC of breast cancer to predict pCR.

METHODS

A total of 801 patients with biopsy-proven breast cancer were retrospectively enrolled from three institutions and were split into a training cohort (242 patients), an internal validation cohort (197 patients), and two external test cohorts (212 and 150 patients). Three imaging signatures were constructed from the serial ultrasonographic features before (pretreatment signature), during the first-second cycle of (early-stage treatment signature), and after (posttreatment signature) NAC based on autosegmentation by U-net. The SUAS was constructed by subsequently integrating the pre, early-stage, and posttreatment signatures, and the incremental performance was analysed.

RESULTS

The SUAS yielded a favourable performance in predicting pCR, with areas under the receiver operating characteristic curve (AUCs) of 0.927 [95% confidence interval (CI) 0.891-0.963] and 0.914 (95% CI 0.853-0.976), compared with those of the clinicopathological prediction model [0.734 (95% CI 0.665-0.804) and 0.610 (95% CI 0.504-0.716)], and radiologist interpretation [0.632 (95% CI 0.570-0.693) and 0.724 (95% CI 0.644-0.804)] in the external test cohorts. Furthermore, similar results were also observed in the early-stage treatment of NAC [AUC 0.874 (0.793-0.955)-0.897 (0.851-0.943) in the external test cohorts].

CONCLUSIONS

We demonstrate that autosegmentation-based SAUS integrating serial ultrasonographic features throughout NAC can predict pCR with favourable performance, which can facilitate individualized treatment strategies.

Quick Annotator: an open-source digital pathology based rapid image annotation tool

Image-based biomarker discovery typically requires accurate segmentation of histologic structures (e.g. cell nuclei, tubules, and epithelial regions) in digital pathology whole slide images (WSIs). Unfortunately, annotating each structure of interest is laborious and often intractable even in moderately sized cohorts. Here, we present an open-source tool, Quick Annotator (QA), designed to improve annotation efficiency of histologic structures by orders of magnitude. While the user annotates regions of interest (ROIs) via an intuitive web interface, a deep learning (DL) model is concurrently optimized using these annotations and applied to the ROI. The user iteratively reviews DL results to either (1) accept accurately annotated regions or (2) correct erroneously segmented structures to improve subsequent model suggestions, before transitioning to other ROIs. We demonstrate the effectiveness of QA over comparable manual efforts via three use cases. These include annotating (1) 337,386 nuclei in 5 pancreatic WSIs, (2) 5,692 tubules in 10 colorectal WSIs, and (3) 14,187 regions of epithelium in 10 breast WSIs. Efficiency gains in terms of annotations per second of 102×, 9×, and 39× were, respectively, witnessed while retaining f-scores >0.95, suggesting that QA may be a valuable tool for efficiently fully annotating WSIs employed in downstream biomarker studies.

A deep learning quantified stroma-immune score to predict survival of patients with stage II-III colorectal cancer

Background

Profound heterogeneity in prognosis has been observed in colorectal cancer (CRC) patients with intermediate levels of disease (stage II–III), advocating the identification of valuable biomarkers that could improve the prognostic stratification. This study aims to develop a deep learning-based pipeline for fully automatic quantification of immune infiltration within the stroma region on immunohistochemical (IHC) whole-slide images (WSIs) and further analyze its prognostic value in CRC.

Methods

Patients from two independent cohorts were divided into three groups: the development group (N = 200), the internal (N = 134), and the external validation group (N = 90). We trained a convolutional neural network for tissue classification of CD3 and CD8 stained WSIs. A scoring system, named stroma-immune score, was established by quantifying the density of CD3⁺ and CD8⁺ T-cells infiltration in the stroma region.

Results

Patients with higher stroma-immune scores had much longer survival. In the development group, 5-year survival rates of the low and high scores were 55.7% and 80.8% (hazard ratio [HR] for high vs. low 0.39, 95% confidence interval [CI] 0.24–0.63, P < 0.001). These results were confirmed in the internal and external validation groups with 5-year survival rates of low and high scores were 57.1% and 78.8%, 63.9% and 88.9%, respectively (internal: HR for high vs. low 0.49, 95% CI 0.28–0.88, P = 0.017; external: HR for high vs. low 0.35, 95% CI 0.15–0.83, P = 0.018). The combination of stroma-immune score and tumor-node-metastasis (TNM) stage showed better discrimination ability for survival prediction than using the TNM stage alone.

Conclusions

We proposed a stroma-immune score via a deep learning-based pipeline to quantify CD3⁺ and CD8⁺ T-cells densities within the stroma region on WSIs of CRC and further predict survival.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12935-021-02297-w.

Predictive models of response to neoadjuvant chemotherapy in muscle-invasive bladder cancer using nuclear morphology and tissue architecture

Characterizing likelihood of response to neoadjuvant chemotherapy (NAC) in muscle-invasive bladder cancer (MIBC) is an important yet unmet challenge. In this study, a machine-learning framework is developed using imaging of biopsy pathology specimens to generate models of likelihood of NAC response. Developed using cross-validation (evaluable N = 66) and an independent validation cohort (evaluable N = 56), our models achieve promising results (65%-73% accuracy). Interestingly, one model-using features derived from hematoxylin and eosin (H&E)-stained tissues in conjunction with clinico-demographic features-is able to stratify the cohort into likely responders in cross-validation and the validation cohort (response rate of 65% for predicted responder compared with the 41% baseline response rate in the validation cohort). The results suggest that computational approaches applied to routine pathology specimens of MIBC can capture differences between responders and non-responders to NAC and should therefore be considered in the future design of precision oncology for MIBC.

Deep Learning of Histopathology Images at the Single Cell Level

The tumor immune microenvironment (TIME) encompasses many heterogeneous cell types that engage in extensive crosstalk among the cancer, immune, and stromal components. The spatial organization of these different cell types in TIME could be used as biomarkers for predicting drug responses, prognosis and metastasis. Recently, deep learning approaches have been widely used for digital histopathology images for cancer diagnoses and prognoses. Furthermore, some recent approaches have attempted to integrate spatial and molecular omics data to better characterize the TIME. In this review we focus on machine learning-based digital histopathology image analysis methods for characterizing tumor ecosystem. In this review, we will consider three different scales of histopathological analyses that machine learning can operate within: whole slide image (WSI)-level, region of interest (ROI)-level, and cell-level. We will systematically review the various machine learning methods in these three scales with a focus on cell-level analysis. We will provide a perspective of workflow on generating cell-level training data sets using immunohistochemistry markers to "weakly-label" the cell types. We will describe some common steps in the workflow of preparing the data, as well as some limitations of this approach. Finally, we will discuss future opportunities of integrating molecular omics data with digital histopathology images for characterizing tumor ecosystem.

Machine Learning and Artificial Intelligence-driven Spatial Analysis of the Tumor Immune Microenvironment in Pathology Slides

A better understanding of the tumor immune microenvironment (TIME) could lead to accurate diagnosis, prognosis, and treatment stratification. Although molecular analyses at the tissue and/or single cell level could reveal the cellular status of the tumor microenvironment, these approaches lack information related to spatial-level cellular distribution, co-organization, and cell-cell interaction in the TIME. With the emergence of computational pathology coupled with machine learning (ML) and artificial intelligence (AI), ML- and AI-driven spatial TIME analyses of pathology images could revolutionize our understanding of the highly heterogeneous and complex molecular architecture of the TIME. In this review we highlight recent studies on spatial TIME analysis of pathology slides using state-of-the-art ML and AI algorithms. PATIENT SUMMARY: This mini-review reports recent advances in machine learning and artificial intelligence for spatial analysis of the tumor immune microenvironment in pathology slides. This information can help in understanding the spatial heterogeneity and organization of cells in patient tumors.

Hot or cold: Bioengineering immune contextures into in vitro patient-derived tumor models

In the past decade, immune checkpoint inhibitors (ICI) have proven to be tremendously effective for a subset of cancer patients. However, it is difficult to predict the response of individual patients and efforts are now directed at understanding the mechanisms of ICI resistance. Current models of patient tumors poorly recapitulate the immune contexture, which describe immune parameters that are associated with patient survival. In this Review, we discuss parameters that influence the induction of different immune contextures found within tumors and how engineering strategies may be leveraged to recapitulate these contextures to develop the next generation of immune-competent patient-derived in vitro models.

Artificial Intelligence for solid tumor diagnosis in digital pathology

Tumor diagnosis relies on the visual examination of histological slides by pathologists through a microscope eyepiece. Digital pathology, the digitalization of histological slides at high magnification with slides scanners, has raised the opportunity to extract quantitative information thanks to image analysis. In the last decade, medical image analysis has made exceptional progress due to the development of artificial intelligence (AI) algorithms. AI has been successfully used in the field of medical imaging and more recently in digital pathology. The feasibility and usefulness of AI assisted pathology tasks have been demonstrated in the very last years and we can expect those developments to be applied on routine histopathology in the future. In this review, we will describe and illustrate this technique and present the most recent applications in the field of tumor histopathology.

Combining texture features of whole slide images improves prognostic prediction of recurrence-free survival for cutaneous melanoma patients

BACKGROUND

Accurate prediction of recurrence-free survival (RFS) is important for the prognosis of cutaneous melanoma patients. The image-based pathological examination remains as the gold standard for diagnosis. It is of clinical interest to account for computer-aided processing of pathology image when performing prognostic analysis.

METHODS

We enrolled in this study a total of 152 patients from TCGA-SKCM (The Cancer Genome Atlas Skin Cutaneous Melanoma project) with complete information in recurrence-related survival time, baseline variables (clinicopathologic variables, mutation status of BRAF and NRAS genes), gene expression data, and whole slide image (WSI) features. We preprocessed WSI to segment global or nucleus areas, and extracted 3 types of texture features from each region. We performed cross validation and used multiple evaluation metrics including C-index and time-dependent AUC to determine the best model of predicting recurrence events. We further performed differential gene expression analysis between the higher and lower-risk groups within AJCC pathologic tumor stage III patients to explore the underlying molecular mechanisms driving risk stratification.

RESULTS

The model combining baseline variables and WSI features had the best performance among models with any other types of data integration. The prognostic risk score generated by this model could provide a higher-resolution risk stratification within pathologically defined subgroups. We found the selected image features captured important immune-related variations, such as the aberration of expression in T cell activation and proliferation gene sets, and therefore contributed to the improved prediction.

CONCLUSIONS

Our study provided a prognostic model based on the combination of baseline variables and computer-processed WSI features. This model provided more accurate prediction than models based on other types of data combination in recurrence-free survival analysis.

TRIAL REGISTRATION

This study was based on public open data from TCGA and hence the study objects were retrospectively registered.