Utilizing Automated Breast Cancer Detection to Identify Spatial Distributions of Tumor-Infiltrating Lymphocytes in Invasive Breast Cancer

Publicly available datasets of breast histopathology H&E whole-slide images: A scoping review

Advancements in digital pathology and computing resources have made a significant impact in the field of computational pathology for breast cancer diagnosis and treatment. However, access to high-quality labeled histopathological images of breast cancer is a big challenge that limits the development of accurate and robust deep learning models. In this scoping review, we identified the publicly available datasets of breast H&E-stained whole-slide images (WSIs) that can be used to develop deep learning algorithms. We systematically searched 9 scientific literature databases and 9 research data repositories and found 17 publicly available datasets containing 10 385 H&E WSIs of breast cancer. Moreover, we reported image metadata and characteristics for each dataset to assist researchers in selecting proper datasets for specific tasks in breast cancer computational pathology. In addition, we compiled 2 lists of breast H&E patches and private datasets as supplementary resources for researchers. Notably, only 28% of the included articles utilized multiple datasets, and only 14% used an external validation set, suggesting that the performance of other developed models may be susceptible to overestimation. The TCGA-BRCA was used in 52% of the selected studies. This dataset has a considerable selection bias that can impact the robustness and generalizability of the trained algorithms. There is also a lack of consistent metadata reporting of breast WSI datasets that can be an issue in developing accurate deep learning models, indicating the necessity of establishing explicit guidelines for documenting breast WSI dataset characteristics and metadata.

Artificial intelligence: illuminating the depths of the tumor microenvironment

Artificial intelligence (AI) can acquire characteristics that are not yet known to humans through extensive learning, enabling to handle large amounts of pathology image data. Divided into machine learning and deep learning, AI has the advantage of handling large amounts of data and processing image analysis, consequently it also has a great potential in accurately assessing tumour microenvironment (TME) models. With the complex composition of the TME, in-depth study of TME contributes to new ideas for treatment, assessment of patient response to postoperative therapy and prognostic prediction. This leads to a review of the development of AI's application in TME assessment in this study, provides an overview of AI techniques applied to medicine, delves into the application of AI in analysing the quantitative and spatial location characteristics of various cells (tumour cells, immune and non-immune cells) in the TME, reveals the predictive prognostic value of TME and provides new ideas for tumour therapy, highlights the great potential for clinical applications. In addition, a discussion of its limitations and encouraging future directions for its practical clinical application is presented.

Tumor-Infiltrating Lymphocyte Scoring in Neoadjuvant-Treated Breast Cancer

Neoadjuvant chemotherapy (NAC) is now the standard of care for patients with locally advanced breast cancer (BC). TIL scoring is prognostic and adds predictive value to the residual cancer burden evaluation after NAC. However, NAC induces changes in the tumor, and the reliability of TIL scoring in post-NAC samples has not yet been studied. H&E- and dual CD3/CD20 chromogenic IHC-stained tissues were scored for stromal and intra-tumoral TIL by two experienced pathologists on pre- and post-treatment BC tissues. Digital TIL scoring was performed using the HALO® image analysis software (version 2.2). In patients with residual disease, we show a good inter-pathologist correlation for stromal TIL on H&E-stained tissues (CCC value 0.73). A good correlation for scoring with both staining methods (CCC 0.81) and the digital TIL scoring (CCC 0.77) was also observed. Overall concordance for TIL scoring in patients with a complete response was however poor. This study reveals there is good reliability for TIL scoring in patients with detectable residual tumors after NAC treatment, which is comparable to the scoring of untreated breast cancer patients. Based on the good consistency observed with digital TIL scoring, the development of a validated algorithm in the future might be advantageous.

CXCR6 expression correlates with radiotherapy response and immune context in triple-negative breast cancer-experimental studies

BACKGROUND

The chemokine receptor CXCR6 is critical for sustained tumor control mediated by CD8+ cytotoxic T cells (CTLs) in tumors. Previous studies have shown that ionizing radiation induces an inflamed immune contexture by upregulating CXCR6. However, the clinical significance of CXCR6 expression in triple-negative breast cancer (TNBC) and its correlation with radiotherapy remains unknown. This study aimed to clarify the prognostic value of CXCR6 and its role in the breast tumor microenvironment (TME).

METHODS

The messenger RNA and protein expression of CXCR6 in human TNBC and their association with survival were analyzed. The role of CXCR6 in the immune context was investigated using a combination of single-cell RNA sequencing, bulk transcriptome sequencing data, and fluorescence-based multiplex immunohistochemistry (mIHC) techniques.

RESULTS

Elevated CXCR6 expression correlated with better clinical outcomes and superior response to adjuvant radiotherapy and immunotherapy in TNBC. CXCR6 fostered an immunostimulatory microenvironment characterized by upregulated cytotoxic markers. We also found that CXCR6 plays a crucial role in regulating the differentiation of CD8+ T cells and the intercellular communication of immune cell subtypes, thus shaping the TME.

CONCLUSIONS

This study highlights the emerging role of CXCR6 in shaping the TME and targeting CXCR6 may be a promising strategy for improving the effectiveness of radiotherapy and immunotherapy in TNBC.

Enhancing AI Research for Breast Cancer: A Comprehensive Review of Tumor-Infiltrating Lymphocyte Datasets

The field of immunology is fundamental to our understanding of the intricate dynamics of the tumor microenvironment. In particular, tumor-infiltrating lymphocyte (TIL) assessment emerges as essential aspect in breast cancer cases. To gain comprehensive insights, the quantification of TILs through computer-assisted pathology (CAP) tools has become a prominent approach, employing advanced artificial intelligence models based on deep learning techniques. The successful recognition of TILs requires the models to be trained, a process that demands access to annotated datasets. Unfortunately, this task is hampered not only by the scarcity of such datasets, but also by the time-consuming nature of the annotation phase required to create them. Our review endeavors to examine publicly accessible datasets pertaining to the TIL domain and thereby become a valuable resource for the TIL community. The overall aim of the present review is thus to make it easier to train and validate current and upcoming CAP tools for TIL assessment by inspecting and evaluating existing publicly available online datasets.

Learning from crowds for automated histopathological image segmentation

Automated semantic segmentation of histopathological images is an essential task in Computational Pathology (CPATH). The main limitation of Deep Learning (DL) to address this task is the scarcity of expert annotations. Crowdsourcing (CR) has emerged as a promising solution to reduce the individual (expert) annotation cost by distributing the labeling effort among a group of (non-expert) annotators. Extracting knowledge in this scenario is challenging, as it involves noisy annotations. Jointly learning the underlying (expert) segmentation and the annotators' expertise is currently a commonly used approach. Unfortunately, this approach is frequently carried out by learning a different neural network for each annotator, which scales poorly when the number of annotators grows. For this reason, this strategy cannot be easily applied to real-world CPATH segmentation. This paper proposes a new family of methods for CR segmentation of histopathological images. Our approach consists of two coupled networks: a segmentation network (for learning the expert segmentation) and an annotator network (for learning the annotators' expertise). We propose to estimate the annotators' behavior with only one network that receives the annotator ID as input, achieving scalability on the number of annotators. Our family is composed of three different models for the annotator network. Within this family, we propose a novel modeling of the annotator network in the CR segmentation literature, which considers the global features of the image. We validate our methods on a real-world dataset of Triple Negative Breast Cancer images labeled by several medical students. Our new CR modeling achieves a Dice coefficient of 0.7827, outperforming the well-known STAPLE (0.7039) and being competitive with the supervised method with expert labels (0.7723). The code is available at https://github.com/wizmik12/CRowd_Seg.

A systematic review on deep learning-based automated cancer diagnosis models

Deep learning is gaining importance due to its wide range of applications. Many researchers have utilized deep learning (DL) models for the automated diagnosis of cancer patients. This paper provides a systematic review of DL models for automated diagnosis of cancer patients. Initially, various DL models for cancer diagnosis are presented. Five major categories of cancers such as breast, lung, liver, brain and cervical cancer are considered. As these categories of cancers have a very high percentage of occurrences with high mortality rate. The comparative analysis of different types of DL models is drawn for the diagnosis of cancer at early stages by considering the latest research articles from 2016 to 2022. After comprehensive comparative analysis, it is found that most of the researchers achieved appreciable accuracy with implementation of the convolutional neural network model. These utilized the pretrained models for automated diagnosis of cancer patients. Various shortcomings with the existing DL-based automated cancer diagnosis models are also been presented. Finally, future directions are discussed to facilitate further research for automated diagnosis of cancer patients.

An Intelligent Search & Retrieval System (IRIS) and Clinical and Research Repository for Decision Support Based on Machine Learning and Joint Kernel-based Supervised Hashing

Large-scale, multi-site collaboration is becoming indispensable for a wide range of research and clinical activities in oncology. To facilitate the next generation of advances in cancer biology, precision oncology and the population sciences it will be necessary to develop and implement data management and analytic tools that empower investigators to reliably and objectively detect, characterize and chronicle the phenotypic and genomic changes that occur during the transformation from the benign to cancerous state and throughout the course of disease progression. To facilitate these efforts it is incumbent upon the informatics community to establish the workflows and architectures that automate the aggregation and organization of a growing range and number of clinical data types and modalities ranging from new molecular and laboratory tests to sophisticated diagnostic imaging studies. In an attempt to meet those challenges, leading health care centers across the country are making steep investments to establish enterprise-wide, data warehouses. A significant limitation of many data warehouses, however, is that they are designed to support only alphanumeric information. In contrast to those traditional designs, the system that we have developed supports automated collection and mining of multimodal data including genomics, digital pathology and radiology images. In this paper, our team describes the design, development and implementation of a multi-modal, Clinical & Research Data Warehouse (CRDW) that is tightly integrated with a suite of computational and machine-learning tools to provide actionable insight into the underlying characteristics of the tumor environment that would not be revealed using standard methods and tools. The System features a flexible Extract, Transform and Load (ETL) interface that enables it to adapt to aggregate data originating from different clinical and research sources depending on the specific EHR and other data sources utilized at a given deployment site.

Open and reusable deep learning for pathology with WSInfer and QuPath

Digital pathology has seen a proliferation of deep learning models in recent years, but many models are not readily reusable. To address this challenge, we developed WSInfer: an open-source software ecosystem designed to streamline the sharing and reuse of deep learning models for digital pathology. The increased access to trained models can augment research on the diagnostic, prognostic, and predictive capabilities of digital pathology.

PathLDM: Text conditioned Latent Diffusion Model for Histopathology

To achieve high-quality results, diffusion models must be trained on large datasets. This can be notably prohibitive for models in specialized domains, such as computational pathology. Conditioning on labeled data is known to help in data-efficient model training. Therefore, histopathology reports, which are rich in valuable clinical information, are an ideal choice as guidance for a histopathology generative model. In this paper, we introduce PathLDM, the first text-conditioned Latent Diffusion Model tailored for generating high-quality histopathology images. Leveraging the rich contextual information provided by pathology text reports, our approach fuses image and textual data to enhance the generation process. By utilizing GPT's capabilities to distill and summarize complex text reports, we establish an effective conditioning mechanism. Through strategic conditioning and necessary architectural enhancements, we achieved a SoTA FID score of 7.64 for text-to-image generation on the TCGA-BRCA dataset, significantly outperforming the closest text-conditioned competitor with FID 30.1.

Advances in Early Breast Cancer Risk Profiling: From Histopathology to Molecular Technologies

Early breast cancer (BC) is the definition applied to breast-confined tumors with or without limited involvement of locoregional lymph nodes. While risk stratification is essential for guiding clinical decisions, it can be a complex endeavor in these patients due to the absence of comprehensive guidelines. Histopathological analysis and biomarker assessment play a pivotal role in defining patient outcomes. Traditional histological criteria such as tumor size, lymph node involvement, histological type and grade, lymphovascular invasion, and immune cell infiltration are significant prognostic indicators. In addition to the hormone receptor, HER2, and-in specific scenarios-BRCA1/2 testing, molecular subtyping through gene expression profiling provides valuable insights to tailor clinical decision-making. The emergence of "omics" technologies, applicable to both tissue and liquid biopsy samples, has broadened our arsenal for evaluating the risk of early BC. However, a pressing need remains for standardized methodologies and integrated pathological models that encompass multiple analytical dimensions. In this study, we provide a detailed examination of the existing strategies for early BC risk stratification, intending to serve as a practical guide for histopathologists and molecular pathologists.

PROACTING: predicting pathological complete response to neoadjuvant chemotherapy in breast cancer from routine diagnostic histopathology biopsies with deep learning

BACKGROUND

Invasive breast cancer patients are increasingly being treated with neoadjuvant chemotherapy; however, only a fraction of the patients respond to it completely. To prevent overtreatment, there is an urgent need for biomarkers to predict treatment response before administering the therapy.

METHODS

In this retrospective study, we developed hypothesis-driven interpretable biomarkers based on deep learning, to predict the pathological complete response (pCR, i.e., the absence of tumor cells in the surgical resection specimens) to neoadjuvant chemotherapy solely using digital pathology H&E images of pre-treatment breast biopsies. Our approach consists of two steps: First, we use deep learning to characterize aspects of the tumor micro-environment by detecting mitoses and segmenting tissue into several morphology compartments including tumor, lymphocytes and stroma. Second, we derive computational biomarkers from the segmentation and detection output to encode slide-level relationships of components of the tumor microenvironment, such as tumor and mitoses, stroma, and tumor infiltrating lymphocytes (TILs).

RESULTS

We developed and evaluated our method on slides from n = 721 patients from three European medical centers with triple-negative and Luminal B breast cancers and performed external independent validation on n = 126 patients from a public dataset. We report the predictive value of the investigated biomarkers for predicting pCR with areas under the receiver operating characteristic curve between 0.66 and 0.88 across the tested cohorts.

CONCLUSION

The proposed computational biomarkers predict pCR, but will require more evaluation and finetuning for clinical application. Our results further corroborate the potential role of deep learning to automate TILs quantification, and their predictive value in breast cancer neoadjuvant treatment planning, along with automated mitoses quantification. We made our method publicly available to extract segmentation-based biomarkers for research purposes.

Evaluation of tumour infiltrating lymphocytes in luminal breast cancer using artificial intelligence

BACKGROUND

Tumour infiltrating lymphocytes (TILs) are a prognostic parameter in triple-negative and human epidermal growth factor receptor 2 (HER2)-positive breast cancer (BC). However, their role in luminal (oestrogen receptor positive and HER2 negative (ER + /HER2-)) BC remains unclear. In this study, we used artificial intelligence (AI) to assess the prognostic significance of TILs in a large well-characterised cohort of luminal BC.

METHODS

Supervised deep learning model analysis of Haematoxylin and Eosin (H&E)-stained whole slide images (WSI) was applied to a cohort of 2231 luminal early-stage BC patients with long-term follow-up. Stromal TILs (sTILs) and intratumoural TILs (tTILs) were quantified and their spatial distribution within tumour tissue, as well as the proportion of stroma involved by sTILs were assessed. The association of TILs with clinicopathological parameters and patient outcome was determined.

RESULTS

A strong positive linear correlation was observed between sTILs and tTILs. High sTILs and tTILs counts, as well as their proximity to stromal and tumour cells (co-occurrence) were associated with poor clinical outcomes and unfavourable clinicopathological parameters including high tumour grade, lymph node metastasis, large tumour size, and young age. AI-based assessment of the proportion of stroma composed of sTILs (as assessed visually in routine practice) was not predictive of patient outcome. tTILs was an independent predictor of worse patient outcome in multivariate Cox Regression analysis.

CONCLUSION

AI-based detection of TILs counts, and their spatial distribution provides prognostic value in luminal early-stage BC patients. The utilisation of AI algorithms could provide a comprehensive assessment of TILs as a morphological variable in WSIs beyond eyeballing assessment.

Enhancing head and neck tumor management with artificial intelligence: Integration and perspectives

Head and neck tumors (HNTs) constitute a multifaceted ensemble of pathologies that primarily involve regions such as the oral cavity, pharynx, and nasal cavity. The intricate anatomical structure of these regions poses considerable challenges to efficacious treatment strategies. Despite the availability of myriad treatment modalities, the overall therapeutic efficacy for HNTs continues to remain subdued. In recent years, the deployment of artificial intelligence (AI) in healthcare practices has garnered noteworthy attention. AI modalities, inclusive of machine learning (ML), neural networks (NNs), and deep learning (DL), when amalgamated into the holistic management of HNTs, promise to augment the precision, safety, and efficacy of treatment regimens. The integration of AI within HNT management is intricately intertwined with domains such as medical imaging, bioinformatics, and medical robotics. This article intends to scrutinize the cutting-edge advancements and prospective applications of AI in the realm of HNTs, elucidating AI's indispensable role in prevention, diagnosis, treatment, prognostication, research, and inter-sectoral integration. The overarching objective is to stimulate scholarly discourse and invigorate insights among medical practitioners and researchers to propel further exploration, thereby facilitating superior therapeutic alternatives for patients.

Computational pathology to improve biomarker testing in breast cancer: how close are we?

The recent advancements in breast cancer precision medicine have highlighted the urgency for the precise and reproducible characterization of clinically actionable biomarkers. Despite numerous standardization efforts, biomarker testing by conventional methodologies is challenged by several issues such as high inter-observer variabilities, the spatial heterogeneity of biomarkers expression, and technological heterogeneity. In this respect, artificial intelligence-based digital pathology approaches are being increasingly recognized as promising methods for biomarker testing and subsequently improved clinical management. Here, we provide an overview on the most recent advances for artificial intelligence-assisted biomarkers testing in breast cancer, with a particular focus on tumor-infiltrating lymphocytes, programmed death-ligand 1, phosphatidylinositol-3 kinase catalytic alpha, and estrogen receptor 1. Challenges and solutions for this integrative analysis in pathology laboratories are also provided.

Identification of lymph node metastasis in pre-operation cervical cancer patients by weakly supervised deep learning from histopathological whole-slide biopsy images

BACKGROUND

Lymph node metastasis (LNM) significantly impacts the prognosis of individuals diagnosed with cervical cancer, as it is closely linked to disease recurrence and mortality, thereby impacting therapeutic schedule choices for patients. However, accurately predicting LNM prior to treatment remains challenging. Consequently, this study seeks to utilize digital pathological features extracted from histopathological slides of primary cervical cancer patients to preoperatively predict the presence of LNM.

METHODS

A deep learning (DL) model was trained using the Vision transformer (ViT) and recurrent neural network (RNN) frameworks to predict LNM. This prediction was based on the analysis of 554 histopathological whole-slide images (WSIs) obtained from Qilu Hospital of Shandong University. To validate the model's performance, an external test was conducted using 336 WSIs from four other hospitals. Additionally, the efficiency of the DL model was evaluated using 190 cervical biopsies WSIs in a prospective set.

RESULTS

In the internal test set, our DL model achieved an area under the curve (AUC) of 0.919, with sensitivity and specificity values of 0.923 and 0.905, respectively, and an accuracy (ACC) of 0.909. The performance of the DL model remained strong in the external test set. In the prospective cohort, the AUC was 0.91, and the ACC was 0.895. Additionally, the DL model exhibited higher accuracy compared to imaging examination in the evaluation of LNM. By utilizing the transformer visualization method, we generated a heatmap that illustrates the local pathological features in primary lesions relevant to LNM.

CONCLUSION

DL-based image analysis has demonstrated efficiency in predicting LNM in early operable cervical cancer through the utilization of biopsies WSI. This approach has the potential to enhance therapeutic decision-making for patients diagnosed with cervical cancer.

ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification

BACKGROUND AND OBJECTIVE

Histopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers.

METHODS

Here we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model.

RESULTS

The main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision.

CONCLUSIONS

Choosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.

Machine learning in computational histopathology: Challenges and opportunities

Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.

Deep learning model improves tumor-infiltrating lymphocyte evaluation and therapeutic response prediction in breast cancer

Tumor-infiltrating lymphocytes (TILs) have been recognized as key players in the tumor microenvironment of breast cancer, but substantial interobserver variability among pathologists has impeded its utility as a biomarker. We developed a deep learning (DL)-based TIL analyzer to evaluate stromal TILs (sTILs) in breast cancer. Three pathologists evaluated 402 whole slide images of breast cancer and interpreted the sTIL scores. A standalone performance of the DL model was evaluated in the 210 cases (52.2%) exhibiting sTIL score differences of less than 10 percentage points, yielding a concordance correlation coefficient of 0.755 (95% confidence interval [CI], 0.693-0.805) in comparison to the pathologists' scores. For the 226 slides (56.2%) showing a 10 percentage points or greater variance between pathologists and the DL model, revisions were made. The number of discordant cases was reduced to 116 (28.9%) with the DL assistance (p < 0.001). The DL assistance also increased the concordance correlation coefficient of the sTIL score among every two pathologists. In triple-negative and human epidermal growth factor receptor 2 (HER2)-positive breast cancer patients who underwent the neoadjuvant chemotherapy, the DL-assisted revision notably accentuated higher sTIL scores in responders (26.8 ± 19.6 vs. 19.0 ± 16.4, p = 0.003). Furthermore, the DL-assistant revision disclosed the correlation of sTIL-high tumors (sTIL ≥ 50) with the chemotherapeutic response (odd ratio 1.28 [95% confidence interval, 1.01-1.63], p = 0.039). Through enhancing inter-pathologist concordance in sTIL interpretation and predicting neoadjuvant chemotherapy response, here we report the utility of the DL-based tool as a reference for sTIL scoring in breast cancer assessment.

Pitfalls in machine learning-based assessment of tumor-infiltrating lymphocytes in breast cancer: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor-immune interaction in breast cancer is now established, and tumor-infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple-negative (estrogen receptor, progesterone receptor, and HER2-negative) breast cancer and HER2-positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state-of-the-art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false-positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in-depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple-negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Immune Biomarkers in Triple-Negative Breast Cancer: Improving the Predictivity of Current Testing Methods

Triple-negative breast cancer (TNBC) poses a significant challenge in terms of prognosis and disease recurrence. The limited treatment options and the development of resistance to chemotherapy make it particularly difficult to manage these patients. However, recent research has been shifting its focus towards biomarker-based approaches for TNBC, with a particular emphasis on the tumor immune landscape. Immune biomarkers in TNBC are now a subject of great interest due to the presence of tumor-infiltrating lymphocytes (TILs) in these tumors. This characteristic often coincides with the presence of PD-L1 expression on both neoplastic cells and immune cells within the tumor microenvironment. Furthermore, a subset of TNBC harbor mismatch repair deficient (dMMR) TNBC, which is frequently accompanied by microsatellite instability (MSI). All of these immune biomarkers hold actionable potential for guiding patient selection in immunotherapy. To fully capitalize on these opportunities, the identification of additional or complementary biomarkers and the implementation of highly customized testing strategies are of paramount importance in TNBC. In this regard, this article aims to provide an overview of the current state of the art in immune-related biomarkers for TNBC. Specifically, it focuses on the various testing methodologies available and sheds light on the immediate future perspectives for patient selection. By delving into the advancements made in understanding the immune landscape of TNBC, this study aims to contribute to the growing body of knowledge in the field. The ultimate goal is to pave the way for the development of more personalized testing strategies, ultimately improving outcomes for TNBC patients.

Deep learning for histopathological segmentation of smooth muscle in the urinary bladder

BACKGROUND

Histological assessment of smooth muscle is a critical step particularly in staging malignant tumors in various internal organs including  the urinary bladder. Nonetheless, manual segmentation and classification of muscular tissues by pathologists is often challenging. Therefore, a fully automated and reliable smooth muscle image segmentation system is in high demand.

METHODS

To characterize muscle fibers in the urinary bladder, including muscularis mucosa (MM) and muscularis propria (MP), we assessed 277 histological images from surgical specimens, using two well-known deep learning (DL) model groups, one including VGG16, ResNet18, SqueezeNet, and MobileNetV2, considered as a patch-based approach, and the other including U-Net, MA-Net, DeepLabv3 + , and FPN, considered as a pixel-based approach. All the trained models in both the groups were evaluated at pixel-level for their performance.

RESULTS

For segmenting MP and non-MP (including MM) regions, MobileNetV2, in the patch-based approach and U-Net, in the pixel-based approach outperformed their peers in the groups with mean Jaccard Index equal to 0.74 and 0.79, and mean Dice co-efficient equal to 0.82 and 0.88, respectively. We also demonstrated the strengths and weaknesses of the models in terms of speed and prediction accuracy.

CONCLUSIONS

This work not only creates a benchmark for future development of tools for the histological segmentation of smooth muscle but also provides an effective DL-based diagnostic system for accurate pathological staging of bladder cancer.

A Deep Learning-Based System Trained for Gastrointestinal Stromal Tumor Screening Can Identify Multiple Types of Soft Tissue Tumors

The accuracy and timeliness of the pathologic diagnosis of soft tissue tumors (STTs) critically affect treatment decision and patient prognosis. Thus, it is crucial to make a preliminary judgement on whether the tumor is benign or malignant with hematoxylin and eosin-stained images. A deep learning-based system, Soft Tissue Tumor Box (STT-BOX), is presented herein, with only hematoxylin and eosin images for malignant STT identification from benign STTs with histopathologic similarity. STT-BOX assumed gastrointestinal stromal tumor as a baseline for malignant STT evaluation, and distinguished gastrointestinal stromal tumor from leiomyoma and schwannoma with 100% area under the curve in patients from three hospitals, which achieved higher accuracy than the interpretation of experienced pathologists. Particularly, this system performed well on six common types of malignant STTs from The Cancer Genome Atlas data set, accurately highlighting the malignant mass lesion. STT-BOX was able to distinguish ovarian malignant sex-cord stromal tumors without any fine-tuning. This study included mesenchymal tumors that originated from the digestive system, bone and soft tissues, and reproductive system, where the high accuracy of migration verification may reveal the morphologic similarity of the nine types of malignant tumors. Further evaluation in a pan-STT setting would be potential and prospective, obviating the overuse of immunohistochemistry and molecular tests, and providing a practical basis for clinical treatment selection in a timely manner.

Image analysis-based tumor infiltrating lymphocytes measurement predicts breast cancer pathologic complete response in SWOG S0800 neoadjuvant chemotherapy trial

We assessed the predictive value of an image analysis-based tumor-infiltrating lymphocytes (TILs) score for pathologic complete response (pCR) and event-free survival in breast cancer (BC). About 113 pretreatment samples were analyzed from patients with stage IIB-IIIC HER-2-negative BC randomized to neoadjuvant chemotherapy ± bevacizumab. TILs quantification was performed on full sections using QuPath open-source software with a convolutional neural network cell classifier (CNN11). We used easTILs% as a digital metric of TILs score defined as [sum of lymphocytes area (mm2)/stromal area(mm2)] × 100. Pathologist-read stromal TILs score (sTILs%) was determined following published guidelines. Mean pretreatment easTILs% was significantly higher in cases with pCR compared to residual disease (median 36.1 vs.14.8%, p < 0.001). We observed a strong positive correlation (r = 0.606, p < 0.0001) between easTILs% and sTILs%. The area under the prediction curve (AUC) was higher for easTILs% than sTILs%, 0.709 and 0.627, respectively. Image analysis-based TILs quantification is predictive of pCR in BC and had better response discrimination than pathologist-read sTILs%.

A pathomic approach for tumor-infiltrating lymphocytes classification on breast cancer digital pathology images

Background and objectives

The detection of tumor-infiltrating lymphocytes (TILs) could aid in the development of objective measures of the infiltration grade and can support decision-making in breast cancer (BC). However, manual quantification of TILs in BC histopathological whole slide images (WSI) is currently based on a visual assessment, thus resulting not standardized, not reproducible, and time-consuming for pathologists. In this work, a novel pathomic approach, aimed to apply high-throughput image feature extraction techniques to analyze the microscopic patterns in WSI, is proposed. In fact, pathomic features provide additional information concerning the underlying biological processes compared to the WSI visual interpretation, thus providing more easily interpretable and explainable results than the most frequently investigated Deep Learning based methods in the literature.

Methods

A dataset containing 1037 regions of interest with tissue compartments and TILs annotated on 195 TNBC and HER2+ BC hematoxylin and eosin (H&E)-stained WSI was used. After segmenting nuclei within tumor-associated stroma using a watershed-based approach, 71 pathomic features were extracted from each nucleus and reduced using a Spearman's correlation filter followed by a nonparametric Wilcoxon rank-sum test and least absolute shrinkage and selection operator. The relevant features were used to classify each candidate nucleus as either TILs or non-TILs using 5 multivariable machine learning classification models trained using 5-fold cross-validation (1) without resampling, (2) with the synthetic minority over-sampling technique and (3) with downsampling. The prediction performance of the models was assessed using ROC curves.

Results

21 features were selected, with most of them related to the well-known TILs properties of having regular shape, clearer margins, high peak intensity, more homogeneous enhancement and different textural pattern than other cells. The best performance was obtained by Random-Forest with ROC AUC of 0.86, regardless of resampling technique.

Conclusions

The presented approach holds promise for the classification of TILs in BC H&E-stained WSI and could provide support to pathologists for a reliable, rapid and interpretable clinical assessment of TILs in BC.

Predicting Response of Triple-Negative Breast Cancer to Neoadjuvant Chemotherapy Using a Deep Convolutional Neural Network-Based Artificial Intelligence Tool

PURPOSE

Achieving a pathological complete response (pCR) to neoadjuvant chemotherapy (NAC) is associated with improved patient outcomes in triple-negative breast cancer (TNBC). Currently, there are no validated predictive biomarkers for the response to NAC in TNBC. We developed and validated a deep convolutional neural network-based artificial intelligence (AI) model to predict the response of TNBC to NAC.

MATERIALS AND METHODS

Whole-slide images (WSIs) of hematoxylin and eosin-stained core biopsies from 165 (pCR in 60 and non-pCR in 105) and 78 (pCR in 31 and non-pCR in 47) patients with TNBC were used to train and validate the model. The model extracts morphometric features from WSIs in an unsupervised manner, thereby generating clusters of morphologically similar patterns. Downstream ranking of clusters provided regions of interest and morphometric scores; a low score close to zero and a high score close to one represented a high or low probability of response to NAC.

RESULTS

The predictive ability of AI score for the entire cohort of 78 patients with TNBC ascertained by receiver operating characteristic analysis demonstrated an area under the curve (AUC) of 0.75. The AUC for stages I, II, and III disease were 0.88, 0.73, and 0.74, respectively. Using a cutoff value of 0.35, the positive predictive value of the AI score for pCR was 73.7%, and the negative predictive value was 76.2% for non-pCR patients.

CONCLUSION

To our knowledge, this study is the first to demonstrate the use of an AI tool on digitized hematoxylin and eosin-stained tissue images to predict the response to NAC in patients with TNBC with high accuracy. If validated in subsequent studies, these results may serve as an ancillary aid for individualized therapeutic decisions in patients with TNBC.

Attention-guided multi-scale deep object detection framework for lymphocyte analysis in IHC histological images

Tumor-infiltrating lymphocytes are specialized lymphocytes that can detect and kill cancerous cells. Their detection poses many challenges due to significant morphological variations, overlapping occurrence, artifact regions and high-class resemblance between clustered areas and artifacts. In this regard, a Lymphocyte Analysis Framework based on Deep Convolutional neural network (DC-Lym-AF) is proposed to analyze lymphocytes in immunohistochemistry images. The proposed framework comprises (i) pre-processing, (ii) screening phase, (iii) localization phase and (iv) post-processing. In the screening phase, a custom convolutional neural network architecture (lymphocyte dilated network) is developed to screen lymphocytic regions by performing a patch-level classification. This proposed architecture uses dilated convolutions and shortcut connections to capture multi-level variations and ensure reference-based learning. In contrast, the localization phase utilizes an attention-guided multi-scale lymphocyte detector to detect lymphocytes. The proposed detector extracts refined and multi-scale features by exploiting dilated convolutions, attention mechanism and feature pyramid network (FPN) using its custom attention-aware backbone. The proposed DC-Lym-AF shows exemplary performance on the NuClick dataset compared with the existing detection models, with an F-score and precision of 0.84 and 0.83, respectively. We verified the generalizability of our proposed framework by participating in a publically open LYON'19 challenge. Results in terms of detection rate (0.76) and F-score (0.73) suggest that the proposed DC-Lym-AF can effectively detect lymphocytes in immunohistochemistry-stained images collected from different laboratories. In addition, its promising generalization on several datasets implies that it can be turned into a medical diagnostic tool to investigate various histopathological problems. Graphical Abstract.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

A semi-supervised multi-task learning framework for cancer classification with weak annotation in whole-slide images

Cancer region detection (CRD) and subtyping are two fundamental tasks in digital pathology image analysis. The development of data-driven models for CRD and subtyping on whole-slide images (WSIs) would mitigate the burden of pathologists and improve their accuracy in diagnosis. However, the existing models are facing two major limitations. Firstly, they typically require large-scale datasets with precise annotations, which contradicts with the original intention of reducing labor effort. Secondly, for the subtyping task, the non-cancerous regions are treated as the same as cancerous regions within a WSI, which confuses a subtyping model in its training process. To tackle the latter limitation, the previous research proposed to perform CRD first for ruling out the non-cancerous region, then train a subtyping model based on the remaining cancerous patches. However, separately training ignores the interaction of these two tasks, also leads to propagating the error of the CRD task to the subtyping task. To address these issues and concurrently improve the performance on both CRD and subtyping tasks, we propose a semi-supervised multi-task learning (MTL) framework for cancer classification. Our framework consists of a backbone feature extractor, two task-specific classifiers, and a weight control mechanism. The backbone feature extractor is shared by two task-specific classifiers, such that the interaction of CRD and subtyping tasks can be captured. The weight control mechanism preserves the sequential relationship of these two tasks and guarantees the error back-propagation from the subtyping task to the CRD task under the MTL framework. We train the overall framework in a semi-supervised setting, where datasets only involve small quantities of annotations produced by our minimal point-based (min-point) annotation strategy. Extensive experiments on four large datasets with different cancer types demonstrate the effectiveness of the proposed framework in both accuracy and generalization.

Artificial intelligence-based multi-omics analysis fuels cancer precision medicine

With biotechnological advancements, innovative omics technologies are constantly emerging that have enabled researchers to access multi-layer information from the genome, epigenome, transcriptome, proteome, metabolome, and more. A wealth of omics technologies, including bulk and single-cell omics approaches, have empowered to characterize different molecular layers at unprecedented scale and resolution, providing a holistic view of tumor behavior. Multi-omics analysis allows systematic interrogation of various molecular information at each biological layer while posing tricky challenges regarding how to extract valuable insights from the exponentially increasing amount of multi-omics data. Therefore, efficient algorithms are needed to reduce the dimensionality of the data while simultaneously dissecting the mysteries behind the complex biological processes of cancer. Artificial intelligence has demonstrated the ability to analyze complementary multi-modal data streams within the oncology realm. The coincident development of multi-omics technologies and artificial intelligence algorithms has fuelled the development of cancer precision medicine. Here, we present state-of-the-art omics technologies and outline a roadmap of multi-omics integration analysis using an artificial intelligence strategy. The advances made using artificial intelligence-based multi-omics approaches are described, especially concerning early cancer screening, diagnosis, response assessment, and prognosis prediction. Finally, we discuss the challenges faced in multi-omics analysis, along with tentative future trends in this field. With the increasing application of artificial intelligence in multi-omics analysis, we anticipate a shifting paradigm in precision medicine becoming driven by artificial intelligence-based multi-omics technologies.

H2G-Net: A multi-resolution refinement approach for segmentation of breast cancer region in gigapixel histopathological images

Over the past decades, histopathological cancer diagnostics has become more complex, and the increasing number of biopsies is a challenge for most pathology laboratories. Thus, development of automatic methods for evaluation of histopathological cancer sections would be of value. In this study, we used 624 whole slide images (WSIs) of breast cancer from a Norwegian cohort. We propose a cascaded convolutional neural network design, called H2G-Net, for segmentation of breast cancer region from gigapixel histopathological images. The design involves a detection stage using a patch-wise method, and a refinement stage using a convolutional autoencoder. To validate the design, we conducted an ablation study to assess the impact of selected components in the pipeline on tumor segmentation. Guiding segmentation, using hierarchical sampling and deep heatmap refinement, proved to be beneficial when segmenting the histopathological images. We found a significant improvement when using a refinement network for post-processing the generated tumor segmentation heatmaps. The overall best design achieved a Dice similarity coefficient of 0.933±0.069 on an independent test set of 90 WSIs. The design outperformed single-resolution approaches, such as cluster-guided, patch-wise high-resolution classification using MobileNetV2 (0.872±0.092) and a low-resolution U-Net (0.874±0.128). In addition, the design performed consistently on WSIs across all histological grades and segmentation on a representative × 400 WSI took ~ 58 s, using only the central processing unit. The findings demonstrate the potential of utilizing a refinement network to improve patch-wise predictions. The solution is efficient and does not require overlapping patch inference or ensembling. Furthermore, we showed that deep neural networks can be trained using a random sampling scheme that balances on multiple different labels simultaneously, without the need of storing patches on disk. Future work should involve more efficient patch generation and sampling, as well as improved clustering.

Proceedings of the fifth international Molecular Pathological Epidemiology (MPE) meeting

Cancer heterogeneities hold the key to a deeper understanding of cancer etiology and progression and the discovery of more precise cancer therapy. Modern pathological and molecular technologies offer a powerful set of tools to profile tumor heterogeneities at multiple levels in large patient populations, from DNA to RNA, protein and epigenetics, and from tumor tissues to tumor microenvironment and liquid biopsy. When coupled with well-validated epidemiologic methodology and well-characterized epidemiologic resources, the rich tumor pathological and molecular tumor information provide new research opportunities at an unprecedented breadth and depth. This is the research space where Molecular Pathological Epidemiology (MPE) emerged over a decade ago and has been thriving since then. As a truly multidisciplinary field, MPE embraces collaborations from diverse fields including epidemiology, pathology, immunology, genetics, biostatistics, bioinformatics, and data science. Since first convened in 2013, the International MPE Meeting series has grown into a dynamic and dedicated platform for experts from these disciplines to communicate novel findings, discuss new research opportunities and challenges, build professional networks, and educate the next-generation scientists. Herein, we share the proceedings of the Fifth International MPE meeting, held virtually online, on May 24 and 25, 2021. The meeting consisted of 21 presentations organized into the three main themes, which were recent integrative MPE studies, novel cancer profiling technologies, and new statistical and data science approaches. Looking forward to the near future, the meeting attendees anticipated continuous expansion and fruition of MPE research in many research fronts, particularly immune-epidemiology, mutational signatures, liquid biopsy, and health disparities.

Deep learning for survival analysis in breast cancer with whole slide image data

MOTIVATION

Whole slide tissue images contain detailed data on the sub-cellular structure of cancer. Quantitative analyses of this data can lead to novel biomarkers for better cancer diagnosis and prognosis and can improve our understanding of cancer mechanisms. Such analyses are challenging to execute because of the sizes and complexity of whole slide image data and relatively limited volume of training data for machine learning methods.

RESULTS

We propose and experimentally evaluate a multi-resolution deep learning method for breast cancer survival analysis. The proposed method integrates image data at multiple resolutions and tumor, lymphocyte and nuclear segmentation results from deep learning models. Our results show that this approach can significantly improve the deep learning model performance compared to using only the original image data. The proposed approach achieves a c-index value of 0.706 compared to a c-index value of 0.551 from an approach that uses only color image data at the highest image resolution. Furthermore, when clinical features (sex, age and cancer stage) are combined with image data, the proposed approach achieves a c-index of 0.773.

AVAILABILITY AND IMPLEMENTATION

https://github.com/SBU-BMI/deep_survival_analysis.

Open-Source Biomedical Image Analysis Models: A Meta-Analysis and Continuous Survey

Open-source research software has proven indispensable in modern biomedical image analysis. A multitude of open-source platforms drive image analysis pipelines and help disseminate novel analytical approaches and algorithms. Recent advances in machine learning allow for unprecedented improvement in these approaches. However, these novel algorithms come with new requirements in order to remain open source. To understand how these requirements are met, we have collected 50 biomedical image analysis models and performed a meta-analysis of their respective papers, source code, dataset, and trained model parameters. We concluded that while there are many positive trends in openness, only a fraction of all publications makes all necessary elements available to the research community.

Deep Learning-Based Pathology Image Analysis Enhances Magee Feature Correlation With Oncotype DX Breast Recurrence Score

Background

Oncotype DX Recurrence Score (RS) has been widely used to predict chemotherapy benefits in patients with estrogen receptor-positive breast cancer. Studies showed that the features used in Magee equations correlate with RS. We aimed to examine whether deep learning (DL)-based histology image analyses can enhance such correlations.

Methods

We retrieved 382 cases with RS diagnosed between 2011 and 2015 from the Emory University and the Ohio State University. All patients received surgery. DL models were developed to detect nuclei of tumor cells and tumor-infiltrating lymphocytes (TILs) and segment tumor cell nuclei in hematoxylin and eosin (H&E) stained histopathology whole slide images (WSIs). Based on the DL-based analysis, we derived image features from WSIs, such as tumor cell number, TIL number variance, and nuclear grades. The entire patient cohorts were divided into one training set (125 cases) and two validation sets (82 and 175 cases) based on the data sources and WSI resolutions. The training set was used to train the linear regression models to predict RS. For prediction performance comparison, we used independent variables from Magee features alone or the combination of WSI-derived image and Magee features.

Results

The Pearson's correlation coefficients between the actual RS and predicted RS by DL-based analysis were 0.7058 (p-value = 1.32 × 10-13) and 0.5041 (p-value = 1.15 × 10-12) for the validation sets 1 and 2, respectively. The adjusted R 2 values using Magee features alone are 0.3442 and 0.2167 in the two validation sets, respectively. In contrast, the adjusted R 2 values were enhanced to 0.4431 and 0.2182 when WSI-derived imaging features were jointly used with Magee features.

Conclusion

Our results suggest that DL-based digital pathological features can enhance Magee feature correlation with RS.

Building Efficient CNN Architectures for Histopathology Images Analysis: A Case-Study in Tumor-Infiltrating Lymphocytes Classification

Background

Deep learning methods have demonstrated remarkable performance in pathology image analysis, but they are computationally very demanding. The aim of our study is to reduce their computational cost to enable their use with large tissue image datasets.

Methods

We propose a method called Network Auto-Reduction (NAR) that simplifies a Convolutional Neural Network (CNN) by reducing the network to minimize the computational cost of doing a prediction. NAR performs a compound scaling in which the width, depth, and resolution dimensions of the network are reduced together to maintain a balance among them in the resulting simplified network. We compare our method with a state-of-the-art solution called ResRep. The evaluation is carried out with popular CNN architectures and a real-world application that identifies distributions of tumor-infiltrating lymphocytes in tissue images.

Results

The experimental results show that both ResRep and NAR are able to generate simplified, more efficient versions of ResNet50 V2. The simplified versions by ResRep and NAR require 1.32× and 3.26× fewer floating-point operations (FLOPs), respectively, than the original network without a loss in classification power as measured by the Area under the Curve (AUC) metric. When applied to a deeper and more computationally expensive network, Inception V4, NAR is able to generate a version that requires 4× lower than the original version with the same AUC performance.

Conclusions

NAR is able to achieve substantial reductions in the execution cost of two popular CNN architectures, while resulting in small or no loss in model accuracy. Such cost savings can significantly improve the use of deep learning methods in digital pathology. They can enable studies with larger tissue image datasets and facilitate the use of less expensive and more accessible graphics processing units (GPUs), thus reducing the computing costs of a study.

Spatial Characterization of Tumor-Infiltrating Lymphocytes and Breast Cancer Progression

Tumor-infiltrating lymphocytes (TILs) have been established as a robust prognostic biomarker in breast cancer, with emerging utility in predicting treatment response in the adjuvant and neoadjuvant settings. In this study, the role of TILs in predicting overall survival and progression-free interval was evaluated in two independent cohorts of breast cancer from the Cancer Genome Atlas (TCGA BRCA) and the Carolina Breast Cancer Study (UNC CBCS). We utilized machine learning and computer vision algorithms to characterize TIL infiltrates in digital whole-slide images (WSIs) of breast cancer stained with hematoxylin and eosin (H&E). Multiple parameters were used to characterize the global abundance and spatial features of TIL infiltrates. Univariate and multivariate analyses show that large aggregates of peritumoral and intratumoral TILs (forests) were associated with longer survival, whereas the absence of intratumoral TILs (deserts) is associated with increased risk of recurrence. Patients with two or more high-risk spatial features were associated with significantly shorter progression-free interval (PFI). This study demonstrates the practical utility of Pathomics in evaluating the clinical significance of the abundance and spatial patterns of distribution of TIL infiltrates as important biomarkers in breast cancer.

Accurate Prediction of Metachronous Liver Metastasis in Stage I-III Colorectal Cancer Patients Using Deep Learning With Digital Pathological Images

Objectives

Metachronous liver metastasis (LM) significantly impacts the prognosis of stage I-III colorectal cancer (CRC) patients. An effective biomarker to predict LM after surgery is urgently needed. We aimed to develop deep learning-based models to assist in predicting LM in stage I-III CRC patients using digital pathological images.

Methods

Six-hundred eleven patients were retrospectively included in the study and randomly divided into training (428 patients) and validation (183 patients) cohorts according to the 7:3 ratio. Digital HE images from training cohort patients were used to construct the LM risk score based on a 50-layer residual convolutional neural network (ResNet-50). An LM prediction model was established by multivariable Cox analysis and confirmed in the validation cohort. The performance of the integrated nomogram was assessed with respect to its calibration, discrimination, and clinical application value.

Results

Patients were divided into low- and high-LM risk score groups according to the cutoff value and significant differences were observed in the LM of the different risk score groups in the training and validation cohorts (P&lt;0.001). Multivariable analysis revealed that the LM risk score, VELIPI, pT stage and pN stage were independent predictors of LM. Then, the prediction model was developed and presented as a nomogram to predict the 1-, 2-, and 3-year probability of LM. The integrated nomogram achieved satisfactory discrimination, with C-indexes of 0.807 (95% CI: 0.787, 0.827) and 0.812 (95% CI: 0.773, 0.850) and AUCs of 0.840 (95% CI: 0.795, 0.885) and 0.848 (95% CI: 0.766, 0.931) in the training and validation cohorts, respectively. Favorable calibration of the nomogram was confirmed in the training and validation cohorts. Integrated discrimination improvement and net reclassification index indicated that the integrated nomogram was superior to the traditional clinicopathological model. Decision curve analysis confirmed that the nomogram has clinical application value.

Conclusions

The LM risk score based on ResNet-50 and digital HE images was significantly associated with LM. The integrated nomogram could identify stage I-III CRC patients at high risk of LM after primary colectomy, so it may serve as a potential tool to choose the appropriate treatment to improve the prognosis of stage I-III CRC patients.

Deep Learning-Based Mapping of Tumor Infiltrating Lymphocytes in Whole Slide Images of 23 Types of Cancer

The role of tumor infiltrating lymphocytes (TILs) as a biomarker to predict disease progression and clinical outcomes has generated tremendous interest in translational cancer research. We present an updated and enhanced deep learning workflow to classify 50x50 um tiled image patches (100x100 pixels at 20x magnification) as TIL positive or negative based on the presence of 2 or more TILs in gigapixel whole slide images (WSIs) from the Cancer Genome Atlas (TCGA). This workflow generates TIL maps to study the abundance and spatial distribution of TILs in 23 different types of cancer. We trained three state-of-the-art, popular convolutional neural network (CNN) architectures (namely VGG16, Inception-V4, and ResNet-34) with a large volume of training data, which combined manual annotations from pathologists (strong annotations) and computer-generated labels from our previously reported first-generation TIL model for 13 cancer types (model-generated annotations). Specifically, this training dataset contains TIL positive and negative patches from cancers in additional organ sites and curated data to help improve algorithmic performance by decreasing known false positives and false negatives. Our new TIL workflow also incorporates automated thresholding to convert model predictions into binary classifications to generate TIL maps. The new TIL models all achieve better performance with improvements of up to 13% in accuracy and 15% in F-score. We report these new TIL models and a curated dataset of TIL maps, referred to as TIL-Maps-23, for 7983 WSIs spanning 23 types of cancer with complex and diverse visual appearances, which will be publicly available along with the code to evaluate performance. Code Available at: https://github.com/ShahiraAbousamra/til_classification.

An Expandable Informatics Framework for Enhancing Central Cancer Registries with Digital Pathology Specimens, Computational Imaging Tools, and Advanced Mining Capabilities

BACKGROUND

Population-based state cancer registries are an authoritative source for cancer statistics in the United States. They routinely collect a variety of data, including patient demographics, primary tumor site, stage at diagnosis, first course of treatment, and survival, on every cancer case that is reported across all U.S. states and territories. The goal of our project is to enrich NCI's Surveillance, Epidemiology, and End Results (SEER) registry data with high-quality population-based biospecimen data in the form of digital pathology, machine-learning-based classifications, and quantitative histopathology imaging feature sets (referred to here as Pathomics features).

MATERIALS AND METHODS

As part of the project, the underlying informatics infrastructure was designed, tested, and implemented through close collaboration with several participating SEER registries to ensure consistency with registry processes, computational scalability, and ability to support creation of population cohorts that span multiple sites. Utilizing computational imaging algorithms and methods to both generate indices and search for matches makes it possible to reduce inter- and intra-observer inconsistencies and to improve the objectivity with which large image repositories are interrogated.

RESULTS

Our team has created and continues to expand a well-curated repository of high-quality digitized pathology images corresponding to subjects whose data are routinely collected by the collaborating registries. Our team has systematically deployed and tested key, visual analytic methods to facilitate automated creation of population cohorts for epidemiological studies and tools to support visualization of feature clusters and evaluation of whole-slide images. As part of these efforts, we are developing and optimizing advanced search and matching algorithms to facilitate automated, content-based retrieval of digitized specimens based on their underlying image features and staining characteristics.

CONCLUSION

To meet the challenges of this project, we established the analytic pipelines, methods, and workflows to support the expansion and management of a growing repository of high-quality digitized pathology and information-rich, population cohorts containing objective imaging and clinical attributes to facilitate studies that seek to discriminate among different subtypes of disease, stratify patient populations, and perform comparisons of tumor characteristics within and across patient cohorts. We have also successfully developed a suite of tools based on a deep-learning method to perform quantitative characterizations of tumor regions, assess infiltrating lymphocyte distributions, and generate objective nuclear feature measurements. As part of these efforts, our team has implemented reliable methods that enable investigators to systematically search through large repositories to automatically retrieve digitized pathology specimens and correlated clinical data based on their computational signatures.

A Comprehensive Survey on Deep-Learning-Based Breast Cancer Diagnosis

Breast cancer is now the most frequently diagnosed cancer in women, and its percentage is gradually increasing. Optimistically, there is a good chance of recovery from breast cancer if identified and treated at an early stage. Therefore, several researchers have established deep-learning-based automated methods for their efficiency and accuracy in predicting the growth of cancer cells utilizing medical imaging modalities. As of yet, few review studies on breast cancer diagnosis are available that summarize some existing studies. However, these studies were unable to address emerging architectures and modalities in breast cancer diagnosis. This review focuses on the evolving architectures of deep learning for breast cancer detection. In what follows, this survey presents existing deep-learning-based architectures, analyzes the strengths and limitations of the existing studies, examines the used datasets, and reviews image pre-processing techniques. Furthermore, a concrete review of diverse imaging modalities, performance metrics and results, challenges, and research directions for future researchers is presented.

Deep embeddings and logistic regression for rapid active learning in histopathological images

BACKGROUND AND OBJECTIVE

Recognizing different tissue components is one of the most fundamental and essential works in digital pathology. Current methods are often based on convolutional neural networks (CNNs), which need numerous annotated samples for training. Creating large-scale histopathological datasets is labor-intensive, where interactive data annotation is a potential solution.

METHODS

We propose DELR (Deep Embedding-based Logistic Regression) to enable rapid model training and inference for histopathological image analysis. DELR utilizes a pretrained CNN to encode images as compact embeddings with low computational cost. The embeddings are then used to train a Logistic Regression model efficiently. We implemented DELR in an active learning framework, and validated it on three histopathological problems (binary, 4-category, and 8-category classification challenge for lung, breast, and colorectal cancer, respectively). We also investigated the influence of active learning strategy and type of the encoder.

RESULTS

On all the three datasets, DELR can achieve an area under curve (AUC) metric higher than 0.95 with only 100 image patches per class. Although its AUC is slightly lower than a fine-tuned CNN counterpart, DELR can be 536, 316, and 1481 times faster after pre-encoding. Moreover, DELR is proved to be compatible with a variety of active learning strategies and encoders.

CONCLUSIONS

DELR can achieve comparable accuracy to CNN with rapid running speed. These advantages make it a potential solution for real-time interactive data annotation.

Deep Learning of Histopathology Images at the Single Cell Level

The tumor immune microenvironment (TIME) encompasses many heterogeneous cell types that engage in extensive crosstalk among the cancer, immune, and stromal components. The spatial organization of these different cell types in TIME could be used as biomarkers for predicting drug responses, prognosis and metastasis. Recently, deep learning approaches have been widely used for digital histopathology images for cancer diagnoses and prognoses. Furthermore, some recent approaches have attempted to integrate spatial and molecular omics data to better characterize the TIME. In this review we focus on machine learning-based digital histopathology image analysis methods for characterizing tumor ecosystem. In this review, we will consider three different scales of histopathological analyses that machine learning can operate within: whole slide image (WSI)-level, region of interest (ROI)-level, and cell-level. We will systematically review the various machine learning methods in these three scales with a focus on cell-level analysis. We will provide a perspective of workflow on generating cell-level training data sets using immunohistochemistry markers to "weakly-label" the cell types. We will describe some common steps in the workflow of preparing the data, as well as some limitations of this approach. Finally, we will discuss future opportunities of integrating molecular omics data with digital histopathology images for characterizing tumor ecosystem.

Intra-Tumour Heterogeneity Is One of the Main Sources of Inter-Observer Variation in Scoring Stromal Tumour Infiltrating Lymphocytes in Triple Negative Breast Cancer

Stromal tumour infiltrating lymphocytes (sTILs) are a strong prognostic marker in triple negative breast cancer (TNBC). Consistency scoring sTILs is good and was excellent when an internet-based scoring aid developed by the TIL-WG was used to score cases in a reproducibility study. This study aimed to evaluate the reproducibility of sTILs assessment using this scoring aid in cases from routine practice and to explore the potential of the tool to overcome variability in scoring. Twenty-three breast pathologists scored sTILs in digitized slides of 49 TNBC biopsies using the scoring aid. Subsequently, fields of view (FOV) from each case were selected by one pathologist and scored by the group using the tool. Inter-observer agreement was good for absolute sTILs (ICC 0.634, 95% CI 0.539-0.735, p < 0.001) but was poor to fair using binary cutpoints. sTILs heterogeneity was the main contributor to disagreement. When pathologists scored the same FOV from each case, inter-observer agreement was excellent for absolute sTILs (ICC 0.798, 95% CI 0.727-0.864, p < 0.001) and good for the 20% (ICC 0.657, 95% CI 0.561-0.756, p < 0.001) and 40% (ICC 0.644, 95% CI 0.546-0.745, p < 0.001) cutpoints. However, there was a wide range of scores for many cases. Reproducibility scoring sTILs is good when the scoring aid is used. Heterogeneity is the main contributor to variance and will need to be overcome for analytic validity to be achieved.

Correlation of manual semi-quantitative and automated quantitative Ki-67 proliferative index with OncotypeDXTM recurrence score in invasive breast carcinoma

BACKGROUND

Ki-67 immunohistochemistry (IHC) staining is a widely used cancer proliferation assay; however, its limitations could be improved with automated scoring. The OncotypeDXTM Recurrence Score (ORS), which primarily evaluates cancer proliferation genes, is a prognostic indicator for breast cancer chemotherapy response; however, it is more expensive and slower than Ki-67.

OBJECTIVE

To compare manual Ki-67 (mKi-67) with automated Ki-67 (aKi-67) algorithm results based on manually selected Ki-67 "hot spots" in breast cancer, and correlate both with ORS.

METHODS

105 invasive breast carcinoma cases from 100 patients at our institution (2011-2013) with available ORS were evaluated. Concordance was assessed via Cohen's Kappa (κ).

RESULTS

57/105 cases showed agreement between mKi-67 and aKi-67 (κ 0.31, 95% CI 0.18-0.45), with 41 cases overestimated by aKi-67. Concordance was higher when estimated on the same image (κ 0.53, 95% CI 0.37-0.69). Concordance between mKi-67 score and ORS was fair (κ 0.27, 95% CI 0.11-0.42), and concordance between aKi-67 and ORS was poor (κ 0.10, 95% CI -0.03-0.23).

CONCLUSIONS

These results highlight the limits of Ki-67 algorithms that use manual "hot spot" selection. Due to suboptimal concordance, Ki-67 is likely most useful as a complement to, rather than a surrogate for ORS, regardless of scoring method.

A Computational Tumor-Infiltrating Lymphocyte Assessment Method Comparable with Visual Reporting Guidelines for Triple-Negative Breast Cancer

Background

Tumor-infiltrating lymphocytes (TILs) are clinically significant in triple-negative breast cancer (TNBC). Although a standardized methodology for visual TILs assessment (VTA) exists, it has several inherent limitations. We established a deep learning-based computational TIL assessment (CTA) method broadly following VTA guideline and compared it with VTA for TNBC to determine the prognostic value of the CTA and a reasonable CTA workflow for clinical practice.

Methods

We trained three deep neural networks for nuclei segmentation, nuclei classification and necrosis classification to establish a CTA workflow. The automatic TIL (aTIL) score generated was compared with manual TIL (mTIL) scores provided by three pathologists in an Asian (n = 184) and a Caucasian (n = 117) TNBC cohort to evaluate scoring concordance and prognostic value.

Findings

The intraclass correlations (ICCs) between aTILs and mTILs varied from 0.40 to 0.70 in two cohorts. Multivariate Cox proportional hazards analysis revealed that the aTIL score was associated with disease free survival (DFS) in both cohorts, as either a continuous [hazard ratio (HR)=0.96, 95% CI 0.94–0.99] or dichotomous variable (HR=0.29, 95% CI 0.12–0.72). A higher C-index was observed in a composite mTIL/aTIL three-tier stratification model than in the dichotomous model, using either mTILs or aTILs alone.

Interpretation

The current study provides a useful tool for stromal TIL assessment and prognosis evaluation for patients with TNBC. A workflow integrating both VTA and CTA may aid pathologists in performing risk management and decision-making tasks.

An automated computational image analysis pipeline for histological grading of cardiac allograft rejection

AIM

Allograft rejection is a serious concern in heart transplant medicine. Though endomyocardial biopsy with histological grading is the diagnostic standard for rejection, poor inter-pathologist agreement creates significant clinical uncertainty. The aim of this investigation is to demonstrate that cellular rejection grades generated via computational histological analysis are on-par with those provided by expert pathologists.

METHODS AND RESULTS

The study cohort consisted of 2472 endomyocardial biopsy slides originating from three major US transplant centres. The 'Computer-Assisted Cardiac Histologic Evaluation (CACHE)-Grader' pipeline was trained using an interpretable, biologically inspired, 'hand-crafted' feature extraction approach. From a menu of 154 quantitative histological features relating the density and orientation of lymphocytes, myocytes, and stroma, a model was developed to reproduce the 4-grade clinical standard for cellular rejection diagnosis. CACHE-grader interpretations were compared with independent pathologists and the 'grade of record', testing for non-inferiority (δ = 6%). Study pathologists achieved a 60.7% agreement [95% confidence interval (CI): 55.2-66.0%] with the grade of record, and pair-wise agreement among all human graders was 61.5% (95% CI: 57.0-65.8%). The CACHE-Grader met the threshold for non-inferiority, achieving a 65.9% agreement (95% CI: 63.4-68.3%) with the grade of record and a 62.6% agreement (95% CI: 60.3-64.8%) with all human graders. The CACHE-Grader demonstrated nearly identical performance in internal and external validation sets (66.1% vs. 65.8%), resilience to inter-centre variations in tissue processing/digitization, and superior sensitivity for high-grade rejection (74.4% vs. 39.5%, P < 0.001).

CONCLUSION

These results show that the CACHE-grader pipeline, derived using intuitive morphological features, can provide expert-quality rejection grading, performing within the range of inter-grader variability seen among human pathologists.

Automated Quantification of sTIL Density with H&E-Based Digital Image Analysis Has Prognostic Potential in Triple-Negative Breast Cancers

Triple-negative breast cancer (TNBC) is an aggressive and difficult-to-treat cancer type that represents approximately 15% of all breast cancers. Recently, stromal tumor-infiltrating lymphocytes (sTIL) resurfaced as a strong prognostic biomarker for overall survival (OS) for TNBC patients. Manual assessment has innate limitations that hinder clinical adoption, and the International Immuno-Oncology Biomarker Working Group (TIL-WG) has therefore envisioned that computational assessment of sTIL could overcome these limitations and recommended that any algorithm should follow the manual guidelines where appropriate. However, no existing studies capture all the concepts of the guideline or have shown the same prognostic evidence as manual assessment. In this study, we present a fully automated digital image analysis pipeline and demonstrate that our hematoxylin and eosin (H&E)-based pipeline can provide a quantitative and interpretable score that correlates with the manual pathologist-derived sTIL status, and importantly, can stratify a retrospective cohort into two significant distinct prognostic groups. We found our score to be prognostic for OS (HR: 0.81 CI: 0.72-0.92 p = 0.001) independent of age, tumor size, nodal status, and tumor type in statistical modeling. While prior studies have followed fragments of the TIL-WG guideline, our approach is the first to follow all complex aspects, where appropriate, supporting the TIL-WG vision of computational assessment of sTIL in the future clinical setting.

An Open Source, Automated Tumor Infiltrating Lymphocyte Algorithm for Prognosis in Triple-Negative Breast Cancer

PURPOSE

Although tumor infiltrating lymphocytes (TIL) assessment has been acknowledged to have both prognostic and predictive importance in triple negative breast cancer (TNBC), it is subject to inter and intra-observer variability that has prevented widespread adoption. Here we constructed a machine-learning based breast cancer TIL scoring approach and validated its prognostic potential in multiple TNBC cohorts.

EXPERIMENTAL DESIGN

Using the QuPath open source software, we built a neural-network classifier for tumor cells, lymphocytes, fibroblasts and "other" cells on hematoxylin-eosin (H&E) stained sections. We analyzed the classifier-derived TIL measurements with five unique constructed TIL variables. A retrospective collection of 171 TNBC cases was used as the discovery set to identify the optimal association of machine-read TIL variables with patient outcome. For validation we evaluated a retrospective collection of 749 TNBC patients comprised of four independent validation subsets.

RESULTS

We found that all five machine TIL variables had significant prognostic association with outcomes (p{less than or equal to}0.01 for all comparisons) but showed cell specific variation in validation sets. Cox regression analysis demonstrated that all five TIL variables were independently associated with improved overall survival after adjusting for clinicopathological factors including stage, age and histological grade (p{less than or equal to}0.003 for all analyses).

CONCLUSIONS

Neural net driven cell classifier defined TIL variables were robust and independent prognostic factors in several independent validation cohorts of TNBC patients. These objective, open source TIL variables are freely available to download and can now be considered for testing in a prospective setting to assess clinical utility.

A Cervical Histopathology Dataset for Computer Aided Diagnosis of Precancerous Lesions

Cervical cancer, as one of the most frequently diagnosed cancers worldwide, is curable when detected early. Histopathology images play an important role in precision medicine of the cervical lesions. However, few computer aided algorithms have been explored on cervical histopathology images due to the lack of public datasets. In this article, we release a new cervical histopathology image dataset for automated precancerous diagnosis. Specifically, 100 slides from 71 patients are annotated by three independent pathologists. To show the difficulty of the task, benchmarks are obtained through both fully and weakly supervised learning. Extensive experiments based on typical classification and semantic segmentation networks are carried out to provide strong baselines. In particular, a strategy of assembling classification, segmentation, and pseudo-labeling is proposed to further improve the performance. The Dice coefficient reaches 0.7833, indicating the feasibility of computer aided diagnosis and the effectiveness of our weakly supervised ensemble algorithm. The dataset and evaluation codes are publicly available. To the best of our knowledge, it is the first public cervical histopathology dataset for automated precancerous segmentation. We believe that this work will attract researchers to explore novel algorithms on cervical automated diagnosis, thereby assisting doctors and patients clinically.