Learning to detect lymphocytes in immunohistochemistry with deep learning

pFedBCC: Personalizing Federated multi-target domain adaptive segmentation via Bi-pole Collaborative Calibration

BACKGROUND AND OBJECTIVE

Multi-target domain adaptation (MTDA) is a well-established technology for unsupervised segmentation. It can significantly reduce the workload of large-scale data annotations, but assumes that each domain data can be freely accessed. However, data privacy limit its deployment in real-world medical scenes. Aiming at this problem, federated learning (FL) commits a paradigm to handle private cross-institution data.

METHODS

This paper makes the first attempt to apply FedMTDA to medical image segmentation by proposing a personalized Federated Bi-pole Collaborative Calibration (pFedBCC) framework, which leverages unannotated private client data and a public source-domain model to learn a global model at the central server for unsupervised multi-type immunohistochemically (IHC) image segmentation. Concretely, pFedBCC tackles two significant challenges in FedMTDA including client-side prediction drift and server-side aggregation drift via Semantic-affinity-driven Personalized Label Calibration (SPLC) and Source-knowledge-oriented Consistent Gradient Calibration (SCGC). To alleviate local prediction drift, SPLC personalizes a cross-domain graph reasoning module for each client, which achieves semantic-affinity alignment between high-level source- and target-domain features to produce pseudo labels that are semantically consistent with source-domain labels to guide client training. To further alleviate global aggregation drift, SCGC develops a new conflict-gradient clipping scheme, which takes the source-domain gradient as a guidance to ensure that all clients update with similar gradient directions and magnitudes, thereby improving the generalization of the global model.

RESULTS

pFedBCC is evaluated on private and public IHC benchmarks, including the proposed MT-IHC dataset, and the panCK, BCData, DLBC-Morph and LYON19 datasets. Overall, pFedBCC achieves the best performance of 88.8% PA on MT-IHC, as well as 88.4% PA on the LYON19 dataset, respectively.

CONCLUSIONS

The proposed pFedBCC performs better than all comparison methods. The ablation study also confirms the contribution of SPLC and SCGC for unsupervised multi-type IHC image segmentation. This paper constructs a MT-IHC dataset containing more than 19,000 IHC images of 10 types (CgA, CK, Syn, CD, Ki67, P40, P53, EMA, TdT and BCL). Extensive experiments on the MT-IHC and public IHC datasets confirm that pFedBCC outperforms existing FL and DA methods.

Leveraging Deep Learning for Immune Cell Quantification and Prognostic Evaluation in Radiotherapy-Treated Oropharyngeal Squamous Cell Carcinomas

The tumor microenvironment plays a critical role in cancer progression and therapeutic responsiveness, with the tumor immune microenvironment (TIME) being a key modulator. In head and neck squamous cell carcinomas (HNSCCs), immune cell infiltration significantly influences the response to radiotherapy (RT). A better understanding of the TIME in HNSCCs could help identify patients most likely to benefit from combining RT with immunotherapy. Standardized, cost-effective methods for studying TIME in HNSCCs are currently lacking. This study aims to leverage deep learning (DL) to quantify immune cell densities using immunohistochemistry in untreated oropharyngeal squamous cell carcinoma (OPSCC) biopsies of patients scheduled for curative RT and assess their prognostic value. We analyzed 84 pretreatment formalin-fixed paraffin-embedded tumor biopsies from OPSCC patients. Immunohistochemistry was performed for CD3, CD8, CD20, CD163, and FOXP3, and whole slide images were digitized for analysis using a U-Net-based DL model. Two quantification approaches were applied: a cell-counting method and an area-based method. These methods were applied to stained regions. The DL model achieved high accuracy in detecting stained cells across all biomarkers. Strong correlations were found between our DL pipeline, the HALO Image Analysis Platform, and the open-source QuPath software for estimating immune cell densities. Our DL pipeline provided an accurate and reproducible approach for quantifying immune cells in OPSCC. The area-based method demonstrated superior prognostic value for recurrence-free survival, when compared with the cell-counting method. Elevated densities of CD3, CD8, CD20, and FOXP3 were associated with improved recurrence-free survival, whereas CD163 showed no significant prognostic association. These results highlight the potential of DL in digital pathology for assessing TIME and predicting patient outcomes.

Automated Detection of Kaposi Sarcoma-Associated Herpesvirus-Infected Cells in Immunohistochemical Images of Skin Biopsies

PURPOSE

Immunohistochemical staining for the antigen of Kaposi sarcoma (KS)-associated herpesvirus, latency-associated nuclear antigen (LANA), is helpful in diagnosing KS. A challenge lies in distinguishing anti-LANA-positive cells from morphologically similar brown counterparts. This work aims to develop an automated framework for localization and quantification of LANA positivity in whole-slide images (WSI) of skin biopsies.

METHODS

The proposed framework leverages weakly supervised multiple-instance learning (MIL) to reduce false-positive predictions. A novel morphology-based slide aggregation method is introduced to improve accuracy. The framework generates interpretable heatmaps for cell localization and provides quantitative values for the percentage of positive tiles. The framework was trained and tested with a KS pathology data set prepared from skin biopsies of KS-suspected patients in Uganda.

RESULTS

The developed MIL framework achieved an area under the receiver operating characteristic curve of 0.99, with a sensitivity of 98.15% and specificity of 96.00% in predicting anti-LANA-positive WSIs in a test data set.

CONCLUSION

The framework shows promise for the automated detection of LANA in skin biopsies, offering a reliable and accurate tool for identifying anti-LANA-positive cells. This method may be especially impactful in resource-limited areas that lack trained pathologists, potentially improving diagnostic capabilities in settings with limited access to expert analysis.

Prediction and analysis of tumor infiltrating lymphocytes across 28 cancers by TILScout using deep learning

The density of tumor-infiltrating lymphocytes (TILs) serves as a valuable indicator for predicting anti-tumor responses, but its broad impact across various types of cancers remains underexplored. We introduce TILScout, a pan-cancer deep-learning approach to compute patch-level TIL scores from whole slide images (WSIs). TILScout achieved accuracies of 0.9787 and 0.9628, and AUCs of 0.9988 and 0.9934 in classifying WSI patches into three categories-TIL-positive, TIL-negative, and other/necrotic-on validation and independent test sets, respectively, surpassing previous studies. The biological significance of TILScout-derived TIL scores across 28 cancers was validated through comprehensive functional and correlational analyses. A consistent decrease in TIL scores with an increase in cancer stage provides direct evidence that the lower TIL content may stimulate cancer progression. Additionally, TIL scores correlated with immune checkpoint gene expression and genomic variation in common cancer driver genes. Our comprehensive pan-cancer survey highlights the critical prognostic significance of TILs within the tumor microenvironment.

The times they are AI-changing: AI-powered advances in the application of extracellular vesicles to liquid biopsy in breast cancer

Artificial intelligence (AI) is revolutionizing scientific research by facilitating a paradigm shift in data analysis and discovery. This transformation is characterized by a fundamental change in scientific methods and concepts due to AI's ability to process vast datasets with unprecedented speed and accuracy. In breast cancer research, AI aids in early detection, prognosis, and personalized treatment strategies. Liquid biopsy, a noninvasive tool for detecting circulating tumor traits, could ideally benefit from AI's analytical capabilities, enhancing the detection of minimal residual disease and improving treatment monitoring. Extracellular vesicles (EVs), which are key elements in cell communication and cancer progression, could be analyzed with AI to identify disease-specific biomarkers. AI combined with EV analysis promises an enhancement in diagnosis precision, aiding in early detection and treatment monitoring. Studies show that AI can differentiate cancer types and predict drug efficacy, exemplifying its potential in personalized medicine. Overall, the integration of AI in biomedical research and clinical practice promises significant changes and advancements in diagnostics, personalized medicine-based approaches, and our understanding of complex diseases like cancer.

AI-based tumor-infiltrating lymphocyte scoring system for assessing HCC prognosis in patients undergoing liver resection

Background & Aims

Tumor-infiltrating lymphocytes (TILs), particularly CD8+ TILs, are key prognostic markers in many cancers. However, their prognostic value in hepatocellular carcinoma (HCC) remains controversial, with different evidence. Given the heterogeneous outcomes in patients with HCC undergoing liver resection, this study aims to develop an AI-based system to quantify CD8+ TILs and assess their prognostic value for patients with HCC.

Methods

We conducted a retrospective multicenter study on patients undergoing liver resection across three cohorts (N = 514). We trained a deep neural network and a random forest model to segment tumor regions and locate CD8+ TILs in H&E and CD8-stained whole-slide images. We quantified CD8+ TIL density and established an Automated CD8+ Tumor-infiltrating Lymphocyte Scoring (ATLS-8) system to assess its prognostic value.

Results

In the discovery cohort, the 5-year overall survival (OS) rates were 34.05% for ATLS-8 low-score and 65.03% for ATLS-8 high-score groups (hazard ratio [HR] 2.40; 95% CI, 1.37-4.19; p = 0.015). These findings were confirmed in validation cohort 1, which had 5-year OS rates of 28.57% and 68.73% (HR 3.38; 95% CI, 1.27-9.02; p = 0.0098), and validation cohort 2, which had 59.26% and 81.48% (HR 2.74; 95% CI, 1.05-7.15; p = 0.031). ATLS-8 improved the prognostic model based on clinical variables (C-index 0.770 vs. 0.757; 0.769 vs. 0.727; 0.712 vs. 0.642 in three cohorts).

Conclusions

We developed an automated system using CD8-stained whole-slide images to assess immune infiltration (ATLS-8). In patients with HCC undergoing resection, higher CD8+ TIL density correlates with better OS, as per ATLS-8 assessment. This system is a promising tool for advancing clinical immune microenvironment assessment and outcome prediction.

Impact and implications

CD8+ tumor-infiltrating lymphocytes (TILs) have been identified as a prognostic factor associated with many cancers. In this study, CD8+ TILs were identified as an independent prognostic factor for overall survival in patients with hepatocellular carcinoma who undergoing liver resection. Therefore, ATLS-8, a novel digital biomarker based on whole-slide image-level CD8+ TILs, could play an important role in the prognostic assessment of patients with HCC and could be integrated into clinicopathological models to participate in the decision-making and prognostic assessment of patients. The scoring system combined with artificial intelligence is essential for automated, quantitative, whole-slide image-level assessment of TILs, which can be widely applied to quantify the immune profile of multi-cancer disease types with the discussion of subsequent immunotherapy.

Computer-assisted diagnosis to improve diagnostic pathology: A review

ABSTRACT

With an increasing demand for accuracy and efficiency in diagnostic pathology, computer-assisted diagnosis (CAD) emerges as a prominent and transformative solution. This review aims to explore the practical applications, implications, strengths, and weaknesses of CAD applied to diagnostic pathology. A comprehensive literature search was conducted to include English-language studies focusing on CAD tools, digital pathology, and Artificial intelligence (AI) applications in pathology. The review underscores the transformative potential of CAD tools in pathology, particularly in streamlining diagnostic processes, reducing turnaround times, and augmenting diagnostic accuracy. It emphasizes the strides made in digital pathology, the integration of AI, and the promising prospects for prognostic biomarker discovery using computational methods. Additionally, ethical considerations regarding data privacy, equity, and trust in AI deployment are examined. CAD has the potential to revolutionize diagnostic pathology. The insights gleaned from this review offer a panoramic view of recent advancements. Ultimately, this review aims to guide future research, influence clinical practice, and inform policy-making by elucidating the promising horizons and potential pitfalls of integrating CAD tools in pathology.

ImmuNet: a segmentation-free machine learning pipeline for immune landscape phenotyping in tumors by multiplex imaging

Tissue specimens taken from primary tumors or metastases contain important information for diagnosis and treatment of cancer patients. Multiplex imaging allows in situ visualization of heterogeneous cell populations, such as immune cells, in tissue samples. Most image processing pipelines first segment cell boundaries and then measure marker expression to assign cell phenotypes. In dense tissue environments, this segmentation-first approach can be inaccurate due to segmentation errors or overlapping cells. Here, we introduce the machine-learning pipeline "ImmuNet", which identifies positions and phenotypes of cells without segmenting them. ImmuNet is easy to train: human annotators only need to click on an immune cell and score its expression of each marker-drawing a full cell outline is not required. We trained and evaluated ImmuNet on multiplex images from human tonsil, lung cancer, prostate cancer, melanoma, and bladder cancer tissue samples and found it to consistently achieve error rates below 5%-10% across tissue types, cell types, and tissue densities, outperforming a segmentation-based baseline method. Furthermore, we externally validate ImmuNet results by comparing them to flow cytometric cell count measurements from the same tissue. In summary, ImmuNet is an effective, simpler alternative to segmentation-based approaches when only cell positions and phenotypes, but not their shapes, are required for downstream analyses. Thus, ImmuNet helps researchers to analyze cell positions in multiplex tissue images more easily and accurately.

A universal immunohistochemistry analyzer for generalizing AI-driven assessment of immunohistochemistry across immunostains and cancer types

Immunohistochemistry (IHC) is the common companion diagnostics in targeted therapies. However, quantifying protein expressions in IHC images present a significant challenge, due to variability in manual scoring and inherent subjective interpretation. Deep learning (DL) offers a promising approach to address these issues, though current models require extensive training for each cancer and IHC type, limiting the practical application. We developed a Universal IHC (UIHC) analyzer, a DL-based tool that quantifies protein expression across different cancers and IHC types. This multi-cohort trained model outperformed conventional single-cohort models in analyzing unseen IHC images (Kappa score 0.578 vs. up to 0.509) and demonstrated consistent performance across varying positive staining cutoff values. In a discovery application, the UIHC model assigned higher tumor proportion scores to MET amplification cases, but not MET exon 14 splicing or other non-small cell lung cancer cases. This UIHC model represents a novel role for DL that further advances quantitative analysis of IHC.

A Digital Workflow for Automated Assessment of Tumor-Infiltrating Lymphocytes in Oral Squamous Cell Carcinoma Using QuPath and a StarDist-Based Model

The search for reliable prognostic markers in oral squamous cell carcinoma (OSCC) remains a critical need. Tumor-infiltrating lymphocytes (TILs), particularly T lymphocytes, play a pivotal role in the immune response against tumors and are strongly correlated with favorable prognoses. Computational pathology has proven highly effective for histopathological image analysis, automating tasks such as cell detection, classification, and segmentation.

In the present study, we developed a StarDist-based model to automatically detect T lymphocytes in hematoxylin and eosin (H&E)-stained whole-slide images (WSIs) of OSCC, bypassing the need for traditional immunohistochemistry (IHC). Using QuPath, we generated training datasets from annotated slides, employing IHC as the ground truth. Our model was validated on Cancer Genome Atlas-derived OSCC images, and survival analyses demonstrated that higher TIL densities correlated with improved patient outcomes.

This work introduces an efficient, AI-powered workflow for automated immune profiling in OSCC, offering a reproducible and scalable approach for diagnostic and prognostic applications.

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Enhancing AI Research for Breast Cancer: A Comprehensive Review of Tumor-Infiltrating Lymphocyte Datasets

The field of immunology is fundamental to our understanding of the intricate dynamics of the tumor microenvironment. In particular, tumor-infiltrating lymphocyte (TIL) assessment emerges as essential aspect in breast cancer cases. To gain comprehensive insights, the quantification of TILs through computer-assisted pathology (CAP) tools has become a prominent approach, employing advanced artificial intelligence models based on deep learning techniques. The successful recognition of TILs requires the models to be trained, a process that demands access to annotated datasets. Unfortunately, this task is hampered not only by the scarcity of such datasets, but also by the time-consuming nature of the annotation phase required to create them. Our review endeavors to examine publicly accessible datasets pertaining to the TIL domain and thereby become a valuable resource for the TIL community. The overall aim of the present review is thus to make it easier to train and validate current and upcoming CAP tools for TIL assessment by inspecting and evaluating existing publicly available online datasets.

Hepatocellular Carcinoma Immune Microenvironment Analysis: A Comprehensive Assessment with Computational and Classical Pathology

PURPOSE

The spatial variability and clinical relevance of the tumor immune microenvironment (TIME) are still poorly understood for hepatocellular carcinoma (HCC). In this study, we aim to develop a deep learning (DL)-based image analysis model for the spatial analysis of immune cell biomarkers and microscopically evaluate the distribution of immune infiltration.

EXPERIMENTAL DESIGN

Ninety-two HCC surgical liver resections and 51 matched needle biopsies were histologically classified according to their immunophenotypes: inflamed, immune-excluded, and immune-desert. To characterize the TIME on immunohistochemistry (IHC)-stained slides, we designed a multistage DL algorithm, IHC-TIME, to automatically detect immune cells and their localization in the TIME in tumor-stroma and center-border segments.

RESULTS

Two models were trained to detect and localize the immune cells on IHC-stained slides. The framework models (i.e., immune cell detection models and tumor-stroma segmentation) reached 98% and 91% accuracy, respectively. Patients with inflamed tumors showed better recurrence-free survival than those with immune-excluded or immune-desert tumors. Needle biopsies were found to be 75% accurate in representing the immunophenotypes of the main tumor. Finally, we developed an algorithm that defines immunophenotypes automatically based on the IHC-TIME analysis, achieving an accuracy of 80%.

CONCLUSIONS

Our DL-based tool can accurately analyze and quantify immune cells on IHC-stained slides of HCC. Microscopic classification of the TIME can stratify HCC according to the patient prognosis. Needle biopsies can provide valuable insights for TIME-related prognostic prediction, albeit with specific constraints. The computational pathology tool provides a new way to study the HCC TIME.

Computational staining of CD3/CD20 positive lymphocytes in human tissues with experimental confirmation in a genetically engineered mouse model

Introduction

Immune dysregulation plays a major role in cancer progression. The quantification of lymphocytic spatial inflammation may enable spatial system biology, improve understanding of therapeutic resistance, and contribute to prognostic imaging biomarkers.

Methods

In this paper, we propose a knowledge-guided deep learning framework to measure the lymphocytic spatial architecture on human H&E tissue, where the fidelity of training labels is maximized through single-cell resolution image registration of H&E to IHC. We demonstrate that such an approach enables pixel-perfect ground-truth labeling of lymphocytes on H&E as measured by IHC. We then experimentally validate our technique in a genetically engineered, immune-compromised Rag2 mouse model, where Rag2 knockout mice lacking mature lymphocytes are used as a negative experimental control. Such experimental validation moves beyond the classical statistical testing of deep learning models and demonstrates feasibility of more rigorous validation strategies that integrate computational science and basic science.

Results

Using our developed approach, we automatically annotated more than 111,000 human nuclei (45,611 CD3/CD20 positive lymphocytes) on H&E images to develop our model, which achieved an AUC of 0.78 and 0.71 on internal hold-out testing data and external testing on an independent dataset, respectively. As a measure of the global spatial architecture of the lymphocytic microenvironment, the average structural similarity between predicted lymphocytic density maps and ground truth lymphocytic density maps was 0.86 ± 0.06 on testing data. On experimental mouse model validation, we measured a lymphocytic density of 96.5 ± %1% in a Rag2+/- control mouse, compared to an average of 16.2 ± %5% in Rag2-/- immune knockout mice (p<0.0001, ANOVA-test).

Discussion

These results demonstrate that CD3/CD20 positive lymphocytes can be accurately detected and characterized on H&E by deep learning and generalized across species. Collectively, these data suggest that our understanding of complex biological systems may benefit from computationally-derived spatial analysis, as well as integration of computational science and basic science.

Automated scoring methods for quantitative interpretation of Tumour infiltrating lymphocytes (TILs) in breast cancer: a systematic review

Tumour microenvironment (TME) of breast cancer mainly comprises malignant, stromal, immune, and tumour infiltrating lymphocyte (TILs). Assessment of TILs is crucial for determining the disease's prognosis. Manual TIL assessments are hampered by multiple limitations, including low precision, poor inter-observer reproducibility, and time consumption. In response to these challenges, automated scoring emerges as a promising approach. The aim of this systematic review is to assess the evidence on the approaches and performance of automated scoring methods for TILs assessment in breast cancer. This review presents a comprehensive compilation of studies related to automated scoring of TILs, sourced from four databases (Web of Science, Scopus, Science Direct, and PubMed), employing three primary keywords (artificial intelligence, breast cancer, and tumor-infiltrating lymphocytes). The PICOS framework was employed for study eligibility, and reporting adhered to the PRISMA guidelines. The initial search yielded a total of 1910 articles. Following screening and examination, 27 studies met the inclusion criteria and data were extracted for the review. The findings indicate a concentration of studies on automated TILs assessment in developed countries, specifically the United States and the United Kingdom. From the analysis, a combination of sematic segmentation and object detection (n = 10, 37%) and convolutional neural network (CNN) (n = 11, 41%), become the most frequent automated task and ML approaches applied for model development respectively. All models developed their own ground truth datasets for training and validation, and 59% of the studies assessed the prognostic value of TILs. In conclusion, this analysis contends that automated scoring methods for TILs assessment of breast cancer show significant promise for commodification and application within clinical settings.

A deep-learning framework to predict cancer treatment response from histopathology images through imputed transcriptomics

Advances in artificial intelligence have paved the way for leveraging hematoxylin and eosin-stained tumor slides for precision oncology. We present ENLIGHT-DeepPT, an indirect two-step approach consisting of (1) DeepPT, a deep-learning framework that predicts genome-wide tumor mRNA expression from slides, and (2) ENLIGHT, which predicts response to targeted and immune therapies from the inferred expression values. We show that DeepPT successfully predicts transcriptomics in all 16 The Cancer Genome Atlas cohorts tested and generalizes well to two independent datasets. ENLIGHT-DeepPT successfully predicts true responders in five independent patient cohorts involving four different treatments spanning six cancer types, with an overall odds ratio of 2.28 and a 39.5% increased response rate among predicted responders versus the baseline rate. Notably, its prediction accuracy, obtained without any training on the treatment data, is comparable to that achieved by directly predicting the response from the images, which requires specific training on the treatment evaluation cohorts.

EpidermaQuant: Unsupervised Detection and Quantification of Epidermal Differentiation Markers on H-DAB-Stained Images of Reconstructed Human Epidermis

The integrity of the reconstructed human epidermis generated in vitro can be assessed using histological analyses combined with immunohistochemical staining of keratinocyte differentiation markers. Technical differences during the preparation and capture of stained images may influence the outcome of computational methods. Due to the specific nature of the analyzed material, no annotated datasets or dedicated methods are publicly available. Using a dataset with 598 unannotated images showing cross-sections of in vitro reconstructed human epidermis stained with DAB-based immunohistochemistry reaction to visualize four different keratinocyte differentiation marker proteins (filaggrin, keratin 10, Ki67, HSPA2) and counterstained with hematoxylin, we developed an unsupervised method for the detection and quantification of immunohistochemical staining. The pipeline consists of the following steps: (i) color normalization; (ii) color deconvolution; (iii) morphological operations; (iv) automatic image rotation; and (v) clustering. The most effective combination of methods includes (i) Reinhard's normalization; (ii) Ruifrok and Johnston color-deconvolution method; (iii) proposed image-rotation method based on boundary distribution of image intensity; and (iv) k-means clustering. The results of the work should enhance the performance of quantitative analyses of protein markers in reconstructed human epidermis samples and enable the comparison of their spatial distribution between different experimental conditions.

Automated detection of Kaposi sarcoma-associated herpesvirus infected cells in immunohistochemical images of skin biopsies

Immunohistochemical (IHC) staining for the antigen of Kaposi sarcoma-associated herpesvirus (KSHV), latency-associated nuclear antigen (LANA), is helpful in diagnosing Kaposi sarcoma (KS). A challenge, however, lies in distinguishing anti-LANA-positive cells from morphologically similar brown counterparts. In this work, we demonstrate a framework for automated localization and quantification of LANA positivity in whole slide images (WSI) of skin biopsies, leveraging weakly supervised multiple instance learning (MIL) while reducing false positive predictions by introducing a novel morphology-based slide aggregation method. Our framework generates interpretable heatmaps, offering insights into precise anti-LANA-positive cell localization within WSIs and a quantitative value for the percentage of positive tiles, which may assist with histological subtyping. We trained and tested our framework with an anti-LANA-stained KS pathology dataset prepared by pathologists in the United States from skin biopsies of KS-suspected patients investigated in Uganda. We achieved an area under the receiver operating characteristic curve (AUC) of 0.99 with a sensitivity and specificity of 98.15% and 96.00% in predicting anti-LANA-positive WSIs in a test dataset. We believe that the framework can provide promise for automated detection of LANA in skin biopsies, which may be especially impactful in resource-limited areas that lack trained pathologists.

Computational methods and biomarker discovery strategies for spatial proteomics: a review in immuno-oncology

Advancements in imaging technologies have revolutionized our ability to deeply profile pathological tissue architectures, generating large volumes of imaging data with unparalleled spatial resolution. This type of data collection, namely, spatial proteomics, offers invaluable insights into various human diseases. Simultaneously, computational algorithms have evolved to manage the increasing dimensionality of spatial proteomics inherent in this progress. Numerous imaging-based computational frameworks, such as computational pathology, have been proposed for research and clinical applications. However, the development of these fields demands diverse domain expertise, creating barriers to their integration and further application. This review seeks to bridge this divide by presenting a comprehensive guideline. We consolidate prevailing computational methods and outline a roadmap from image processing to data-driven, statistics-informed biomarker discovery. Additionally, we explore future perspectives as the field moves toward interfacing with other quantitative domains, holding significant promise for precision care in immuno-oncology.

Exploring immune status in peripheral blood and tumor tissue in association with survival in patients with multi-organ metastatic colorectal cancer

Colorectal cancer (CRC) raises considerable clinical challenges, including a high mortality rate once the tumor spreads to distant sites. At this advanced stage, more accurate prediction of prognosis and treatment outcome is urgently needed. The role of cancer immunity in metastatic CRC (mCRC) is poorly understood. Here, we explore cellular immune cell status in patients with multi-organ mCRC. We analyzed T cell infiltration in primary tumor sections, surveyed the lymphocytic landscape of liver metastases, and assessed circulating mononuclear immune cells. Besides asking whether immune cells are associated with survival at this stage of the disease, we investigated correlations between the different tissue types; as this could indicate a dominant immune phenotype. Taken together, our analyses corroborate previous observations that higher levels of CD8+ T lymphocytes link to better survival outcomes. Our findings therefore extend evidence from earlier stages of CRC to indicate an important role for cancer immunity in disease control even after metastatic spreading to multiple organs. This finding may help to improve predicting outcome of patients with mCRC and suggests a future role for immunotherapeutic strategies.

A systematic review of machine learning-based tumor-infiltrating lymphocytes analysis in colorectal cancer: Overview of techniques, performance metrics, and clinical outcomes

The incidence of colorectal cancer (CRC), one of the deadliest cancers around the world, is increasing. Tissue microenvironment (TME) features such as tumor-infiltrating lymphocytes (TILs) can have a crucial impact on diagnosis or decision-making for treating patients with CRC. While clinical studies showed that TILs improve the host immune response, leading to a better prognosis, inter-observer agreement for quantifying TILs is not perfect. Incorporating machine learning (ML) based applications in clinical routine may promote diagnosis reliability. Recently, ML has shown potential for making progress in routine clinical procedures. We aim to systematically review the TILs analysis based on ML in CRC histological images. Deep learning (DL) and non-DL techniques can aid pathologists in identifying TILs, and automated TILs are associated with patient outcomes. However, a large multi-institutional CRC dataset with a diverse and multi-ethnic population is necessary to generalize ML methods.

Advances of machine learning-assisted small extracellular vesicles detection strategy

Detection of extracellular vesicles (EVs), particularly small EVs (sEVs), is of great significance in exploring their physiological characteristics and clinical applications. The heterogeneity of sEVs plays a crucial role in distinguishing different types of cells and diseases. Machine learning, with its exceptional data processing capabilities, offers a solution to overcome the limitations of conventional detection methods for accurately classifying sEV subtypes and sources. Principal component analysis, linear discriminant analysis, partial least squares discriminant analysis, XGBoost, support vector machine, k-nearest neighbor, and deep learning, along with some combined methods such as principal component-linear discriminant analysis, have been successfully applied in the detection and identification of sEVs. This review focuses on machine learning-assisted detection strategies for cell identification and disease prediction via sEVs, and summarizes the integration of these strategies with surface-enhanced Raman scattering, electrochemistry, inductively coupled plasma mass spectrometry and fluorescence. The performance of different machine learning-based detection strategies is compared, and the advantages and limitations of various machine learning models are also evaluated. Finally, we discuss the merits and limitations of the current approaches and briefly outline the perspective of potential research directions in the field of sEV analysis based on machine learning.

Lymph node metastases and recurrence in pT1 colorectal cancer: Prediction with the International Budding Consortium Score-A retrospective, multi-centric study

BACKGROUND

The International Collaboration on Cancer Reporting proposes histological tumour type, lymphovascular invasion, tumour grade, perineural invasion, extent, and dimensions of invasion as risk factors for lymph node metastases and tumour progression in completely endoscopically resected pT1 colorectal cancer (CRC).

OBJECTIVE

The aim of the study was to propose a predictive and reliable score to optimise the clinical management of endoscopically resected pT1 CRC patients.

METHODS

This multi-centric, retrospective International Budding Consortium (IBC) study included an international pT1 CRC cohort of 565 patients. All cases were reviewed by eight expert gastrointestinal pathologists. All risk factors were reported according to international guidelines. Tumour budding and immune response (CD8+ T-cells) were assessed with automated models using artificial intelligence. We used the information on risk factors and least absolute shrinkage and selection operator logistic regression to develop a prediction model and generate a score to predict the occurrence of lymph node metastasis or cancer recurrence.

RESULTS

The IBC prediction score included the following parameters: lymphovascular invasion, tumour buds, infiltration depth and tumour grade. The score has an acceptable discrimination power (area under the curve of 0.68 [95% confidence intervals (CI) 0.61-0.75]; 0.64 [95% CI 0.57-0.71] after internal validation). At a cut-off of 6.8 points to discriminate high-and low-risk patients, the score had a sensitivity and specificity of 0.9 [95% CI 0.8-0.95] and 0.26 [95% 0.22, 0.3], respectively.

CONCLUSION

The IBC score is based on well-established risk factors and is a promising tool with clinical utility to support the management of pT1 CRC patients.

GravityNet for end-to-end small lesion detection

This paper introduces a novel one-stage end-to-end detector specifically designed to detect small lesions in medical images. Precise localization of small lesions presents challenges due to their appearance and the diverse contextual backgrounds in which they are found. To address this, our approach introduces a new type of pixel-based anchor that dynamically moves towards the targeted lesion for detection. We refer to this new architecture as GravityNet, and the novel anchors as gravity points since they appear to be "attracted" by the lesions. We conducted experiments on two well-established medical problems involving small lesions to evaluate the performance of the proposed approach: microcalcifications detection in digital mammograms and microaneurysms detection in digital fundus images. Our method demonstrates promising results in effectively detecting small lesions in these medical imaging tasks.

HoLy-Net: Segmentation of histological images of diffuse large B-cell lymphoma

Over the last years, there has been large progress in automated segmentation and classification methods in histological whole slide images (WSIs) stained with hematoxylin and eosin (H&E). Current state-of-the-art (SOTA) techniques are based on diverse datasets of H&E-stained WSIs of different types of predominantly solid cancer. However, there is a scarcity of methods and datasets enabling segmentation of tumors of the lymphatic system (lymphomas). Here, we propose a solution for segmentation of diffuse large B-cell lymphoma (DLBCL), the most common non-Hodgkin's lymphoma. Our method applies to both H&E-stained slides and to a broad range of markers stained with immunohistochemistry (IHC). While IHC staining is an important tool in cancer diagnosis and treatment decisions, there are few automated segmentation and classification methods for IHC-stained WSIs. To address the challenges of nuclei segmentation in H&E- and IHC-stained DLBCL images, we propose HoLy-Net - a HoVer-Net-based deep learning model for lymphoma segmentation. We train two different models, one for segmenting H&E- and one for IHC-stained images and compare the test results with the SOTA methods as well as with the original version of HoVer-Net. Subsequently, we segment patient WSIs and perform single cell-level analysis of different cell types to identify patient-specific tumor characteristics such as high level of immune infiltration. Our method outperforms general-purpose segmentation methods for H&E staining in lymphoma WSIs (with an F1 score of 0.899) and is also a unique automated method for IHC slide segmentation (with an F1 score of 0.913). With our solution, we provide a new dataset we denote LyNSeC (lymphoma nuclear segmentation and classification) containing 73,931 annotated cell nuclei from H&E and 87,316 from IHC slides. Our method and dataset open up new avenues for quantitative, large-scale studies of morphology and microenvironment of lymphomas overlooked by the current automated segmentation methods.

LYSTO: The Lymphocyte Assessment Hackathon and Benchmark Dataset

We introduce LYSTO, the Lymphocyte Assessment Hackathon, which was held in conjunction with the MICCAI 2019 Conference in Shenzhen (China). The competition required participants to automatically assess the number of lymphocytes, in particular T-cells, in images of colon, breast, and prostate cancer stained with CD3 and CD8 immunohistochemistry. Differently from other challenges setup in medical image analysis, LYSTO participants were solely given a few hours to address this problem. In this paper, we describe the goal and the multi-phase organization of the hackathon; we describe the proposed methods and the on-site results. Additionally, we present post-competition results where we show how the presented methods perform on an independent set of lung cancer slides, which was not part of the initial competition, as well as a comparison on lymphocyte assessment between presented methods and a panel of pathologists. We show that some of the participants were capable to achieve pathologist-level performance at lymphocyte assessment. After the hackathon, LYSTO was left as a lightweight plug-and-play benchmark dataset on grand-challenge website, together with an automatic evaluation platform.

Convolutional Neural Networks in the Diagnosis of Colon Adenocarcinoma

Colorectal cancer is one of the most lethal cancers because of late diagnosis and challenges in the selection of therapy options. The histopathological diagnosis of colon adenocarcinoma is hindered by poor reproducibility and a lack of standard examination protocols required for appropriate treatment decisions. In the current study, using state-of-the-art approaches on benchmark datasets, we analyzed different architectures and ensembling strategies to develop the most efficient network combinations to improve binary and ternary classification. We propose an innovative two-stage pipeline approach to diagnose colon adenocarcinoma grading from histological images in a similar manner to a pathologist. The glandular regions were first segmented by a transformer architecture with subsequent classification using a convolutional neural network (CNN) ensemble, which markedly improved the learning efficiency and shortened the learning time. Moreover, we prepared and published a dataset for clinical validation of the developed artificial neural network, which suggested the discovery of novel histological phenotypic alterations in adenocarcinoma sections that could have prognostic value. Therefore, AI could markedly improve the reproducibility, efficiency, and accuracy of colon cancer diagnosis, which are required for precision medicine to personalize the treatment of cancer patients.

One label is all you need: Interpretable AI-enhanced histopathology for oncology

Artificial Intelligence (AI)-enhanced histopathology presents unprecedented opportunities to benefit oncology through interpretable methods that require only one overall label per hematoxylin and eosin (H&E) slide with no tissue-level annotations. We present a structured review of these methods organized by their degree of verifiability and by commonly recurring application areas in oncological characterization. First, we discuss morphological markers (tumor presence/absence, metastases, subtypes, grades) in which AI-identified regions of interest (ROIs) within whole slide images (WSIs) verifiably overlap with pathologist-identified ROIs. Second, we discuss molecular markers (gene expression, molecular subtyping) that are not verified via H&E but rather based on overlap with positive regions on adjacent tissue. Third, we discuss genetic markers (mutations, mutational burden, microsatellite instability, chromosomal instability) that current technologies cannot verify if AI methods spatially resolve specific genetic alterations. Fourth, we discuss the direct prediction of survival to which AI-identified histopathological features quantitatively correlate but are nonetheless not mechanistically verifiable. Finally, we discuss in detail several opportunities and challenges for these one-label-per-slide methods within oncology. Opportunities include reducing the cost of research and clinical care, reducing the workload of clinicians, personalized medicine, and unlocking the full potential of histopathology through new imaging-based biomarkers. Current challenges include explainability and interpretability, validation via adjacent tissue sections, reproducibility, data availability, computational needs, data requirements, domain adaptability, external validation, dataset imbalances, and finally commercialization and clinical potential. Ultimately, the relative ease and minimum upfront cost with which relevant data can be collected in addition to the plethora of available AI methods for outcome-driven analysis will surmount these current limitations and achieve the innumerable opportunities associated with AI-driven histopathology for the benefit of oncology.

An AI-Ready Multiplex Staining Dataset for Reproducible and Accurate Characterization of Tumor Immune Microenvironment

We introduce a new AI-ready computational pathology dataset containing restained and co-registered digitized images from eight head-and-neck squamous cell carcinoma patients. Specifically, the same tumor sections were stained with the expensive multiplex immunofluorescence (mIF) assay first and then restained with cheaper multiplex immunohistochemistry (mIHC). This is a first public dataset that demonstrates the equivalence of these two staining methods which in turn allows several use cases; due to the equivalence, our cheaper mIHC staining protocol can offset the need for expensive mIF staining/scanning which requires highly-skilled lab technicians. As opposed to subjective and error-prone immune cell annotations from individual pathologists (disagreement > 50%) to drive SOTA deep learning approaches, this dataset provides objective immune and tumor cell annotations via mIF/mIHC restaining for more reproducible and accurate characterization of tumor immune microenvironment (e.g. for immunotherapy). We demonstrate the effectiveness of this dataset in three use cases: (1) IHC quantification of CD3/CD8 tumor-infiltrating lymphocytes via style transfer, (2) virtual translation of cheap mIHC stains to more expensive mIF stains, and (3) virtual tumor/immune cellular phenotyping on standard hematoxylin images. The dataset is available at https://github.com/nadeemlab/DeepLIIF.

Machine learning in computational histopathology: Challenges and opportunities

Digital histopathological images, high-resolution images of stained tissue samples, are a vital tool for clinicians to diagnose and stage cancers. The visual analysis of patient state based on these images are an important part of oncology workflow. Although pathology workflows have historically been conducted in laboratories under a microscope, the increasing digitization of histopathological images has led to their analysis on computers in the clinic. The last decade has seen the emergence of machine learning, and deep learning in particular, a powerful set of tools for the analysis of histopathological images. Machine learning models trained on large datasets of digitized histopathology slides have resulted in automated models for prediction and stratification of patient risk. In this review, we provide context for the rise of such models in computational histopathology, highlight the clinical tasks they have found success in automating, discuss the various machine learning techniques that have been applied to this domain, and underscore open problems and opportunities.

Lymphocyte detection for cancer analysis using a novel fusion block based channel boosted CNN

Tumor-infiltrating lymphocytes, specialized immune cells, are considered an important biomarker in cancer analysis. Automated lymphocyte detection is challenging due to its heterogeneous morphology, variable distribution, and presence of artifacts. In this work, we propose a novel Boosted Channels Fusion-based CNN "BCF-Lym-Detector" for lymphocyte detection in multiple cancer histology images. The proposed network initially selects candidate lymphocytic regions at the tissue level and then detects lymphocytes at the cellular level. The proposed "BCF-Lym-Detector" generates diverse boosted channels by utilizing the feature learning capability of different CNN architectures. In this connection, a new adaptive fusion block is developed to combine and select the most relevant lymphocyte-specific features from the generated enriched feature space. Multi-level feature learning is used to retain lymphocytic spatial information and detect lymphocytes with variable appearances. The assessment of the proposed "BCF-Lym-Detector" show substantial improvement in terms of F-score (0.93 and 0.84 on LYSTO and NuClick, respectively), which suggests that the diverse feature extraction and dynamic feature selection enhanced the feature learning capacity of the proposed network. Moreover, the proposed technique's generalization on unseen test sets with a good recall (0.75) and F-score (0.73) shows its potential use for pathologists' assistance.

Pitfalls in machine learning-based assessment of tumor-infiltrating lymphocytes in breast cancer: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor-immune interaction in breast cancer is now established, and tumor-infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple-negative (estrogen receptor, progesterone receptor, and HER2-negative) breast cancer and HER2-positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state-of-the-art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false-positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in-depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple-negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Biological insights and novel biomarker discovery through deep learning approaches in breast cancer histopathology

Breast cancer remains a highly prevalent disease with considerable inter- and intra-tumoral heterogeneity complicating prognostication and treatment decisions. The utilization and depth of genomic, transcriptomic and proteomic data for cancer has exploded over recent times and the addition of spatial context to this information, by understanding the correlating morphologic and spatial patterns of cells in tissue samples, has created an exciting frontier of research, histo-genomics. At the same time, deep learning (DL), a class of machine learning algorithms employing artificial neural networks, has rapidly progressed in the last decade with a confluence of technical developments - including the advent of modern graphic processing units (GPU), allowing efficient implementation of increasingly complex architectures at scale; advances in the theoretical and practical design of network architectures; and access to larger datasets for training - all leading to sweeping advances in image classification and object detection. In this review, we examine recent developments in the application of DL in breast cancer histology with particular emphasis of those producing biologic insights or novel biomarkers, spanning the extraction of genomic information to the use of stroma to predict cancer recurrence, with the aim of suggesting avenues for further advancing this exciting field.

A pathomic approach for tumor-infiltrating lymphocytes classification on breast cancer digital pathology images

Background and objectives

The detection of tumor-infiltrating lymphocytes (TILs) could aid in the development of objective measures of the infiltration grade and can support decision-making in breast cancer (BC). However, manual quantification of TILs in BC histopathological whole slide images (WSI) is currently based on a visual assessment, thus resulting not standardized, not reproducible, and time-consuming for pathologists. In this work, a novel pathomic approach, aimed to apply high-throughput image feature extraction techniques to analyze the microscopic patterns in WSI, is proposed. In fact, pathomic features provide additional information concerning the underlying biological processes compared to the WSI visual interpretation, thus providing more easily interpretable and explainable results than the most frequently investigated Deep Learning based methods in the literature.

Methods

A dataset containing 1037 regions of interest with tissue compartments and TILs annotated on 195 TNBC and HER2+ BC hematoxylin and eosin (H&E)-stained WSI was used. After segmenting nuclei within tumor-associated stroma using a watershed-based approach, 71 pathomic features were extracted from each nucleus and reduced using a Spearman's correlation filter followed by a nonparametric Wilcoxon rank-sum test and least absolute shrinkage and selection operator. The relevant features were used to classify each candidate nucleus as either TILs or non-TILs using 5 multivariable machine learning classification models trained using 5-fold cross-validation (1) without resampling, (2) with the synthetic minority over-sampling technique and (3) with downsampling. The prediction performance of the models was assessed using ROC curves.

Results

21 features were selected, with most of them related to the well-known TILs properties of having regular shape, clearer margins, high peak intensity, more homogeneous enhancement and different textural pattern than other cells. The best performance was obtained by Random-Forest with ROC AUC of 0.86, regardless of resampling technique.

Conclusions

The presented approach holds promise for the classification of TILs in BC H&E-stained WSI and could provide support to pathologists for a reliable, rapid and interpretable clinical assessment of TILs in BC.

An open source pipeline for quantitative immunohistochemistry image analysis of inflammatory skin disease using artificial intelligence

BACKGROUND

The application of artificial intelligence (AI) to whole slide images has the potential to improve research reliability and ultimately diagnostic efficiency and service capacity. Image annotation plays a key role in AI and digital pathology. However, the work-streams required for tissue-specific (skin) and immunostain-specific annotation has not been extensively studied compared with the development of AI algorithms.

OBJECTIVES

The objective of this study is to develop a common workflow for annotating whole slide images of biopsies from inflammatory skin disease immunostained with a variety of epidermal and dermal markers prior to the development of the AI-assisted analysis pipeline.

METHODS

A total of 45 slides containing 3-5 sections each were scanned using Aperio AT2 slide scanner (Leica Biosystems). These slides were annotated by hand using a commonly used image analysis tool which resulted in more than 4000 images blocks. We used deep learning (DL) methodology to first sequentially segment (epidermis and upper dermis), with the exclusion of common artefacts and second to quantify the immunostained signal in those two compartments of skin biopsies and the ratio of positive cells.

RESULTS

We validated two DL models using 10-fold validation runs and by comparing to ground truth manually annotated data. The models achieved an average (global) accuracy of 95.0% for the segmentation of epidermis and dermis and 86.1% for the segmentation of positive/negative cells.

CONCLUSIONS

The application of two DL models in sequence facilitates accurate segmentation of epidermal and dermal structures, exclusion of common artefacts and enables the quantitative analysis of the immunostained signal. However, inaccurate annotation of the slides for training the DL model can decrease the accuracy of the output. Our open source code will facilitate further external validation across different immunostaining platforms and slide scanners.

Pan-tumor T-lymphocyte detection using deep neural networks: Recommendations for transfer learning in immunohistochemistry

The success of immuno-oncology treatments promises long-term cancer remission for an increasing number of patients. The response to checkpoint inhibitor drugs has shown a correlation with the presence of immune cells in the tumor and tumor microenvironment. An in-depth understanding of the spatial localization of immune cells is therefore critical for understanding the tumor's immune landscape and predicting drug response. Computer-aided systems are well suited for efficiently quantifying immune cells in their spatial context. Conventional image analysis approaches are often based on color features and therefore require a high level of manual interaction. More robust image analysis methods based on deep learning are expected to decrease this reliance on human interaction and improve the reproducibility of immune cell scoring. However, these methods require sufficient training data and previous work has reported low robustness of these algorithms when they are tested on out-of-distribution data from different pathology labs or samples from different organs. In this work, we used a new image analysis pipeline to explicitly evaluate the robustness of marker-labeled lymphocyte quantification algorithms depending on the number of training samples before and after being transferred to a new tumor indication. For these experiments, we adapted the RetinaNet architecture for the task of T-lymphocyte detection and employed transfer learning to bridge the domain gap between tumor indications and reduce the annotation costs for unseen domains. On our test set, we achieved human-level performance for almost all tumor indications with an average precision of 0.74 in-domain and 0.72-0.74 cross-domain. From our results, we derive recommendations for model development regarding annotation extent, training sample selection, and label extraction for the development of robust algorithms for immune cell scoring. By extending the task of marker-labeled lymphocyte quantification to a multi-class detection task, the pre-requisite for subsequent analyses, e.g., distinguishing lymphocytes in the tumor stroma from tumor-infiltrating lymphocytes, is met.

Attention-guided multi-scale deep object detection framework for lymphocyte analysis in IHC histological images

Tumor-infiltrating lymphocytes are specialized lymphocytes that can detect and kill cancerous cells. Their detection poses many challenges due to significant morphological variations, overlapping occurrence, artifact regions and high-class resemblance between clustered areas and artifacts. In this regard, a Lymphocyte Analysis Framework based on Deep Convolutional neural network (DC-Lym-AF) is proposed to analyze lymphocytes in immunohistochemistry images. The proposed framework comprises (i) pre-processing, (ii) screening phase, (iii) localization phase and (iv) post-processing. In the screening phase, a custom convolutional neural network architecture (lymphocyte dilated network) is developed to screen lymphocytic regions by performing a patch-level classification. This proposed architecture uses dilated convolutions and shortcut connections to capture multi-level variations and ensure reference-based learning. In contrast, the localization phase utilizes an attention-guided multi-scale lymphocyte detector to detect lymphocytes. The proposed detector extracts refined and multi-scale features by exploiting dilated convolutions, attention mechanism and feature pyramid network (FPN) using its custom attention-aware backbone. The proposed DC-Lym-AF shows exemplary performance on the NuClick dataset compared with the existing detection models, with an F-score and precision of 0.84 and 0.83, respectively. We verified the generalizability of our proposed framework by participating in a publically open LYON'19 challenge. Results in terms of detection rate (0.76) and F-score (0.73) suggest that the proposed DC-Lym-AF can effectively detect lymphocytes in immunohistochemistry-stained images collected from different laboratories. In addition, its promising generalization on several datasets implies that it can be turned into a medical diagnostic tool to investigate various histopathological problems. Graphical Abstract.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

Artificial intelligence applications in predicting the behavior of gastrointestinal cancers in pathology

Recent research has provided a wealth of data supporting the application of artificial intelligence (AI)-based applications in routine pathology practice. Indeed, it is clear that these methods can significantly support an accurate and rapid diagnosis by eliminating errors, increasing reliability, and improving workflow. In addition, the effectiveness of AI in the pathological evaluation of prognostic parameters associated with behavior, course, and treatment in many types of tumors has also been noted. Regarding gastrointestinal system (GIS) cancers, the contribution of AI methods to pathological diagnosis has been investigated in many studies. On the other hand, studies focusing on AI applications in evaluating parameters to determine tumor behavior are relatively few. For this purpose, the potential of AI models has been studied over a broad spectrum, from tumor subtyping to the identification of new digital biomarkers. The capacity of AI to infer genetic alterations of cancer tissues from digital slides has been demonstrated. Although current data suggest the merit of AI-based approaches in assessing tumor behavior in GIS cancers, a wide range of challenges still need to be solved, from laboratory infrastructure to improving the robustness of algorithms, before incorporating AI applications into real-life GIS pathology practice. This review aims to present data from AI applications in evaluating pathological parameters related to the behavior of GIS cancer with an overview of the opportunities and challenges encountered in implementing AI in pathology.

Artificial Intelligence-Based Prediction of Recurrence after Curative Resection for Colorectal Cancer from Digital Pathological Images

BACKGROUND

To develop an artificial intelligence-based model to predict recurrence after curative resection for stage I-III colorectal cancer from digitized pathological slides.

PATIENTS AND METHODS

In this retrospective study, 471 consecutive patients who underwent curative resection for stage I-III colorectal cancer at our institution from 2004 to 2015 were enrolled, and 512 randomly selected tiles from digitally scanned images of hematoxylin and eosin-stained tumor tissue sections were used to train a convolutional neural network. Five-fold cross-validation was used to validate the model. The association between recurrence and the model's output scores were analyzed in the test cohorts.

RESULTS

The area under the receiver operating characteristic curve of the cross-validation was 0.7245 [95% confidence interval (CI) 0.6707-0.7783; P < 0.0001]. The score successfully classified patients into those with better and worse recurrence free survival (P < 0.0001). Multivariate analysis revealed that a high score was significantly associated with worse recurrence free survival [odds ratio (OR) 1.857; 95% CI 1.248-2.805; P = 0.0021], which was independent from other predictive factors: male sex (P = 0.0238), rectal cancer (P = 0.0396), preoperative abnormal carcinoembryonic antigen (CEA) level (P = 0.0216), pathological T3/T4 stage (P = 0.0162), and pathological positive lymph node metastasis (P < 0.0001).

CONCLUSIONS

The artificial intelligence-based prediction model discriminated patients with a high risk of recurrence. This approach could help decision-makers consider the benefits of adjuvant chemotherapy.

Development of a deep learning model for the histologic diagnosis of dysplasia in Barrett's esophagus

BACKGROUND AND AIMS

The risk of progression in Barrett's esophagus (BE) increases with development of dysplasia. There is a critical need to improve the diagnosis of BE dysplasia, given substantial interobserver disagreement among expert pathologists and overdiagnosis of dysplasia by community pathologists. We developed a deep learning model to predict dysplasia grade on whole-slide imaging.

METHODS

We digitized nondysplastic BE (NDBE), low-grade dysplasia (LGD), and high-grade dysplasia (HGD) histology slides. Two expert pathologists confirmed all histology and digitally annotated areas of dysplasia. Training, validation, and test sets were created (by a random 70/20/10 split). We used an ensemble approach combining a "you only look once" model to identify regions of interest and histology class (NDBE, LGD, or HGD) followed by a ResNet101 model pretrained on ImageNet applied to the regions of interest. Diagnostic performance was determined for the whole slide.

RESULTS

We included slides from 542 patients (164 NDBE, 226 LGD, and 152 HGD) yielding 8596 bounding boxes in the training set, 1946 bounding boxes in the validation set, and 840 boxes in the test set. When the ensemble model was used, sensitivity and specificity for LGD was 81.3% and 100%, respectively, and >90% for NDBE and HGD. The overall positive predictive value and sensitivity metric (calculated as F1 score) was .91 for NDBE, .90 for LGD, and 1.0 for HGD.

CONCLUSIONS

We successfully trained and validated a deep learning model to accurately identify dysplasia on whole-slide images. This model can potentially help improve the histologic diagnosis of BE dysplasia and the appropriate application of endoscopic therapy.

Contrastive Multiple Instance Learning: An Unsupervised Framework for Learning Slide-Level Representations of Whole Slide Histopathology Images without Labels

Recent methods in computational pathology have trended towards semi- and weakly-supervised methods requiring only slide-level labels. Yet, even slide-level labels may be absent or irrelevant to the application of interest, such as in clinical trials. Hence, we present a fully unsupervised method to learn meaningful, compact representations of WSIs. Our method initially trains a tile-wise encoder using SimCLR, from which subsets of tile-wise embeddings are extracted and fused via an attention-based multiple-instance learning framework to yield slide-level representations. The resulting set of intra-slide-level and inter-slide-level embeddings are attracted and repelled via contrastive loss, respectively. This resulted in slide-level representations with self-supervision. We applied our method to two tasks- (1) non-small cell lung cancer subtyping (NSCLC) as a classification prototype and (2) breast cancer proliferation scoring (TUPAC16) as a regression prototype-and achieved an AUC of 0.8641 ± 0.0115 and correlation (R2) of 0.5740 ± 0.0970, respectively. Ablation experiments demonstrate that the resulting unsupervised slide-level feature space can be fine-tuned with small datasets for both tasks. Overall, our method approaches computational pathology in a novel manner, where meaningful features can be learned from whole-slide images without the need for annotations of slide-level labels. The proposed method stands to benefit computational pathology, as it theoretically enables researchers to benefit from completely unlabeled whole-slide images.

Predicting colorectal cancer tumor mutational burden from histopathological images and clinical information using multi-modal deep learning

MOTIVATION

Tumor mutational burden (TMB) is an indicator of the efficacy and prognosis of immune checkpoint therapy in colorectal cancer (CRC). In general, patients with higher TMB values are more likely to benefit from immunotherapy. Though whole-exome sequencing is considered the gold standard for determining TMB, it is difficult to be applied in clinical practice due to its high cost. There are also a few DNA panel-based methods to estimate TMB; however, their detection cost is also high, and the associated wet-lab experiments usually take days, which emphasize the need for faster and cheaper alternatives.

RESULTS

In this study, we propose a multi-modal deep learning model based on a residual network (ResNet) and multi-modal compact bilinear pooling to predict TMB status (i.e. TMB high (TMB_H) or TMB low(TMB_L)) directly from histopathological images and clinical data. We applied the model to CRC data from The Cancer Genome Atlas and compared it with four other popular methods, namely, ResNet18, ResNet50, VGG19 and AlexNet. We tested different TMB thresholds, namely, percentiles of 10%, 14.3%, 15%, 16.3%, 20%, 30% and 50%, to differentiate TMB_H and TMB_L.For the percentile of 14.3% (i.e. TMB value 20) and ResNet18, our model achieved an area under the receiver operating characteristic curve of 0.817 after 5-fold cross-validation, which was better than that of other compared models. In addition, we also found that TMB values were significantly associated with the tumor stage and N and M stages. Our study shows that deep learning models can predict TMB status from histopathological images and clinical information only, which is worth clinical application.

Convolutional Neural Networks for the Evaluation of Chronic and Inflammatory Lesions in Kidney Transplant Biopsies

In kidney transplant biopsies, both inflammation and chronic changes are important features that predict long-term graft survival. Quantitative scoring of these features is important for transplant diagnostics and kidney research. However, visual scoring is poorly reproducible and labor intensive. The goal of this study was to investigate the potential of convolutional neural networks (CNNs) to quantify inflammation and chronic features in kidney transplant biopsies. A structure segmentation CNN and a lymphocyte detection CNN were applied on 125 whole-slide image pairs of periodic acid-Schiff- and CD3-stained slides. The CNN results were used to quantify healthy and sclerotic glomeruli, interstitial fibrosis, tubular atrophy, and inflammation within both nonatrophic and atrophic tubuli, and in areas of interstitial fibrosis. The computed tissue features showed high correlation with Banff lesion scores of five pathologists (A.A., A.Dend., J.H.B., J.K., and T.N.). Analyses on a small subset showed a moderate correlation toward higher CD3⁺ cell density within scarred regions and higher CD3⁺ cell count inside atrophic tubuli correlated with long-term change of estimated glomerular filtration rate. The presented CNNs are valid tools to yield objective quantitative information on glomeruli number, fibrotic tissue, and inflammation within scarred and non-scarred kidney parenchyma in a reproducible manner. CNNs have the potential to improve kidney transplant diagnostics and will benefit the community as a novel method to generate surrogate end points for large-scale clinical studies.

Development and Validation of a Digital Analysis Method to Quantify CD3-immunostained T Lymphocytes in Whole Slide Images of Crohn's Disease Biopsies

The T-lymphocyte-mediated inflammation in Crohn's disease can be assessed by quantifying CD3-positive T-lymphocyte counts in colonic sections. We developed and validated a process to reliably quantify immunohistochemical marker-positive cells in a high-throughput setting using whole slide images (WSIs) of CD3-immunostained colonic and ileal tissue sections. In regions of interest (ROIs) and/or whole tissue sections of 40 WSIs from 36 patients with Crohn's disease, CD3-positive cells were quantified by an expert gastrointestinal pathologist (gold standard) and by image analysis algorithms developed with software from 3 independent vendors. Semiautomated quantification of CD3-positive cell counts estimated in 1 ROI per section were accurate when compared with manual analysis (Pearson correlation coefficient, 0.877 to 0.925). Biological variability was acceptable in digitally determined CD3-positive cell measures between 2 to 5 ROIs annotated on the same tissue section (coefficient of variation <25%). Results from computer-aided analysis of CD3-positive T lymphocytes in a whole tissue section and the average of results from 2 to 5 ROIs per tissue section lacked reliability (overestimation or underestimation and systematic bias), suggesting that absolute quantification of CD3-positive T lymphocytes in a whole tissue section may be more accurate. Semiautomated image analysis in WSIs demonstrated reproducible CD3-positive cell measures across 3 independent algorithms. A computer-aided digital image analysis method was developed and validated to quantify CD3-positive T lymphocytes in colonic and ileal biopsy sections from patients with Crohn's disease. Results support consideration of this digital analysis method for use in future Crohn's disease clinical studies.

Cross-Boosted Multi-Target Domain Adaptation for Multi-Modality Histopathology Image Translation and Segmentation

Recent digital pathology workflows mainly focus on mono-modality histopathology image analysis. However, they ignore the complementarity between Haematoxylin & Eosin (H&E) and Immunohistochemically (IHC) stained images, which can provide comprehensive gold standard for cancer diagnosis. To resolve this issue, we propose a cross-boosted multi-target domain adaptation pipeline for multi-modality histopathology images, which contains Cross-frequency Style-auxiliary Translation Network (CSTN) and Dual Cross-boosted Segmentation Network (DCSN). Firstly, CSTN achieves the one-to-many translation from fluorescence microscopy images to H&E and IHC images for providing source domain training data. To generate images with realistic color and texture, Cross-frequency Feature Transfer Module (CFTM) is developed to pertinently restructure and normalize high-frequency content and low-frequency style features from different domains. Then, DCSN fulfills multi-target domain adaptive segmentation, where a dual-branch encoder is introduced, and Bidirectional Cross-domain Boosting Module (BCBM) is designed to implement cross-modality information complementation through bidirectional inter-domain collaboration. Finally, we establish Multi-modality Thymus Histopathology (MThH) dataset, which is the largest publicly available H&E and IHC image benchmark. Experiments on MThH dataset and several public datasets show that the proposed pipeline outperforms state-of-the-art methods on both histopathology image translation and segmentation.

Recent advances and clinical applications of deep learning in medical image analysis

Deep learning has received extensive research interest in developing new medical image processing algorithms, and deep learning based models have been remarkably successful in a variety of medical imaging tasks to support disease detection and diagnosis. Despite the success, the further improvement of deep learning models in medical image analysis is majorly bottlenecked by the lack of large-sized and well-annotated datasets. In the past five years, many studies have focused on addressing this challenge. In this paper, we reviewed and summarized these recent studies to provide a comprehensive overview of applying deep learning methods in various medical image analysis tasks. Especially, we emphasize the latest progress and contributions of state-of-the-art unsupervised and semi-supervised deep learning in medical image analysis, which are summarized based on different application scenarios, including classification, segmentation, detection, and image registration. We also discuss major technical challenges and suggest possible solutions in the future research efforts.

Artificial Intelligence in Pediatric Pathology: The Extinction of a Medical Profession or the Key to a Bright Future?

Artificial Intelligence (AI) has become of increasing interest over the past decade. While digital image analysis (DIA) is already being used in radiology, it is still in its infancy in pathology. One of the reasons is that large-scale digitization of glass slides has only recently become available. With the advent of digital slide scanners, that digitize glass slides into whole slide images, many labs are now in a transition phase towards digital pathology. However, only few departments worldwide are currently fully digital. Digital pathology provides the ability to annotate large datasets and train computers to develop and validate robust algorithms, similar to radiology. In this opinionated overview, we will give a brief introduction into AI in pathology, discuss the potential positive and negative implications and speculate about the future role of AI in the field of pediatric pathology.

Jointly Optimized Spatial Histogram UNET Architecture (JOSHUA) for Adipose Tissue Segmentation

Objective. We aim to develop a machine learning algorithm to quantify adipose tissue deposition at surgical sites as a function of biomaterial implantation. Impact Statement. To our knowledge, this study is the first investigation to apply convolutional neural network (CNN) models to identify and segment adipose tissue in histological images from silk fibroin biomaterial implants. Introduction. When designing biomaterials for the treatment of various soft tissue injuries and diseases, one must consider the extent of adipose tissue deposition. In this work, we analyzed adipose tissue accumulation in histological images of sectioned silk fibroin-based biomaterials excised from rodents following subcutaneous implantation for 1, 2, 4, or 8 weeks. Current strategies for quantifying adipose tissue after biomaterial implantation are often tedious and prone to human bias during analysis. Methods. We used CNN models with novel spatial histogram layer(s) that can more accurately identify and segment regions of adipose tissue in hematoxylin and eosin (H&E) and Masson's trichrome stained images, allowing for determination of the optimal biomaterial formulation. We compared the method, Jointly Optimized Spatial Histogram UNET Architecture (JOSHUA), to the baseline UNET model and an extension of the baseline model, attention UNET, as well as to versions of the models with a supplemental attention-inspired mechanism (JOSHUA+ and UNET+). Results. The inclusion of histogram layer(s) in our models shows improved performance through qualitative and quantitative evaluation. Conclusion. Our results demonstrate that the proposed methods, JOSHUA and JOSHUA+, are highly beneficial for adipose tissue identification and localization. The new histological dataset and code used in our experiments are publicly available.