Optimization of an automated tumor-infiltrating lymphocyte algorithm for improved prognostication in primary melanoma

Advancements in computer vision and pathology: Unraveling the potential of artificial intelligence for precision diagnosis and beyond

The integration of computer vision into pathology through slide digitalization represents a transformative leap in the field's evolution. Traditional pathology methods, while reliable, are often time-consuming and susceptible to intra- and interobserver variability. In contrast, computer vision, empowered by artificial intelligence (AI) and machine learning (ML), promises revolutionary changes, offering consistent, reproducible, and objective results with ever-increasing speed and scalability. The applications of advanced algorithms and deep learning architectures like CNNs and U-Nets augment pathologists' diagnostic capabilities, opening new frontiers in automated image analysis. As these technologies mature and integrate into digital pathology workflows, they are poised to provide deeper insights into disease processes, quantify and standardize biomarkers, enhance patient outcomes, and automate routine tasks, reducing pathologists' workload. However, this transformative force calls for cross-disciplinary collaboration between pathologists, computer scientists, and industry innovators to drive research and development. While acknowledging its potential, this chapter addresses the limitations of AI in pathology, encompassing technical, practical, and ethical considerations during development and implementation.

Current status of artificial intelligence methods for skin cancer survival analysis: a scoping review

Skin cancer mortality rates continue to rise, and survival analysis is increasingly needed to understand who is at risk and what interventions improve outcomes. However, current statistical methods are limited by inability to synthesize multiple data types, such as patient genetics, clinical history, demographics, and pathology and reveal significant multimodal relationships through predictive algorithms. Advances in computing power and data science enabled the rise of artificial intelligence (AI), which synthesizes vast amounts of data and applies algorithms that enable personalized diagnostic approaches. Here, we analyze AI methods used in skin cancer survival analysis, focusing on supervised learning, unsupervised learning, deep learning, and natural language processing. We illustrate strengths and weaknesses of these approaches with examples. Our PubMed search yielded 14 publications meeting inclusion criteria for this scoping review. Most publications focused on melanoma, particularly histopathologic interpretation with deep learning. Such concentration on a single type of skin cancer amid increasing focus on deep learning highlight growing areas for innovation; however, it also demonstrates opportunity for additional analysis that addresses other types of cutaneous malignancies and expands the scope of prognostication to combine both genetic, histopathologic, and clinical data. Moreover, researchers may leverage multiple AI methods for enhanced benefit in analyses. Expanding AI to this arena may enable improved survival analysis, targeted treatments, and outcomes.

Tumor-Infiltrating Lymphocyte Recognition in Primary Melanoma by Deep Learning Convolutional Neural Network

The presence of tumor-infiltrating lymphocytes (TILs) is associated with a favorable prognosis of primary melanoma (PM). Recently, artificial intelligence (AI)-based approach in digital pathology was proposed for the standardized assessment of TILs on hematoxylin and eosin-stained whole slide images (WSIs). Herein, the study applied a new convolution neural network (CNN) analysis of PM WSIs to automatically assess the infiltration of TILs and extract a TIL score. A CNN was trained and validated in a retrospective cohort of 307 PMs including a training set (237 WSIs, 57,758 patches) and an independent testing set (70 WSIs, 29,533 patches). An AI-based TIL density index (AI-TIL) was identified after the classification of tumor patches by the presence or absence of TILs. The proposed CNN showed high performance in recognizing TILs in PM WSIs, showing 100% specificity and sensitivity on the testing set. The AI-based TIL index correlated with conventional TIL evaluation and clinical outcome. The AI-TIL index was an independent prognostic marker associated directly with a favorable prognosis. A fully automated and standardized AI-TIL appeared to be superior to conventional methods at differentiating the PM clinical outcome. Further studies are required to develop an easy-to-use tool to assist pathologists to assess TILs in the clinical evaluation of solid tumors.

Enhancing personalized immune checkpoint therapy by immune archetyping and pharmacological targeting

Immune checkpoint inhibitors (ICIs) are an expanding class of immunotherapeutic agents with the potential to cure cancer. Despite the outstanding clinical response in patient subsets, most individuals become refractory or develop resistance. Patient stratification and personalized immunotherapies are limited by the absence of predictive response markers. Recent findings show that dominant patterns of immune cell composition, T-cell status and heterogeneity, and spatiotemporal distribution of immune cells within the tumor microenvironment (TME) are becoming essential determinants of prognosis and therapeutic response. In this context, ICIs also function as investigational tools and proof of concept, allowing the validation of the identified mechanisms. After reviewing the current state of ICIs, this article will explore new comprehensive predictive markers for ICIs based on recent discoveries. We will discuss the recent establishment of a classification of TMEs into immune archetypes as a tool for personalized immune profiling, allowing patient stratification before ICI treatment. We will discuss the developing comprehension of T-cell diversity and its role in shaping the immune profile of patients. We describe the potential of strategies that score the mutual spatiotemporal modulation between T-cells and other cellular components of the TME. Additionally, we will provide an overview of a range of synthetic and naturally occurring or derived small molecules. We will compare compounds that were recently identified by in silico prediction to wet lab-validated drug candidates with the potential to function as ICIs and/or modulators of the cellular components of the TME.

Artificial intelligence-assisted dermatology diagnosis: From unimodal to multimodal

Artificial Intelligence (AI) is progressively permeating medicine, notably in the realm of assisted diagnosis. However, the traditional unimodal AI models, reliant on large volumes of accurately labeled data and single data type usage, prove insufficient to assist dermatological diagnosis. Augmenting these models with text data from patient narratives, laboratory reports, and image data from skin lesions, dermoscopy, and pathologies could significantly enhance their diagnostic capacity. Large-scale pre-training multimodal models offer a promising solution, exploiting the burgeoning reservoir of clinical data and amalgamating various data types. This paper delves into unimodal models' methodologies, applications, and shortcomings while exploring how multimodal models can enhance accuracy and reliability. Furthermore, integrating cutting-edge technologies like federated learning and multi-party privacy computing with AI can substantially mitigate patient privacy concerns in dermatological datasets and further fosters a move towards high-precision self-diagnosis. Diagnostic systems underpinned by large-scale pre-training multimodal models can facilitate dermatology physicians in formulating effective diagnostic and treatment strategies and herald a transformative era in healthcare.

Deep learning in computational dermatopathology of melanoma: A technical systematic literature review

Deep learning (DL) has become one of the major approaches in computational dermatopathology, evidenced by a significant increase in this topic in the current literature. We aim to provide a structured and comprehensive overview of peer-reviewed publications on DL applied to dermatopathology focused on melanoma. In comparison to well-published DL methods on non-medical images (e.g., classification on ImageNet), this field of application comprises a specific set of challenges, such as staining artifacts, large gigapixel images, and various magnification levels. Thus, we are particularly interested in the pathology-specific technical state-of-the-art. We also aim to summarize the best performances achieved thus far with respect to accuracy, along with an overview of self-reported limitations. Accordingly, we conducted a systematic literature review of peer-reviewed journal and conference articles published between 2012 and 2022 in the databases ACM Digital Library, Embase, IEEE Xplore, PubMed, and Scopus, expanded by forward and backward searches to identify 495 potentially eligible studies. After screening for relevance and quality, a total of 54 studies were included. We qualitatively summarized and analyzed these studies from technical, problem-oriented, and task-oriented perspectives. Our findings suggest that the technical aspects of DL for histopathology in melanoma can be further improved. The DL methodology was adopted later in this field, and still lacks the wider adoption of DL methods already shown to be effective for other applications. We also discuss upcoming trends toward ImageNet-based feature extraction and larger models. While DL has achieved human-competitive accuracy in routine pathological tasks, its performance on advanced tasks is still inferior to wet-lab testing (for example). Finally, we discuss the challenges impeding the translation of DL methods to clinical practice and provide insight into future research directions.

Spatial analyses of immune cell infiltration in cancer: current methods and future directions: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

Modern histologic imaging platforms coupled with machine learning methods have provided new opportunities to map the spatial distribution of immune cells in the tumor microenvironment. However, there exists no standardized method for describing or analyzing spatial immune cell data, and most reported spatial analyses are rudimentary. In this review, we provide an overview of two approaches for reporting and analyzing spatial data (raster versus vector-based). We then provide a compendium of spatial immune cell metrics that have been reported in the literature, summarizing prognostic associations in the context of a variety of cancers. We conclude by discussing two well-described clinical biomarkers, the breast cancer stromal tumor infiltrating lymphocytes score and the colon cancer Immunoscore, and describe investigative opportunities to improve clinical utility of these spatial biomarkers. © 2023 The Pathological Society of Great Britain and Ireland.

Deep learning-based scoring of tumour-infiltrating lymphocytes is prognostic in primary melanoma and predictive to PD-1 checkpoint inhibition in melanoma metastases

BACKGROUND

Recent advances in digital pathology have enabled accurate and standardised enumeration of tumour-infiltrating lymphocytes (TILs). Here, we aim to evaluate TILs as a percentage electronic TIL score (eTILs) and investigate its prognostic and predictive relevance in cutaneous melanoma.

METHODS

We included stage I to IV cutaneous melanoma patients and used hematoxylin-eosin-stained slides for TIL analysis. We assessed eTILs as a continuous and categorical variable using the published cut-off of 16.6% and applied Cox regression models to evaluate associations of eTILs with relapse-free, distant metastasis-free, and overall survival. We compared eTILs of the primaries with matched metastasis. Moreover, we assessed the predictive relevance of eTILs in therapy-naïve metastases according to the first-line therapy.

FINDINGS

We analysed 321 primary cutaneous melanomas and 191 metastatic samples. In simple Cox regression, tumour thickness (p < 0.0001), presence of ulceration (p = 0.0001) and eTILs ≤16.6% (p = 0.0012) were found to be significant unfavourable prognostic factors for RFS. In multiple Cox regression, eTILs ≤16.6% (p = 0.0161) remained significant and downgraded the current staging. Lower eTILs in the primary tissue was associated with unfavourable relapse-free (p = 0.0014) and distant metastasis-free survival (p = 0.0056). In multiple Cox regression adjusted for tumour thickness and ulceration, eTILs as continuous remained significant (p = 0.019). When comparing TILs in primary tissue and corresponding metastasis of the same patient, eTILs in metastases was lower than in primary melanomas (p < 0.0001). In therapy-naïve metastases, an eTILs >12.2% was associated with longer progression-free survival (p = 0.037) and melanoma-specific survival (p = 0.0038) in patients treated with anti-PD-1-based immunotherapy. In multiple Cox regression, lactate dehydrogenase (p < 0.0001) and eTILs ≤12.2% (p = 0.0130) were significantly associated with unfavourable melanoma-specific survival.

INTERPRETATION

Assessment of TILs is prognostic in primary melanoma samples, and the eTILs complements staging. In therapy-naïve metastases, eTILs ≤12.2% is predictive of unfavourable survival outcomes in patients receiving anti-PD-1-based therapy.

FUNDING

See a detailed list of funding bodies in the Acknowledgements section at the end of the manuscript.

Diagnostic and Prognostic Deep Learning Applications for Histological Assessment of Cutaneous Melanoma

Melanoma is among the most devastating human malignancies. Accurate diagnosis and prognosis are essential to offer optimal treatment. Histopathology is the gold standard for establishing melanoma diagnosis and prognostic features. However, discrepancies often exist between pathologists, and analysis is costly and time-consuming. Deep-learning algorithms are deployed to improve melanoma diagnosis and prognostication from histological images of melanoma. In recent years, the development of these machine-learning tools has accelerated, and machine learning is poised to become a clinical tool to aid melanoma histology. Nevertheless, a review of the advances in machine learning in melanoma histology was lacking. We performed a comprehensive literature search to provide a complete overview of the recent advances in machine learning in the assessment of melanoma based on hematoxylin eosin digital pathology images. In our work, we review 37 recent publications, compare the methods and performance of the reviewed studies, and highlight the variety of promising machine-learning applications in melanoma histology.

Quantitative Assessment of Tumor-Infiltrating Lymphocytes Using Machine Learning Predicts Survival in Muscle-Invasive Bladder Cancer

(1) Purpose: Although assessment of tumor-infiltrating lymphocytes (TILs) has been acknowledged to have important predictive prognostic value in muscle-invasive bladder cancer (MIBC), it is limited by inter- and intra-observer variability, hampering widespread clinical application. We aimed to evaluate the prognostic value of quantitative TILs score based on a machine learning (ML) algorithm to identify MIBC patients who might benefit from immunotherapy or the de-escalation of therapy. (2) Methods: We constructed an artificial neural network classifier for tumor cells, lymphocytes, stromal cells, and “ignore” cells from hematoxylin-and-eosin-stained slide images using the QuPath open source software. We defined four unique TILs variables based on ML to analyze TILs measurements. Pathological slide images from 133 MIBC patients were retrospectively collected as the discovery set to determine the optimal association of ML-read TILs variables with patient survival outcomes. For validation, we evaluated an independent external validation set consisting of 247 MIBC patients. (3) Results: We found that all four TILs variables had significant prognostic associations with survival outcomes in MIBC patients (p < 0.001 for all comparisons), with higher TILs score being associated with better prognosis. Univariate and multivariate Cox regression analyses demonstrated that electronic TILs (eTILs) variables were independently associated with overall survival after adjustment for clinicopathological factors including age, sex, and pathological stage (p < 0.001 for all analyses). Results analyzed in different subgroups showed that the eTILs variable was a strong prognostic factor that was not redundant with pre-existing clinicopathological features (p < 0.05 for all analyses). (4) Conclusion: ML-driven cell classifier-defined TILs variables were robust and independent prognostic factors in two independent cohorts of MIBC patients. eTILs have the potential to identify a subset of high-risk stage II or stage III-IV MIBC patients who might benefit from adjuvant immunotherapy.

Multi-modality artificial intelligence in digital pathology

In common medical procedures, the time-consuming and expensive nature of obtaining test results plagues doctors and patients. Digital pathology research allows using computational technologies to manage data, presenting an opportunity to improve the efficiency of diagnosis and treatment. Artificial intelligence (AI) has a great advantage in the data analytics phase. Extensive research has shown that AI algorithms can produce more up-to-date and standardized conclusions for whole slide images. In conjunction with the development of high-throughput sequencing technologies, algorithms can integrate and analyze data from multiple modalities to explore the correspondence between morphological features and gene expression. This review investigates using the most popular image data, hematoxylin-eosin stained tissue slide images, to find a strategic solution for the imbalance of healthcare resources. The article focuses on the role that the development of deep learning technology has in assisting doctors' work and discusses the opportunities and challenges of AI.

Multiplexed Immunohistochemistry and Digital Pathology as the Foundation for Next-Generation Pathology in Melanoma: Methodological Comparison and Future Clinical Applications

The state-of-the-art for melanoma treatment has recently witnessed an enormous revolution, evolving from a chemotherapeutic, "one-drug-for-all" approach, to a tailored molecular- and immunological-based approach with the potential to make personalized therapy a reality. Nevertheless, methods still have to improve a lot before these can reliably characterize all the tumoral features that make each patient unique. While the clinical introduction of next-generation sequencing has made it possible to match mutational profiles to specific targeted therapies, improving response rates to immunotherapy will similarly require a deep understanding of the immune microenvironment and the specific contribution of each component in a patient-specific way. Recent advancements in artificial intelligence and single-cell profiling of resected tumor samples are paving the way for this challenging task. In this review, we provide an overview of the state-of-the-art in artificial intelligence and multiplexed immunohistochemistry in pathology, and how these bear the potential to improve diagnostics and therapy matching in melanoma. A major asset of in-situ single-cell profiling methods is that these preserve the spatial distribution of the cells in the tissue, allowing researchers to not only determine the cellular composition of the tumoral microenvironment, but also study tissue sociology, making inferences about specific cell-cell interactions and visualizing distinctive cellular architectures - all features that have an impact on anti-tumoral response rates. Despite the many advantages, the introduction of these approaches requires the digitization of tissue slides and the development of standardized analysis pipelines which pose substantial challenges that need to be addressed before these can enter clinical routine.