Deep-Learning-Based Characterization of Tumor-Infiltrating Lymphocytes in Breast Cancers From Histopathology Images and Multiomics Data

Leveraging Deep Learning for Immune Cell Quantification and Prognostic Evaluation in Radiotherapy-Treated Oropharyngeal Squamous Cell Carcinomas

The tumor microenvironment plays a critical role in cancer progression and therapeutic responsiveness, with the tumor immune microenvironment (TIME) being a key modulator. In head and neck squamous cell carcinomas (HNSCCs), immune cell infiltration significantly influences the response to radiotherapy (RT). A better understanding of the TIME in HNSCCs could help identify patients most likely to benefit from combining RT with immunotherapy. Standardized, cost-effective methods for studying TIME in HNSCCs are currently lacking. This study aims to leverage deep learning (DL) to quantify immune cell densities using immunohistochemistry in untreated oropharyngeal squamous cell carcinoma (OPSCC) biopsies of patients scheduled for curative RT and assess their prognostic value. We analyzed 84 pretreatment formalin-fixed paraffin-embedded tumor biopsies from OPSCC patients. Immunohistochemistry was performed for CD3, CD8, CD20, CD163, and FOXP3, and whole slide images were digitized for analysis using a U-Net-based DL model. Two quantification approaches were applied: a cell-counting method and an area-based method. These methods were applied to stained regions. The DL model achieved high accuracy in detecting stained cells across all biomarkers. Strong correlations were found between our DL pipeline, the HALO Image Analysis Platform, and the open-source QuPath software for estimating immune cell densities. Our DL pipeline provided an accurate and reproducible approach for quantifying immune cells in OPSCC. The area-based method demonstrated superior prognostic value for recurrence-free survival, when compared with the cell-counting method. Elevated densities of CD3, CD8, CD20, and FOXP3 were associated with improved recurrence-free survival, whereas CD163 showed no significant prognostic association. These results highlight the potential of DL in digital pathology for assessing TIME and predicting patient outcomes.

Automated deep learning-based assessment of tumour-infiltrating lymphocyte density determines prognosis in colorectal cancer

BACKGROUND

The presence of tumour-infiltrating lymphocytes (TILs) is a well-established prognostic biomarker across multiple cancer types, with higher TIL counts being associated with lower recurrence rates and improved patient survival. We aimed to examine whether an automated intraepithelial TIL (iTIL) assessment could stratify patients by risk, with the ability to generalise across independent patient cohorts, using routine H&E slides of colorectal cancer (CRC). To our knowledge, no other existing fully automated iTIL system has demonstrated this capability.

METHODS

An automated method employing deep neural networks was developed to enumerate iTILs in H&E slides of CRC. The method was applied to a Stage III discovery cohort (n = 353) to identify an optimal threshold of 17 iTILs per-mm2 tumour for stratifying relapse-free survival. Using this threshold, patients from two independent Stage II-III validation cohorts (n = 1070, n = 885) were classified as "TIL-High" or "TIL-Low".

RESULTS

Significant stratification was observed in terms of overall survival for a combined validation cohort univariate (HR 1.67, 95%CI 1.39-2.00; p < 0.001) and multivariate (HR 1.37, 95%CI 1.13-1.66; p = 0.001) analysis. Our iTIL classifier was an independent prognostic factor within proficient DNA mismatch repair (pMMR) Stage II CRC cases with clinical high-risk features. Of these, those classified as TIL-High had outcomes similar to pMMR clinical low risk cases, and those classified TIL-Low had significantly poorer outcomes (univariate HR 2.38, 95%CI 1.57-3.61; p < 0.001, multivariate HR 2.17, 95%CI 1.42-3.33; p < 0.001).

CONCLUSIONS

Our deep learning method is the first fully automated system to stratify patient outcome by analysing TILs in H&E slides of CRC, that has shown generalisation capabilities across multiple independent cohorts.

Computational pathology: A survey review and the way forward

Computational Pathology (CPath) is an interdisciplinary science that augments developments of computational approaches to analyze and model medical histopathology images. The main objective for CPath is to develop infrastructure and workflows of digital diagnostics as an assistive CAD system for clinical pathology, facilitating transformational changes in the diagnosis and treatment of cancer that are mainly address by CPath tools. With evergrowing developments in deep learning and computer vision algorithms, and the ease of the data flow from digital pathology, currently CPath is witnessing a paradigm shift. Despite the sheer volume of engineering and scientific works being introduced for cancer image analysis, there is still a considerable gap of adopting and integrating these algorithms in clinical practice. This raises a significant question regarding the direction and trends that are undertaken in CPath. In this article we provide a comprehensive review of more than 800 papers to address the challenges faced in problem design all-the-way to the application and implementation viewpoints. We have catalogued each paper into a model-card by examining the key works and challenges faced to layout the current landscape in CPath. We hope this helps the community to locate relevant works and facilitate understanding of the field's future directions. In a nutshell, we oversee the CPath developments in cycle of stages which are required to be cohesively linked together to address the challenges associated with such multidisciplinary science. We overview this cycle from different perspectives of data-centric, model-centric, and application-centric problems. We finally sketch remaining challenges and provide directions for future technical developments and clinical integration of CPath. For updated information on this survey review paper and accessing to the original model cards repository, please refer to GitHub. Updated version of this draft can also be found from arXiv.

Enhancing AI Research for Breast Cancer: A Comprehensive Review of Tumor-Infiltrating Lymphocyte Datasets

The field of immunology is fundamental to our understanding of the intricate dynamics of the tumor microenvironment. In particular, tumor-infiltrating lymphocyte (TIL) assessment emerges as essential aspect in breast cancer cases. To gain comprehensive insights, the quantification of TILs through computer-assisted pathology (CAP) tools has become a prominent approach, employing advanced artificial intelligence models based on deep learning techniques. The successful recognition of TILs requires the models to be trained, a process that demands access to annotated datasets. Unfortunately, this task is hampered not only by the scarcity of such datasets, but also by the time-consuming nature of the annotation phase required to create them. Our review endeavors to examine publicly accessible datasets pertaining to the TIL domain and thereby become a valuable resource for the TIL community. The overall aim of the present review is thus to make it easier to train and validate current and upcoming CAP tools for TIL assessment by inspecting and evaluating existing publicly available online datasets.

Deep learning-based approaches for multi-omics data integration and analysis

BACKGROUND

The rapid growth of deep learning, as well as the vast and ever-growing amount of available data, have provided ample opportunity for advances in fusion and analysis of complex and heterogeneous data types. Different data modalities provide complementary information that can be leveraged to gain a more complete understanding of each subject. In the biomedical domain, multi-omics data includes molecular (genomics, transcriptomics, proteomics, epigenomics, metabolomics, etc.) and imaging (radiomics, pathomics) modalities which, when combined, have the potential to improve performance on prediction, classification, clustering and other tasks. Deep learning encompasses a wide variety of methods, each of which have certain strengths and weaknesses for multi-omics integration.

METHOD

In this review, we categorize recent deep learning-based approaches by their basic architectures and discuss their unique capabilities in relation to one another. We also discuss some emerging themes advancing the field of multi-omics integration.

RESULTS

Deep learning-based multi-omics integration methods were categorized broadly into non-generative (feedforward neural networks, graph convolutional neural networks, and autoencoders) and generative (variational methods, generative adversarial models, and a generative pretrained model). Generative methods have the advantage of being able to impose constraints on the shared representations to enforce certain properties or incorporate prior knowledge. They can also be used to generate or impute missing modalities. Recent advances achieved by these methods include the ability to handle incomplete data as well as going beyond the traditional molecular omics data types to integrate other modalities such as imaging data.

CONCLUSION

We expect to see further growth in methods that can handle missingness, as this is a common challenge in working with complex and heterogeneous data. Additionally, methods that integrate more data types are expected to improve performance on downstream tasks by capturing a comprehensive view of each sample.

Automated scoring methods for quantitative interpretation of Tumour infiltrating lymphocytes (TILs) in breast cancer: a systematic review

Tumour microenvironment (TME) of breast cancer mainly comprises malignant, stromal, immune, and tumour infiltrating lymphocyte (TILs). Assessment of TILs is crucial for determining the disease's prognosis. Manual TIL assessments are hampered by multiple limitations, including low precision, poor inter-observer reproducibility, and time consumption. In response to these challenges, automated scoring emerges as a promising approach. The aim of this systematic review is to assess the evidence on the approaches and performance of automated scoring methods for TILs assessment in breast cancer. This review presents a comprehensive compilation of studies related to automated scoring of TILs, sourced from four databases (Web of Science, Scopus, Science Direct, and PubMed), employing three primary keywords (artificial intelligence, breast cancer, and tumor-infiltrating lymphocytes). The PICOS framework was employed for study eligibility, and reporting adhered to the PRISMA guidelines. The initial search yielded a total of 1910 articles. Following screening and examination, 27 studies met the inclusion criteria and data were extracted for the review. The findings indicate a concentration of studies on automated TILs assessment in developed countries, specifically the United States and the United Kingdom. From the analysis, a combination of sematic segmentation and object detection (n = 10, 37%) and convolutional neural network (CNN) (n = 11, 41%), become the most frequent automated task and ML approaches applied for model development respectively. All models developed their own ground truth datasets for training and validation, and 59% of the studies assessed the prognostic value of TILs. In conclusion, this analysis contends that automated scoring methods for TILs assessment of breast cancer show significant promise for commodification and application within clinical settings.

Artificial intelligence: illuminating the depths of the tumor microenvironment

Artificial intelligence (AI) can acquire characteristics that are not yet known to humans through extensive learning, enabling to handle large amounts of pathology image data. Divided into machine learning and deep learning, AI has the advantage of handling large amounts of data and processing image analysis, consequently it also has a great potential in accurately assessing tumour microenvironment (TME) models. With the complex composition of the TME, in-depth study of TME contributes to new ideas for treatment, assessment of patient response to postoperative therapy and prognostic prediction. This leads to a review of the development of AI's application in TME assessment in this study, provides an overview of AI techniques applied to medicine, delves into the application of AI in analysing the quantitative and spatial location characteristics of various cells (tumour cells, immune and non-immune cells) in the TME, reveals the predictive prognostic value of TME and provides new ideas for tumour therapy, highlights the great potential for clinical applications. In addition, a discussion of its limitations and encouraging future directions for its practical clinical application is presented.

Tumor-Infiltrating Lymphocyte Scoring in Neoadjuvant-Treated Breast Cancer

Neoadjuvant chemotherapy (NAC) is now the standard of care for patients with locally advanced breast cancer (BC). TIL scoring is prognostic and adds predictive value to the residual cancer burden evaluation after NAC. However, NAC induces changes in the tumor, and the reliability of TIL scoring in post-NAC samples has not yet been studied. H&E- and dual CD3/CD20 chromogenic IHC-stained tissues were scored for stromal and intra-tumoral TIL by two experienced pathologists on pre- and post-treatment BC tissues. Digital TIL scoring was performed using the HALO® image analysis software (version 2.2). In patients with residual disease, we show a good inter-pathologist correlation for stromal TIL on H&E-stained tissues (CCC value 0.73). A good correlation for scoring with both staining methods (CCC 0.81) and the digital TIL scoring (CCC 0.77) was also observed. Overall concordance for TIL scoring in patients with a complete response was however poor. This study reveals there is good reliability for TIL scoring in patients with detectable residual tumors after NAC treatment, which is comparable to the scoring of untreated breast cancer patients. Based on the good consistency observed with digital TIL scoring, the development of a validated algorithm in the future might be advantageous.

From Pixels to Prognosis: A Survey on AI-Driven Cancer Patient Survival Prediction Using Digital Histology Images

Survival analysis is an integral part of medical statistics that is extensively utilized to establish prognostic indices for mortality or disease recurrence, assess treatment efficacy, and tailor effective treatment plans. The identification of prognostic biomarkers capable of predicting patient survival is a primary objective in the field of cancer research. With the recent integration of digital histology images into routine clinical practice, a plethora of Artificial Intelligence (AI)-based methods for digital pathology has emerged in scholarly literature, facilitating patient survival prediction. These methods have demonstrated remarkable proficiency in analyzing and interpreting whole slide images, yielding results comparable to those of expert pathologists. The complexity of AI-driven techniques is magnified by the distinctive characteristics of digital histology images, including their gigapixel size and diverse tissue appearances. Consequently, advanced patch-based methods are employed to effectively extract features that correlate with patient survival. These computational methods significantly enhance survival prediction accuracy and augment prognostic capabilities in cancer patients. The review discusses the methodologies employed in the literature, their performance metrics, ongoing challenges, and potential solutions for future advancements. This paper explains survival analysis and feature extraction methods for analyzing cancer patients. It also compiles essential acronyms related to cancer precision medicine. Furthermore, it is noteworthy that this is the inaugural review paper in the field. The target audience for this interdisciplinary review comprises AI practitioners, medical statisticians, and progressive oncologists who are enthusiastic about translating AI-driven solutions into clinical practice. We expect this comprehensive review article to guide future research directions in the field of cancer research.

PseudoCell: Hard Negative Mining as Pseudo Labeling for Deep Learning-Based Centroblast Cell Detection

Background: Deep learning models for patch classification in whole-slide images (WSIs) have shown promise in assisting follicular lymphoma grading. However, these models often require pathologists to identify centroblasts and manually provide refined labels for model optimization. Objective: To address this limitation, we propose PseudoCell, an object detection framework for automated centroblast detection in WSI, eliminating the need for extensive pathologist's refined labels. Methods: PseudoCell leverages a combination of pathologist-provided centroblast labels and pseudo-negative labels generated from undersampled false-positive predictions based on cell morphology features. This approach reduces the reliance on time-consuming manual annotations. Results: Our framework significantly reduces the workload for pathologists by accurately identifying and narrowing down areas of interest containing centroblasts. Depending on the confidence threshold, PseudoCell can eliminate 58.18-99.35% of irrelevant tissue areas on WSI, streamlining the diagnostic process. Conclusion: This study presents PseudoCell as a practical and efficient prescreening method for centroblast detection, eliminating the need for refined labels from pathologists. The discussion section provides detailed guidance for implementing PseudoCell in clinical practice.

Image-based multiplex immune profiling of cancer tissues: translational implications. A report of the International Immuno-oncology Biomarker Working Group on Breast Cancer

Recent advances in the field of immuno-oncology have brought transformative changes in the management of cancer patients. The immune profile of tumours has been found to have key value in predicting disease prognosis and treatment response in various cancers. Multiplex immunohistochemistry and immunofluorescence have emerged as potent tools for the simultaneous detection of multiple protein biomarkers in a single tissue section, thereby expanding opportunities for molecular and immune profiling while preserving tissue samples. By establishing the phenotype of individual tumour cells when distributed within a mixed cell population, the identification of clinically relevant biomarkers with high-throughput multiplex immunophenotyping of tumour samples has great potential to guide appropriate treatment choices. Moreover, the emergence of novel multi-marker imaging approaches can now provide unprecedented insights into the tumour microenvironment, including the potential interplay between various cell types. However, there are significant challenges to widespread integration of these technologies in daily research and clinical practice. This review addresses the challenges and potential solutions within a structured framework of action from a regulatory and clinical trial perspective. New developments within the field of immunophenotyping using multiplexed tissue imaging platforms and associated digital pathology are also described, with a specific focus on translational implications across different subtypes of cancer. © 2024 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Invasive carcinoma segmentation in whole slide images using MS-ResMTUNet

Identifying the invasive cancer area is a crucial step in the automated diagnosis of digital pathology slices of the breast. When examining the pathological sections of patients with invasive ductal carcinoma, several evaluations are required specifically for the invasive cancer area. However, currently there is little work that can effectively distinguish the invasive cancer area from the ductal carcinoma in situ in whole slide images. To address this issue, we propose a novel architecture named ResMTUnet that combines the strengths of vision transformer and CNN, and uses multi-task learning to achieve accurate invasive carcinoma recognition and segmentation in breast cancer. Furthermore, we introduce a multi-scale input model based on ResMTUnet with conditional random field, named MS-ResMTUNet, to perform segmentation on WSIs. Our systematic experimentation has shown that the proposed network outperforms other competitive methods and effectively segments invasive carcinoma regions in WSIs. This lays a solid foundation for subsequent analysis of breast pathological slides in the future. The code is available at: https://github.com/liuyiqing2018/MS-ResMTUNet.

A systematic analysis of deep learning in genomics and histopathology for precision oncology

BACKGROUND

Digitized histopathological tissue slides and genomics profiling data are available for many patients with solid tumors. In the last 5 years, Deep Learning (DL) has been broadly used to extract clinically actionable information and biological knowledge from pathology slides and genomic data in cancer. In addition, a number of recent studies have introduced multimodal DL models designed to simultaneously process both images from pathology slides and genomic data as inputs. By comparing patterns from one data modality with those in another, multimodal DL models are capable of achieving higher performance compared to their unimodal counterparts. However, the application of these methodologies across various tumor entities and clinical scenarios lacks consistency.

METHODS

Here, we present a systematic survey of the academic literature from 2010 to November 2023, aiming to quantify the application of DL for pathology, genomics, and the combined use of both data types. After filtering 3048 publications, our search identified 534 relevant articles which then were evaluated by basic (diagnosis, grading, subtyping) and advanced (mutation, drug response and survival prediction) application types, publication year and addressed cancer tissue.

RESULTS

Our analysis reveals a predominant application of DL in pathology compared to genomics. However, there is a notable surge in DL incorporation within both domains. Furthermore, while DL applied to pathology primarily targets the identification of histology-specific patterns in individual tissues, DL in genomics is more commonly used in a pan-cancer context. Multimodal DL, on the contrary, remains a niche topic, evidenced by a limited number of publications, primarily focusing on prognosis predictions.

CONCLUSION

In summary, our quantitative analysis indicates that DL not only has a well-established role in histopathology but is also being successfully integrated into both genomic and multimodal applications. In addition, there is considerable potential in multimodal DL for harnessing further advanced tasks, such as predicting drug response. Nevertheless, this review also underlines the need for further research to bridge the existing gaps in these fields.

Improving Tumor-Infiltrating Lymphocytes Score Prediction in Breast Cancer with Self-Supervised Learning

Tumor microenvironment (TME) plays a pivotal role in immuno-oncology, which investigates the intricate interactions between tumors and the human immune system. Specifically, tumor-infiltrating lymphocytes (TILs) are crucial biomarkers for evaluating the prognosis of breast cancer patients and have the potential to refine immunotherapy precision and accurately identify tumor cells in specific cancer types. In this study, we conducted tissue segmentation and lymphocyte detection tasks to predict TIL scores by employing self-supervised learning (SSL) model-based approaches capable of addressing limited labeling data issues. Our experiments showed a 1.9% improvement in tissue segmentation and a 2% improvement in lymphocyte detection over the ImageNet pre-training model. Using these SSL-based models, we achieved a TIL score of 0.718 with a 4.4% improvement. In particular, when trained with only 10% of the entire dataset, the SwAV pre-trained model exhibited a superior performance over other models. Our work highlights improved tissue segmentation and lymphocyte detection using the SSL model with less labeled data for TIL score prediction.

Built to Last? Reproducibility and Reusability of Deep Learning Algorithms in Computational Pathology

Recent progress in computational pathology has been driven by deep learning. While code and data availability are essential to reproduce findings from preceding publications, ensuring a deep learning model's reusability is more challenging. For that, the codebase should be well-documented and easy to integrate into existing workflows and models should be robust toward noise and generalizable toward data from different sources. Strikingly, only a few computational pathology algorithms have been reused by other researchers so far, let alone employed in a clinical setting. To assess the current state of reproducibility and reusability of computational pathology algorithms, we evaluated peer-reviewed articles available in PubMed, published between January 2019 and March 2021, in 5 use cases: stain normalization; tissue type segmentation; evaluation of cell-level features; genetic alteration prediction; and inference of grading, staging, and prognostic information. We compiled criteria for data and code availability and statistical result analysis and assessed them in 160 publications. We found that only one-quarter (41 of 160 publications) made code publicly available. Among these 41 studies, three-quarters (30 of 41) analyzed their results statistically, half of them (20 of 41) released their trained model weights, and approximately a third (16 of 41) used an independent cohort for evaluation. Our review is intended for both pathologists interested in deep learning and researchers applying algorithms to computational pathology challenges. We provide a detailed overview of publications with published code in the field, list reusable data handling tools, and provide criteria for reproducibility and reusability.

A Review of AI-Based Radiomics and Computational Pathology Approaches in Triple-Negative Breast Cancer: Current Applications and Perspectives

Breast cancer is one of the most common and deadly cancers worldwide. Approximately, 20% of all breast cancers are characterized as triple negative (TNBC). TNBC typically is associated with a poorer prognosis relative to other breast cancer subtypes. Due to its aggressiveness and lack of response to hormonal therapy, conventional cytotoxic chemotherapy is the usual treatment; however, this treatment is not always effective, and an important percentage of patients develop recurrence. More recently, immunotherapy has started to be used on some populations with TNBC showing promising results. Unfortunately, immunotherapy is only applicable to a minority of patients and responses in metastatic TNBC have overall been modest in comparison to other cancer types. This situation evidences the need for developing effective biomarkers that help to stratify and personalize patient management. Thanks to recent advances in artificial intelligence (AI), there has been an increasing interest in its use for medical applications aiming at supporting clinical decision making. Several works have used AI in combination with diagnostic medical imaging, more specifically radiology and digitized histopathological tissue samples, aiming to extract disease-specific information that is difficult to quantify by the human eye. These works have demonstrated that analysis of such images in the context of TNBC has great potential for (1) risk-stratifying patients to identify those patients who are more likely to experience disease recurrence or die from the disease and (2) predicting pathologic complete response. In this manuscript, we present an overview on AI and its integration with radiology and histopathological images for developing prognostic and predictive approaches for TNBC. We present state of the art approaches in the literature and discuss the opportunities and challenges with developing AI algorithms regarding further development and clinical deployment, including identifying those patients who may benefit from certain treatments (e.g., adjuvant chemotherapy) from those who may not and thereby should be directed toward other therapies, discovering potential differences between populations, and identifying disease subtypes.

Recent advancements in artificial intelligence for breast cancer: Image augmentation, segmentation, diagnosis, and prognosis approaches

Breast cancer is a significant global health burden, with increasing morbidity and mortality worldwide. Early screening and accurate diagnosis are crucial for improving prognosis. Radiographic imaging modalities such as digital mammography (DM), digital breast tomosynthesis (DBT), magnetic resonance imaging (MRI), ultrasound (US), and nuclear medicine techniques, are commonly used for breast cancer assessment. And histopathology (HP) serves as the gold standard for confirming malignancy. Artificial intelligence (AI) technologies show great potential for quantitative representation of medical images to effectively assist in segmentation, diagnosis, and prognosis of breast cancer. In this review, we overview the recent advancements of AI technologies for breast cancer, including 1) improving image quality by data augmentation, 2) fast detection and segmentation of breast lesions and diagnosis of malignancy, 3) biological characterization of the cancer such as staging and subtyping by AI-based classification technologies, 4) prediction of clinical outcomes such as metastasis, treatment response, and survival by integrating multi-omics data. Then, we then summarize large-scale databases available to help train robust, generalizable, and reproducible deep learning models. Furthermore, we conclude the challenges faced by AI in real-world applications, including data curating, model interpretability, and practice regulations. Besides, we expect that clinical implementation of AI will provide important guidance for the patient-tailored management.

End-to-end deep learning radiomics: development and validation of a novel attention-based aggregate convolutional neural network to distinguish breast diffuse large B-cell lymphoma from breast invasive ductal carcinoma

Background

Apart from invasive pathological examination, there is no effective method to differentiate breast diffuse large B-cell lymphoma (DLBCL) from breast invasive ductal carcinoma (IDC). In this study, we aimed to develop and validate an effective deep learning radiomics model to discriminate between DLBCL and IDC.

Methods

A total of 324 breast nodules from 236 patients with baseline 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography/computed tomography (PET/CT) were retrospectively analyzed. After grouping breast DLBCL and breast IDC patients, external and internal datasets were divided according to the data collected by different centers. Preprocessing was then used to process the original PET/CT images and an attention-based aggregate convolutional neural network (AACNN) model was designed. The AACNN model was trained using patches of CT or PET tumor images and optimized with an improved loss function. The final ensemble predictive model was built using distance weight voting. Finally, the model performance was evaluated and statistically verified.

Results

A total of 249 breast nodules from Fudan University Shanghai Cancer Center (FUSCC) and 75 breast nodules from Shanghai Proton and Heavy Ion Center (SPHIC) were selected as internal and external datasets, respectively. On the internal testing, our method yielded an area under the curve (AUC), accuracy (ACC), sensitivity (SEN), specificity (SPE), positive predictive value (PPV), negative predictive value (NPV), and harmonic mean of precision and sensitivity (F1) of 0.886, 83.0%, 80.9%, 85.0%, 84.8%, 81.2%, and 0.828, respectively. Meanwhile on the external testing, the results were 0.788, 71.6%, 61.4%, 84.7%, 84.0%, 62.6%, and 0.709, respectively.

Conclusions

Our study outlines a deep learning radiomics method which can automatically, noninvasively, and accurately differentiate breast DLBCL from breast IDC, which will be more in line with the needs and strategies of precision medicine, individualized diagnosis, and treatment.

Advances in the application of computational pathology in diagnosis, immunomicroenvironment recognition, and immunotherapy evaluation of breast cancer: a narrative review

BACKGROUND

Breast cancer (BC) is a prevalent and highly lethal malignancy affecting women worldwide. Immunotherapy has emerged as a promising therapeutic strategy for BC, offering potential improvements in patient survival. Neoadjuvant therapy (NAT) has also gained significant clinical traction. With the advancement of computer technology, Artificial Intelligence (AI) has been increasingly applied in pathology research, expanding and redefining the scope of the field. This narrative review aims to provide a comprehensive overview of the current literature on the application of computational pathology in BC, specifically focusing on diagnosis, immune microenvironment recognition, and the evaluation of immunotherapy and NAT response.

METHODS

A thorough examination of relevant literature was conducted, focusing on studies investigating the role of computational pathology in BC diagnosis, immune microenvironment recognition, and immunotherapy and NAT assessment.

RESULTS

The application of computational pathology has shown significant potential in BC management. AI-based techniques enable improved diagnosis and classification of BC subtypes, enhance the identification and characterization of the immune microenvironment, and facilitate the evaluation of immunotherapy and NAT response. However, challenges related to data quality, standardization, and algorithm development still need to be addressed.

CONCLUSION

The integration of computational pathology and AI has transformative implications for BC patient care. By leveraging AI-based technologies, clinicians can make more informed decisions in diagnosis, treatment planning, and therapeutic response assessment. Future research should focus on refining AI algorithms, addressing technical challenges, and conducting large-scale clinical validation studies to facilitate the translation of computational pathology into routine clinical practice for BC patients.

Prediction of pathological complete response to neoadjuvant chemotherapy in locally advanced breast cancer by using a deep learning model with 18F-FDG PET/CT

OBJECTIVES

The aim of the study is 18F-FDG PET/CT imaging by using deep learning method are predictive for pathological complete response pCR after Neoadjuvant chemotherapy (NAC) in locally advanced breast cancer (LABC).

INTRODUCTION

NAC is the standard treatment for locally advanced breast cancer (LABC). Pathological complete response (pCR) after NAC is considered a good predictor of disease-free survival (DFS) and overall survival (OS).Therefore, there is a need to develop methods that can predict the pCR at the time of diagnosis.

METHODS

This article was designed as a retrospective chart study.For the convolutional neural network model, a total of 355 PET/CT images of 31 patients were used. All patients had primary breast surgery after completing NAC.

RESULTS

Pathological complete response was obtained in a total of 9 patients. The study results show that our proposed deep convolutional neural networks model achieved a remarkable success with an accuracy of 84.79% to predict pathological complete response.

CONCLUSION

It was concluded that deep learning methods can predict breast cancer treatment.

Computational pathology to improve biomarker testing in breast cancer: how close are we?

The recent advancements in breast cancer precision medicine have highlighted the urgency for the precise and reproducible characterization of clinically actionable biomarkers. Despite numerous standardization efforts, biomarker testing by conventional methodologies is challenged by several issues such as high inter-observer variabilities, the spatial heterogeneity of biomarkers expression, and technological heterogeneity. In this respect, artificial intelligence-based digital pathology approaches are being increasingly recognized as promising methods for biomarker testing and subsequently improved clinical management. Here, we provide an overview on the most recent advances for artificial intelligence-assisted biomarkers testing in breast cancer, with a particular focus on tumor-infiltrating lymphocytes, programmed death-ligand 1, phosphatidylinositol-3 kinase catalytic alpha, and estrogen receptor 1. Challenges and solutions for this integrative analysis in pathology laboratories are also provided.

ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification

BACKGROUND AND OBJECTIVE

Histopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers.

METHODS

Here we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model.

RESULTS

The main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision.

CONCLUSIONS

Choosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.

Deep learning model improves tumor-infiltrating lymphocyte evaluation and therapeutic response prediction in breast cancer

Tumor-infiltrating lymphocytes (TILs) have been recognized as key players in the tumor microenvironment of breast cancer, but substantial interobserver variability among pathologists has impeded its utility as a biomarker. We developed a deep learning (DL)-based TIL analyzer to evaluate stromal TILs (sTILs) in breast cancer. Three pathologists evaluated 402 whole slide images of breast cancer and interpreted the sTIL scores. A standalone performance of the DL model was evaluated in the 210 cases (52.2%) exhibiting sTIL score differences of less than 10 percentage points, yielding a concordance correlation coefficient of 0.755 (95% confidence interval [CI], 0.693-0.805) in comparison to the pathologists' scores. For the 226 slides (56.2%) showing a 10 percentage points or greater variance between pathologists and the DL model, revisions were made. The number of discordant cases was reduced to 116 (28.9%) with the DL assistance (p < 0.001). The DL assistance also increased the concordance correlation coefficient of the sTIL score among every two pathologists. In triple-negative and human epidermal growth factor receptor 2 (HER2)-positive breast cancer patients who underwent the neoadjuvant chemotherapy, the DL-assisted revision notably accentuated higher sTIL scores in responders (26.8 ± 19.6 vs. 19.0 ± 16.4, p = 0.003). Furthermore, the DL-assistant revision disclosed the correlation of sTIL-high tumors (sTIL ≥ 50) with the chemotherapeutic response (odd ratio 1.28 [95% confidence interval, 1.01-1.63], p = 0.039). Through enhancing inter-pathologist concordance in sTIL interpretation and predicting neoadjuvant chemotherapy response, here we report the utility of the DL-based tool as a reference for sTIL scoring in breast cancer assessment.

Lymphocyte detection for cancer analysis using a novel fusion block based channel boosted CNN

Tumor-infiltrating lymphocytes, specialized immune cells, are considered an important biomarker in cancer analysis. Automated lymphocyte detection is challenging due to its heterogeneous morphology, variable distribution, and presence of artifacts. In this work, we propose a novel Boosted Channels Fusion-based CNN "BCF-Lym-Detector" for lymphocyte detection in multiple cancer histology images. The proposed network initially selects candidate lymphocytic regions at the tissue level and then detects lymphocytes at the cellular level. The proposed "BCF-Lym-Detector" generates diverse boosted channels by utilizing the feature learning capability of different CNN architectures. In this connection, a new adaptive fusion block is developed to combine and select the most relevant lymphocyte-specific features from the generated enriched feature space. Multi-level feature learning is used to retain lymphocytic spatial information and detect lymphocytes with variable appearances. The assessment of the proposed "BCF-Lym-Detector" show substantial improvement in terms of F-score (0.93 and 0.84 on LYSTO and NuClick, respectively), which suggests that the diverse feature extraction and dynamic feature selection enhanced the feature learning capacity of the proposed network. Moreover, the proposed technique's generalization on unseen test sets with a good recall (0.75) and F-score (0.73) shows its potential use for pathologists' assistance.

Pitfalls in machine learning-based assessment of tumor-infiltrating lymphocytes in breast cancer: A report of the International Immuno-Oncology Biomarker Working Group on Breast Cancer

The clinical significance of the tumor-immune interaction in breast cancer is now established, and tumor-infiltrating lymphocytes (TILs) have emerged as predictive and prognostic biomarkers for patients with triple-negative (estrogen receptor, progesterone receptor, and HER2-negative) breast cancer and HER2-positive breast cancer. How computational assessments of TILs might complement manual TIL assessment in trial and daily practices is currently debated. Recent efforts to use machine learning (ML) to automatically evaluate TILs have shown promising results. We review state-of-the-art approaches and identify pitfalls and challenges of automated TIL evaluation by studying the root cause of ML discordances in comparison to manual TIL quantification. We categorize our findings into four main topics: (1) technical slide issues, (2) ML and image analysis aspects, (3) data challenges, and (4) validation issues. The main reason for discordant assessments is the inclusion of false-positive areas or cells identified by performance on certain tissue patterns or design choices in the computational implementation. To aid the adoption of ML for TIL assessment, we provide an in-depth discussion of ML and image analysis, including validation issues that need to be considered before reliable computational reporting of TILs can be incorporated into the trial and routine clinical management of patients with triple-negative breast cancer. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Recent Advancements in Deep Learning Using Whole Slide Imaging for Cancer Prognosis

This review furnishes an exhaustive analysis of the latest advancements in deep learning techniques applied to whole slide images (WSIs) in the context of cancer prognosis, focusing specifically on publications from 2019 through 2023. The swiftly maturing field of deep learning, in combination with the burgeoning availability of WSIs, manifests significant potential in revolutionizing the predictive modeling of cancer prognosis. In light of the swift evolution and profound complexity of the field, it is essential to systematically review contemporary methodologies and critically appraise their ramifications. This review elucidates the prevailing landscape of this intersection, cataloging major developments, evaluating their strengths and weaknesses, and providing discerning insights into prospective directions. In this paper, a comprehensive overview of the field aims to be presented, which can serve as a critical resource for researchers and clinicians, ultimately enhancing the quality of cancer care outcomes. This review's findings accentuate the need for ongoing scrutiny of recent studies in this rapidly progressing field to discern patterns, understand breakthroughs, and navigate future research trajectories.

GPR81-mediated reprogramming of glucose metabolism contributes to the immune landscape in breast cancer

BACKGROUND

Local tumor microenvironment (TME) plays a crucial role in immunotherapy for breast cancer (BC). Whereas, the molecular mechanism responsible for the crosstalk between BC cells and surrounding immune cells remains unclear. The present study aimed to determine the interplay between GPR81-mediated glucometabolic reprogramming of BC and the immune landscape in TME.

MATERIALS AND METHODS

Immunohistochemistry (IHC) assay was first performed to evaluate the association between GPR81 and the immune landscape. Then, several stable BC cell lines with down-regulated GPR81 expression were established to directly identify the role of GPR81 in glucometabolic reprogramming, and western blotting assay was used to detect the underlying molecular mechanism. Finally, a transwell co-culture system confirmed the crosstalk between glucometabolic regulation mediated by GPR81 in BC and induced immune attenuation.

RESULTS

IHC analysis demonstrated that the representation of infiltrating CD8⁺ T cells and FOXP3⁺ T cells were dramatically higher in BC with a triple negative (TN) subtype in comparison with that with a non-TN subtype (P < 0.001). Additionally, the ratio of infiltrating CD8⁺ to FOXP3⁺ T cells was significantly negatively associated with GPR81 expression in BC with a TN subtype (P < 0.001). Furthermore, GPR81 was found to be substantially correlated with the glycolytic capability (P < 0.001) of BC cells depending on a Hippo-YAP signaling pathway (P < 0.001). In the transwell co-culture system, GPR81-mediated reprogramming of glucose metabolism in BC significantly contributed to a decreased proportion of CD8⁺ T (P < 0.001) and an increased percentage of FOXP3⁺ T (P < 0.001) in the co-cultured lymphocytes.

CONCLUSION

Glucometabolic reprogramming through a GPR81-mediated Hippo-YAP signaling pathway was responsible for the distinct immune landscape in BC. GPR81 was a potential biomarker to stratify patients before immunotherapy to improve BC's clinical prospect.

SPARTIN: a Bayesian method for the quantification and characterization of cell type interactions in spatial pathology data

Introduction: The acquisition of high-resolution digital pathology imaging data has sparked the development of methods to extract context-specific features from such complex data. In the context of cancer, this has led to increased exploration of the tumor microenvironment with respect to the presence and spatial composition of immune cells. Spatial statistical modeling of the immune microenvironment may yield insights into the role played by the immune system in the natural development of cancer as well as downstream therapeutic interventions. Methods: In this paper, we present SPatial Analysis of paRtitioned Tumor-Immune imagiNg (SPARTIN), a Bayesian method for the spatial quantification of immune cell infiltration from pathology images. SPARTIN uses Bayesian point processes to characterize a novel measure of local tumor-immune cell interaction, Cell Type Interaction Probability (CTIP). CTIP allows rigorous incorporation of uncertainty and is highly interpretable, both within and across biopsies, and can be used to assess associations with genomic and clinical features. Results: Through simulations, we show SPARTIN can accurately distinguish various patterns of cellular interactions as compared to existing methods. Using SPARTIN, we characterized the local spatial immune cell infiltration within and across 335 melanoma biopsies and evaluated their association with genomic, phenotypic, and clinical outcomes. We found that CTIP was significantly (negatively) associated with deconvolved immune cell prevalence scores including CD8+ T-Cells and Natural Killer cells. Furthermore, average CTIP scores differed significantly across previously established transcriptomic classes and significantly associated with survival outcomes. Discussion: SPARTIN provides a general framework for investigating spatial cellular interactions in high-resolution digital histopathology imaging data and its associations with patient level characteristics. The results of our analysis have potential implications relevant to both treatment and prognosis in the context of Skin Cutaneous Melanoma. The R-package for SPARTIN is available at https://github.com/bayesrx/SPARTIN along with a visualization tool for the images and results at: https://nateosher.github.io/SPARTIN.

A pathomic approach for tumor-infiltrating lymphocytes classification on breast cancer digital pathology images

Background and objectives

The detection of tumor-infiltrating lymphocytes (TILs) could aid in the development of objective measures of the infiltration grade and can support decision-making in breast cancer (BC). However, manual quantification of TILs in BC histopathological whole slide images (WSI) is currently based on a visual assessment, thus resulting not standardized, not reproducible, and time-consuming for pathologists. In this work, a novel pathomic approach, aimed to apply high-throughput image feature extraction techniques to analyze the microscopic patterns in WSI, is proposed. In fact, pathomic features provide additional information concerning the underlying biological processes compared to the WSI visual interpretation, thus providing more easily interpretable and explainable results than the most frequently investigated Deep Learning based methods in the literature.

Methods

A dataset containing 1037 regions of interest with tissue compartments and TILs annotated on 195 TNBC and HER2+ BC hematoxylin and eosin (H&E)-stained WSI was used. After segmenting nuclei within tumor-associated stroma using a watershed-based approach, 71 pathomic features were extracted from each nucleus and reduced using a Spearman's correlation filter followed by a nonparametric Wilcoxon rank-sum test and least absolute shrinkage and selection operator. The relevant features were used to classify each candidate nucleus as either TILs or non-TILs using 5 multivariable machine learning classification models trained using 5-fold cross-validation (1) without resampling, (2) with the synthetic minority over-sampling technique and (3) with downsampling. The prediction performance of the models was assessed using ROC curves.

Results

21 features were selected, with most of them related to the well-known TILs properties of having regular shape, clearer margins, high peak intensity, more homogeneous enhancement and different textural pattern than other cells. The best performance was obtained by Random-Forest with ROC AUC of 0.86, regardless of resampling technique.

Conclusions

The presented approach holds promise for the classification of TILs in BC H&E-stained WSI and could provide support to pathologists for a reliable, rapid and interpretable clinical assessment of TILs in BC.

Attention-guided multi-scale deep object detection framework for lymphocyte analysis in IHC histological images

Tumor-infiltrating lymphocytes are specialized lymphocytes that can detect and kill cancerous cells. Their detection poses many challenges due to significant morphological variations, overlapping occurrence, artifact regions and high-class resemblance between clustered areas and artifacts. In this regard, a Lymphocyte Analysis Framework based on Deep Convolutional neural network (DC-Lym-AF) is proposed to analyze lymphocytes in immunohistochemistry images. The proposed framework comprises (i) pre-processing, (ii) screening phase, (iii) localization phase and (iv) post-processing. In the screening phase, a custom convolutional neural network architecture (lymphocyte dilated network) is developed to screen lymphocytic regions by performing a patch-level classification. This proposed architecture uses dilated convolutions and shortcut connections to capture multi-level variations and ensure reference-based learning. In contrast, the localization phase utilizes an attention-guided multi-scale lymphocyte detector to detect lymphocytes. The proposed detector extracts refined and multi-scale features by exploiting dilated convolutions, attention mechanism and feature pyramid network (FPN) using its custom attention-aware backbone. The proposed DC-Lym-AF shows exemplary performance on the NuClick dataset compared with the existing detection models, with an F-score and precision of 0.84 and 0.83, respectively. We verified the generalizability of our proposed framework by participating in a publically open LYON'19 challenge. Results in terms of detection rate (0.76) and F-score (0.73) suggest that the proposed DC-Lym-AF can effectively detect lymphocytes in immunohistochemistry-stained images collected from different laboratories. In addition, its promising generalization on several datasets implies that it can be turned into a medical diagnostic tool to investigate various histopathological problems. Graphical Abstract.

Prognostic Significance of Tumor-Infiltrating Lymphocytes Determined Using LinkNet on Colorectal Cancer Pathology Images

PURPOSE

Tumor-infiltrating lymphocytes (TILs) have a significant prognostic value in cancers. However, very few automated, deep learning-based TIL scoring algorithms have been developed for colorectal cancer (CRC).

MATERIALS AND METHODS

We developed an automated, multiscale LinkNet workflow for quantifying TILs at the cellular level in CRC tumors using H&E-stained images from the Lizard data set with annotations of lymphocytes. The predictive performance of the automatic TIL scores (TILsLink) for disease progression and overall survival (OS) was evaluated using two international data sets, including 554 patients with CRC from The Cancer Genome Atlas (TCGA) and 1,130 patients with CRC from Molecular and Cellular Oncology (MCO).

RESULTS

The LinkNet model provided outstanding precision (0.9508), recall (0.9185), and overall F1 score (0.9347). Clear continuous TIL-hazard relationships were observed between TILsLink and the risk of disease progression or death in both TCGA and MCO cohorts. Both univariate and multivariate Cox regression analyses for the TCGA data demonstrated that patients with high TIL abundance had a significant (approximately 75%) reduction in risk for disease progression. In both the MCO and TCGA cohorts, the TIL-high group was significantly associated with improved OS in univariate analysis (30% and 54% reduction in risk, respectively). The favorable effects of high TIL levels were consistently observed in different subgroups (classified according to known risk factors).

CONCLUSION

The proposed deep-learning workflow for automatic TIL quantification on the basis of LinkNet can be a useful tool for CRC. TILsLink is likely an independent risk factor for disease progression and carries predictive information of disease progression beyond the current clinical risk factors and biomarkers. The prognostic significance of TILsLink for OS is also evident.

Predicting melanoma survival and metastasis with interpretable histopathological features and machine learning models

Introduction

Melanoma is the fifth most common cancer in US, and the incidence is increasing 1.4% annually. The overall survival rate for early-stage disease is 99.4%. However, melanoma can recur years later (in the same region of the body or as distant metastasis), and results in a dramatically lower survival rate. Currently there is no reliable method to predict tumor recurrence and metastasis on early primary tumor histological images.

Methods

To identify rapid, accurate, and cost-effective predictors of metastasis and survival, in this work, we applied various interpretable machine learning approaches to analyze melanoma histopathological H&E images. The result is a set of image features that can help clinicians identify high-risk-of-metastasis patients for increased clinical follow-up and precision treatment. We use simple models (i.e., logarithmic classification and KNN) and "human-interpretable" measures of cell morphology and tissue architecture (e.g., cell size, staining intensity, and cell density) to predict the melanoma survival on public and local Stage I-III cohorts as well as the metastasis risk on a local cohort.

Results

We use penalized survival regression to limit features available to downstream classifiers and investigate the utility of convolutional neural networks in isolating tumor regions to focus morphology extraction on only the tumor region. This approach allows us to predict survival and metastasis with a maximum F1 score of 0.72 and 0.73, respectively, and to visualize several high-risk cell morphologies.

Discussion

This lays the foundation for future work, which will focus on using our interpretable pipeline to predict metastasis in Stage I & II melanoma.

Deep Learning Based Methods for Breast Cancer Diagnosis: A Systematic Review and Future Direction

Breast cancer is one of the precarious conditions that affect women, and a substantive cure has not yet been discovered for it. With the advent of Artificial intelligence (AI), recently, deep learning techniques have been used effectively in breast cancer detection, facilitating early diagnosis and therefore increasing the chances of patients' survival. Compared to classical machine learning techniques, deep learning requires less human intervention for similar feature extraction. This study presents a systematic literature review on the deep learning-based methods for breast cancer detection that can guide practitioners and researchers in understanding the challenges and new trends in the field. Particularly, different deep learning-based methods for breast cancer detection are investigated, focusing on the genomics and histopathological imaging data. The study specifically adopts the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA), which offer a detailed analysis and synthesis of the published articles. Several studies were searched and gathered, and after the eligibility screening and quality evaluation, 98 articles were identified. The results of the review indicated that the Convolutional Neural Network (CNN) is the most accurate and extensively used model for breast cancer detection, and the accuracy metrics are the most popular method used for performance evaluation. Moreover, datasets utilized for breast cancer detection and the evaluation metrics are also studied. Finally, the challenges and future research direction in breast cancer detection based on deep learning models are also investigated to help researchers and practitioners acquire in-depth knowledge of and insight into the area.

Artificial intelligence in breast cancer histopathology

This is a review on the use of artificial intelligence for digital breast pathology. A systematic search on PubMed was conducted, identifying 17,324 research papers related to breast cancer pathology. Following a semimanual screening, 664 papers were retrieved and pursued. The papers are grouped into six major tasks performed by pathologists-namely, molecular and hormonal analysis, grading, mitotic figure counting, ki-67 indexing, tumour-infiltrating lymphocyte assessment, and lymph node metastases identification. Under each task, open-source datasets for research to build artificial intelligence (AI) tools are also listed. Many AI tools showed promise and demonstrated feasibility in the automation of routine pathology investigations. We expect continued growth of AI in this field as new algorithms mature.

Association of Machine Learning-Based Assessment of Tumor-Infiltrating Lymphocytes on Standard Histologic Images With Outcomes of Immunotherapy in Patients With NSCLC

Importance

Currently, predictive biomarkers for response to immune checkpoint inhibitor (ICI) therapy in lung cancer are limited. Identifying such biomarkers would be useful to refine patient selection and guide precision therapy.

Objective

To develop a machine-learning (ML)-based tumor-infiltrating lymphocytes (TILs) scoring approach, and to evaluate TIL association with clinical outcomes in patients with advanced non-small cell lung cancer (NSCLC).

Design, Setting, and Participants

This multicenter retrospective discovery-validation cohort study included 685 ICI-treated patients with NSCLC with median follow-up of 38.1 and 43.3 months for the discovery (n = 446) and validation (n = 239) cohorts, respectively. Patients were treated between February 2014 and September 2021. We developed an ML automated method to count tumor, stroma, and TIL cells in whole-slide hematoxylin-eosin-stained images of NSCLC tumors. Tumor mutational burden (TMB) and programmed death ligand-1 (PD-L1) expression were assessed separately, and clinical response to ICI therapy was determined by medical record review. Data analysis was performed from June 2021 to April 2022.

Exposures

All patients received anti-PD-(L)1 monotherapy.

Main Outcomes and Measures

Objective response rate (ORR), progression-free survival (PFS), and overall survival (OS) were determined by blinded medical record review. The area under curve (AUC) of TIL levels, TMB, and PD-L1 in predicting ICI response were calculated using ORR.

Results

Overall, there were 248 (56%) women in the discovery cohort and 97 (41%) in the validation cohort. In a multivariable analysis, high TIL level (≥250 cells/mm2) was independently associated with ICI response in both the discovery (PFS: HR, 0.71; P = .006; OS: HR, 0.74; P = .03) and validation (PFS: HR = 0.80; P = .01; OS: HR = 0.75; P = .001) cohorts. Survival benefit was seen in both first- and subsequent-line ICI treatments in patients with NSCLC. In the discovery cohort, the combined models of TILs/PD-L1 or TMB/PD-L1 had additional specificity in differentiating ICI responders compared with PD-L1 alone. In the PD-L1 negative (<1%) subgroup, TIL levels had superior classification accuracy for ICI response (AUC = 0.77) compared with TMB (AUC = 0.65).

Conclusions and Relevance

In these cohorts, TIL levels were robustly and independently associated with response to ICI treatment. Patient TIL assessment is relatively easily incorporated into the workflow of pathology laboratories at minimal additional cost, and may enhance precision therapy.

Spatial Transcriptomic Analysis Reveals Associations between Genes and Cellular Topology in Breast and Prostate Cancers

BACKGROUND

Cancer is the leading cause of death worldwide with breast and prostate cancer the most common among women and men, respectively. Gene expression and image features are independently prognostic of patient survival; but until the advent of spatial transcriptomics (ST), it was not possible to determine how gene expression of cells was tied to their spatial relationships (i.e., topology).

METHODS

We identify topology-associated genes (TAGs) that correlate with 700 image topological features (ITFs) in breast and prostate cancer ST samples. Genes and image topological features are independently clustered and correlated with each other. Themes among genes correlated with ITFs are investigated by functional enrichment analysis.

RESULTS

Overall, topology-associated genes (TAG) corresponding to extracellular matrix (ECM) and Collagen Type I Trimer gene ontology terms are common to both prostate and breast cancer. In breast cancer specifically, we identify the ZAG-PIP Complex as a TAG. In prostate cancer, we identify distinct TAGs that are enriched for GI dysmotility and the IgA immunoglobulin complex. We identified TAGs in every ST slide regardless of cancer type.

CONCLUSIONS

These TAGs are enriched for ontology terms, illustrating the biological relevance to our image topology features and their potential utility in diagnostic and prognostic models.

Artificial intelligence to identify genetic alterations in conventional histopathology

Precision oncology relies on the identification of targetable molecular alterations in tumor tissues. In many tumor types, a limited set of molecular tests is currently part of standard diagnostic workflows. However, universal testing for all targetable alterations, especially rare ones, is limited by the cost and availability of molecular assays. From 2017 to 2021, multiple studies have shown that artificial intelligence (AI) methods can predict the probability of specific genetic alterations directly from conventional hematoxylin and eosin (H&E) tissue slides. Although these methods are currently less accurate than gold-standard testing (e.g. immunohistochemistry, polymerase chain reaction or next-generation sequencing), they could be used as pre-screening tools to reduce the workload of genetic analyses. In this systematic literature review, we summarize the state of the art in predicting molecular alterations from H&E using AI. We found that AI methods perform reasonably well across multiple tumor types, although few algorithms have been broadly validated. In addition, we found that genetic alterations in FGFR, IDH, PIK3CA, BRAF, TP53 and DNA repair pathways are predictable from H&E in multiple tumor types, while many other genetic alterations have rarely been investigated or were only poorly predictable. Finally, we discuss the next steps for the implementation of AI-based surrogate tests in diagnostic workflows. This article is protected by copyright. All rights reserved.

A deep learning- and CT image-based prognostic model for the prediction of survival in non-small cell lung cancer

OBJECTIVE

To assist clinicians in arranging personalized treatment, planning follow-up programs and extending survival times for non-small cell lung cancer (NSCLC) patients, a method of deep learning combined with computed tomography (CT) imaging for survival prediction was designed.

METHODS

Data were collected from 484 patients from four research centers. The data from 344 patients were utilized to build the A_CNN survival prognosis model to classify 2-year overall survival time ranges (730 days cut-off). Data from 140 patients, including independent internal and external test sets, were utilized for model testing. First, a series of preprocessing techniques were used to process the original CT images and generate training and test data sets from the axial, coronal, and sagittal planes. Second, the structure of the A_CNN model was designed based on asymmetric convolution, bottleneck blocks, the uniform cross-entropy (UC) loss function, and other advanced techniques. After that, the A_CNN model was trained, and numerous comparative experiments were designed to obtain the best prognostic survival model. Last, the model performance was evaluated, and the predicted survival curves were analyzed.

RESULTS

The A_CNN survival prognosis model yielded a high patient-level accuracy of 88.8%, a patch-level accuracy of 82.9%, and an area under the receiver operating characteristic (ROC) curve (AUC) of 0.932. When tested on an external data set, the maximum patient-level accuracy was 80.0%.

CONCLUSIONS

The results suggest that using a deep learning method can improve prognosis in patients with NSCLC and has important application value in establishing individualized prognostic models.

Tertiary lymphoid structures (TLS) identification and density assessment on H&E-stained digital slides of lung cancer

Tertiary lymphoid structures (TLS) are ectopic aggregates of lymphoid cells in inflamed, infected, or tumoral tissues that are easily recognized on an H&E histology slide as discrete entities, distinct from lymphocytes. TLS are associated with improved cancer prognosis but there is no standardised method available to quantify their presence. Previous studies have used immunohistochemistry to determine the presence of specific cells as a marker of the TLS. This has now been proven to be an underestimate of the true number of TLS. Thus, we propose a methodology for the automated identification and quantification of TLS, based on H&E slides. We subsequently determined the mathematical criteria defining a TLS. TLS regions were identified through a deep convolutional neural network and segmentation of lymphocytes was performed through an ellipsoidal model. This methodology had a 92.87% specificity at 95% sensitivity, 88.79% specificity at 98% sensitivity and 84.32% specificity at 99% sensitivity level based on 144 TLS annotated H&E slides implying that the automated approach was able to reproduce the histopathologists’ assessment with great accuracy. We showed that the minimum number of lymphocytes within TLS is 45 and the minimum TLS area is 6,245μm². Furthermore, we have shown that the density of the lymphocytes is more than 3 times those outside of the TLS. The mean density and standard deviation of lymphocytes within a TLS area are 0.0128/μm² and 0.0026/μm² respectively compared to 0.004/μm² and 0.001/μm² in non-TLS regions. The proposed methodology shows great potential for automated identification and quantification of the TLS density on digital H&E slides.

Technological advances in cancer immunity: from immunogenomics to single-cell analysis and artificial intelligence

Immunotherapies play critical roles in cancer treatment. However, given that only a few patients respond to immune checkpoint blockades and other immunotherapeutic strategies, more novel technologies are needed to decipher the complicated interplay between tumor cells and the components of the tumor immune microenvironment (TIME). Tumor immunomics refers to the integrated study of the TIME using immunogenomics, immunoproteomics, immune-bioinformatics, and other multi-omics data reflecting the immune states of tumors, which has relied on the rapid development of next-generation sequencing. High-throughput genomic and transcriptomic data may be utilized for calculating the abundance of immune cells and predicting tumor antigens, referring to immunogenomics. However, as bulk sequencing represents the average characteristics of a heterogeneous cell population, it fails to distinguish distinct cell subtypes. Single-cell-based technologies enable better dissection of the TIME through precise immune cell subpopulation and spatial architecture investigations. In addition, radiomics and digital pathology-based deep learning models largely contribute to research on cancer immunity. These artificial intelligence technologies have performed well in predicting response to immunotherapy, with profound significance in cancer therapy. In this review, we briefly summarize conventional and state-of-the-art technologies in the field of immunogenomics, single-cell and artificial intelligence, and present prospects for future research.

Correlation of manual semi-quantitative and automated quantitative Ki-67 proliferative index with OncotypeDXTM recurrence score in invasive breast carcinoma

BACKGROUND

Ki-67 immunohistochemistry (IHC) staining is a widely used cancer proliferation assay; however, its limitations could be improved with automated scoring. The OncotypeDXTM Recurrence Score (ORS), which primarily evaluates cancer proliferation genes, is a prognostic indicator for breast cancer chemotherapy response; however, it is more expensive and slower than Ki-67.

OBJECTIVE

To compare manual Ki-67 (mKi-67) with automated Ki-67 (aKi-67) algorithm results based on manually selected Ki-67 "hot spots" in breast cancer, and correlate both with ORS.

METHODS

105 invasive breast carcinoma cases from 100 patients at our institution (2011-2013) with available ORS were evaluated. Concordance was assessed via Cohen's Kappa (κ).

RESULTS

57/105 cases showed agreement between mKi-67 and aKi-67 (κ 0.31, 95% CI 0.18-0.45), with 41 cases overestimated by aKi-67. Concordance was higher when estimated on the same image (κ 0.53, 95% CI 0.37-0.69). Concordance between mKi-67 score and ORS was fair (κ 0.27, 95% CI 0.11-0.42), and concordance between aKi-67 and ORS was poor (κ 0.10, 95% CI -0.03-0.23).

CONCLUSIONS

These results highlight the limits of Ki-67 algorithms that use manual "hot spot" selection. Due to suboptimal concordance, Ki-67 is likely most useful as a complement to, rather than a surrogate for ORS, regardless of scoring method.

Machine Learning and Artificial Intelligence-driven Spatial Analysis of the Tumor Immune Microenvironment in Pathology Slides

A better understanding of the tumor immune microenvironment (TIME) could lead to accurate diagnosis, prognosis, and treatment stratification. Although molecular analyses at the tissue and/or single cell level could reveal the cellular status of the tumor microenvironment, these approaches lack information related to spatial-level cellular distribution, co-organization, and cell-cell interaction in the TIME. With the emergence of computational pathology coupled with machine learning (ML) and artificial intelligence (AI), ML- and AI-driven spatial TIME analyses of pathology images could revolutionize our understanding of the highly heterogeneous and complex molecular architecture of the TIME. In this review we highlight recent studies on spatial TIME analysis of pathology slides using state-of-the-art ML and AI algorithms. PATIENT SUMMARY: This mini-review reports recent advances in machine learning and artificial intelligence for spatial analysis of the tumor immune microenvironment in pathology slides. This information can help in understanding the spatial heterogeneity and organization of cells in patient tumors.

Body mass index, adiposity and tumour infiltrating lymphocytes as prognostic biomarkers in patients treated with immunotherapy: A multi-parametric analysis

BACKGROUND

We performed a multi-parametric analysis investigating the association between adiposity (as measured using body mass index [BMI] and computed tomography [CT]-based body composition), tumour infiltrating lymphocytes (TILs) and clinical outcomes in patients with advanced-stage cancer treated with immunotherapy in phase I clinical trials.

MATERIAL AND METHODS

All consecutive patients (N = 153) with metastatic solid tumours treated within immunotherapy-based phase I clinical trials between August 2014 and May 2019 at our institution were included. Baseline characteristics, BMI, TILs value and CT-assessed fat indices (total fat area [TFA], subcutaneous fat area [SFA] and visceral fat [VFA]) were collected. The primary endpoints were to evaluate the impact of these parameters on overall survival (OS) and progression-free survival (PFS). Kaplan-Meier method and Cox proportional-hazards model were used for survival analyses.

RESULTS

At both univariate and multivariate analyses, BMI was not associated with PFS neither when considered as continuous variable (HR 0.90, 95% CI 0.74-1.09, P = 0.28) nor as dichotomous variable (underweight/normal versus overweight/obese) (HR 0.79, 95% CI 0.55-1.14, P = 0.21). Interestingly, patients diagnosed with 'immunogenic' tumours and higher VFA/SFA ratio (1st and 2nd tertile versus 3rd tertile) presented an increased OS (HR 0.88, 95% CI 0.78-1.00, P = 0.047).

CONCLUSION

Our analysis showed that patients with tumours that are already known as responsive to ICIs with higher VFA/SFA ratio presented an increased OS. Further studies are needed to elucidate the effect of adiposity on the host immune response to immunotherapy.

Artificial Intelligence for Histology-Based Detection of Microsatellite Instability and Prediction of Response to Immunotherapy in Colorectal Cancer

Microsatellite instability (MSI) is a molecular marker of deficient DNA mismatch repair (dMMR) that is found in approximately 15% of colorectal cancer (CRC) patients. Testing all CRC patients for MSI/dMMR is recommended as screening for Lynch Syndrome and, more recently, to determine eligibility for immune checkpoint inhibitors in advanced disease. However, universal testing for MSI/dMMR has not been uniformly implemented because of cost and resource limitations. Artificial intelligence has been used to predict MSI/dMMR directly from hematoxylin and eosin (H&E) stained tissue slides. We review the emerging data regarding the utility of machine learning for MSI classification, focusing on CRC. We also provide the clinician with an introduction to image analysis with machine learning and convolutional neural networks. Machine learning can predict MSI/dMMR with high accuracy in high quality, curated datasets. Accuracy can be significantly decreased when applied to cohorts with different ethnic and/or clinical characteristics, or different tissue preparation protocols. Research is ongoing to determine the optimal machine learning methods for predicting MSI, which will need to be compared to current clinical practices, including next-generation sequencing. Predicting response to immunotherapy remains an unmet need.

AF-SENet: Classification of Cancer in Cervical Tissue Pathological Images Based on Fusing Deep Convolution Features

Cervical cancer is the fourth most common cancer in the world. Whole-slide images (WSIs) are an important standard for the diagnosis of cervical cancer. Missed diagnoses and misdiagnoses often occur due to the high similarity in pathological cervical images, the large number of readings, the long reading time, and the insufficient experience levels of pathologists. Existing models have insufficient feature extraction and representation capabilities, and they suffer from insufficient pathological classification. Therefore, this work first designs an image processing algorithm for data augmentation. Second, the deep convolutional features are extracted by fine-tuning pre-trained deep network models, including ResNet50 v2, DenseNet121, Inception v3, VGGNet19, and Inception-ResNet, and then local binary patterns and a histogram of the oriented gradient to extract traditional image features are used. Third, the features extracted by the fine-tuned models are serially fused according to the feature representation ability parameters and the accuracy of multiple experiments proposed in this paper, and spectral embedding is used for dimension reduction. Finally, the fused features are inputted into the Analysis of Variance-F value-Spectral Embedding Net (AF-SENet) for classification. There are four different pathological images of the dataset: normal, low-grade squamous intraepithelial lesion (LSIL), high-grade squamous intraepithelial lesion (HSIL), and cancer. The dataset is divided into a training set (90%) and a test set (10%). The serial fusion effect of the deep features extracted by Resnet50v2 and DenseNet121 (C₅) is the best, with average classification accuracy reaching 95.33%, which is 1.07% higher than ResNet50 v2 and 1.05% higher than DenseNet121. The recognition ability is significantly improved, especially in LSIL, reaching 90.89%, which is 2.88% higher than ResNet50 v2 and 2.1% higher than DenseNet121. Thus, this method significantly improves the accuracy and generalization ability of pathological cervical WSI recognition by fusing deep features.

Predicting and Overcoming Taxane Chemoresistance

Taxanes are microtubule-targeting drugs used as cytotoxic chemotherapy to treat most solid tumors. The development of resistance to taxanes is a major cause of therapeutic failure and overcoming chemoresistance remains an important challenge to improve patient's outcome. Extensive efforts have been made recently to identify predictive biomarkers to select populations of patients who will benefit from taxane-based chemotherapy and avoid inefficient treatment of patients with innate resistance. This, together with the discovery of new mechanisms of resistance that include metabolic reprogramming and dialogue between tumor and its microenvironment, pave the way to a new era of personalized medicine. In this review, we recapitulate recent insights into taxane resistance and present promising emerging strategies to overcome chemoresistance in the future.