1
|
Hilgers L, Ghaffari Laleh N, West NP, Westwood A, Hewitt KJ, Quirke P, Grabsch HI, Carrero ZI, Matthaei E, Loeffler CML, Brinker TJ, Yuan T, Brenner H, Brobeil A, Hoffmeister M, Kather JN. Automated curation of large-scale cancer histopathology image datasets using deep learning. Histopathology 2024; 84:1139-1153. [PMID: 38409878 DOI: 10.1111/his.15159] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2023] [Revised: 12/29/2023] [Accepted: 02/09/2024] [Indexed: 02/28/2024]
Abstract
BACKGROUND Artificial intelligence (AI) has numerous applications in pathology, supporting diagnosis and prognostication in cancer. However, most AI models are trained on highly selected data, typically one tissue slide per patient. In reality, especially for large surgical resection specimens, dozens of slides can be available for each patient. Manually sorting and labelling whole-slide images (WSIs) is a very time-consuming process, hindering the direct application of AI on the collected tissue samples from large cohorts. In this study we addressed this issue by developing a deep-learning (DL)-based method for automatic curation of large pathology datasets with several slides per patient. METHODS We collected multiple large multicentric datasets of colorectal cancer histopathological slides from the United Kingdom (FOXTROT, N = 21,384 slides; CR07, N = 7985 slides) and Germany (DACHS, N = 3606 slides). These datasets contained multiple types of tissue slides, including bowel resection specimens, endoscopic biopsies, lymph node resections, immunohistochemistry-stained slides, and tissue microarrays. We developed, trained, and tested a deep convolutional neural network model to predict the type of slide from the slide overview (thumbnail) image. The primary statistical endpoint was the macro-averaged area under the receiver operating curve (AUROCs) for detection of the type of slide. RESULTS In the primary dataset (FOXTROT), with an AUROC of 0.995 [95% confidence interval [CI]: 0.994-0.996] the algorithm achieved a high classification performance and was able to accurately predict the type of slide from the thumbnail image alone. In the two external test cohorts (CR07, DACHS) AUROCs of 0.982 [95% CI: 0.979-0.985] and 0.875 [95% CI: 0.864-0.887] were observed, which indicates the generalizability of the trained model on unseen datasets. With a confidence threshold of 0.95, the model reached an accuracy of 94.6% (7331 classified cases) in CR07 and 85.1% (2752 classified cases) for the DACHS cohort. CONCLUSION Our findings show that using the low-resolution thumbnail image is sufficient to accurately classify the type of slide in digital pathology. This can support researchers to make the vast resource of existing pathology archives accessible to modern AI models with only minimal manual annotations.
Collapse
Affiliation(s)
- Lars Hilgers
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Narmin Ghaffari Laleh
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Nicholas P West
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Alice Westwood
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Katherine J Hewitt
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Philip Quirke
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Heike I Grabsch
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Department of Pathology, GROW - Research Institute for Oncology and Reproduction, Maastricht University Medical Center+, Maastricht, The Netherlands
| | - Zunamys I Carrero
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Emylou Matthaei
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Chiara M L Loeffler
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Brobeil
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany
- Tissue Bank, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| |
Collapse
|
2
|
Hoang DT, Shulman ED, Turakulov R, Abdullaev Z, Singh O, Campagnolo EM, Lalchungnunga H, Stone EA, Nasrallah MP, Ruppin E, Aldape K. Prediction of DNA methylation-based tumor types from histopathology in central nervous system tumors with deep learning. Nat Med 2024:10.1038/s41591-024-02995-8. [PMID: 38760587 DOI: 10.1038/s41591-024-02995-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2023] [Accepted: 04/11/2024] [Indexed: 05/19/2024]
Abstract
Precision in the diagnosis of diverse central nervous system (CNS) tumor types is crucial for optimal treatment. DNA methylation profiles, which capture the methylation status of thousands of individual CpG sites, are state-of-the-art data-driven means to enhance diagnostic accuracy but are also time consuming and not widely available. Here, to address these limitations, we developed Deep lEarning from histoPathoLOgy and methYlation (DEPLOY), a deep learning model that classifies CNS tumors to ten major categories from histopathology. DEPLOY integrates three distinct components: the first classifies CNS tumors directly from slide images ('direct model'), the second initially generates predictions for DNA methylation beta values, which are subsequently used for tumor classification ('indirect model'), and the third classifies tumor types directly from routinely available patient demographics. First, we find that DEPLOY accurately predicts beta values from histopathology images. Second, using a ten-class model trained on an internal dataset of 1,796 patients, we predict the tumor categories in three independent external test datasets including 2,156 patients, achieving an overall accuracy of 95% and balanced accuracy of 91% on samples that are predicted with high confidence. These results showcase the potential future use of DEPLOY to assist pathologists in diagnosing CNS tumors within a clinically relevant short time frame.
Collapse
Affiliation(s)
- Danh-Tai Hoang
- Biological Data Science Institute, College of Science, Australian National University, Canberra, Australian Capital Territory, Australia
| | - Eldad D Shulman
- Cancer Data Science Laboratory, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
| | - Rust Turakulov
- Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
| | - Zied Abdullaev
- Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
| | - Omkar Singh
- Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
| | - Emma M Campagnolo
- Cancer Data Science Laboratory, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
| | - H Lalchungnunga
- Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA
| | - Eric A Stone
- Biological Data Science Institute, College of Science, Australian National University, Canberra, Australian Capital Territory, Australia
| | - MacLean P Nasrallah
- Division of Neuropathology, Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Eytan Ruppin
- Cancer Data Science Laboratory, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA.
| | - Kenneth Aldape
- Laboratory of Pathology, Center for Cancer Research, National Cancer Institute, Bethesda, MD, USA.
| |
Collapse
|
3
|
Zamanitajeddin N, Jahanifar M, Bilal M, Eastwood M, Rajpoot N. Social network analysis of cell networks improves deep learning for prediction of molecular pathways and key mutations in colorectal cancer. Med Image Anal 2024; 93:103071. [PMID: 38199068 DOI: 10.1016/j.media.2023.103071] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 11/14/2023] [Accepted: 12/19/2023] [Indexed: 01/12/2024]
Abstract
Colorectal cancer (CRC) is a primary global health concern, and identifying the molecular pathways, genetic subtypes, and mutations associated with CRC is crucial for precision medicine. However, traditional measurement techniques such as gene sequencing are costly and time-consuming, while most deep learning methods proposed for this task lack interpretability. This study offers a new approach to enhance the state-of-the-art deep learning methods for molecular pathways and key mutation prediction by incorporating cell network information. We build cell graphs with nuclei as nodes and nuclei connections as edges of the network and leverage Social Network Analysis (SNA) measures to extract abstract, perceivable, and interpretable features that explicitly describe the cell network characteristics in an image. Our approach does not rely on precise nuclei segmentation or feature extraction, is computationally efficient, and is easily scalable. In this study, we utilize the TCGA-CRC-DX dataset, comprising 499 patients and 502 diagnostic slides from primary colorectal tumours, sourced from 36 distinct medical centres in the United States. By incorporating the SNA features alongside deep features in two multiple instance learning frameworks, we demonstrate improved performance for chromosomal instability (CIN), hypermutated tumour (HM), TP53 gene, BRAF gene, and Microsatellite instability (MSI) status prediction tasks (2.4%-4% and 7-8.8% improvement in AUROC and AUPRC on average). Additionally, our method achieves outstanding performance on MSI prediction in an external PAIP dataset (99% AUROC and 98% AUPRC), demonstrating its generalizability. Our findings highlight the discrimination power of SNA features and how they can be beneficial to deep learning models' performance and provide insights into the correlation of cell network profiles with molecular pathways and key mutations.
Collapse
Affiliation(s)
- Neda Zamanitajeddin
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK.
| | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mohsin Bilal
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Mark Eastwood
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK; Histofy Ltd., Birmingham, UK.
| |
Collapse
|
4
|
Godson L, Alemi N, Nsengimana J, Cook GP, Clarke EL, Treanor D, Bishop DT, Newton-Bishop J, Gooya A, Magee D. Immune subtyping of melanoma whole slide images using multiple instance learning. Med Image Anal 2024; 93:103097. [PMID: 38325154 DOI: 10.1016/j.media.2024.103097] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 01/15/2024] [Accepted: 01/25/2024] [Indexed: 02/09/2024]
Abstract
Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential predictive biomarkers. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here, we attempt to overcome this by developing deep learning models to classify gigapixel haematoxylin and eosin (H&E) stained pathology slides, which are well established in clinical workflows, into these immune subgroups. We systematically assess six different multiple instance learning (MIL) frameworks, using five different image resolutions and three different feature extraction methods. We show that pathology-specific self-supervised models using 10x resolution patches generate superior representations for the classification of immune subtypes. In addition, in a primary melanoma dataset, we achieve a mean area under the receiver operating characteristic curve (AUC) of 0.80 for classifying histopathology images into 'high' or 'low immune' subgroups and a mean AUC of 0.82 in an independent TCGA melanoma dataset. Furthermore, we show that these models are able to stratify patients into 'high' and 'low immune' subgroups with significantly different melanoma specific survival outcomes (log rank test, P< 0.005). We anticipate that MIL methods will allow us to find new biomarkers of high importance, act as a tool for clinicians to infer the immune landscape of tumours and stratify patients, without needing to carry out additional expensive genetic tests.
Collapse
Affiliation(s)
- Lucy Godson
- School of Computing, University of Leeds, Woodhouse, Leeds, LS2 9JT, United Kingdom.
| | - Navid Alemi
- School of Computing, University of Leeds, Woodhouse, Leeds, LS2 9JT, United Kingdom
| | - Jérémie Nsengimana
- Population Health Sciences Institute, Newcastle University, Newcastle upon Tyne, NE1 7RU, United Kingdom
| | - Graham P Cook
- Leeds Institute of Medical Research, University of Leeds School of Medicine, St. James's University Hospital, Leeds, United Kingdom
| | - Emily L Clarke
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, United Kingdom; Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, United Kingdom
| | - Darren Treanor
- Department of Histopathology, Leeds Teaching Hospitals NHS Trust, Leeds, United Kingdom; Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, United Kingdom; Department of Clinical Pathology and Department of Clinical and Experimental Medicine, Linköping University, Linköping, Sweden; Centre for Medical Image Science and Visualization (CMIV), Linköping University, Linköping, Sweden
| | - D Timothy Bishop
- Leeds Institute of Medical Research, University of Leeds School of Medicine, St. James's University Hospital, Leeds, United Kingdom
| | - Julia Newton-Bishop
- Division of Pathology and Data Analytics, Leeds Institute of Cancer and Pathology, University of Leeds, Beckett Street, Leeds, LS9 7TF, United Kingdom
| | - Ali Gooya
- School of Computing, University of Glasgow, Glasgow, G12 8QQ, United Kingdom
| | - Derek Magee
- School of Computing, University of Leeds, Woodhouse, Leeds, LS2 9JT, United Kingdom
| |
Collapse
|
5
|
Arslan S, Schmidt J, Bass C, Mehrotra D, Geraldes A, Singhal S, Hense J, Li X, Raharja-Liu P, Maiques O, Kather JN, Pandya P. A systematic pan-cancer study on deep learning-based prediction of multi-omic biomarkers from routine pathology images. Commun Med (Lond) 2024; 4:48. [PMID: 38491101 PMCID: PMC10942985 DOI: 10.1038/s43856-024-00471-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 02/29/2024] [Indexed: 03/18/2024] Open
Abstract
BACKGROUND The objective of this comprehensive pan-cancer study is to evaluate the potential of deep learning (DL) for molecular profiling of multi-omic biomarkers directly from hematoxylin and eosin (H&E)-stained whole slide images. METHODS A total of 12,093 DL models predicting 4031 multi-omic biomarkers across 32 cancer types were trained and validated. The study included a broad range of genetic, transcriptomic, and proteomic biomarkers, as well as established prognostic markers, molecular subtypes, and clinical outcomes. RESULTS Here we show that 50% of the models achieve an area under the curve (AUC) of 0.644 or higher. The observed AUC for 25% of the models is at least 0.719 and exceeds 0.834 for the top 5%. Molecular profiling with image-based histomorphological features is generally considered feasible for most of the investigated biomarkers and across different cancer types. The performance appears to be independent of tumor purity, sample size, and class ratio (prevalence), suggesting a degree of inherent predictability in histomorphology. CONCLUSIONS The results demonstrate that DL holds promise to predict a wide range of biomarkers across the omics spectrum using only H&E-stained histological slides of solid tumors. This paves the way for accelerating diagnosis and developing more precise treatments for cancer patients.
Collapse
Affiliation(s)
| | | | | | - Debapriya Mehrotra
- Panakeia Technologies, London, UK
- Department of Pathology, Barking, Havering and Redbridge University NHS Trust, Romford, UK
| | | | - Shikha Singhal
- Panakeia Technologies, London, UK
- Department of Pathology, The Royal Wolverhampton NHS Trust, Wolverhampton, UK
| | | | - Xiusi Li
- Panakeia Technologies, London, UK
| | | | - Oscar Maiques
- Cytoskeleton and Cancer Metastasis Group, Breast Cancer Now Toby Robins Breast Cancer Research Centre, The Institute of Cancer Research, London, UK
- Cancer Biomarkers & Biotherapeutics, Barts Cancer Institute, Queen Mary University of London, John Vane Science Building, London, UK
| | - Jakob Nikolas Kather
- Medical Oncology, National Center for Tumor Diseases, University Hospital Heidelberg, Heidelberg, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | | |
Collapse
|
6
|
Coombs A, Ong S. Four change-makers seek impact in medical research. Nature 2024; 627:S8-S10. [PMID: 38480969 DOI: 10.1038/d41586-024-00754-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/17/2024]
|
7
|
Angeloni M, van Doeveren T, Lindner S, Volland P, Schmelmer J, Foersch S, Matek C, Stoehr R, Geppert CI, Heers H, Wach S, Taubert H, Sikic D, Wullich B, van Leenders GJLH, Zaburdaev V, Eckstein M, Hartmann A, Boormans JL, Ferrazzi F, Bahlinger V. A deep-learning workflow to predict upper tract urothelial carcinoma protein-based subtypes from H&E slides supporting the prioritization of patients for molecular testing. J Pathol Clin Res 2024; 10:e12369. [PMID: 38504364 PMCID: PMC10951050 DOI: 10.1002/2056-4538.12369] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2023] [Revised: 02/08/2024] [Accepted: 02/26/2024] [Indexed: 03/21/2024]
Abstract
Upper tract urothelial carcinoma (UTUC) is a rare and aggressive, yet understudied, urothelial carcinoma (UC). The more frequent UC of the bladder comprises several molecular subtypes, associated with different targeted therapies and overlapping with protein-based subtypes. However, if and how these findings extend to UTUC remains unclear. Artificial intelligence-based approaches could help elucidate UTUC's biology and extend access to targeted treatments to a wider patient audience. Here, UTUC protein-based subtypes were identified, and a deep-learning (DL) workflow was developed to predict them directly from routine histopathological H&E slides. Protein-based subtypes in a retrospective cohort of 163 invasive tumors were assigned by hierarchical clustering of the immunohistochemical expression of three luminal (FOXA1, GATA3, and CK20) and three basal (CD44, CK5, and CK14) markers. Cluster analysis identified distinctive luminal (N = 80) and basal (N = 42) subtypes. The luminal subtype mostly included pushing, papillary tumors, whereas the basal subtype diffusely infiltrating, non-papillary tumors. DL model building relied on a transfer-learning approach by fine-tuning a pre-trained ResNet50. Classification performance was measured via three-fold repeated cross-validation. A mean area under the receiver operating characteristic curve of 0.83 (95% CI: 0.67-0.99), 0.8 (95% CI: 0.62-0.99), and 0.81 (95% CI: 0.65-0.96) was reached in the three repetitions. High-confidence DL-based predicted subtypes showed significant associations (p < 0.001) with morphological features, i.e. tumor type, histological subtypes, and infiltration type. Furthermore, a significant association was found with programmed cell death ligand 1 (PD-L1) combined positive score (p < 0.001) and FGFR3 mutational status (p = 0.002), with high-confidence basal predictions containing a higher proportion of PD-L1 positive samples and high-confidence luminal predictions a higher proportion of FGFR3-mutated samples. Testing of the DL model on an independent cohort highlighted the importance to accommodate histological subtypes. Taken together, our DL workflow can predict protein-based UTUC subtypes, associated with the presence of targetable alterations, directly from H&E slides.
Collapse
Affiliation(s)
- Miriam Angeloni
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Thomas van Doeveren
- Department of UrologyErasmus MC Urothelial Cancer Research GroupRotterdamThe Netherlands
| | - Sebastian Lindner
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Patrick Volland
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Jorina Schmelmer
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | | | - Christian Matek
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Robert Stoehr
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Carol I Geppert
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Hendrik Heers
- Department of UrologyPhilipps‐Universität MarburgMarburgGermany
| | - Sven Wach
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
- Department of Urology and Pediatric UrologyUniversity Hospital Erlangen, Friedrich‐Alexander Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
| | - Helge Taubert
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
- Department of Urology and Pediatric UrologyUniversity Hospital Erlangen, Friedrich‐Alexander Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
| | - Danijel Sikic
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
- Department of Urology and Pediatric UrologyUniversity Hospital Erlangen, Friedrich‐Alexander Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
| | - Bernd Wullich
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
- Department of Urology and Pediatric UrologyUniversity Hospital Erlangen, Friedrich‐Alexander Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
| | - Geert JLH van Leenders
- Department of PathologyErasmus MC Cancer Institute, University Medical CentreRotterdamthe Netherlands
| | - Vasily Zaburdaev
- Department of BiologyFriedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Max‐Planck‐Zentrum für Physik und MedizinErlangenGermany
| | - Markus Eckstein
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Arndt Hartmann
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
| | - Joost L Boormans
- Department of UrologyErasmus MC Urothelial Cancer Research GroupRotterdamThe Netherlands
| | - Fulvia Ferrazzi
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
- Department of NephropathologyInstitute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
| | - Veronika Bahlinger
- Institute of Pathology, University Hospital Erlangen‐Nürnberg, Friedrich‐Alexander‐Universität Erlangen‐Nürnberg (FAU)ErlangenGermany
- Comprehensive Cancer Center Erlangen‐EMN (CCC ER‐EMN)ErlangenGermany
- Bavarian Cancer Research Center (BZKF)ErlangenGermany
- Department of Pathology and NeuropathologyUniversity Hospital and Comprehensive Cancer Center TübingenTübingenGermany
| |
Collapse
|
8
|
Beaufrère A, Ouzir N, Zafar PE, Laurent-Bellue A, Albuquerque M, Lubuela G, Grégory J, Guettier C, Mondet K, Pesquet JC, Paradis V. Primary liver cancer classification from routine tumour biopsy using weakly supervised deep learning. JHEP Rep 2024; 6:101008. [PMID: 38379584 PMCID: PMC10877109 DOI: 10.1016/j.jhepr.2024.101008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/31/2023] [Revised: 12/09/2023] [Accepted: 12/17/2023] [Indexed: 02/22/2024] Open
Abstract
Background & Aims The diagnosis of primary liver cancers (PLCs) can be challenging, especially on biopsies and for combined hepatocellular-cholangiocarcinoma (cHCC-CCA). We automatically classified PLCs on routine-stained biopsies using a weakly supervised learning method. Method We selected 166 PLC biopsies divided into training, internal and external validation sets: 90, 29 and 47 samples, respectively. Two liver pathologists reviewed each whole-slide hematein eosin saffron (HES)-stained image (WSI). After annotating the tumour/non-tumour areas, tiles of 256x256 pixels were extracted from the WSIs and used to train a ResNet18 neural network. The tumour/non-tumour annotations served as labels during training, and the network's last convolutional layer was used to extract new tumour tile features. Without knowledge of the precise labels of the malignancies, we then applied an unsupervised clustering algorithm. Results Pathological review classified the training and validation sets into hepatocellular carcinoma (HCC, 33/90, 11/29 and 26/47), intrahepatic cholangiocarcinoma (iCCA, 28/90, 9/29 and 15/47), and cHCC-CCA (29/90, 9/29 and 6/47). In the two-cluster model, Clusters 0 and 1 contained mainly HCC and iCCA histological features. The diagnostic agreement between the pathological diagnosis and the two-cluster model predictions (major contingent) in the internal and external validation sets was 100% (11/11) and 96% (25/26) for HCC and 78% (7/9) and 87% (13/15) for iCCA, respectively. For cHCC-CCA, we observed a highly variable proportion of tiles from each cluster (cluster 0: 5-97%; cluster 1: 2-94%). Conclusion Our method applied to PLC HES biopsy could identify specific morphological features of HCC and iCCA. Although no specific features of cHCC-CCA were recognized, assessing the proportion of HCC and iCCA tiles within a slide could facilitate the identification of cHCC-CCA. Impact and implications The diagnosis of primary liver cancers can be challenging, especially on biopsies and for combined hepatocellular-cholangiocarcinoma (cHCC-CCA). We automatically classified primary liver cancers on routine-stained biopsies using a weakly supervised learning method. Our model identified specific features of hepatocellular carcinoma and intrahepatic cholangiocarcinoma. Despite no specific features of cHCC-CCA being recognized, the identification of hepatocellular carcinoma and intrahepatic cholangiocarcinoma tiles within a slide could facilitate the diagnosis of primary liver cancers, and particularly cHCC-CCA.
Collapse
Affiliation(s)
- Aurélie Beaufrère
- AP-HP. Nord, Department of Pathology, FHU MOSAIC, Beaujon Hospital, Clichy, France
- Université Paris Cité, Paris, France
- Centre de Recherche sur l'Inflammation, INSERM UMR 1149, Paris, France
| | - Nora Ouzir
- University of Paris-Saclay, CentraleSupélec, CVN, OPIS Inria, Gif-sur-Yvette 91190, France
| | - Paul Emile Zafar
- AP-HP. Nord, Department of Pathology, FHU MOSAIC, Beaujon Hospital, Clichy, France
- University of Paris-Saclay, CentraleSupélec, CVN, OPIS Inria, Gif-sur-Yvette 91190, France
| | - Astrid Laurent-Bellue
- AP-HP, Department of Pathology, Hôpital Bicêtre, Le Kremlin- Bicêtre, France; UMR-S 1193, Université Paris-Saclay, Kremlin-Bicêtre, France
| | - Miguel Albuquerque
- AP-HP. Nord, Department of Pathology, FHU MOSAIC, Beaujon Hospital, Clichy, France
| | | | - Jules Grégory
- AP-HP. Nord, Department of Pathology, FHU MOSAIC, Beaujon Hospital, Clichy, France
- Université Paris Cité, Paris, France
- AP-HP.Nord, Department of Imaging, Beaujon Hospital, Clichy, France
| | - Catherine Guettier
- AP-HP, Department of Pathology, Hôpital Bicêtre, Le Kremlin- Bicêtre, France; UMR-S 1193, Université Paris-Saclay, Kremlin-Bicêtre, France
| | - Kévin Mondet
- AP-HP. Nord, Department of Pathology, FHU MOSAIC, Beaujon Hospital, Clichy, France
| | | | - Valérie Paradis
- AP-HP. Nord, Department of Pathology, FHU MOSAIC, Beaujon Hospital, Clichy, France
- Université Paris Cité, Paris, France
- Centre de Recherche sur l'Inflammation, INSERM UMR 1149, Paris, France
| |
Collapse
|
9
|
Irmakci I, Nateghi R, Zhou R, Vescovo M, Saft M, Ross AE, Yang XJ, Cooper LAD, Goldstein JA. Tissue Contamination Challenges the Credibility of Machine Learning Models in Real World Digital Pathology. Mod Pathol 2024; 37:100422. [PMID: 38185250 PMCID: PMC10960671 DOI: 10.1016/j.modpat.2024.100422] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 11/13/2023] [Accepted: 12/15/2023] [Indexed: 01/09/2024]
Abstract
Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. Although human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole-slide models. Three operate in placenta for the following functions: (1) detection of decidual arteriopathy, (2) estimation of gestational age, and (3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in the t-distributed stochastic neighbor embedding feature space. Every model showed performance degradation in response to one or more tissue contaminants. Decidual arteriopathy detection--balanced accuracy decreased from 0.74 to 0.69 ± 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant, raised the mean absolute error in estimating gestational age from 1.626 weeks to 2.371 ± 0.003 weeks. Blood, incorporated into placental sections, induced false-negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033 mm2, and resulted in a 97% false-positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.
Collapse
Affiliation(s)
- Ismail Irmakci
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ramin Nateghi
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Rujoi Zhou
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Mariavittoria Vescovo
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Madeline Saft
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ashley E Ross
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ximing J Yang
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Lee A D Cooper
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Jeffery A Goldstein
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois.
| |
Collapse
|
10
|
Adachi M, Taki T, Sakamoto N, Kojima M, Hirao A, Matsuura K, Hayashi R, Tabuchi K, Ishikawa S, Ishii G, Sakashita S. Extracting interpretable features for pathologists using weakly supervised learning to predict p16 expression in oropharyngeal cancer. Sci Rep 2024; 14:4506. [PMID: 38402356 PMCID: PMC10894206 DOI: 10.1038/s41598-024-55288-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Accepted: 02/22/2024] [Indexed: 02/26/2024] Open
Abstract
One drawback of existing artificial intelligence (AI)-based histopathological prediction models is the lack of interpretability. The objective of this study is to extract p16-positive oropharyngeal squamous cell carcinoma (OPSCC) features in a form that can be interpreted by pathologists using AI model. We constructed a model for predicting p16 expression using a dataset of whole-slide images from 114 OPSCC biopsy cases. We used the clustering-constrained attention-based multiple-instance learning (CLAM) model, a weakly supervised learning approach. To improve performance, we incorporated tumor annotation into the model (Annot-CLAM) and achieved the mean area under the receiver operating characteristic curve of 0.905. Utilizing the image patches on which the model focused, we examined the features of model interest via histopathologic morphological analysis and cycle-consistent adversarial network (CycleGAN) image translation. The histopathologic morphological analysis evaluated the histopathological characteristics of image patches, revealing significant differences in the numbers of nuclei, the perimeters of the nuclei, and the intercellular bridges between p16-negative and p16-positive image patches. By using the CycleGAN-converted images, we confirmed that the sizes and densities of nuclei are significantly converted. This novel approach improves interpretability in histopathological morphology-based AI models and contributes to the advancement of clinically valuable histopathological morphological features.
Collapse
Affiliation(s)
- Masahiro Adachi
- Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan
- Department of Otolaryngology, Head and Neck Surgery, University of Tsukuba, Tsukuba, Japan
| | - Tetsuro Taki
- Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan
| | - Naoya Sakamoto
- Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan
- Division of Pathology, National Cancer Center Exploratory Oncology Research and Clinical Trial Center, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Motohiro Kojima
- Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan
- Division of Pathology, National Cancer Center Exploratory Oncology Research and Clinical Trial Center, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Akihiko Hirao
- Division of Pathology, National Cancer Center Exploratory Oncology Research and Clinical Trial Center, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
| | - Kazuto Matsuura
- Department of Head and Neck Surgery, National Cancer Center Hospital East, Kashiwa, Japan
| | - Ryuichi Hayashi
- Department of Head and Neck Surgery, National Cancer Center Hospital East, Kashiwa, Japan
| | - Keiji Tabuchi
- Department of Otolaryngology, Head and Neck Surgery, University of Tsukuba, Tsukuba, Japan
| | - Shumpei Ishikawa
- Division of Pathology, National Cancer Center Exploratory Oncology Research and Clinical Trial Center, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan
- Department of Preventive Medicine, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan
| | - Genichiro Ishii
- Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan
- Division of Innovative Pathology and Laboratory Medicine, National Cancer Center Exploratory Oncology Research and Clinical Trial Center, Kashiwa, Japan
| | - Shingo Sakashita
- Department of Pathology and Clinical Laboratories, National Cancer Center Hospital East, Kashiwa, Japan.
- Division of Pathology, National Cancer Center Exploratory Oncology Research and Clinical Trial Center, 6-5-1, Kashiwanoha, Kashiwa, Chiba, 277-8577, Japan.
| |
Collapse
|
11
|
Murchan P, Baird AM, Ó Broin P, Sheils O, Finn SP. Surrogate Biomarker Prediction from Whole-Slide Images for Evaluating Overall Survival in Lung Adenocarcinoma. Diagnostics (Basel) 2024; 14:462. [PMID: 38472935 DOI: 10.3390/diagnostics14050462] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Revised: 02/14/2024] [Accepted: 02/16/2024] [Indexed: 03/14/2024] Open
Abstract
BACKGROUND Recent advances in computational pathology have shown potential in predicting biomarkers from haematoxylin and eosin (H&E) whole-slide images (WSI). However, predicting the outcome directly from WSIs remains a substantial challenge. In this study, we aimed to investigate how gene expression, predicted from WSIs, could be used to evaluate overall survival (OS) in patients with lung adenocarcinoma (LUAD). METHODS Differentially expressed genes (DEGs) were identified from The Cancer Genome Atlas (TCGA)-LUAD cohort. Cox regression analysis was performed on DEGs to identify the gene prognostics of OS. Attention-based multiple instance learning (AMIL) models were trained to predict the expression of identified prognostic genes from WSIs using the TCGA-LUAD dataset. Models were externally validated in the Clinical Proteomic Tumour Analysis Consortium (CPTAC)-LUAD dataset. The prognostic value of predicted gene expression values was then compared to the true gene expression measurements. RESULTS The expression of 239 prognostic genes could be predicted in TCGA-LUAD with cross-validated Pearson's R > 0.4. Predicted gene expression demonstrated prognostic performance, attaining a cross-validated concordance index of up to 0.615 in TCGA-LUAD through Cox regression. In total, 36 genes had predicted expression in the external validation cohort that was prognostic of OS. CONCLUSIONS Gene expression predicted from WSIs is an effective method of evaluating OS in patients with LUAD. These results may open up new avenues of cost- and time-efficient prognosis assessment in LUAD treatment.
Collapse
Affiliation(s)
- Pierre Murchan
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland
- The SFI Centre for Research Training in Genomics Data Science, University of Galway, H91 CF50 Galway, Ireland
- Trinity St. James's Cancer Institute (TSJCI), St. James's Hospital, D08 RX0X Dublin, Ireland
| | - Anne-Marie Baird
- Trinity St. James's Cancer Institute (TSJCI), St. James's Hospital, D08 RX0X Dublin, Ireland
- School of Medicine, Trinity Translational Medicine Institute, Trinity College Dublin, D02 A440 Dublin, Ireland
| | - Pilib Ó Broin
- School of Mathematical & Statistical Sciences, University of Galway, H91 TK33 Galway, Ireland
| | - Orla Sheils
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland
- Trinity St. James's Cancer Institute (TSJCI), St. James's Hospital, D08 RX0X Dublin, Ireland
| | - Stephen P Finn
- Department of Histopathology and Morbid Anatomy, Trinity Translational Medicine Institute, Trinity College Dublin, D08 W9RT Dublin, Ireland
- Trinity St. James's Cancer Institute (TSJCI), St. James's Hospital, D08 RX0X Dublin, Ireland
- Department of Histopathology, St. James's Hospital, James's Street, D08 X4RX Dublin, Ireland
| |
Collapse
|
12
|
El Nahhas OSM, Loeffler CML, Carrero ZI, van Treeck M, Kolbinger FR, Hewitt KJ, Muti HS, Graziani M, Zeng Q, Calderaro J, Ortiz-Brüchle N, Yuan T, Hoffmeister M, Brenner H, Brobeil A, Reis-Filho JS, Kather JN. Regression-based Deep-Learning predicts molecular biomarkers from pathology slides. Nat Commun 2024; 15:1253. [PMID: 38341402 PMCID: PMC10858881 DOI: 10.1038/s41467-024-45589-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Accepted: 01/29/2024] [Indexed: 02/12/2024] Open
Abstract
Deep Learning (DL) can predict biomarkers from cancer histopathology. Several clinically approved applications use this technology. Most approaches, however, predict categorical labels, whereas biomarkers are often continuous measurements. We hypothesize that regression-based DL outperforms classification-based DL. Therefore, we develop and evaluate a self-supervised attention-based weakly supervised regression method that predicts continuous biomarkers directly from 11,671 images of patients across nine cancer types. We test our method for multiple clinically and biologically relevant biomarkers: homologous recombination deficiency score, a clinically used pan-cancer biomarker, as well as markers of key biological processes in the tumor microenvironment. Using regression significantly enhances the accuracy of biomarker prediction, while also improving the predictions' correspondence to regions of known clinical relevance over classification. In a large cohort of colorectal cancer patients, regression-based prediction scores provide a higher prognostic value than classification-based scores. Our open-source regression approach offers a promising alternative for continuous biomarker analysis in computational pathology.
Collapse
Grants
- P30 CA008748 NCI NIH HHS
- JNK is supported by the German Federal Ministry of Health (DEEP LIVER, ZMVI1-2520DAT111) and the Max-Eder-Programme of the German Cancer Aid (grant #70113864), the German Federal Ministry of Education and Research (PEARL, 01KD2104C; CAMINO, 01EO2101; SWAG, 01KD2215A; TRANSFORM LIVER, 031L0312A), the German Academic Exchange Service (SECAI, 57616814), the German Federal Joint Committee (Transplant.KI, 01VSF21048) the European Union (ODELIA, 101057091; GENIAL, 101096312) and the National Institute for Health and Care Research (NIHR, NIHR213331) Leeds Biomedical Research Centre. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.
Collapse
Affiliation(s)
- Omar S M El Nahhas
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Chiara M L Loeffler
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
- Department of Medicine 1, University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Zunamys I Carrero
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Marko van Treeck
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Fiona R Kolbinger
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Katherine J Hewitt
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Hannah S Muti
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany
| | - Mara Graziani
- University of Applied Sciences of Western Switzerland (HES-SO Valais), Rue du Technopole 3, 3960, Sierre, Valais, Switzerland
| | - Qinghe Zeng
- Centre d'Histologie, d'Imagerie et de Cytométrie (CHIC), Centre de Recherche des Cordeliers, INSERM, Sorbonne Université, Université Paris Cité, Paris, France
| | - Julien Calderaro
- Assistance Publique-Hôpitaux de Paris, Département de Pathologie, CHU Henri Mondor, F-94000, Créteil, France
| | - Nadina Ortiz-Brüchle
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
- Center for Integrated Oncology Aachen Bonn Cologne Duesseldorf (CIO ABCD), Cologne, Germany
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Brobeil
- Institute of Pathology, University Hospital Heidelberg, 69120, Heidelberg, Germany
- Tissue Bank, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, 69120, Heidelberg, Germany
| | - Jorge S Reis-Filho
- Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
- Department of Medicine 1, University Hospital and Faculty of Medicine Carl Gustav Carus, TUD Dresden University of Technology, Dresden, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
13
|
Knudsen JE, Rich JM, Ma R. Artificial Intelligence in Pathomics and Genomics of Renal Cell Carcinoma. Urol Clin North Am 2024; 51:47-62. [PMID: 37945102 DOI: 10.1016/j.ucl.2023.06.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2023]
Abstract
The integration of artificial intelligence (AI) with histopathology images and gene expression patterns has led to the emergence of the dynamic fields of pathomics and genomics. These fields have revolutionized renal cell carcinoma (RCC) diagnosis and subtyping and improved survival prediction models. Machine learning has identified unique gene patterns across RCC subtypes and grades, providing insights into RCC origins and potential treatments, as targeted therapies. The combination of pathomics and genomics using AI opens new avenues in RCC research, promising future breakthroughs and innovations that patients and physicians can anticipate.
Collapse
Affiliation(s)
- J Everett Knudsen
- Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, Center for Robotic Simulation & Education, University of Southern California, Los Angeles, CA, USA
| | - Joseph M Rich
- Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, Center for Robotic Simulation & Education, University of Southern California, Los Angeles, CA, USA
| | - Runzhuo Ma
- Catherine & Joseph Aresty Department of Urology, USC Institute of Urology, Center for Robotic Simulation & Education, University of Southern California, Los Angeles, CA, USA.
| |
Collapse
|
14
|
Truhn D, Tayebi Arasteh S, Saldanha OL, Müller-Franzes G, Khader F, Quirke P, West NP, Gray R, Hutchins GGA, James JA, Loughrey MB, Salto-Tellez M, Brenner H, Brobeil A, Yuan T, Chang-Claude J, Hoffmeister M, Foersch S, Han T, Keil S, Schulze-Hagen M, Isfort P, Bruners P, Kaissis G, Kuhl C, Nebelung S, Kather JN. Encrypted federated learning for secure decentralized collaboration in cancer image analysis. Med Image Anal 2024; 92:103059. [PMID: 38104402 PMCID: PMC10804934 DOI: 10.1016/j.media.2023.103059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 04/28/2023] [Accepted: 12/05/2023] [Indexed: 12/19/2023]
Abstract
Artificial intelligence (AI) has a multitude of applications in cancer research and oncology. However, the training of AI systems is impeded by the limited availability of large datasets due to data protection requirements and other regulatory obstacles. Federated and swarm learning represent possible solutions to this problem by collaboratively training AI models while avoiding data transfer. However, in these decentralized methods, weight updates are still transferred to the aggregation server for merging the models. This leaves the possibility for a breach of data privacy, for example by model inversion or membership inference attacks by untrusted servers. Somewhat-homomorphically-encrypted federated learning (SHEFL) is a solution to this problem because only encrypted weights are transferred, and model updates are performed in the encrypted space. Here, we demonstrate the first successful implementation of SHEFL in a range of clinically relevant tasks in cancer image analysis on multicentric datasets in radiology and histopathology. We show that SHEFL enables the training of AI models which outperform locally trained models and perform on par with models which are centrally trained. In the future, SHEFL can enable multiple institutions to co-train AI models without forsaking data governance and without ever transmitting any decryptable data to untrusted servers.
Collapse
Affiliation(s)
- Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany.
| | - Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Oliver Lester Saldanha
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany; Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Philip Quirke
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Nicholas P West
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Richard Gray
- Clinical Trial Service Unit, University of Oxford, Oxford, United Kingdom
| | - Gordon G A Hutchins
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
| | - Jacqueline A James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, United Kingdom; Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, United Kingdom; The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, United Kingdom
| | - Maurice B Loughrey
- The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, United Kingdom; Department of Cellular Pathology, Belfast Health and Social Care Trust, Belfast, United Kingdom; Centre for Public Health, Queen's University Belfast, Belfast, United Kingdom
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, United Kingdom; Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, United Kingdom; The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, United Kingdom
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany; Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Alexander Brobeil
- Institute of Pathology, University Hospital Heidelberg, Heidelberg, Germany; Tissue Bank, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany; Medical Faculty Heidelberg, Heidelberg University, Heidelberg, Germany
| | - Jenny Chang-Claude
- Cancer Epidemiology Group, University Cancer Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany; Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Sebastian Foersch
- Institute of Pathology, University Medical Center Mainz, Mainz, Germany
| | - Tianyu Han
- Physics of Molecular Imaging Systems, Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany
| | - Sebastian Keil
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Maximilian Schulze-Hagen
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Peter Isfort
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Philipp Bruners
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Georgios Kaissis
- Institute of Diagnostic and Interventional Radiology, Technical University of Munich, Munich, Germany; Artificial Intelligence in Medicine and Healthcare, Technical University of Munich, Munich, Germany; Department of Computing, Imperial College London, London, United Kingdom
| | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany; Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany; Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| |
Collapse
|
15
|
Ligero M, Serna G, El Nahhas OS, Sansano I, Mauchanski S, Viaplana C, Calderaro J, Toledo RA, Dienstmann R, Vanguri RS, Sauter JL, Sanchez-Vega F, Shah SP, Ramón y Cajal S, Garralda E, Nuciforo P, Perez-Lopez R, Kather JN. Weakly Supervised Deep Learning Predicts Immunotherapy Response in Solid Tumors Based on PD-L1 Expression. Cancer Res Commun 2024; 4:92-102. [PMID: 38126740 PMCID: PMC10782919 DOI: 10.1158/2767-9764.crc-23-0287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Revised: 10/11/2023] [Accepted: 12/12/2023] [Indexed: 12/23/2023]
Abstract
Programmed death-ligand 1 (PD-L1) IHC is the most commonly used biomarker for immunotherapy response. However, quantification of PD-L1 status in pathology slides is challenging. Neither manual quantification nor a computer-based mimicking of manual readouts is perfectly reproducible, and the predictive performance of both approaches regarding immunotherapy response is limited. In this study, we developed a deep learning (DL) method to predict PD-L1 status directly from raw IHC image data, without explicit intermediary steps such as cell detection or pigment quantification. We trained the weakly supervised model on PD-L1-stained slides from the non-small cell lung cancer (NSCLC)-Memorial Sloan Kettering (MSK) cohort (N = 233) and validated it on the pan-cancer-Vall d'Hebron Institute of Oncology (VHIO) cohort (N = 108). We also investigated the performance of the model to predict response to immune checkpoint inhibitors (ICI) in terms of progression-free survival. In the pan-cancer-VHIO cohort, the performance was compared with tumor proportion score (TPS) and combined positive score (CPS). The DL model showed good performance in predicting PD-L1 expression (TPS ≥ 1%) in both NSCLC-MSK and pan-cancer-VHIO cohort (AUC 0.88 ± 0.06 and 0.80 ± 0.03, respectively). The predicted PD-L1 status showed an improved association with response to ICIs [HR: 1.5 (95% confidence interval: 1-2.3), P = 0.049] compared with TPS [HR: 1.4 (0.96-2.2), P = 0.082] and CPS [HR: 1.2 (0.79-1.9), P = 0.386]. Notably, our explainability analysis showed that the model does not just look at the amount of brown pigment in the IHC slides, but also considers morphologic factors such as lymphocyte conglomerates. Overall, end-to-end weakly supervised DL shows potential for improving patient stratification for cancer immunotherapy by analyzing PD-L1 IHC, holistically integrating morphology and PD-L1 staining intensity. SIGNIFICANCE The weakly supervised DL model to predict PD-L1 status from raw IHC data, integrating tumor staining intensity and morphology, enables enhanced patient stratification in cancer immunotherapy compared with traditional pathologist assessment.
Collapse
Affiliation(s)
- Marta Ligero
- Radiomics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Garazi Serna
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Omar S.M. El Nahhas
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Irene Sansano
- Pathology Department, Vall d'Hebron University Hospital (VHUH), Barcelona, Spain
| | - Siarhei Mauchanski
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Cristina Viaplana
- Oncology Data Science (ODysSey) Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Julien Calderaro
- Assistance Publique-Hôpitaux de Paris, Département de Pathologie, CHU Henri Mondor, Créteil, France
- Université Paris-Est Créteil, Faculté de Médecine, Créteil, France
| | - Rodrigo A. Toledo
- Biomakers and Clonal Dynamics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Rodrigo Dienstmann
- Oncology Data Science (ODysSey) Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Rami S. Vanguri
- Department of Pathology and Laboratory Medicine, Children's Hospital of Philadelphia, Philadelphia, Pennsylvania
| | - Jennifer L. Sauter
- Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York
| | | | - Sohrab P. Shah
- Computational Oncology, Memorial Sloan Kettering Cancer Center, New York, New York
| | | | - Elena Garralda
- Department of Medical Oncology, Vall d'Hebron University Hospital and Institute of Oncology (VHIO), Barcelona, Spain
| | - Paolo Nuciforo
- Molecular Oncology Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Raquel Perez-Lopez
- Radiomics Group, Vall d'Hebron Institute of Oncology (VHIO), Barcelona, Spain
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom
- Department of Medicine I, University Hospital Dresden, Dresden, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| |
Collapse
|
16
|
Liu P, Ji L, Ye F, Fu B. AdvMIL: Adversarial multiple instance learning for the survival analysis on whole-slide images. Med Image Anal 2024; 91:103020. [PMID: 37926034 DOI: 10.1016/j.media.2023.103020] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2022] [Revised: 04/05/2023] [Accepted: 10/31/2023] [Indexed: 11/07/2023]
Abstract
The survival analysis on histological whole-slide images (WSIs) is one of the most important means to estimate patient prognosis. Although many weakly-supervised deep learning models have been developed for gigapixel WSIs, their potential is generally restricted by classical survival analysis rules and fully-supervised learning requirements. As a result, these models provide patients only with a completely-certain point estimation of time-to-event, and they could only learn from the labeled WSI data currently at a small scale. To tackle these problems, we propose a novel adversarial multiple instance learning (AdvMIL) framework. This framework is based on adversarial time-to-event modeling, and integrates the multiple instance learning (MIL) that is much necessary for WSI representation learning. It is a plug-and-play one, so that most existing MIL-based end-to-end methods can be easily upgraded by applying this framework, gaining the improved abilities of survival distribution estimation and semi-supervised learning. Our extensive experiments show that AdvMIL not only could often bring performance improvement to mainstream WSI survival analysis methods at a relatively low computational cost, but also enables these methods to effectively utilize unlabeled data via semi-supervised learning. Moreover, it is observed that AdvMIL could help improving the robustness of models against patch occlusion and two representative image noises. The proposed AdvMIL framework could promote the research of survival analysis in computational pathology with its novel adversarial MIL paradigm.
Collapse
Affiliation(s)
- Pei Liu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| | - Luping Ji
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| | - Feng Ye
- Institute of Clinical Pathology, West China Hospital, Sichuan University, Guo Xue Xiang, Chengdu, 610041, Sichuan, China.
| | - Bo Fu
- School of Computer Science and Engineering, University of Electronic Science and Technology of China, Xiyuan Ave, Chengdu, 611731, Sichuan, China.
| |
Collapse
|
17
|
Mukashyaka P, Sheridan TB, Foroughi Pour A, Chuang JH. SAMPLER: unsupervised representations for rapid analysis of whole slide tissue images. EBioMedicine 2024; 99:104908. [PMID: 38101298 PMCID: PMC10733087 DOI: 10.1016/j.ebiom.2023.104908] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 11/27/2023] [Accepted: 11/27/2023] [Indexed: 12/17/2023] Open
Abstract
BACKGROUND Deep learning has revolutionized digital pathology, allowing automatic analysis of hematoxylin and eosin (H&E) stained whole slide images (WSIs) for diverse tasks. WSIs are broken into smaller images called tiles, and a neural network encodes each tile. Many recent works use supervised attention-based models to aggregate tile-level features into a slide-level representation, which is then used for downstream analysis. Training supervised attention-based models is computationally intensive, architecture optimization of the attention module is non-trivial, and labeled data are not always available. Therefore, we developed an unsupervised and fast approach called SAMPLER to generate slide-level representations. METHODS Slide-level representations of SAMPLER are generated by encoding the cumulative distribution functions of multiscale tile-level features. To assess effectiveness of SAMPLER, slide-level representations of breast carcinoma (BRCA), non-small cell lung carcinoma (NSCLC), and renal cell carcinoma (RCC) WSIs of The Cancer Genome Atlas (TCGA) were used to train separate classifiers distinguishing tumor subtypes in FFPE and frozen WSIs. In addition, BRCA and NSCLC classifiers were externally validated on frozen WSIs. Moreover, SAMPLER's attention maps identify regions of interest, which were evaluated by a pathologist. To determine time efficiency of SAMPLER, we compared runtime of SAMPLER with two attention-based models. SAMPLER concepts were used to improve the design of a context-aware multi-head attention model (context-MHA). FINDINGS SAMPLER-based classifiers were comparable to state-of-the-art attention deep learning models to distinguish subtypes of BRCA (AUC = 0.911 ± 0.029), NSCLC (AUC = 0.940 ± 0.018), and RCC (AUC = 0.987 ± 0.006) on FFPE WSIs (internal test sets). However, training SAMLER-based classifiers was >100 times faster. SAMPLER models successfully distinguished tumor subtypes on both internal and external test sets of frozen WSIs. Histopathological review confirmed that SAMPLER-identified high attention tiles contained subtype-specific morphological features. The improved context-MHA distinguished subtypes of BRCA and RCC (BRCA-AUC = 0.921 ± 0.027, RCC-AUC = 0.988 ± 0.010) with increased accuracy on internal test FFPE WSIs. INTERPRETATION Our unsupervised statistical approach is fast and effective for analyzing WSIs, with greatly improved scalability over attention-based deep learning methods. The high accuracy of SAMPLER-based classifiers and interpretable attention maps suggest that SAMPLER successfully encodes the distinct morphologies within WSIs and will be applicable to general histology image analysis problems. FUNDING This study was supported by the National Cancer Institute (Grant No. R01CA230031 and P30CA034196).
Collapse
Affiliation(s)
- Patience Mukashyaka
- The Jackson Laboratory for Genomic Medicine, Farmington, CT, USA; Department of Genetics and Genome Sciences, University of Connecticut Health Center, Farmington, CT, USA
| | - Todd B Sheridan
- The Jackson Laboratory for Genomic Medicine, Farmington, CT, USA; Department of Pathology, Hartford Hospital, Hartford, CT, USA
| | | | - Jeffrey H Chuang
- The Jackson Laboratory for Genomic Medicine, Farmington, CT, USA; Department of Genetics and Genome Sciences, University of Connecticut Health Center, Farmington, CT, USA.
| |
Collapse
|
18
|
Vafaei Sadr A, Bülow R, von Stillfried S, Schmitz NEJ, Pilva P, Hölscher DL, Ha PP, Schweiker M, Boor P. Operational greenhouse-gas emissions of deep learning in digital pathology: a modelling study. Lancet Digit Health 2024; 6:e58-e69. [PMID: 37996339 PMCID: PMC10728828 DOI: 10.1016/s2589-7500(23)00219-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 10/04/2023] [Accepted: 10/16/2023] [Indexed: 11/25/2023]
Abstract
BACKGROUND Deep learning is a promising way to improve health care. Image-processing medical disciplines, such as pathology, are expected to be transformed by deep learning. The first clinically applicable deep-learning diagnostic support tools are already available in cancer pathology, and their number is increasing. However, data on the environmental sustainability of these tools are scarce. We aimed to conduct an environmental-sustainability analysis of a theoretical implementation of deep learning in patient-care pathology. METHODS For this modelling study, we first assembled and calculated relevant data and parameters of a digital-pathology workflow. Data were breast and prostate specimens from the university clinic at the Institute of Pathology of the Rheinisch-Westfälische Technische Hochschule Aachen (Aachen, Germany), for which commercially available deep learning was already available. Only specimens collected between Jan 1 and Dec 31, 2019 were used, to omit potential biases due to the COVID-19 pandemic. Our final selection was based on 2 representative weeks outside holidays, covering different types of specimens. To calculate carbon dioxide (CO2) or CO2 equivalent (CO2 eq) emissions of deep learning in pathology, we gathered relevant data for exact numbers and sizes of whole-slide images (WSIs), which were generated by scanning histopathology samples of prostate and breast specimens. We also evaluated different data input scenarios (including all slide tiles, only tiles containing tissue, or only tiles containing regions of interest). To convert estimated energy consumption from kWh to CO2 eq, we used the internet protocol address of the computational server and the Electricity Maps database to obtain information on the sources of the local electricity grid (ie, renewable vs non-renewable), and estimated the number of trees and proportion of the local and world's forests needed to sequester the CO2 eq emissions. We calculated the computational requirements and CO2 eq emissions of 30 deep-learning models that varied in task and size. The first scenario represented the use of one commercially available deep-learning model for one task in one case (1-task), the second scenario considered two deep-learning models for two tasks per case (2-task), the third scenario represented a future, potentially automated workflow that could handle 7 tasks per case (7-task), and the fourth scenario represented the use of a single potential, large, computer-vision model that could conduct multiple tasks (multitask). We also compared the performance (ie, accuracy) and CO2 eq emissions of different deep-learning models for the classification of renal cell carcinoma on WSIs, also from Rheinisch-Westfälische Technische Hochschule Aachen. We also tested other approaches to reducing CO2 eq emissions, including model pruning and an alternative method for histopathology analysis (pathomics). FINDINGS The pathology database contained 35 552 specimens (237 179 slides), 6420 of which were prostate specimens (10 115 slides) and 11 801 of which were breast specimens (19 763 slides). We selected and subsequently digitised 140 slides from eight breast-cancer cases and 223 slides from five prostate-cancer cases. Applying large deep-learning models on all WSI tiles of prostate and breast pathology cases would result in yearly CO2 eq emissions of 7·65 metric tons (t; 95% CI 7·62-7·68) with the use of a single deep-learning model per case; yearly CO2 eq emissions were up to 100·56 t (100·21-100·99) with the use of seven deep-learning models per case. CO2 eq emissions for different deep-learning model scenarios, data inputs, and deep-learning model sizes for all slides varied from 3·61 t (3·59-3·63) to 2795·30 t (1177·51-6482·13. For the estimated number of overall pathology cases worldwide, the yearly CO2 eq emissions varied, reaching up to 16 megatons (Mt) of CO2 eq, requiring up to 86 590 km2 (0·22%) of world forest to sequester the CO2 eq emissions. Use of the 7-task scenario and small deep-learning models on slides containing tissue only could substantially reduce CO2 eq emissions worldwide by up to 141 times (0·1 Mt, 95% CI 0·1-0·1). Considering the local environment in Aachen, Germany, the maximum CO2 eq emission from the use of deep learning in digital pathology only would require 32·8% (95% CI 13·8-76·6) of the local forest to sequester the CO2 eq emissions. A single pathomics run on a tissue could provide information that was comparable to or even better than the output of multitask deep-learning models, but with 147 times reduced CO2 eq emissions. INTERPRETATION Our findings suggest that widespread use of deep learning in pathology might have considerable global-warming potential. The medical community, policy decision makers, and the public should be aware of this potential and encourage the use of CO2 eq emissions reduction strategies where possible. FUNDING German Research Foundation, European Research Council, German Federal Ministry of Education and Research, Health, Economic Affairs and Climate Action, and the Innovation Fund of the Federal Joint Committee.
Collapse
Affiliation(s)
- Alireza Vafaei Sadr
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany; Department of Public Health Sciences, College of Medicine, Pennsylvania State University, Hershey, PA, USA
| | - Roman Bülow
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Saskia von Stillfried
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Nikolas E J Schmitz
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Pourya Pilva
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - David L Hölscher
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Peiman Pilehchi Ha
- Healthy Living Spaces Lab, Institute for Occupational, Social and Environmental Medicine, Medical Faculty, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Marcel Schweiker
- Healthy Living Spaces Lab, Institute for Occupational, Social and Environmental Medicine, Medical Faculty, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany
| | - Peter Boor
- Institute of Pathology, University Hospital Aachen, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany; Department of Nephrology and Immunology, Rheinisch-Westfälische Technische Hochschule Aachen, Aachen, Germany.
| |
Collapse
|
19
|
Kumar N, Marée R, Geurts P, Muller M. Recent Advances in Bioimage Analysis Methods for Detecting Skeletal Deformities in Biomedical and Aquaculture Fish Species. Biomolecules 2023; 13:1797. [PMID: 38136667 PMCID: PMC10742266 DOI: 10.3390/biom13121797] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 12/05/2023] [Accepted: 12/09/2023] [Indexed: 12/24/2023] Open
Abstract
Detecting skeletal or bone-related deformities in model and aquaculture fish is vital for numerous biomedical studies. In biomedical research, model fish with bone-related disorders are potential indicators of various chemically induced toxins in their environment or poor dietary conditions. In aquaculture, skeletal deformities are affecting fish health, and economic losses are incurred by fish farmers. This survey paper focuses on showcasing the cutting-edge image analysis tools and techniques based on artificial intelligence that are currently applied in the analysis of bone-related deformities in aquaculture and model fish. These methods and tools play a significant role in improving research by automating various aspects of the analysis. This paper also sheds light on some of the hurdles faced when dealing with high-content bioimages and explores potential solutions to overcome these challenges.
Collapse
Affiliation(s)
- Navdeep Kumar
- Department of Computer Science and Electrical Engineering, Montefiore Institute, University of Liège, 4000 Liège, Belgium; (R.M.); (P.G.)
| | - Raphaël Marée
- Department of Computer Science and Electrical Engineering, Montefiore Institute, University of Liège, 4000 Liège, Belgium; (R.M.); (P.G.)
| | - Pierre Geurts
- Department of Computer Science and Electrical Engineering, Montefiore Institute, University of Liège, 4000 Liège, Belgium; (R.M.); (P.G.)
| | - Marc Muller
- Laboratory for Organogenesis and Regeneration (LOR), GIGA Institute, University of Liège, 4000 Liège, Belgium;
| |
Collapse
|
20
|
Calderaro J, Ghaffari Laleh N, Zeng Q, Maille P, Favre L, Pujals A, Klein C, Bazille C, Heij LR, Uguen A, Luedde T, Di Tommaso L, Beaufrère A, Chatain A, Gastineau D, Nguyen CT, Nguyen-Canh H, Thi KN, Gnemmi V, Graham RP, Charlotte F, Wendum D, Vij M, Allende DS, Aucejo F, Diaz A, Rivière B, Herrero A, Evert K, Calvisi DF, Augustin J, Leow WQ, Leung HHW, Boleslawski E, Rela M, François A, Cha AWH, Forner A, Reig M, Allaire M, Scatton O, Chatelain D, Boulagnon-Rombi C, Sturm N, Menahem B, Frouin E, Tougeron D, Tournigand C, Kempf E, Kim H, Ningarhari M, Michalak-Provost S, Gopal P, Brustia R, Vibert E, Schulze K, Rüther DF, Weidemann SA, Rhaiem R, Pawlotsky JM, Zhang X, Luciani A, Mulé S, Laurent A, Amaddeo G, Regnault H, De Martin E, Sempoux C, Navale P, Westerhoff M, Lo RCL, Bednarsch J, Gouw A, Guettier C, Lequoy M, Harada K, Sripongpun P, Wetwittayaklang P, Loménie N, Tantipisit J, Kaewdech A, Shen J, Paradis V, Caruso S, Kather JN. Deep learning-based phenotyping reclassifies combined hepatocellular-cholangiocarcinoma. Nat Commun 2023; 14:8290. [PMID: 38092727 PMCID: PMC10719304 DOI: 10.1038/s41467-023-43749-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2023] [Accepted: 11/17/2023] [Indexed: 12/17/2023] Open
Abstract
Primary liver cancer arises either from hepatocytic or biliary lineage cells, giving rise to hepatocellular carcinoma (HCC) or intrahepatic cholangiocarcinoma (ICCA). Combined hepatocellular- cholangiocarcinomas (cHCC-CCA) exhibit equivocal or mixed features of both, causing diagnostic uncertainty and difficulty in determining proper management. Here, we perform a comprehensive deep learning-based phenotyping of multiple cohorts of patients. We show that deep learning can reproduce the diagnosis of HCC vs. CCA with a high performance. We analyze a series of 405 cHCC-CCA patients and demonstrate that the model can reclassify the tumors as HCC or ICCA, and that the predictions are consistent with clinical outcomes, genetic alterations and in situ spatial gene expression profiling. This type of approach could improve treatment decisions and ultimately clinical outcome for patients with rare and biphenotypic cancers such as cHCC-CCA.
Collapse
Affiliation(s)
- Julien Calderaro
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France.
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France.
- Inserm, U955, Team 18, Créteil, France.
- European Reference Network (ERN) RARE-LIVER, Créteil, France.
| | - Narmin Ghaffari Laleh
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, RWTH Aachen university, Aachen, Germany
| | - Qinghe Zeng
- Centre d'Histologie, d'Imagerie et de Cytométrie (CHIC), Centre de Recherche des Cordeliers, Paris, France
- Laboratoire d'Informatique Paris Descartes (LIPADE), Université Paris Cité, Paris, France
| | - Pascale Maille
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- Inserm, U955, Team 18, Créteil, France
| | - Loetitia Favre
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- Inserm, U955, Team 18, Créteil, France
| | - Anaïs Pujals
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- Inserm, U955, Team 18, Créteil, France
| | - Christophe Klein
- Centre d'Histologie, d'Imagerie et de Cytométrie (CHIC), Centre de Recherche des Cordeliers, Paris, France
- INSERM, Sorbonne Université, Université Paris Cité, Paris, France
| | - Céline Bazille
- Caen University Hospital, Department of Pathology, Caen, France
| | - Lara R Heij
- Department of Surgery and Transplantation, University Hospital RWTH Aachen, Aachen, Germany
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Arnaud Uguen
- CHRU Brest, Department of Pathology, Brest, 29220, France
- Univ Brest, Inserm, CHU de Brest, LBAI, UMR1227, Brest, France
| | - Tom Luedde
- Clinic for Gastroenterology, Hepatology and Infectious Diseases, University Hospital Düsseldorf, Medical Faculty of Heinrich Heine University Düsseldorf, Düsseldorf, Germany
| | - Luca Di Tommaso
- Department of Pathology, Humanitas University, Humanitas Clinical and Research Center, IRCCS, Rozzano, Milan, Italy
| | - Aurélie Beaufrère
- Assistance Publique-Hôpitaux de Paris, Beaujon University Hospital, Department of Pathology, F-92110, Clichy, France
- Université de Paris, Inflammation Research Center, Inserm, U1149, CNRS, ERL8252, F-75018, Paris, France
| | | | | | - Cong Trung Nguyen
- Department of Pathology, E Hospital, Hanoi Medical University, Hanoi, Vietnam
| | - Hiep Nguyen-Canh
- Pathology Center, Bachmai Hospital, Hanoi, Vietnam
- Department of Human Pathology, Kanazawa University Graduate School of Medicine, Kanazawa, Japan
| | - Khuyen Nguyen Thi
- Pathology and Molecular biology Center, National Cancer Hospital, Hanoi, Vietnam
| | - Viviane Gnemmi
- University Lille, UMR9020-U1277, Cancer Heterogeneity Plasticity and Resistance to Therapies, Lille, France
- CHU Lille, Institute of Pathology, Lille, France
| | - Rondell P Graham
- Department of Laboratory Medicine and Pathology, Mayo Clinic Rochester, Rochester, MN, USA
| | - Frédéric Charlotte
- Assistance Publique-Hôpitaux de Paris, Pitié-Salpêtrière University Hospital, Department of Pathology, Paris, France
| | - Dominique Wendum
- Assistance Publique-Hôpitaux de Paris, Saint-Antoine University Hospital, Department of Pathology, Paris, France
| | - Mukul Vij
- Department of Pathology, Dr Rela Institute and Medical Centre, Bharath Institute of Higher Education and Research, Chennai, India
| | - Daniela S Allende
- Department of Hepatobiliary Pathology, Cleveland Clinic Foundation, Cleveland, OH, USA
- Robert J. Tomsich Pathology & Laboratory Medicine Institute, Cleveland Clinic, 9500 Euclid Avenue, L25, Cleveland, OH, 44195, USA
| | - Federico Aucejo
- Department of Gastrointestinal and Hepatobiliary Surgery, Cleveland Clinic Foundation, Cleveland, OH, USA
| | - Alba Diaz
- Barcelona Clinic Liver Cancer (BCLC) Group, Department of Pathology, Hospital Clínic de Barcelona, Universitat de Barcelona, Barcelona, Spain
| | - Benjamin Rivière
- Department of Pathology, Gui-de-Chauliac University Hospital, 80, avenue Augustin-Fliche, 34295, Montpellier, France
| | - Astrid Herrero
- Department of Digestive and Hepatobiliary Surgery, Gui-de-Chauliac University Hospital, 80, avenue Augustin-Fliche, 34295, Montpellier, France
| | - Katja Evert
- Institute of Pathology, University of Regensburg, Franz-Josef-Strauß-Allee 11, 93053, Regensburg, Germany
| | - Diego Francesco Calvisi
- Institute of Pathology, University of Regensburg, Franz-Josef-Strauß-Allee 11, 93053, Regensburg, Germany
| | - Jérémy Augustin
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- Inserm, U955, Team 18, Créteil, France
| | - Wei Qiang Leow
- Department of Anatomical Pathology, Singapore General Hospital, Singapore, Singapore
| | - Howard Ho Wai Leung
- Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | | | - Mohamed Rela
- Dr Rela Institute and Medical Centre, Bharath Institute of Higher Education and Research, Chennai, India
| | - Arnaud François
- Rouen University Hospital, Department of Pathology, Rouen, France
| | - Anthony Wing-Hung Cha
- Department of Anatomical and Cellular Pathology, The Chinese University of Hong Kong, Shatin, Hong Kong
| | - Alejandro Forner
- Barcelona Clinic Liver Cancer (BCLC), Liver Unit, Hospital Clinic of Barcelona, IDIBAPS, CIBEREHD, Universidad de Barcelona, Barcelona, Spain
| | - Maria Reig
- Barcelona Clinic Liver Cancer (BCLC), Liver Unit, Hospital Clinic of Barcelona, IDIBAPS, CIBEREHD, Universidad de Barcelona, Barcelona, Spain
| | - Manon Allaire
- Assistance Publique-Hôpitaux de Paris, Pitié-Salpêtrière University Hospital, Department of Hepatology, Paris, France
| | - Olivier Scatton
- Assistance Publique-Hôpitaux de Paris, Pitié-Salpêtrière University Hospital, Department of Digestive and Hepatobiliary Surgery, Paris, France
| | - Denis Chatelain
- Centre Hospitalier Universitaire d'Amiens, Département de Pathologie, Amiens, France
| | | | - Nathalie Sturm
- Department of Pathology, University Hospital, Grenoble, France
- Translational Innovation in Medicine and Complexity, Centre National de la Recherche Scientifique UMR5525, La Tronche, France
| | - Benjamin Menahem
- Caen University Hospital, Department of Digestive and Hepatobiliary Surgery, Caen, France
| | - Eric Frouin
- Poitiers University Hospital, Department of Pathology, Poitiers, France
- LITEC, Université de Poitiers, Poitiers, France
| | - David Tougeron
- Poitiers University Hospital, Department of Hepatogastroenterology and Oncology, Poitiers, France
| | - Christophe Tournigand
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Medical Oncology, Créteil, France
| | - Emmanuelle Kempf
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Medical Oncology, Créteil, France
| | - Haeryoung Kim
- Department of Pathology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Korea
| | | | | | - Purva Gopal
- Department of Pathology, UT Southwestern Medical Center, Dallas, TX, USA
| | - Raffaele Brustia
- Assistance Publique-Hôpitaux de Paris, Henri Mondor University Hospital, Department of Digestive and Hepatobiliary Surgery, Créteil, France
| | - Eric Vibert
- Assistance Publique-Hôpitaux de Paris, Paul Brousse University Hospital, Department of Digestive and Hepatobiliary Surgery, Paris, France
| | - Kornelius Schulze
- Department of Internal Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Darius F Rüther
- Department of Internal Medicine, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Sören A Weidemann
- Department of Pathology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Rami Rhaiem
- Reims University Hospital, Department of Digestive and Hepatobiliary Surgery, Reims, France
| | - Jean-Michel Pawlotsky
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France
- Inserm, U955, Team 18, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Paul Brousse University Hospital, Department of Digestive and Hepatobiliary Surgery, Paris, France
| | - Xuchen Zhang
- Department of Pathology, Yale University School of Medicine, New Haven, CT, USA
| | - Alain Luciani
- Assistance Publique-Hôpitaux de Paris, Henri Mondor University Hospital, Department of Medical Imaging, Créteil, France
| | - Sébastien Mulé
- Assistance Publique-Hôpitaux de Paris, Henri Mondor University Hospital, Department of Medical Imaging, Créteil, France
| | - Alexis Laurent
- Assistance Publique-Hôpitaux de Paris, Henri Mondor University Hospital, Department of Digestive and Hepatobiliary Surgery, Créteil, France
| | - Giuliana Amaddeo
- Assistance Publique-Hôpitaux de Paris, Henri Mondor University Hospital, Department of Hepatology, Créteil, France
| | - Hélène Regnault
- Assistance Publique-Hôpitaux de Paris, Henri Mondor University Hospital, Department of Hepatology, Créteil, France
| | - Eleonora De Martin
- Assistance Publique-Hôpitaux de Paris, Paul Brousse University Hospital, Department of Hepatology, Paris, France
| | - Christine Sempoux
- Institute of Pathology, Lausanne University Hospital (CHUV), University of Lausanne (UNIL), Lausanne, Switzerland
| | - Pooja Navale
- Department of Pathology and Immunology, Washington University in St. Louis, St. Louis, MO, 63110, USA
| | - Maria Westerhoff
- Department of Pathology University of Michigan, Ann Arbor, MI, USA
| | - Regina Cheuk-Lam Lo
- Department of Pathology, The University of Hong Kong, Pok Fu Lam, Hong Kong, China
- State Key Laboratory of Liver Research, (The University of Hong Kong), Pok Fu Lam, Hong Kong, China
| | - Jan Bednarsch
- Department of Surgery and Transplantation, University Hospital RWTH Aachen, Aachen, Germany
| | - Annette Gouw
- Department of Pathology and Medical Biology, University Medical Center Groningen, Groningen, the Netherlands
| | - Catherine Guettier
- Department of Surgery and Transplantation, University Hospital RWTH Aachen, Aachen, Germany
- Department of Pathology and Medical Biology, University Medical Center Groningen, Groningen, the Netherlands
- Assistance Publique-Hôpitaux de Paris, Paul Brousse University Hospital, Department of Pathology, Villejuif, France
| | - Marie Lequoy
- Assistance Publique-Hôpitaux de Paris, Saint Antoine University Hospital, Department of Hepatology, Paris, France
| | - Kenichi Harada
- Department of Human Pathology, Kanazawa University Graduate School of Medicine, Kanazawa, Japan
| | - Pimsiri Sripongpun
- Gastroenterology and Hepatology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Hat Yai, Thailand
| | | | - Nicolas Loménie
- Laboratoire d'Informatique Paris Descartes (LIPADE), Université Paris Cité, Paris, France
| | - Jarukit Tantipisit
- Prince of Songkla University, Department of Pathology, Hat Yai, Thailand
| | - Apichat Kaewdech
- Gastroenterology and Hepatology Unit, Division of Internal Medicine, Faculty of Medicine, Prince of Songkla University, Hat Yai, Thailand
| | - Jeanne Shen
- Center for Artificial Intelligence in Medicine and Imaging, Stanford University, 1701 Page Mill Road, Palo Alto, CA, 94304, USA
- Department of Pathology, Stanford University School of Medicine, 300 Pasteur Drive, Stanford, CA, 94305, USA
| | - Valérie Paradis
- Assistance Publique-Hôpitaux de Paris, Beaujon University Hospital, Department of Pathology, F-92110, Clichy, France
- Université de Paris, Inflammation Research Center, Inserm, U1149, CNRS, ERL8252, F-75018, Paris, France
| | - Stefano Caruso
- Université Paris Est Créteil, INSERM, IMRB, F-94010, Créteil, France
- Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Department of Pathology, Créteil, France
- Inserm, U955, Team 18, Créteil, France
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
21
|
Cen M, Li X, Guo B, Jonnagaddala J, Zhang H, Xu XS. A Novel and Efficient Digital Pathology Classifier for Predicting Cancer Biomarkers Using Sequencer Architecture. Am J Pathol 2023; 193:2122-2132. [PMID: 37775043 DOI: 10.1016/j.ajpath.2023.09.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/21/2023] [Revised: 08/16/2023] [Accepted: 09/01/2023] [Indexed: 10/01/2023]
Abstract
In digital pathology tasks, transformers have achieved state-of-the-art results, surpassing convolutional neural networks (CNNs). However, transformers are usually complex and resource intensive. This study developed a novel and efficient digital pathology classifier called DPSeq to predict cancer biomarkers through fine-tuning a sequencer architecture integrating horizontal and vertical bidirectional long short-term memory networks. Using hematoxylin and eosin-stained histopathologic images of colorectal cancer from two international data sets (The Cancer Genome Atlas and Molecular and Cellular Oncology), the predictive performance of DPSeq was evaluated in a series of experiments. DPSeq demonstrated exceptional performance for predicting key biomarkers in colorectal cancer (microsatellite instability status, hypermutation, CpG island methylator phenotype status, BRAF mutation, TP53 mutation, and chromosomal instability), outperforming most published state-of-the-art classifiers in a within-cohort internal validation and a cross-cohort external validation. In addition, under the same experimental conditions using the same set of training and testing data sets, DPSeq surpassed four CNNs (ResNet18, ResNet50, MobileNetV2, and EfficientNet) and two transformer (Vision Transformer and Swin Transformer) models, achieving the highest area under the receiver operating characteristic curve and area under the precision-recall curve values in predicting microsatellite instability status, BRAF mutation, and CpG island methylator phenotype status. Furthermore, DPSeq required less time for both training and prediction because of its simple architecture. Therefore, DPSeq appears to be the preferred choice over transformer and CNN models for predicting cancer biomarkers.
Collapse
Affiliation(s)
- Min Cen
- School of Data Science, University of Science and Technology of China, Hefei, China
| | - Xingyu Li
- Department of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei, China
| | - Bangwei Guo
- School of Data Science, University of Science and Technology of China, Hefei, China
| | - Jitendra Jonnagaddala
- School of Population Health, University of New South Wales, Sydney, New South Wales, Australia
| | - Hong Zhang
- Department of Statistics and Finance, School of Management, University of Science and Technology of China, Hefei, China.
| | - Xu Steven Xu
- Clinical Pharmacology and Quantitative Science, Genmab Inc., Princeton, New Jersey.
| |
Collapse
|
22
|
Chatterji S, Niehues JM, van Treeck M, Loeffler CML, Saldanha OL, Veldhuizen GP, Cifci D, Carrero ZI, Abu-Eid R, Speirs V, Kather JN. Prediction models for hormone receptor status in female breast cancer do not extend to males: further evidence of sex-based disparity in breast cancer. NPJ Breast Cancer 2023; 9:91. [PMID: 37940649 PMCID: PMC10632426 DOI: 10.1038/s41523-023-00599-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2023] [Accepted: 10/27/2023] [Indexed: 11/10/2023] Open
Abstract
Breast cancer prognosis and management for both men and women are reliant upon estrogen receptor alpha (ERα) and progesterone receptor (PR) expression to inform therapy. Previous studies have shown that there are sex-specific binding characteristics of ERα and PR in breast cancer and, counterintuitively, ERα expression is more common in male than female breast cancer. We hypothesized that these differences could have morphological manifestations that are undetectable to human observers but could be elucidated computationally. To investigate this, we trained attention-based multiple instance learning prediction models for ERα and PR using H&E-stained images of female breast cancer from the Cancer Genome Atlas (TCGA) (n = 1085) and deployed them on external female (n = 192) and male breast cancer images (n = 245). Both targets were predicted in the internal (AUROC for ERα prediction: 0.86 ± 0.02, p < 0.001; AUROC for PR prediction = 0.76 ± 0.03, p < 0.001) and external female cohorts (AUROC for ERα prediction: 0.78 ± 0.03, p < 0.001; AUROC for PR prediction = 0.80 ± 0.04, p < 0.001) but not the male cohort (AUROC for ERα prediction: 0.66 ± 0.14, p = 0.43; AUROC for PR prediction = 0.63 ± 0.04, p = 0.05). This suggests that subtle morphological differences invisible upon visual inspection may exist between the sexes, supporting previous immunohistochemical, genomic, and transcriptomic analyses.
Collapse
Affiliation(s)
- Subarnarekha Chatterji
- Institute of Medical Sciences, University of Aberdeen, Aberdeen, UK
- Aberdeen Cancer Centre, University of Aberdeen, Aberdeen, UK
| | - Jan Moritz Niehues
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
| | - Marko van Treeck
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
| | - Chiara Maria Lavinia Loeffler
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
- Department of Medicine I, University Hospital and Faculty of Medicine, Technical University of Dresden, Dresden, Germany
| | - Oliver Lester Saldanha
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
| | - Gregory Patrick Veldhuizen
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
| | - Didem Cifci
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
| | - Zunamys Itzell Carrero
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
| | - Rasha Abu-Eid
- Institute of Medical Sciences, University of Aberdeen, Aberdeen, UK
- Aberdeen Cancer Centre, University of Aberdeen, Aberdeen, UK
- Institute of Dentistry, University of Aberdeen, Aberdeen, UK
| | - Valerie Speirs
- Institute of Medical Sciences, University of Aberdeen, Aberdeen, UK.
- Aberdeen Cancer Centre, University of Aberdeen, Aberdeen, UK.
| | - Jakob Nikolas Kather
- Else Kröner Fresenius Centre for Digital Health, Carl Gustav Carus Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH (Rheinisch-Westfälische Technische Hochschule) Aachen, Aachen, Germany
- Department of Medicine I, University Hospital and Faculty of Medicine, Technical University of Dresden, Dresden, Germany
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St. James's, University of Leeds, Leeds, UK
| |
Collapse
|
23
|
Muti HS, Röcken C, Behrens HM, Löffler CML, Reitsam NG, Grosser B, Märkl B, Stange DE, Jiang X, Velduizen GP, Truhn D, Ebert MP, Grabsch HI, Kather JN. Deep learning trained on lymph node status predicts outcome from gastric cancer histopathology: a retrospective multicentric study. Eur J Cancer 2023; 194:113335. [PMID: 37862795 DOI: 10.1016/j.ejca.2023.113335] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Revised: 08/15/2023] [Accepted: 09/03/2023] [Indexed: 10/22/2023]
Abstract
AIM Gastric cancer (GC) is a tumour entity with highly variant outcomes. Lymph node metastasis is a prognostically adverse biomarker. We hypothesised that GC primary tissue contains information that is predictive of lymph node status and patient prognosis and that this information can be extracted using deep learning (DL). METHODS Using three patient cohorts comprising 1146 patients, we trained and validated a DL system to predict lymph node status directly from haematoxylin and eosin-stained GC tissue sections. We investigated the concordance between the DL-based prediction from the primary tumour slides (aiN score) and the histopathological lymph node status (pN). Furthermore, we assessed the prognostic value of the aiN score alone and when combined with the pN status. RESULTS The aiN score predicted the pN status reaching area under the receiver operating characteristic curves of 0.71 in the training cohort and 0.69 and 0.65 in the two test cohorts. In a multivariate Cox analysis, the aiN score was an independent predictor of patient survival with hazard ratios of 1.5 in the training cohort and of 1.3 and 2.2 in the two test cohorts. A combination of the aiN score and the pN status prognostically stratified patients by survival with p-values <0.05 in logrank tests. CONCLUSION GC primary tumour tissue contains additional prognostic information that is accessible using the aiN score. In combination with the pN status, this can be used for personalised management of GC patients after prospective validation.
Collapse
Affiliation(s)
- Hannah S Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Visceral, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus Dresden, Dresden, Germany
| | - Christoph Röcken
- Department of Pathology, University Hospital Schleswig-Holstein, Kiel, Germany
| | | | - Chiara M L Löffler
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine I, University Hospital Dresden, Dresden, Germany
| | - Nic G Reitsam
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
| | - Bianca Grosser
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
| | - Bruno Märkl
- Pathology, Faculty of Medicine, University of Augsburg, Augsburg, Germany
| | - Daniel E Stange
- Department of Visceral, Thoracic and Vascular Surgery, University Hospital Carl Gustav Carus Dresden, Dresden, Germany
| | - Xiaofeng Jiang
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Gregory P Velduizen
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Germany
| | - Matthias P Ebert
- Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany; DKFZ-Hector Cancer Institute at the University Medical Center, Mannheim, Germany; Clinical Cooperation Unit Healthy Metabolism, Center for Preventive Medicine and Digital Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany; Mannheim Institute for Innate Immunoscience (MI3), Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | - Heike I Grabsch
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Department of Pathology, GROW School for Oncology and Reproduction, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Jakob N Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany; Department of Medicine I, University Hospital Dresden, Dresden, Germany; Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
24
|
Wang W, Zhao Y, Teng L, Yan J, Guo Y, Qiu Y, Ji Y, Yu B, Pei D, Duan W, Wang M, Wang L, Duan J, Sun Q, Wang S, Duan H, Sun C, Guo Y, Luo L, Guo Z, Guan F, Wang Z, Xing A, Liu Z, Zhang H, Cui L, Zhang L, Jiang G, Yan D, Liu X, Zheng H, Liang D, Li W, Li ZC, Zhang Z. Neuropathologist-level integrated classification of adult-type diffuse gliomas using deep learning from whole-slide pathological images. Nat Commun 2023; 14:6359. [PMID: 37821431 PMCID: PMC10567721 DOI: 10.1038/s41467-023-41195-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2022] [Accepted: 08/16/2023] [Indexed: 10/13/2023] Open
Abstract
Current diagnosis of glioma types requires combining both histological features and molecular characteristics, which is an expensive and time-consuming procedure. Determining the tumor types directly from whole-slide images (WSIs) is of great value for glioma diagnosis. This study presents an integrated diagnosis model for automatic classification of diffuse gliomas from annotation-free standard WSIs. Our model is developed on a training cohort (n = 1362) and a validation cohort (n = 340), and tested on an internal testing cohort (n = 289) and two external cohorts (n = 305 and 328, respectively). The model can learn imaging features containing both pathological morphology and underlying biological clues to achieve the integrated diagnosis. Our model achieves high performance with area under receiver operator curve all above 0.90 in classifying major tumor types, in identifying tumor grades within type, and especially in distinguishing tumor genotypes with shared histological features. This integrated diagnosis model has the potential to be used in clinical scenarios for automated and unbiased classification of adult-type diffuse gliomas.
Collapse
Affiliation(s)
- Weiwei Wang
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Yuanshen Zhao
- Institute of Biomedical and Health Engineering, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Lianghong Teng
- Department of Pathology, Xuanwu Hospital, Capital Medical University, Beijing, China
| | - Jing Yan
- Department of MRI, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Yang Guo
- Department of Neurosurgery, Henan Provincial People's Hospital, Zhengzhou, Henan, China
| | - Yuning Qiu
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Yuchen Ji
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Bin Yu
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Dongling Pei
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Wenchao Duan
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Minkai Wang
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Li Wang
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Jingxian Duan
- Institute of Biomedical and Health Engineering, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Qiuchang Sun
- Institute of Biomedical and Health Engineering, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
| | - Shengnan Wang
- Department of Pathology, Xuanwu Hospital, Capital Medical University, Beijing, China
| | - Huanli Duan
- Department of Pathology, Xuanwu Hospital, Capital Medical University, Beijing, China
| | - Chen Sun
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Yu Guo
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Lin Luo
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Zhixuan Guo
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Fangzhan Guan
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Zilong Wang
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Aoqi Xing
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Zhongyi Liu
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Hongyan Zhang
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Li Cui
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Lan Zhang
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Guozhong Jiang
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Dongming Yan
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Xianzhi Liu
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Hairong Zheng
- Institute of Biomedical and Health Engineering, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
- The Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, Shenzhen, China
- National Innovation Center for Advanced Medical Devices, Shenzhen, China
| | - Dong Liang
- Institute of Biomedical and Health Engineering, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- University of Chinese Academy of Sciences, Beijing, China
- The Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, Shenzhen, China
- National Innovation Center for Advanced Medical Devices, Shenzhen, China
| | - Wencai Li
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China.
| | - Zhi-Cheng Li
- Institute of Biomedical and Health Engineering, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
- University of Chinese Academy of Sciences, Beijing, China.
- The Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, Shenzhen, China.
- National Innovation Center for Advanced Medical Devices, Shenzhen, China.
| | - Zhenyu Zhang
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China.
| |
Collapse
|
25
|
Wagner SJ, Reisenbüchler D, West NP, Niehues JM, Zhu J, Foersch S, Veldhuizen GP, Quirke P, Grabsch HI, van den Brandt PA, Hutchins GGA, Richman SD, Yuan T, Langer R, Jenniskens JCA, Offermans K, Mueller W, Gray R, Gruber SB, Greenson JK, Rennert G, Bonner JD, Schmolze D, Jonnagaddala J, Hawkins NJ, Ward RL, Morton D, Seymour M, Magill L, Nowak M, Hay J, Koelzer VH, Church DN, Matek C, Geppert C, Peng C, Zhi C, Ouyang X, James JA, Loughrey MB, Salto-Tellez M, Brenner H, Hoffmeister M, Truhn D, Schnabel JA, Boxberg M, Peng T, Kather JN. Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study. Cancer Cell 2023; 41:1650-1661.e4. [PMID: 37652006 PMCID: PMC10507381 DOI: 10.1016/j.ccell.2023.08.002] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 01/04/2023] [Revised: 06/18/2023] [Accepted: 08/07/2023] [Indexed: 09/02/2023]
Abstract
Deep learning (DL) can accelerate the prediction of prognostic biomarkers from routine pathology slides in colorectal cancer (CRC). However, current approaches rely on convolutional neural networks (CNNs) and have mostly been validated on small patient cohorts. Here, we develop a new transformer-based pipeline for end-to-end biomarker prediction from pathology slides by combining a pre-trained transformer encoder with a transformer network for patch aggregation. Our transformer-based approach substantially improves the performance, generalizability, data efficiency, and interpretability as compared with current state-of-the-art algorithms. After training and evaluating on a large multicenter cohort of over 13,000 patients from 16 colorectal cancer cohorts, we achieve a sensitivity of 0.99 with a negative predictive value of over 0.99 for prediction of microsatellite instability (MSI) on surgical resection specimens. We demonstrate that resection specimen-only training reaches clinical-grade performance on endoscopic biopsy tissue, solving a long-standing diagnostic problem.
Collapse
Affiliation(s)
- Sophia J Wagner
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany; Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany
| | - Daniel Reisenbüchler
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany
| | - Nicholas P West
- Institute of Pathology, University Medical Center Mainz, Mainz, Germany
| | - Jan Moritz Niehues
- Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany
| | - Jiefu Zhu
- Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany
| | - Sebastian Foersch
- Institute of Pathology, University Medical Center Mainz, Mainz, Germany
| | | | - Philip Quirke
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Heike I Grabsch
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Department of Pathology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Piet A van den Brandt
- Department of Epidemiology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Gordon G A Hutchins
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Susan D Richman
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Tanwei Yuan
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Rupert Langer
- Institute of Pathology und Molecular Pathology, Johannes Kepler University Hospital Linz, Linz, Österreich
| | - Josien C A Jenniskens
- Department of Epidemiology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Kelly Offermans
- Department of Epidemiology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | | | - Richard Gray
- Nuffield Department of Population Health, University of Oxford, Oxford, UK
| | - Stephen B Gruber
- Center for Precision Medicine and Department of Medical Oncology, City of Hope National Medical Center, Duarte, CA, USA
| | - Joel K Greenson
- Department of Pathology, City of Hope Comprehensive Cancer Center, Duarte, CA, USA
| | - Gad Rennert
- Department of Community Medicine & Epidemiology, Lady Davis Carmel Medical Center, Ruth & Bruce Rappaport Faculty of Medicine, Technion-Israel Institute of Technology, Haifa, Israel; Steve and Cindy Rasmussen Institute for Genomic Medicine, Lady Davis Carmel Medical Center and Technion Faculty of Medicine, Clalit National Cancer Control Center, Haifa, Israel
| | - Joseph D Bonner
- Department of Community Medicine & Epidemiology, Lady Davis Carmel Medical Center, Ruth & Bruce Rappaport Faculty of Medicine, Technion-Israel Institute of Technology, Haifa, Israel
| | - Daniel Schmolze
- Center for Precision Medicine and Department of Medical Oncology, City of Hope National Medical Center, Duarte, CA, USA
| | - Jitendra Jonnagaddala
- School of Population Health, Faculty of Medicine and Health, UNSW Sydney, Sydney, NSW, Australia
| | - Nicholas J Hawkins
- School of Medical Sciences, Faculty of Medicine and Health, UNSW Sydney, Sydney, NSW, Australia
| | - Robyn L Ward
- School of Medical Sciences, Faculty of Medicine and Health, UNSW Sydney, Sydney, NSW, Australia; Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, Australia
| | - Dion Morton
- University Hospital Birmingham, Birmingham, UK
| | | | - Laura Magill
- University of Birmingham Clinical Trials Unit, Birmingham, UK
| | - Marta Nowak
- Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Jennifer Hay
- Glasgow Tissue Research Facility, University of Glasgow, Queen Elizabeth University Hospital, Glasgow, UK
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, University Hospital Zurich, University of Zurich, Zurich, Switzerland; Department of Oncology, University of Oxford, Oxford, UK; Nuffield Department of Medicine, University of Oxford, Roosevelt Drive, Oxford, UK
| | - David N Church
- Nuffield Department of Medicine, University of Oxford, Roosevelt Drive, Oxford, UK; Oxford NIHR Comprehensive Biomedical Research Centre, Oxford University Hospitals NHS Foundation Trust, Oxford, UK
| | - Christian Matek
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; Institute of Pathology, University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany; Comprehensive Cancer Center Erlangen-EMN (CCC), University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany
| | - Carol Geppert
- Institute of Pathology, University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany; Comprehensive Cancer Center Erlangen-EMN (CCC), University Hospital Erlangen, FAU Erlangen-Nuremberg, Erlangen, Germany
| | - Chaolong Peng
- Medical School, Jianggang Shan University, Jiangxi, China
| | - Cheng Zhi
- Department of Pathology, the Second Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
| | - Xiaoming Ouyang
- Department of Pathology, the Second Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
| | - Jacqueline A James
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK; Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, UK; The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK
| | - Maurice B Loughrey
- The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK; Department of Cellular Pathology, Belfast Health and Social Care Trust, Belfast, UK; Centre for Public Health, Queen's University Belfast, Belfast, UK
| | - Manuel Salto-Tellez
- Precision Medicine Centre of Excellence, Health Sciences Building, The Patrick G Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK; Regional Molecular Diagnostic Service, Belfast Health and Social Care Trust, Belfast, UK; Integrated Pathology Unit, Institute for Cancer Research and Royal Marsden Hospital, London, UK
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany; Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Julia A Schnabel
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany; School of Computation, Information and Technology, Technical University of Munich, Munich, Germany; School of Biomedical Engineering and Imaging Sciences, King's College London, London, UK
| | - Melanie Boxberg
- Institute of Pathology, Technical University Munich, Munich, Germany; Institute of Pathology Munich-North, Munich, Germany
| | - Tingying Peng
- Helmholtz Munich - German Research Center for Environment and Health, Munich, Germany.
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health (EFFZ), Technical University Dresden, Dresden, Germany; Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg.
| |
Collapse
|
26
|
Shao Z, Dai L, Jonnagaddala J, Chen Y, Wang Y, Fang Z, Zhang Y. Generalizability of Self-Supervised Training Models for Digital Pathology: A Multicountry Comparison in Colorectal Cancer. JCO Clin Cancer Inform 2023; 7:e2200178. [PMID: 37703507 DOI: 10.1200/cci.22.00178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2022] [Revised: 06/15/2023] [Accepted: 06/27/2023] [Indexed: 09/15/2023] Open
Abstract
PURPOSE In this multicountry study, we aim to explore the effectiveness of self-supervised learning (SSL) in colorectal cancer (CRC)-related predictive tasks using large amount of unlabeled digital pathology imaging data. METHODS We adopted SimSiam to conduct self-supervised pretraining on two large whole-slide image CRC data sets from the United States and Australia. The SSL pretrained encoder is then used in several predictive tasks, including supervised predictive tasks (tissue classification, microsatellite instability v microsatellite stability classification), and weakly supervised predictive tasks (polyp type classification and adenoma grading, and 5-year survival prediction). Performance on the tasks was compared between models using SSL pretraining and those using ImageNet pretraining, and performance for one-country pretraining was compared with two-country pretraining. RESULTS We demonstrate that SSL pretraining outperforms ImageNet pretraining in predictive tasks, that is, SSL pretraining outperforms the ImageNet pretraining by 3.01% of F 1 score on average over supervised predictive tasks and 1.53% of AUC on average over weakly supervised predictive tasks. Furthermore, two-country SSL pretraining has shown more stable performance than single-country pretraining, that is, two-country pretraining outperforms at least one of the single-country pretrainings by 1.93% of F 1 on average over supervised predictive tasks and 1.36% of AUC on average over weakly-supervised predictive tasks. CONCLUSION We find that using unlabeled image data for SSL pretraining in CRC related tasks is more effective than using ImageNet pretraining. Furthermore, SSL pretraining using data from multiple countries achieve more stable performance and better generalization than single-country pretraining.
Collapse
Affiliation(s)
- Zhuchen Shao
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Liuxi Dai
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
| | | | - Yang Chen
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Yifeng Wang
- School of Science, Harbin Institute of Technology, Shenzhen, China
| | - Zijie Fang
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
| | - Yongbing Zhang
- School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen, China
| |
Collapse
|
27
|
Pizurica M, Larmuseau M, Van der Eecken K, de Schaetzen van Brienen L, Carrillo-Perez F, Isphording S, Lumen N, Van Dorpe J, Ost P, Verbeke S, Gevaert O, Marchal K. Whole Slide Imaging-Based Prediction of TP53 Mutations Identifies an Aggressive Disease Phenotype in Prostate Cancer. Cancer Res 2023; 83:2970-2984. [PMID: 37352385 PMCID: PMC10538366 DOI: 10.1158/0008-5472.can-22-3113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2022] [Revised: 03/08/2023] [Accepted: 06/20/2023] [Indexed: 06/25/2023]
Abstract
In prostate cancer, there is an urgent need for objective prognostic biomarkers that identify the metastatic potential of a tumor at an early stage. While recent analyses indicated TP53 mutations as candidate biomarkers, molecular profiling in a clinical setting is complicated by tumor heterogeneity. Deep learning models that predict the spatial presence of TP53 mutations in whole slide images (WSI) offer the potential to mitigate this issue. To assess the potential of WSIs as proxies for spatially resolved profiling and as biomarkers for aggressive disease, we developed TiDo, a deep learning model that achieves state-of-the-art performance in predicting TP53 mutations from WSIs of primary prostate tumors. In an independent multifocal cohort, the model showed successful generalization at both the patient and lesion level. Analysis of model predictions revealed that false positive (FP) predictions could at least partially be explained by TP53 deletions, suggesting that some FP carry an alteration that leads to the same histological phenotype as TP53 mutations. Comparative expression and histologic cell type analyses identified a TP53-like cellular phenotype triggered by expression of pathways affecting stromal composition. Together, these findings indicate that WSI-based models might not be able to perfectly predict the spatial presence of individual TP53 mutations but they have the potential to elucidate the prognosis of a tumor by depicting a downstream phenotype associated with aggressive disease biomarkers. SIGNIFICANCE Deep learning models predicting TP53 mutations from whole slide images of prostate cancer capture histologic phenotypes associated with stromal composition, lymph node metastasis, and biochemical recurrence, indicating their potential as in silico prognostic biomarkers. See related commentary by Bordeleau, p. 2809.
Collapse
Affiliation(s)
- Marija Pizurica
- Internet Technology and Data Science Lab (IDLab/IMEC), Ghent University, Gent, Belgium
- Department of Plant biotechnology and Bioinformatics, Ghent University, Gent, Belgium
- Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California
| | - Maarten Larmuseau
- Internet Technology and Data Science Lab (IDLab/IMEC), Ghent University, Gent, Belgium
- Department of Plant biotechnology and Bioinformatics, Ghent University, Gent, Belgium
| | | | - Louise de Schaetzen van Brienen
- Internet Technology and Data Science Lab (IDLab/IMEC), Ghent University, Gent, Belgium
- Department of Plant biotechnology and Bioinformatics, Ghent University, Gent, Belgium
| | - Francisco Carrillo-Perez
- Department of Architecture and Computer Technology (ATC), University of Granada, Granada, Spain
- Stanford Center for Biomedical Informatics Research (BMIR), Stanford University, School of Medicine, Stanford, California
| | - Simon Isphording
- Internet Technology and Data Science Lab (IDLab/IMEC), Ghent University, Gent, Belgium
- Department of Plant biotechnology and Bioinformatics, Ghent University, Gent, Belgium
| | - Nicolaas Lumen
- Department of Urology, Ghent University Hospital, Ghent, Belgium
| | - Jo Van Dorpe
- Department of Urology, Ghent University Hospital, Ghent, Belgium
| | - Piet Ost
- Department of Radiotherapy, Ghent University Hospital, Ghent, Belgium
| | - Sofie Verbeke
- Department of Urology, Ghent University Hospital, Ghent, Belgium
| | - Olivier Gevaert
- Department of Biomedical Data Science, Stanford University, School of Medicine, Stanford, California
- Stanford Center for Biomedical Informatics Research (BMIR), Stanford University, School of Medicine, Stanford, California
| | - Kathleen Marchal
- Internet Technology and Data Science Lab (IDLab/IMEC), Ghent University, Gent, Belgium
- Department of Plant biotechnology and Bioinformatics, Ghent University, Gent, Belgium
| |
Collapse
|
28
|
Kaczmarzyk JR, Gupta R, Kurc TM, Abousamra S, Saltz JH, Koo PK. ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification. Comput Methods Programs Biomed 2023; 239:107631. [PMID: 37271050 PMCID: PMC11093625 DOI: 10.1016/j.cmpb.2023.107631] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Revised: 04/23/2023] [Accepted: 05/28/2023] [Indexed: 06/06/2023]
Abstract
BACKGROUND AND OBJECTIVE Histopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers. METHODS Here we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model. RESULTS The main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision. CONCLUSIONS Choosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.
Collapse
Affiliation(s)
- Jakub R Kaczmarzyk
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA; Simons Center for Quantitative Biology, 1 Bungtown Rd, Cold Spring Harbor, 11724, NY, USA.
| | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA
| | - Tahsin M Kurc
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA
| | - Shahira Abousamra
- Department of Computer Science, Stony Brook University, Stony Brook, NY, USA
| | - Joel H Saltz
- Department of Biomedical Informatics, Stony Brook Medicine, 101 Nicolls Rd, Stony Brook, 11794, NY, USA.
| | - Peter K Koo
- Simons Center for Quantitative Biology, 1 Bungtown Rd, Cold Spring Harbor, 11724, NY, USA.
| |
Collapse
|
29
|
Veldhuizen GP, Röcken C, Behrens HM, Cifci D, Muti HS, Yoshikawa T, Arai T, Oshima T, Tan P, Ebert MP, Pearson AT, Calderaro J, Grabsch HI, Kather JN. Deep learning-based subtyping of gastric cancer histology predicts clinical outcome: a multi-institutional retrospective study. Gastric Cancer 2023; 26:708-720. [PMID: 37269416 PMCID: PMC10361890 DOI: 10.1007/s10120-023-01398-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Accepted: 05/09/2023] [Indexed: 06/05/2023]
Abstract
INTRODUCTION The Laurén classification is widely used for Gastric Cancer (GC) histology subtyping. However, this classification is prone to interobserver variability and its prognostic value remains controversial. Deep Learning (DL)-based assessment of hematoxylin and eosin (H&E) stained slides is a potentially useful tool to provide an additional layer of clinically relevant information, but has not been systematically assessed in GC. OBJECTIVE We aimed to train, test and externally validate a deep learning-based classifier for GC histology subtyping using routine H&E stained tissue sections from gastric adenocarcinomas and to assess its potential prognostic utility. METHODS We trained a binary classifier on intestinal and diffuse type GC whole slide images for a subset of the TCGA cohort (N = 166) using attention-based multiple instance learning. The ground truth of 166 GC was obtained by two expert pathologists. We deployed the model on two external GC patient cohorts, one from Europe (N = 322) and one from Japan (N = 243). We assessed classification performance using the Area Under the Receiver Operating Characteristic Curve (AUROC) and prognostic value (overall, cancer specific and disease free survival) of the DL-based classifier with uni- and multivariate Cox proportional hazard models and Kaplan-Meier curves with log-rank test statistics. RESULTS Internal validation using the TCGA GC cohort using five-fold cross-validation achieved a mean AUROC of 0.93 ± 0.07. External validation showed that the DL-based classifier can better stratify GC patients' 5-year survival compared to pathologist-based Laurén classification for all survival endpoints, despite frequently divergent model-pathologist classifications. Univariate overall survival Hazard Ratios (HRs) of pathologist-based Laurén classification (diffuse type versus intestinal type) were 1.14 (95% Confidence Interval (CI) 0.66-1.44, p-value = 0.51) and 1.23 (95% CI 0.96-1.43, p-value = 0.09) in the Japanese and European cohorts, respectively. DL-based histology classification resulted in HR of 1.46 (95% CI 1.18-1.65, p-value < 0.005) and 1.41 (95% CI 1.20-1.57, p-value < 0.005), in the Japanese and European cohorts, respectively. In diffuse type GC (as defined by the pathologist), classifying patients using the DL diffuse and intestinal classifications provided a superior survival stratification, and demonstrated statistically significant survival stratification when combined with pathologist classification for both the Asian (overall survival log-rank test p-value < 0.005, HR 1.43 (95% CI 1.05-1.66, p-value = 0.03) and European cohorts (overall survival log-rank test p-value < 0.005, HR 1.56 (95% CI 1.16-1.76, p-value < 0.005)). CONCLUSION Our study shows that gastric adenocarcinoma subtyping using pathologist's Laurén classification as ground truth can be performed using current state of the art DL techniques. Patient survival stratification seems to be better by DL-based histology typing compared with expert pathologist histology typing. DL-based GC histology typing has potential as an aid in subtyping. Further investigations are warranted to fully understand the underlying biological mechanisms for the improved survival stratification despite apparent imperfect classification by the DL algorithm.
Collapse
Affiliation(s)
| | - Christoph Röcken
- Department of Pathology, Christian-Albrechts University, Kiel, Germany
| | | | - Didem Cifci
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Hannah Sophie Muti
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Department of Visceral, Thoracic and Vascular Surgery, Technical University Dresden, University Hospital Carl Gustav Carus, Dresden, Germany
| | - Takaki Yoshikawa
- Department of Gastric Surgery, National Cancer Center Hospital, Tokyo, Japan
| | - Tomio Arai
- Department of Pathology, Tokyo Metropolitan Geriatric Hospital and Institute of Gerontology, Tokyo, Japan
| | - Takashi Oshima
- Department of Gastrointestinal Surgery, Kanagawa Cancer Center, Yokohama, Japan
| | - Patrick Tan
- Duke-NUS Medical School, Singapore, Singapore
| | - Matthias P Ebert
- Department of Medicine II, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
- DKFZ-Hector Cancer Institute at the University Medical Center, Mannheim, Germany
- Clinical Cooperation Unit Healthy Metabolism, Center for Preventive Medicine and Digital Health, Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
- Mannheim Institute for Innate Immunoscience (MI3), Medical Faculty Mannheim, Heidelberg University, Mannheim, Germany
| | - Alexander T Pearson
- Department of Medicine, Section of Hematology/Oncology, The University of Chicago, Chicago, IL, USA
| | - Julien Calderaro
- Université Paris Est Créteil, INSERM, IMRB, Créteil, France
- Department of Pathology, Assistance Publique-Hôpitaux de Paris, Henri Mondor-Albert Chenevier University Hospital, Créteil, France
| | - Heike I Grabsch
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Department of Pathology, GROW School for Oncology and Reproduction, Maastricht University Medical Center+, Maastricht, The Netherlands
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK.
- Department of Medicine I, University Hospital Dresden, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| |
Collapse
|
30
|
Bussola N, Xu J, Wu L, Gorini L, Zhang Y, Furlanello C, Tong W. A Weakly Supervised Deep Learning Framework for Whole Slide Classification to Facilitate Digital Pathology in Animal Study. Chem Res Toxicol 2023; 36:1321-1331. [PMID: 37540590 PMCID: PMC10445282 DOI: 10.1021/acs.chemrestox.3c00058] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Indexed: 08/06/2023]
Abstract
The pathology of animal studies is crucial for toxicity evaluations and regulatory assessments, but the manual examination of slides by pathologists remains time-consuming and requires extensive training. One inherent challenge in this process is the interobserver variability, which can compromise the consistency and accuracy of a study. Artificial intelligence (AI) has demonstrated its ability to automate similar examinations in clinical applications with enhanced efficiency, consistency, and accuracy. However, training AI models typically relies on costly pixel-level annotation of injured regions and is often not available for animal pathology. To address this, we developed the PathologAI system, a "weakly" supervised approach for WSI classification in rat images without explicit lesion annotation at the pixel level. Using rat liver imaging data from the Open TG-GATEs system, PathologAI was applied to predict necrosis of n = 816 WSIs (377 controls). TG-GATEs studied 170 compounds at three dose levels (low, middle, and high) for four time points (3, 7, 14, and 28 days). PathologAI first preprocessed WSIs at the tile level to generate a high-level representation with a Generative Adversarial Network architecture. The prediction of liver necrosis relied on an ensemble model of 5 CNN classifiers trained on 335 WSIs. The ensemble model achieved notable classification accuracy on the holdout test set: 87% among 87 control slides free of findings, 83% among 120 controls with spontaneous necrosis, 67% among 147 treated animals with spontaneous minimal or slight necrosis, and 59% among 127 treated animals with minimal or slight necrosis caused by the treatment. Importantly, PathologAI was able to discriminate WSIs with spontaneous necrosis from those with treatment related necrosis and discriminated mild lesion level findings (slight vs minimal) and between treatment dose levels. PathologAI could provide an inexpensive and rapid screening tool to assist the digital pathology analysis in preclinical applications and general toxicological studies.
Collapse
Affiliation(s)
- Nicole Bussola
- Center
for Integrative Biology, University of Trento, Trento 38123, Italy
| | - Joshua Xu
- Division
of Bioinformatics and Biostatistics National
Center for Toxicological Research, Food
and Drug Administration, Jefferson, Arkansas 72079, United States
| | - Leihong Wu
- Division
of Bioinformatics and Biostatistics National
Center for Toxicological Research, Food
and Drug Administration, Jefferson, Arkansas 72079, United States
| | | | - Yifan Zhang
- Division
of Bioinformatics and Biostatistics National
Center for Toxicological Research, Food
and Drug Administration, Jefferson, Arkansas 72079, United States
| | | | - Weida Tong
- Division
of Bioinformatics and Biostatistics National
Center for Toxicological Research, Food
and Drug Administration, Jefferson, Arkansas 72079, United States
| |
Collapse
|
31
|
Nasrallah MP, Zhao J, Tsai CC, Meredith D, Marostica E, Ligon KL, Golden JA, Yu KH. Machine learning for cryosection pathology predicts the 2021 WHO classification of glioma. Med 2023; 4:526-540.e4. [PMID: 37421953 PMCID: PMC10527821 DOI: 10.1016/j.medj.2023.06.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2022] [Revised: 04/17/2023] [Accepted: 06/06/2023] [Indexed: 07/10/2023]
Abstract
BACKGROUND Timely and accurate intraoperative cryosection evaluations remain the gold standard for guiding surgical treatments for gliomas. However, the tissue-freezing process often generates artifacts that make histologic interpretation difficult. In addition, the 2021 WHO Classification of Tumors of the Central Nervous System incorporates molecular profiles in the diagnostic categories, so standard visual evaluation of cryosections alone cannot completely inform diagnoses based on the new classification system. METHODS To address these challenges, we develop the context-aware Cryosection Histopathology Assessment and Review Machine (CHARM) using samples from 1,524 glioma patients from three different patient populations to systematically analyze cryosection slides. FINDINGS Our CHARM models successfully identified malignant cells (AUROC = 0.98 ± 0.01 in the independent validation cohort), distinguished isocitrate dehydrogenase (IDH)-mutant tumors from wild type (AUROC = 0.79-0.82), classified three major types of molecularly defined gliomas (AUROC = 0.88-0.93), and identified the most prevalent subtypes of IDH-mutant tumors (AUROC = 0.89-0.97). CHARM further predicts clinically important genetic alterations in low-grade glioma, including ATRX, TP53, and CIC mutations, CDKN2A/B homozygous deletion, and 1p/19q codeletion via cryosection images. CONCLUSIONS Our approaches accommodate the evolving diagnostic criteria informed by molecular studies, provide real-time clinical decision support, and will democratize accurate cryosection diagnoses. FUNDING Supported in part by the National Institute of General Medical Sciences grant R35GM142879, the Google Research Scholar Award, the Blavatnik Center for Computational Biomedicine Award, the Partners' Innovation Discovery Grant, and the Schlager Family Award for Early Stage Digital Health Innovations.
Collapse
Affiliation(s)
- MacLean P Nasrallah
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA 19104, USA
| | - Junhan Zhao
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA
| | - Cheng Che Tsai
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA
| | - David Meredith
- Department of Pathology, Brigham and Women's Hospital, Boston, MA 02115, USA
| | - Eliana Marostica
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA; Division of Health Sciences and Technology, Harvard-Massachusetts Institute of Technology, Boston, MA 02139, USA
| | - Keith L Ligon
- Department of Pathology, Dana-Farber Cancer Institute, Boston, MA 02215, USA
| | - Jeffrey A Golden
- Department of Pathology, Brigham and Women's Hospital, Boston, MA 02115, USA; Department of Pathology, Cedars-Sinai Medical Center, Los Angeles, CA 90048, USA
| | - Kun-Hsing Yu
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA; Department of Pathology, Brigham and Women's Hospital, Boston, MA 02115, USA.
| |
Collapse
|
32
|
Mukashyaka P, Sheridan TB, Foroughi Pour A, Chuang JH. SAMPLER: Empirical distribution representations for rapid analysis of whole slide tissue images. bioRxiv 2023:2023.08.01.551468. [PMID: 37577691 PMCID: PMC10418159 DOI: 10.1101/2023.08.01.551468] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/15/2023]
Abstract
Deep learning has revolutionized digital pathology, allowing for automatic analysis of hematoxylin and eosin (H&E) stained whole slide images (WSIs) for diverse tasks. In such analyses, WSIs are typically broken into smaller images called tiles, and a neural network backbone encodes each tile in a feature space. Many recent works have applied attention based deep learning models to aggregate tile-level features into a slide-level representation, which is then used for slide-level prediction tasks. However, training attention models is computationally intensive, necessitating hyperparameter optimization and specialized training procedures. Here, we propose SAMPLER, a fully statistical approach to generate efficient and informative WSI representations by encoding the empirical cumulative distribution functions (CDFs) of multiscale tile features. We demonstrate that SAMPLER-based classifiers are as accurate or better than state-of-the-art fully deep learning attention models for classification tasks including distinction of: subtypes of breast carcinoma (BRCA: AUC=0.911 ± 0.029); subtypes of non-small cell lung carcinoma (NSCLC: AUC=0.940±0.018); and subtypes of renal cell carcinoma (RCC: AUC=0.987±0.006). A major advantage of the SAMPLER representation is that predictive models are >100X faster compared to attention models. Histopathological review confirms that SAMPLER-identified high attention tiles contain tumor morphological features specific to the tumor type, while low attention tiles contain fibrous stroma, blood, or tissue folding artifacts. We further apply SAMPLER concepts to improve the design of attention-based neural networks, yielding a context aware multi-head attention model with increased accuracy for subtype classification within BRCA and RCC (BRCA: AUC=0.921±0.027, and RCC: AUC=0.988±0.010). Finally, we provide theoretical results identifying sufficient conditions for which SAMPLER is optimal. SAMPLER is a fast and effective approach for analyzing WSIs, with greatly improved scalability over attention methods to benefit digital pathology analysis.
Collapse
Affiliation(s)
- Patience Mukashyaka
- The Jackson Laboratory for Genomic Medicine, Farmington, CT
- University of Connecticut Health Center, Department of Genetics and Genome Sciences, Farmington, CT
| | - Todd B Sheridan
- The Jackson Laboratory for Genomic Medicine, Farmington, CT
- Department of Pathology, Hartford hospital, Hartford, CT
| | | | - Jeffrey H Chuang
- The Jackson Laboratory for Genomic Medicine, Farmington, CT
- University of Connecticut Health Center, Department of Genetics and Genome Sciences, Farmington, CT
| |
Collapse
|
33
|
Verghese G, Lennerz JK, Ruta D, Ng W, Thavaraj S, Siziopikou KP, Naidoo T, Rane S, Salgado R, Pinder SE, Grigoriadis A. Computational pathology in cancer diagnosis, prognosis, and prediction - present day and prospects. J Pathol 2023; 260:551-563. [PMID: 37580849 PMCID: PMC10785705 DOI: 10.1002/path.6163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2023] [Revised: 06/12/2023] [Accepted: 06/17/2023] [Indexed: 08/16/2023]
Abstract
Computational pathology refers to applying deep learning techniques and algorithms to analyse and interpret histopathology images. Advances in artificial intelligence (AI) have led to an explosion in innovation in computational pathology, ranging from the prospect of automation of routine diagnostic tasks to the discovery of new prognostic and predictive biomarkers from tissue morphology. Despite the promising potential of computational pathology, its integration in clinical settings has been limited by a range of obstacles including operational, technical, regulatory, ethical, financial, and cultural challenges. Here, we focus on the pathologists' perspective of computational pathology: we map its current translational research landscape, evaluate its clinical utility, and address the more common challenges slowing clinical adoption and implementation. We conclude by describing contemporary approaches to drive forward these techniques. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Collapse
Affiliation(s)
- Gregory Verghese
- School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
- The Breast Cancer Now Research Unit, School of Cancer and Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
| | - Jochen K Lennerz
- Center for Integrated Diagnostics, Department of PathologyMassachusetts General Hospital/Harvard Medical SchoolBostonMAUSA
| | - Danny Ruta
- Guy's CancerGuy's and St Thomas’ NHS Foundation TrustLondonUK
| | - Wen Ng
- Department of Cellular PathologyGuy's and St Thomas NHS Foundation TrustLondonUK
| | - Selvam Thavaraj
- Head & Neck PathologyGuy's and St Thomas NHS Foundation TrustLondonUK
- Centre for Clinical, Oral & Translational Science, Faculty of Dentistry, Oral & Craniofacial SciencesKing's College LondonLondonUK
| | - Kalliopi P Siziopikou
- Department of Pathology, Section of Breast PathologyNorthwestern University Feinberg School of MedicineChicagoILUSA
| | - Threnesan Naidoo
- Department of Laboratory Medicine and Pathology, Walter Sisulu University, Mthatha, Eastern CapeSouth Africa and Africa Health Research InstituteDurbanSouth Africa
| | - Swapnil Rane
- Department of PathologyTata Memorial Centre – ACTRECHBNINavi MumbaiIndia
- Computational Pathology, AI & Imaging LaboratoryTata Memorial Centre – ACTREC, HBNINavi MumbaiIndia
| | - Roberto Salgado
- Department of PathologyGZA–ZNA ZiekenhuizenAntwerpBelgium
- Division of ResearchPeter MacCallum Cancer CentreMelbourneVictoriaAustralia
| | - Sarah E Pinder
- School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
- Department of Cellular PathologyGuy's and St Thomas NHS Foundation TrustLondonUK
| | - Anita Grigoriadis
- School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
- The Breast Cancer Now Research Unit, School of Cancer and Pharmaceutical Sciences, Faculty of Life Sciences and MedicineKing's College LondonLondonUK
| |
Collapse
|
34
|
Asif A, Rajpoot K, Graham S, Snead D, Minhas F, Rajpoot N. Unleashing the potential of AI for pathology: challenges and recommendations. J Pathol 2023; 260:564-577. [PMID: 37550878 PMCID: PMC10952719 DOI: 10.1002/path.6168] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Revised: 06/21/2023] [Accepted: 06/22/2023] [Indexed: 08/09/2023]
Abstract
Computational pathology is currently witnessing a surge in the development of AI techniques, offering promise for achieving breakthroughs and significantly impacting the practices of pathology and oncology. These AI methods bring with them the potential to revolutionize diagnostic pipelines as well as treatment planning and overall patient care. Numerous peer-reviewed studies reporting remarkable performance across diverse tasks serve as a testimony to the potential of AI in the field. However, widespread adoption of these methods in clinical and pre-clinical settings still remains a challenge. In this review article, we present a detailed analysis of the major obstacles encountered during the development of effective models and their deployment in practice. We aim to provide readers with an overview of the latest developments, assist them with insights into identifying some specific challenges that may require resolution, and suggest recommendations and potential future research directions. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.
Collapse
Affiliation(s)
- Amina Asif
- Tissue Image Analytics Centre, Department of Computer ScienceUniversity of WarwickCoventryUK
| | - Kashif Rajpoot
- School of Computer ScienceUniversity of BirminghamBirminghamUK
| | - Simon Graham
- Histofy Ltd, Birmingham Business ParkBirminghamUK
| | - David Snead
- Histofy Ltd, Birmingham Business ParkBirminghamUK
- Department of PathologyUniversity Hospitals Coventry & Warwickshire NHS TrustCoventryUK
| | - Fayyaz Minhas
- Tissue Image Analytics Centre, Department of Computer ScienceUniversity of WarwickCoventryUK
- Cancer Research CentreUniversity of WarwickCoventryUK
| | - Nasir Rajpoot
- Tissue Image Analytics Centre, Department of Computer ScienceUniversity of WarwickCoventryUK
- Histofy Ltd, Birmingham Business ParkBirminghamUK
- Cancer Research CentreUniversity of WarwickCoventryUK
- The Alan Turing InstituteLondonUK
| |
Collapse
|
35
|
Wu Y, Li Y, Xiong X, Liu X, Lin B, Xu B. Recent advances of pathomics in colorectal cancer diagnosis and prognosis. Front Oncol 2023; 13:1094869. [PMID: 37538112 PMCID: PMC10396402 DOI: 10.3389/fonc.2023.1094869] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2022] [Accepted: 06/13/2023] [Indexed: 08/05/2023] Open
Abstract
Colorectal cancer (CRC) is one of the most common malignancies, with the third highest incidence and the second highest mortality in the world. To improve the therapeutic outcome, the risk stratification and prognosis predictions would help guide clinical treatment decisions. Achieving these goals have been facilitated by the fast development of artificial intelligence (AI) -based algorithms using radiological and pathological data, in combination with genomic information. Among them, features extracted from pathological images, termed pathomics, are able to reflect sub-visual characteristics linking to better stratification and prediction of therapeutic responses. In this paper, we review recent advances in pathological image-based algorithms in CRC, focusing on diagnosis of benign and malignant lesions, micro-satellite instability, as well as prediction of neoadjuvant chemoradiotherapy and the prognosis of CRC patients.
Collapse
Affiliation(s)
- Yihan Wu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Yi Li
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College, Chongqing University, Chongqing, China
| | - Xiaomin Xiong
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
- Bioengineering College, Chongqing University, Chongqing, China
| | - Xiaohua Liu
- Bioengineering College, Chongqing University, Chongqing, China
| | - Bo Lin
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| | - Bo Xu
- School of Medicine, Chongqing University, Chongqing, China
- Chongqing Key Laboratory of Intelligent Oncology for Breast Cancer, Chongqing University Cancer Hospital, Chongqing, China
| |
Collapse
|
36
|
Dooper S, Pinckaers H, Aswolinskiy W, Hebeda K, Jarkman S, van der Laak J, Litjens G. Gigapixel end-to-end training using streaming and attention. Med Image Anal 2023; 88:102881. [PMID: 37437452 DOI: 10.1016/j.media.2023.102881] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 05/04/2023] [Accepted: 06/22/2023] [Indexed: 07/14/2023]
Abstract
Current hardware limitations make it impossible to train convolutional neural networks on gigapixel image inputs directly. Recent developments in weakly supervised learning, such as attention-gated multiple instance learning, have shown promising results, but often use multi-stage or patch-wise training strategies risking suboptimal feature extraction, which can negatively impact performance. In this paper, we propose to train a ResNet-34 encoder with an attention-gated classification head in an end-to-end fashion, which we call StreamingCLAM, using a streaming implementation of convolutional layers. This allows us to train end-to-end on 4-gigapixel microscopic images using only slide-level labels. We achieve a mean area under the receiver operating characteristic curve of 0.9757 for metastatic breast cancer detection (CAMELYON16), close to fully supervised approaches using pixel-level annotations. Our model can also detect MYC-gene translocation in histologic slides of diffuse large B-cell lymphoma, achieving a mean area under the ROC curve of 0.8259. Furthermore, we show that our model offers a degree of interpretability through the attention mechanism.
Collapse
Affiliation(s)
- Stephan Dooper
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands.
| | - Hans Pinckaers
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| | - Witali Aswolinskiy
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| | - Konnie Hebeda
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| | - Sofia Jarkman
- Department of Clinical Pathology, and Department of Biomedical and Clinical Sciences, Linköping University, Linköping 581 83, Sweden; Center for Medical Image Science and Visualization, Linköping University, Linköping 581 85, Sweden
| | - Jeroen van der Laak
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping 581 85, Sweden
| | - Geert Litjens
- Computational Pathology Group, Department of Pathology, Radboud University Medical Center, Nijmegen 6525 GA, The Netherlands
| |
Collapse
|
37
|
Zheng Q, Yang R, Xu H, Fan J, Jiao P, Ni X, Yuan J, Wang L, Chen Z, Liu X. A Weakly Supervised Deep Learning Model and Human-Machine Fusion for Accurate Grading of Renal Cell Carcinoma from Histopathology Slides. Cancers (Basel) 2023; 15:3198. [PMID: 37370808 DOI: 10.3390/cancers15123198] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Revised: 05/23/2023] [Accepted: 06/14/2023] [Indexed: 06/29/2023] Open
Abstract
(1) Background: The Fuhrman grading (FG) system is widely used in the management of clear cell renal cell carcinoma (ccRCC). However, it is affected by observer variability and irreproducibility in clinical practice. We aimed to use a deep learning multi-class model called SSL-CLAM to assist in diagnosing the FG status of ccRCC patients using digitized whole slide images (WSIs). (2) Methods: We recruited 504 eligible ccRCC patients from The Cancer Genome Atlas (TCGA) cohort and obtained 708 hematoxylin and eosin-stained WSIs for the development and internal validation of the SSL-CLAM model. Additionally, we obtained 445 WSIs from 188 ccRCC eligible patients in the Clinical Proteomic Tumor Analysis Consortium (CPTAC) cohort as an independent external validation set. A human-machine fusion approach was used to validate the added value of the SSL-CLAM model for pathologists. (3) Results: The SSL-CLAM model successfully diagnosed the five FG statuses (Grade-0, 1, 2, 3, and 4) of ccRCC, and achieved AUCs of 0.917 and 0.887 on the internal and external validation sets, respectively, outperforming a junior pathologist. For the normal/tumor classification (Grade-0, Grade-1/2/3/4) task, the SSL-CLAM model yielded AUCs close to 1 on both the internal and external validation sets. The SSL-CLAM model achieved a better performance for the two-tiered FG (Grade-0, Grade-1/2, and Grade-3/4) task, with AUCs of 0.936 and 0.915 on the internal and external validation sets, respectively. The human-machine diagnostic performance was superior to that of the SSL-CLAM model, showing promising prospects. In addition, the high-attention regions of the SSL-CLAM model showed that with an increasing FG status, the cell nuclei in the tumor region become larger, with irregular contours and increased cellular pleomorphism. (4) Conclusions: Our findings support the feasibility of using deep learning and human-machine fusion methods for FG classification on WSIs from ccRCC patients, which may assist pathologists in making diagnostic decisions.
Collapse
Affiliation(s)
- Qingyuan Zheng
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Rui Yang
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Huazhen Xu
- Department of Pharmacology, School of Basic Medical Sciences, Wuhan University, Wuhan 430072, China
| | - Junjie Fan
- University of Chinese Academy of Sciences, Beijing 100049, China
- Trusted Computing and Information Assurance Laboratory, Institute of Software, Chinese Academy of Sciences, Beijing 100190, China
| | - Panpan Jiao
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Xinmiao Ni
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Jingping Yuan
- Department of Pathology, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Lei Wang
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Zhiyuan Chen
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| | - Xiuheng Liu
- Department of Urology, Renmin Hospital of Wuhan University, Wuhan 430060, China
- Institute of Urologic Disease, Renmin Hospital of Wuhan University, Wuhan 430060, China
| |
Collapse
|
38
|
Wagner P, Springenberg M, Kröger M, Moritz RKC, Schleusener J, Meinke MC, Ma J. Semantic modeling of cell damage prediction: a machine learning approach at human-level performance in dermatology. Sci Rep 2023; 13:8336. [PMID: 37221254 DOI: 10.1038/s41598-023-35370-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Accepted: 05/17/2023] [Indexed: 05/25/2023] Open
Abstract
Machine learning is transforming the field of histopathology. Especially in classification related tasks, there have been many successful applications of deep learning already. Yet, in tasks that rely on regression and many niche applications, the domain lacks cohesive procedures that are adapted to the learning processes of neural networks. In this work, we investigate cell damage in whole slide images of the epidermis. A common way for pathologists to annotate a score, characterizing the degree of damage for these samples, is the ratio between healthy and unhealthy nuclei. The annotation procedure of these scores, however, is expensive and prone to be noisy among pathologists. We propose a new measure of damage, that is the total area of damage, relative to the total area of the epidermis. In this work, we present results of regression and segmentation models, predicting both scores on a curated and public dataset. We have acquired the dataset in collaborative efforts with medical professionals. Our study resulted in a comprehensive evaluation of the proposed damage metrics in the epidermis, with recommendations, emphasizing practical relevance for real world applications.
Collapse
Affiliation(s)
- Patrick Wagner
- Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587, Berlin, Germany
| | - Maximilian Springenberg
- Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587, Berlin, Germany
| | - Marius Kröger
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Rose K C Moritz
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Johannes Schleusener
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Martina C Meinke
- Department of Dermatology, Venereology and Allergology, Charité - Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin and Humboldt-Universität zu Berlin, Charitéplatz 1, 10117, Berlin, Germany
| | - Jackie Ma
- Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587, Berlin, Germany.
| |
Collapse
|
39
|
Irmakci I, Nateghi R, Zhou R, Ross AE, Yang XJ, Cooper LAD, Goldstein JA. Tissue contamination challenges the credibility of machine learning models in real world digital pathology. medRxiv 2023:2023.04.28.23289287. [PMID: 37205404 PMCID: PMC10187357 DOI: 10.1101/2023.04.28.23289287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm2, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Jeffery A. Goldstein
- To whom correspondence should be addressed: Olson 2-455, 710 N. Fairbanks Ave, Chicago IL, 60611,
| |
Collapse
|
40
|
Seraphin TP, Luedde M, Roderburg C, van Treeck M, Scheider P, Buelow RD, Boor P, Loosen SH, Provaznik Z, Mendelsohn D, Berisha F, Magnussen C, Westermann D, Luedde T, Brochhausen C, Sossalla S, Kather JN. Prediction of heart transplant rejection from routine pathology slides with self-supervised deep learning. Eur Heart J Digit Health 2023; 4:265-274. [PMID: 37265858 PMCID: PMC10232288 DOI: 10.1093/ehjdh/ztad016] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 02/07/2023] [Indexed: 06/03/2023]
Abstract
Aims One of the most important complications of heart transplantation is organ rejection, which is diagnosed on endomyocardial biopsies by pathologists. Computer-based systems could assist in the diagnostic process and potentially improve reproducibility. Here, we evaluated the feasibility of using deep learning in predicting the degree of cellular rejection from pathology slides as defined by the International Society for Heart and Lung Transplantation (ISHLT) grading system. Methods and results We collected 1079 histopathology slides from 325 patients from three transplant centres in Germany. We trained an attention-based deep neural network to predict rejection in the primary cohort and evaluated its performance using cross-validation and by deploying it to three cohorts. For binary prediction (rejection yes/no), the mean area under the receiver operating curve (AUROC) was 0.849 in the cross-validated experiment and 0.734, 0.729, and 0.716 in external validation cohorts. For a prediction of the ISHLT grade (0R, 1R, 2/3R), AUROCs were 0.835, 0.633, and 0.905 in the cross-validated experiment and 0.764, 0.597, and 0.913; 0.631, 0.633, and 0.682; and 0.722, 0.601, and 0.805 in the validation cohorts, respectively. The predictions of the artificial intelligence model were interpretable by human experts and highlighted plausible morphological patterns. Conclusion We conclude that artificial intelligence can detect patterns of cellular transplant rejection in routine pathology, even when trained on small cohorts.
Collapse
Affiliation(s)
| | | | | | - Marko van Treeck
- Department of Medicine III, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Pascal Scheider
- Institute of Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Roman D Buelow
- Institute of Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Peter Boor
- Institute of Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Sven H Loosen
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Medical Faculty at Heinrich-Heine-University, Moorenstr. 5, 40225 Dusseldorf, Germany
| | - Zdenek Provaznik
- Department of Cardiothoracic Surgery, University Medical Center Regensburg, Franz-Josef-Strauß-Allee 11, 93053 Regensburg, Germany
| | - Daniel Mendelsohn
- Institute of Pathology, University of Regensburg, Franz-Josef-Strauß-Allee 11, 93053 Regensburg, Germany
| | - Filip Berisha
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Hospital Eppendorf, Martinistraße 52, 20251 Hamburg, Germany
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Potsdamer Str. 58, 10785 Berlin, Germany
| | - Christina Magnussen
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Hospital Eppendorf, Martinistraße 52, 20251 Hamburg, Germany
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Potsdamer Str. 58, 10785 Berlin, Germany
| | - Dirk Westermann
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Hospital Eppendorf, Martinistraße 52, 20251 Hamburg, Germany
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Potsdamer Str. 58, 10785 Berlin, Germany
| | - Tom Luedde
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Medical Faculty at Heinrich-Heine-University, Moorenstr. 5, 40225 Dusseldorf, Germany
| | | | | | | |
Collapse
|
41
|
Tayebi Arasteh S, Isfort P, Saehn M, Mueller-Franzes G, Khader F, Kather JN, Kuhl C, Nebelung S, Truhn D. Collaborative training of medical artificial intelligence models with non-uniform labels. Sci Rep 2023; 13:6046. [PMID: 37055456 PMCID: PMC10102221 DOI: 10.1038/s41598-023-33303-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Accepted: 04/11/2023] [Indexed: 04/15/2023] Open
Abstract
Due to the rapid advancements in recent years, medical image analysis is largely dominated by deep learning (DL). However, building powerful and robust DL models requires training with large multi-party datasets. While multiple stakeholders have provided publicly available datasets, the ways in which these data are labeled vary widely. For Instance, an institution might provide a dataset of chest radiographs containing labels denoting the presence of pneumonia, while another institution might have a focus on determining the presence of metastases in the lung. Training a single AI model utilizing all these data is not feasible with conventional federated learning (FL). This prompts us to propose an extension to the widespread FL process, namely flexible federated learning (FFL) for collaborative training on such data. Using 695,000 chest radiographs from five institutions from across the globe-each with differing labels-we demonstrate that having heterogeneously labeled datasets, FFL-based training leads to significant performance increase compared to conventional FL training, where only the uniformly annotated images are utilized. We believe that our proposed algorithm could accelerate the process of bringing collaborative training methods from research and simulation phase to the real-world applications in healthcare.
Collapse
Affiliation(s)
- Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Peter Isfort
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Marwin Saehn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Gustav Mueller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
- Medical Faculty Carl Gustav Carus, Else Kroener Fresenius Center for Digital Health, Technical University Dresden, Dresden, Germany
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Pauwelsstr. 30, 52074, Aachen, Germany.
| |
Collapse
|
42
|
Saldanha OL, Loeffler CML, Niehues JM, van Treeck M, Seraphin TP, Hewitt KJ, Cifci D, Veldhuizen GP, Ramesh S, Pearson AT, Kather JN. Self-supervised attention-based deep learning for pan-cancer mutation prediction from histopathology. NPJ Precis Oncol 2023; 7:35. [PMID: 36977919 PMCID: PMC10050159 DOI: 10.1038/s41698-023-00365-0] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/22/2022] [Accepted: 02/17/2023] [Indexed: 03/30/2023] Open
Abstract
The histopathological phenotype of tumors reflects the underlying genetic makeup. Deep learning can predict genetic alterations from pathology slides, but it is unclear how well these predictions generalize to external datasets. We performed a systematic study on Deep-Learning-based prediction of genetic alterations from histology, using two large datasets of multiple tumor types. We show that an analysis pipeline that integrates self-supervised feature extraction and attention-based multiple instance learning achieves a robust predictability and generalizability.
Collapse
Affiliation(s)
- Oliver Lester Saldanha
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Chiara M L Loeffler
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Jan Moritz Niehues
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Marko van Treeck
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Tobias P Seraphin
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Medical Faculty at Heinrich-Heine-University Duesseldorf, Düsseldorf, Germany
| | - Katherine Jane Hewitt
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Didem Cifci
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Gregory Patrick Veldhuizen
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Siddhi Ramesh
- Pritzker School of Medicine, University of Chicago, Chicago, IL, USA
| | - Alexander T Pearson
- Biological Sciences Division, University of Chicago, Chicago, IL, USA
- University of Chicago Comprehensive Cancer Center, University of Chicago, Chicago, IL, USA
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany.
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
- Department of Medicine 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Technische Universität Dresden, Dresden, Germany.
| |
Collapse
|
43
|
Jiménez-Sánchez D, López-Janeiro Á, Villalba-Esparza M, Ariz M, Kadioglu E, Masetto I, Goubert V, Lozano MD, Melero I, Hardisson D, Ortiz-de-Solórzano C, de Andrea CE. Weakly supervised deep learning to predict recurrence in low-grade endometrial cancer from multiplexed immunofluorescence images. NPJ Digit Med 2023; 6:48. [PMID: 36959234 PMCID: PMC10036616 DOI: 10.1038/s41746-023-00795-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2022] [Accepted: 03/10/2023] [Indexed: 03/25/2023] Open
Abstract
Predicting recurrence in low-grade, early-stage endometrial cancer (EC) is both challenging and clinically relevant. We present a weakly-supervised deep learning framework, NaroNet, that can learn, without manual expert annotation, the complex tumor-immune interrelations at three levels: local phenotypes, cellular neighborhoods, and tissue areas. It uses multiplexed immunofluorescence for the simultaneous visualization and quantification of CD68 + macrophages, CD8 + T cells, FOXP3 + regulatory T cells, PD-L1/PD-1 protein expression, and tumor cells. We used 489 tumor cores from 250 patients to train a multilevel deep-learning model to predict tumor recurrence. Using a tenfold cross-validation strategy, our model achieved an area under the curve of 0.90 with a 95% confidence interval of 0.83-0.95. Our model predictions resulted in concordance for 96,8% of cases (κ = 0.88). This method could accurately assess the risk of recurrence in EC, outperforming current prognostic factors, including molecular subtyping.
Collapse
Affiliation(s)
- Daniel Jiménez-Sánchez
- Program of Solid Tumors and Biomarkers, Center for Applied Medical Research (CIMA), University of Navarra, Pamplona, Spain
- Department of Pathology, Clínica Universidad de Navarra, Pamplona, Spain
| | - Álvaro López-Janeiro
- Department of Pathology, Clínica Universidad de Navarra, Pamplona, Spain
- Department of Pathology, Hospital Universitario La Paz, IdiPAZ, Madrid, Spain
| | - María Villalba-Esparza
- Department of Pathology, Clínica Universidad de Navarra, Pamplona, Spain
- Navarra Institute for Health Research (IdISNA), Pamplona, Spain
| | - Mikel Ariz
- Program of Solid Tumors and Biomarkers, Center for Applied Medical Research (CIMA), University of Navarra, Pamplona, Spain
- Navarra Institute for Health Research (IdISNA), Pamplona, Spain
| | - Ece Kadioglu
- Lunaphore Technologies SA, Tolochenaz, Switzerland
| | | | | | - Maria D Lozano
- Department of Pathology, Clínica Universidad de Navarra, Pamplona, Spain
- Navarra Institute for Health Research (IdISNA), Pamplona, Spain
- Center for Biomedical Research in the Cancer Network (CIBERONC), Madrid, Spain
| | - Ignacio Melero
- Navarra Institute for Health Research (IdISNA), Pamplona, Spain
- Center for Biomedical Research in the Cancer Network (CIBERONC), Madrid, Spain
- Department of Immunology and Immunotherapy, Clínica Universidad de Navarra, Pamplona, Spain
- Program of Immunology and Immunotherapy, Center for Applied Medical Research (CIMA), University of Navarra, Pamplona, Spain
| | - David Hardisson
- Department of Pathology, Hospital Universitario La Paz, IdiPAZ, Madrid, Spain
- Center for Biomedical Research in the Cancer Network (CIBERONC), Madrid, Spain
- Molecular Pathology and Therapeutic Targets Group, La Paz University Hospital, IdiPAZ, Madrid, Spain
- Faculty of Medicine, Universidad Autónoma de Madrid, Madrid, Spain
| | - Carlos Ortiz-de-Solórzano
- Program of Solid Tumors and Biomarkers, Center for Applied Medical Research (CIMA), University of Navarra, Pamplona, Spain
- Navarra Institute for Health Research (IdISNA), Pamplona, Spain
- Center for Biomedical Research in the Cancer Network (CIBERONC), Madrid, Spain
| | - Carlos E de Andrea
- Department of Pathology, Clínica Universidad de Navarra, Pamplona, Spain.
- Navarra Institute for Health Research (IdISNA), Pamplona, Spain.
- Center for Biomedical Research in the Cancer Network (CIBERONC), Madrid, Spain.
| |
Collapse
|
44
|
Niehues JM, Quirke P, West NP, Grabsch HI, van Treeck M, Schirris Y, Veldhuizen GP, Hutchins GGA, Richman SD, Foersch S, Brinker TJ, Fukuoka J, Bychkov A, Uegami W, Truhn D, Brenner H, Brobeil A, Hoffmeister M, Kather JN. Generalizable biomarker prediction from cancer pathology slides with self-supervised deep learning: A retrospective multi-centric study. Cell Rep Med 2023; 4:100980. [PMID: 36958327 PMCID: PMC10140458 DOI: 10.1016/j.xcrm.2023.100980] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2022] [Revised: 12/28/2022] [Accepted: 02/24/2023] [Indexed: 03/25/2023]
Abstract
Deep learning (DL) can predict microsatellite instability (MSI) from routine histopathology slides of colorectal cancer (CRC). However, it is unclear whether DL can also predict other biomarkers with high performance and whether DL predictions generalize to external patient populations. Here, we acquire CRC tissue samples from two large multi-centric studies. We systematically compare six different state-of-the-art DL architectures to predict biomarkers from pathology slides, including MSI and mutations in BRAF, KRAS, NRAS, and PIK3CA. Using a large external validation cohort to provide a realistic evaluation setting, we show that models using self-supervised, attention-based multiple-instance learning consistently outperform previous approaches while offering explainable visualizations of the indicative regions and morphologies. While the prediction of MSI and BRAF mutations reaches a clinical-grade performance, mutation prediction of PIK3CA, KRAS, and NRAS was clinically insufficient.
Collapse
Affiliation(s)
- Jan Moritz Niehues
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, 01307 Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, 52074 Aachen, Germany
| | - Philip Quirke
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds LS9 7TF, UK
| | - Nicholas P West
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds LS9 7TF, UK
| | - Heike I Grabsch
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds LS9 7TF, UK; Department of Pathology, GROW School for Oncology and Reproduction, Maastricht University Medical Center+, 6229 HX Maastricht, the Netherlands
| | - Marko van Treeck
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, 01307 Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, 52074 Aachen, Germany
| | - Yoni Schirris
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, 01307 Dresden, Germany; Netherlands Cancer Institute, 1066 CX Amsterdam, the Netherlands; University of Amsterdam, 1012 WP Amsterdam, the Netherlands
| | - Gregory P Veldhuizen
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, 01307 Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, 52074 Aachen, Germany
| | - Gordon G A Hutchins
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds LS9 7TF, UK
| | - Susan D Richman
- Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds LS9 7TF, UK
| | - Sebastian Foersch
- Institute of Pathology, University Medical Center Mainz, 55131 Mainz, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany
| | - Junya Fukuoka
- Department of Pathology Informatics, Nagasaki University Graduate School of Biomedical Sciences, Nagasaki 852-8523, Japan; Department of Pathology, Kameda Medical Center, Kamogawa 296-8602, Chiba, Japan
| | - Andrey Bychkov
- Department of Pathology, Kameda Medical Center, Kamogawa 296-8602, Chiba, Japan
| | - Wataru Uegami
- Department of Pathology, Kameda Medical Center, Kamogawa 296-8602, Chiba, Japan
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, 52074 Aachen, Germany
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany; Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), 69120 Heidelberg, Germany; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany
| | - Alexander Brobeil
- Institute of Pathology, University Hospital Heidelberg, 69120 Heidelberg, Germany; Tissue Bank, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, 69120 Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), 69120 Heidelberg, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University Dresden, 01307 Dresden, Germany; Department of Medicine III, University Hospital RWTH Aachen, 52074 Aachen, Germany; Pathology & Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds LS9 7TF, UK; Department of Medicine I, University Hospital Dresden, 01307 Dresden, Germany; Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, 69120 Heidelberg, Germany.
| |
Collapse
|
45
|
Vorndran J, Neuner C, Coras R, Hoffmann L, Geffers S, Honke J, Herms J, Roeber S, Hamer H, Brandner S, Hartlieb T, Pieper T, Kudernatsch M, Bien CG, Kalbhenn T, Simon M, Adle-biassette H, Cienfuegos J, Di Giacomo R, Garbelli R, Miyata H, Mühlebner A, Raicevic S, Rauramaa T, Rogerio F, Blümcke I, Jabari S. A deep learning-based histopathology classifier for Focal Cortical Dysplasia. Neural Comput Appl 2023. [DOI: 10.1007/s00521-023-08364-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/11/2023]
Abstract
AbstractA light microscopy-based histopathology diagnosis of human brain specimens obtained from epilepsy surgery remains the gold standard to confirm the underlying cause of a patient’s focal epilepsy and further inform postsurgical patient management. The differential diagnosis of neocortical specimens in the realm of epilepsy surgery remains, however, challenging. Herein, we developed an open access, deep learning-based classifier to histopathologically assess whole slide microscopy images (WSI) and to automatically recognize various subtypes of Focal Cortical Dysplasia (FCD), according to the ILAE consensus classification update of 2022. We trained a convolutional neuronal network (CNN) with fully digitalized WSI of hematoxylin–eosin stainings obtained from 125 patients covering the spectrum of mild malformation of cortical development (mMCD), mMCD with oligodendroglial hyperplasia in epilepsy (MOGHE), FCD ILAE Type 1a, 2a and 2b using 414 formalin-fixed and paraffin-embedded archival tissue blocks. An additional series of 198 postmortem tissue blocks from 59 patients without neurological disorders served as control to train the CNN for homotypic frontal, temporal and occipital areas and heterotypic Brodmann areas 4 and 17, entorhinal cortex and dentate gyrus. Special stains and immunohistochemical reactions were used to comprehensively annotate the region of interest. We then programmed a novel tile extraction pipeline and graphical dashboard to visualize all areas on the WSI recognized by the CNN. Our deep learning-based classifier is able to compute 1000 × 1000 µm large tiles and recognizes 25 anatomical regions and FCD categories with an accuracy of 98.8% (F1 score = 0.82). Microscopic review of regions predicted by the network confirmed these results. This deep learning-based classifier will be made available as online web application to support the differential histopathology diagnosis in neocortical human brain specimens obtained from epilepsy surgery. It will also serve as blueprint to build a digital histopathology slide suite addressing all major brain diseases encountered in patients with surgically amenable focal epilepsy.
Collapse
|
46
|
Shao J, Ma J, Zhang Q, Li W, Wang C. Predicting gene mutation status via artificial intelligence technologies based on multimodal integration (MMI) to advance precision oncology. Semin Cancer Biol 2023; 91:1-15. [PMID: 36801447 DOI: 10.1016/j.semcancer.2023.02.006] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/29/2022] [Revised: 01/30/2023] [Accepted: 02/15/2023] [Indexed: 02/21/2023]
Abstract
Personalized treatment strategies for cancer frequently rely on the detection of genetic alterations which are determined by molecular biology assays. Historically, these processes typically required single-gene sequencing, next-generation sequencing, or visual inspection of histopathology slides by experienced pathologists in a clinical context. In the past decade, advances in artificial intelligence (AI) technologies have demonstrated remarkable potential in assisting physicians with accurate diagnosis of oncology image-recognition tasks. Meanwhile, AI techniques make it possible to integrate multimodal data such as radiology, histology, and genomics, providing critical guidance for the stratification of patients in the context of precision therapy. Given that the mutation detection is unaffordable and time-consuming for a considerable number of patients, predicting gene mutations based on routine clinical radiological scans or whole-slide images of tissue with AI-based methods has become a hot issue in actual clinical practice. In this review, we synthesized the general framework of multimodal integration (MMI) for molecular intelligent diagnostics beyond standard techniques. Then we summarized the emerging applications of AI in the prediction of mutational and molecular profiles of common cancers (lung, brain, breast, and other tumor types) pertaining to radiology and histology imaging. Furthermore, we concluded that there truly exist multiple challenges of AI techniques in the way of its real-world application in the medical field, including data curation, feature fusion, model interpretability, and practice regulations. Despite these challenges, we still prospect the clinical implementation of AI as a highly potential decision-support tool to aid oncologists in future cancer treatment management.
Collapse
|
47
|
Familiar AM, Mahtabfar A, Fathi Kazerooni A, Kiani M, Vossough A, Viaene A, Storm PB, Resnick AC, Nabavizadeh A. Radio-pathomic approaches in pediatric neuro-oncology: Opportunities and challenges. Neurooncol Adv 2023; 5:vdad119. [PMID: 37841693 PMCID: PMC10576517 DOI: 10.1093/noajnl/vdad119] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2023] Open
Abstract
With medical software platforms moving to cloud environments with scalable storage and computing, the translation of predictive artificial intelligence (AI) models to aid in clinical decision-making and facilitate personalized medicine for cancer patients is becoming a reality. Medical imaging, namely radiologic and histologic images, has immense analytical potential in neuro-oncology, and models utilizing integrated radiomic and pathomic data may yield a synergistic effect and provide a new modality for precision medicine. At the same time, the ability to harness multi-modal data is met with challenges in aggregating data across medical departments and institutions, as well as significant complexity in modeling the phenotypic and genotypic heterogeneity of pediatric brain tumors. In this paper, we review recent pathomic and integrated pathomic, radiomic, and genomic studies with clinical applications. We discuss current challenges limiting translational research on pediatric brain tumors and outline technical and analytical solutions. Overall, we propose that to empower the potential residing in radio-pathomics, systemic changes in cross-discipline data management and end-to-end software platforms to handle multi-modal data sets are needed, in addition to embracing modern AI-powered approaches. These changes can improve the performance of predictive models, and ultimately the ability to advance brain cancer treatments and patient outcomes through the development of such models.
Collapse
Affiliation(s)
- Ariana M Familiar
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Aria Mahtabfar
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Thomas Jefferson University Hospital, Philadelphia, PA, USA
| | - Anahita Fathi Kazerooni
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Mahsa Kiani
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Arastoo Vossough
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Angela Viaene
- Department of Pathology and Laboratory Medicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Phillip B Storm
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Adam C Resnick
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Ali Nabavizadeh
- Center for Data-Driven Discovery in Biomedicine, Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
48
|
Couture HD. Deep Learning-Based Prediction of Molecular Tumor Biomarkers from H&E: A Practical Review. J Pers Med 2022; 12:2022. [PMID: 36556243 PMCID: PMC9784641 DOI: 10.3390/jpm12122022] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Revised: 10/26/2022] [Accepted: 12/05/2022] [Indexed: 12/12/2022] Open
Abstract
Molecular and genomic properties are critical in selecting cancer treatments to target individual tumors, particularly for immunotherapy. However, the methods to assess such properties are expensive, time-consuming, and often not routinely performed. Applying machine learning to H&E images can provide a more cost-effective screening method. Dozens of studies over the last few years have demonstrated that a variety of molecular biomarkers can be predicted from H&E alone using the advancements of deep learning: molecular alterations, genomic subtypes, protein biomarkers, and even the presence of viruses. This article reviews the diverse applications across cancer types and the methodology to train and validate these models on whole slide images. From bottom-up to pathologist-driven to hybrid approaches, the leading trends include a variety of weakly supervised deep learning-based approaches, as well as mechanisms for training strongly supervised models in select situations. While results of these algorithms look promising, some challenges still persist, including small training sets, rigorous validation, and model explainability. Biomarker prediction models may yield a screening method to determine when to run molecular tests or an alternative when molecular tests are not possible. They also create new opportunities in quantifying intratumoral heterogeneity and predicting patient outcomes.
Collapse
|
49
|
Miao S, Jia H, Cheng K, Hu X, Li J, Huang W, Wang R. Deep learning radiomics under multimodality explore association between muscle/fat and metastasis and survival in breast cancer patients. Brief Bioinform 2022; 23:6748489. [PMID: 36198668 DOI: 10.1093/bib/bbac432] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2022] [Revised: 09/03/2022] [Accepted: 09/06/2022] [Indexed: 12/14/2022] Open
Abstract
Sarcopenia is correlated with poor clinical outcomes in breast cancer (BC) patients. However, there is no precise quantitative study on the correlation between body composition changes and BC metastasis and survival. The present study proposed a deep learning radiomics (DLR) approach to investigate the effects of muscle and fat on distant metastasis and death outcomes in BC patients. Image feature extraction was performed on 4th thoracic vertebra (T4) and 11th thoracic vertebra (T11) on computed tomography (CT) image levels by DLR, and image features were combined with clinical information to predict distant metastasis in BC patients. Clinical information combined with DLR significantly predicted distant metastasis in BC patients. In the test cohort, the area under the curve of model performance on clinical information combined with DLR was 0.960 (95% CI: 0.942-0.979, P < 0.001). The patients with distant metastases had a lower pectoral muscle index in T4 (PMI/T4) than in patients without metastases. PMI/T4 and visceral fat tissue area in T11 (VFA/T11) were independent prognostic factors for the overall survival in BC patients. The pectoralis muscle area in T4 (PMA/T4) and PMI/T4 is an independent prognostic factor for distant metastasis-free survival in BC patients. The current study further confirmed that muscle/fat of T4 and T11 levels have a significant effect on the distant metastasis of BC. Appending the network features of T4 and T11 to the model significantly enhances the prediction performance of distant metastasis of BC, providing a valuable biomarker for the early treatment of BC patients.
Collapse
Affiliation(s)
- Shidi Miao
- School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
| | - Haobo Jia
- School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
| | - Ke Cheng
- School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
| | - Xiaohui Hu
- School of Computer Science and Technology, Harbin University of Science and Technology, Harbin, China
| | - Jing Li
- Department of Geriatrics, the Second Affiliated Hospital, Harbin Medical University, Harbin, China
| | - Wenjuan Huang
- Department of Internal Medicine, Harbin Medical University Cancer Hospital, Harbin Medical University, Harbin, China
| | - Ruitao Wang
- Department of Internal Medicine, Harbin Medical University Cancer Hospital, Harbin Medical University, Harbin, China
| |
Collapse
|
50
|
Zhu H, Liu M, Tang Z, Wang S. Editorial: Weakly supervised deep learning-based methods for brain image analysis. Front Neuroinform 2022; 16:1079240. [DOI: 10.3389/fninf.2022.1079240] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2022] [Accepted: 11/01/2022] [Indexed: 11/13/2022] Open
|