1
|
Seraphin TP, Luedde M, Roderburg C, van Treeck M, Scheider P, Buelow RD, Boor P, Loosen SH, Provaznik Z, Mendelsohn D, Berisha F, Magnussen C, Westermann D, Luedde T, Brochhausen C, Sossalla S, Kather JN. Prediction of heart transplant rejection from routine pathology slides with self-supervised deep learning. Eur Heart J Digit Health 2023; 4:265-274. [PMID: 37265858 PMCID: PMC10232288 DOI: 10.1093/ehjdh/ztad016] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/19/2022] [Revised: 02/07/2023] [Indexed: 06/03/2023]
Abstract
Aims One of the most important complications of heart transplantation is organ rejection, which is diagnosed on endomyocardial biopsies by pathologists. Computer-based systems could assist in the diagnostic process and potentially improve reproducibility. Here, we evaluated the feasibility of using deep learning in predicting the degree of cellular rejection from pathology slides as defined by the International Society for Heart and Lung Transplantation (ISHLT) grading system. Methods and results We collected 1079 histopathology slides from 325 patients from three transplant centres in Germany. We trained an attention-based deep neural network to predict rejection in the primary cohort and evaluated its performance using cross-validation and by deploying it to three cohorts. For binary prediction (rejection yes/no), the mean area under the receiver operating curve (AUROC) was 0.849 in the cross-validated experiment and 0.734, 0.729, and 0.716 in external validation cohorts. For a prediction of the ISHLT grade (0R, 1R, 2/3R), AUROCs were 0.835, 0.633, and 0.905 in the cross-validated experiment and 0.764, 0.597, and 0.913; 0.631, 0.633, and 0.682; and 0.722, 0.601, and 0.805 in the validation cohorts, respectively. The predictions of the artificial intelligence model were interpretable by human experts and highlighted plausible morphological patterns. Conclusion We conclude that artificial intelligence can detect patterns of cellular transplant rejection in routine pathology, even when trained on small cohorts.
Collapse
Affiliation(s)
| | | | | | - Marko van Treeck
- Department of Medicine III, University Hospital RWTH Aachen, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Pascal Scheider
- Institute of Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Roman D Buelow
- Institute of Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Peter Boor
- Institute of Pathology, RWTH Aachen University Hospital, Pauwelsstraße 30, 52074 Aachen, Germany
| | - Sven H Loosen
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Medical Faculty at Heinrich-Heine-University, Moorenstr. 5, 40225 Dusseldorf, Germany
| | - Zdenek Provaznik
- Department of Cardiothoracic Surgery, University Medical Center Regensburg, Franz-Josef-Strauß-Allee 11, 93053 Regensburg, Germany
| | - Daniel Mendelsohn
- Institute of Pathology, University of Regensburg, Franz-Josef-Strauß-Allee 11, 93053 Regensburg, Germany
| | - Filip Berisha
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Hospital Eppendorf, Martinistraße 52, 20251 Hamburg, Germany
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Potsdamer Str. 58, 10785 Berlin, Germany
| | - Christina Magnussen
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Hospital Eppendorf, Martinistraße 52, 20251 Hamburg, Germany
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Potsdamer Str. 58, 10785 Berlin, Germany
| | - Dirk Westermann
- Department of Cardiology, University Heart and Vascular Center Hamburg, University Medical Center Hamburg-Hospital Eppendorf, Martinistraße 52, 20251 Hamburg, Germany
- German Center for Cardiovascular Research (DZHK), Partner Site Hamburg/Kiel/Lübeck, Potsdamer Str. 58, 10785 Berlin, Germany
| | - Tom Luedde
- Department of Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Medical Faculty at Heinrich-Heine-University, Moorenstr. 5, 40225 Dusseldorf, Germany
| | | | | | | |
Collapse
|
2
|
Ghaffari Laleh N, Truhn D, Veldhuizen GP, Han T, van Treeck M, Buelow RD, Langer R, Dislich B, Boor P, Schulz V, Kather JN. Adversarial attacks and adversarial robustness in computational pathology. Nat Commun 2022; 13:5711. [PMID: 36175413 PMCID: PMC9522657 DOI: 10.1038/s41467-022-33266-0] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2022] [Accepted: 09/09/2022] [Indexed: 11/09/2022] Open
Abstract
Artificial Intelligence (AI) can support diagnostic workflows in oncology by aiding diagnosis and providing biomarkers directly from routine pathology slides. However, AI applications are vulnerable to adversarial attacks. Hence, it is essential to quantify and mitigate this risk before widespread clinical use. Here, we show that convolutional neural networks (CNNs) are highly susceptible to white- and black-box adversarial attacks in clinically relevant weakly-supervised classification tasks. Adversarially robust training and dual batch normalization (DBN) are possible mitigation strategies but require precise knowledge of the type of attack used in the inference. We demonstrate that vision transformers (ViTs) perform equally well compared to CNNs at baseline, but are orders of magnitude more robust to white- and black-box attacks. At a mechanistic level, we show that this is associated with a more robust latent representation of clinically relevant categories in ViTs compared to CNNs. Our results are in line with previous theoretical studies and provide empirical evidence that ViTs are robust learners in computational pathology. This implies that large-scale rollout of AI models in computational pathology should rely on ViTs rather than CNN-based classifiers to provide inherent protection against perturbation of the input data, especially adversarial attacks. Artificial Intelligence can support diagnostic workflows in oncology, but they are vulnerable to adversarial attacks. Here, the authors show that convolutional neural networks are highly susceptible to white- and black-box adversarial attacks in clinically relevant classification tasks.
Collapse
Affiliation(s)
- Narmin Ghaffari Laleh
- Department of Medicine III, University Hospital RWTH Aachen, RWTH Aachen university, Aachen, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Gregory Patrick Veldhuizen
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
| | - Tianyu Han
- Department of Physics of Molecular Imaging Systems, Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany
| | - Marko van Treeck
- Department of Medicine III, University Hospital RWTH Aachen, RWTH Aachen university, Aachen, Germany
| | - Roman D Buelow
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Rupert Langer
- Institute of Pathology, University of Bern, Bern, Switzerland.,Institute of Pathology and Molecular Pathology, Kepler University Hospital, Johannes Kepler University Linz, Linz, Austria
| | - Bastian Dislich
- Institute of Pathology, University of Bern, Bern, Switzerland
| | - Peter Boor
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Volkmar Schulz
- Department of Physics of Molecular Imaging Systems, Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany.,Physics Institute III B, RWTH Aachen University, Aachen, Germany.,Fraunhofer Institute for Digital Medicine MEVIS, Aachen, Germany.,Hyperion Hybrid Imaging Systems GmbH, Aachen, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, RWTH Aachen university, Aachen, Germany. .,Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany. .,Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany. .,Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK. .,Department of Medicine 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
| |
Collapse
|
3
|
Ghaffari Laleh N, Muti HS, Loeffler CML, Echle A, Saldanha OL, Mahmood F, Lu MY, Trautwein C, Langer R, Dislich B, Buelow RD, Grabsch HI, Brenner H, Chang-Claude J, Alwers E, Brinker TJ, Khader F, Truhn D, Gaisa NT, Boor P, Hoffmeister M, Schulz V, Kather JN. Erratum to 'Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology' Medical Image Analysis, Volume 79, July 2022, 102474. Med Image Anal 2022; 82:102622. [PMID: 36130464 DOI: 10.1016/j.media.2022.102622] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Affiliation(s)
| | - Hannah Sophie Muti
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | | | - Amelie Echle
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | | | - Faisal Mahmood
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Ming Y Lu
- Department of Pathology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Christian Trautwein
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Rupert Langer
- Institute of Pathology and Molecular Pathology, Kepler University Hospital, Johannes Kepler University Linz, Linz, Austria
| | | | - Roman D Buelow
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Heike Irmgard Grabsch
- Department of Pathology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center+, Maastricht, The Netherlands.; Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Hermann Brenner
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany; Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany; German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Jenny Chang-Claude
- Division of Cancer Epidemiology, German Cancer Research Center (DKFZ), Heidelberg, Germany; Cancer Epidemiology Group, University Cancer Center Hamburg, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| | - Elizabeth Alwers
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Titus J Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Firas Khader
- Department of Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Daniel Truhn
- Department of Radiology, University Hospital RWTH Aachen, Aachen, Germany
| | - Nadine T Gaisa
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Peter Boor
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Volkmar Schulz
- Department of Physics of Molecular Imaging Systems, Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany; Comprehensive Diagnostic Center Aachen (CDCA), University Hospital Aachen, Aachen, Germany; Hyperion Hybrid Imaging Systems GmbH, Aachen, Germany
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany; Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK; Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany.
| |
Collapse
|
4
|
Laleh NG, Muti HS, Loeffler CML, Echle A, Saldanha OL, Mahmood F, Lu MY, Trautwein C, Langer R, Dislich B, Buelow RD, Grabsch HI, Brenner H, Chang-Claude J, Alwers E, Brinker TJ, Khader F, Truhn D, Gaisa NT, Boor P, Hoffmeister M, Schulz V, Kather JN. Benchmarking weakly-supervised deep learning pipelines for whole slide classification in computational pathology. Med Image Anal 2022; 79:102474. [DOI: 10.1016/j.media.2022.102474] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2021] [Revised: 04/07/2022] [Accepted: 05/03/2022] [Indexed: 02/07/2023]
|
5
|
Kather JN, Heij LR, Grabsch HI, Loeffler C, Echle A, Muti HS, Krause J, Niehues JM, Sommer KAJ, Bankhead P, Kooreman LFS, Schulte JJ, Cipriani NA, Buelow RD, Boor P, Ortiz-Brüchle N, Hanby AM, Speirs V, Kochanny S, Patnaik A, Srisuwananukorn A, Brenner H, Hoffmeister M, van den Brandt PA, Jäger D, Trautwein C, Pearson AT, Luedde T. Author Correction: Pan-cancer image-based detection of clinically actionable genetic alterations. Nat Cancer 2020; 1:1129. [PMID: 35122072 DOI: 10.1038/s43018-020-00149-6] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Affiliation(s)
- Jakob Nikolas Kather
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany.
- German Cancer Consortium (DKTK), Heidelberg, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
| | - Lara R Heij
- Department of Surgery and Transplantation, University Hospital RWTH Aachen, Aachen, Germany
- Department of Surgery, School of Nutrition and Translational Research in Metabolism (NUTRIM), Maastricht University, Maastricht, the Netherlands
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Heike I Grabsch
- Department of Pathology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center+, Maastricht, the Netherlands
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Chiara Loeffler
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Amelie Echle
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Hannah Sophie Muti
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Jeremias Krause
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Jan M Niehues
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Kai A J Sommer
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | - Peter Bankhead
- MRC Institute of Genetics and Molecular Medicine, University of Edinburgh, Edinburgh, UK
| | - Loes F S Kooreman
- Department of Pathology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center+, Maastricht, the Netherlands
| | - Jefree J Schulte
- Department of Pathology, University of Chicago Medicine, Chicago, IL, USA
| | - Nicole A Cipriani
- Department of Pathology, University of Chicago Medicine, Chicago, IL, USA
| | - Roman D Buelow
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | - Peter Boor
- Institute of Pathology, University Hospital RWTH Aachen, Aachen, Germany
| | | | - Andrew M Hanby
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
| | - Valerie Speirs
- Institute of Medical Sciences, School of Medicine, Medical Sciences and Nutrition, University of Aberdeen, Aberdeen, UK
| | - Sara Kochanny
- Department of Medicine, University of Chicago Medicine, Chicago, IL, USA
| | - Akash Patnaik
- Department of Medicine, University of Chicago Medicine, Chicago, IL, USA
| | | | - Hermann Brenner
- German Cancer Consortium (DKTK), Heidelberg, Germany
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Division of Preventive Oncology, German Cancer Research Center (DKFZ) and National Center for Tumor Diseases (NCT), Heidelberg, Germany
| | - Michael Hoffmeister
- Division of Clinical Epidemiology and Aging Research, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Piet A van den Brandt
- Department of Epidemiology, GROW School for Oncology and Developmental Biology, Maastricht University Medical Center+, Maastricht, The Netherlands
| | - Dirk Jäger
- German Cancer Consortium (DKTK), Heidelberg, Germany
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Christian Trautwein
- Department of Medicine III, University Hospital RWTH Aachen, Aachen, Germany
| | | | - Tom Luedde
- Division of Gastroenterology, Hepatology and Gastrointestinal Oncology, University Hospital RWTH Aachen, Aachen, Germany.
- Clinic for Gastroenterology, Hepatology and Infectious Diseases, University Hospital Duesseldorf, Duesseldorf, Germany.
| |
Collapse
|