1
|
Irmakci I, Nateghi R, Zhou R, Vescovo M, Saft M, Ross AE, Yang XJ, Cooper LAD, Goldstein JA. Tissue Contamination Challenges the Credibility of Machine Learning Models in Real World Digital Pathology. Mod Pathol 2024; 37:100422. [PMID: 38185250 PMCID: PMC10960671 DOI: 10.1016/j.modpat.2024.100422] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 11/13/2023] [Accepted: 12/15/2023] [Indexed: 01/09/2024]
Abstract
Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. Although human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole-slide models. Three operate in placenta for the following functions: (1) detection of decidual arteriopathy, (2) estimation of gestational age, and (3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in the t-distributed stochastic neighbor embedding feature space. Every model showed performance degradation in response to one or more tissue contaminants. Decidual arteriopathy detection--balanced accuracy decreased from 0.74 to 0.69 ± 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant, raised the mean absolute error in estimating gestational age from 1.626 weeks to 2.371 ± 0.003 weeks. Blood, incorporated into placental sections, induced false-negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033 mm2, and resulted in a 97% false-positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.
Collapse
Affiliation(s)
- Ismail Irmakci
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ramin Nateghi
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Rujoi Zhou
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Mariavittoria Vescovo
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Madeline Saft
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ashley E Ross
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Ximing J Yang
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Lee A D Cooper
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois
| | - Jeffery A Goldstein
- Department of Pathology, Northwestern University, Feinberg School of Medicine, Chicago, Illinois.
| |
Collapse
|
2
|
Kim K, Lee K, Cho S, Kang DU, Park S, Kang Y, Kim H, Choe G, Moon KC, Lee KS, Park JH, Hong C, Nateghi R, Pourakpour F, Wang X, Yang S, Jahromi SAF, Khani A, Kim HR, Choi DH, Han CH, Kwak JT, Zhang F, Han B, Ho DJ, Kang GH, Chun SY, Jeong WK, Park P, Choi J. PAIP 2020: Microsatellite instability prediction in colorectal cancer. Med Image Anal 2023; 89:102886. [PMID: 37494811 DOI: 10.1016/j.media.2023.102886] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2022] [Revised: 06/21/2023] [Accepted: 06/28/2023] [Indexed: 07/28/2023]
Abstract
Microsatellite instability (MSI) refers to alterations in the length of simple repetitive genomic sequences. MSI status serves as a prognostic and predictive factor in colorectal cancer. The MSI-high status is a good prognostic factor in stage II/III cancer, and predicts a lack of benefit to adjuvant fluorouracil chemotherapy in stage II cancer but a good response to immunotherapy in stage IV cancer. Therefore, determining MSI status in patients with colorectal cancer is important for identifying the appropriate treatment protocol. In the Pathology Artificial Intelligence Platform (PAIP) 2020 challenge, artificial intelligence researchers were invited to predict MSI status based on colorectal cancer slide images. Participants were required to perform two tasks. The primary task was to classify a given slide image as belonging to either the MSI-high or the microsatellite-stable group. The second task was tumor area segmentation to avoid ties with the main task. A total of 210 of the 495 participants enrolled in the challenge downloaded the images, and 23 teams submitted their final results. Seven teams from the top 10 participants agreed to disclose their algorithms, most of which were convolutional neural network-based deep learning models, such as EfficientNet and UNet. The top-ranked system achieved the highest F1 score (0.9231). This paper summarizes the various methods used in the PAIP 2020 challenge. This paper supports the effectiveness of digital pathology for identifying the relationship between colorectal cancer and the MSI characteristics.
Collapse
Affiliation(s)
- Kyungmo Kim
- Interdisciplinary program in Bioengineering, Seoul National University, Seoul 110-799, Republic of Korea
| | - Kyoungbun Lee
- Department of Pathology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Sungduk Cho
- Korea University, College of Informatics, Department of Computer Science and Engineering, Seoul, Republic of Korea
| | - Dong Un Kang
- Department of Electrical and Computer Engineering, Seoul National University, Seoul, Republic of Korea
| | - Seongkeun Park
- Department of Biomedical Engineering, Seoul National University Hospital, Seoul, Republic of Korea
| | - Yunsook Kang
- Department of Biomedical Engineering, Seoul National University Hospital, Seoul, Republic of Korea
| | - Hyunjeong Kim
- Department of Biomedical Engineering, Seoul National University Hospital, Seoul, Republic of Korea
| | - Gheeyoung Choe
- Department of Pathology, Seoul National University Bundang Hospital, Seongnam, Republic of Korea; Department of Pathology, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Kyung Chul Moon
- Department of Pathology, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Kyu Sang Lee
- Department of Pathology, Seoul National University Bundang Hospital, Seongnam, Republic of Korea; Department of Pathology, Seoul National University College of Medicine, Seoul, Republic of Korea
| | - Jeong Hwan Park
- Department of Pathology, Seoul National University College of Medicine, SMG-SNU Boramae Medical Center, Seoul, Republic of Korea
| | - Choyeon Hong
- Department of Pathology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Ramin Nateghi
- Electrical and Electronics Engineering Department, Shiraz University of Technology, Shiraz, Iran
| | - Fattaneh Pourakpour
- Iranian Brain Mapping Biobank, National Brain Mapping Laboratory, Tehran, Iran
| | - Xiyue Wang
- College of Computer Science, Sichuan University, China
| | - Sen Yang
- College of Biomedical Engineering, Sichuan University, China; Tencent AI Lab, Shenzhen, China
| | | | - Aliasghar Khani
- Department of Computer Engineering, Sharif University of Technology, Tehran, Iran
| | - Hwa-Rang Kim
- Graduate School of Electronic and Electrical Engineering, Kyungpook National University, Republic of Korea
| | - Doo-Hyun Choi
- Graduate School of Electronic and Electrical Engineering, Kyungpook National University, Republic of Korea
| | - Chang Hee Han
- Department of Computer Science and Engineering, Sejong University, Seoul, Republic of Korea
| | - Jin Tae Kwak
- School of Electrical Engineering, Korea University, Seoul, Republic of Korea
| | - Fan Zhang
- Research and Development Center, Canon Medical Systems (China) Co., Ltd, Beijing, China
| | - Bing Han
- Research and Development Center, Canon Medical Systems (China) Co., Ltd, Beijing, China
| | - David Joon Ho
- Department of Pathology, Memorial Sloan Kettering Cancer Center, New York, NY, USA
| | - Gyeong Hoon Kang
- Department of Pathology, Seoul National University Bundang Hospital, Seongnam, Republic of Korea; Laboratory of Epigenetics, Cancer Research Institute, Seoul National University College of Medicine, Republic of Korea.
| | - Se Young Chun
- Department of Electrical and Computer Engineering, INMC, Seoul National University, Seoul, Republic of Korea.
| | - Won-Ki Jeong
- Korea University, College of Informatics, Department of Computer Science and Engineering, Seoul, Republic of Korea.
| | | | - Jinwook Choi
- Department of Biomedical Engineering, Seoul National University College of Medicine, Seoul, Republic of Korea.
| |
Collapse
|
3
|
Irmakci I, Nateghi R, Zhou R, Ross AE, Yang XJ, Cooper LAD, Goldstein JA. Tissue contamination challenges the credibility of machine learning models in real world digital pathology. medRxiv 2023:2023.04.28.23289287. [PMID: 37205404 PMCID: PMC10187357 DOI: 10.1101/2023.04.28.23289287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Machine learning (ML) models are poised to transform surgical pathology practice. The most successful use attention mechanisms to examine whole slides, identify which areas of tissue are diagnostic, and use them to guide diagnosis. Tissue contaminants, such as floaters, represent unexpected tissue. While human pathologists are extensively trained to consider and detect tissue contaminants, we examined their impact on ML models. We trained 4 whole slide models. Three operate in placenta for 1) detection of decidual arteriopathy (DA), 2) estimation of gestational age (GA), and 3) classification of macroscopic placental lesions. We also developed a model to detect prostate cancer in needle biopsies. We designed experiments wherein patches of contaminant tissue are randomly sampled from known slides and digitally added to patient slides and measured model performance. We measured the proportion of attention given to contaminants and examined the impact of contaminants in T-distributed Stochastic Neighbor Embedding (tSNE) feature space. Every model showed performance degradation in response to one or more tissue contaminants. DA detection balanced accuracy decreased from 0.74 to 0.69 +/- 0.01 with addition of 1 patch of prostate tissue for every 100 patches of placenta (1% contaminant). Bladder, added at 10% contaminant raised the mean absolute error in estimating gestation age from 1.626 weeks to 2.371 +/ 0.003 weeks. Blood, incorporated into placental sections, induced false negative diagnoses of intervillous thrombi. Addition of bladder to prostate cancer needle biopsies induced false positives, a selection of high-attention patches, representing 0.033mm2, resulted in a 97% false positive rate when added to needle biopsies. Contaminant patches received attention at or above the rate of the average patch of patient tissue. Tissue contaminants induce errors in modern ML models. The high level of attention given to contaminants indicates a failure to encode biological phenomena. Practitioners should move to quantify and ameliorate this problem.
Collapse
Affiliation(s)
| | | | | | | | | | | | - Jeffery A. Goldstein
- To whom correspondence should be addressed: Olson 2-455, 710 N. Fairbanks Ave, Chicago IL, 60611,
| |
Collapse
|
4
|
Goldstein JA, Nateghi R, Irmakci I, Cooper LAD. Machine learning classification of placental villous infarction, perivillous fibrin deposition, and intervillous thrombus. Placenta 2023; 135:43-50. [PMID: 36958179 PMCID: PMC10156426 DOI: 10.1016/j.placenta.2023.03.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 12/02/2022] [Revised: 03/09/2023] [Accepted: 03/11/2023] [Indexed: 03/17/2023]
Abstract
INTRODUCTION Placental parenchymal lesions are commonly encountered and carry significant clinical associations. However, they are frequently missed or misclassified by general practice pathologists. Interpretation of pathology slides has emerged as one of the most successful applications of machine learning (ML) in medicine with applications ranging from cancer detection and prognostication to transplant medicine. The goal of this study was to use a whole-slide learning model to identify and classify placental parenchymal lesions including villous infarctions, intervillous thrombi (IVT), and perivillous fibrin deposition (PVFD). METHODS We generated whole slide images from placental discs examined at our institution with infarct, IVT, PVFD, or no macroscopic lesion. Slides were analyzed as a set of overlapping patches. We extracted feature vectors from each patch using a pretrained convolutional neural network (EfficientNetV2L). We trained a model to assign attention to each vector and used the attentions as weights to produce a pooled feature vector. The pooled vector was classified as normal or 1 of 3 lesions using a fully connected network. Patch attention was plotted to highlight informative areas of the slide. RESULTS Overall balanced accuracy in a test set of held-out slides was 0.86 with receiver-operator characteristic areas under the curve of 0.917-0.993. Cases of PVFD were frequently miscalled as normal or infarcts, the latter possibly due to the perivillous fibrin found at the periphery of infarctions. We used attention maps to further understand some errors, including one most likely due to poor tissue fixation and processing. DISCUSSION We used a whole-slide learning paradigm to train models to recognize three of the most common placental parenchymal lesions. We used attention maps to gain insight into model function, which differed from intuitive explanations.
Collapse
Affiliation(s)
| | - Ramin Nateghi
- Northwestern University, Department of Pathology, Chicago, IL, USA
| | - Ismail Irmakci
- Northwestern University, Department of Pathology, Chicago, IL, USA
| | - Lee A D Cooper
- Northwestern University, Department of Pathology, Chicago, IL, USA; Northwestern University, McCormick School of Engineering, Evanston, IL, USA
| |
Collapse
|
5
|
Aubreville M, Stathonikos N, Bertram CA, Klopfleisch R, Ter Hoeve N, Ciompi F, Wilm F, Marzahl C, Donovan TA, Maier A, Breen J, Ravikumar N, Chung Y, Park J, Nateghi R, Pourakpour F, Fick RHJ, Ben Hadj S, Jahanifar M, Shephard A, Dexl J, Wittenberg T, Kondo S, Lafarge MW, Koelzer VH, Liang J, Wang Y, Long X, Liu J, Razavi S, Khademi A, Yang S, Wang X, Erber R, Klang A, Lipnik K, Bolfa P, Dark MJ, Wasinger G, Veta M, Breininger K. Mitosis domain generalization in histopathology images - The MIDOG challenge. Med Image Anal 2023; 84:102699. [PMID: 36463832 DOI: 10.1016/j.media.2022.102699] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Revised: 10/28/2022] [Accepted: 11/17/2022] [Indexed: 11/27/2022]
Abstract
The density of mitotic figures (MF) within tumor tissue is known to be highly correlated with tumor proliferation and thus is an important marker in tumor grading. Recognition of MF by pathologists is subject to a strong inter-rater bias, limiting its prognostic value. State-of-the-art deep learning methods can support experts but have been observed to strongly deteriorate when applied in a different clinical environment. The variability caused by using different whole slide scanners has been identified as one decisive component in the underlying domain shift. The goal of the MICCAI MIDOG 2021 challenge was the creation of scanner-agnostic MF detection algorithms. The challenge used a training set of 200 cases, split across four scanning systems. As test set, an additional 100 cases split across four scanning systems, including two previously unseen scanners, were provided. In this paper, we evaluate and compare the approaches that were submitted to the challenge and identify methodological factors contributing to better performance. The winning algorithm yielded an F1 score of 0.748 (CI95: 0.704-0.781), exceeding the performance of six experts on the same task.
Collapse
Affiliation(s)
| | | | - Christof A Bertram
- Institute of Pathology, University of Veterinary Medicine, Vienna, Austria
| | - Robert Klopfleisch
- Institute of Veterinary Pathology, Freie Universität Berlin, Berlin, Germany
| | | | - Francesco Ciompi
- Computational Pathology Group, Radboud UMC, Nijmegen, The Netherlands
| | - Frauke Wilm
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany; Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Christian Marzahl
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Taryn A Donovan
- Department of Anatomic Pathology, Schwarzman Animal Medical Center, NY, USA
| | - Andreas Maier
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Jack Breen
- CISTIB Center for Computational Imaging and Simulation Technologies in Biomedicine, School of Computing, University of Leeds, Leeds, UK
| | - Nishant Ravikumar
- CISTIB Center for Computational Imaging and Simulation Technologies in Biomedicine, School of Computing, University of Leeds, Leeds, UK
| | - Youjin Chung
- Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Jinah Park
- Korea Advanced Institute of Science and Technology, Daejeon, South Korea
| | - Ramin Nateghi
- Electrical and Electronics Engineering Department, Shiraz University of Technology, Shiraz, Iran
| | - Fattaneh Pourakpour
- Iranian Brain Mapping Biobank (IBMB), National Brain Mapping Laboratory (NBML), Tehran, Iran
| | | | | | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Warwick, UK
| | - Adam Shephard
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Warwick, UK
| | - Jakob Dexl
- Fraunhofer-Institute for Integrated Circuits IIS, Erlangen, Germany
| | | | | | - Maxime W Lafarge
- Department of Pathology and Molecular Pathology, University Hospital and University of Zurich, Zurich, Switzerland
| | - Viktor H Koelzer
- Department of Pathology and Molecular Pathology, University Hospital and University of Zurich, Zurich, Switzerland
| | - Jingtang Liang
- School of Life Science and Technology, Xidian University, Shannxi, China
| | - Yubo Wang
- School of Life Science and Technology, Xidian University, Shannxi, China
| | - Xi Long
- Histo Pathology Diagnostic Center, Shanghai, China
| | - Jingxin Liu
- Xi'an Jiaotong-Liverpool University, Suzhou, China
| | - Salar Razavi
- Image Analysis in Medicine Lab (IAMLAB), Electrical, Computer and Biomedical Engineering, Ryerson University, Toronto, ON, Canada
| | - April Khademi
- Image Analysis in Medicine Lab (IAMLAB), Electrical, Computer and Biomedical Engineering, Ryerson University, Toronto, ON, Canada
| | - Sen Yang
- Tencent AI Lab, Shenzhen 518057, China
| | - Xiyue Wang
- College of Computer Science, Sichuan University, Chengdu 610065, China
| | - Ramona Erber
- Institute of Pathology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Andrea Klang
- Institute of Pathology, University of Veterinary Medicine, Vienna, Austria
| | - Karoline Lipnik
- Institute of Pathology, University of Veterinary Medicine, Vienna, Austria
| | - Pompei Bolfa
- Ross University School of Veterinary Medicine, Basseterre, Saint Kitts and Nevis
| | - Michael J Dark
- College of Veterinary Medicine, University of Florida, Gainesville, FL, USA
| | - Gabriel Wasinger
- Department of Pathology, General Hospital of Vienna, Medical University of Vienna, Vienna, Austria
| | - Mitko Veta
- Medical Image Analysis Group, TU Eindhoven, Eindhoven, The Netherlands
| | - Katharina Breininger
- Department Artificial Intelligence in Biomedical Engineering, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| |
Collapse
|
6
|
Nateghi R, Danyali H, Helfroush MS. A deep learning approach for mitosis detection: Application in tumor proliferation prediction from whole slide images. Artif Intell Med 2021; 114:102048. [PMID: 33875159 DOI: 10.1016/j.artmed.2021.102048] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2020] [Revised: 02/25/2021] [Accepted: 02/28/2021] [Indexed: 02/07/2023]
Abstract
The tumor proliferation, which is correlated with tumor grade, is a crucial biomarker indicative of breast cancer patients' prognosis. The most commonly used method in predicting tumor proliferation speed is the counting of mitotic figures in Hematoxylin and Eosin (H&E) histological slides. Manual mitosis counting is known to suffer from reproducibility problems. This paper presents a fully automated system for tumor proliferation prediction from whole slide images via mitosis counting. First, by considering the epithelial tissue as mitosis activity regions, we build a deep-learning-based region of interest detection method to select the high mitosis activity regions from whole slide images. Second, we learned a set of deep neural networks to detect mitosis detection from selected areas. The proposed mitosis detection system is designed to effectively overcome the mitosis detection challenges by two novel deep preprocessing and two-step hard negative mining approaches. Third, we trained a Support Vector Machine (SVM) classifier to predict the final tumor proliferation score. The proposed method was evaluated on the dataset of the Tumor Proliferation Assessment Challenge (TUPAC16) and achieved a 73.81 % F-measure and 0.612 weighted kappa score, respectively, outperforming all previous approaches significantly. Experimental results demonstrate that the proposed system considerably improves the tumor proliferation prediction accuracy and provides a reliable automated tool to support health care make-decisions.
Collapse
Affiliation(s)
- Ramin Nateghi
- Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran.
| | - Habibollah Danyali
- Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran.
| | - Mohammad Sadegh Helfroush
- Department of Electrical and Electronics Engineering, Shiraz University of Technology, Shiraz, Iran.
| |
Collapse
|
7
|
Fatehi M, Nateghi R, Pourakpour F. Automatic Bone Age Determination Using Wrist MRI Based on FIFA Grading System for Athletes: Deep Learning Approach. Semin Musculoskelet Radiol 2019. [DOI: 10.1055/s-0039-1692580] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/26/2022]
|