1
|
Raciti P, Sue J, Retamero JA, Ceballos R, Godrich R, Kunz JD, Casson A, Thiagarajan D, Ebrahimzadeh Z, Viret J, Lee D, Schüffler PJ, DeMuth G, Gulturk E, Kanan C, Rothrock B, Reis-Filho J, Klimstra DS, Reuter V, Fuchs TJ. Clinical Validation of Artificial Intelligence-Augmented Pathology Diagnosis Demonstrates Significant Gains in Diagnostic Accuracy in Prostate Cancer Detection. Arch Pathol Lab Med 2023; 147:1178-1185. [PMID: 36538386 DOI: 10.5858/arpa.2022-0066-oa] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/28/2022] [Indexed: 09/29/2023]
Abstract
CONTEXT.— Prostate cancer diagnosis rests on accurate assessment of tissue by a pathologist. The application of artificial intelligence (AI) to digitized whole slide images (WSIs) can aid pathologists in cancer diagnosis, but robust, diverse evidence in a simulated clinical setting is lacking. OBJECTIVE.— To compare the diagnostic accuracy of pathologists reading WSIs of prostatic biopsy specimens with and without AI assistance. DESIGN.— Eighteen pathologists, 2 of whom were genitourinary subspecialists, evaluated 610 prostate needle core biopsy WSIs prepared at 218 institutions, with the option for deferral. Two evaluations were performed sequentially for each WSI: initially without assistance, and immediately thereafter aided by Paige Prostate (PaPr), a deep learning-based system that provides a WSI-level binary classification of suspicious for cancer or benign and pinpoints the location that has the greatest probability of harboring cancer on suspicious WSIs. Pathologists' changes in sensitivity and specificity between the assisted and unassisted modalities were assessed, together with the impact of PaPr output on the assisted reads. RESULTS.— Using PaPr, pathologists improved their sensitivity and specificity across all histologic grades and tumor sizes. Accuracy gains on both benign and cancerous WSIs could be attributed to PaPr, which correctly classified 100% of the WSIs showing corrected diagnoses in the PaPr-assisted phase. CONCLUSIONS.— This study demonstrates the effectiveness and safety of an AI tool for pathologists in simulated diagnostic practice, bridging the gap between computational pathology research and its clinical application, and resulted in the first US Food and Drug Administration authorization of an AI system in pathology.
Collapse
Affiliation(s)
- Patricia Raciti
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Jillian Sue
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Juan A Retamero
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Rodrigo Ceballos
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Ran Godrich
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Jeremy D Kunz
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Adam Casson
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Dilip Thiagarajan
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Zahra Ebrahimzadeh
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Julian Viret
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Donghun Lee
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Peter J Schüffler
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | | | - Emre Gulturk
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Christopher Kanan
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Brandon Rothrock
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Jorge Reis-Filho
- The Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York (Reis-Filho, Reuter)
| | - David S Klimstra
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| | - Victor Reuter
- The Department of Pathology and Laboratory Medicine, Memorial Sloan Kettering Cancer Center, New York, New York (Reis-Filho, Reuter)
| | - Thomas J Fuchs
- From Paige (Raciti, Sue, Retamero, Ceballos, Godrich, Kunz, Casson, Thiagarajan, Ebrahimzadeh, Viret, Lee, Schüffler, Gulturk, Kanan, Rothrock, Klimstra, Fuchs), New York, New York
| |
Collapse
|
2
|
Reis-Filho JS, Pareja F, Derakhshan F, Brown DN, Sue J, Selenica P, Wang YK, Paula ADC, Banerjee M, Ebrahimzadeh Z, Isava M, Lee M, Godrich R, Casson A, Padron R, Shaikovski G, van Eck A, Marra A, Dopeso H, Wen HY, Brogi E, Hanna MG, Kanan C, Kunz JD, Geyer FC, Leibowitz C, Klimstra D, Grady L, Fuchs TJ. Abstract PD11-01: An artificial intelligence-based predictor of CDH1 biallelic mutations and invasive lobular carcinoma. Cancer Res 2022. [DOI: 10.1158/1538-7445.sabcs21-pd11-01] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Abstract
Introduction: Invasive lobular carcinoma (ILC) is the most frequent special histologic subtype of breast cancer (BC). ILC is identifiable by pathologic assessment given its distinctive discohesive growth pattern, largely caused by CDH1 inactivation. Compared to common forms of BC, ILCs display lower responses to chemotherapy and selective estrogen receptor modulators. The low interobserver agreement for the diagnosis of ILC, however, renders the inclusion of histologic subtyping in therapeutic decision-making challenging. Artificial intelligence (AI)-based algorithms hold promise for improving pathologic diagnosis; their performance, however, depends on the ground truth labeling used. Here, we seek to develop an AI-based methodology for detection of ILC using ‘CDH1 biallelic mutations’ (i.e., mutation + loss-of-heterozygosity of the wild-type allele or two pathogenic somatic mutations) as ground truth, reasoning that in BC, >95% of CDH1 bi-allelic inactivation is found in ILCs.Materials and methods: We developed a convolutional neural network system to detect CDH1 biallelic genetic inactivation (AI-CDH1) using whole slide images (WSI) of 1,100 primary BCs with available targeted sequencing data. The model was trained using a 10-fold cross-validation method to detect biallelic mutations. The mean number of positive and negative samples in the training set was 85.2 (SD=2.57) and 562.8 (SD=10.51) per fold, respectively. The evaluation set consisted of a mean of 14.2 (SD=2.04) positive and 93.8 (SD=9.13) negative samples. We evaluated the performance of the AI-CDH1 classifier to predict the lobular phenotype and CDH1 status using original and revised labels, following a histopathologic re-review of the histologic type and CDH1 status curation. The latter was conducted by incorporating information on biallelic CDH1 inactivation beyond CDH1 mutations (homozygous deletions, deleterious structural rearrangements, and loss-of-heterozygosity and gene promoter methylation).Results: The AI-CDH1 classifier predicted biallelic CDH1 mutations with an area under the curve (AUC)=0.944 (95 CI: 0.925-0.963), sensitivity=91.6% and specificity=85.9%, PPV=49.8%, NPV=98.5% and accuracy=86.7%, and the original ‘lobular phenotype’ with an AUC=0.941 (95 CI: 0.922-0.960), sensitivity=89%, specificity=86.7%, PPV=55.6%, NPV=97.7% and accuracy=87.1%. Review of the CDH1 gene status revealed that 7/957 BCs lacking CDH1 biallelic mutations harbored biallelic CDH1 inactivation by promoter methylation, homozygous deletions or structural rearrangements. The AI-CDH1 classifier detected all seven reclassified BCs and predicted the revised CDH1 biallelic inactivation with an AUC=0.948 (95 CI: 0.930-0.966), sensitivity=92%, specificity=86.5%, PPV=52.3%, NPV=98.5% and accuracy=87.2%. Upon histologic re-review, which resulted in reclassification of 36/927 non-lobular BCs as ‘lobular’ and 5/173 ‘lobular’ BCs as ‘non-lobular’, the AI-CDH1 classifier detected the ‘lobular phenotype’ with an AUC=0.953 (95 CI: 0.935-0.971), sensitivity=90.7%, specificity=89.7%, PPV=66.8%, NPV=97.7% and accuracy=89.9%. Using the revised histologic re-classification and CDH1 biallelic inactivation status labels, the AI-CDH1 classifier predicted the lobular phenotype irrespective of CDH1 status (P>0.05).Conclusions: By training a machine learning system to detect ‘CDH1 biallelic mutations’, as ground truth rather than histologic diagnosis of lobular carcinoma, which might be confounded by human subjectivity, we developed an AI-based system that can detect ILCs accurately, providing a new paradigm for the development of AI-based cancer classification systems.
Citation Format: Jorge S Reis-Filho, Fresia Pareja, Fatemeh Derakhshan, David N Brown, Jillian Sue, Pier Selenica, Yi Kan Wang, Arnaud Da Cruz Paula, Monami Banerjee, Zahra Ebrahimzadeh, Manuel Isava, Matthew Lee, Ran Godrich, Adam Casson, Ruben Padron, George Shaikovski, Alexander van Eck, Antonio Marra, Higinio Dopeso, Hannah Y Wen, Edi Brogi, Matthew G Hanna, Chris Kanan, Jeremy D Kunz, Felipe C Geyer, Carla Leibowitz, David Klimstra, Leo Grady, Thomas J Fuchs. An artificial intelligence-based predictor of CDH1 biallelic mutations and invasive lobular carcinoma [abstract]. In: Proceedings of the 2021 San Antonio Breast Cancer Symposium; 2021 Dec 7-10; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2022;82(4 Suppl):Abstract nr PD11-01.
Collapse
Affiliation(s)
| | - Fresia Pareja
- Memorial Sloan Kettering Cancer Center, New York, NY
| | | | - David N Brown
- Memorial Sloan Kettering Cancer Center, New York, NY
| | | | - Pier Selenica
- Memorial Sloan Kettering Cancer Center, New York, NY
| | | | | | | | | | | | | | | | | | | | | | | | - Antonio Marra
- Memorial Sloan Kettering Cancer Center, New York, NY
| | | | - Hannah Y Wen
- Memorial Sloan Kettering Cancer Center, New York, NY
| | - Edi Brogi
- Memorial Sloan Kettering Cancer Center, New York, NY
| | | | | | | | | | | | | | | | | |
Collapse
|