1
|
Rizzo PC, Girolami I, Marletta S, Pantanowitz L, Antonini P, Brunelli M, Santonicco N, Vacca P, Tumino N, Moretta L, Parwani A, Satturwar S, Eccher A, Munari E. Technical and Diagnostic Issues in Whole Slide Imaging Published Validation Studies. Front Oncol 2022; 12:918580. [PMID: 35785212 PMCID: PMC9246412 DOI: 10.3389/fonc.2022.918580] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Accepted: 05/24/2022] [Indexed: 01/07/2023] Open
Abstract
ObjectiveDigital pathology with whole-slide imaging (WSI) has many potential clinical and non-clinical applications. In the past two decades, despite significant advances in WSI technology adoption remains slow for primary diagnosis. The aim of this study was to identify common pitfalls of WSI reported in validation studies and offer measures to overcome these challenges.MethodsA systematic search was conducted in the electronic databases Pubmed-MEDLINE and Embase. Inclusion criteria were all validation studies designed to evaluate the feasibility of WSI for diagnostic clinical use in pathology. Technical and diagnostic problems encountered with WSI in these studies were recorded.ResultsA total of 45 studies were identified in which technical issues were reported in 15 (33%), diagnostic issues in 8 (18%), and 22 (49%) reported both. Key technical problems encompassed slide scan failure, prolonged time for pathologists to review cases, and a need for higher image resolution. Diagnostic challenges encountered were concerned with grading dysplasia, reliable assessment of mitoses, identification of microorganisms, and clearly defining the invasive front of tumors.ConclusionDespite technical advances with WSI technology, some critical concerns remain that need to be addressed to ensure trustworthy clinical diagnostic use. More focus on the quality of the pre-scanning phase and training of pathologists could help reduce the negative impact of WSI technical difficulties. WSI also seems to exacerbate specific diagnostic tasks that are already challenging among pathologists even when examining glass slides with conventional light microscopy.
Collapse
Affiliation(s)
- Paola Chiara Rizzo
- Department of Pathology and Diagnostics and Public Health, Section of Pathology, University Hospital of Verona, Verona, Italy
| | | | - Stefano Marletta
- Department of Pathology and Diagnostics and Public Health, Section of Pathology, University Hospital of Verona, Verona, Italy
- Department of Pathology, Pederzoli Hospital, Peschiera del Garda, Italy
| | - Liron Pantanowitz
- Department of Pathology & Clinical Labs, University of Michigan, Ann Arbor, MI, United States
| | - Pietro Antonini
- Department of Pathology and Diagnostics and Public Health, Section of Pathology, University Hospital of Verona, Verona, Italy
| | - Matteo Brunelli
- Department of Pathology and Diagnostics and Public Health, Section of Pathology, University Hospital of Verona, Verona, Italy
| | - Nicola Santonicco
- Department of Pathology and Diagnostics and Public Health, Section of Pathology, University Hospital of Verona, Verona, Italy
| | - Paola Vacca
- Bambino Gesù Children’s Hospital, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS), Rome, Italy
| | - Nicola Tumino
- Bambino Gesù Children’s Hospital, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS), Rome, Italy
| | - Lorenzo Moretta
- Bambino Gesù Children’s Hospital, Istituto di Ricovero e Cura a Carattere Scientifico (IRCCS), Rome, Italy
| | - Anil Parwani
- Department of Pathology, Ohio State University Medical Center, Columbus, OH, United States
| | - Swati Satturwar
- Department of Pathology, Ohio State University Medical Center, Columbus, OH, United States
| | - Albino Eccher
- Department of Pathology and Diagnostics, University and Hospital Trust of Verona, Verona, Italy
- *Correspondence: Albino Eccher,
| | - Enrico Munari
- Department of Molecular and Translational Medicine, University of Brescia, Brescia, Italy
| |
Collapse
|
2
|
Evans AJ, Brown RW, Bui MM, Chlipala EA, Lacchetti C, Milner DA, Pantanowitz L, Parwani AV, Reid K, Riben MW, Reuter VE, Stephens L, Stewart RL, Thomas NE. Validating Whole Slide Imaging Systems for Diagnostic Purposes in Pathology. Arch Pathol Lab Med 2022; 146:440-450. [PMID: 34003251 DOI: 10.5858/arpa.2020-0723-cp] [Citation(s) in RCA: 61] [Impact Index Per Article: 30.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 03/26/2021] [Indexed: 11/06/2022]
Abstract
CONTEXT.— The original guideline, "Validating Whole Slide Imaging for Diagnostic Purposes in Pathology," was published in 2013 and included 12 guideline statements. The College of American Pathologists convened an expert panel to update the guideline following standards established by the National Academies of Medicine for developing trustworthy clinical practice guidelines. OBJECTIVE.— To assess evidence published since the release of the original guideline and provide updated recommendations for validating whole slide imaging (WSI) systems used for diagnostic purposes. DESIGN.— An expert panel performed a systematic review of the literature. Frozen sections, anatomic pathology specimens (biopsies, curettings, and resections), and hematopathology cases were included. Cytology cases were excluded. Using the Grading of Recommendations Assessment, Development, and Evaluation approach, the panel reassessed and updated the original guideline recommendations. RESULTS.— Three strong recommendations and 9 good practice statements are offered to assist laboratories with validating WSI digital pathology systems. CONCLUSIONS.— Systematic review of literature following release of the 2013 guideline reaffirms the use of a validation set of at least 60 cases, establishing intraobserver diagnostic concordance between WSI and glass slides and the use of a 2-week washout period between modalities. Although all discordances between WSI and glass slide diagnoses discovered during validation need to be reconciled, laboratories should be particularly concerned if their overall WSI-glass slide concordance is less than 95%.
Collapse
Affiliation(s)
- Andrew J Evans
- From the Department of Pathology, Mackenzie Health, Richmond Hill, Ontario, Canada (Evans)
| | - Richard W Brown
- The Department of Pathology, Memorial Hermann Southwest Hospital, Houston, Texas (Brown)
| | - Marilyn M Bui
- The Department of Pathology, Moffitt Cancer Center, Tampa, Florida (Bui)
| | | | - Christina Lacchetti
- Policy and Advocacy, American Society of Clinical Oncology, Alexandria, Virginia (Lacchetti)
| | - Danny A Milner
- American Society for Clinical Pathology, Chicago, Illinois (Milner)
| | - Liron Pantanowitz
- The Department of Pathology, University of Michigan, Ann Arbor (Pantanowitz)
| | - Anil V Parwani
- The Department of Pathology, Ohio State University Medical Center, Columbus (Parwani)
| | | | - Michael W Riben
- The Department of Pathology, University of Texas MD Anderson Cancer Center, Houston (Riben)
| | - Victor E Reuter
- The Department of Pathology, Memorial Sloan-Kettering Cancer Center, New York, New York (Reuter)
| | - Lisa Stephens
- The Department of Anatomic Pathology, Cleveland Clinic, Cleveland, Ohio (Stephens)
| | - Rachel L Stewart
- Janssen Research & Development, Spring House, Pennsylvania (Stewart)
| | - Nicole E Thomas
- Surveys (Thomas), College of American Pathologists, Northfield, Illinois
| |
Collapse
|
3
|
Sturm B, Creytens D, Smits J, Ooms AHAG, Eijken E, Kurpershoek E, Küsters-Vandevelde HVN, Wauters C, Blokx WAM, van der Laak JAWM. Computer-Aided Assessment of Melanocytic Lesions by Means of a Mitosis Algorithm. Diagnostics (Basel) 2022; 12:diagnostics12020436. [PMID: 35204526 PMCID: PMC8871065 DOI: 10.3390/diagnostics12020436] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2021] [Revised: 12/31/2021] [Accepted: 01/14/2022] [Indexed: 11/16/2022] Open
Abstract
An increasing number of pathology laboratories are now fully digitised, using whole slide imaging (WSI) for routine diagnostics. WSI paves the road to use artificial intelligence (AI) that will play an increasing role in computer-aided diagnosis (CAD). In melanocytic skin lesions, the presence of a dermal mitosis may be an important clue for an intermediate or a malignant lesion and may indicate worse prognosis. In this study a mitosis algorithm primarily developed for breast carcinoma is applied to melanocytic skin lesions. This study aimed to assess whether the algorithm could be used in diagnosing melanocytic lesions, and to study the added value in diagnosing melanocytic lesions in a practical setting. WSI’s of a set of hematoxylin and eosin (H&E) stained slides of 99 melanocytic lesions (35 nevi, 4 intermediate melanocytic lesions, and 60 malignant melanomas, including 10 nevoid melanomas), for which a consensus diagnosis was reached by three academic pathologists, were subjected to a mitosis algorithm based on AI. Two academic and six general pathologists specialized in dermatopathology examined the WSI cases two times, first without mitosis annotations and after a washout period of at least 2 months with mitosis annotations based on the algorithm. The algorithm indicated true mitosis in lesional cells, i.e., melanocytes, and non-lesional cells, i.e., mainly keratinocytes and inflammatory cells. A high number of false positive mitosis was indicated as well, comprising melanin pigment, sebaceous glands nuclei, and spindle cell nuclei such as stromal cells and neuroid differentiated melanocytes. All but one pathologist reported more often a dermal mitosis with the mitosis algorithm, which on a regular basis, was incorrectly attributed to mitoses from mainly inflammatory cells. The overall concordance of the pathologists with the consensus diagnosis for all cases excluding nevoid melanoma (n = 89) appeared to be comparable with and without the use of AI (89% vs. 90%). However, the concordance increased by using AI in nevoid melanoma cases (n = 10) (75% vs. 68%). This study showed that in general cases, pathologists perform similarly with the aid of a mitosis algorithm developed primarily for breast cancer. In nevoid melanoma cases, pathologists perform better with the algorithm. From this study, it can be learned that pathologists need to be aware of potential pitfalls using CAD on H&E slides, e.g., misinterpreting dermal mitoses in non-melanotic cells.
Collapse
Affiliation(s)
- Bart Sturm
- Department of Pathology, Radboud University Medical Center, 6500 HB Nijmegen, The Netherlands;
- Pathan B.V., 3045 PM Rotterdam, The Netherlands; (J.S.); (A.H.A.G.O.); (E.K.)
| | - David Creytens
- Department of Pathology, Ghent University Hospital, 9000 Ghent, Belgium;
| | - Jan Smits
- Pathan B.V., 3045 PM Rotterdam, The Netherlands; (J.S.); (A.H.A.G.O.); (E.K.)
| | | | - Erik Eijken
- Laboratory for Pathology Oost Nederland (LabPON), 7550 AM Hengelo, The Netherlands;
| | - Eline Kurpershoek
- Pathan B.V., 3045 PM Rotterdam, The Netherlands; (J.S.); (A.H.A.G.O.); (E.K.)
| | | | - Carla Wauters
- Department of Pathology, Canisius Wilhelmina Hospital, 6500 GS Nijmegen, The Netherlands; (H.V.N.K.-V.); (C.W.)
| | - Willeke A. M. Blokx
- Division Laboratories, Pharmacy and Biomedical Genetics, University Medical Center Utrecht, 3508 GA Utrecht, The Netherlands;
| | - Jeroen A. W. M. van der Laak
- Department of Pathology, Radboud University Medical Center, 6500 HB Nijmegen, The Netherlands;
- Center for Medical Image Science and Visualization, Linköping University, 581 83 Linköping, Sweden
- Correspondence: ; Tel.: +31-638-814-869
| |
Collapse
|
4
|
Bertram CA, Stathonikos N, Donovan TA, Bartel A, Fuchs-Baumgartinger A, Lipnik K, van Diest PJ, Bonsembiante F, Klopfleisch R. Validation of digital microscopy: Review of validation methods and sources of bias. Vet Pathol 2021; 59:26-38. [PMID: 34433345 PMCID: PMC8761960 DOI: 10.1177/03009858211040476] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Digital microscopy (DM) is increasingly replacing traditional light microscopy (LM) for performing routine diagnostic and research work in human and veterinary pathology. The DM workflow encompasses specimen preparation, whole-slide image acquisition, slide retrieval, and the workstation, each of which has the potential (depending on the technical parameters) to introduce limitations and artifacts into microscopic examination by pathologists. Performing validation studies according to guidelines established in human pathology ensures that the best-practice approaches for patient care are not deteriorated by implementing DM. Whereas current publications on validation studies suggest an overall high reliability of DM, each laboratory is encouraged to perform an individual validation study to ensure that the DM workflow performs as expected in the respective clinical or research environment. With the exception of validation guidelines developed by the College of American Pathologists in 2013 and its update in 2021, there is no current review of the application of methods fundamental to validation. We highlight that there is high methodological variation between published validation studies, each having advantages and limitations. The diagnostic concordance rate between DM and LM is the most relevant outcome measure, which is influenced (regardless of the viewing modality used) by different sources of bias including complexity of the cases examined, diagnostic experience of the study pathologists, and case recall. Here, we review 3 general study designs used for previous publications on DM validation as well as different approaches for avoiding bias.
Collapse
Affiliation(s)
- Christof A Bertram
- University of Veterinary Medicine, Vienna, Austria.,Freie Universität Berlin, Berlin, Germany
| | | | | | | | | | | | | | | | | |
Collapse
|
5
|
Sanghvi AB, Allen EZ, Callenberg KM, Pantanowitz L. Performance of an artificial intelligence algorithm for reporting urine cytopathology. Cancer Cytopathol 2019; 127:658-666. [PMID: 31412169 DOI: 10.1002/cncy.22176] [Citation(s) in RCA: 50] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2019] [Revised: 04/29/2019] [Accepted: 05/21/2019] [Indexed: 11/06/2022]
Abstract
BACKGROUND Unlike Papanicolaou tests, there are no commercially available computer-assisted automated screening systems for urine specimens. Despite The Paris System for Reporting Urinary Cytology, there still is poor interobserver agreement with urine cytology and many cases in which a definitive diagnosis cannot be made. In the current study, the authors have reported on the development of an image algorithm that applies computational methods to digitized liquid-based urine cytology slides. METHODS A total of 2405 archival ThinPrep glass slides, including voided and instrumented urine cytology cases, were digitized. A deep learning computational pipeline with multiple tiers of convolutional neural network models was developed for processing whole slide images (WSIs) and predicting diagnoses. The algorithm was validated using a separate test data set comprised of consecutive cases encountered in routine clinical practice. RESULTS There were 1.9 million urothelial cells analyzed. An average of 5400 urothelial cells were identified in each WSI. The algorithm achieved an area under the curve of 0.88 (95% CI, 0.83-0.93). Using the optimal operating point, the algorithm's sensitivity was 79.5% (95% CI, 64.7%-90.2%) and the specificity was 84.5% (95% CI, 81.6%-87.1%) for high-grade urothelial carcinoma. CONCLUSIONS The authors successfully developed a computational algorithm capable of accurately analyzing WSIs of urine cytology cases. Compared with prior studies, this effort used a much larger data set, exploited whole slide-level and not just cell-level features, and used a cell gallery to display the algorithm's output for easy end-user review. This algorithm provides computer-assisted interpretation of urine cytology cases, akin to the machine learning technology currently used for automated Papanicolaou test screening.
Collapse
Affiliation(s)
| | | | | | - Liron Pantanowitz
- Department of Pathology, University of Pittsburgh Medical Center, Pittsburgh, Pennsylvania
| |
Collapse
|