1
|
Elfer K, Dudgeon S, Garcia V, Blenman K, Hytopoulos E, Wen S, Li X, Ly A, Werness B, Sheth MS, Amgad M, Gupta R, Saltz J, Hanna MG, Ehinger A, Peeters D, Salgado R, Gallas BD. Pilot study to evaluate tools to collect pathologist annotations for validating machine learning algorithms. J Med Imaging (Bellingham) 2022; 9:047501. [PMID: 35911208 PMCID: PMC9326105 DOI: 10.1117/1.jmi.9.4.047501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Accepted: 06/28/2022] [Indexed: 11/14/2022] Open
Abstract
Purpose: Validation of artificial intelligence (AI) algorithms in digital pathology with a reference standard is necessary before widespread clinical use, but few examples focus on creating a reference standard based on pathologist annotations. This work assesses the results of a pilot study that collects density estimates of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer biopsy specimens. This work will inform the creation of a validation dataset for the evaluation of AI algorithms fit for a regulatory purpose. Approach: Collaborators and crowdsourced pathologists contributed glass slides, digital images, and annotations. Here, "annotations" refer to any marks, segmentations, measurements, or labels a pathologist adds to a report, image, region of interest (ROI), or biological feature. Pathologists estimated sTILs density in 640 ROIs from hematoxylin and eosin stained slides of 64 patients via two modalities: an optical light microscope and two digital image viewing platforms. Results: The pilot study generated 7373 sTILs density estimates from 29 pathologists. Analysis of annotations found the variability of density estimates per ROI increases with the mean; the root mean square differences were 4.46, 14.25, and 26.25 as the mean density ranged from 0% to 10%, 11% to 40%, and 41% to 100%, respectively. The pilot study informs three areas of improvement for future work: technical workflows, annotation platforms, and agreement analysis methods. Upgrades to the workflows and platforms will improve operability and increase annotation speed and consistency. Conclusions: Exploratory data analysis demonstrates the need to develop new statistical approaches for agreement. The pilot study dataset and analysis methods are publicly available to allow community feedback. The development and results of the validation dataset will be publicly available to serve as an instructive tool that can be replicated by developers and researchers.
Collapse
Affiliation(s)
- Katherine Elfer
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics & Software Reliability, Silver Spring, Maryland, United States
- National Institutes of Health, National Cancer Institute, Division of Cancer Prevention, Cancer Prevention Fellowship Program, Bethesda, Maryland, United States
| | - Sarah Dudgeon
- Yale University Computational Biology and Bioinformatics, New Haven, Connecticut, United States
- Yale New Haven Hospital, Center for Outcomes Research and Evaluation, New Haven, Connecticut, United States
| | - Victor Garcia
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics & Software Reliability, Silver Spring, Maryland, United States
| | - Kim Blenman
- School of Medicine, Yale Cancer Center, Department of Internal Medicine, Section of Medical Oncology, New Haven, Connecticut, United States
- Yale University, School of Engineering and Applied Science, Department of Computer Science, New Haven, Connecticut, United States
| | | | - Si Wen
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics & Software Reliability, Silver Spring, Maryland, United States
| | - Xiaoxian Li
- Emory University School of Medicine, Department of Pathology and Laboratory Medicine, Atlanta, Georgia, United States
| | - Amy Ly
- Massachusetts General Hospital, Boston, Massachusetts, United States
| | - Bruce Werness
- Inova Health System Department of Pathology, Falls Church, Virginia, United States
- Arrive Bio LLC, San Francisco, California, United States
| | - Manasi S. Sheth
- United States Food and Drug Administration (FDA), Center for Devices and Radiologic Health, Office of Product Evaluation and Quality, Office of Clinical Evidence and Analysis, Division of Biostatistics, White Oak, Maryland, United States
| | - Mohamed Amgad
- Northwestern University Feinberg School of Medicine, Department of Pathology, Chicago, Illinois, United States
| | - Rajarsi Gupta
- SUNY Stony Brook Medicine, Department of Biomedical Informatics, Stony Brook, New York, United States
| | - Joel Saltz
- SUNY Stony Brook Medicine, Department of Biomedical Informatics, Stony Brook, New York, United States
- SUNY Stony Brook Medicine, Department of Pathology, Stony Brook, New York, United States
| | - Matthew G. Hanna
- Memorial Sloan Kettering Cancer Center, New York, New York, United States
| | - Anna Ehinger
- Lund University, Laboratory Medicine, Region Skåne, Department of Genetics and Pathology, Lund, Sweden
| | - Dieter Peeters
- Sint-Maarten Hospital, Department of Pathology, Mechelen, Belgium
- University of Antwerp, Department of Biomedical Sciences, Antwerp, Belgium
| | - Roberto Salgado
- Peter Mac Callum Cancer Centre, Division of Research, Melbourne, Australia
- GZA-ZNA Hospitals, Department of Pathology, Antwerp, Belgium
| | - Brandon D. Gallas
- United States Food and Drug Administration, Center for Devices and Radiological Health, Office of Science and Engineering Laboratories, Division of Imaging Diagnostics & Software Reliability, Silver Spring, Maryland, United States
- Address all correspondence to Brandon D. Gallas,
| |
Collapse
|