1
|
Dudgeon SN, Wen S, Hanna MG, Gupta R, Amgad M, Sheth M, Marble H, Huang R, Herrmann MD, Szu CH, Tong D, Werness B, Szu E, Larsimont D, Madabhushi A, Hytopoulos E, Chen W, Singh R, Hart SN, Sharma A, Saltz J, Salgado R, Gallas BD. A Pathologist-Annotated Dataset for Validating Artificial Intelligence: A Project Description and Pilot Study. J Pathol Inform 2021; 12:45. [PMID: 34881099 PMCID: PMC8609287 DOI: 10.4103/jpi.jpi_83_20] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2020] [Revised: 01/23/2021] [Accepted: 03/16/2021] [Indexed: 12/13/2022] Open
Abstract
Purpose: Validating artificial intelligence algorithms for clinical use in medical images is a challenging endeavor due to a lack of standard reference data (ground truth). This topic typically occupies a small portion of the discussion in research papers since most of the efforts are focused on developing novel algorithms. In this work, we present a collaboration to create a validation dataset of pathologist annotations for algorithms that process whole slide images. We focus on data collection and evaluation of algorithm performance in the context of estimating the density of stromal tumor-infiltrating lymphocytes (sTILs) in breast cancer. Methods: We digitized 64 glass slides of hematoxylin- and eosin-stained invasive ductal carcinoma core biopsies prepared at a single clinical site. A collaborating pathologist selected 10 regions of interest (ROIs) per slide for evaluation. We created training materials and workflows to crowdsource pathologist image annotations on two modes: an optical microscope and two digital platforms. The microscope platform allows the same ROIs to be evaluated in both modes. The workflows collect the ROI type, a decision on whether the ROI is appropriate for estimating the density of sTILs, and if appropriate, the sTIL density value for that ROI. Results: In total, 19 pathologists made 1645 ROI evaluations during a data collection event and the following 2 weeks. The pilot study yielded an abundant number of cases with nominal sTIL infiltration. Furthermore, we found that the sTIL densities are correlated within a case, and there is notable pathologist variability. Consequently, we outline plans to improve our ROI and case sampling methods. We also outline statistical methods to account for ROI correlations within a case and pathologist variability when validating an algorithm. Conclusion: We have built workflows for efficient data collection and tested them in a pilot study. As we prepare for pivotal studies, we will investigate methods to use the dataset as an external validation tool for algorithms. We will also consider what it will take for the dataset to be fit for a regulatory purpose: study size, patient population, and pathologist training and qualifications. To this end, we will elicit feedback from the Food and Drug Administration via the Medical Device Development Tool program and from the broader digital pathology and AI community. Ultimately, we intend to share the dataset, statistical methods, and lessons learned.
Collapse
Affiliation(s)
- Sarah N Dudgeon
- Division of Imaging Diagnostics and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
| | - Si Wen
- Division of Imaging Diagnostics and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
| | | | - Rajarsi Gupta
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Mohamed Amgad
- Department of Pathology, Northwestern University, Chicago, IL, USA
| | - Manasi Sheth
- Division of Biostatistics, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
| | - Hetal Marble
- Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | - Richard Huang
- Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | - Markus D Herrmann
- Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | | | | | | | - Evan Szu
- Arrive Bio, San Francisco, CA, USA
| | - Denis Larsimont
- Department of Pathology, Institute Jules Bordet, Brussels, Belgium
| | - Anant Madabhushi
- Louis Stokes Cleveland Veterans Administration Medical Center, Cleveland, OH, USA
| | | | - Weijie Chen
- Division of Imaging Diagnostics and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
| | - Rajendra Singh
- Northwell Health and Zucker School of Medicine, New York, NY, USA
| | - Steven N Hart
- Department of Pathology, Massachusetts General Hospital and Harvard Medical School, Boston, MA, USA
| | - Ashish Sharma
- Department of Biomedical Informatics, Emory University, Atlanta, GA, USA
| | - Joel Saltz
- Department of Biomedical Informatics, Stony Brook University, Stony Brook, NY, USA
| | - Roberto Salgado
- Division of Research, Peter Mac Callum Cancer Centre, Melbourne, Australia.,Department of Pathology, GZA-ZNA Hospitals, Antwerp, Belgium
| | - Brandon D Gallas
- Division of Imaging Diagnostics and Software Reliability, Office of Science and Engineering Laboratories, Center for Devices and Radiologic Health, United States Food and Drug Administration, White Oak, MD, USA
| |
Collapse
|
2
|
Leong TKM, Lo WS, Lee WEZ, Tan B, Lee XZ, Lee LWJN, Lee JYJ, Suresh N, Loo LH, Szu E, Yeong J. Leveraging advances in immunopathology and artificial intelligence to analyze in vitro tumor models in composition and space. Adv Drug Deliv Rev 2021; 177:113959. [PMID: 34481035 DOI: 10.1016/j.addr.2021.113959] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 08/17/2021] [Accepted: 08/30/2021] [Indexed: 12/12/2022]
Abstract
Cancer is the leading cause of death worldwide. Unfortunately, efforts to understand this disease are confounded by the complex, heterogenous tumor microenvironment (TME). Better understanding of the TME could lead to novel diagnostic, prognostic, and therapeutic discoveries. One way to achieve this involves in vitro tumor models that recapitulate the in vivo TME composition and spatial arrangement. Here, we review the potential of harnessing in vitro tumor models and artificial intelligence to delineate the TME. This includes (i) identification of novel features, (ii) investigation of higher-order relationships, and (iii) analysis and interpretation of multiomics data in a (iv) holistic, objective, reproducible, and efficient manner, which surpasses previous methods of TME analysis. We also discuss limitations of this approach, namely inadequate datasets, indeterminate biological correlations, ethical concerns, and logistical constraints; finally, we speculate on future avenues of research that could overcome these limitations, ultimately translating to improved clinical outcomes.
Collapse
|
3
|
Szu E, Osborne J, Patterson AD. Factual accuracy and the cultural context of science in popular media: Perspectives of media makers, middle school students, and university students on an entertainment television program. Public Underst Sci 2017; 26:596-611. [PMID: 27340172 DOI: 10.1177/0963662516655685] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/06/2023]
Abstract
Popular media influences ideas about science constructed by the public. To sway media productions, public policy organizations have increasingly promoted use of science consultants. This study contributes to understanding the connection from science consultants to popular media to public outcomes. A science-based television series was examined for intended messages of the creator and consulting scientist, and received messages among middle school and non-science university students. The results suggest the consulting scientist missed an opportunity to influence the portrayal of the cultural contexts of science and that middle school students may be reading these aspects uncritically-a deficiency educators could potentially address. In contrast, all groups discussed the science content and practices of the show, indicating that scientific facts were salient to both media makers and audiences. This suggests popular media may influence the public knowledge of science, supporting concerns of scientists about the accuracy of fictional television and film.
Collapse
|