1
|
Dwivedi K, Sharkey M, Alabed S, Langlotz CP, Swift AJ, Bluethgen C. External validation, radiological evaluation, and development of deep learning automatic lung segmentation in contrast-enhanced chest CT. Eur Radiol 2024; 34:2727-2737. [PMID: 37775589 PMCID: PMC10957646 DOI: 10.1007/s00330-023-10235-9] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 06/25/2023] [Accepted: 07/24/2023] [Indexed: 10/01/2023]
Abstract
OBJECTIVES There is a need for CT pulmonary angiography (CTPA) lung segmentation models. Clinical translation requires radiological evaluation of model outputs, understanding of limitations, and identification of failure points. This multicentre study aims to develop an accurate CTPA lung segmentation model, with evaluation of outputs in two diverse patient cohorts with pulmonary hypertension (PH) and interstitial lung disease (ILD). METHODS This retrospective study develops an nnU-Net-based segmentation model using data from two specialist centres (UK and USA). Model was trained (n = 37), tested (n = 12), and clinically evaluated (n = 176) on a diverse 'real-world' cohort of 225 PH patients with volumetric CTPAs. Dice score coefficient (DSC) and normalised surface distance (NSD) were used for testing. Clinical evaluation of outputs was performed by two radiologists who assessed clinical significance of errors. External validation was performed on heterogenous contrast and non-contrast scans from 28 ILD patients. RESULTS A total of 225 PH and 28 ILD patients with diverse demographic and clinical characteristics were evaluated. Mean accuracy, DSC, and NSD scores were 0.998 (95% CI 0.9976, 0.9989), 0.990 (0.9840, 0.9962), and 0.983 (0.9686, 0.9972) respectively. There were no segmentation failures. On radiological review, 82% and 71% of internal and external cases respectively had no errors. Eighteen percent and 25% respectively had clinically insignificant errors. Peripheral atelectasis and consolidation were common causes for suboptimal segmentation. One external case (0.5%) with patulous oesophagus had a clinically significant error. CONCLUSION State-of-the-art CTPA lung segmentation model provides accurate outputs with minimal clinical errors on evaluation across two diverse cohorts with PH and ILD. CLINICAL RELEVANCE Clinical translation of artificial intelligence models requires radiological review and understanding of model limitations. This study develops an externally validated state-of-the-art model with robust radiological review. Intended clinical use is in techniques such as lung volume or parenchymal disease quantification. KEY POINTS • Accurate, externally validated CT pulmonary angiography (CTPA) lung segmentation model tested in two large heterogeneous clinical cohorts (pulmonary hypertension and interstitial lung disease). • No segmentation failures and robust review of model outputs by radiologists found 1 (0.5%) clinically significant segmentation error. • Intended clinical use of this model is a necessary step in techniques such as lung volume, parenchymal disease quantification, or pulmonary vessel analysis.
Collapse
Affiliation(s)
- Krit Dwivedi
- Department of Infection, Immunity & Cardiovascular Disease, Medical School, University of Sheffield, Sheffield, UK.
- Academic Department of Radiology, Royal Hallamshire Hospital, Glossop Road, Sheffield, S10 2JF, USA.
| | - Michael Sharkey
- 3DLab, Sheffield Teaching Hospitals NHS Trust, Sheffield, UK
| | - Samer Alabed
- Department of Infection, Immunity & Cardiovascular Disease, Medical School, University of Sheffield, Sheffield, UK
| | - Curtis P Langlotz
- Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI), Stanford University, Sheffield, USA
| | - Andy J Swift
- Department of Infection, Immunity & Cardiovascular Disease, Medical School, University of Sheffield, Sheffield, UK
| | - Christian Bluethgen
- Stanford Center for Artificial Intelligence in Medicine and Imaging (AIMI), Stanford University, Sheffield, USA
| |
Collapse
|
2
|
Van Veen D, Van Uden C, Blankemeier L, Delbrouck JB, Aali A, Bluethgen C, Pareek A, Polacin M, Reis EP, Seehofnerová A, Rohatgi N, Hosamani P, Collins W, Ahuja N, Langlotz CP, Hom J, Gatidis S, Pauly J, Chaudhari AS. Adapted large language models can outperform medical experts in clinical text summarization. Nat Med 2024; 30:1134-1142. [PMID: 38413730 DOI: 10.1038/s41591-024-02855-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Accepted: 02/02/2024] [Indexed: 02/29/2024]
Abstract
Analyzing vast textual data and summarizing key information from electronic health records imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown promise in natural language processing (NLP) tasks, their effectiveness on a diverse range of clinical summarization tasks remains unproven. Here we applied adaptation methods to eight LLMs, spanning four distinct clinical summarization tasks: radiology reports, patient questions, progress notes and doctor-patient dialogue. Quantitative assessments with syntactic, semantic and conceptual NLP metrics reveal trade-offs between models and adaptation methods. A clinical reader study with 10 physicians evaluated summary completeness, correctness and conciseness; in most cases, summaries from our best-adapted LLMs were deemed either equivalent (45%) or superior (36%) compared with summaries from medical experts. The ensuing safety analysis highlights challenges faced by both LLMs and medical experts, as we connect errors to potential medical harm and categorize types of fabricated information. Our research provides evidence of LLMs outperforming medical experts in clinical text summarization across multiple tasks. This suggests that integrating LLMs into clinical workflows could alleviate documentation burden, allowing clinicians to focus more on patient care.
Collapse
Affiliation(s)
- Dave Van Veen
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA.
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA.
| | - Cara Van Uden
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Louis Blankemeier
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
| | - Jean-Benoit Delbrouck
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
| | - Asad Aali
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA
| | - Christian Bluethgen
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Diagnostic and Interventional Radiology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Anuj Pareek
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Copenhagen University Hospital, Copenhagen, Denmark
| | - Malgorzata Polacin
- Diagnostic and Interventional Radiology, University Hospital Zurich, University of Zurich, Zurich, Switzerland
| | - Eduardo Pontes Reis
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Albert Einstein Israelite Hospital, São Paulo, Brazil
| | - Anna Seehofnerová
- Department of Medicine, Stanford University, Stanford, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - Nidhi Rohatgi
- Department of Medicine, Stanford University, Stanford, CA, USA
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | - Poonam Hosamani
- Department of Medicine, Stanford University, Stanford, CA, USA
| | - William Collins
- Department of Medicine, Stanford University, Stanford, CA, USA
| | - Neera Ahuja
- Department of Medicine, Stanford University, Stanford, CA, USA
| | - Curtis P Langlotz
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Medicine, Stanford University, Stanford, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
| | - Jason Hom
- Department of Medicine, Stanford University, Stanford, CA, USA
| | - Sergios Gatidis
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - John Pauly
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Akshay S Chaudhari
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford University, Stanford, CA, USA
- Stanford Cardiovascular Institute, Stanford, CA, USA
| |
Collapse
|
3
|
Langlotz CP, Kahn CE. 2023 Manuscript Reviewers: A Note of Thanks. Radiol Artif Intell 2024; 6:e240138. [PMID: 38535965 PMCID: PMC10982905 DOI: 10.1148/ryai.240138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
|
4
|
Langlotz CP, Luker GD. 2023 Manuscript Reviewers: A Note of Thanks. Radiol Imaging Cancer 2024; 6:e240054. [PMID: 38488497 PMCID: PMC10988324 DOI: 10.1148/rycan.240054] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/19/2024]
|
5
|
Langlotz CP, Abbara S. 2023 Manuscript Reviewers: A Note of Thanks. Radiol Cardiothorac Imaging 2024; 6:e240046. [PMID: 38385760 PMCID: PMC10912858 DOI: 10.1148/ryct.240046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/23/2024]
|
6
|
Gefter WB, Prokop M, Seo JB, Raoof S, Langlotz CP, Hatabu H. Human-AI Symbiosis: A Path Forward to Improve Chest Radiography and the Role of Radiologists in Patient Care. Radiology 2024; 310:e232778. [PMID: 38259206 PMCID: PMC10831473 DOI: 10.1148/radiol.232778] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 12/08/2023] [Accepted: 12/18/2023] [Indexed: 01/24/2024]
Affiliation(s)
- Warren B. Gefter
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Mathias Prokop
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Joon Beom Seo
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Suhail Raoof
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Curtis P. Langlotz
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| | - Hiroto Hatabu
- From the Department of Radiology, Penn Medicine, University of Pennsylvania, Philadelphia, Pa (W.B.G.); Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (M.P.); Department of Radiology, Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea (J.B.S.); Department of Medicine and Radiology, Zucker School of Medicine, Hofstra/Northwell and Lung Institute, Lenox Hill Hospital, New York, NY (S.R.); Department of Radiology and Biomedical Informatics and Center for Artificial Intelligence in Medicine and Imaging, Stanford University, Palo Alto, Calif (C.P.L.); and Center for Pulmonary Functional Imaging, Department of Radiology, Brigham and Women’s Hospital and Harvard Medical School, 75 Francis St, Boston, MA 02215 (H.H.)
| |
Collapse
|
7
|
Youssef A, Ng MY, Long J, Hernandez-Boussard T, Shah N, Miner A, Larson D, Langlotz CP. Organizational Factors in Clinical Data Sharing for Artificial Intelligence in Health Care. JAMA Netw Open 2023; 6:e2348422. [PMID: 38113040 PMCID: PMC10731479 DOI: 10.1001/jamanetworkopen.2023.48422] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 08/29/2023] [Accepted: 11/03/2023] [Indexed: 12/21/2023] Open
Abstract
Importance Limited sharing of data sets that accurately represent disease and patient diversity limits the generalizability of artificial intelligence (AI) algorithms in health care. Objective To explore the factors associated with organizational motivation to share health data for AI development. Design, Setting, and Participants This qualitative study investigated organizational readiness for sharing health data across the academic, governmental, nonprofit, and private sectors. Using a multiple case studies approach, 27 semistructured interviews were conducted with leaders in data-sharing roles from August 29, 2022, to January 9, 2023. The interviews were conducted in the English language using a video conferencing platform. Using a purposive and nonprobabilistic sampling strategy, 78 individuals across 52 unique organizations were identified. Of these, 35 participants were enrolled. Participant recruitment concluded after 27 interviews, as theoretical saturation was reached and no additional themes emerged. Main Outcome and Measure Concepts defining organizational readiness for data sharing and the association between data-sharing factors and organizational behavior were mapped through iterative qualitative analysis to establish a framework defining organizational readiness for sharing clinical data for AI development. Results Interviews included 27 leaders from 18 organizations (academia: 10, government: 7, nonprofit: 8, and private: 2). Organizational readiness for data sharing centered around 2 main constructs: motivation and capabilities. Motivation related to the alignment of an organization's values with data-sharing priorities and was associated with its engagement in data-sharing efforts. However, organizational motivation could be modulated by extrinsic incentives for financial or reputational gains. Organizational capabilities comprised infrastructure, people, expertise, and access to data. Cross-sector collaboration was a key strategy to mitigate barriers to access health data. Conclusions and Relevance This qualitative study identified sector-specific factors that may affect the data-sharing behaviors of health organizations. External incentives may bolster cross-sector collaborations by helping overcome barriers to accessing health data for AI development. The findings suggest that tailored incentives may boost organizational motivation and facilitate sustainable flow of health data for AI development.
Collapse
Affiliation(s)
- Alaa Youssef
- Department of Radiology, Stanford University School of Medicine, Stanford, California
- Department of Medicine, Biomedical Informatics Research, Stanford University School of Medicine, California
| | - Madelena Y. Ng
- Department of Medicine, Biomedical Informatics Research, Stanford University School of Medicine, California
| | - Jin Long
- Department of Pediatrics, Stanford University School of Medicine, Stanford, California
| | - Tina Hernandez-Boussard
- Department of Medicine, Biomedical Informatics Research, Stanford University School of Medicine, California
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California
| | - Nigam Shah
- Department of Medicine, Biomedical Informatics Research, Stanford University School of Medicine, California
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California
| | - Adam Miner
- Department of Psychiatry, Stanford University School of Medicine, Stanford, California
| | - David Larson
- Department of Radiology, Stanford University School of Medicine, Stanford, California
| | - Curtis P. Langlotz
- Department of Radiology, Stanford University School of Medicine, Stanford, California
- Department of Medicine, Biomedical Informatics Research, Stanford University School of Medicine, California
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California
| |
Collapse
|
8
|
Ng MY, Youssef A, Miner AS, Sarellano D, Long J, Larson DB, Hernandez-Boussard T, Langlotz CP. Perceptions of Data Set Experts on Important Characteristics of Health Data Sets Ready for Machine Learning: A Qualitative Study. JAMA Netw Open 2023; 6:e2345892. [PMID: 38039004 PMCID: PMC10692863 DOI: 10.1001/jamanetworkopen.2023.45892] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 10/20/2023] [Indexed: 12/02/2023] Open
Abstract
Importance The lack of data quality frameworks to guide the development of artificial intelligence (AI)-ready data sets limits their usefulness for machine learning (ML) research in health care and hinders the diagnostic excellence of developed clinical AI applications for patient care. Objective To discern what constitutes high-quality and useful data sets for health and biomedical ML research purposes according to subject matter experts. Design, Setting, and Participants This qualitative study interviewed data set experts, particularly those who are creators and ML researchers. Semistructured interviews were conducted in English and remotely through a secure video conferencing platform between August 23, 2022, and January 5, 2023. A total of 93 experts were invited to participate. Twenty experts were enrolled and interviewed. Using purposive sampling, experts were affiliated with a diverse representation of 16 health data sets/databases across organizational sectors. Content analysis was used to evaluate survey information and thematic analysis was used to analyze interview data. Main Outcomes and Measures Data set experts' perceptions on what makes data sets AI ready. Results Participants included 20 data set experts (11 [55%] men; mean [SD] age, 42 [11] years), of whom all were health data set creators, and 18 of the 20 were also ML researchers. Themes (3 main and 11 subthemes) were identified and integrated into an AI-readiness framework to show their association within the health data ecosystem. Participants partially determined the AI readiness of data sets using priority appraisal elements of accuracy, completeness, consistency, and fitness. Ethical acquisition and societal impact emerged as appraisal considerations in that participant samples have not been described to date in prior data quality frameworks. Factors that drive creation of high-quality health data sets and mitigate risks associated with data reuse in ML research were also relevant to AI readiness. The state of data availability, data quality standards, documentation, team science, and incentivization were associated with elements of AI readiness and the overall perception of data set usefulness. Conclusions and Relevance In this qualitative study of data set experts, participants contributed to the development of a grounded framework for AI data set quality. Data set AI readiness required the concerted appraisal of many elements and the balancing of transparency and ethical reflection against pragmatic constraints. The movement toward more reliable, relevant, and ethical AI and ML applications for patient care will inevitably require strategic updates to data set creation practices.
Collapse
Affiliation(s)
- Madelena Y. Ng
- Department of Medicine (Biomedical Informatics), Stanford University School of Medicine, Stanford, California
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California
| | - Alaa Youssef
- Department of Radiology, Stanford University School of Medicine, Stanford, California
| | - Adam S. Miner
- Department of Psychiatry and Behavioral Sciences, Stanford University School of Medicine, Stanford, California
| | - Daniela Sarellano
- Department of Radiology, Stanford University School of Medicine, Stanford, California
| | - Jin Long
- Department of Pediatrics, Stanford University School of Medicine, Stanford, California
| | - David B. Larson
- Department of Radiology, Stanford University School of Medicine, Stanford, California
| | - Tina Hernandez-Boussard
- Department of Medicine (Biomedical Informatics), Stanford University School of Medicine, Stanford, California
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California
| | - Curtis P. Langlotz
- Department of Medicine (Biomedical Informatics), Stanford University School of Medicine, Stanford, California
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, California
- Department of Radiology, Stanford University School of Medicine, Stanford, California
| |
Collapse
|
9
|
Van Veen D, Van Uden C, Blankemeier L, Delbrouck JB, Aali A, Bluethgen C, Pareek A, Polacin M, Reis EP, Seehofnerová A, Rohatgi N, Hosamani P, Collins W, Ahuja N, Langlotz CP, Hom J, Gatidis S, Pauly J, Chaudhari AS. Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts. Res Sq 2023:rs.3.rs-3483777. [PMID: 37961377 PMCID: PMC10635391 DOI: 10.21203/rs.3.rs-3483777/v1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2023]
Abstract
Sifting through vast textual data and summarizing key information from electronic health records (EHR) imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown immense promise in natural language processing (NLP) tasks, their efficacy on a diverse range of clinical summarization tasks has not yet been rigorously demonstrated. In this work, we apply domain adaptation methods to eight LLMs, spanning six datasets and four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Our thorough quantitative assessment reveals trade-offs between models and adaptation methods in addition to instances where recent advances in LLMs may not improve results. Further, in a clinical reader study with ten physicians, we show that summaries from our best-adapted LLMs are preferable to human summaries in terms of completeness and correctness. Our ensuing qualitative analysis highlights challenges faced by both LLMs and human experts. Lastly, we correlate traditional quantitative NLP metrics with reader study scores to enhance our understanding of how these metrics align with physician preferences. Our research marks the first evidence of LLMs outperforming human experts in clinical text summarization across multiple tasks. This implies that integrating LLMs into clinical workflows could alleviate documentation burden, empowering clinicians to focus more on personalized patient care and the inherently human aspects of medicine.
Collapse
Affiliation(s)
- Dave Van Veen
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
| | - Cara Van Uden
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Computer Science, Stanford University, Stanford, CA, USA
| | - Louis Blankemeier
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
| | - Jean-Benoit Delbrouck
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
| | - Asad Aali
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA
| | - Christian Bluethgen
- Department of Medicine, Stanford, CA, USA
- University Hospital Zurich, Zurich, Switzerland
| | - Anuj Pareek
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Copenhagen University Hospital, Copenhagen, Denmark
| | - Malgorzata Polacin
- Department of Medicine, Stanford, CA, USA
- University Hospital Zurich, Zurich, Switzerland
| | - Eduardo Pontes Reis
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Albert Einstein Israelite Hospital, São Paulo, Brazil
| | - Anna Seehofnerová
- Department of Medicine, Stanford, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - Nidhi Rohatgi
- Department of Medicine, Stanford, CA, USA
- Department of Neurosurgery, Stanford University, Stanford, CA, USA
| | | | | | | | - Curtis P. Langlotz
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Medicine, Stanford, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford, CA, USA
| | - Jason Hom
- Department of Medicine, Stanford, CA, USA
| | - Sergios Gatidis
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - John Pauly
- Department of Electrical Engineering, Stanford University, Stanford, CA, USA
| | - Akshay S. Chaudhari
- Stanford Center for Artificial Intelligence in Medicine and Imaging, Palo Alto, CA, USA
- Department of Radiology, Stanford University, Stanford, CA, USA
- Department of Biomedical Data Science, Stanford, CA, USA
| |
Collapse
|
10
|
Affiliation(s)
- Curtis P. Langlotz
- From the Departments of Radiology, Medicine, and Biomedical Data
Science, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA
94305
| |
Collapse
|
11
|
Callahan A, Ashley E, Datta S, Desai P, Ferris TA, Fries JA, Halaas M, Langlotz CP, Mackey S, Posada JD, Pfeffer MA, Shah NH. The Stanford Medicine data science ecosystem for clinical and translational research. JAMIA Open 2023; 6:ooad054. [PMID: 37545984 PMCID: PMC10397535 DOI: 10.1093/jamiaopen/ooad054] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 03/14/2023] [Accepted: 07/19/2023] [Indexed: 08/08/2023] Open
Abstract
Objective To describe the infrastructure, tools, and services developed at Stanford Medicine to maintain its data science ecosystem and research patient data repository for clinical and translational research. Materials and Methods The data science ecosystem, dubbed the Stanford Data Science Resources (SDSR), includes infrastructure and tools to create, search, retrieve, and analyze patient data, as well as services for data deidentification, linkage, and processing to extract high-value information from healthcare IT systems. Data are made available via self-service and concierge access, on HIPAA compliant secure computing infrastructure supported by in-depth user training. Results The Stanford Medicine Research Data Repository (STARR) functions as the SDSR data integration point, and includes electronic medical records, clinical images, text, bedside monitoring data and HL7 messages. SDSR tools include tools for electronic phenotyping, cohort building, and a search engine for patient timelines. The SDSR supports patient data collection, reproducible research, and teaching using healthcare data, and facilitates industry collaborations and large-scale observational studies. Discussion Research patient data repositories and their underlying data science infrastructure are essential to realizing a learning health system and advancing the mission of academic medical centers. Challenges to maintaining the SDSR include ensuring sufficient financial support while providing researchers and clinicians with maximal access to data and digital infrastructure, balancing tool development with user training, and supporting the diverse needs of users. Conclusion Our experience maintaining the SDSR offers a case study for academic medical centers developing data science and research informatics infrastructure.
Collapse
Affiliation(s)
- Alison Callahan
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Euan Ashley
- Department of Medicine, School of Medicine, Stanford University, Stanford, California, USA
- Department of Genetics, School of Medicine, Stanford University, Stanford, California, USA
- Department of Biomedical Data Science, School of Medicine, Stanford University, Stanford, California, USA
| | - Somalee Datta
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Priyamvada Desai
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Todd A Ferris
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Jason A Fries
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
| | - Michael Halaas
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Curtis P Langlotz
- Department of Radiology, School of Medicine, Stanford University, Stanford, California, USA
| | - Sean Mackey
- Department of Anesthesia, School of Medicine, Stanford University, Stanford, California, USA
| | - José D Posada
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Michael A Pfeffer
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
| | - Nigam H Shah
- Stanford Center for Biomedical Informatics Research, Stanford University, Stanford, California, USA
- Technology and Digital Solutions, Stanford Medicine, Stanford University, Stanford, California, USA
- Clinical Excellence Research Center, School of Medicine, Stanford University, Stanford, California, USA
| |
Collapse
|
12
|
Yu F, Endo M, Krishnan R, Pan I, Tsai A, Reis EP, Fonseca EKUN, Lee HMH, Abad ZSH, Ng AY, Langlotz CP, Venugopal VK, Rajpurkar P. Evaluating progress in automatic chest X-ray radiology report generation. Patterns (N Y) 2023; 4:100802. [PMID: 37720336 PMCID: PMC10499844 DOI: 10.1016/j.patter.2023.100802] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Revised: 02/03/2023] [Accepted: 06/29/2023] [Indexed: 09/19/2023]
Abstract
Artificial intelligence (AI) models for automatic generation of narrative radiology reports from images have the potential to enhance efficiency and reduce the workload of radiologists. However, evaluating the correctness of these reports requires metrics that can capture clinically pertinent differences. In this study, we investigate the alignment between automated metrics and radiologists' scoring of errors in report generation. We address the limitations of existing metrics by proposing new metrics, RadGraph F1 and RadCliQ, which demonstrate stronger correlation with radiologists' evaluations. In addition, we analyze the failure modes of the metrics to understand their limitations and provide guidance for metric selection and interpretation. This study establishes RadGraph F1 and RadCliQ as meaningful metrics for guiding future research in radiology report generation.
Collapse
Affiliation(s)
- Feiyang Yu
- Department of Computer Science, Stanford University, Stanford, CA 94305, USA
| | - Mark Endo
- Department of Computer Science, Stanford University, Stanford, CA 94305, USA
| | - Rayan Krishnan
- Department of Computer Science, Stanford University, Stanford, CA 94305, USA
| | - Ian Pan
- Department of Radiology, Brigham and Women’s Hospital, Boston, MA 02115, USA
| | - Andy Tsai
- Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, MA 02115, USA
| | - Eduardo Pontes Reis
- Cardiothoracic Radiology Group, Hospital Israelita Albert Einstein, São Paulo, São Paulo 05652, Brazil
| | | | - Henrique Min Ho Lee
- Cardiothoracic Radiology Group, Hospital Israelita Albert Einstein, São Paulo, São Paulo 05652, Brazil
| | | | - Andrew Y. Ng
- Department of Computer Science, Stanford University, Stanford, CA 94305, USA
| | | | | | - Pranav Rajpurkar
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA 02115, USA
| |
Collapse
|
13
|
Langlotz CP, Mauro MA, Mahmood U, Klein JS, Meltzer CC, Bhalla S, Heller RE, Scott JA, Flanders AE, Pandharipande PV. Truth and Transformation: RSNA's Journey Toward Equity. Radiol Cardiothorac Imaging 2023; 5:e239001. [PMID: 37124648 PMCID: PMC10141329 DOI: 10.1148/ryct.239001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/02/2023]
|
14
|
Langlotz CP, Mauro MA, Mahmood U, Klein JS, Meltzer CC, Bhalla S, Heller RE, Scott JA, Flanders AE, Pandharipande PV. Truth and Transformation: RSNA's Journey Toward Equity. Radiographics 2023; 43:e239005. [PMID: 36862085 DOI: 10.1148/rg.239005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/03/2023]
|
15
|
Langlotz CP, Mauro MA, Mahmood U, Klein JS, Meltzer CC, Bhalla S, Heller RE, Scott JA, Flanders AE, Pandharipande PV. Truth and Transformation: RSNA's Journey Toward Equity. Radiology 2023; 307:e239008. [PMID: 36862088 DOI: 10.1148/radiol.239008] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/03/2023]
|
16
|
Langlotz CP, Mauro MA, Mahmood U, Klein JS, Meltzer CC, Bhalla S, Heller RE, Scott JA, Flanders AE, Pandharipande PV. Truth and Transformation: RSNA's Journey Toward Equity. Radiol Artif Intell 2023; 5:e239001. [PMID: 37035432 PMCID: PMC10077070 DOI: 10.1148/ryai.239001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/06/2023]
|
17
|
Langlotz CP, Mauro MA, Mahmood U, Klein JS, Meltzer CC, Bhalla S, Heller RE, Scott JA, Flanders AE, Pandharipande PV. Truth and Transformation: RSNA's Journey Toward Equity. Radiol Imaging Cancer 2023; 5:e239005. [PMID: 36862089 PMCID: PMC10077064 DOI: 10.1148/rycan.239005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/03/2023]
|
18
|
Chambon PJ, Wu C, Steinkamp JM, Adleberg J, Cook TS, Langlotz CP. Automated deidentification of radiology reports combining transformer and "hide in plain sight" rule-based methods. J Am Med Inform Assoc 2023; 30:318-328. [PMID: 36416419 PMCID: PMC9846681 DOI: 10.1093/jamia/ocac219] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2022] [Revised: 10/11/2022] [Accepted: 11/09/2022] [Indexed: 11/24/2022] Open
Abstract
OBJECTIVE To develop an automated deidentification pipeline for radiology reports that detect protected health information (PHI) entities and replaces them with realistic surrogates "hiding in plain sight." MATERIALS AND METHODS In this retrospective study, 999 chest X-ray and CT reports collected between November 2019 and November 2020 were annotated for PHI at the token level and combined with 3001 X-rays and 2193 medical notes previously labeled, forming a large multi-institutional and cross-domain dataset of 6193 documents. Two radiology test sets, from a known and a new institution, as well as i2b2 2006 and 2014 test sets, served as an evaluation set to estimate model performance and to compare it with previously released deidentification tools. Several PHI detection models were developed based on different training datasets, fine-tuning approaches and data augmentation techniques, and a synthetic PHI generation algorithm. These models were compared using metrics such as precision, recall and F1 score, as well as paired samples Wilcoxon tests. RESULTS Our best PHI detection model achieves 97.9 F1 score on radiology reports from a known institution, 99.6 from a new institution, 99.5 on i2b2 2006, and 98.9 on i2b2 2014. On reports from a known institution, it achieves 99.1 recall of detecting the core of each PHI span. DISCUSSION Our model outperforms all deidentifiers it was compared to on all test sets as well as human labelers on i2b2 2014 data. It enables accurate and automatic deidentification of radiology reports. CONCLUSIONS A transformer-based deidentification pipeline can achieve state-of-the-art performance for deidentifying radiology reports and other medical documents.
Collapse
Affiliation(s)
- Pierre J Chambon
- Department of Radiology, Stanford University, Stanford, California, USA
- Department of Applied Mathematics and Engineering, Paris-Saclay University, Ecole Centrale Paris, Paris, France
| | - Christopher Wu
- Department of Radiology, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Jackson M Steinkamp
- Department of Radiology, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Jason Adleberg
- Department of Radiology, Mount Sinai Health System, New York, New York, USA
| | - Tessa S Cook
- Department of Radiology, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Curtis P Langlotz
- Department of Radiology, Stanford University, Stanford, California, USA
| |
Collapse
|
19
|
Daye D, Wiggins WF, Lungren MP, Alkasab T, Kottler N, Allen B, Roth CJ, Bizzo BC, Durniak K, Brink JA, Larson DB, Dreyer KJ, Langlotz CP. Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How? Radiology 2022; 305:555-563. [PMID: 35916673 PMCID: PMC9713445 DOI: 10.1148/radiol.212151] [Citation(s) in RCA: 38] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2021] [Revised: 03/30/2022] [Accepted: 04/12/2022] [Indexed: 01/03/2023]
Abstract
As the role of artificial intelligence (AI) in clinical practice evolves, governance structures oversee the implementation, maintenance, and monitoring of clinical AI algorithms to enhance quality, manage resources, and ensure patient safety. In this article, a framework is established for the infrastructure required for clinical AI implementation and presents a road map for governance. The road map answers four key questions: Who decides which tools to implement? What factors should be considered when assessing an application for implementation? How should applications be implemented in clinical practice? Finally, how should tools be monitored and maintained after clinical implementation? Among the many challenges for the implementation of AI in clinical practice, devising flexible governance structures that can quickly adapt to a changing environment will be essential to ensure quality patient care and practice improvement objectives.
Collapse
Affiliation(s)
- Dania Daye
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Walter F. Wiggins
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Matthew P. Lungren
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Tarik Alkasab
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Nina Kottler
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Bibb Allen
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Christopher J. Roth
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Bernardo C. Bizzo
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - Kimberly Durniak
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - James A. Brink
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | - David B. Larson
- From the Department of Radiology, Massachusetts General Hospital,
Harvard Medical School, 55 Fruit St, GRB 297, Boston, MA 02155 (D.D., T.A.,
B.C.B., K.D., J.A.B., K.J.D.); Department of Radiology, Duke University, Durham,
NC (W.F.W., C.J.R.); Department of Radiology, Stanford University, Stanford,
Calif (M.P.L., D.B.L., C.P.L.); Radiology Partners, El Segundo, Calif (N.K.);
and Department of Radiology, Grandview Medical Center, Birmingham, Ala
(B.A.)
| | | | | |
Collapse
|
20
|
Daye D, Wiggins WF, Lungren MP, Alkasab T, Kottler N, Allen B, Roth CJ, Bizzo BC, Durniak K, Brink JA, Larson DB, Dreyer KJ, Langlotz CP. Implementation of Clinical Artificial Intelligence in Radiology: Who Decides and How? Radiology 2022; 305:E62. [PMID: 36154286 DOI: 10.1148/radiol.229021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
21
|
McGrath AL, McGinty G, Berg WA, Mendelson EB, Drotman MB, Ellis RL, Langlotz CP. Optimizing the Breast Imaging Report for Today and Tomorrow. J Breast Imaging 2022; 4:343-345. [PMID: 38416981 DOI: 10.1093/jbi/wbac033] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/28/2021] [Indexed: 03/01/2024]
Affiliation(s)
- Anika L McGrath
- Weill Cornell Medicine at New York-Presbyterian, Department of Radiology, New York, NY, USA
| | - Geraldine McGinty
- Weill Cornell Medicine at New York-Presbyterian, Department of Radiology, New York, NY, USA
| | - Wendie A Berg
- Magee-Womens Hospital of University of Pittsburgh School of Medicine, Department of Radiology, Pittsburgh, PA, USA
| | - Ellen B Mendelson
- Feinberg School of Medicine Northwestern at University, Department of Radiology, Chicago, IL, USA
| | - Michele B Drotman
- Weill Cornell Medicine at New York-Presbyterian, Department of Radiology, New York, NY, USA
| | - Richard L Ellis
- Mayo Clinic Health System, Department of Radiology, La Crosse, WI, USA
| | - Curtis P Langlotz
- Stanford University School of Medicine, Department of Radiology, Stanford, CA, USA
| |
Collapse
|
22
|
Larson DB, Krishnaraj A, Mendelson DS, Langlotz CP, Wald C. Moving Toward Seamless Interinstitutional Electronic Image Transfer. J Am Coll Radiol 2022; 19:460-468. [PMID: 35114138 DOI: 10.1016/j.jacr.2021.11.017] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2021] [Revised: 11/12/2021] [Accepted: 11/17/2021] [Indexed: 11/25/2022]
Abstract
The fact that medical images are still predominately exchanged between institutions via physical media is unacceptable in the era of value-driven health care. Although better solutions are technically possible, problems of coordination and market dynamics may be inhibiting progress more than technical factors. We provide a macrosystem analysis of the problem of interinstitutional medical image exchange and propose a strategy for nudging the market toward a patient-friendly solution. The system can be viewed as a network, with autonomous nodes interconnected by links through which information is exchanged. A variety of potential network configurations include those that depend on individual carriers, peer-to-peer links, one or multiple hubs, or a hybrid of models. We find the linked multihub model, in which individual institutions are connected to other institutions via image exchange companies, to be the configuration most likely to create a patient-friendly electronic image exchange system. To achieve this configuration, image exchange companies, which operate in a competitive marketplace, must exchange images with each other. We call on these vendors to immediately commit to coordinating in this manner. We call on all other stakeholders, including medical societies, payers, and regulators, to actively encourage and facilitate this behavior. Specifically, we call on institutions to create appropriate market incentives by only contracting with image exchange vendors who are committed to begin vendor-to-vendor image exchange by no later than 2024.
Collapse
Affiliation(s)
- David B Larson
- Chair, Commission on Quality and Safety, ACR; Member, Board of Chancellors, ACR; and Vice Chair, Education and Clinical Operations, Department of Radiology, Stanford University School of Medicine, Stanford, California.
| | - Arun Krishnaraj
- Chair, Commission on Patient- and Family-Centered Care, ACR; Member, Board of Chancellors, ACR; and Chief, Division of Body Imaging, Department of Radiology and Medical Imaging, University of Virginia School of Medicine, Charlottesville, Virginia
| | - David S Mendelson
- Vice Chair, Informatics, Department of Radiology, The Mount Sinai Medical Center, New York, New York
| | - Curtis P Langlotz
- Member, Board of Directors, RSNA, and Associate Chair, Information Systems, Department of Radiology, Stanford University School of Medicine, Stanford, California
| | - Christoph Wald
- Chair, Commission on Informatics, ACR; Member, Board of Chancellors, ACR; and Chair, Department of Radiology, Lahey Hospital and Medical Center, Burlington, Massachusetts; Tufts University Medical School, Boston, Massachusetts
| |
Collapse
|
23
|
Tiu E, Talius E, Patel P, Langlotz CP, Ng AY, Rajpurkar P. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat Biomed Eng 2022; 6:1399-1406. [PMID: 36109605 PMCID: PMC9792370 DOI: 10.1038/s41551-022-00936-9] [Citation(s) in RCA: 33] [Impact Index Per Article: 16.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2021] [Accepted: 08/07/2022] [Indexed: 01/14/2023]
Abstract
In tasks involving the interpretation of medical images, suitably trained machine-learning models often exceed the performance of medical experts. Yet such a high-level of performance typically requires that the models be trained with relevant datasets that have been painstakingly annotated by experts. Here we show that a self-supervised model trained on chest X-ray images that lack explicit annotations performs pathology-classification tasks with accuracies comparable to those of radiologists. On an external validation dataset of chest X-rays, the self-supervised model outperformed a fully supervised model in the detection of three pathologies (out of eight), and the performance generalized to pathologies that were not explicitly annotated for model training, to multiple image-interpretation tasks and to datasets from multiple institutions.
Collapse
Affiliation(s)
- Ekin Tiu
- grid.168010.e0000000419368956Stanford University Department of Computer Science, Stanford, CA USA ,grid.38142.3c000000041936754XDepartment of Biomedical Informatics, Harvard University, Boston, MA USA
| | - Ellie Talius
- grid.168010.e0000000419368956Stanford University Department of Computer Science, Stanford, CA USA ,grid.38142.3c000000041936754XDepartment of Biomedical Informatics, Harvard University, Boston, MA USA
| | - Pujan Patel
- grid.168010.e0000000419368956Stanford University Department of Computer Science, Stanford, CA USA ,grid.38142.3c000000041936754XDepartment of Biomedical Informatics, Harvard University, Boston, MA USA
| | - Curtis P. Langlotz
- grid.168010.e0000000419368956AIMI Center, Stanford University, Palo Alto, CA USA
| | - Andrew Y. Ng
- grid.168010.e0000000419368956Stanford University Department of Computer Science, Stanford, CA USA
| | - Pranav Rajpurkar
- grid.38142.3c000000041936754XDepartment of Biomedical Informatics, Harvard University, Boston, MA USA
| |
Collapse
|
24
|
Eng DK, Khandwala NB, Long J, Fefferman NR, Lala SV, Strubel NA, Milla SS, Filice RW, Sharp SE, Towbin AJ, Francavilla ML, Kaplan SL, Ecklund K, Prabhu SP, Dillon BJ, Everist BM, Anton CG, Bittman ME, Dennis R, Larson DB, Seekins JM, Silva CT, Zandieh AR, Langlotz CP, Lungren MP, Halabi SS. Artificial Intelligence Algorithm Improves Radiologist Performance in Skeletal Age Assessment: A Prospective Multicenter Randomized Controlled Trial. Radiology 2021; 301:692-699. [PMID: 34581608 DOI: 10.1148/radiol.2021204021] [Citation(s) in RCA: 38] [Impact Index Per Article: 12.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
Background Previous studies suggest that use of artificial intelligence (AI) algorithms as diagnostic aids may improve the quality of skeletal age assessment, though these studies lack evidence from clinical practice. Purpose To compare the accuracy and interpretation time of skeletal age assessment on hand radiograph examinations with and without the use of an AI algorithm as a diagnostic aid. Materials and Methods In this prospective randomized controlled trial, the accuracy of skeletal age assessment on hand radiograph examinations was performed with (n = 792) and without (n = 739) the AI algorithm as a diagnostic aid. For examinations with the AI algorithm, the radiologist was shown the AI interpretation as part of their routine clinical work and was permitted to accept or modify it. Hand radiographs were interpreted by 93 radiologists from six centers. The primary efficacy outcome was the mean absolute difference between the skeletal age dictated into the radiologists' signed report and the average interpretation of a panel of four radiologists not using a diagnostic aid. The secondary outcome was the interpretation time. A linear mixed-effects regression model with random center- and radiologist-level effects was used to compare the two experimental groups. Results Overall mean absolute difference was lower when radiologists used the AI algorithm compared with when they did not (5.36 months vs 5.95 months; P = .04). The proportions at which the absolute difference exceeded 12 months (9.3% vs 13.0%, P = .02) and 24 months (0.5% vs 1.8%, P = .02) were lower with the AI algorithm than without it. Median radiologist interpretation time was lower with the AI algorithm than without it (102 seconds vs 142 seconds, P = .001). Conclusion Use of an artificial intelligence algorithm improved skeletal age assessment accuracy and reduced interpretation times for radiologists, although differences were observed between centers. Clinical trial registration no. NCT03530098 © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Rubin in this issue.
Collapse
Affiliation(s)
- David K Eng
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Nishith B Khandwala
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Jin Long
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Nancy R Fefferman
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Shailee V Lala
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Naomi A Strubel
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Sarah S Milla
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Ross W Filice
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Susan E Sharp
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Alexander J Towbin
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Michael L Francavilla
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Summer L Kaplan
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Kirsten Ecklund
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Sanjay P Prabhu
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Brian J Dillon
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Brian M Everist
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Christopher G Anton
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Mark E Bittman
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Rebecca Dennis
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - David B Larson
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Jayne M Seekins
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Cicero T Silva
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Arash R Zandieh
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Curtis P Langlotz
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Matthew P Lungren
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| | - Safwan S Halabi
- From the Department of Computer Science, Stanford University, 300 N Pasteur Dr, Stanford, CA 94305 (D.K.E., N.B.K.); Departments of Pediatrics (J.L.) and Radiology (D.B.L., J.M.S., C.P.L., M.P.L., S.S.H.), Stanford University School of Medicine, Stanford, Calif; Department of Radiology, New York University School of Medicine, New York, NY (N.R.F., S.V.L., N.A.S., M.E.B.); Department of Radiology, Emory School of Medicine and Children's Healthcare of Atlanta, Atlanta, Ga (S.S.M.); Department of Radiology, MedStar Health and Georgetown University School of Medicine, Washington, DC (R.W.F., A.R.Z.); Department of Radiology, Cincinnati Children's Hospital Medical Center, Cincinnati, Ohio (S.E.S., A.J.T., C.G.A.); Department of Radiology, Children's Hospital of Philadelphia, Philadelphia, Pa (M.L.F., S.L.K., R.D.); Department of Radiology, Harvard Medical School and Boston Children's Hospital, Boston, Mass (K.E., S.P.P.); Department of Radiology, Yale School of Medicine, New Haven, Conn (B.J.D., C.T.S.); and Department of Radiology, Kansas University School of Medicine, Kansas City, Kan (B.M.E.)
| |
Collapse
|
25
|
Shad R, Quach N, Fong R, Kasinpila P, Bowles C, Castro M, Guha A, Suarez EE, Jovinge S, Lee S, Boeve T, Amsallem M, Tang X, Haddad F, Shudo Y, Woo YJ, Teuteberg J, Cunningham JP, Langlotz CP, Hiesinger W. Predicting post-operative right ventricular failure using video-based deep learning. Nat Commun 2021; 12:5192. [PMID: 34465780 PMCID: PMC8408163 DOI: 10.1038/s41467-021-25503-9] [Citation(s) in RCA: 21] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2021] [Accepted: 08/11/2021] [Indexed: 11/22/2022] Open
Abstract
Despite progressive improvements over the decades, the rich temporally resolved data in an echocardiogram remain underutilized. Human assessments reduce the complex patterns of cardiac wall motion, to a small list of measurements of heart function. All modern echocardiography artificial intelligence (AI) systems are similarly limited by design - automating measurements of the same reductionist metrics rather than utilizing the embedded wealth of data. This underutilization is most evident where clinical decision making is guided by subjective assessments of disease acuity. Predicting the likelihood of developing post-operative right ventricular failure (RV failure) in the setting of mechanical circulatory support is one such example. Here we describe a video AI system trained to predict post-operative RV failure using the full spatiotemporal density of information in pre-operative echocardiography. We achieve an AUC of 0.729, and show that this ML system significantly outperforms a team of human experts at the same task on independent evaluation.
Collapse
Affiliation(s)
- Rohan Shad
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Nicolas Quach
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Robyn Fong
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Patpilai Kasinpila
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Cayley Bowles
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Miguel Castro
- Department of Cardiovascular Medicine, Houston Methodist DeBakey Heart Centre, Houston, TX, USA
| | - Ashrith Guha
- Department of Cardiovascular Medicine, Houston Methodist DeBakey Heart Centre, Houston, TX, USA
| | - Erik E Suarez
- Department of Cardiothoracic Surgery, Houston Methodist DeBakey Heart Centre, Houston, TX, USA
| | - Stefan Jovinge
- Department of Cardiovascular Surgery, Spectrum Health Grand Rapids, Grand Rapids, MI, USA
| | - Sangjin Lee
- Department of Cardiovascular Surgery, Spectrum Health Grand Rapids, Grand Rapids, MI, USA
| | - Theodore Boeve
- Department of Cardiovascular Surgery, Spectrum Health Grand Rapids, Grand Rapids, MI, USA
| | - Myriam Amsallem
- Department of Cardiovascular Medicine, Stanford University, Stanford, CA, USA
| | - Xiu Tang
- Department of Cardiovascular Medicine, Stanford University, Stanford, CA, USA
| | - Francois Haddad
- Department of Cardiovascular Medicine, Stanford University, Stanford, CA, USA
| | - Yasuhiro Shudo
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Y Joseph Woo
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA
| | - Jeffrey Teuteberg
- Department of Cardiovascular Medicine, Stanford University, Stanford, CA, USA
- Stanford Artificial Intelligence in Medicine Centre, Stanford, CA, USA
| | | | - Curtis P Langlotz
- Stanford Artificial Intelligence in Medicine Centre, Stanford, CA, USA
- Department of Radiology and Biomedical Informatics, Stanford University, Stanford, CA, USA
| | - William Hiesinger
- Department of Cardiothoracic Surgery, Stanford University, Stanford, CA, USA.
- Stanford Artificial Intelligence in Medicine Centre, Stanford, CA, USA.
| |
Collapse
|
26
|
Shad R, Fong R, Quach N, Bowles C, Kasinpila P, Li M, Callon K, Castro M, Guha A, Suarez EE, Lee S, Jovinge S, Boeve T, Shudo Y, Langlotz CP, Teuteberg J, Hiesinger W. Long-term survival in patients with post-LVAD right ventricular failure: multi-state modelling with competing outcomes of heart transplant. J Heart Lung Transplant 2021; 40:778-785. [PMID: 34167863 DOI: 10.1016/j.healun.2021.05.002] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2021] [Revised: 04/19/2021] [Accepted: 05/12/2021] [Indexed: 10/21/2022] Open
Abstract
BACKGROUND Multicenter data on long term survival following LVAD implantation that make use of contemporary definitions of RV failure are limited. Furthermore, traditional survival analyses censor patients who receive a bridge to heart transplant. Here we compare the outcomes of LVAD patients who develop post-operative RV failure accounting for the transitional probability of receiving an interim heart transplantation. METHODS We use a retrospective cohort of LVAD patients sourced from multiple high-volume centers based in the United States. Five- and ten-year survival accounting for transition probabilities of receiving a heart transplant were calculated using a multi-state Aalen Johansen survival model. RESULTS Of the 897 patients included in the study, 238 (26.5%) developed post-operative RV failure at index hospitalization. At 10 years the probability of death with post-op RV failure was 79.28% vs 61.70% in patients without (HR 2.10; 95% CI 1.72 - 2.57; p = < .001). Though not significant, patients with RV failure were less likely to be bridged to a heart transplant (HR 0.87, p = .4). Once transplanted the risk of death between both patient groups remained equivalent; the probability of death after a heart transplant was 3.97% in those with post-operative RV failure shortly after index LVAD implant, as compared to 14.71% in those without. CONCLUSIONS AND RELEVANCE Long-term durable mechanical circulatory support is associated with significantly higher mortality in patients who develop post-operative RV failure. Improving outcomes may necessitate expeditious bridge to heart transplant wherever appropriate, along with critical reassessment of organ allocation policies.
Collapse
Affiliation(s)
- Rohan Shad
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Robyn Fong
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Nicolas Quach
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Cayley Bowles
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Patpilai Kasinpila
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Michelle Li
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Kate Callon
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Miguel Castro
- Department of Cardiovascular Medicine, Houston Methodist DeBakey Heart Center, Texas
| | - Ashrith Guha
- Department of Cardiovascular Medicine, Houston Methodist DeBakey Heart Center, Texas
| | - Erik E Suarez
- Department of Cardiothoracic Surgery, Houston Methodist DeBakey Heart Center, Texas
| | - Sangjin Lee
- Department of Cardiothoracic Surgery, Spectrum Health Grand Rapids Michigan, Michigan
| | - Stefan Jovinge
- Department of Cardiothoracic Surgery, Spectrum Health Grand Rapids Michigan, Michigan
| | - Theodore Boeve
- Department of Cardiothoracic Surgery, Spectrum Health Grand Rapids Michigan, Michigan
| | - Yasuhiro Shudo
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California
| | - Curtis P Langlotz
- Stanford Artificial intelligence in Medicine and Imaging Center, Stanford University School of Medicine, Califorina; Department of Radiology, Stanford University School of Medicine, California
| | - Jeffrey Teuteberg
- Stanford Artificial intelligence in Medicine and Imaging Center, Stanford University School of Medicine, Califorina; Department of Cardiovascular Medicine, Stanford University School of Medicine, California
| | - William Hiesinger
- Department of Cardiothoracic Surgery, Stanford University School of Medicine, California; Stanford Artificial intelligence in Medicine and Imaging Center, Stanford University School of Medicine, Califorina.
| |
Collapse
|
27
|
Chaudhari AS, Sandino CM, Cole EK, Larson DB, Gold GE, Vasanawala SS, Lungren MP, Hargreaves BA, Langlotz CP. Prospective Deployment of Deep Learning in MRI: A Framework for Important Considerations, Challenges, and Recommendations for Best Practices. J Magn Reson Imaging 2021; 54:357-371. [PMID: 32830874 PMCID: PMC8639049 DOI: 10.1002/jmri.27331] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2020] [Revised: 07/27/2020] [Accepted: 07/31/2020] [Indexed: 12/16/2022] Open
Abstract
Artificial intelligence algorithms based on principles of deep learning (DL) have made a large impact on the acquisition, reconstruction, and interpretation of MRI data. Despite the large number of retrospective studies using DL, there are fewer applications of DL in the clinic on a routine basis. To address this large translational gap, we review the recent publications to determine three major use cases that DL can have in MRI, namely, that of model-free image synthesis, model-based image reconstruction, and image or pixel-level classification. For each of these three areas, we provide a framework for important considerations that consist of appropriate model training paradigms, evaluation of model robustness, downstream clinical utility, opportunities for future advances, as well recommendations for best current practices. We draw inspiration for this framework from advances in computer vision in natural imaging as well as additional healthcare fields. We further emphasize the need for reproducibility of research studies through the sharing of datasets and software. LEVEL OF EVIDENCE: 5 TECHNICAL EFFICACY STAGE: 2.
Collapse
Affiliation(s)
| | - Christopher M Sandino
- Department of Radiology, Stanford University, Stanford, California, USA
- Department of Electrical Engineering, Stanford University, Stanford, California, USA
| | - Elizabeth K Cole
- Department of Radiology, Stanford University, Stanford, California, USA
- Department of Electrical Engineering, Stanford University, Stanford, California, USA
| | - David B Larson
- Department of Radiology, Stanford University, Stanford, California, USA
| | - Garry E Gold
- Department of Radiology, Stanford University, Stanford, California, USA
- Department of Orthopaedic Surgery, Stanford University, Stanford, California, USA
- Department of Bioengineering, Stanford University, Stanford, California, USA
| | | | - Matthew P Lungren
- Department of Radiology, Stanford University, Stanford, California, USA
| | - Brian A Hargreaves
- Department of Radiology, Stanford University, Stanford, California, USA
- Department of Electrical Engineering, Stanford University, Stanford, California, USA
- Department of Biomedical Informatics, Stanford University, Stanford, California, USA
| | - Curtis P Langlotz
- Department of Radiology, Stanford University, Stanford, California, USA
- Department of Biomedical Informatics, Stanford University, Stanford, California, USA
| |
Collapse
|
28
|
Zhang Y, Zhang Y, Qi P, Manning CD, Langlotz CP. Biomedical and clinical English model packages for the Stanza Python NLP library. J Am Med Inform Assoc 2021; 28:1892-1899. [PMID: 34157094 PMCID: PMC8363782 DOI: 10.1093/jamia/ocab090] [Citation(s) in RCA: 31] [Impact Index Per Article: 10.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2021] [Revised: 04/05/2021] [Accepted: 05/03/2021] [Indexed: 11/13/2022] Open
Abstract
Objective The study sought to develop and evaluate neural natural language processing (NLP) packages for the syntactic analysis and named entity recognition of biomedical and clinical English text. Materials and Methods We implement and train biomedical and clinical English NLP pipelines by extending the widely used Stanza library originally designed for general NLP tasks. Our models are trained with a mix of public datasets such as the CRAFT treebank as well as with a private corpus of radiology reports annotated with 5 radiology-domain entities. The resulting pipelines are fully based on neural networks, and are able to perform tokenization, part-of-speech tagging, lemmatization, dependency parsing, and named entity recognition for both biomedical and clinical text. We compare our systems against popular open-source NLP libraries such as CoreNLP and scispaCy, state-of-the-art models such as the BioBERT models, and winning systems from the BioNLP CRAFT shared task. Results For syntactic analysis, our systems achieve much better performance compared with the released scispaCy models and CoreNLP models retrained on the same treebanks, and are on par with the winning system from the CRAFT shared task. For NER, our systems substantially outperform scispaCy, and are better or on par with the state-of-the-art performance from BioBERT, while being much more computationally efficient. Conclusions We introduce biomedical and clinical NLP packages built for the Stanza library. These packages offer performance that is similar to the state of the art, and are also optimized for ease of use. To facilitate research, we make all our models publicly available. We also provide an online demonstration (http://stanza.run/bio).
Collapse
Affiliation(s)
- Yuhao Zhang
- Biomedical Informatics Training Program, Stanford University, Stanford, California, USA
| | - Yuhui Zhang
- Computer Science Department, Stanford University, Stanford, California, USA
| | - Peng Qi
- Computer Science Department, Stanford University, Stanford, California, USA
| | - Christopher D Manning
- Computer Science and Linguistics Departments, Stanford University, Stanford, California, USA
| | - Curtis P Langlotz
- Department of Radiology, Stanford University, Stanford, California, USA
| |
Collapse
|
29
|
Eng D, Chute C, Khandwala N, Rajpurkar P, Long J, Shleifer S, Khalaf MH, Sandhu AT, Rodriguez F, Maron DJ, Seyyedi S, Marin D, Golub I, Budoff M, Kitamura F, Takahashi MS, Filice RW, Shah R, Mongan J, Kallianos K, Langlotz CP, Lungren MP, Ng AY, Patel BN. Automated coronary calcium scoring using deep learning with multicenter external validation. NPJ Digit Med 2021; 4:88. [PMID: 34075194 PMCID: PMC8169744 DOI: 10.1038/s41746-021-00460-1] [Citation(s) in RCA: 50] [Impact Index Per Article: 16.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2019] [Accepted: 04/26/2021] [Indexed: 02/05/2023] Open
Abstract
Coronary artery disease (CAD), the most common manifestation of cardiovascular disease, remains the most common cause of mortality in the United States. Risk assessment is key for primary prevention of coronary events and coronary artery calcium (CAC) scoring using computed tomography (CT) is one such non-invasive tool. Despite the proven clinical value of CAC, the current clinical practice implementation for CAC has limitations such as the lack of insurance coverage for the test, need for capital-intensive CT machines, specialized imaging protocols, and accredited 3D imaging labs for analysis (including personnel and software). Perhaps the greatest gap is the millions of patients who undergo routine chest CT exams and demonstrate coronary artery calcification, but their presence is not often reported or quantitation is not feasible. We present two deep learning models that automate CAC scoring demonstrating advantages in automated scoring for both dedicated gated coronary CT exams and routine non-gated chest CTs performed for other reasons to allow opportunistic screening. First, we trained a gated coronary CT model for CAC scoring that showed near perfect agreement (mean difference in scores = -2.86; Cohen's Kappa = 0.89, P < 0.0001) with current conventional manual scoring on a retrospective dataset of 79 patients and was found to perform the task faster (average time for automated CAC scoring using a graphics processing unit (GPU) was 3.5 ± 2.1 s vs. 261 s for manual scoring) in a prospective trial of 55 patients with little difference in scores compared to three technologists (mean difference in scores = 3.24, 5.12, and 5.48, respectively). Then using CAC scores from paired gated coronary CT as a reference standard, we trained a deep learning model on our internal data and a cohort from the Multi-Ethnic Study of Atherosclerosis (MESA) study (total training n = 341, Stanford test n = 42, MESA test n = 46) to perform CAC scoring on routine non-gated chest CT exams with validation on external datasets (total n = 303) obtained from four geographically disparate health systems. On identifying patients with any CAC (i.e., CAC ≥ 1), sensitivity and PPV was high across all datasets (ranges: 80-100% and 87-100%, respectively). For CAC ≥ 100 on routine non-gated chest CTs, which is the latest recommended threshold to initiate statin therapy, our model showed sensitivities of 71-94% and positive predictive values in the range of 88-100% across all the sites. Adoption of this model could allow more patients to be screened with CAC scoring, potentially allowing opportunistic early preventive interventions.
Collapse
Affiliation(s)
- David Eng
- grid.168010.e0000000419368956Department of Computer Science, Stanford University School of Medicine, Stanford, CA USA ,Bunkerhill, Palo Alto, CA USA
| | - Christopher Chute
- grid.168010.e0000000419368956Department of Computer Science, Stanford University School of Medicine, Stanford, CA USA
| | | | - Pranav Rajpurkar
- grid.168010.e0000000419368956Department of Computer Science, Stanford University School of Medicine, Stanford, CA USA
| | - Jin Long
- grid.168010.e0000000419368956Department of Pediatrics, Stanford University School of Medicine, Stanford, CA USA
| | - Sam Shleifer
- grid.168010.e0000000419368956Department of Computer Science, Stanford University School of Medicine, Stanford, CA USA
| | - Mohamed H. Khalaf
- grid.168010.e0000000419368956Department of Radiology, Stanford University School of Medicine, Stanford, CA USA
| | - Alexander T. Sandhu
- grid.168010.e0000000419368956Division of Cardiovascular Medicine and Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Palo Alto, CA USA
| | - Fatima Rodriguez
- grid.168010.e0000000419368956Division of Cardiovascular Medicine and Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Palo Alto, CA USA
| | - David J. Maron
- grid.168010.e0000000419368956Division of Cardiovascular Medicine and Stanford Prevention Research Center, Department of Medicine, Stanford University School of Medicine, Palo Alto, CA USA
| | - Saeed Seyyedi
- grid.168010.e0000000419368956Department of Radiology, Stanford University School of Medicine, Stanford, CA USA
| | - Daniele Marin
- grid.189509.c0000000100241216Department of Radiology, Duke University Medical Center, Durham, NC USA
| | - Ilana Golub
- grid.239844.00000 0001 0157 6501Lundquist Institute at Harbor-UCLA Medical Center, Torrance, CA USA
| | - Matthew Budoff
- grid.239844.00000 0001 0157 6501Lundquist Institute at Harbor-UCLA Medical Center, Torrance, CA USA
| | - Felipe Kitamura
- Diagnósticos da América SA (Dasa), Alphaville Barueri, SP Brazil ,grid.411249.b0000 0001 0514 7202Department of Diagnostic Imaging, Universidade Federal de São Paulo (Unifesp), São Paulo, SP Brazil
| | | | - Ross W. Filice
- grid.411663.70000 0000 8937 0972Department of Radiology, MedStar Georgetown University Hospital, Washington, DC USA
| | - Rajesh Shah
- grid.280747.e0000 0004 0419 2556Radiology Service, VA Palo Alto Health Care System, Palo Alto, CA USA
| | - John Mongan
- grid.266102.10000 0001 2297 6811Department of Radiology and Biomedical Imaging and Center for Intelligent Imaging, University of California, San Francisco, School of Medicine, San Francisco, CA USA
| | - Kimberly Kallianos
- grid.266102.10000 0001 2297 6811Department of Radiology and Biomedical Imaging and Center for Intelligent Imaging, University of California, San Francisco, School of Medicine, San Francisco, CA USA
| | - Curtis P. Langlotz
- grid.168010.e0000000419368956Department of Radiology, Stanford University School of Medicine, Stanford, CA USA
| | - Matthew P. Lungren
- grid.168010.e0000000419368956Department of Radiology, Stanford University School of Medicine, Stanford, CA USA
| | - Andrew Y. Ng
- grid.168010.e0000000419368956Department of Computer Science, Stanford University School of Medicine, Stanford, CA USA
| | - Bhavik N. Patel
- grid.417468.80000 0000 8875 6339Department of Radiology, Mayo Clinic, Scottsdale, AZ USA
| |
Collapse
|
30
|
|
31
|
Alvarez JB, Bibault JE, Burgun A, Cai J, Cao Z, Chang K, Chen JH, Chen WC, Cho M, Cho PJ, Cornish TC, Costa A, Dekker A, Drukker K, Dunn J, Eminaga O, Erickson BJ, Fournier L, Gambhir SS, Gennatas ED, Giger ML, Halilaj I, Harrison AP, He B, Hong JC, Jin D, Jin MC, Jochems A, Kalpathy-Cramer J, Kapp DS, Karimzadeh M, Karnes W, Lambin P, Langlotz CP, Lee J, Li H, Liao JC, Lin AL, Lin RY, Liu Y, Lu L, Magnus D, McIntosh C, Miao S, Min JK, Neill DB, Oermann EK, Ouyang D, Peng L, Phene S, Poirot MG, Quon JL, Ranti D, Rao A, Raskar R, Rombaoa C, Rubin DL, Samarasena J, Seekins J, Seetharam K, Shearer E, Sibley A, Singh K, Singh P, Sordo M, Suraweera D, Valliani AAA, van Wijk Y, Vepakomma P, Wang B, Wang G, Wang N, Wang Y, Warner E, Welch M, Wong K, Wu Z, Xing F, Xing L, Yan K, Yan P, Yang L, Yeom KW, Zachariah R, Zeng D, Zhang L, Zhang L, Zhang X, Zhou L, Zou J. List of contributors. Artif Intell Med 2021. [DOI: 10.1016/b978-0-12-821259-2.00035-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
|
32
|
Larson DB, Harvey H, Rubin DL, Irani N, Tse JR, Langlotz CP. Regulatory Frameworks for Development and Evaluation of Artificial Intelligence-Based Diagnostic Imaging Algorithms: Summary and Recommendations. J Am Coll Radiol 2020; 18:413-424. [PMID: 33096088 PMCID: PMC7574690 DOI: 10.1016/j.jacr.2020.09.060] [Citation(s) in RCA: 42] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2020] [Revised: 09/23/2020] [Accepted: 09/23/2020] [Indexed: 12/28/2022]
Abstract
Although artificial intelligence (AI)-based algorithms for diagnosis hold promise for improving care, their safety and effectiveness must be ensured to facilitate wide adoption. Several recently proposed regulatory frameworks provide a solid foundation but do not address a number of issues that may prevent algorithms from being fully trusted. In this article, we review the major regulatory frameworks for software as a medical device applications, identify major gaps, and propose additional strategies to improve the development and evaluation of diagnostic AI algorithms. We identify the following major shortcomings of the current regulatory frameworks: (1) conflation of the diagnostic task with the diagnostic algorithm, (2) superficial treatment of the diagnostic task definition, (3) no mechanism to directly compare similar algorithms, (4) insufficient characterization of safety and performance elements, (5) lack of resources to assess performance at each installed site, and (6) inherent conflicts of interest. We recommend the following additional measures: (1) separate the diagnostic task from the algorithm, (2) define performance elements beyond accuracy, (3) divide the evaluation process into discrete steps, (4) encourage assessment by a third-party evaluator, (5) incorporate these elements into the manufacturers’ development process. Specifically, we recommend four phases of development and evaluation, analogous to those that have been applied to pharmaceuticals and proposed for software applications, to help ensure world-class performance of all algorithms at all installed sites. In the coming years, we anticipate the emergence of a substantial body of research dedicated to ensuring the accuracy, reliability, and safety of the algorithms.
Collapse
Affiliation(s)
- David B Larson
- Vice Chair, Education and Clinical Operations, Department of Radiology, Stanford University School of Medicine, Stanford, California.
| | - Hugh Harvey
- Institute for Cognitive Neuroscience, University College, London, UK
| | - Daniel L Rubin
- Director of Biomedical Informatics at Stanford Cancer Institute, Departments of Biomedical Data Science, Radiology, and Medicine, Stanford University School of Medicine, Stanford, California
| | - Neville Irani
- Department of Radiology, University of Kansas Medical Center, Kansas City, Kansas
| | - Justin R Tse
- Department of Radiological Sciences, David Geffen School of Medicine, University of California, Los Angeles, California
| | - Curtis P Langlotz
- Associate Chair, Information Systems, Department of Radiology, Stanford University School of Medicine, Stanford, California
| |
Collapse
|
33
|
Ouyang D, He B, Ghorbani A, Yuan N, Ebinger J, Langlotz CP, Heidenreich PA, Harrington RA, Liang DH, Ashley EA, Zou JY. Video-based AI for beat-to-beat assessment of cardiac function. Nature 2020; 580:252-256. [DOI: 10.1038/s41586-020-2145-8] [Citation(s) in RCA: 183] [Impact Index Per Article: 45.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2019] [Accepted: 02/20/2020] [Indexed: 12/18/2022]
|
34
|
Larson DB, Magnus DC, Lungren MP, Shah NH, Langlotz CP. Ethics of Using and Sharing Clinical Imaging Data for Artificial Intelligence: A Proposed Framework. Radiology 2020; 295:675-682. [PMID: 32208097 DOI: 10.1148/radiol.2020192536] [Citation(s) in RCA: 77] [Impact Index Per Article: 19.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Abstract
In this article, the authors propose an ethical framework for using and sharing clinical data for the development of artificial intelligence (AI) applications. The philosophical premise is as follows: when clinical data are used to provide care, the primary purpose for acquiring the data is fulfilled. At that point, clinical data should be treated as a form of public good, to be used for the benefit of future patients. In their 2013 article, Faden et al argued that all who participate in the health care system, including patients, have a moral obligation to contribute to improving that system. The authors extend that framework to questions surrounding the secondary use of clinical data for AI applications. Specifically, the authors propose that all individuals and entities with access to clinical data become data stewards, with fiduciary (or trust) responsibilities to patients to carefully safeguard patient privacy, and to the public to ensure that the data are made widely available for the development of knowledge and tools to benefit future patients. According to this framework, the authors maintain that it is unethical for providers to "sell" clinical data to other parties by granting access to clinical data, especially under exclusive arrangements, in exchange for monetary or in-kind payments that exceed costs. The authors also propose that patient consent is not required before the data are used for secondary purposes when obtaining such consent is prohibitively costly or burdensome, as long as mechanisms are in place to ensure that ethical standards are strictly followed. Rather than debate whether patients or provider organizations "own" the data, the authors propose that clinical data are not owned at all in the traditional sense, but rather that all who interact with or control the data have an obligation to ensure that the data are used for the benefit of future patients and society.
Collapse
Affiliation(s)
- David B Larson
- From the Department of Radiology, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA 94305-5105
| | - David C Magnus
- From the Department of Radiology, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA 94305-5105
| | - Matthew P Lungren
- From the Department of Radiology, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA 94305-5105
| | - Nigam H Shah
- From the Department of Radiology, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA 94305-5105
| | - Curtis P Langlotz
- From the Department of Radiology, Stanford University School of Medicine, 300 Pasteur Dr, Stanford, CA 94305-5105
| |
Collapse
|
35
|
Rajpurkar P, Park A, Irvin J, Chute C, Bereket M, Mastrodicasa D, Langlotz CP, Lungren MP, Ng AY, Patel BN. AppendiXNet: Deep Learning for Diagnosis of Appendicitis from A Small Dataset of CT Exams Using Video Pretraining. Sci Rep 2020; 10:3958. [PMID: 32127625 PMCID: PMC7054445 DOI: 10.1038/s41598-020-61055-6] [Citation(s) in RCA: 39] [Impact Index Per Article: 9.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2019] [Accepted: 02/17/2020] [Indexed: 12/13/2022] Open
Abstract
The development of deep learning algorithms for complex tasks in digital medicine has relied on the availability of large labeled training datasets, usually containing hundreds of thousands of examples. The purpose of this study was to develop a 3D deep learning model, AppendiXNet, to detect appendicitis, one of the most common life-threatening abdominal emergencies, using a small training dataset of less than 500 training CT exams. We explored whether pretraining the model on a large collection of natural videos would improve the performance of the model over training the model from scratch. AppendiXNet was pretrained on a large collection of YouTube videos called Kinetics, consisting of approximately 500,000 video clips and annotated for one of 600 human action classes, and then fine-tuned on a small dataset of 438 CT scans annotated for appendicitis. We found that pretraining the 3D model on natural videos significantly improved the performance of the model from an AUC of 0.724 (95% CI 0.625, 0.823) to 0.810 (95% CI 0.725, 0.895). The application of deep learning to detect abnormalities on CT examinations using video pretraining could generalize effectively to other challenging cross-sectional medical imaging tasks when training data is limited.
Collapse
Affiliation(s)
- Pranav Rajpurkar
- Stanford University Department of Computer Science, Stanford, USA
| | - Allison Park
- Stanford University Department of Computer Science, Stanford, USA
| | - Jeremy Irvin
- Stanford University Department of Computer Science, Stanford, USA
| | - Chris Chute
- Stanford University Department of Computer Science, Stanford, USA
| | - Michael Bereket
- Stanford University Department of Computer Science, Stanford, USA
| | | | | | | | - Andrew Y Ng
- Stanford University Department of Computer Science, Stanford, USA
| | - Bhavik N Patel
- Stanford University Department of Radiology, Stanford, USA.
- Stanford University AIMI Center, Stanford, USA.
| |
Collapse
|
36
|
Kiani A, Uyumazturk B, Rajpurkar P, Wang A, Gao R, Jones E, Yu Y, Langlotz CP, Ball RL, Montine TJ, Martin BA, Berry GJ, Ozawa MG, Hazard FK, Brown RA, Chen SB, Wood M, Allard LS, Ylagan L, Ng AY, Shen J. Impact of a deep learning assistant on the histopathologic classification of liver cancer. NPJ Digit Med 2020; 3:23. [PMID: 32140566 PMCID: PMC7044422 DOI: 10.1038/s41746-020-0232-8] [Citation(s) in RCA: 108] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/03/2019] [Accepted: 02/06/2020] [Indexed: 12/15/2022] Open
Abstract
Artificial intelligence (AI) algorithms continue to rival human performance on a variety of clinical tasks, while their actual impact on human diagnosticians, when incorporated into clinical workflows, remains relatively unexplored. In this study, we developed a deep learning-based assistant to help pathologists differentiate between two subtypes of primary liver cancer, hepatocellular carcinoma and cholangiocarcinoma, on hematoxylin and eosin-stained whole-slide images (WSI), and evaluated its effect on the diagnostic performance of 11 pathologists with varying levels of expertise. Our model achieved accuracies of 0.885 on a validation set of 26 WSI, and 0.842 on an independent test set of 80 WSI. Although use of the assistant did not change the mean accuracy of the 11 pathologists (p = 0.184, OR = 1.281), it significantly improved the accuracy (p = 0.045, OR = 1.499) of a subset of nine pathologists who fell within well-defined experience levels (GI subspecialists, non-GI subspecialists, and trainees). In the assisted state, model accuracy significantly impacted the diagnostic decisions of all 11 pathologists. As expected, when the model's prediction was correct, assistance significantly improved accuracy (p = 0.000, OR = 4.289), whereas when the model's prediction was incorrect, assistance significantly decreased accuracy (p = 0.000, OR = 0.253), with both effects holding across all pathologist experience levels and case difficulty levels. Our results highlight the challenges of translating AI models into the clinical setting, and emphasize the importance of taking into account potential unintended negative consequences of model assistance when designing and testing medical AI-assistance tools.
Collapse
Affiliation(s)
- Amirhossein Kiani
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Bora Uyumazturk
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Pranav Rajpurkar
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Alex Wang
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Rebecca Gao
- Stanford University School of Medicine, Stanford, CA USA
| | - Erik Jones
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Yifan Yu
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Curtis P. Langlotz
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA USA
- Department of Radiology, Stanford University, Stanford, CA USA
| | - Robyn L. Ball
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA USA
| | - Thomas J. Montine
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA USA
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Brock A. Martin
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Gerald J. Berry
- Department of Pathology, Stanford University, Stanford, CA USA
| | | | | | - Ryanne A. Brown
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Simon B. Chen
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Mona Wood
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Libby S. Allard
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Lourdes Ylagan
- Department of Pathology, Stanford University, Stanford, CA USA
| | - Andrew Y. Ng
- Department of Computer Science, Stanford University, Stanford, CA USA
| | - Jeanne Shen
- Center for Artificial Intelligence in Medicine & Imaging, Stanford University, Stanford, CA USA
- Department of Pathology, Stanford University, Stanford, CA USA
| |
Collapse
|
37
|
Vreeman DJ, Abhyankar S, Wang KC, Carr C, Collins B, Rubin DL, Langlotz CP. The LOINC RSNA radiology playbook - a unified terminology for radiology procedures. J Am Med Inform Assoc 2019; 25:885-893. [PMID: 29850823 PMCID: PMC6016707 DOI: 10.1093/jamia/ocy053] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2017] [Accepted: 05/01/2018] [Indexed: 11/30/2022] Open
Abstract
Objective This paper describes the unified LOINC/RSNA Radiology Playbook and the process by which it was produced. Methods The Regenstrief Institute and the Radiological Society of North America (RSNA) developed a unification plan consisting of six objectives 1) develop a unified model for radiology procedure names that represents the attributes with an extensible set of values, 2) transform existing LOINC procedure codes into the unified model representation, 3) create a mapping between all the attribute values used in the unified model as coded in LOINC (ie, LOINC Parts) and their equivalent concepts in RadLex, 4) create a mapping between the existing procedure codes in the RadLex Core Playbook and the corresponding codes in LOINC, 5) develop a single integrated governance process for managing the unified terminology, and 6) publicly distribute the terminology artifacts. Results We developed a unified model and instantiated it in a new LOINC release artifact that contains the LOINC codes and display name (ie LONG_COMMON_NAME) for each procedure, mappings between LOINC and the RSNA Playbook at the procedure code level, and connections between procedure terms and their attribute values that are expressed as LOINC Parts and RadLex IDs. We transformed all the existing LOINC content into the new model and publicly distributed it in standard releases. The organizations have also developed a joint governance process for ongoing maintenance of the terminology. Conclusions The LOINC/RSNA Radiology Playbook provides a universal terminology standard for radiology orders and results.
Collapse
Affiliation(s)
- Daniel J Vreeman
- Center for Biomedical Informatics, Regenstrief Institute, Inc., Indianapolis, Indiana, USA.,Department of Medicine, Indiana University School of Medicine, Indianapolis, Indiana, USA
| | - Swapna Abhyankar
- Center for Biomedical Informatics, Regenstrief Institute, Inc., Indianapolis, Indiana, USA
| | - Kenneth C Wang
- Imaging Service, VA Maryland Health Care System, Baltimore, Maryland, USA.,Diagnostic Radiology and Nuclear Medicine, University of Maryland School of Medicine, Baltimore, Maryland, USA
| | - Christopher Carr
- Informatics Department, Radiological Society of North America, Oak Brook, Illinois, USA
| | - Beverly Collins
- Department of Radiology, Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Daniel L Rubin
- Department of Biomedical Data Science, Stanford University, Stanford, California, USA and.,Department of Radiology, Stanford University, Stanford, California, USA
| | - Curtis P Langlotz
- Department of Radiology, Stanford University, Stanford, California, USA
| |
Collapse
|
38
|
Nass SJ, Cogle CR, Brink JA, Langlotz CP, Balogh EP, Muellner A, Siegal D, Schilsky RL, Hricak H. Improving Cancer Diagnosis and Care: Patient Access to Oncologic Imaging Expertise. J Clin Oncol 2019; 37:1690-1694. [PMID: 31050908 PMCID: PMC6638597 DOI: 10.1200/jco.18.01970] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/29/2019] [Indexed: 12/20/2022] Open
Affiliation(s)
- Sharyl J. Nass
- National Academies of Sciences, Engineering, and Medicine, Washington, DC
| | | | - James A. Brink
- Massachusetts General Hospital, Harvard Medical School, Boston, MA
| | | | - Erin P. Balogh
- National Academies of Sciences, Engineering, and Medicine, Washington, DC
| | - Ada Muellner
- Memorial Sloan Kettering Cancer Center, New York, NY
| | - Dana Siegal
- CRICO Strategies, The Risk Management Foundation, Harvard Medical Institutions, Boston, MA
| | | | - Hedvig Hricak
- Memorial Sloan Kettering Cancer Center, New York, NY
| |
Collapse
|
39
|
Langlotz CP, Allen B, Erickson BJ, Kalpathy-Cramer J, Bigelow K, Cook TS, Flanders AE, Lungren MP, Mendelson DS, Rudie JD, Wang G, Kandarpa K. A Roadmap for Foundational Research on Artificial Intelligence in Medical Imaging: From the 2018 NIH/RSNA/ACR/The Academy Workshop. Radiology 2019; 291:781-791. [PMID: 30990384 PMCID: PMC6542624 DOI: 10.1148/radiol.2019190613] [Citation(s) in RCA: 163] [Impact Index Per Article: 32.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2019] [Revised: 03/24/2019] [Accepted: 03/25/2019] [Indexed: 01/08/2023]
Abstract
Imaging research laboratories are rapidly creating machine learning systems that achieve expert human performance using open-source methods and tools. These artificial intelligence systems are being developed to improve medical image reconstruction, noise reduction, quality assurance, triage, segmentation, computer-aided detection, computer-aided classification, and radiogenomics. In August 2018, a meeting was held in Bethesda, Maryland, at the National Institutes of Health to discuss the current state of the art and knowledge gaps and to develop a roadmap for future research initiatives. Key research priorities include: 1, new image reconstruction methods that efficiently produce images suitable for human interpretation from source data; 2, automated image labeling and annotation methods, including information extraction from the imaging report, electronic phenotyping, and prospective structured image reporting; 3, new machine learning methods for clinical imaging data, such as tailored, pretrained model architectures, and federated machine learning methods; 4, machine learning methods that can explain the advice they provide to human users (so-called explainable artificial intelligence); and 5, validated methods for image de-identification and data sharing to facilitate wide availability of clinical imaging data sets. This research roadmap is intended to identify and prioritize these needs for academic research laboratories, funding agencies, professional societies, and industry.
Collapse
Affiliation(s)
- Curtis P. Langlotz
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Bibb Allen
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Bradley J. Erickson
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Jayashree Kalpathy-Cramer
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Keith Bigelow
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Tessa S. Cook
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Adam E. Flanders
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Matthew P. Lungren
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - David S. Mendelson
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Jeffrey D. Rudie
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Ge Wang
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| | - Krishna Kandarpa
- From the Department of Radiology, Stanford University, Stanford, CA 94305 (C.P.L., M.P.L.); Department of Radiology, Grandview Medical Center, Birmingham, Ala (B.A.); Department of Radiology, Mayo Clinic, Rochester, Minn (B.J.E.); Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, Mass (J.K.C.); GE Healthcare, Chicago, Ill (K.B.); Department of Radiology, Hospital of the University of Pennsylvania, Philadelphia, Pa (T.S.C., J.D.R.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, Icahn School of Medicine at Mount Sinai, New York, NY (D.S.M.); Biomedical Imaging Center, Rensselaer Polytechnic Institute, Troy, NY (G.W.); and National Institute of Biomedical Imaging and Bioengineering, National Institutes of Health, Washington, DC (K.K.)
| |
Collapse
|
40
|
Allen B, Seltzer SE, Langlotz CP, Dreyer KP, Summers RM, Petrick N, Marinac-Dabic D, Cruz M, Alkasab TK, Hanisch RJ, Nilsen WJ, Burleson J, Lyman K, Kandarpa K. A Road Map for Translational Research on Artificial Intelligence in Medical Imaging: From the 2018 National Institutes of Health/RSNA/ACR/The Academy Workshop. J Am Coll Radiol 2019; 16:1179-1189. [PMID: 31151893 DOI: 10.1016/j.jacr.2019.04.014] [Citation(s) in RCA: 56] [Impact Index Per Article: 11.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2019] [Revised: 04/22/2019] [Accepted: 04/22/2019] [Indexed: 12/29/2022]
Abstract
Advances in machine learning in medical imaging are occurring at a rapid pace in research laboratories both at academic institutions and in industry. Important artificial intelligence (AI) tools for diagnostic imaging include algorithms for disease detection and classification, image optimization, radiation reduction, and workflow enhancement. Although advances in foundational research are occurring rapidly, translation to routine clinical practice has been slower. In August 2018, the National Institutes of Health assembled multiple relevant stakeholders at a public meeting to discuss the current state of knowledge, infrastructure gaps, and challenges to wider implementation. The conclusions of that meeting are summarized in two publications that identify and prioritize initiatives to accelerate foundational and translational research in AI for medical imaging. This publication summarizes key priorities for translational research developed at the workshop including: (1) creating structured AI use cases, defining and highlighting clinical challenges potentially solvable by AI; (2) establishing methods to encourage data sharing for training and testing AI algorithms to promote generalizability to widespread clinical practice and mitigate unintended bias; (3) establishing tools for validation and performance monitoring of AI algorithms to facilitate regulatory approval; and (4) developing standards and common data elements for seamless integration of AI tools into existing clinical workflows. An important goal of the resulting road map is to grow an ecosystem, facilitated by professional societies, industry, and government agencies, that will allow robust collaborations between practicing clinicians and AI researchers to advance foundational and translational research relevant to medical imaging.
Collapse
Affiliation(s)
- Bibb Allen
- Department of Radiology, Grandview Medical Center, Birmingham, Alabama.
| | - Steven E Seltzer
- Radiology Department, Brigham and Women's Hospital, Boston, Massachusetts; Radiology, Harvard Medical School, Boston, Massachusetts
| | | | - Keith P Dreyer
- Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts
| | - Ronald M Summers
- Radiology and Imaging Sciences, National Institutes of Health Clinical Center, Bethesda, Maryland
| | - Nicholas Petrick
- Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland
| | - Danica Marinac-Dabic
- Division of Epidemiology, Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland
| | - Marisa Cruz
- Digital Health Unit, Center for Devices and Radiological Health, US Food and Drug Administration, Silver Spring, Maryland
| | - Tarik K Alkasab
- Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts
| | - Robert J Hanisch
- Office of Data and Informatics, Material Measurement Laboratory, National Institute of Standards and Technology, Gaithersburg, Maryland
| | - Wendy J Nilsen
- National Science Foundation, Division of Information and Intelligent Systems, Alexandria, Virginia
| | - Judy Burleson
- American College of Radiology, Department of Quality and Safety, Reston, Virginia
| | | | - Krishna Kandarpa
- Research Sciences & Strategic Directions, Office of the Director, National Institute of Biomedical Imaging and Bioengineering, The National Institutes of Health, Bethesda, Maryland
| |
Collapse
|
41
|
Langlotz CP. Will Artificial Intelligence Replace Radiologists? Radiol Artif Intell 2019; 1:e190058. [PMID: 33937794 PMCID: PMC8017417 DOI: 10.1148/ryai.2019190058] [Citation(s) in RCA: 79] [Impact Index Per Article: 15.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2019] [Revised: 04/16/2019] [Accepted: 04/17/2019] [Indexed: 01/02/2023]
Affiliation(s)
- Curtis P. Langlotz
- From the Department of Radiology, Stanford University, 300 Pasteur Dr, Room H1330D, Stanford, CA 94305
| |
Collapse
|
42
|
Huhdanpaa HT, Tan WK, Rundell SD, Suri P, Chokshi FH, Comstock BA, Heagerty PJ, James KT, Avins AL, Nedeljkovic SS, Nerenz DR, Kallmes DF, Luetmer PH, Sherman KJ, Organ NL, Griffith B, Langlotz CP, Carrell D, Hassanpour S, Jarvik JG. Using Natural Language Processing of Free-Text Radiology Reports to Identify Type 1 Modic Endplate Changes. J Digit Imaging 2019; 31:84-90. [PMID: 28808792 DOI: 10.1007/s10278-017-0013-3] [Citation(s) in RCA: 28] [Impact Index Per Article: 5.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022] Open
Abstract
Electronic medical record (EMR) systems provide easy access to radiology reports and offer great potential to support quality improvement efforts and clinical research. Harnessing the full potential of the EMR requires scalable approaches such as natural language processing (NLP) to convert text into variables used for evaluation or analysis. Our goal was to determine the feasibility of using NLP to identify patients with Type 1 Modic endplate changes using clinical reports of magnetic resonance (MR) imaging examinations of the spine. Identifying patients with Type 1 Modic change who may be eligible for clinical trials is important as these findings may be important targets for intervention. Four annotators identified all reports that contained Type 1 Modic change, using N = 458 randomly selected lumbar spine MR reports. We then implemented a rule-based NLP algorithm in Java using regular expressions. The prevalence of Type 1 Modic change in the annotated dataset was 10%. Results were recall (sensitivity) 35/50 = 0.70 (95% confidence interval (C.I.) 0.52-0.82), specificity 404/408 = 0.99 (0.97-1.0), precision (positive predictive value) 35/39 = 0.90 (0.75-0.97), negative predictive value 404/419 = 0.96 (0.94-0.98), and F1-score 0.79 (0.43-1.0). Our evaluation shows the efficacy of rule-based NLP approach for identifying patients with Type 1 Modic change if the emphasis is on identifying only relevant cases with low concern regarding false negatives. As expected, our results show that specificity is higher than recall. This is due to the inherent difficulty of eliciting all possible keywords given the enormous variability of lumbar spine reporting, which decreases recall, while availability of good negation algorithms improves specificity.
Collapse
Affiliation(s)
| | - W Katherine Tan
- Department of Biostatistics, University of Washington, Seattle, WA, USA.,Center for Biomedical Statistics, University of Washington, Seattle, WA, USA
| | - Sean D Rundell
- Department of Rehabilitation Medicine, University of Washington, Seattle, WA, USA.,Comparative Effectiveness, Cost and Outcomes Research Center, University of Washington, Seattle, WA, USA
| | - Pradeep Suri
- Department of Rehabilitation Medicine, University of Washington, Seattle, WA, USA.,Comparative Effectiveness, Cost and Outcomes Research Center, University of Washington, Seattle, WA, USA.,Division of Rehabilitation Care Services, Seattle Epidemiologic Research and Information Center, VA Puget Sound Health Care System, Seattle, WA, USA
| | - Falgun H Chokshi
- Department of Radiology and Imaging Sciences, Emory University School of Medicine, Atlanta, GA, USA
| | - Bryan A Comstock
- Department of Biostatistics, University of Washington, Seattle, WA, USA.,Center for Biomedical Statistics, University of Washington, Seattle, WA, USA
| | - Patrick J Heagerty
- Department of Biostatistics, University of Washington, Seattle, WA, USA.,Center for Biomedical Statistics, University of Washington, Seattle, WA, USA
| | - Kathryn T James
- Comparative Effectiveness, Cost and Outcomes Research Center, University of Washington, Seattle, WA, USA.,Department of Radiology, University of Washington, Box 359728, 325 Ninth Ave., Seattle, WA, 98104-2499, USA
| | - Andrew L Avins
- Division of Research, Kaiser Permanente Northern California, Oakland, CA, USA
| | - Srdjan S Nedeljkovic
- Department of Anesthesiology, Perioperative and Pain Medicine, Harvard Vanguard Medical Associates, Brigham and Women's Hospital and Spine Unit, Boston, MA, USA
| | - David R Nerenz
- Henry Ford Hospital, Neuroscience Institute, Detroit, MI, USA
| | | | | | - Karen J Sherman
- Kaiser Permanente of Washington Research Institute, Seattle, WA, USA
| | - Nancy L Organ
- Department of Biostatistics, University of Washington, Seattle, WA, USA.,Center for Biomedical Statistics, University of Washington, Seattle, WA, USA
| | - Brent Griffith
- Department of Radiology, Henry Ford Hospital, Detroit, MI, USA
| | | | - David Carrell
- Kaiser Permanente of Washington Research Institute, Seattle, WA, USA
| | - Saeed Hassanpour
- Department of Biomedical Data Science, Dartmouth College, Lebanon, NH, USA
| | - Jeffrey G Jarvik
- Comparative Effectiveness, Cost and Outcomes Research Center, University of Washington, Seattle, WA, USA. .,Department of Radiology, University of Washington, Box 359728, 325 Ninth Ave., Seattle, WA, 98104-2499, USA. .,Department of Neurological Surgery, University of Washington, Seattle, WA, USA. .,Department of Health Services, University of Washington, Seattle, WA, USA.
| |
Collapse
|
43
|
Chokshi FH, Flanders AE, Prevedello LM, Langlotz CP. Fostering a Healthy AI Ecosystem for Radiology: Conclusions of the 2018 RSNA Summit on AI in Radiology. Radiol Artif Intell 2019; 1:190021. [PMID: 33937789 PMCID: PMC8017423 DOI: 10.1148/ryai.2019190021] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2019] [Revised: 03/01/2019] [Accepted: 03/04/2019] [Indexed: 05/03/2023]
Abstract
The 2018 RSNA Summit on AI in Radiology brought together a diverse group of stakeholders to identify and prioritize areas of need related to artificial intelligence in radiology. This article presents the proceedings of the summit with emphasis on RSNA's role in leading, organizing, and catalyzing change during this important time in radiology. © RSNA, 2019.
Collapse
Affiliation(s)
- Falgun H. Chokshi
- From the Departments of Radiology and Imaging Sciences and Biomedical Informatics, Emory University School of Medicine, 1364 Clifton Rd NE, Atlanta, GA 30322 (F.H.C.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); and Departments of Radiology and Biomedical Informatics, Stanford University School of Medicine, Stanford, Calif (C.P.L.)
| | - Adam E. Flanders
- From the Departments of Radiology and Imaging Sciences and Biomedical Informatics, Emory University School of Medicine, 1364 Clifton Rd NE, Atlanta, GA 30322 (F.H.C.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); and Departments of Radiology and Biomedical Informatics, Stanford University School of Medicine, Stanford, Calif (C.P.L.)
| | - Luciano M. Prevedello
- From the Departments of Radiology and Imaging Sciences and Biomedical Informatics, Emory University School of Medicine, 1364 Clifton Rd NE, Atlanta, GA 30322 (F.H.C.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); and Departments of Radiology and Biomedical Informatics, Stanford University School of Medicine, Stanford, Calif (C.P.L.)
| | - Curtis P. Langlotz
- From the Departments of Radiology and Imaging Sciences and Biomedical Informatics, Emory University School of Medicine, 1364 Clifton Rd NE, Atlanta, GA 30322 (F.H.C.); Department of Radiology, Thomas Jefferson University Hospital, Philadelphia, Pa (A.E.F.); Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, Ohio (L.M.P.); and Departments of Radiology and Biomedical Informatics, Stanford University School of Medicine, Stanford, Calif (C.P.L.)
| |
Collapse
|
44
|
Dunnmon JA, Yi D, Langlotz CP, Ré C, Rubin DL, Lungren MP. Assessment of Convolutional Neural Networks for Automated Classification of Chest Radiographs. Radiology 2019; 290:537-544. [PMID: 30422093 PMCID: PMC6358056 DOI: 10.1148/radiol.2018181422] [Citation(s) in RCA: 106] [Impact Index Per Article: 21.2] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2018] [Revised: 08/25/2018] [Accepted: 09/17/2018] [Indexed: 11/11/2022]
Abstract
Purpose To assess the ability of convolutional neural networks (CNNs) to enable high-performance automated binary classification of chest radiographs. Materials and Methods In a retrospective study, 216 431 frontal chest radiographs obtained between 1998 and 2012 were procured, along with associated text reports and a prospective label from the attending radiologist. This data set was used to train CNNs to classify chest radiographs as normal or abnormal before evaluation on a held-out set of 533 images hand-labeled by expert radiologists. The effects of development set size, training set size, initialization strategy, and network architecture on end performance were assessed by using standard binary classification metrics; detailed error analysis, including visualization of CNN activations, was also performed. Results Average area under the receiver operating characteristic curve (AUC) was 0.96 for a CNN trained with 200 000 images. This AUC value was greater than that observed when the same model was trained with 2000 images (AUC = 0.84, P < .005) but was not significantly different from that observed when the model was trained with 20 000 images (AUC = 0.95, P > .05). Averaging the CNN output score with the binary prospective label yielded the best-performing classifier, with an AUC of 0.98 (P < .005). Analysis of specific radiographs revealed that the model was heavily influenced by clinically relevant spatial regions but did not reliably generalize beyond thoracic disease. Conclusion CNNs trained with a modestly sized collection of prospectively labeled chest radiographs achieved high diagnostic performance in the classification of chest radiographs as normal or abnormal; this function may be useful for automated prioritization of abnormal chest radiographs. © RSNA, 2018 Online supplemental material is available for this article. See also the editorial by van Ginneken in this issue.
Collapse
Affiliation(s)
- Jared A. Dunnmon
- From the Departments of Computer Science (J.A.D., C.R.), Biomedical
Data Science (D.Y., D.L.R.), and Radiology (C.P.L., D.L.R., M.P.L.), Stanford
University, 300 Pasteur Dr, Stanford, CA 94305
| | - Darvin Yi
- From the Departments of Computer Science (J.A.D., C.R.), Biomedical
Data Science (D.Y., D.L.R.), and Radiology (C.P.L., D.L.R., M.P.L.), Stanford
University, 300 Pasteur Dr, Stanford, CA 94305
| | - Curtis P. Langlotz
- From the Departments of Computer Science (J.A.D., C.R.), Biomedical
Data Science (D.Y., D.L.R.), and Radiology (C.P.L., D.L.R., M.P.L.), Stanford
University, 300 Pasteur Dr, Stanford, CA 94305
| | - Christopher Ré
- From the Departments of Computer Science (J.A.D., C.R.), Biomedical
Data Science (D.Y., D.L.R.), and Radiology (C.P.L., D.L.R., M.P.L.), Stanford
University, 300 Pasteur Dr, Stanford, CA 94305
| | - Daniel L. Rubin
- From the Departments of Computer Science (J.A.D., C.R.), Biomedical
Data Science (D.Y., D.L.R.), and Radiology (C.P.L., D.L.R., M.P.L.), Stanford
University, 300 Pasteur Dr, Stanford, CA 94305
| | - Matthew P. Lungren
- From the Departments of Computer Science (J.A.D., C.R.), Biomedical
Data Science (D.Y., D.L.R.), and Radiology (C.P.L., D.L.R., M.P.L.), Stanford
University, 300 Pasteur Dr, Stanford, CA 94305
| |
Collapse
|
45
|
Banerjee I, Ling Y, Chen MC, Hasan SA, Langlotz CP, Moradzadeh N, Chapman B, Amrhein T, Mong D, Rubin DL, Farri O, Lungren MP. Comparative effectiveness of convolutional neural network (CNN) and recurrent neural network (RNN) architectures for radiology text report classification. Artif Intell Med 2018; 97:79-88. [PMID: 30477892 DOI: 10.1016/j.artmed.2018.11.004] [Citation(s) in RCA: 98] [Impact Index Per Article: 16.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2017] [Revised: 08/06/2018] [Accepted: 11/13/2018] [Indexed: 01/11/2023]
Abstract
This paper explores cutting-edge deep learning methods for information extraction from medical imaging free text reports at a multi-institutional scale and compares them to the state-of-the-art domain-specific rule-based system - PEFinder and traditional machine learning methods - SVM and Adaboost. We proposed two distinct deep learning models - (i) CNN Word - Glove, and (ii) Domain phrase attention-based hierarchical recurrent neural network (DPA-HNN), for synthesizing information on pulmonary emboli (PE) from over 7370 clinical thoracic computed tomography (CT) free-text radiology reports collected from four major healthcare centers. Our proposed DPA-HNN model encodes domain-dependent phrases into an attention mechanism and represents a radiology report through a hierarchical RNN structure composed of word-level, sentence-level and document-level representations. Experimental results suggest that the performance of the deep learning models that are trained on a single institutional dataset, are better than rule-based PEFinder on our multi-institutional test sets. The best F1 score for the presence of PE in an adult patient population was 0.99 (DPA-HNN) and for a pediatrics population was 0.99 (HNN) which shows that the deep learning models being trained on adult data, demonstrated generalizability to pediatrics population with comparable accuracy. Our work suggests feasibility of broader usage of neural network models in automated classification of multi-institutional imaging text reports for a variety of applications including evaluation of imaging utilization, imaging yield, clinical decision support tools, and as part of automated classification of large corpus for medical imaging deep learning work.
Collapse
Affiliation(s)
- Imon Banerjee
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA.
| | - Yuan Ling
- Artificial Intelligence Laboratory, Philips Research North America, Cambridge, MA, USA
| | - Matthew C Chen
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Sadid A Hasan
- Artificial Intelligence Laboratory, Philips Research North America, Cambridge, MA, USA
| | - Curtis P Langlotz
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Nathaniel Moradzadeh
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Brian Chapman
- Department of Bioinformatics, University of Utah Medical Center, UT, USA
| | - Timothy Amrhein
- Department of Neuroradiology, Duke University School of Medicine, NC, USA
| | - David Mong
- Department of Radiology, Children Hospital Colorado, CO, USA
| | - Daniel L Rubin
- Department of Biomedical Data Science, Stanford University School of Medicine, Stanford, CA, USA; Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| | - Oladimeji Farri
- Artificial Intelligence Laboratory, Philips Research North America, Cambridge, MA, USA
| | - Matthew P Lungren
- Department of Radiology, Stanford University School of Medicine, Stanford, CA, USA
| |
Collapse
|
46
|
Rajpurkar P, Irvin J, Ball RL, Zhu K, Yang B, Mehta H, Duan T, Ding D, Bagul A, Langlotz CP, Patel BN, Yeom KW, Shpanskaya K, Blankenberg FG, Seekins J, Amrhein TJ, Mong DA, Halabi SS, Zucker EJ, Ng AY, Lungren MP. Deep learning for chest radiograph diagnosis: A retrospective comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 2018; 15:e1002686. [PMID: 30457988 PMCID: PMC6245676 DOI: 10.1371/journal.pmed.1002686] [Citation(s) in RCA: 496] [Impact Index Per Article: 82.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/29/2018] [Accepted: 10/03/2018] [Indexed: 12/14/2022] Open
Abstract
BACKGROUND Chest radiograph interpretation is critical for the detection of thoracic diseases, including tuberculosis and lung cancer, which affect millions of people worldwide each year. This time-consuming task typically requires expert radiologists to read the images, leading to fatigue-based diagnostic error and lack of diagnostic expertise in areas of the world where radiologists are not available. Recently, deep learning approaches have been able to achieve expert-level performance in medical image interpretation tasks, powered by large network architectures and fueled by the emergence of large labeled datasets. The purpose of this study is to investigate the performance of a deep learning algorithm on the detection of pathologies in chest radiographs compared with practicing radiologists. METHODS AND FINDINGS We developed CheXNeXt, a convolutional neural network to concurrently detect the presence of 14 different pathologies, including pneumonia, pleural effusion, pulmonary masses, and nodules in frontal-view chest radiographs. CheXNeXt was trained and internally validated on the ChestX-ray8 dataset, with a held-out validation set consisting of 420 images, sampled to contain at least 50 cases of each of the original pathology labels. On this validation set, the majority vote of a panel of 3 board-certified cardiothoracic specialist radiologists served as reference standard. We compared CheXNeXt's discriminative performance on the validation set to the performance of 9 radiologists using the area under the receiver operating characteristic curve (AUC). The radiologists included 6 board-certified radiologists (average experience 12 years, range 4-28 years) and 3 senior radiology residents, from 3 academic institutions. We found that CheXNeXt achieved radiologist-level performance on 11 pathologies and did not achieve radiologist-level performance on 3 pathologies. The radiologists achieved statistically significantly higher AUC performance on cardiomegaly, emphysema, and hiatal hernia, with AUCs of 0.888 (95% confidence interval [CI] 0.863-0.910), 0.911 (95% CI 0.866-0.947), and 0.985 (95% CI 0.974-0.991), respectively, whereas CheXNeXt's AUCs were 0.831 (95% CI 0.790-0.870), 0.704 (95% CI 0.567-0.833), and 0.851 (95% CI 0.785-0.909), respectively. CheXNeXt performed better than radiologists in detecting atelectasis, with an AUC of 0.862 (95% CI 0.825-0.895), statistically significantly higher than radiologists' AUC of 0.808 (95% CI 0.777-0.838); there were no statistically significant differences in AUCs for the other 10 pathologies. The average time to interpret the 420 images in the validation set was substantially longer for the radiologists (240 minutes) than for CheXNeXt (1.5 minutes). The main limitations of our study are that neither CheXNeXt nor the radiologists were permitted to use patient history or review prior examinations and that evaluation was limited to a dataset from a single institution. CONCLUSIONS In this study, we developed and validated a deep learning algorithm that classified clinically important abnormalities in chest radiographs at a performance level comparable to practicing radiologists. Once tested prospectively in clinical settings, the algorithm could have the potential to expand patient access to chest radiograph diagnostics.
Collapse
Affiliation(s)
- Pranav Rajpurkar
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Jeremy Irvin
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Robyn L. Ball
- Department of Medicine, Quantitative Sciences Unit, Stanford University, Stanford, California, United States of America
| | - Kaylie Zhu
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Brandon Yang
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Hershel Mehta
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Tony Duan
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Daisy Ding
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Aarti Bagul
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Curtis P. Langlotz
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Bhavik N. Patel
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Kristen W. Yeom
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Katie Shpanskaya
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Francis G. Blankenberg
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Jayne Seekins
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Timothy J. Amrhein
- Department of Radiology, Duke University, Durham, North Carolina, United States of America
| | - David A. Mong
- Department of Radiology, University of Colorado, Denver, Colorado, United States of America
| | - Safwan S. Halabi
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Evan J. Zucker
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Andrew Y. Ng
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Matthew P. Lungren
- Department of Radiology, Stanford University, Stanford, California, United States of America
| |
Collapse
|
47
|
Bien N, Rajpurkar P, Ball RL, Irvin J, Park A, Jones E, Bereket M, Patel BN, Yeom KW, Shpanskaya K, Halabi S, Zucker E, Fanton G, Amanatullah DF, Beaulieu CF, Riley GM, Stewart RJ, Blankenberg FG, Larson DB, Jones RH, Langlotz CP, Ng AY, Lungren MP. Deep-learning-assisted diagnosis for knee magnetic resonance imaging: Development and retrospective validation of MRNet. PLoS Med 2018; 15:e1002699. [PMID: 30481176 PMCID: PMC6258509 DOI: 10.1371/journal.pmed.1002699] [Citation(s) in RCA: 281] [Impact Index Per Article: 46.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/02/2018] [Accepted: 10/23/2018] [Indexed: 12/22/2022] Open
Abstract
BACKGROUND Magnetic resonance imaging (MRI) of the knee is the preferred method for diagnosing knee injuries. However, interpretation of knee MRI is time-intensive and subject to diagnostic error and variability. An automated system for interpreting knee MRI could prioritize high-risk patients and assist clinicians in making diagnoses. Deep learning methods, in being able to automatically learn layers of features, are well suited for modeling the complex relationships between medical images and their interpretations. In this study we developed a deep learning model for detecting general abnormalities and specific diagnoses (anterior cruciate ligament [ACL] tears and meniscal tears) on knee MRI exams. We then measured the effect of providing the model's predictions to clinical experts during interpretation. METHODS AND FINDINGS Our dataset consisted of 1,370 knee MRI exams performed at Stanford University Medical Center between January 1, 2001, and December 31, 2012 (mean age 38.0 years; 569 [41.5%] female patients). The majority vote of 3 musculoskeletal radiologists established reference standard labels on an internal validation set of 120 exams. We developed MRNet, a convolutional neural network for classifying MRI series and combined predictions from 3 series per exam using logistic regression. In detecting abnormalities, ACL tears, and meniscal tears, this model achieved area under the receiver operating characteristic curve (AUC) values of 0.937 (95% CI 0.895, 0.980), 0.965 (95% CI 0.938, 0.993), and 0.847 (95% CI 0.780, 0.914), respectively, on the internal validation set. We also obtained a public dataset of 917 exams with sagittal T1-weighted series and labels for ACL injury from Clinical Hospital Centre Rijeka, Croatia. On the external validation set of 183 exams, the MRNet trained on Stanford sagittal T2-weighted series achieved an AUC of 0.824 (95% CI 0.757, 0.892) in the detection of ACL injuries with no additional training, while an MRNet trained on the rest of the external data achieved an AUC of 0.911 (95% CI 0.864, 0.958). We additionally measured the specificity, sensitivity, and accuracy of 9 clinical experts (7 board-certified general radiologists and 2 orthopedic surgeons) on the internal validation set both with and without model assistance. Using a 2-sided Pearson's chi-squared test with adjustment for multiple comparisons, we found no significant differences between the performance of the model and that of unassisted general radiologists in detecting abnormalities. General radiologists achieved significantly higher sensitivity in detecting ACL tears (p-value = 0.002; q-value = 0.019) and significantly higher specificity in detecting meniscal tears (p-value = 0.003; q-value = 0.019). Using a 1-tailed t test on the change in performance metrics, we found that providing model predictions significantly increased clinical experts' specificity in identifying ACL tears (p-value < 0.001; q-value = 0.006). The primary limitations of our study include lack of surgical ground truth and the small size of the panel of clinical experts. CONCLUSIONS Our deep learning model can rapidly generate accurate clinical pathology classifications of knee MRI exams from both internal and external datasets. Moreover, our results support the assertion that deep learning models can improve the performance of clinical experts during medical imaging interpretation. Further research is needed to validate the model prospectively and to determine its utility in the clinical setting.
Collapse
Affiliation(s)
- Nicholas Bien
- Department of Computer Science, Stanford University, Stanford, California, United States of America
- * E-mail:
| | - Pranav Rajpurkar
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Robyn L. Ball
- Quantitative Sciences Unit, Department of Medicine, Stanford University, Stanford, California, United States of America
| | - Jeremy Irvin
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Allison Park
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Erik Jones
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Michael Bereket
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Bhavik N. Patel
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Kristen W. Yeom
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Katie Shpanskaya
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Safwan Halabi
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Evan Zucker
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Gary Fanton
- Department of Orthopedic Surgery, Stanford University, Stanford, California, United States of America
| | - Derek F. Amanatullah
- Department of Orthopedic Surgery, Stanford University, Stanford, California, United States of America
| | - Christopher F. Beaulieu
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Geoffrey M. Riley
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Russell J. Stewart
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Francis G. Blankenberg
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - David B. Larson
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Ricky H. Jones
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Curtis P. Langlotz
- Department of Radiology, Stanford University, Stanford, California, United States of America
| | - Andrew Y. Ng
- Department of Computer Science, Stanford University, Stanford, California, United States of America
| | - Matthew P. Lungren
- Department of Radiology, Stanford University, Stanford, California, United States of America
| |
Collapse
|
48
|
Tan WK, Hassanpour S, Heagerty PJ, Rundell SD, Suri P, Huhdanpaa HT, James K, Carrell DS, Langlotz CP, Organ NL, Meier EN, Sherman KJ, Kallmes DF, Luetmer PH, Griffith B, Nerenz DR, Jarvik JG. Comparison of Natural Language Processing Rules-based and Machine-learning Systems to Identify Lumbar Spine Imaging Findings Related to Low Back Pain. Acad Radiol 2018; 25:1422-1432. [PMID: 29605561 DOI: 10.1016/j.acra.2018.03.008] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [What about the content of this article? (0)] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2018] [Revised: 03/09/2018] [Accepted: 03/09/2018] [Indexed: 12/28/2022]
Abstract
RATIONALE AND OBJECTIVES To evaluate a natural language processing (NLP) system built with open-source tools for identification of lumbar spine imaging findings related to low back pain on magnetic resonance and x-ray radiology reports from four health systems. MATERIALS AND METHODS We used a limited data set (de-identified except for dates) sampled from lumbar spine imaging reports of a prospectively assembled cohort of adults. From N = 178,333 reports, we randomly selected N = 871 to form a reference-standard dataset, consisting of N = 413 x-ray reports and N = 458 MR reports. Using standardized criteria, four spine experts annotated the presence of 26 findings, where 71 reports were annotated by all four experts and 800 were each annotated by two experts. We calculated inter-rater agreement and finding prevalence from annotated data. We randomly split the annotated data into development (80%) and testing (20%) sets. We developed an NLP system from both rule-based and machine-learned models. We validated the system using accuracy metrics such as sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). RESULTS The multirater annotated dataset achieved inter-rater agreement of Cohen's kappa > 0.60 (substantial agreement) for 25 of 26 findings, with finding prevalence ranging from 3% to 89%. In the testing sample, rule-based and machine-learned predictions both had comparable average specificity (0.97 and 0.95, respectively). The machine-learned approach had a higher average sensitivity (0.94, compared to 0.83 for rules-based), and a higher overall AUC (0.98, compared to 0.90 for rules-based). CONCLUSIONS Our NLP system performed well in identifying the 26 lumbar spine findings, as benchmarked by reference-standard annotation by medical experts. Machine-learned models provided substantial gains in model sensitivity with slight loss of specificity, and overall higher AUC.
Collapse
|
49
|
Abstract
Deep learning is a form of machine learning using a convolutional neural network architecture that shows tremendous promise for imaging applications. It is increasingly being adapted from its original demonstration in computer vision applications to medical imaging. Because of the high volume and wealth of multimodal imaging information acquired in typical studies, neuroradiology is poised to be an early adopter of deep learning. Compelling deep learning research applications have been demonstrated, and their use is likely to grow rapidly. This review article describes the reasons, outlines the basic methods used to train and test deep learning models, and presents a brief overview of current and potential clinical applications with an emphasis on how they are likely to change future neuroradiology practice. Facility with these methods among neuroimaging researchers and clinicians will be important to channel and harness the vast potential of this new method.
Collapse
Affiliation(s)
- G Zaharchuk
- From the Departments of Radiology (G.Z., M.W., D.R., C.P.L.)
| | - E Gong
- Electrical Engineering (E.G.), Stanford University and Stanford University Medical Center, Stanford, California
| | - M Wintermark
- From the Departments of Radiology (G.Z., M.W., D.R., C.P.L.)
| | - D Rubin
- From the Departments of Radiology (G.Z., M.W., D.R., C.P.L.)
| | - C P Langlotz
- From the Departments of Radiology (G.Z., M.W., D.R., C.P.L.)
| |
Collapse
|
50
|
Percha B, Zhang Y, Bozkurt S, Rubin D, Altman RB, Langlotz CP. Expanding a radiology lexicon using contextual patterns in radiology reports. J Am Med Inform Assoc 2018; 25:679-685. [PMID: 29329435 PMCID: PMC5978019 DOI: 10.1093/jamia/ocx152] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/25/2017] [Revised: 11/01/2017] [Accepted: 12/18/2017] [Indexed: 11/14/2022] Open
Abstract
Objective Distributional semantics algorithms, which learn vector space representations of words and phrases from large corpora, identify related terms based on contextual usage patterns. We hypothesize that distributional semantics can speed up lexicon expansion in a clinical domain, radiology, by unearthing synonyms from the corpus. Materials and Methods We apply word2vec, a distributional semantics software package, to the text of radiology notes to identify synonyms for RadLex, a structured lexicon of radiology terms. We stratify performance by term category, term frequency, number of tokens in the term, vector magnitude, and the context window used in vector building. Results Ranking candidates based on distributional similarity to a target term results in high curation efficiency: on a ranked list of 775 249 terms, >50% of synonyms occurred within the first 25 terms. Synonyms are easier to find if the target term is a phrase rather than a single word, if it occurs at least 100× in the corpus, and if its vector magnitude is between 4 and 5. Some RadLex categories, such as anatomical substances, are easier to identify synonyms for than others. Discussion The unstructured text of clinical notes contains a wealth of information about human diseases and treatment patterns. However, searching and retrieving information from clinical notes often suffer due to variations in how similar concepts are described in the text. Biomedical lexicons address this challenge, but are expensive to produce and maintain. Distributional semantics algorithms can assist lexicon curation, saving researchers time and money.
Collapse
Affiliation(s)
- Bethany Percha
- Department of Genetics and Genomic Sciences, Icahn School of Medicine at Mount Sinai, New York, NY, USA
- Biomedical Informatics Training Program, Stanford University, Stanford, CA, USA
| | - Yuhao Zhang
- Biomedical Informatics Training Program, Stanford University, Stanford, CA, USA
| | - Selen Bozkurt
- Department of Biostatistics and Medical Informatics, Akdeniz University Faculty of Medicine, Antalya, Turkey
| | - Daniel Rubin
- Department of Radiology, Stanford University, Stanford, CA, USA
| | - Russ B Altman
- Department of Medicine, Stanford University, Stanford, CA, USA
- Department of Genetics and Bioengineering, Stanford University, Stanford, CA, USA
| | | |
Collapse
|