1
|
Peng C, Yang X, Yu Z, Bian J, Hogan WR, Wu Y. Clinical concept and relation extraction using prompt-based machine reading comprehension. J Am Med Inform Assoc 2023; 30:1486-1493. [PMID: 37316988 PMCID: PMC10436141 DOI: 10.1093/jamia/ocad107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 05/08/2023] [Accepted: 06/05/2023] [Indexed: 06/16/2023] Open
Abstract
OBJECTIVE To develop a natural language processing system that solves both clinical concept extraction and relation extraction in a unified prompt-based machine reading comprehension (MRC) architecture with good generalizability for cross-institution applications. METHODS We formulate both clinical concept extraction and relation extraction using a unified prompt-based MRC architecture and explore state-of-the-art transformer models. We compare our MRC models with existing deep learning models for concept extraction and end-to-end relation extraction using 2 benchmark datasets developed by the 2018 National NLP Clinical Challenges (n2c2) challenge (medications and adverse drug events) and the 2022 n2c2 challenge (relations of social determinants of health [SDoH]). We also evaluate the transfer learning ability of the proposed MRC models in a cross-institution setting. We perform error analyses and examine how different prompting strategies affect the performance of MRC models. RESULTS AND CONCLUSION The proposed MRC models achieve state-of-the-art performance for clinical concept and relation extraction on the 2 benchmark datasets, outperforming previous non-MRC transformer models. GatorTron-MRC achieves the best strict and lenient F1-scores for concept extraction, outperforming previous deep learning models on the 2 datasets by 1%-3% and 0.7%-1.3%, respectively. For end-to-end relation extraction, GatorTron-MRC and BERT-MIMIC-MRC achieve the best F1-scores, outperforming previous deep learning models by 0.9%-2.4% and 10%-11%, respectively. For cross-institution evaluation, GatorTron-MRC outperforms traditional GatorTron by 6.4% and 16% for the 2 datasets, respectively. The proposed method is better at handling nested/overlapped concepts, extracting relations, and has good portability for cross-institute applications. Our clinical MRC package is publicly available at https://github.com/uf-hobi-informatics-lab/ClinicalTransformerMRC.
Collapse
Affiliation(s)
- Cheng Peng
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA
| | - Xi Yang
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA
| | - Zehao Yu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA
| | - Jiang Bian
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA
| | - William R Hogan
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA
| | - Yonghui Wu
- Department of Health Outcomes and Biomedical Informatics, College of Medicine, University of Florida, Gainesville, Florida, USA
- Cancer Informatics Shared Resource, University of Florida Health Cancer Center, Gainesville, Florida, USA
| |
Collapse
|
2
|
Mueller B, Kinoshita T, Peebles A, Graber MA, Lee S. Artificial intelligence and machine learning in emergency medicine: a narrative review. Acute Med Surg 2022; 9:e740. [PMID: 35251669 PMCID: PMC8887797 DOI: 10.1002/ams2.740] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2021] [Revised: 01/26/2022] [Accepted: 02/06/2022] [Indexed: 12/20/2022] Open
Abstract
AIM The emergence and evolution of artificial intelligence (AI) has generated increasing interest in machine learning applications for health care. Specifically, researchers are grasping the potential of machine learning solutions to enhance the quality of care in emergency medicine. METHODS We undertook a narrative review of published works on machine learning applications in emergency medicine and provide a synopsis of recent developments. RESULTS This review describes fundamental concepts of machine learning and presents clinical applications for triage, risk stratification specific to disease, medical imaging, and emergency department operations. Additionally, we consider how machine learning models could contribute to the improvement of causal inference in medicine, and to conclude, we discuss barriers to safe implementation of AI. CONCLUSION We intend that this review serves as an introduction to AI and machine learning in emergency medicine.
Collapse
Affiliation(s)
- Brianna Mueller
- Department of Business Analytics The University of Iowa Tippie College of Business Iowa City Iowa USA
| | | | - Alexander Peebles
- Department of Emergency Medicine The University of Iowa Carver College of Medicine Iowa City Iowa USA
| | - Mark A Graber
- Department of Emergency Medicine The University of Iowa Carver College of Medicine Iowa City Iowa USA
| | - Sangil Lee
- Department of Emergency Medicine The University of Iowa Carver College of Medicine Iowa City Iowa USA
| |
Collapse
|
3
|
Lee K, Dobbins NJ, McInnes B, Yetisgen M, Uzuner Ö. Transferability of neural network clinical deidentification systems. J Am Med Inform Assoc 2021; 28:2661-2669. [PMID: 34586386 DOI: 10.1093/jamia/ocab207] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2021] [Revised: 07/19/2021] [Accepted: 09/10/2021] [Indexed: 11/14/2022] Open
Abstract
OBJECTIVE Neural network deidentification studies have focused on individual datasets. These studies assume the availability of a sufficient amount of human-annotated data to train models that can generalize to corresponding test data. In real-world situations, however, researchers often have limited or no in-house training data. Existing systems and external data can help jump-start deidentification on in-house data; however, the most efficient way of utilizing existing systems and external data is unclear. This article investigates the transferability of a state-of-the-art neural clinical deidentification system, NeuroNER, across a variety of datasets, when it is modified architecturally for domain generalization and when it is trained strategically for domain transfer. MATERIALS AND METHODS We conducted a comparative study of the transferability of NeuroNER using 4 clinical note corpora with multiple note types from 2 institutions. We modified NeuroNER architecturally to integrate 2 types of domain generalization approaches. We evaluated each architecture using 3 training strategies. We measured transferability from external sources; transferability across note types; the contribution of external source data when in-domain training data are available; and transferability across institutions. RESULTS AND CONCLUSIONS Transferability from a single external source gave inconsistent results. Using additional external sources consistently yielded an F1-score of approximately 80%. Fine-tuning emerged as a dominant transfer strategy, with or without domain generalization. We also found that external sources were useful even in cases where in-domain training data were available. Transferability across institutions differed by note type and annotation label but resulted in improved performance.
Collapse
Affiliation(s)
- Kahyun Lee
- Department of Information Science and Technology, George Mason University, Fairfax, Virginia, USA
| | - Nicholas J Dobbins
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, Washington, USA
| | - Bridget McInnes
- Department of Computer Science, Virginia Commonwealth University, Richmond, Virginia, USA
| | - Meliha Yetisgen
- Department of Biomedical Informatics and Medical Education, University of Washington, Seattle, Washington, USA
| | - Özlem Uzuner
- Department of Information Science and Technology, George Mason University, Fairfax, Virginia, USA
| |
Collapse
|
4
|
Santiso S, Pérez A, Casillas A. Adverse Drug Reaction extraction: Tolerance to entity recognition errors and sub-domain variants. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 199:105891. [PMID: 33333368 DOI: 10.1016/j.cmpb.2020.105891] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/17/2020] [Accepted: 11/24/2020] [Indexed: 06/12/2023]
Affiliation(s)
- Sara Santiso
- IXA research group, University of the Basque Country (UPV/EHU) Manuel Lardizabal 1, 20080 Donostia, Spain.
| | - Alicia Pérez
- IXA research group, University of the Basque Country (UPV/EHU) Manuel Lardizabal 1, 20080 Donostia, Spain.
| | - Arantza Casillas
- IXA research group, University of the Basque Country (UPV/EHU) Manuel Lardizabal 1, 20080 Donostia, Spain.
| |
Collapse
|
5
|
Burke PC, Shirley RB, Raciniewski J, Simon JF, Wyllie R, Fraser TG. Development and Evaluation of a Fully Automated Surveillance System for Influenza-Associated Hospitalization at a Multihospital Health System in Northeast Ohio. Appl Clin Inform 2020; 11:564-569. [PMID: 32851617 DOI: 10.1055/s-0040-1715651] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022] Open
Abstract
BACKGROUND Performing high-quality surveillance for influenza-associated hospitalization (IAH) is challenging, time-consuming, and essential. OBJECTIVES Our objectives were to develop a fully automated surveillance system for laboratory-confirmed IAH at our multihospital health system, to evaluate the performance of the automated system during the 2018 to 2019 influenza season at eight hospitals by comparing its sensitivity and positive predictive value to that of manual surveillance, and to estimate the time and cost savings associated with reliance on the automated surveillance system. METHODS Infection preventionists (IPs) perform manual surveillance for IAH by reviewing laboratory records and making a determination about each result. For automated surveillance, we programmed a query against our Enterprise Data Vault (EDV) for cases of IAH. The EDV query was established as a dynamic data source to feed our data visualization software, automatically updating every 24 hours.To establish a gold standard of cases of IAH against which to evaluate the performance of manual and automated surveillance systems, we generated a master list of possible IAH by querying four independent information systems. We reviewed medical records and adjudicated whether each possible case represented a true case of IAH. RESULTS We found 844 true cases of IAH, 577 (68.4%) of which were detected by the manual system and 774 (91.7%) of which were detected by the automated system. The positive predictive values of the manual and automated systems were 89.3 and 88.3%, respectively.Relying on the automated surveillance system for IAH resulted in an average recoup of 82 minutes per day for each IP and an estimated system-wide payroll redirection of $32,880 over the four heaviest weeks of influenza activity. CONCLUSION Surveillance for IAH can be entirely automated at multihospital health systems, saving time, and money while improving case detection.
Collapse
Affiliation(s)
- Patrick C Burke
- Department of Infection Prevention, Enterprise Quality and Patient Safety, Cleveland Clinic, Cleveland, Ohio, United States
| | - Rachel Benish Shirley
- Enterprise Quality and Patient Safety, Cleveland Clinic, Cleveland, Ohio, United States
| | - Jacob Raciniewski
- Department of Enterprise Analytics, Cleveland Clinic, Cleveland, Ohio, United States
| | - James F Simon
- Medical Operations Department, Cleveland Clinic, Cleveland, Ohio, United States
| | - Robert Wyllie
- Medical Operations Department, Cleveland Clinic, Cleveland, Ohio, United States
| | - Thomas G Fraser
- Department of Infectious Diseases, Cleveland Clinic, Cleveland, Ohio, United States
| |
Collapse
|
6
|
Feller DJ, Bear Don't Walk Iv OJ, Zucker J, Yin MT, Gordon P, Elhadad N. Detecting Social and Behavioral Determinants of Health with Structured and Free-Text Clinical Data. Appl Clin Inform 2020; 11:172-181. [PMID: 32131117 DOI: 10.1055/s-0040-1702214] [Citation(s) in RCA: 38] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022] Open
Abstract
BACKGROUND Social and behavioral determinants of health (SBDH) are environmental and behavioral factors that often impede disease management and result in sexually transmitted infections. Despite their importance, SBDH are inconsistently documented in electronic health records (EHRs) and typically collected only in an unstructured format. Evidence suggests that structured data elements present in EHRs can contribute further to identify SBDH in the patient record. OBJECTIVE Explore the automated inference of both the presence of SBDH documentation and individual SBDH risk factors in patient records. Compare the relative ability of clinical notes and structured EHR data, such as laboratory measurements and diagnoses, to support inference. METHODS We attempt to infer the presence of SBDH documentation in patient records, as well as patient status of 11 SBDH, including alcohol abuse, homelessness, and sexual orientation. We compare classification performance when considering clinical notes only, structured data only, and notes and structured data together. We perform an error analysis across several SBDH risk factors. RESULTS Classification models inferring the presence of SBDH documentation achieved good performance (F1 score: 92.7-78.7; F1 considered as the primary evaluation metric). Performance was variable for models inferring patient SBDH risk status; results ranged from F1 = 82.7 for LGBT (lesbian, gay, bisexual, and transgender) status to F1 = 28.5 for intravenous drug use. Error analysis demonstrated that lexical diversity and documentation of historical SBDH status challenge inference of patient SBDH status. Three of five classifiers inferring topic-specific SBDH documentation and 10 of 11 patient SBDH status classifiers achieved highest performance when trained using both clinical notes and structured data. CONCLUSION Our findings suggest that combining clinical free-text notes and structured data provide the best approach in classifying patient SBDH status. Inferring patient SBDH status is most challenging among SBDH with low prevalence and high lexical diversity.
Collapse
Affiliation(s)
- Daniel J Feller
- Department of Biomedical Informatics, Columbia University, New York, New York, United States
| | | | - Jason Zucker
- Division of Infectious Diseases, Department of Internal Medicine, Columbia University Irving Medical Center, New York, New York, United States
| | - Michael T Yin
- Division of Infectious Diseases, Department of Internal Medicine, Columbia University Irving Medical Center, New York, New York, United States
| | - Peter Gordon
- Division of Infectious Diseases, Department of Internal Medicine, Columbia University Irving Medical Center, New York, New York, United States
| | - Noémie Elhadad
- Department of Biomedical Informatics, Columbia University, New York, New York, United States
| |
Collapse
|
7
|
Horng S, Greenbaum NR, Nathanson LA, McClay JC, Goss FR, Nielson JA. Consensus Development of a Modern Ontology of Emergency Department Presenting Problems-The Hierarchical Presenting Problem Ontology (HaPPy). Appl Clin Inform 2019; 10:409-420. [PMID: 31189204 DOI: 10.1055/s-0039-1691842] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022] Open
Abstract
OBJECTIVE Numerous attempts have been made to create a standardized "presenting problem" or "chief complaint" list to characterize the nature of an emergency department visit. Previous attempts have failed to gain widespread adoption as they were not freely shareable or did not contain the right level of specificity, structure, and clinical relevance to gain acceptance by the larger emergency medicine community. Using real-world data, we constructed a presenting problem list that addresses these challenges. MATERIALS AND METHODS We prospectively captured the presenting problems for 180,424 consecutive emergency department patient visits at an urban, academic, Level I trauma center in the Boston metro area. No patients were excluded. We used a consensus process to iteratively derive our system using real-world data. We used the first 70% of consecutive visits to derive our ontology, followed by a 6-month washout period, and the remaining 30% for validation. All concepts were mapped to Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT). RESULTS Our system consists of a polyhierarchical ontology containing 692 unique concepts, 2,118 synonyms, and 30,613 nonvisible descriptions to correct misspellings and nonstandard terminology. Our ontology successfully captured structured data for 95.9% of visits in our validation data set. DISCUSSION AND CONCLUSION We present the HierArchical Presenting Problem ontologY (HaPPy). This ontology was empirically derived and then iteratively validated by an expert consensus panel. HaPPy contains 692 presenting problem concepts, each concept being mapped to SNOMED CT. This freely sharable ontology can help to facilitate presenting problem-based quality metrics, research, and patient care.
Collapse
Affiliation(s)
- Steven Horng
- Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States.,Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States
| | - Nathaniel R Greenbaum
- Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States.,Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States
| | - Larry A Nathanson
- Division of Clinical Informatics, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States.,Department of Emergency Medicine, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, Massachusetts, United States
| | - James C McClay
- Department of Emergency Medicine, College of Medicine, University of Nebraska Medical Center, Omaha, Nebraska, United States
| | - Foster R Goss
- Department of Emergency Medicine, University of Colorado Hospital, University of Colorado School of Medicine, Aurora, Colorado, United States
| | - Jeffrey A Nielson
- Northeastern Ohio Medical University, University Hospitals Samaritan Medical Center, Ashland, Ohio, United States
| |
Collapse
|