1
|
Jeffery AD, Fabbri D, Reeves RM, Matheny ME. Use of noisy labels as weak learners to identify incompletely ascertainable outcomes: A Feasibility study with opioid-induced respiratory depression. Heliyon 2024; 10:e26434. [PMID: 38444495 PMCID: PMC10912240 DOI: 10.1016/j.heliyon.2024.e26434] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Revised: 02/09/2024] [Accepted: 02/13/2024] [Indexed: 03/07/2024] Open
Abstract
Objective Assigning outcome labels to large observational data sets in a timely and accurate manner, particularly when outcomes are rare or not directly ascertainable, remains a significant challenge within biomedical informatics. We examined whether noisy labels generated from subject matter experts' heuristics using heterogenous data types within a data programming paradigm could provide outcomes labels to a large, observational data set. We chose the clinical condition of opioid-induced respiratory depression for our use case because it is rare, has no administrative codes to easily identify the condition, and typically requires at least some unstructured text to ascertain its presence. Materials and methods Using de-identified electronic health records of 52,861 post-operative encounters, we applied a data programming paradigm (implemented in the Snorkel software) for the development of a machine learning classifier for opioid-induced respiratory depression. Our approach included subject matter experts creating 14 labeling functions that served as noisy labels for developing a probabilistic Generative model. We used probabilistic labels from the Generative model as outcome labels for training a Discriminative model on the source data. We evaluated performance of the Discriminative model with a hold-out test set of 599 independently-reviewed patient records. Results The final Discriminative classification model achieved an accuracy of 0.977, an F1 score of 0.417, a sensitivity of 1.0, and an AUC of 0.988 in the hold-out test set with a prevalence of 0.83% (5/599). Discussion All of the confirmed Cases were identified by the classifier. For rare outcomes, this finding is encouraging because it reduces the number of manual reviews needed by excluding visits/patients with low probabilities. Conclusion Application of a data programming paradigm with expert-informed labeling functions might have utility for phenotyping clinical phenomena that are not easily ascertainable from highly-structured data.
Collapse
Affiliation(s)
- Alvin D. Jeffery
- Vanderbilt University School of Nursing, Nashville, TN, USA
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
- Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA
| | - Daniel Fabbri
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Ruth M. Reeves
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
- Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA
| | - Michael E. Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
- Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA
| |
Collapse
|
2
|
Meerwijk EL, Jones GA, Shotqara AS, Reyes S, Tamang SR, Eddington HS, Reeves RM, Finlay AK, Harris AHS. Development of a 3-Step theory of suicide ontology to facilitate 3ST factor extraction from clinical progress notes. J Biomed Inform 2024; 150:104582. [PMID: 38160758 DOI: 10.1016/j.jbi.2023.104582] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2023] [Revised: 11/21/2023] [Accepted: 12/22/2023] [Indexed: 01/03/2024]
Abstract
OBJECTIVE Suicide risk prediction algorithms at the Veterans Health Administration (VHA) do not include predictors based on the 3-Step Theory of suicide (3ST), which builds on hopelessness, psychological pain, connectedness, and capacity for suicide. These four factors are not available from structured fields in VHA electronic health records, but they are found in unstructured clinical text. An ontology and controlled vocabulary that maps psychosocial and behavioral terms to these factors does not exist. The objectives of this study were 1) to develop an ontology with a controlled vocabulary of terms that map onto classes that represent the 3ST factors as identified within electronic clinical progress notes, and 2) to determine the accuracy of automated extractions based on terms in the controlled vocabulary. METHODS A team of four annotators did linguistic annotation of 30,000 clinical progress notes from 231 Veterans in VHA electronic health records who attempted suicide or who died by suicide for terms relating to the 3ST factors. Annotation involved manually assigning a label to words or phrases that indicated presence or absence of the factor (polarity). These words and phrases were entered into a controlled vocabulary that was then used by our computational system to tag 14 million clinical progress notes from Veterans who attempted or died by suicide after 2013. Tagged text was extracted and machine-labelled for presence or absence of the 3ST factors. Accuracy of these machine-labels was determined for 1000 randomly selected extractions for each factor against a ground truth created by our annotators. RESULTS Linguistic annotation identified 8486 terms that related to 33 subclasses across the four factors and polarities. Precision of machine-labeled extractions ranged from 0.73 to 1.00 for most factor-polarity combinations, whereas recall was somewhat lower 0.65-0.91. CONCLUSION The ontology that was developed consists of classes that represent each of the four 3ST factors, subclasses, relationships, and terms that map onto those classes which are stored in a controlled vocabulary (https://bioportal.bioontology.org/ontologies/THREE-ST). The use case that we present shows how scores based on clinical notes tagged for terms in the controlled vocabulary capture meaningful change in the 3ST factors during weeks preceding a suicidal event.
Collapse
Affiliation(s)
- Esther L Meerwijk
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA.
| | - Gabrielle A Jones
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA
| | - Asqar S Shotqara
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA
| | - Sofia Reyes
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA
| | - Suzanne R Tamang
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA; Department of Medicine, Stanford University, Stanford, CA, USA
| | - Hyrum S Eddington
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA; Department of Surgery, Stanford University, Stanford, CA, USA
| | - Ruth M Reeves
- VA Tennessee Valley Healthcare System, Nashville, TN, USA; Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Andrea K Finlay
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA; VA National Center on Homelessness Among Veterans, USA; Schar School of Policy and Government, George Mason University, Arlington, VA, USA
| | - Alex H S Harris
- VA Health Services Research & Development, Center for Innovation to Implementation (Ci2i), VA Palo Alto Health Care System, Menlo Park, CA, USA; Department of Surgery, Stanford University, Stanford, CA, USA
| |
Collapse
|
3
|
Jeffery AD, Fabbri D, Reeves RM, Matheny ME. Use of Noisy Labels as Weak Learners to Identify Incompletely Ascertainable Outcomes: A Feasibility Study with Opioid-Induced Respiratory Depression. medRxiv 2024:2024.01.29.24301963. [PMID: 38352435 PMCID: PMC10863026 DOI: 10.1101/2024.01.29.24301963] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/19/2024]
Abstract
Objective Assigning outcome labels to large observational data sets in a timely and accurate manner, particularly when outcomes are rare or not directly ascertainable, remains a significant challenge within biomedical informatics. We examined whether noisy labels generated from subject matter experts' heuristics using heterogenous data types within a data programming paradigm could provide outcomes labels to a large, observational data set. We chose the clinical condition of opioid-induced respiratory depression for our use case because it is rare, has no administrative codes to easily identify the condition, and typically requires at least some unstructured text to ascertain its presence. Materials and Methods Using de-identified electronic health records of 52,861 post-operative encounters, we applied a data programming paradigm (implemented in the Snorkel software) for the development of a machine learning classifier for opioid-induced respiratory depression. Our approach included subject matter experts creating 14 labeling functions that served as noisy labels for developing a probabilistic Generative model. We used probabilistic labels from the Generative model as outcome labels for training a Discriminative model on the source data. We evaluated performance of the Discriminative model with a hold-out test set of 599 independently-reviewed patient records. Results The final Discriminative classification model achieved an accuracy of 0.977, an F1 score of 0.417, a sensitivity of 1.0, and an AUC of 0.988 in the hold-out test set with a prevalence of 0.83% (5/599). Discussion All of the confirmed Cases were identified by the classifier. For rare outcomes, this finding is encouraging because it reduces the number of manual reviews needed by excluding visits/patients with low probabilities. Conclusion Application of a data programming paradigm with expert-informed labeling functions might have utility for phenotyping clinical phenomena that are not easily ascertainable from highly-structured data.
Collapse
Affiliation(s)
- Alvin D Jeffery
- School of Nursing, Vanderbilt University, Department of Biomedical Informatics, Vanderbilt University Medical Center, Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA
| | - Daniel Fabbri
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Ruth M Reeves
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA
| | - Michael E Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Tennessee Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA
| |
Collapse
|
4
|
Keloth VK, Banda JM, Gurley M, Heider PM, Kennedy G, Liu H, Liu F, Miller T, Natarajan K, V Patterson O, Peng Y, Raja K, Reeves RM, Rouhizadeh M, Shi J, Wang X, Wang Y, Wei WQ, Williams AE, Zhang R, Belenkaya R, Reich C, Blacketer C, Ryan P, Hripcsak G, Elhadad N, Xu H. Representing and utilizing clinical textual data for real world studies: An OHDSI approach. J Biomed Inform 2023; 142:104343. [PMID: 36935011 PMCID: PMC10428170 DOI: 10.1016/j.jbi.2023.104343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Revised: 01/21/2023] [Accepted: 03/13/2023] [Indexed: 03/19/2023]
Abstract
Clinical documentation in electronic health records contains crucial narratives and details about patients and their care. Natural language processing (NLP) can unlock the information conveyed in clinical notes and reports, and thus plays a critical role in real-world studies. The NLP Working Group at the Observational Health Data Sciences and Informatics (OHDSI) consortium was established to develop methods and tools to promote the use of textual data and NLP in real-world observational studies. In this paper, we describe a framework for representing and utilizing textual data in real-world evidence generation, including representations of information from clinical text in the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), the workflow and tools that were developed to extract, transform and load (ETL) data from clinical notes into tables in OMOP CDM, as well as current applications and specific use cases of the proposed OHDSI NLP solution at large consortia and individual institutions with English textual data. Challenges faced and lessons learned during the process are also discussed to provide valuable insights for researchers who are planning to implement NLP solutions in real-world studies.
Collapse
Affiliation(s)
- Vipina K Keloth
- Section of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT, USA
| | - Juan M Banda
- Department of Computer Science, Georgia State University, Atlanta, GA, USA
| | - Michael Gurley
- Lurie Cancer Center, Northwestern University, Chicago, Illinois, USA
| | - Paul M Heider
- Biomedical Informatics Center, Medical University of South Carolina, Charleston, SC, USA
| | - Georgina Kennedy
- Ingham Institute for Applied Medical Research, Sydney, Australia
| | - Hongfang Liu
- Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, MN, USA
| | - Feifan Liu
- Department of Population and Quantitative Health Sciences, University of Massachusetts Chan Medical School, Worcester, MA, USA
| | - Timothy Miller
- Computational Health Informatics Program, Boston Children's Hospital, and Department of Pediatrics, Harvard Medical School, Boston, MA, USA
| | - Karthik Natarajan
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, NY, USA
| | - Olga V Patterson
- VA Informatics and Computing Infrastructure, Department of Veterans Affairs Salt Lake City Health Care System, Salt Lake City, Utah, USA; Division of Epidemiology, Department of Internal Medicine, School of Medicine, University of Utah, Salt Lake City, Utah, USA; Verily Life Sciences, Mountain View, CA, USA
| | - Yifan Peng
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA
| | - Kalpana Raja
- Section of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT, USA
| | - Ruth M Reeves
- TN Valley Healthcare System, U.S. Department of Veterans Affairs, Nashville, TN, USA; Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Masoud Rouhizadeh
- Department of Pharmaceutical Outcomes & Policy, University of Florida, Gainesville, FL, USA; Biomedical Informatics and Data Science, Johns Hopkins University, Baltimore, MD, USA
| | - Jianlin Shi
- VA Informatics and Computing Infrastructure, Department of Veterans Affairs Salt Lake City Health Care System, Salt Lake City, Utah, USA; Division of Epidemiology, Department of Internal Medicine, School of Medicine, University of Utah, Salt Lake City, Utah, USA; Department of Biomedical Informatics, University of Utah, Salt Lake City, USA
| | - Xiaoyan Wang
- Sema4 Mount Sinai Genomics Incorporation, Stamford, CT, USA
| | - Yanshan Wang
- Department of Health Information Management, Department of Biomedical Informatics, and Intelligent Systems Program, University of Pittsburgh, Pittsburgh, PA, USA
| | - Wei-Qi Wei
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, USA
| | | | - Rui Zhang
- Institute for Health Informatics, and Department of Pharmaceutical Care & Health Systems, University of Minnesota, Minneapolis, MN, USA
| | | | | | - Clair Blacketer
- Janssen Pharmaceutical Research and Development LLC, Titusville, NJ, USA; Department of Medical Informatics, Erasmus University Medical Center, Rotterdam, the Netherlands
| | - Patrick Ryan
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, NY, USA; Janssen Pharmaceutical Research and Development LLC, Titusville, NJ, USA
| | - George Hripcsak
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, NY, USA
| | - Noémie Elhadad
- Department of Biomedical Informatics, Columbia University Irving Medical Center, New York, NY, USA.
| | - Hua Xu
- Section of Biomedical Informatics and Data Science, Yale School of Medicine, Yale University, New Haven, CT, USA.
| |
Collapse
|
5
|
Meerwijk EL, Tamang SR, Finlay AK, Ilgen MA, Reeves RM, Harris AHS. Suicide theory-guided natural language processing of clinical progress notes to improve prediction of veteran suicide risk: protocol for a mixed-method study. BMJ Open 2022; 12:e065088. [PMID: 36002210 PMCID: PMC9413184 DOI: 10.1136/bmjopen-2022-065088] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Accepted: 08/02/2022] [Indexed: 11/17/2022] Open
Abstract
INTRODUCTION The state-of-the-art 3-step Theory of Suicide (3ST) describes why people consider suicide and who will act on their suicidal thoughts and attempt suicide. The central concepts of 3ST-psychological pain, hopelessness, connectedness, and capacity for suicide-are among the most important drivers of suicidal behaviour but they are missing from clinical suicide risk prediction models in use at the US Veterans Health Administration (VHA). These four concepts are not systematically recorded in structured fields of VHA's electronic healthcare records. Therefore, this study will develop a domain-specific ontology that will enable automated extraction of these concepts from clinical progress notes using natural language processing (NLP), and test whether NLP-based predictors for these concepts improve accuracy of existing VHA suicide risk prediction models. METHODS AND ANALYSIS Our mixed-method study has an exploratory sequential design where a qualitative component (aim 1) will inform quantitative analyses (aims 2 and 3). For aim 1, subject matter experts will manually annotate progress notes of clinical encounters with veterans who attempted or died by suicide to develop a domain-specific ontology for the 3ST concepts. During aim 2, we will use NLP to machine-annotate clinical progress notes and derive longitudinal representations for each patient with respect to the presence and intensity of hopelessness, psychological pain, connectedness and capacity for suicide in temporal proximity of suicide attempts and deaths by suicide. These longitudinal representations will be evaluated during aim 3 for their ability to improve existing VHA prediction models of suicide and suicide attempts, STORM (Stratification Tool for Opioid Risk Mitigation) and REACHVET (Recovery Engagement and Coordination for Health - Veterans Enhanced Treatment). ETHICS AND DISSEMINATION Ethics approval for this study was granted by the Stanford University Institutional Review Board and the Research and Development Committee of the VA Palo Alto Health Care System. Results of the study will be disseminated through several outlets, including peer-reviewed publications and presentations at national conferences.
Collapse
Affiliation(s)
- Esther Lydia Meerwijk
- VA Health Services Research & Development, Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California, USA
| | - Suzanne R Tamang
- VA Health Services Research & Development, Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California, USA
- Department of Biomedical Data Science, Stanford University, Stanford, California, USA
| | - Andrea K Finlay
- VA Health Services Research & Development, Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California, USA
- Schar School of Policy and Government, George Mason University, Arlington, Virginia, USA
- VA National Center on Homelessness Among Veterans, Durham, North Carolina, USA
| | - Mark A Ilgen
- Department of Psychiatry, University of Michigan, Ann Arbor, Michigan, USA
- VA Health Services Research & Development, Center for Clinical Management Research, VA Ann Arbor Health Care System, Ann Arbor, Michigan, USA
| | - Ruth M Reeves
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
- VA Health Sevices Research & Development, VA Tennessee Valley Health Care System, Nashville, Tennessee, USA
| | - Alex H S Harris
- VA Health Services Research & Development, Center for Innovation to Implementation, VA Palo Alto Health Care System, Palo Alto, California, USA
- Stanford-Surgical Policy Improvement Research and Education Center, Stanford University School of Medicine, Stanford, California, USA
| |
Collapse
|
6
|
Brown JR, Ricket IM, Reeves RM, Shah RU, Goodrich CA, Gobbel G, Stabler ME, Perkins AM, Minter F, Cox KC, Dorn C, Denton J, Bray BE, Gouripeddi R, Higgins J, Chapman WW, MacKenzie T, Matheny ME. Information Extraction From Electronic Health Records to Predict Readmission Following Acute Myocardial Infarction: Does Natural Language Processing Using Clinical Notes Improve Prediction of Readmission? J Am Heart Assoc 2022; 11:e024198. [PMID: 35322668 PMCID: PMC9075435 DOI: 10.1161/jaha.121.024198] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
Background Social risk factors influence rehospitalization rates yet are challenging to incorporate into prediction models. Integration of social risk factors using natural language processing (NLP) and machine learning could improve risk prediction of 30‐day readmission following an acute myocardial infarction. Methods and Results Patients were enrolled into derivation and validation cohorts. The derivation cohort included inpatient discharges from Vanderbilt University Medical Center between January 1, 2007, and December 31, 2016, with a primary diagnosis of acute myocardial infarction, who were discharged alive, and not transferred from another facility. The validation cohort included patients from Dartmouth‐Hitchcock Health Center between April 2, 2011, and December 31, 2016, meeting the same eligibility criteria described above. Data from both sites were linked to Centers for Medicare & Medicaid Services administrative data to supplement 30‐day hospital readmissions. Clinical notes from each cohort were extracted, and an NLP model was deployed, counting mentions of 7 social risk factors. Five machine learning models were run using clinical and NLP‐derived variables. Model discrimination and calibration were assessed, and receiver operating characteristic comparison analyses were performed. The 30‐day rehospitalization rates among the derivation (n=6165) and validation (n=4024) cohorts were 15.1% (n=934) and 10.2% (n=412), respectively. The derivation models demonstrated no statistical improvement in model performance with the addition of the selected NLP‐derived social risk factors. Conclusions Social risk factors extracted using NLP did not significantly improve 30‐day readmission prediction among hospitalized patients with acute myocardial infarction. Alternative methods are needed to capture social risk factors.
Collapse
Affiliation(s)
- Jeremiah R Brown
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Iben M Ricket
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Ruth M Reeves
- Department of Biomedical Informatics Vanderbilt University Medical Center Nashville TN.,Geriatric Research Education and Clinical Care Center Tennessee Valley Healthcare System VA Nashville TN
| | - Rashmee U Shah
- Division of Cardiovascular Medicine University of Utah School of Medicine Salt Lake City UT
| | - Christine A Goodrich
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Glen Gobbel
- Department of Biomedical Informatics Vanderbilt University Medical Center Nashville TN.,Geriatric Research Education and Clinical Care Center Tennessee Valley Healthcare System VA Nashville TN.,Department of Biostatistics Vanderbilt University Medical Center Nashville TN.,Division of General Internal Medicine Vanderbilt University Medical Center Nashville TN
| | - Meagan E Stabler
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Amy M Perkins
- Geriatric Research Education and Clinical Care Center Tennessee Valley Healthcare System VA Nashville TN.,Department of Biostatistics Vanderbilt University Medical Center Nashville TN
| | - Freneka Minter
- Department of Biomedical Informatics Vanderbilt University Medical Center Nashville TN
| | - Kevin C Cox
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Chad Dorn
- Department of Biomedical Informatics Vanderbilt University Medical Center Nashville TN
| | - Jason Denton
- Department of Biomedical Informatics Vanderbilt University Medical Center Nashville TN
| | - Bruce E Bray
- Division of General Internal Medicine Vanderbilt University Medical Center Nashville TN.,Department of Biomedical Informatics University of Utah School of Medicine Salt Lake City UT
| | - Ramkiran Gouripeddi
- Department of Biomedical Informatics University of Utah School of Medicine Salt Lake City UT.,Utah Clinical & Translational Science InstituteUniversity of Utah Salt Lake City UT
| | - John Higgins
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Wendy W Chapman
- Centre for Digital Transformation of Health University of Melbourne Melbourne Victoria Australia
| | - Todd MacKenzie
- Departments of Epidemiology and Biomedical Data Science Dartmouth Geisel School of Medicine Hanover NH
| | - Michael E Matheny
- Department of Biomedical Informatics Vanderbilt University Medical Center Nashville TN.,Geriatric Research Education and Clinical Care Center Tennessee Valley Healthcare System VA Nashville TN.,Department of Biostatistics Vanderbilt University Medical Center Nashville TN.,Division of General Internal Medicine Vanderbilt University Medical Center Nashville TN
| |
Collapse
|
7
|
Reeves RM, Christensen L, Brown JR, Conway M, Levis M, Gobbel GT, Shah RU, Goodrich C, Ricket I, Minter F, Bohm A, Bray BE, Matheny ME, Chapman W. Adaptation of an NLP system to a new healthcare environment to identify social determinants of health. J Biomed Inform 2021; 120:103851. [PMID: 34174396 DOI: 10.1016/j.jbi.2021.103851] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2021] [Revised: 06/16/2021] [Accepted: 06/21/2021] [Indexed: 11/18/2022]
Abstract
Social determinants of health (SDoH) are increasingly important factors for population health, healthcare outcomes, and care delivery. However, many of these factors are not reliably captured within structured electronic health record (EHR) data. In this work, we evaluated and adapted a previously published NLP tool to include additional social risk factors for deployment at Vanderbilt University Medical Center in an Acute Myocardial Infarction cohort. We developed a transformation of the SDoH outputs of the tool into the OMOP common data model (CDM) for re-use across many potential use cases, yielding performance measures across 8 SDoH classes of precision 0.83 recall 0.74 and F-measure of 0.78.
Collapse
Affiliation(s)
- Ruth M Reeves
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States; Geriatric Research Education and Clinical Care Center, Tennessee Valley Healthcare System VA, Nashville, TN, United States.
| | - Lee Christensen
- Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT, United States
| | - Jeremiah R Brown
- Department of Epidemiology and Biomedical Data Science, Dartmouth Geisel School of Medicine, Hanover, NH, United States
| | - Michael Conway
- Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT, United States
| | - Maxwell Levis
- Department of Epidemiology and Biomedical Data Science, Dartmouth Geisel School of Medicine, Hanover, NH, United States
| | - Glenn T Gobbel
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States; Division of General Internal Medicine, Vanderbilt University Medical Center, Nashville, TN, United States; Geriatric Research Education and Clinical Care Center, Tennessee Valley Healthcare System VA, Nashville, TN, United States
| | - Rashmee U Shah
- Division of Cardiovascular Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States
| | - Christine Goodrich
- Department of Epidemiology and Biomedical Data Science, Dartmouth Geisel School of Medicine, Hanover, NH, United States
| | - Iben Ricket
- Department of Epidemiology and Biomedical Data Science, Dartmouth Geisel School of Medicine, Hanover, NH, United States
| | - Freneka Minter
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States
| | - Andrew Bohm
- Department of Epidemiology and Biomedical Data Science, Dartmouth Geisel School of Medicine, Hanover, NH, United States
| | - Bruce E Bray
- Division of Cardiovascular Medicine, University of Utah School of Medicine, Salt Lake City, UT, United States; Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT, United States
| | - Michael E Matheny
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, United States; Department of Biostatistics, Vanderbilt University Medical Center, Nashville, TN, United States; Division of General Internal Medicine, Vanderbilt University Medical Center, Nashville, TN, United States; Geriatric Research Education and Clinical Care Center, Tennessee Valley Healthcare System VA, Nashville, TN, United States
| | - Wendy Chapman
- Department of Biomedical Informatics, University of Utah School of Medicine, Salt Lake City, UT, United States; Centre for Clinical and Public Health Informatics, University of Melbourne, Melbourne, Australia
| |
Collapse
|
8
|
Levy AE, Shah NR, Matheny ME, Reeves RM, Gobbel GT, Bradley SM. Determining post-test risk in a national sample of stress nuclear myocardial perfusion imaging reports: Implications for natural language processing tools. J Nucl Cardiol 2019; 26:1878-1885. [PMID: 29696484 PMCID: PMC6202272 DOI: 10.1007/s12350-018-1275-y] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2017] [Accepted: 02/26/2018] [Indexed: 12/23/2022]
Abstract
BACKGROUND Reporting standards promote clarity and consistency of stress myocardial perfusion imaging (MPI) reports, but do not require an assessment of post-test risk. Natural Language Processing (NLP) tools could potentially help estimate this risk, yet it is unknown whether reports contain adequate descriptive data to use NLP. METHODS Among VA patients who underwent stress MPI and coronary angiography between January 1, 2009 and December 31, 2011, 99 stress test reports were randomly selected for analysis. Two reviewers independently categorized each report for the presence of critical data elements essential to describing post-test ischemic risk. RESULTS Few stress MPI reports provided a formal assessment of post-test risk within the impression section (3%) or the entire document (4%). In most cases, risk was determinable by combining critical data elements (74% impression, 98% whole). If ischemic risk was not determinable (25% impression, 2% whole), inadequate description of systolic function (9% impression, 1% whole) and inadequate description of ischemia (5% impression, 1% whole) were most commonly implicated. CONCLUSIONS Post-test ischemic risk was determinable but rarely reported in this sample of stress MPI reports. This supports the potential use of NLP to help clarify risk. Further study of NLP in this context is needed.
Collapse
Affiliation(s)
- Andrew E. Levy
- Division of Cardiology, Department of Medicine, University of Colorado School of Medicine, Aurora, CO, USA
| | - Nishant R. Shah
- Division of Cardiology, Department of Medicine, Brown University Alpert Medical School, Providence, RI, USA
- Center for Evidence Synthesis in Health, Department of Health Services, Policy and Practice, Brown University School of Public Health, Providence, RI, USA
| | - Michael E. Matheny
- Health Services Research & Development; VA Tennessee Valley Healthcare System, Nashville, TN, USA
- Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA
| | - Ruth M. Reeves
- Health Services Research & Development; VA Tennessee Valley Healthcare System, Nashville, TN, USA
- Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA
| | - Glenn T. Gobbel
- Health Services Research & Development; VA Tennessee Valley Healthcare System, Nashville, TN, USA
- Department of Biomedical Informatics, Vanderbilt University School of Medicine, Nashville, TN, USA
| | - Steven M. Bradley
- Cardiovascular Medicine, VA Eastern Colorado Healthcare System, Denver, CO, USA
- Center for Healthcare Delivery Innovation, Minneapolis Heart Institute, Minneapolis, MN, USA
| |
Collapse
|
9
|
Reeves RM, FitzHenry F, Brown SH, Kotter K, Gobbel GT, Montella D, Murff HJ, Speroff T, Matheny ME. Who said it? Establishing professional attribution among authors of Veterans' Electronic Health Records. AMIA Annu Symp Proc 2012; 2012:753-762. [PMID: 23304349 PMCID: PMC3540586] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 06/01/2023]
Abstract
BACKGROUND A practical data point for assessing information quality and value in the Electronic Health Record (EHR) is the professional category of the EHR author. We evaluated and compared free form electronic signatures against LOINC note titles in categorizing the profession of EHR authors. METHODS A random 1000 clinical document sample was selected and divided into 500 document sets for training and testing. The gold standard for provider classification was generated by dual clinician manual review, disagreements resolved by a third reviewer. Text matching algorithms composed of document titles and author electronic signatures for provider classification were developed on the training set. RESULTS Overall, detection of professional classification by note titles alone resulted in 76.1% sensitivity and 69.4% specificity. The aggregate of note titles with electronic signatures resulted in 95.7% sensitivity and 98.5% specificity. CONCLUSIONS Note titles alone provided fair professional classification. Inclusion of author electronic signatures significantly boosted classification performance.
Collapse
Affiliation(s)
- Ruth M Reeves
- Health Services Research and Development, Tennessee Valley Healthcare System, Department of Veterans Affairs, 1310 24 Avenue South, Nashville, TN 37212, USA.
| | | | | | | | | | | | | | | | | |
Collapse
|
10
|
Reeves RM, Ong FR, Matheny ME, Denny JC, Aronsky D, Gobbel GT, Montella D, Speroff T, Brown SH. Detecting temporal expressions in medical narratives. Int J Med Inform 2012; 82:118-27. [PMID: 22595284 DOI: 10.1016/j.ijmedinf.2012.04.006] [Citation(s) in RCA: 19] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2012] [Revised: 03/30/2012] [Accepted: 04/12/2012] [Indexed: 12/27/2022]
Abstract
BACKGROUND Clinical practice and epidemiological information aggregation require knowing when, how long, and in what sequence medically relevant events occur. The Temporal Awareness and Reasoning Systems for Question Interpretation (TARSQI) Toolkit (TTK) is a complete, open source software package for the temporal ordering of events within narrative text documents. TTK was developed on newspaper articles. We extended TTK to support medical notes using veterans' affairs (VA) clinical notes and compared it to TTK. METHODS We used a development set consisting of 200 VA clinical notes to modify and append rules to TTK's time tagger, creating Med-TTK. We then evaluated the performances of TTK and Med-TTK on an independent random selection of 100 clinical notes. Evaluation tasks were to identify and classify time-referring expressions as one of four temporal classes (DATE, TIME, DURATION, and SET). The reference standard for this test set was generated by dual human manual review with disagreements resolved by a third reviewer. Outcome measures included recall and precision for each class, and inter-rater agreement scores. RESULTS There were 3146 temporal expressions in the reference standard. TTK identified 1595 temporal expressions. Recall was 0.15 (95% confidence interval [CI] 0.12-0.15) and precision was 0.27 (95% CI 0.25-0.29) for TTK. Med-TTK identified 3174 expressions. Recall was 0.86 (95% CI 0.84-0.87) and precision was 0.85 (95% CI 0.84-0.86) for Med-TTK. CONCLUSION The algorithms for identifying and classifying temporal expressions in medical narratives developed within Med-TTK significantly improved performance compared to TTK. Natural language processing applications such as Med-TTK provide a foundation for meaningful longitudinal mapping of patient history events among electronic health records. The tool can be accessed at the following site: http://code.google.com/p/med-ttk/.
Collapse
Affiliation(s)
- Ruth M Reeves
- Geriatric Research Education and Clinical Center, Tennessee Valley Healthcare System, Department of Veterans Affairs, Nashville, TN, USA.
| | | | | | | | | | | | | | | | | |
Collapse
|
11
|
Reeves RM. Omission of bupropion as a recommended treatment for PTSD. J Clin Psychiatry 2000; 61:786. [PMID: 11078042 DOI: 10.4088/jcp.v61n1010e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|