Identifying Cases of Shoulder Injury Related to Vaccine Administration (SIRVA) in the United States: Development and Validation of a Natural Language Processing Method (Preprint)

LEVERAGING UNSTRUCTURED DATA IN ELECTRONIC HEALTH RECORDS TO DETECT ADVERSE EVENTS FROM PEDIATRIC DRUG USE - A SCOPING REVIEW

Adverse drug events (ADEs) in pediatric populations pose significant public health challenges, yet research on their detection and monitoring remains limited. This scoping review evaluates the use of unstructured data from electronic health records (EHRs) to identify ADEs in children. We searched six databases, including MEDLINE, Embase and IEEE Xplore, in September 2024. From 984 records, only nine studies met our inclusion criteria, indicating a significant gap in research towards identify ADEs in children. We found that unstructured data in EHRs can indeed be of value and enhance pediatric pharmacovigilance, although its use has been so far very limited. Traditional Natural Language Processing (NLP) methods have been employed to extract ADEs, but the approaches utilized face challenges in generalizability and context interpretation. These challenges could be addressed with recent advances in transformer-based models and large language models (LLMs), unlocking the use of EHR data at scale for pediatric pharmacovigilance.

Medical Marijuana Documentation Practices in Patient Electronic Health Records: Retrospective Observational Study Using Smart Data Elements and a Review of Medical Records

Background

Medical marijuana (MMJ) is available in Pennsylvania, and participation in the state-regulated program requires patient registration and receiving certification by an approved physician. Currently, no integration of MMJ certification data with health records exists in Pennsylvania that would allow clinicians to rapidly identify patients using MMJ, as exists with other scheduled drugs. This absence of a formal data sharing structure necessitates tools aiding in consistent documentation practices to enable comprehensive patient care. Customized smart data elements (SDEs) were made available to clinicians at an integrated health system, Geisinger, following MMJ legalization in Pennsylvania.

Objective

The purpose of this project was to examine and contextualize the use of MMJ SDEs in the Geisinger population. We accomplished this goal by developing a systematic protocol for review of medical records and creating a tool that resulted in consistent human data extraction.

Methods

We developed a protocol for reviewing medical records for extracting MMJ-related information. The protocol was developed between August and December of 2022 and focused on a patient group that received one of several MMJ SDEs between January 25, 2019, and May 26, 2022. Characteristics were first identified on a pilot sample (n=5), which were then iteratively reviewed to optimize for consistency. Following the pilot, 2 reviewers were assigned 200 randomly selected patients' medical records, with a third reviewer examining a subsample (n=30) to determine reliability. We then summarized the clinician- and patient-level features from 156 medical records with a table-format SDE that best captured MMJ information.

Results

We found the review protocol for medical records was feasible for those with minimal medical background to complete, with high interrater reliability (κ=0.966; P<.001; odds ratio 0.97, 95% CI 0.954-0.978). MMJ certification was largely documented by nurses and medical assistants (n=138, 88.5%) and typically within primary care settings (n=107, 68.6%). The SDE has 6 preset field prompts with heterogeneous documentation completion rates, including certifying conditions (n=146, 93.6%), product (n=145, 92.9%), authorized dispensary (n=137, 87.8%), active ingredient (n=130, 83.3%), certifying provider (n=96, 61.5%), and dosage (n=48, 30.8%). We found preset fields were overall well-recorded (mean 76.6%, SD 23.7% across all fields). Primary diagnostic codes recorded at documentation encounters varied, with the most frequent being routine examinations and testing (n=34, 21.8%), musculoskeletal or nervous conditions, and signs and symptoms not classified elsewhere (n=21, 13.5%).

Conclusions

This method of reviewing medical records yields high-quality data extraction that can serve as a model for other health record inquiries. Our evaluation showed relatively high completeness of SDE fields, primarily by clinical staff responsible for rooming patients, with an overview of conditions under which MMJ is documented. Improving the adoption and fidelity of SDE data collection may present a valuable data source for future research on patient MMJ use, treatment efficacy, and outcomes.

Natural Language Processing Versus Diagnosis Code-Based Methods for Postherpetic Neuralgia Identification: Algorithm Development and Validation

Background

Diagnosis codes and prescription data are used in algorithms to identify postherpetic neuralgia (PHN), a debilitating complication of herpes zoster (HZ). Because of the questionable accuracy of codes and prescription data, manual chart review is sometimes used to identify PHN in electronic health records (EHRs), which can be costly and time-consuming.

Objective

This study aims to develop and validate a natural language processing (NLP) algorithm for automatically identifying PHN from unstructured EHR data and to compare its performance with that of code-based methods.

Methods

This retrospective study used EHR data from Kaiser Permanente Southern California, a large integrated health care system that serves over 4.8 million members. The source population included members aged ≥50 years who received an incident HZ diagnosis and accompanying antiviral prescription between 2018 and 2020 and had ≥1 encounter within 90-180 days of the incident HZ diagnosis. The study team manually reviewed the EHR and identified PHN cases. For NLP development and validation, 500 and 800 random samples from the source population were selected, respectively. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F-score, and Matthews correlation coefficient (MCC) of NLP and the code-based methods were evaluated using chart-reviewed results as the reference standard.

Results

The NLP algorithm identified PHN cases with a 90.9% sensitivity, 98.5% specificity, 82% PPV, and 99.3% NPV. The composite scores of the NLP algorithm were 0.89 (F-score) and 0.85 (MCC). The prevalences of PHN in the validation data were 6.9% (reference standard), 7.6% (NLP), and 5.4%-13.1% (code-based). The code-based methods achieved a 52.7%-61.8% sensitivity, 89.8%-98.4% specificity, 27.6%-72.1% PPV, and 96.3%-97.1% NPV. The F-scores and MCCs ranged between 0.45 and 0.59 and between 0.32 and 0.61, respectively.

Conclusions

The automated NLP-based approach identified PHN cases from the EHR with good accuracy. This method could be useful in population-based PHN research.

Current Approaches in Postapproval Vaccine Safety Studies Using Real-World Data: A Systematic Review of Published Literature

PURPOSE

Well-designed observational postmarketing studies using real-world data (RWD) are critical in supporting an evidence base and bolstering public confidence in vaccine safety. This systematic review presents current research methodologies in vaccine safety research in postapproval settings, technological advancements contributing to research resources and capabilities, and their major strengths and limitations.

METHODS

A comprehensive search was conducted using PubMed to identify relevant articles published from January 1, 2019, to December 31, 2022. Eligible studies were summarized overall by study design and other study characteristics (eg, country, vaccine studied, types of data source, and study population). An in-depth review of select studies representative of conventional or new designs, analytical approaches, or data collection methods was conducted to summarize current methods in vaccine safety research.

FINDINGS

Out of 977 articles screened for inclusion, 135 were reviewed. The review shows that recent advancements in scientific methods, digital technology, and analytic approaches have significantly contributed to postapproval vaccine safety studies using RWD. "Near real-time surveillance" using large datasets (via collaborative or distributed databases) has been used to facilitate rapid signal detection that complements passive surveillance. There was increasing appreciation for self-controlled case-only designs (self-controlled case series and self-controlled risk interval) to assess acute-onset safety outcomes, artificial intelligence, and natural language processing to improve outcome accuracy and study timeliness and emerging artificial intelligence-based analysis to capture adverse events from social media platforms.

IMPLICATIONS

Continued development in the area of vaccine safety research methodologies using RWD is warranted. The future of successful vaccine safety research, especially evaluation of rare safety events, is likely to comprise digital technologies including linking RWD networks, machine learning, and advanced analytic methods to generate rapid and robust real-world safety information.

Identification of recurrent atrial fibrillation using natural language processing applied to electronic health records

AIMS

This study aimed to develop and apply natural language processing (NLP) algorithms to identify recurrent atrial fibrillation (AF) episodes following rhythm control therapy initiation using electronic health records (EHRs).

METHODS AND RESULTS

We included adults with new-onset AF who initiated rhythm control therapies (ablation, cardioversion, or antiarrhythmic medication) within two US integrated healthcare delivery systems. A code-based algorithm identified potential AF recurrence using diagnosis and procedure codes. An automated NLP algorithm was developed and validated to capture AF recurrence from electrocardiograms, cardiac monitor reports, and clinical notes. Compared with the reference standard cases confirmed by physicians' adjudication, the F-scores, sensitivity, and specificity were all above 0.90 for the NLP algorithms at both sites. We applied the NLP and code-based algorithms to patients with incident AF (n = 22 970) during the 12 months after initiating rhythm control therapy. Applying the NLP algorithms, the percentages of patients with AF recurrence for sites 1 and 2 were 60.7% and 69.9% (ablation), 64.5% and 73.7% (cardioversion), and 49.6% and 55.5% (antiarrhythmic medication), respectively. In comparison, the percentages of patients with code-identified AF recurrence for sites 1 and 2 were 20.2% and 23.7% for ablation, 25.6% and 28.4% for cardioversion, and 20.0% and 27.5% for antiarrhythmic medication, respectively.

CONCLUSION

When compared with a code-based approach alone, this study's high-performing automated NLP method identified significantly more patients with recurrent AF. The NLP algorithms could enable efficient evaluation of treatment effectiveness of AF therapies in large populations and help develop tailored interventions.