Detecting adverse drug reactions in discharge summaries of electronic medical records using Readpeer

Effectiveness of Transformer-Based Large Language Models in Identifying Adverse Drug Reaction Relations from Unstructured Discharge Summaries in Singapore

INTRODUCTION

Transformer-based large language models (LLMs) have transformed the field of natural language processing and led to significant advancements in various text processing tasks. However, the applicability of these LLMs in identifying related drug-adverse event (AE) pairs within clinical context may be limited by the prevalent use of non-standard sentence structures and grammar.

METHOD

Nine transformer-based LLMs pre-trained on biomedical domain corpora are fine-tuned on annotated data (n = 5088) to classify drug-AE pairs in unstructured discharge summaries as causally related or unrelated. These LLMs are then validated on text segments from deidentified hospital discharge summaries from Singapore (n = 1647). To assess generalisability, the models are validated on annotated segments (n = 4418) from the Medical Information Mart for Intensive Care (MIMIC-III) database. Performance of LLMs in identifying related drug-AE pairs is then compared against a prior benchmark set by traditional machine learning models on the same data.

RESULTS

Using an LLM-Bidirectional long short-term memory (LLM-BiLSTM) architecture, transformer-based LLMs improve F1 score as compared to prior benchmark with BioM-ELECTRA-Large-BiLSTM showing an average F1 score improvement of 16.1% (increase from 0.64 to 0.74). Applying additional rules on the LLM-based predictions, like ignoring drug-AE pairs when the AE is a known indication of the drug, results in a further reduction in false positive rates with precision increases of up to 5.6% (0.04 increment).

CONCLUSION

Transformer-based LLMs outperform traditional machine learning methods in identifying causally related drug-AE pairs embedded within unstructured discharge summaries. Nonetheless the improvement in performance with rules indicates that LLMs still possess some degree of imperfection for this causal relation detection task.

Paediatric medication incident reporting: a multicentre comparison study of medication errors identified at audit, detected by staff and reported to an incident system

OBJECTIVES

To compare medication errors identified at audit and via direct observation with medication errors reported to an incident reporting system at paediatric hospitals and to investigate differences in types and severity of errors detected and reported by staff.

METHODS

This is a comparison study at two tertiary referral paediatric hospitals between 2016 and 2020 in Australia. Prescribing errors were identified from a medication chart audit of 7785 patient records. Medication administration errors were identified from a prospective direct observational study of 5137 medication administration doses to 1530 patients. Medication errors reported to the hospitals' incident reporting system were identified and matched with errors identified at audit and observation.

RESULTS

Of 11 302 clinical prescribing errors identified at audit, 3.2 per 1000 errors (95% CI 2.3 to 4.4, n=36) had an incident report. Of 2224 potentially serious prescribing errors from audit, 26.1% (95% CI 24.3 to 27.9, n=580) were detected by staff and 11.2 per 1000 errors (95% CI 7.6 to 16.5, n=25) were reported to the incident system. Although the prescribing error detection rates varied between the two hospitals, there was no difference in incident reporting rates regardless of error severity. Of 40 errors associated with actual patient harm, only 7 (17.5%; 95% CI 8.7% to 31.9%) were detected by staff and 4 (10.0%; 95% CI 4.0% to 23.1%) had an incident report. None of the 2883 clinical medication administration errors observed, including 903 potentially serious errors and 144 errors associated with actual patient harm, had incident reports.

CONCLUSION

Incident reporting data do not provide an accurate reflection of medication errors and related harm to children in hospitals. Failure to detect medication errors is likely to be a significant contributor to low error reporting rates. In an era of electronic health records, new automated approaches to monitor medication safety should be pursued to provide real-time monitoring.

Extracting adverse drug events from clinical Notes: A systematic review of approaches used

BACKGROUND

An adverse drug event (ADE) is any unfavorable effect that occurs due to the use of a drug. Extracting ADEs from unstructured clinical notes is essential to biomedical text extraction research because it helps with pharmacovigilance and patient medication studies.

OBJECTIVE

From the considerable amount of clinical narrative text, natural language processing (NLP) researchers have developed methods for extracting ADEs and their related attributes. This work presents a systematic review of current methods.

METHODOLOGY

Two biomedical databases have been searched from June 2022 until December 2023 for relevant publications regarding this review, namely the databases PubMed and Medline. Similarly, we searched the multi-disciplinary databases IEEE Xplore, Scopus, ScienceDirect, and the ACL Anthology. We adopted the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 statement guidelines and recommendations for reporting systematic reviews in conducting this review. Initially, we obtained 5,537 articles from the search results from the various databases between 2015 and 2023. Based on predefined inclusion and exclusion criteria for article selection, 100 publications have undergone full-text review, of which we consider 82 for our analysis.

RESULTS

We determined the general pattern for extracting ADEs from clinical notes, with named entity recognition (NER) and relation extraction (RE) being the dual tasks considered. Researchers that tackled both NER and RE simultaneously have approached ADE extraction as a "pipeline extraction" problem (n = 22), as a "joint task extraction" problem (n = 7), and as a "multi-task learning" problem (n = 6), while others have tackled only NER (n = 27) or RE (n = 20). We further grouped the reviews based on the approaches for data extraction, namely rule-based (n = 8), machine learning (n = 11), deep learning (n = 32), comparison of two or more approaches (n = 11), hybrid (n = 12) and large language models (n = 8). The most used datasets are MADE 1.0, TAC 2017 and n2c2 2018.

CONCLUSION

Extracting ADEs is crucial, especially for pharmacovigilance studies and patient medications. This survey showcases advances in ADE extraction research, approaches, datasets, and state-of-the-art performance in them. Challenges and future research directions are highlighted. We hope this review will guide researchers in gaining background knowledge and developing more innovative ways to address the challenges.

mtADENet: A novel interpretable method integrating multiple types of network-based inference approaches for prediction of adverse drug events

Identification of adverse drug events (ADEs) is crucial to reduce human health risks and accelerate drug safety assessment. ADEs are mainly caused by unintended interactions with primary or additional targets (off-targets). In this study, we proposed a novel interpretable method named mtADENet, which integrates multiple types of network-based inference approaches for ADE prediction. Different from phenotype-based methods, mtADENet introduced computational target profiles predicted by network-based methods to bridge the gap between chemical structures and ADEs, and hence can not only predict ADEs for drugs and novel compounds within or outside the drug-ADE association network, but also provide insights for the elucidation of molecular mechanisms of the ADEs caused by drugs. We constructed a series of network-based prediction models for 23 ADE categories. These models achieved high AUC values ranging from 0.865 to 0.942 in 10-fold cross validation. The best model further showed high performance on four external validation sets, which outperformed two previous network-based methods. To show the practical value of mtADENet, we performed case studies on developmental neurotoxicity and cardio-oncology, and over 50 % of predicted ADEs and targets for drugs and novel compounds were validated by literature. Moreover, mtADENet is freely available at our web server named NetInfer (http://lmmd.ecust.edu.cn/netinfer/). In summary, mtADENet would be a powerful tool for ADE prediction and drug safety assessment in drug discovery and development.

Review of Natural Language Processing in Pharmacology

Natural language processing (NLP) is an area of artificial intelligence that applies information technologies to process the human language, understand it to a certain degree, and use it in various applications. This area has rapidly developed in the past few years and now employs modern variants of deep neural networks to extract relevant patterns from large text corpora. The main objective of this work is to survey the recent use of NLP in the field of pharmacology. As our work shows, NLP is a highly relevant information extraction and processing approach for pharmacology. It has been used extensively, from intelligent searches through thousands of medical documents to finding traces of adversarial drug interactions in social media. We split our coverage into five categories to survey modern NLP: methodology, commonly addressed tasks, relevant textual data, knowledge bases, and useful programming libraries. We split each of the five categories into appropriate subcategories, describe their main properties and ideas, and summarize them in a tabular form. The resulting survey presents a comprehensive overview of the area, useful to practitioners and interested observers. SIGNIFICANCE STATEMENT: The main objective of this work is to survey the recent use of NLP in the field of pharmacology in order to provide a comprehensive overview of the current state in the area after the rapid developments that occurred in the past few years. The resulting survey will be useful to practitioners and interested observers in the domain.

Artificial Intelligence-Enabled Software Prototype to Inform Opioid Pharmacovigilance From Electronic Health Records: Development and Usability Study

Background

The use of patient health and treatment information captured in structured and unstructured formats in computerized electronic health record (EHR) repositories could potentially augment the detection of safety signals for drug products regulated by the US Food and Drug Administration (FDA). Natural language processing and other artificial intelligence (AI) techniques provide novel methodologies that could be leveraged to extract clinically useful information from EHR resources.

Objective

Our aim is to develop a novel AI-enabled software prototype to identify adverse drug event (ADE) safety signals from free-text discharge summaries in EHRs to enhance opioid drug safety and research activities at the FDA.

Methods

We developed a prototype for web-based software that leverages keyword and trigger-phrase searching with rule-based algorithms and deep learning to extract candidate ADEs for specific opioid drugs from discharge summaries in the Medical Information Mart for Intensive Care III (MIMIC III) database. The prototype uses MedSpacy components to identify relevant sections of discharge summaries and a pretrained natural language processing (NLP) model, Spark NLP for Healthcare, for named entity recognition. Fifteen FDA staff members provided feedback on the prototype's features and functionalities.

Results

Using the prototype, we were able to identify known, labeled, opioid-related adverse drug reactions from text in EHRs. The AI-enabled model achieved accuracy, recall, precision, and F1-scores of 0.66, 0.69, 0.64, and 0.67, respectively. FDA participants assessed the prototype as highly desirable in user satisfaction, visualizations, and in the potential to support drug safety signal detection for opioid drugs from EHR data while saving time and manual effort. Actionable design recommendations included (1) enlarging the tabs and visualizations; (2) enabling more flexibility and customizations to fit end users' individual needs; (3) providing additional instructional resources; (4) adding multiple graph export functionality; and (5) adding project summaries.

Conclusions

The novel prototype uses innovative AI-based techniques to automate searching for, extracting, and analyzing clinically useful information captured in unstructured text in EHRs. It increases efficiency in harnessing real-world data for opioid drug safety and increases the usability of the data to support regulatory review while decreasing the manual research burden.

Introducing AI to the molecular tumor board: one direction toward the establishment of precision medicine using large-scale cancer clinical and biological information

Since U.S. President Barack Obama announced the Precision Medicine Initiative in his New Year's State of the Union address in 2015, the establishment of a precision medicine system has been emphasized worldwide, particularly in the field of oncology. With the advent of next-generation sequencers specifically, genome analysis technology has made remarkable progress, and there are active efforts to apply genome information to diagnosis and treatment. Generally, in the process of feeding back the results of next-generation sequencing analysis to patients, a molecular tumor board (MTB), consisting of experts in clinical oncology, genetic medicine, etc., is established to discuss the results. On the other hand, an MTB currently involves a large amount of work, with humans searching through vast databases and literature, selecting the best drug candidates, and manually confirming the status of available clinical trials. In addition, as personalized medicine advances, the burden on MTB members is expected to increase in the future. Under these circumstances, introducing cutting-edge artificial intelligence (AI) technology and information and communication technology to MTBs while reducing the burden on MTB members and building a platform that enables more accurate and personalized medical care would be of great benefit to patients. In this review, we introduced the latest status of elemental technologies that have potential for AI utilization in MTB, and discussed issues that may arise in the future as we progress with AI implementation.

Combining Machine Learning with a Rule-Based Algorithm to Detect and Identify Related Entities of Documented Adverse Drug Reactions on Hospital Discharge Summaries

INTRODUCTION

Discharge summaries contain valuable information about adverse drug reactions, but their unstructured nature makes them challenging to analyse and use as a signal source for pharmacovigilance. Machine learning has shown promise in identifying discharge summaries that contain related drug-adverse event pairs but has fared relatively poorer in entity extraction.

METHODS

A hybrid model is developed combining rule-based and machine learning algorithms using discharge summaries with the aim of maximising capture of related drug-adverse event pairs. The rule first identifies segments containing adverse event entities within a 100-character distance from a drug term; machine learning subsequently estimates the relatedness of the drug and adverse event entities contained. The approach is validated on four independent datasets that are temporally and geographically separated from model development data. The impact of restricted drug-adverse event pair detection on recall is evaluated by using two of the four validation datasets that do not impose rule-based restrictions to annotations.

RESULTS

The hybrid model achieves a recall of 0.80 (fivefold cross validation), 0.80 (temporal) and 0.76 (geographical) on validation using datasets containing only pre-identified target text segments that fulfil the rule-based algorithm criteria. When tested on datasets that additionally contained drug-adverse event pairs not restricted by the rule-based criteria, recall of the model declines to 0.68 and 0.62 on temporally and geographically separated datasets, respectively.

CONCLUSIONS

The proposed hybrid model demonstrates reasonable generalisability on external validation. Rule-based restriction of the detection space results in an approximately 12-14% reduction in recall but improves identification of the related drug and adverse event terms.

Development of a Pipeline for Adverse Drug Reaction Identification in Clinical Notes: Word Embedding Models and String Matching

Background

Knowledge about adverse drug reactions (ADRs) in the population is limited because of underreporting, which hampers surveillance and assessment of drug safety. Therefore, gathering accurate information that can be retrieved from clinical notes about the incidence of ADRs is of great relevance. However, manual labeling of these notes is time-consuming, and automatization can improve the use of free-text clinical notes for the identification of ADRs. Furthermore, tools for language processing in languages other than English are not widely available.

Objective

The aim of this study is to design and evaluate a method for automatic extraction of medication and Adverse Drug Reaction Identification in Clinical Notes (ADRIN).

Methods

Dutch free-text clinical notes (N=277,398) and medication registrations (N=499,435) from the Cardiology Centers of the Netherlands database were used. All clinical notes were used to develop word embedding models. Vector representations of word embedding models and string matching with a medical dictionary (Medical Dictionary for Regulatory Activities [MedDRA]) were used for identification of ADRs and medication in a test set of clinical notes that were manually labeled. Several settings, including search area and punctuation, could be adjusted in the prototype to evaluate the optimal version of the prototype.

Results

The ADRIN method was evaluated using a test set of 988 clinical notes written on the stop date of a drug. Multiple versions of the prototype were evaluated for a variety of tasks. Binary classification of ADR presence achieved the highest accuracy of 0.84. Reduced search area and inclusion of punctuation improved performance, whereas incorporation of the MedDRA did not improve the performance of the pipeline.

Conclusions

The ADRIN method and prototype are effective in recognizing ADRs in Dutch clinical notes from cardiac diagnostic screening centers. Surprisingly, incorporation of the MedDRA did not result in improved identification on top of word embedding models. The implementation of the ADRIN tool may help increase the identification of ADRs, resulting in better care and saving substantial health care costs.

Towards Pharma 4.0 in clinical trials: A future-orientated perspective

Pharma 4.0, a technology ecosystem in drug development analogous to Industry 4.0 in healthcare, is transforming the traditional approach to drug discovery and development, aligning product quality with less time to market, and creating intelligent stakeholder networks through effective collaborations. The wide range of potential Pharma 4.0 networks have produced several conceptualizations, which have led to a lack of clarity and definition. The main emphasis of this paper is on the clinical trial stage of drug development in the Pharma 4.0 era. It highlights the merged computerized technologies that are currently used in clinical research, and proposes a framework for integrating Pharma 4.0 technologies. The impact of and barriers to employing the proposed framework are discussed, highlighting its potential and some future research applications.

Data Science Methods for Nursing-Relevant Patient Outcomes and Clinical Processes: The 2019 Literature Year in Review

Data science continues to be recognized and used within healthcare due to the increased availability of large data sets and advanced analytics. It can be challenging for nurse leaders to remain apprised of this rapidly changing landscape. In this article, we describe our findings from a scoping literature review of papers published in 2019 that use data science to explore, explain, and/or predict 15 phenomena of interest to nurses. Fourteen of the 15 phenomena were associated with at least one paper published in 2019. We identified the use of many contemporary data science methods (eg, natural language processing, neural networks) for many of the outcomes. We found many studies exploring Readmissions and Pressure Injuries. The topics of Artificial Intelligence/Machine Learning Acceptance, Burnout, Patient Safety, and Unit Culture were poorly represented. We hope that the studies described in this article help readers: (1) understand the breadth and depth of data science's ability to improve clinical processes and patient outcomes that are relevant to nurses and (2) identify gaps in the literature that are in need of exploration.

Role of Artificial Intelligence in Patient Safety Outcomes: Systematic Literature Review

BACKGROUND

Artificial intelligence (AI) provides opportunities to identify the health risks of patients and thus influence patient safety outcomes.

OBJECTIVE

The purpose of this systematic literature review was to identify and analyze quantitative studies utilizing or integrating AI to address and report clinical-level patient safety outcomes.

METHODS

We restricted our search to the PubMed, PubMed Central, and Web of Science databases to retrieve research articles published in English between January 2009 and August 2019. We focused on quantitative studies that reported positive, negative, or intermediate changes in patient safety outcomes using AI apps, specifically those based on machine-learning algorithms and natural language processing. Quantitative studies reporting only AI performance but not its influence on patient safety outcomes were excluded from further review.

RESULTS

We identified 53 eligible studies, which were summarized concerning their patient safety subcategories, the most frequently used AI, and reported performance metrics. Recognized safety subcategories were clinical alarms (n=9; mainly based on decision tree models), clinical reports (n=21; based on support vector machine models), and drug safety (n=23; mainly based on decision tree models). Analysis of these 53 studies also identified two essential findings: (1) the lack of a standardized benchmark and (2) heterogeneity in AI reporting.

CONCLUSIONS

This systematic review indicates that AI-enabled decision support systems, when implemented correctly, can aid in enhancing patient safety by improving error detection, patient stratification, and drug management. Future work is still needed for robust validation of these systems in prospective and real-world clinical environments to understand how well AI can predict safety outcomes in health care settings.

The economics of drug allergy

PURPOSE OF REVIEW

As drug allergy research aims to inform clinical practice, implementation of best practices may be influenced by financial resources required to incorporate new interventions and the resulting clinical and economic returns on those resource investments. The present review summarizes new insights into the economics of drug allergy over the past year.

RECENT FINDINGS

While considering economic implications of recent drug allergy research, many studies have addressed different contextual factors related to the setting, provider, or outcomes. Advances in technology have enabled specialized allergists to support remote settings through telemedicine consultation. Training opportunities and interdisciplinary approaches to address drug allergy challenges have enabled multiple provider types to play a role in screening, diagnosis, and management. Penicillin allergy testing has been a major focus for many institutions, with several studies focused on de-labeling strategies including confirmatory skin testing and direct oral challenges.

SUMMARY

Studies over the past year provide new opportunities for the field of drug allergy research. The focus of current research to capture direct health costs or savings associated with drug allergy interventions demonstrates opportunity for more cost-effective care delivery and opportunity to explore greater benefits to society.

Automated screening of natural language in electronic health records for the diagnosis septic shock is feasible and outperforms an approach based on explicit administrative codes

PURPOSE

Identification of patients for epidemiologic research through administrative coding has important limitations. We investigated the feasibility of a search based on natural language processing (NLP) on the text sections of electronic health records for identification of patients with septic shock.

MATERIALS AND METHODS

Results of an explicit search strategy (using explicit concept retrieval) and a combined search strategy (using both explicit and implicit concept retrieval) were compared to hospital ICD-9 based administrative coding and to our department's own prospectively compiled infection database.

RESULTS

Of 8911 patients admitted to the medical or surgical ICU, 1023 (11.5%) suffered from septic shock according to the combined search strategy. This was significantly more than those identified by the explicit strategy (518, 5.8%), by hospital administrative coding (549, 5.8%) or by our own prospectively compiled database (609, 6.8%) (p < .001). Sensitivity and specificity of the automated combined search strategy were 72.7% (95%CI 69.0%-76.2%) and 93.0% (95%CI 92.4%-93.6%), compared to 56.0% (95%CI 52.0%-60.0%) and 97.5% (95%CI 97.1%-97.8%) for hospital administrative coding.

CONCLUSIONS

An automated search strategy based on a combination of explicit and implicit concept retrieval is feasible to screen electronic health records for septic shock and outperforms an administrative coding based explicit approach.