Enhancing timeliness of drug overdose mortality surveillance: A machine learning approach

Precision public health in the era of genomics and big data

Precision public health (PPH) considers the interplay between genetics, lifestyle and the environment to improve disease prevention, diagnosis and treatment on a population level-thereby delivering the right interventions to the right populations at the right time. In this Review, we explore the concept of PPH as the next generation of public health. We discuss the historical context of using individual-level data in public health interventions and examine recent advancements in how data from human and pathogen genomics and social, behavioral and environmental research, as well as artificial intelligence, have transformed public health. Real-world examples of PPH are discussed, emphasizing how these approaches are becoming a mainstay in public health, as well as outstanding challenges in their development, implementation and sustainability. Data sciences, ethical, legal and social implications research, capacity building, equity research and implementation science will have a crucial role in realizing the potential for 'precision' to enhance traditional public health approaches.

Question-answering system extracts information on injection drug use from clinical notes

BACKGROUND

Injection drug use (IDU) can increase mortality and morbidity. Therefore, identifying IDU early and initiating harm reduction interventions can benefit individuals at risk. However, extracting IDU behaviors from patients' electronic health records (EHR) is difficult because there is no other structured data available, such as International Classification of Disease (ICD) codes, and IDU is most often documented in unstructured free-text clinical notes. Although natural language processing can efficiently extract this information from unstructured data, there are no validated tools.

METHODS

To address this gap in clinical information, we design a question-answering (QA) framework to extract information on IDU from clinical notes for use in clinical operations. Our framework involves two main steps: (1) generating a gold-standard QA dataset and (2) developing and testing the QA model. We use 2323 clinical notes of 1145 patients curated from the US Department of Veterans Affairs (VA) Corporate Data Warehouse to construct the gold-standard dataset for developing and evaluating the QA model. We also demonstrate the QA model's ability to extract IDU-related information from temporally out-of-distribution data.

RESULTS

Here, we show that for a strict match between gold-standard and predicted answers, the QA model achieves a 51.65% F1 score. For a relaxed match between the gold-standard and predicted answers, the QA model obtains a 78.03% F1 score, along with 85.38% Precision and 79.02% Recall scores. Moreover, the QA model demonstrates consistent performance when subjected to temporally out-of-distribution data.

CONCLUSIONS

Our study introduces a QA framework designed to extract IDU information from clinical notes, aiming to enhance the accurate and efficient detection of people who inject drugs, extract relevant information, and ultimately facilitate informed patient care.

PROVIDENT: Development and Validation of a Machine Learning Model to Predict Neighborhood-level Overdose Risk in Rhode Island

BACKGROUND

Drug overdose persists as a leading cause of death in the United States, but resources to address it remain limited. As a result, health authorities must consider where to allocate scarce resources within their jurisdictions. Machine learning offers a strategy to identify areas with increased future overdose risk to proactively allocate overdose prevention resources. This modeling study is embedded in a randomized trial to measure the effect of proactive resource allocation on statewide overdose rates in Rhode Island (RI).

METHODS

We used statewide data from RI from 2016 to 2020 to develop an ensemble machine learning model predicting neighborhood-level fatal overdose risk. Our ensemble model integrated gradient boosting machine and super learner base models in a moving window framework to make predictions in 6-month intervals. Our performance target, developed a priori with the RI Department of Health, was to identify the 20% of RI neighborhoods containing at least 40% of statewide overdose deaths, including at least one neighborhood per municipality. The model was validated after trial launch.

RESULTS

Our model selected priority neighborhoods capturing 40.2% of statewide overdose deaths during the test periods and 44.1% of statewide overdose deaths during validation periods. Our ensemble outperformed the base models during the test periods and performed comparably to the best-performing base model during the validation periods.

CONCLUSIONS

We demonstrated the capacity for machine learning models to predict neighborhood-level fatal overdose risk to a degree of accuracy suitable for practitioners. Jurisdictions may consider predictive modeling as a tool to guide allocation of scarce resources.

An interpretable machine learning framework for opioid overdose surveillance from emergency medical services records

The goal of this study is to develop and validate a lightweight, interpretable machine learning (ML) classifier to identify opioid overdoses in emergency medical services (EMS) records. We conducted a comparative assessment of three feature engineering approaches designed for use with unstructured narrative data. Opioid overdose annotations were provided by two harm reduction paramedics and two supporting annotators trained to reliably match expert annotations. Candidate feature engineering techniques included term frequency-inverse document frequency (TF-IDF), a highly performant approach to concept vectorization, and a custom approach based on the count of empirically-identified keywords. Each feature set was trained using four model architectures: generalized linear model (GLM), Naïve Bayes, neural network, and Extreme Gradient Boost (XGBoost). Ensembles of trained models were also evaluated. The custom feature models were also assessed for variable importance to aid interpretation. Models trained using TF-IDF feature engineering ranged from AUROC = 0.59 (95% CI: 0.53-0.66) for the Naïve Bayes to AUROC = 0.76 (95% CI: 0.71-0.81) for the neural network. Models trained using concept vectorization features ranged from AUROC = 0.83 (95% 0.78-0.88)for the Naïve Bayes to AUROC = 0.89 (95% CI: 0.85-0.94) for the ensemble. Models trained using custom features were the most performant, with benchmarks ranging from AUROC = 0.92 (95% CI: 0.88-0.95) with the GLM to 0.93 (95% CI: 0.90-0.96) for the ensemble. The custom features model achieved positive predictive values (PPV) ranging for 80 to 100%, which represent substantial improvements over previously published EMS encounter opioid overdose classifiers. The application of this approach to county EMS data can productively inform local and targeted harm reduction initiatives.

Assessing and Addressing the Determinants of Appalachian Population Health: A Scoping Review

Introduction

Residents of Appalachia experience elevated rates of morbidity and mortality compared to national averages, and these disparities are associated with inequitable exposures to various determinants of population health. Social and environmental determinants of health are a useful lens through which to develop and evaluate programs to mitigate regional health disparities.

Methods

This 2023 scoping review was conducted of studies linking determinants of Appalachian health with leading causes of regional mortality and morbidity. The search strategy employed a keyword search that included geographic terms for the Appalachian Region and the primary adverse health outcomes in that region. Studies meeting the following inclusion criteria were reviewed: original article, published in the last five years, involving an Appalachian population, and includes a rigorous assessment of an association between a population health determinant and one or more leading causes of Appalachian morbidity and mortality.

Results

The search returned 221 research articles, including 30 interventional studies. The top three health outcomes included cancer (43.59%), diseases of despair (23.08%), and diabetes (12.82). Access to care (27.3%), rurality (18.9%), and education (14.8%) were the most common population health determinants identified. Interventional studies were categorized by program types: education, technology, partnerships, and multilevel interventions. Due to the heterogeneity of study types, the studies were combined using a narrative synthesis.

Implications

The results of this work can inform the development and evaluation of additional programs to promote Appalachian population health. Our study team will use these results to inform community-based discussions that develop strategic plans to mitigate health disparities in Central and Southcentral Appalachian Virginia.

Using Natural Language Processing to Predict Fatal Drug Overdose From Autopsy Narrative Text: Algorithm Development and Validation Study

BACKGROUND

Fatal drug overdose surveillance informs prevention but is often delayed because of autopsy report processing and death certificate coding. Autopsy reports contain narrative text describing scene evidence and medical history (similar to preliminary death scene investigation reports) and may serve as early data sources for identifying fatal drug overdoses. To facilitate timely fatal overdose reporting, natural language processing was applied to narrative texts from autopsies.

OBJECTIVE

This study aimed to develop a natural language processing-based model that predicts the likelihood that an autopsy report narrative describes an accidental or undetermined fatal drug overdose.

METHODS

Autopsy reports of all manners of death (2019-2021) were obtained from the Tennessee Office of the State Chief Medical Examiner. The text was extracted from autopsy reports (PDFs) using optical character recognition. Three common narrative text sections were identified, concatenated, and preprocessed (bag-of-words) using term frequency-inverse document frequency scoring. Logistic regression, support vector machine (SVM), random forest, and gradient boosted tree classifiers were developed and validated. Models were trained and calibrated using autopsies from 2019 to 2020 and tested using those from 2021. Model discrimination was evaluated using the area under the receiver operating characteristic, precision, recall, F1-score, and F2-score (prioritizes recall over precision). Calibration was performed using logistic regression (Platt scaling) and evaluated using the Spiegelhalter z test. Shapley additive explanations values were generated for models compatible with this method. In a post hoc subgroup analysis of the random forest classifier, model discrimination was evaluated by forensic center, race, age, sex, and education level.

RESULTS

A total of 17,342 autopsies (n=5934, 34.22% cases) were used for model development and validation. The training set included 10,215 autopsies (n=3342, 32.72% cases), the calibration set included 538 autopsies (n=183, 34.01% cases), and the test set included 6589 autopsies (n=2409, 36.56% cases). The vocabulary set contained 4002 terms. All models showed excellent performance (area under the receiver operating characteristic ≥0.95, precision ≥0.94, recall ≥0.92, F1-score ≥0.94, and F2-score ≥0.92). The SVM and random forest classifiers achieved the highest F2-scores (0.948 and 0.947, respectively). The logistic regression and random forest were calibrated (P=.95 and P=.85, respectively), whereas the SVM and gradient boosted tree classifiers were miscalibrated (P=.03 and P<.001, respectively). "Fentanyl" and "accident" had the highest Shapley additive explanations values. Post hoc subgroup analyses revealed lower F2-scores for autopsies from forensic centers D and E. Lower F2-score were observed for the American Indian, Asian, ≤14 years, and ≥65 years subgroups, but larger sample sizes are needed to validate these findings.

CONCLUSIONS

The random forest classifier may be suitable for identifying potential accidental and undetermined fatal overdose autopsies. Further validation studies should be conducted to ensure early detection of accidental and undetermined fatal drug overdoses across all subgroups.

Surveillance of communicable diseases using social media: A systematic review

BACKGROUND

Communicable diseases pose a severe threat to public health and economic growth. The traditional methods that are used for public health surveillance, however, involve many drawbacks, such as being labor intensive to operate and resulting in a lag between data collection and reporting. To effectively address the limitations of these traditional methods and to mitigate the adverse effects of these diseases, a proactive and real-time public health surveillance system is needed. Previous studies have indicated the usefulness of performing text mining on social media.

OBJECTIVE

To conduct a systematic review of the literature that used textual content published to social media for the purpose of the surveillance and prediction of communicable diseases.

METHODOLOGY

Broad search queries were formulated and performed in four databases. Both journal articles and conference materials were included. The quality of the studies, operationalized as reliability and validity, was assessed. This qualitative systematic review was guided by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.

RESULTS

Twenty-three publications were included in this systematic review. All studies reported positive results for using textual social media content to surveille communicable diseases. Most studies used Twitter as a source for these data. Influenza was studied most frequently, while other communicable diseases received far less attention. Journal articles had a higher quality (reliability and validity) than conference papers. However, studies often failed to provide important information about procedures and implementation.

CONCLUSION

Text mining of health-related content published on social media can serve as a novel and powerful tool for the automated, real-time, and remote monitoring of public health and for the surveillance and prediction of communicable diseases in particular. This tool can address limitations related to traditional surveillance methods, and it has the potential to supplement traditional methods for public health surveillance.

Using GPT-3 to Build a Lexicon of Drugs of Abuse Synonyms for Social Media Pharmacovigilance

Drug abuse is a serious problem in the United States, with over 90,000 drug overdose deaths nationally in 2020. A key step in combating drug abuse is detecting, monitoring, and characterizing its trends over time and location, also known as pharmacovigilance. While federal reporting systems accomplish this to a degree, they often have high latency and incomplete coverage. Social-media-based pharmacovigilance has zero latency, is easily accessible and unfiltered, and benefits from drug users being willing to share their experiences online pseudo-anonymously. However, unlike highly structured official data sources, social media text is rife with misspellings and slang, making automated analysis difficult. Generative Pretrained Transformer 3 (GPT-3) is a large autoregressive language model specialized for few-shot learning that was trained on text from the entire internet. We demonstrate that GPT-3 can be used to generate slang and common misspellings of terms for drugs of abuse. We repeatedly queried GPT-3 for synonyms of drugs of abuse and filtered the generated terms using automated Google searches and cross-references to known drug names. When generated terms for alprazolam were manually labeled, we found that our method produced 269 synonyms for alprazolam, 221 of which were new discoveries not included in an existing drug lexicon for social media. We repeated this process for 98 drugs of abuse, of which 22 are widely-discussed drugs of abuse, building a lexicon of colloquial drug synonyms that can be used for pharmacovigilance on social media.

Deep Neural Networks for Fine-Grained Surveillance of Overdose Mortality

Surveillance of drug overdose deaths relies on death certificates for identification of the substances that caused death. Drugs and drug classes can be identified through the International Classification of Diseases, Tenth Revision (ICD-10), codes present on death certificates. However, ICD-10 codes do not always provide high levels of specificity in drug identification. To achieve more fine-grained identification of substances on death certificate, the free-text cause-of-death section, completed by the medical certifier, must be analyzed. Current methods for analyzing free-text death certificates rely solely on lookup tables for identifying specific substances, which must be frequently updated and maintained. To improve identification of drugs on death certificates, a deep-learning named-entity recognition model was developed, utilizing data from the Kentucky Drug Overdose Fatality Surveillance System (2014-2019), which achieved an F1-score of 99.13%. This model can identify new drug misspellings and novel substances that are not present on current surveillance lookup tables, enhancing the surveillance of drug overdose deaths.

Priorities for successful use of artificial intelligence by public health organizations: a literature review

Artificial intelligence (AI) has the potential to improve public health's ability to promote the health of all people in all communities. To successfully realize this potential and use AI for public health functions it is important for public health organizations to thoughtfully develop strategies for AI implementation. Six key priorities for successful use of AI technologies by public health organizations are discussed: 1) Contemporary data governance; 2) Investment in modernized data and analytic infrastructure and procedures; 3) Addressing the skills gap in the workforce; 4) Development of strategic collaborative partnerships; 5) Use of good AI practices for transparency and reproducibility, and; 6) Explicit consideration of equity and bias.

Natural Language Processing and Machine Learning to Identify People Who Inject Drugs in Electronic Health Records

Background

Improving the identification of people who inject drugs (PWID) in electronic medical records can improve clinical decision making, risk assessment and mitigation, and health service research. Identification of PWID currently consists of heterogeneous, nonspecific International Classification of Diseases (ICD) codes as proxies. Natural language processing (NLP) and machine learning (ML) methods may have better diagnostic metrics than nonspecific ICD codes for identifying PWID.

Methods

We manually reviewed 1000 records of patients diagnosed with Staphylococcus aureus bacteremia admitted to Veterans Health Administration hospitals from 2003 through 2014. The manual review was the reference standard. We developed and trained NLP/ML algorithms with and without regular expression filters for negation (NegEx) and compared these with 11 proxy combinations of ICD codes to identify PWID. Data were split 70% for training and 30% for testing. We calculated diagnostic metrics and estimated 95% confidence intervals (CIs) by bootstrapping the hold-out test set. Best models were determined by best F-score, a summary of sensitivity and positive predictive value.

Results

Random forest with and without NegEx were the best-performing NLP/ML algorithms in the training set. Random forest with NegEx outperformed all ICD-based algorithms. F-score for the best NLP/ML algorithm was 0.905 (95% CI, .786-.967) and 0.592 (95% CI, .550-.632) for the best ICD-based algorithm. The NLP/ML algorithm had a sensitivity of 92.6% and specificity of 95.4%.

Conclusions

NLP/ML outperformed ICD-based coding algorithms at identifying PWID in electronic health records. NLP/ML models should be considered in identifying cohorts of PWID to improve clinical decision making, health services research, and administrative surveillance.

Development and Validation of Machine Models Using Natural Language Processing to Classify Substances Involved in Overdose Deaths

IMPORTANCE

Overdose is one of the leading causes of death in the US; however, surveillance data lag considerably from medical examiner determination of the death to reporting in national surveillance reports.

OBJECTIVE

To automate the classification of deaths related to substances in medical examiner data using natural language processing (NLP) and machine learning (ML).

DESIGN, SETTING, AND PARTICIPANTS

Diagnostic study comparing different natural language processing and machine learning algorithms to identify substances related to overdose in 10 health jurisdictions in the US from January 1, 2020, to December 31, 2020. Unstructured text from 35 433 medical examiner and coroners' death records was examined.

EXPOSURES

Text from each case was manually classified to a substance that was related to the death. Three feature representation methods were used and compared: text frequency-inverse document frequency (TF-IDF), global vectors for word representations (GloVe), and concept unique identifier (CUI) embeddings. Several ML algorithms were trained and best models were selected based on F-scores. The best models were tested on a hold-out test set and results were reported with 95% CIs.

MAIN OUTCOMES AND MEASURES

Text data from death certificates were classified as any opioid, fentanyl, alcohol, cocaine, methamphetamine, heroin, prescription opioid, and an aggregate of other substances. Diagnostic metrics and 95% CIs were calculated for each combination of feature extraction method and machine learning classifier.

RESULTS

Of 35 433 death records analyzed (decedent median age, 58 years [IQR, 41-72 years]; 24 449 [69%] were male), the most common substances related to deaths included any opioid (5739 [16%]), fentanyl (4758 [13%]), alcohol (2866 [8%]), cocaine (2247 [6%]), methamphetamine (1876 [5%]), heroin (1613 [5%]), prescription opioids (1197 [3%]), and any benzodiazepine (1076 [3%]). The CUI embeddings had similar or better diagnostic metrics compared with word embeddings and TF-IDF for all substances except alcohol. ML classifiers had perfect or near perfect performance in classifying deaths related to any opioids, heroin, fentanyl, prescription opioids, methamphetamine, cocaine, and alcohol. Classification of benzodiazepines was suboptimal using all 3 feature extraction methods.

CONCLUSIONS AND RELEVANCE

In this diagnostic study, NLP/ML algorithms demonstrated excellent diagnostic performance at classifying substances related to overdoses. These algorithms should be integrated into workflows to decrease the lag time in reporting overdose surveillance data.

Preventing Overdose Using Information and Data from the Environment (PROVIDENT): protocol for a randomized, population-based, community intervention trial

BACKGROUND AND AIMS

In light of the accelerating drug overdose epidemic in North America, new strategies are needed to identify communities most at risk to prioritize geographically the existing public health resources (e.g. street outreach, naloxone distribution efforts). We aimed to develop PROVIDENT (Preventing Overdose using Information and Data from the Environment), a machine learning-based forecasting tool to predict future overdose deaths at the census block group (i.e. neighbourhood) level.

DESIGN

Randomized, population-based, community intervention trial.

SETTING

Rhode Island, USA.

PARTICIPANTS

All people who reside in Rhode Island during the study period may contribute data to either the model or the trial outcomes.

INTERVENTION

Each of the state's 39 municipalities will be randomized to the intervention (PROVIDENT) or comparator condition. An interactive, web-based tool will be developed to visualize the PROVIDENT model predictions. Municipalities assigned to the treatment arm will receive neighbourhood risk predictions from the PROVIDENT model, and state agencies and community-based organizations will direct resources to neighbourhoods identified as high risk. Municipalities assigned to the control arm will continue to receive surveillance information and overdose prevention resources, but they will not receive neighbourhood risk predictions.

MEASUREMENTS

The primary outcome is the municipal-level rate of fatal and non-fatal drug overdoses. Fatal overdoses will be defined as unintentional drug-related death; non-fatal overdoses will be defined as an emergency department visit for a suspected overdose reported through the state's syndromic surveillance system. Intervention efficacy will be assessed using Poisson or negative binomial regression to estimate incidence rate ratios comparing fatal and non-fatal overdose rates in treatment vs. control municipalities.

COMMENTS

The findings will inform the utility of predictive modelling as a tool to improve public health decision-making and inform resource allocation to communities that should be prioritized for prevention, treatment, recovery and overdose rescue services.

Using Machine Learning for Pharmacovigilance: A Systematic Review

Pharmacovigilance is a science that involves the ongoing monitoring of adverse drug reactions to existing medicines. Traditional approaches in this field can be expensive and time-consuming. The application of natural language processing (NLP) to analyze user-generated content is hypothesized as an effective supplemental source of evidence. In this systematic review, a broad and multi-disciplinary literature search was conducted involving four databases. A total of 5318 publications were initially found. Studies were considered relevant if they reported on the application of NLP to understand user-generated text for pharmacovigilance. A total of 16 relevant publications were included in this systematic review. All studies were evaluated to have medium reliability and validity. For all types of drugs, 14 publications reported positive findings with respect to the identification of adverse drug reactions, providing consistent evidence that natural language processing can be used effectively and accurately on user-generated textual content that was published to the Internet to identify adverse drug reactions for the purpose of pharmacovigilance. The evidence presented in this review suggest that the analysis of textual data has the potential to complement the traditional system of pharmacovigilance.

Big data and predictive modelling for the opioid crisis: existing research and future potential

A need exists to accurately estimate overdose risk and improve understanding of how to deliver treatments and interventions in people with opioid use disorder in a way that reduces such risk. We consider opportunities for predictive analytics and routinely collected administrative data to evaluate how overdose could be reduced among people with opioid use disorder. Specifically, we summarise global trends in opioid use and overdoses; describe the use of big data in research into opioid overdose; consider the potential for predictive modelling, including machine learning, for prevention and monitoring of opioid overdoses; and outline the challenges and risks relating to the use of big data and machine learning in reducing harms that are related to opioid use. Future research for improving the coverage and provision of existing interventions, treatments, and resources for opioid use disorder requires collaboration of multiple agencies. Predictive modelling could transport the concept of stratified medicine to public health through novel methods, such as predictive modelling and emulated trials for evaluating diagnoses and prognoses of opioid use disorder, predicting treatment response, and providing targeted treatment recommendations.

Challenges and Barriers in Applying Natural Language Processing to Medical Examiner Notes from Fatal Opioid Poisoning Cases

We detail the challenges and barriers in applying natural language processing techniques to a collection of medical examiner case investigation notes related to fatal opioid poisonings. Major advances in biomedical informatics have made natural language processing (NLP) of medical texts both a realistic and useful task. Biomedical NLP tools are typically designed to process documents originating from biomedical libraries or electronic health records (EHRs). The usefulness of biomedical NLP tools on texts authored outside of EHRs is unclear, despite an abundance of medicolegal documents existing at the intersection of medicine and law. In particular, we detail our experiences processing unstructured text and extracting semantic concepts using case investigation notes; these notes were authored by trained investigative professionals working in a medical examiner's office and describe cases containing deaths related to fatal opioid poisonings. Applying NLP to case notes is a particularly important step in generalizing the advances of biomedical NLP for other related domains and giving guidance to data scientists working with unstructured data generated outside of EHRs.