Development and Validation of Machine Models Using Natural Language Processing to Classify Substances Involved in Overdose Deaths

Leveraging pooled medical examiner records to surveil complex and emerging patterns of polysubstance use in the United States

BACKGROUND

The United States (US) is an extreme global outlier for drug-related death rates. However, data describing drug-related deaths are generally available only on an 8-13-month lag. Furthermore, granular details about substance-involvement are often not available, which particularly stymies efforts to track fatal polysubstance and novel psychoactive substance use. Detailed medical examiner records provide a powerful source of information for drug-related death surveillance, but have been underutilized.

METHODS

We pooled medical examiner data from five US states and 14 counties that together comprise 18% of the US population to examine demographic, geographic, and drug-specific trends in polysubstance drug-related deaths. We employed mixed effects logistic regression to identify demographic factors associated with polysubstance rather than single substance drug-related deaths. We assessed the correlations between drug classes and described geographic variation in the prevalence of specific drugs and the presence of novel and emerging psychoactive substances.

RESULTS

Our sample included 73,077 drug-related deaths from 2012 through early 2022. Nearly two-thirds of drug-related deaths were polysubstance-involved, with the number and percentage growing annually. High percentages of polysubstance drug-related deaths were observed in both urban and rural jurisdictions. After adjusting for year and jurisdiction, female, American Indian and Alaska Native, and White individuals had the most elevated odds of polysubstance drug-related deaths. Drug-related deaths involving benzodiazepines or opioids, whether pharmaceutical or illicit, and other pharmaceutical drugs were most likely to have polysubstance involvement, while methamphetamine-involved deaths were least likely to involve multiple substances. Strong correlations were observed between prescription opioids and prescription benzodiazepines, fentanyl and xylazine, and designer benzodiazepines and novel synthetic opioids.

CONCLUSIONS

Analysis of detailed medical examiner records reveals the breadth and complexity of polysubstance drug-related deaths in the US. Future efforts to use this unique resource can improve population-based surveillance of drug-related deaths to better tailor interventions and solutions to this critical health crisis.

Question-answering system extracts information on injection drug use from clinical notes

BACKGROUND

Injection drug use (IDU) can increase mortality and morbidity. Therefore, identifying IDU early and initiating harm reduction interventions can benefit individuals at risk. However, extracting IDU behaviors from patients' electronic health records (EHR) is difficult because there is no other structured data available, such as International Classification of Disease (ICD) codes, and IDU is most often documented in unstructured free-text clinical notes. Although natural language processing can efficiently extract this information from unstructured data, there are no validated tools.

METHODS

To address this gap in clinical information, we design a question-answering (QA) framework to extract information on IDU from clinical notes for use in clinical operations. Our framework involves two main steps: (1) generating a gold-standard QA dataset and (2) developing and testing the QA model. We use 2323 clinical notes of 1145 patients curated from the US Department of Veterans Affairs (VA) Corporate Data Warehouse to construct the gold-standard dataset for developing and evaluating the QA model. We also demonstrate the QA model's ability to extract IDU-related information from temporally out-of-distribution data.

RESULTS

Here, we show that for a strict match between gold-standard and predicted answers, the QA model achieves a 51.65% F1 score. For a relaxed match between the gold-standard and predicted answers, the QA model obtains a 78.03% F1 score, along with 85.38% Precision and 79.02% Recall scores. Moreover, the QA model demonstrates consistent performance when subjected to temporally out-of-distribution data.

CONCLUSIONS

Our study introduces a QA framework designed to extract IDU information from clinical notes, aiming to enhance the accurate and efficient detection of people who inject drugs, extract relevant information, and ultimately facilitate informed patient care.

Medical examiner response to the drug overdose epidemic in King County Washington: "Real-time" surveillance, data science, and applied forensic epidemiology

As the overdose epidemic overwhelmed medicolegal death investigation offices and toxicology laboratories, the King County Medical Examiner's Office responded with "real-time" fatal overdose surveillance to expedite death certification and information dissemination through assembling a team including a dedicated medicolegal death investigator, an information coordinator, and student interns. In-house testing of blood, urine, and drug evidence from scenes was performed using equipment and supplies purchased for surveillance. Collaboration with state laboratories allowed validation. Applied forensic epidemiology accelerated data dissemination. From 2010 to 2022, the epidemic claimed 5815 lives in King County; the last 4 years accounted for 47% of those deaths. After initiating the surveillance project, in-house testing was performed on blood from 2836 decedents, urine from 2807, and 4238 drug evidence items from 1775 death scenes. Time to complete death certificates decreased from weeks to months to hours to days. Overdose-specific information was distributed weekly to a network of law enforcement and public health agencies. As the surveillance project tracked the epidemic, fentanyl and methamphetamine became dominant and were associated with other indicators of social deterioration. In 2022, fentanyl was involved in 68% of 1021 overdose deaths. Homeless deaths increased sixfold; in 2022, 67% of 311 homeless deaths were due to overdose; fentanyl was involved in 49% and methamphetamine in 44%. Homicides increased 250%; in 2021, methamphetamine was positive in 35% of 149 homicides. The results are relevant to the value of rapid surveillance, its impact on standard operations, selection of cases requiring autopsy, and collaboration with other agencies in overdose prevention.

Speak and you shall predict: speech at initial cocaine abstinence as a biomarker of long-term drug use behavior

Importance

Valid biomarkers that can predict longitudinal clinical outcomes at low cost are a holy grail in psychiatric research, promising to ultimately be used to optimize and tailor intervention and prevention efforts.

Objective

To determine if baseline linguistic markers in natural speech, as compared to non-speech clinical and demographic measures, can predict drug use severity measures at future sessions in initially abstinent individuals with cocaine use disorder (iCUD).

Design

A longitudinal cohort study (August 2017 - March 2020), where baseline measures were used to predict outcomes collected at three-month intervals for up to one year of follow-up.

Participants

Eighty-eight initially abstinent iCUD were studied at baseline; 57 (46 male, age 50.7+/-7.9 years) came back for at least another session.

Main Outcomes and Measures

Outcomes were self-reported symptoms of withdrawal, craving, abstinence duration and frequency of cocaine use in the past 90 days at each study session. The predictors were derived from 5-min recordings of vocal descriptions of the positive consequences of abstinence and the negative consequences of using cocaine; the baseline cocaine and other common drug use measures, demographic and neuropsychological variables were used for comparison.

Results

Models using the non-speech variables showed the best predictive performance at three(r>0.45, P<2×10-3) and six months follow-up (r>0.37, P<3×10-2). At 12 months, the natural language processing-based model showed significant correlations with withdrawal (r=0.43, P=3×10-2), craving (r=0.72, P=5×10-5), days of abstinence (r=0.76, P=1×10-5), and cocaine use in the past 90 days (r=0.61, P=2×10-3), significantly outperforming the other models for abstinence prediction.

Conclusions and Relevance

At short time intervals, maximal predictive power was obtained with models that used baseline drug use (in addition to demographic and neuropsychological) measures, potentially reflecting a slow rate of change in these measures, which could be estimated by linear functions. In contrast, short speech samples predicted longer-term changes in drug use, implying deeper penetrance by potentially capturing non-linear dynamics over longer intervals. Results suggest that, compared to the common outcome measures used in clinical trials, speech-based measures could be leveraged as better predictors of longitudinal drug use outcomes in initially abstinent iCUD, as potentially generalizable to other substance use disorders and related comorbidity.

Natural Language Processing and Machine Learning to Identify People Who Inject Drugs in Electronic Health Records

Background

Improving the identification of people who inject drugs (PWID) in electronic medical records can improve clinical decision making, risk assessment and mitigation, and health service research. Identification of PWID currently consists of heterogeneous, nonspecific International Classification of Diseases (ICD) codes as proxies. Natural language processing (NLP) and machine learning (ML) methods may have better diagnostic metrics than nonspecific ICD codes for identifying PWID.

Methods

We manually reviewed 1000 records of patients diagnosed with Staphylococcus aureus bacteremia admitted to Veterans Health Administration hospitals from 2003 through 2014. The manual review was the reference standard. We developed and trained NLP/ML algorithms with and without regular expression filters for negation (NegEx) and compared these with 11 proxy combinations of ICD codes to identify PWID. Data were split 70% for training and 30% for testing. We calculated diagnostic metrics and estimated 95% confidence intervals (CIs) by bootstrapping the hold-out test set. Best models were determined by best F-score, a summary of sensitivity and positive predictive value.

Results

Random forest with and without NegEx were the best-performing NLP/ML algorithms in the training set. Random forest with NegEx outperformed all ICD-based algorithms. F-score for the best NLP/ML algorithm was 0.905 (95% CI, .786-.967) and 0.592 (95% CI, .550-.632) for the best ICD-based algorithm. The NLP/ML algorithm had a sensitivity of 92.6% and specificity of 95.4%.

Conclusions

NLP/ML outperformed ICD-based coding algorithms at identifying PWID in electronic health records. NLP/ML models should be considered in identifying cohorts of PWID to improve clinical decision making, health services research, and administrative surveillance.