OUP accepted manuscript

Extracting patient lifestyle characteristics from Dutch clinical text with BERT models

BACKGROUND

BERT models have seen widespread use on unstructured text within the clinical domain. However, little to no research has been conducted into classifying unstructured clinical notes on the basis of patient lifestyle indicators, especially in Dutch. This article aims to test the feasibility of deep BERT models on the task of patient lifestyle classification, as well as introducing an experimental framework that is easily reproducible in future research.

METHODS

This study makes use of unstructured general patient text data from HagaZiekenhuis, a large hospital in The Netherlands. Over 148 000 notes were provided to us, which were each automatically labelled on the basis of the respective patients' smoking, alcohol usage and drug usage statuses. In this paper we test feasibility of automatically assigning labels, and justify it using hand-labelled input. Ultimately, we compare macro F1-scores of string matching, SGD and several BERT models on the task of classifying smoking, alcohol and drug usage. We test Dutch BERT models and English models with translated input.

RESULTS

We find that our further pre-trained MedRoBERTa.nl-HAGA model outperformed every other model on smoking (0.93) and drug usage (0.77). Interestingly, our ClinicalBERT model that was merely fine-tuned on translated text performed best on the alcohol task (0.80). In t-SNE visualisations, we show our MedRoBERTa.nl-HAGA model is the best model to differentiate between classes in the embedding space, explaining its superior classification performance.

CONCLUSIONS

We suggest MedRoBERTa.nl-HAGA to be used as a baseline in future research on Dutch free text patient lifestyle classification. We furthermore strongly suggest further exploring the application of translation to input text in non-English clinical BERT research, as we only translated a subset of the full set and yet achieved very promising results.

Artificial intelligence in epilepsy - applications and pathways to the clinic

Artificial intelligence (AI) is rapidly transforming health care, and its applications in epilepsy have increased exponentially over the past decade. Integration of AI into epilepsy management promises to revolutionize the diagnosis and treatment of this complex disorder. However, translation of AI into neurology clinical practice has not yet been successful, emphasizing the need to consider progress to date and assess challenges and limitations of AI. In this Review, we provide an overview of AI applications that have been developed in epilepsy using a variety of data modalities: neuroimaging, electroencephalography, electronic health records, medical devices and multimodal data integration. For each, we consider potential applications, including seizure detection and prediction, seizure lateralization, localization of the seizure-onset zone and assessment for surgical or neurostimulation interventions, and review the performance of AI tools developed to date. We also discuss methodological considerations and challenges that must be addressed to successfully integrate AI into clinical practice. Our goal is to provide an overview of the current state of the field and provide guidance for leveraging AI in future to improve management of epilepsy.

Disparities in seizure outcomes revealed by large language models

OBJECTIVE

Large-language models (LLMs) can potentially revolutionize health care delivery and research, but risk propagating existing biases or introducing new ones. In epilepsy, social determinants of health are associated with disparities in care access, but their impact on seizure outcomes among those with access remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to determine if different demographic groups have different seizure outcomes.

MATERIALS AND METHODS

We tested our LLM for differences and equivalences in prediction accuracy and confidence across demographic groups defined by race, ethnicity, sex, income, and health insurance, using manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for demographic outcome disparities, using univariable and multivariable analyses.

RESULTS

We analyzed 84 675 clinic visits from 25 612 unique patients seen at our epilepsy center. We found little evidence of bias in the prediction accuracy or confidence of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, P ≤ .001), those with public insurance (OR 1.53, P ≤ .001), and those from lower-income zip codes (OR ≥1.22, P  ≤ .007). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, P = .66).

CONCLUSION

We found little evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings quantify the critical need to reduce disparities in the care of people with epilepsy.

Potential merits and flaws of large language models in epilepsy care: A critical review

The current pace of development and applications of large language models (LLMs) is unprecedented and will impact future medical care significantly. In this critical review, we provide the background to better understand these novel artificial intelligence (AI) models and how LLMs can be of future use in the daily care of people with epilepsy. Considering the importance of clinical history taking in diagnosing and monitoring epilepsy-combined with the established use of electronic health records-a great potential exists to integrate LLMs in epilepsy care. We present the current available LLM studies in epilepsy. Furthermore, we highlight and compare the most commonly used LLMs and elaborate on how these models can be applied in epilepsy. We further discuss important drawbacks and risks of LLMs, and we provide recommendations for overcoming these limitations.

Alcohol for seizure induction in the epilepsy monitoring unit

RATIONALE

Seizure induction techniques are used in the epilepsy monitoring unit (EMU) to increase diagnostic yield and reduce length of stay. There are insufficient data on the efficacy of alcohol as an induction technique.

METHODS

We performed a retrospective cohort study using six years of EMU data at our institution. We compared cases who received alcohol for seizure induction to matched controls who did not. The groups were matched on the following variables: age, reason for admission, length of stay, number of antiseizure medications (ASM) at admission, whether ASMs were tapered during admission, and presence of interictal epileptiform discharges. We used both propensity score and exact matching strategies. We compared the likelihood of epileptic seizures and nonepileptic events in cases versus controls using Kaplan-Meier time-to-event analysis, as well as odds ratios for these outcomes occurring at any time during the admission.

RESULTS

We analyzed 256 cases who received alcohol (median dose 2.5 standard drinks) and 256 propensity score-matched controls. Cases who received alcohol were no more likely than controls to have an epileptic seizure (X2(1) = 0.01, p = 0.93) or nonepileptic event (X2(1) = 2.1, p = 0.14) in the first 48 h after alcohol administration. For the admission overall, cases were no more likely to have an epileptic seizure (OR 0.89, 95 % CI 0.61-1.28, p = 0.58), nonepileptic event (OR 0.97, CI 0.62-1.53, p = 1.00), nor require rescue benzodiazepine (OR 0.63, CI 0.35-1.12, p = 0.15). Stratified analyses revealed no increased risk of epileptic seizure in any subgroups. Sensitivity analysis using exact matching showed that results were robust to matching strategy.

CONCLUSIONS

Alcohol was not an effective induction technique in the EMU. This finding has implications for counseling patients with epilepsy about the risks of drinking alcohol in moderation in their daily lives.

Large Language Models in Neurology Research and Future Practice

Recent advancements in generative artificial intelligence, particularly using large language models (LLMs), are gaining increased public attention. We provide a perspective on the potential of LLMs to analyze enormous amounts of data from medical records and gain insights on specific topics in neurology. In addition, we explore use cases for LLMs, such as early diagnosis, supporting patient and caregivers, and acting as an assistant for clinicians. We point to the potential ethical and technical challenges raised by LLMs, such as concerns about privacy and data security, potential biases in the data for model training, and the need for careful validation of results. Researchers must consider these challenges and take steps to address them to ensure that their work is conducted in a safe and responsible manner. Despite these challenges, LLMs offer promising opportunities for improving care and treatment of various neurologic disorders.

Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist

PURPOSE OF REVIEW

Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice.

RECENT FINDINGS

ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.

Improving Neurology Clinical Care With Natural Language Processing Tools

The integration of natural language processing (NLP) tools into neurology workflows has the potential to significantly enhance clinical care. However, it is important to address the limitations and risks associated with integrating this new technology. Recent advances in transformer-based NLP algorithms (e.g., GPT, BERT) could augment neurology clinical care by summarizing patient health information, suggesting care options, and assisting research involving large datasets. However, these NLP platforms have potential risks including fabricated facts and data security and substantial barriers for implementation. Although these risks and barriers need to be considered, the benefits for providers, patients, and communities are substantial. With these systems achieving greater functionality and the pace of medical need increasing, integrating these tools into clinical care may prove not only beneficial but necessary. Further investigation is needed to design implementation strategies, mitigate risks, and overcome barriers.

Artificial intelligence in epilepsy phenotyping

Artificial intelligence (AI) allows data analysis and integration at an unprecedented granularity and scale. Here we review the technological advances, challenges, and future perspectives of using AI for electro-clinical phenotyping of animal models and patients with epilepsy. In translational research, AI models accurately identify behavioral states in animal models of epilepsy, allowing identification of correlations between neural activity and interictal and ictal behavior. Clinical applications of AI-based automated and semi-automated analysis of audio and video recordings of people with epilepsy, allow significant data reduction and reliable detection and classification of major motor seizures. AI models can accurately identify electrographic biomarkers of epilepsy, such as spikes, high-frequency oscillations, and seizure patterns. Integrating AI analysis of electroencephalographic, clinical, and behavioral data will contribute to optimizing therapy for patients with epilepsy.

Generalization of finetuned transformer language models to new clinical contexts

Objective

We have previously developed a natural language processing pipeline using clinical notes written by epilepsy specialists to extract seizure freedom, seizure frequency text, and date of last seizure text for patients with epilepsy. It is important to understand how our methods generalize to new care contexts.

Materials and methods

We evaluated our pipeline on unseen notes from nonepilepsy-specialist neurologists and non-neurologists without any additional algorithm training. We tested the pipeline out-of-institution using epilepsy specialist notes from an outside medical center with only minor preprocessing adaptations. We examined reasons for discrepancies in performance in new contexts by measuring physical and semantic similarities between documents.

Results

Our ability to classify patient seizure freedom decreased by at least 0.12 agreement when moving from epilepsy specialists to nonspecialists or other institutions. On notes from our institution, textual overlap between the extracted outcomes and the gold standard annotations attained from manual chart review decreased by at least 0.11 F1 when an answer existed but did not change when no answer existed; here our models generalized on notes from the outside institution, losing at most 0.02 agreement. We analyzed textual differences and found that syntactic and semantic differences in both clinically relevant sentences and surrounding contexts significantly influenced model performance.

Discussion and conclusion

Model generalization performance decreased on notes from nonspecialists; out-of-institution generalization on epilepsy specialist notes required small changes to preprocessing but was especially good for seizure frequency text and date of last seizure text, opening opportunities for multicenter collaborations using these outcomes.

Disparities in seizure outcomes revealed by large language models

Objective

Large-language models (LLMs) in healthcare have the potential to propagate existing biases or introduce new ones. For people with epilepsy, social determinants of health are associated with disparities in access to care, but their impact on seizure outcomes among those with access to specialty care remains unclear. Here we (1) evaluated our validated, epilepsy-specific LLM for intrinsic bias, and (2) used LLM-extracted seizure outcomes to test the hypothesis that different demographic groups have different seizure outcomes.

Methods

First, we tested our LLM for intrinsic bias in the form of differential performance in demographic groups by race, ethnicity, sex, income, and health insurance in manually annotated notes. Next, we used LLM-classified seizure freedom at each office visit to test for outcome disparities in the same demographic groups, using univariable and multivariable analyses.

Results

We analyzed 84,675 clinic visits from 25,612 patients seen at our epilepsy center 2005-2022. We found no differences in the accuracy, or positive or negative class balance of outcome classifications across demographic groups. Multivariable analysis indicated worse seizure outcomes for female patients (OR 1.33, p = 3×10-8), those with public insurance (OR 1.53, p = 2×10-13), and those from lower-income zip codes (OR ≥ 1.22, p ≤ 6.6×10-3). Black patients had worse outcomes than White patients in univariable but not multivariable analysis (OR 1.03, p = 0.66).

Significance

We found no evidence that our LLM was intrinsically biased against any demographic group. Seizure freedom extracted by LLM revealed disparities in seizure outcomes across several demographic groups. These findings highlight the critical need to reduce disparities in the care of people with epilepsy.

Long-term epilepsy outcome dynamics revealed by natural language processing of clinic notes

OBJECTIVE

Electronic medical records allow for retrospective clinical research with large patient cohorts. However, epilepsy outcomes are often contained in free text notes that are difficult to mine. We recently developed and validated novel natural language processing (NLP) algorithms to automatically extract key epilepsy outcome measures from clinic notes. In this study, we assessed the feasibility of extracting these measures to study the natural history of epilepsy at our center.

METHODS

We applied our previously validated NLP algorithms to extract seizure freedom, seizure frequency, and date of most recent seizure from outpatient visits at our epilepsy center from 2010 to 2022. We examined the dynamics of seizure outcomes over time using Markov model-based probability and Kaplan-Meier analyses.

RESULTS

Performance of our algorithms on classifying seizure freedom was comparable to that of human reviewers (algorithm F1 = .88 vs. human annotatorκ = .86). We extracted seizure outcome data from 55 630 clinic notes from 9510 unique patients written by 53 unique authors. Of these, 30% were classified as seizure-free since the last visit, 48% of non-seizure-free visits contained a quantifiable seizure frequency, and 47% of all visits contained the date of most recent seizure occurrence. Among patients with at least five visits, the probabilities of seizure freedom at the next visit ranged from 12% to 80% in patients having seizures or seizure-free at the prior three visits, respectively. Only 25% of patients who were seizure-free for 6 months remained seizure-free after 10 years.

SIGNIFICANCE

Our findings demonstrate that epilepsy outcome measures can be extracted accurately from unstructured clinical note text using NLP. At our tertiary center, the disease course often followed a remitting and relapsing pattern. This method represents a powerful new tool for clinical research with many potential uses and extensions to other clinical questions.

Heart disease risk factors detection from electronic health records using advanced NLP and deep learning techniques

Heart disease remains the major cause of death, despite recent improvements in prediction and prevention. Risk factor identification is the main step in diagnosing and preventing heart disease. Automatically detecting risk factors for heart disease in clinical notes can help with disease progression modeling and clinical decision-making. Many studies have attempted to detect risk factors for heart disease, but none have identified all risk factors. These studies have proposed hybrid systems that combine knowledge-driven and data-driven techniques, based on dictionaries, rules, and machine learning methods that require significant human effort. The National Center for Informatics for Integrating Biology and Beyond (i2b2) proposed a clinical natural language processing (NLP) challenge in 2014, with a track (track2) focused on detecting risk factors for heart disease risk factors in clinical notes over time. Clinical narratives provide a wealth of information that can be extracted using NLP and Deep Learning techniques. The objective of this paper is to improve on previous work in this area as part of the 2014 i2b2 challenge by identifying tags and attributes relevant to disease diagnosis, risk factors, and medications by providing advanced techniques of using stacked word embeddings. The i2b2 heart disease risk factors challenge dataset has shown significant improvement by using the approach of stacking embeddings, which combines various embeddings. Our model achieved an F1 score of 93.66% by using BERT and character embeddings (CHARACTER-BERT Embedding) stacking. The proposed model has significant results compared to all other models and systems that we developed for the 2014 i2b2 challenge.

A pharmacokinetic model of antiseizure medication load to guide care in the epilepsy monitoring unit

OBJECTIVE

Evaluating patients with drug-resistant epilepsy often requires inducing seizures by tapering antiseizure medications (ASMs) in the epilepsy monitoring unit (EMU). The relationship between ASM taper strategy, seizure timing, and severity remains unclear. In this study, we developed and validated a pharmacokinetic model of total ASM load and tested its association with seizure occurrence and severity in the EMU.

METHODS

We studied 80 patients who underwent intracranial electroencephalographic recording for epilepsy surgery planning. We developed a first order pharmacokinetic model of the ASMs administered in the EMU to generate a continuous metric of overall ASM load. We then related modeled ASM load to seizure likelihood and severity. We determined the association between the rate of ASM load reduction, the length of hospital stay, and the probability of having a severe seizure. Finally, we used modeled ASM load to predict oncoming seizures.

RESULTS

Seizures occurred in the bottom 50th percentile of sampled ASM loads across the cohort (p < .0001, Wilcoxon signed-rank test), and seizures requiring rescue therapy occurred at lower ASM loads than seizures that did not require rescue therapy (logistic regression mixed effects model, odds ratio = .27, p = .01). Greater ASM decrease early in the EMU was not associated with an increased likelihood of having a severe seizure, nor with a shorter length of stay.

SIGNIFICANCE

A pharmacokinetic model can accurately estimate ASM levels for patients in the EMU. Lower modeled ASM levels are associated with increased seizure likelihood and seizure severity. We show that ASM load, rather than ASM taper speed, is associated with severe seizures. ASM modeling has the potential to help optimize taper strategy to minimize severe seizures while maximizing diagnostic yield.

Review of Machine Learning and Artificial Intelligence (ML/AI) for the Pediatric Neurologist

Artificial intelligence (AI) and a popular branch of AI known as machine learning (ML) are increasingly being utilized in medicine and to inform medical research. This review provides an overview of AI and ML (AI/ML), including definitions of common terms. We discuss the history of AI and provide instances of how AI/ML can be applied to pediatric neurology. Examples include imaging in neuro-oncology, autism diagnosis, diagnosis from charts, epilepsy, cerebral palsy, and neonatal neurology. Topics such as supervised learning, unsupervised learning, and reinforcement learning are discussed.

Transforming epilepsy research: A systematic review on natural language processing applications

Despite improved ancillary investigations in epilepsy care, patients' narratives remain indispensable for diagnosing and treatment monitoring. This wealth of information is typically stored in electronic health records and accumulated in medical journals in an unstructured manner, thereby restricting complete utilization in clinical decision-making. To this end, clinical researchers increasing apply natural language processing (NLP)-a branch of artificial intelligence-as it removes ambiguity, derives context, and imbues standardized meaning from free-narrative clinical texts. This systematic review presents an overview of the current NLP applications in epilepsy and discusses the opportunities and drawbacks of NLP alongside its future implications. We searched the PubMed and Embase databases with a "natural language processing" and "epilepsy" query (March 4, 2022) and included original research articles describing the application of NLP techniques for textual analysis in epilepsy. Twenty-six studies were included. Fifty-eight percent of these studies used NLP to classify clinical records into predefined categories, improving patient identification and treatment decisions. Other applications of NLP had structured clinical information retrieval from electronic health records, scientific papers, and online posts of patients. Challenges and opportunities of NLP applications for enhancing epilepsy care and research are discussed. The field could further benefit from NLP by replicating successes in other health care domains, such as NLP-aided quality evaluation for clinical decision-making, outcome prediction, and clinical record summarization.

Machine learning approaches for electronic health records phenotyping: a methodical review

OBJECTIVE

Accurate and rapid phenotyping is a prerequisite to leveraging electronic health records for biomedical research. While early phenotyping relied on rule-based algorithms curated by experts, machine learning (ML) approaches have emerged as an alternative to improve scalability across phenotypes and healthcare settings. This study evaluates ML-based phenotyping with respect to (1) the data sources used, (2) the phenotypes considered, (3) the methods applied, and (4) the reporting and evaluation methods used.

MATERIALS AND METHODS

We searched PubMed and Web of Science for articles published between 2018 and 2022. After screening 850 articles, we recorded 37 variables on 100 studies.

RESULTS

Most studies utilized data from a single institution and included information in clinical notes. Although chronic conditions were most commonly considered, ML also enabled the characterization of nuanced phenotypes such as social determinants of health. Supervised deep learning was the most popular ML paradigm, while semi-supervised and weakly supervised learning were applied to expedite algorithm development and unsupervised learning to facilitate phenotype discovery. ML approaches did not uniformly outperform rule-based algorithms, but deep learning offered a marginal improvement over traditional ML for many conditions.

DISCUSSION

Despite the progress in ML-based phenotyping, most articles focused on binary phenotypes and few articles evaluated external validity or used multi-institution data. Study settings were infrequently reported and analytic code was rarely released.

CONCLUSION

Continued research in ML-based phenotyping is warranted, with emphasis on characterizing nuanced phenotypes, establishing reporting and evaluation standards, and developing methods to accommodate misclassified phenotypes due to algorithm errors in downstream applications.

Machine understanding surgical actions from intervention procedure textbooks

The automatic extraction of procedural surgical knowledge from surgery manuals, academic papers or other high-quality textual resources, is of the utmost importance to develop knowledge-based clinical decision support systems, to automatically execute some procedure's step or to summarize the procedural information, spread throughout the texts, in a structured form usable as a study resource by medical students. In this work, we propose a first benchmark on extracting detailed surgical actions from available intervention procedure textbooks and papers. We frame the problem as a Semantic Role Labeling task. Exploiting a manually annotated dataset, we apply different Transformer-based information extraction methods. Starting from RoBERTa and BioMedRoBERTa pre-trained language models, we first investigate a zero-shot scenario and compare the obtained results with a full fine-tuning setting. We then introduce a new ad-hoc surgical language model, named SurgicBERTa, pre-trained on a large collection of surgical materials, and we compare it with the previous ones. In the assessment, we explore different dataset splits (one in-domain and two out-of-domain) and we investigate also the effectiveness of the approach in a few-shot learning scenario. Performance is evaluated on three correlated sub-tasks: predicate disambiguation, semantic argument disambiguation and predicate-argument disambiguation. Results show that the fine-tuning of a pre-trained domain-specific language model achieves the highest performance on all splits and on all sub-tasks. All models are publicly released.

Development of a natural language processing algorithm to extract seizure types and frequencies from the electronic health record

OBJECTIVE

To develop a natural language processing (NLP) algorithm to abstract seizure types and frequencies from electronic health records (EHR).

BACKGROUND

Seizure frequency measurement is an epilepsy quality metric. Yet, abstraction of seizure frequency from the EHR is laborious. We present an NLP algorithm to extract seizure data from unstructured text of clinic notes. Algorithm performance was assessed at two epilepsy centers.

METHODS

We developed a rules-based NLP algorithm to recognize terms related to seizures and frequency within the text of an outpatient encounter. Algorithm output (e.g. number of seizures of a particular type within a time interval) was compared to seizure data manually annotated by two expert reviewers ("gold standard"). The algorithm was developed from 150 clinic notes from institution #1 (development set), then tested on a separate set of 219 notes from institution #1 (internal test set) with 248 unique seizure frequency elements. The algorithm was separately applied to 100 notes from institution #2 (external test set) with 124 unique seizure frequency elements. Algorithm performance was measured by recall (sensitivity), precision (positive predictive value), and F1 score (geometric mean of precision and recall).

RESULTS

In the internal test set, the algorithm demonstrated 70% recall (173/248), 95% precision (173/182), and 0.82 F1 score compared to manual review. Algorithm performance in the external test set was lower with 22% recall (27/124), 73% precision (27/37), and 0.40 F1 score.

CONCLUSIONS

These results suggest NLP extraction of seizure types and frequencies is feasible, though not without challenges in generalizability for large-scale implementation.