Elevated Mortality Rate in Patients With Functional Seizures After Diagnosis and Referral

Supervised machine learning compared to large language models for identifying functional seizures from medical records

OBJECTIVE

The Functional Seizures Likelihood Score (FSLS) is a supervised machine learning-based diagnostic score that was developed to differentiate functional seizures (FS) from epileptic seizures (ES). In contrast to this targeted approach, large language models (LLMs) can identify patterns in data for which they were not specifically trained. To evaluate the relative benefits of each approach, we compared the diagnostic performance of the FSLS to two LLMs: ChatGPT and GPT-4.

METHODS

In total, 114 anonymized cases were constructed based on patients with documented FS, ES, mixed ES and FS, or physiologic seizure-like events (PSLEs). Text-based data were presented in three sequential prompts to the LLMs, showing the history of present illness (HPI), electroencephalography (EEG) results, and neuroimaging results. We compared the accuracy (number of correct predictions/number of cases) and area under the receiver-operating characteristic (ROC) curves (AUCs) of the LLMs to the FSLS using mixed-effects logistic regression.

RESULTS

The accuracy of FSLS was 74% (95% confidence interval [CI] 65%-82%) and the AUC was 85% (95% CI 77%-92%). GPT-4 was superior to both the FSLS and ChatGPT (p <.001), with an accuracy of 85% (95% CI 77%-91%) and AUC of 87% (95% CI 79%-95%). Cohen's kappa between the FSLS and GPT-4 was 40% (fair). The LLMs provided different predictions on different days when the same note was provided for 33% of patients, and the LLM's self-rated certainty was moderately correlated with this observed variability (Spearman's rho2: 30% [fair, ChatGPT] and 63% [substantial, GPT-4]).

SIGNIFICANCE

Both GPT-4 and the FSLS identified a substantial subset of patients with FS based on clinical history. The fair agreement in predictions highlights that the LLMs identified patients differently from the structured score. The inconsistency of the LLMs' predictions across days and incomplete insight into their own consistency was concerning. This comparison highlights both benefits and cautions about how machine learning and artificial intelligence could identify patients with FS in clinical practice.

Do we intervene less and slower in the epilepsy monitoring unit for psychogenic seizures?

PURPOSE

Individuals with psychogenic non-epileptic seizures (PNES) can be stigmatized in healthcare settings. We aimed to compare intervention rate (IR), intervention time (IT), and adverse event (AE) rate between PNES and epileptic seizures (ES) in the epilepsy monitoring unit (EMU).

METHODS

We used a prospective database of consecutive admissions to our centre's EMU between August 2021 and September 2022. We excluded purely electric seizures and vague, minor spells with no EEG correlate. We therefore only included electroclinical seizures and PNES. We compared the IR, IT, and AE rate between PNES and ES, as diagnosed by an epileptologist during EEG monitoring. We performed the same comparisons between spells occurring in people admitted with a high vs low suspicion of PNES (HSP vs LSP). We also verified if ITs became longer with repeated PNES.

RESULTS

We analyzed 586 spells: 43 PNES vs 543 ES, or 133 HSP vs 453 LSP. Our univariate analyses showed that IR was higher for PNES than for ES (93 % vs 61 %, p <.001) but that IT and AE rate were similar across groups. This higher IR was only apparent outside weekday daytime hours, when EEG technologists were not present. HSP did not differ from LSP in terms of IR, IT, and AE rate. As PNES accumulated in individual patients, IT tended to be longer (Spearman's correlation = 0.42; p =.012).

SIGNIFICANCE

Our EMU staff did not intervene less or slower for PNES. Rather, IR was higher for PNES than for ES, but IT tended to be longer with repeat PNES.