1
|
van Diessen E, van Amerongen RA, Zijlmans M, Otte WM. Potential merits and flaws of large language models in epilepsy care: A critical review. Epilepsia 2024; 65:873-886. [PMID: 38305763 DOI: 10.1111/epi.17907] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2023] [Revised: 12/30/2023] [Accepted: 01/19/2024] [Indexed: 02/03/2024]
Abstract
The current pace of development and applications of large language models (LLMs) is unprecedented and will impact future medical care significantly. In this critical review, we provide the background to better understand these novel artificial intelligence (AI) models and how LLMs can be of future use in the daily care of people with epilepsy. Considering the importance of clinical history taking in diagnosing and monitoring epilepsy-combined with the established use of electronic health records-a great potential exists to integrate LLMs in epilepsy care. We present the current available LLM studies in epilepsy. Furthermore, we highlight and compare the most commonly used LLMs and elaborate on how these models can be applied in epilepsy. We further discuss important drawbacks and risks of LLMs, and we provide recommendations for overcoming these limitations.
Collapse
Affiliation(s)
- Eric van Diessen
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
- Department of Pediatrics, Franciscus Gasthuis & Vlietland, Rotterdam, The Netherlands
| | - Ramon A van Amerongen
- Faculty of Science, Bioinformatics and Biocomplexity, Utrecht University, Utrecht, The Netherlands
| | - Maeike Zijlmans
- Department of Neurology and Neurosurgery, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
- Stichting Epilepsie Instellingen Nederland, Heemstede, The Netherlands
| | - Willem M Otte
- Department of Child Neurology, UMC Utrecht Brain Center, University Medical Center Utrecht and Utrecht University, Utrecht, The Netherlands
| |
Collapse
|
2
|
Kerr WT, McFarlane KN. Machine Learning and Artificial Intelligence Applications to Epilepsy: a Review for the Practicing Epileptologist. Curr Neurol Neurosci Rep 2023; 23:869-879. [PMID: 38060133 DOI: 10.1007/s11910-023-01318-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 10/24/2023] [Indexed: 12/08/2023]
Abstract
PURPOSE OF REVIEW Machine Learning (ML) and Artificial Intelligence (AI) are data-driven techniques to translate raw data into applicable and interpretable insights that can assist in clinical decision making. Some of these tools have extremely promising initial results, earning both great excitement and creating hype. This non-technical article reviews recent developments in ML/AI in epilepsy to assist the current practicing epileptologist in understanding both the benefits and limitations of integrating ML/AI tools into their clinical practice. RECENT FINDINGS ML/AI tools have been developed to assist clinicians in almost every clinical decision including (1) predicting future epilepsy in people at risk, (2) detecting and monitoring for seizures, (3) differentiating epilepsy from mimics, (4) using data to improve neuroanatomic localization and lateralization, and (5) tracking and predicting response to medical and surgical treatments. We also discuss practical, ethical, and equity considerations in the development and application of ML/AI tools including chatbots based on Large Language Models (e.g., ChatGPT). ML/AI tools will change how clinical medicine is practiced, but, with rare exceptions, the transferability to other centers, effectiveness, and safety of these approaches have not yet been established rigorously. In the future, ML/AI will not replace epileptologists, but epileptologists with ML/AI will replace epileptologists without ML/AI.
Collapse
Affiliation(s)
- Wesley T Kerr
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Biomedical Informatics, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA.
- Department of Neurology, Michigan Medicine, University of Michigan, Ann Arbor, MI, USA.
| | - Katherine N McFarlane
- Department of Neurology, University of Pittsburgh, 3471 Fifth Ave, Kaufmann 811.22, Pittsburgh, PA, 15213, USA
| |
Collapse
|
3
|
Knight A, Gschwind T, Galer P, Worrell GA, Litt B, Soltesz I, Beniczky S. Artificial intelligence in epilepsy phenotyping. Epilepsia 2023. [PMID: 37983589 DOI: 10.1111/epi.17833] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2023] [Accepted: 11/17/2023] [Indexed: 11/22/2023]
Abstract
Artificial intelligence (AI) allows data analysis and integration at an unprecedented granularity and scale. Here we review the technological advances, challenges, and future perspectives of using AI for electro-clinical phenotyping of animal models and patients with epilepsy. In translational research, AI models accurately identify behavioral states in animal models of epilepsy, allowing identification of correlations between neural activity and interictal and ictal behavior. Clinical applications of AI-based automated and semi-automated analysis of audio and video recordings of people with epilepsy, allow significant data reduction and reliable detection and classification of major motor seizures. AI models can accurately identify electrographic biomarkers of epilepsy, such as spikes, high-frequency oscillations, and seizure patterns. Integrating AI analysis of electroencephalographic, clinical, and behavioral data will contribute to optimizing therapy for patients with epilepsy.
Collapse
Affiliation(s)
- Andrew Knight
- Neuro Event Labs, Tampere University, Tampere, Finland
| | - Tilo Gschwind
- Department of Neurosurgery, Stanford University, Stanford, California, USA
| | - Peter Galer
- Center for Neuroengineering and Therapeutics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Bioengineering, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | | | - Brian Litt
- Center for Neuroengineering and Therapeutics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Bioengineering, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania, USA
| | - Ivan Soltesz
- Department of Neurosurgery, Stanford University, Stanford, California, USA
| | - Sándor Beniczky
- Danish Epilepsy Centre, Filadelfia, Dianalund, Denmark
- Aarhus University Hospital, Aarhus University, Aarhus, Denmark
| |
Collapse
|
4
|
Xie K, Terman SW, Gallagher RS, Hill CE, Davis KA, Litt B, Roth D, Ellis CA. Generalization of finetuned transformer language models to new clinical contexts. JAMIA Open 2023; 6:ooad070. [PMID: 37600072 PMCID: PMC10432353 DOI: 10.1093/jamiaopen/ooad070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 08/01/2023] [Accepted: 08/03/2023] [Indexed: 08/22/2023] Open
Abstract
Objective We have previously developed a natural language processing pipeline using clinical notes written by epilepsy specialists to extract seizure freedom, seizure frequency text, and date of last seizure text for patients with epilepsy. It is important to understand how our methods generalize to new care contexts. Materials and methods We evaluated our pipeline on unseen notes from nonepilepsy-specialist neurologists and non-neurologists without any additional algorithm training. We tested the pipeline out-of-institution using epilepsy specialist notes from an outside medical center with only minor preprocessing adaptations. We examined reasons for discrepancies in performance in new contexts by measuring physical and semantic similarities between documents. Results Our ability to classify patient seizure freedom decreased by at least 0.12 agreement when moving from epilepsy specialists to nonspecialists or other institutions. On notes from our institution, textual overlap between the extracted outcomes and the gold standard annotations attained from manual chart review decreased by at least 0.11 F1 when an answer existed but did not change when no answer existed; here our models generalized on notes from the outside institution, losing at most 0.02 agreement. We analyzed textual differences and found that syntactic and semantic differences in both clinically relevant sentences and surrounding contexts significantly influenced model performance. Discussion and conclusion Model generalization performance decreased on notes from nonspecialists; out-of-institution generalization on epilepsy specialist notes required small changes to preprocessing but was especially good for seizure frequency text and date of last seizure text, opening opportunities for multicenter collaborations using these outcomes.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Samuel W Terman
- Department of Neurology, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Ryan S Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Chloe E Hill
- Department of Neurology, University of Michigan, Ann Arbor, Michigan 48109, USA
| | - Kathryn A Davis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| | - Colin A Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Perelman School of Medicine, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, Pennsylvania 19104, USA
| |
Collapse
|
5
|
Xie K, Gallagher RS, Shinohara RT, Xie SX, Hill CE, Conrad EC, Davis KA, Roth D, Litt B, Ellis CA. Long-term epilepsy outcome dynamics revealed by natural language processing of clinic notes. Epilepsia 2023; 64:1900-1909. [PMID: 37114472 PMCID: PMC10523917 DOI: 10.1111/epi.17633] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/16/2022] [Revised: 04/26/2023] [Accepted: 04/26/2023] [Indexed: 04/29/2023]
Abstract
OBJECTIVE Electronic medical records allow for retrospective clinical research with large patient cohorts. However, epilepsy outcomes are often contained in free text notes that are difficult to mine. We recently developed and validated novel natural language processing (NLP) algorithms to automatically extract key epilepsy outcome measures from clinic notes. In this study, we assessed the feasibility of extracting these measures to study the natural history of epilepsy at our center. METHODS We applied our previously validated NLP algorithms to extract seizure freedom, seizure frequency, and date of most recent seizure from outpatient visits at our epilepsy center from 2010 to 2022. We examined the dynamics of seizure outcomes over time using Markov model-based probability and Kaplan-Meier analyses. RESULTS Performance of our algorithms on classifying seizure freedom was comparable to that of human reviewers (algorithm F1 = .88 vs. human annotatorκ = .86). We extracted seizure outcome data from 55 630 clinic notes from 9510 unique patients written by 53 unique authors. Of these, 30% were classified as seizure-free since the last visit, 48% of non-seizure-free visits contained a quantifiable seizure frequency, and 47% of all visits contained the date of most recent seizure occurrence. Among patients with at least five visits, the probabilities of seizure freedom at the next visit ranged from 12% to 80% in patients having seizures or seizure-free at the prior three visits, respectively. Only 25% of patients who were seizure-free for 6 months remained seizure-free after 10 years. SIGNIFICANCE Our findings demonstrate that epilepsy outcome measures can be extracted accurately from unstructured clinical note text using NLP. At our tertiary center, the disease course often followed a remitting and relapsing pattern. This method represents a powerful new tool for clinical research with many potential uses and extensions to other clinical questions.
Collapse
Affiliation(s)
- Kevin Xie
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Ryan S. Gallagher
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Russell T. Shinohara
- Penn Statistics in Imaging and Visualization Center, Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Sharon X. Xie
- Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Chloe E. Hill
- Department of Neurology, University of Michigan, Ann Arbor, MI, 48109, USA
| | - Erin C. Conrad
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Kathryn A. Davis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Dan Roth
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Brian Litt
- Department of Bioengineering, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| | - Colin A. Ellis
- Center for Neuroengineering and Therapeutics, University of Pennsylvania, Philadelphia, PA, 19104, USA
- Department of Neurology, University of Pennsylvania, Philadelphia, PA, 19104, USA
| |
Collapse
|
6
|
Mummaneni PV, Bydon M. Clinical Databases in Spine Surgery: Strength in Numbers. Neurosurgery 2023; 93:1-3. [PMID: 37318222 DOI: 10.1227/neu.0000000000002465] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/16/2023] Open
Affiliation(s)
- Praveen V Mummaneni
- Department of Neurological Surgery, University of California, San Francisco, San Francisco, California, USA
| | - Mohamad Bydon
- Department of Neurologic Surgery, Mayo Clinic, Rochester, Minnesota, USA
| |
Collapse
|
7
|
Boßelmann CM, Leu C, Lal D. Are AI language models such as ChatGPT ready to improve the care of individuals with epilepsy? Epilepsia 2023; 64:1195-1199. [PMID: 36869421 DOI: 10.1111/epi.17570] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2023] [Revised: 02/23/2023] [Accepted: 03/01/2023] [Indexed: 03/05/2023]
Affiliation(s)
- Christian M Boßelmann
- Genomic Medicine Institute, Lerner Research Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Epilepsy Center, Neurological Institute, Cleveland Clinic, Cleveland, Ohio, USA
| | - Costin Leu
- Genomic Medicine Institute, Lerner Research Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Department of Clinical and Experimental Epilepsy, Institute of Neurology, University College London, London, UK
| | - Dennis Lal
- Genomic Medicine Institute, Lerner Research Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Epilepsy Center, Neurological Institute, Cleveland Clinic, Cleveland, Ohio, USA.,Stanley Center for Psychiatric Research, Broad Institute of Harvard and M.I.T., Cambridge, Massachusetts, USA.,Cologne Center for Genomics (CCG), University of Cologne, Cologne, Delaware, USA
| |
Collapse
|
8
|
López-Ojeda W, Hurley RA. Medical Metaverse, Part 2: Artificial Intelligence Algorithms and Large Language Models in Psychiatry and Clinical Neurosciences. J Neuropsychiatry Clin Neurosci 2023; 35:316-320. [PMID: 37840258 DOI: 10.1176/appi.neuropsych.20230117] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [What about the content of this article? (0)] [Affiliation(s)] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/17/2023]
Affiliation(s)
- Wilfredo López-Ojeda
- Veterans Affairs Mid-Atlantic Mental Illness Research, Education and Clinical Center (MIRECC) and Research and Academic Affairs Service Line, W.G. Hefner Veterans Affairs Medical Center, Salisbury, N.C. (López-Ojeda, Hurley); Department of Psychiatry and Behavioral Medicine (López-Ojeda, Hurley) and Department of Radiology (Hurley), Wake Forest School of Medicine, Winston-Salem, N.C.; Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston (Hurley)
| | - Robin A Hurley
- Veterans Affairs Mid-Atlantic Mental Illness Research, Education and Clinical Center (MIRECC) and Research and Academic Affairs Service Line, W.G. Hefner Veterans Affairs Medical Center, Salisbury, N.C. (López-Ojeda, Hurley); Department of Psychiatry and Behavioral Medicine (López-Ojeda, Hurley) and Department of Radiology (Hurley), Wake Forest School of Medicine, Winston-Salem, N.C.; Menninger Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston (Hurley)
| |
Collapse
|