Diagnostic accuracy of an artificial intelligence online engine in migraine: A multi‐center study

The Ethical Stewardship of Artificial Intelligence in Chronic Pain and Headache: A Narrative Review

PURPOSE OF REVIEW

As artificial intelligence (AI) and machine learning (ML) are becoming more pervasive in medicine, understanding their ethical considerations for chronic pain and headache management is crucial for optimizing their safety.

RECENT FINDINGS

We reviewed thirty-eight editorial and original research articles published between 2018 and 2023, focusing on the application of AI and ML to chronic pain or headache. The core medical principles of beneficence, non-maleficence, autonomy, and justice constituted the evaluation framework. The AI applications addressed topics such as pain intensity prediction, diagnostic aides, risk assessment for medication misuse, empowering patients to self-manage their conditions, and optimizing access to care. Virtually all AI applications aligned both positively and negatively with specific medical ethics principles. This review highlights the potential of AI to enhance patient outcomes and physicians' experiences in managing chronic pain and headache. We emphasize the importance of carefully considering the advantages, disadvantages, and unintended consequences of utilizing AI tools in chronic pain and headache, and propose the four core principles of medical ethics as an evaluation framework.

Approach to the Patient With Headache

OBJECTIVE

The evaluation of patients with headache relies heavily on the history. This article reviews key questions for diagnosing primary and secondary headache disorders with a rationale for each and phrasing to optimize the information obtained and the patient's experience.

LATEST DEVELOPMENTS

The availability of online resources for clinicians and patients continues to increase, including sites that use artificial intelligence to generate a diagnosis and report based on patient responses online. Patient-friendly headache apps include calendars that help track treatment response, identify triggers, and provide educational information.

ESSENTIAL POINTS

A structured approach to taking the history, incorporating online resources and other technologies when needed, facilitates making an accurate diagnosis and often eliminates the need for unnecessary testing. A detailed yet empathetic approach incorporating interpersonal skills enhances relationship building and trust, both of which are integral to successful treatment.

A large language model-based generative natural language processing framework fine-tuned on clinical notes accurately extracts headache frequency from electronic health records

OBJECTIVE

To develop a natural language processing (NLP) algorithm that can accurately extract headache frequency from free-text clinical notes.

BACKGROUND

Headache frequency, defined as the number of days with any headache in a month (or 4 weeks), remains a key parameter in the evaluation of treatment response to migraine preventive medications. However, due to the variations and inconsistencies in documentation by clinicians, significant challenges exist to accurately extract headache frequency from the electronic health record (EHR) by traditional NLP algorithms.

METHODS

This was a retrospective cross-sectional study with patients identified from two tertiary headache referral centers, Mayo Clinic Arizona and Mayo Clinic Rochester. All neurology consultation notes written by 15 specialized clinicians (11 headache specialists and 4 nurse practitioners) between 2012 and 2022 were extracted and 1915 notes were used for model fine-tuning (90%) and testing (10%). We employed four different NLP frameworks: (1) ClinicalBERT (Bidirectional Encoder Representations from Transformers) regression model, (2) Generative Pre-Trained Transformer-2 (GPT-2) Question Answering (QA) model zero-shot, (3) GPT-2 QA model few-shot training fine-tuned on clinical notes, and (4) GPT-2 generative model few-shot training fine-tuned on clinical notes to generate the answer by considering the context of included text.

RESULTS

The mean (standard deviation) headache frequency of our training and testing datasets were 13.4 (10.9) and 14.4 (11.2), respectively. The GPT-2 generative model was the best-performing model with an accuracy of 0.92 (0.91, 0.93, 95% confidence interval [CI]) and R2 score of 0.89 (0.87, 0.90, 95% CI), and all GPT-2-based models outperformed the ClinicalBERT model in terms of exact matching accuracy. Although the ClinicalBERT regression model had the lowest accuracy of 0.27 (0.26, 0.28), it demonstrated a high R2 score of 0.88 (0.85, 0.89), suggesting the ClinicalBERT model can reasonably predict the headache frequency within a range of ≤ ± 3 days, and the R2 score was higher than the GPT-2 QA zero-shot model or GPT-2 QA model few-shot training fine-tuned model.

CONCLUSION

We developed a robust information extraction model based on a state-of-the-art large language model, a GPT-2 generative model that can extract headache frequency from EHR free-text clinical notes with high accuracy and R2 score. It overcame several challenges related to different ways clinicians document headache frequency that were not easily achieved by traditional NLP models. We also showed that GPT-2-based frameworks outperformed ClinicalBERT in terms of accuracy in extracting headache frequency from clinical notes. To facilitate research in the field, we released the GPT-2 generative model and inference code with open-source license of community use in GitHub. Additional fine-tuning of the algorithm might be required when applied to different health-care systems for various clinical use cases.

Preliminary Evidence of the Use of Generative AI in Health Care Clinical Services: Systematic Narrative Review

BACKGROUND

Generative artificial intelligence tools and applications (GenAI) are being increasingly used in health care. Physicians, specialists, and other providers have started primarily using GenAI as an aid or tool to gather knowledge, provide information, train, or generate suggestive dialogue between physicians and patients or between physicians and patients' families or friends. However, unless the use of GenAI is oriented to be helpful in clinical service encounters that can improve the accuracy of diagnosis, treatment, and patient outcomes, the expected potential will not be achieved. As adoption continues, it is essential to validate the effectiveness of the infusion of GenAI as an intelligent technology in service encounters to understand the gap in actual clinical service use of GenAI.

OBJECTIVE

This study synthesizes preliminary evidence on how GenAI assists, guides, and automates clinical service rendering and encounters in health care The review scope was limited to articles published in peer-reviewed medical journals.

METHODS

We screened and selected 0.38% (161/42,459) of articles published between January 1, 2020, and May 31, 2023, identified from PubMed. We followed the protocols outlined in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to select highly relevant studies with at least 1 element on clinical use, evaluation, and validation to provide evidence of GenAI use in clinical services. The articles were classified based on their relevance to clinical service functions or activities using the descriptive and analytical information presented in the articles.

RESULTS

Of 161 articles, 141 (87.6%) reported using GenAI to assist services through knowledge access, collation, and filtering. GenAI was used for disease detection (19/161, 11.8%), diagnosis (14/161, 8.7%), and screening processes (12/161, 7.5%) in the areas of radiology (17/161, 10.6%), cardiology (12/161, 7.5%), gastrointestinal medicine (4/161, 2.5%), and diabetes (6/161, 3.7%). The literature synthesis in this study suggests that GenAI is mainly used for diagnostic processes, improvement of diagnosis accuracy, and screening and diagnostic purposes using knowledge access. Although this solves the problem of knowledge access and may improve diagnostic accuracy, it is oriented toward higher value creation in health care.

CONCLUSIONS

GenAI informs rather than assisting or automating clinical service functions in health care. There is potential in clinical service, but it has yet to be actualized for GenAI. More clinical service-level evidence that GenAI is used to streamline some functions or provides more automated help than only information retrieval is needed. To transform health care as purported, more studies related to GenAI applications must automate and guide human-performed services and keep up with the optimism that forward-thinking health care organizations will take advantage of GenAI.

The Clinical Relevance of Artificial Intelligence in Migraine

Migraine is a burdensome neurological disorder that still lacks clear and easily accessible diagnostic biomarkers. Furthermore, a straightforward pathway is hard to find for migraineurs' management, so the search for response predictors has become urgent. Nowadays, artificial intelligence (AI) has pervaded almost every aspect of our lives, and medicine has not been missed. Its applications are nearly limitless, and the ability to use machine learning approaches has given researchers a chance to give huge amounts of data new insights. When it comes to migraine, AI may play a fundamental role, helping clinicians and patients in many ways. For example, AI-based models can increase diagnostic accuracy, especially for non-headache specialists, and may help in correctly classifying the different groups of patients. Moreover, AI models analysing brain imaging studies reveal promising results in identifying disease biomarkers. Regarding migraine management, AI applications showed value in identifying outcome measures, the best treatment choices, and therapy response prediction. In the present review, the authors introduce the various and most recent clinical applications of AI regarding migraine.

Developing an artificial intelligence-based diagnostic model of headaches from a dataset of clinic patients' records

OBJECTIVE

We developed an artificial intelligence (AI)-based headache diagnosis model using a large questionnaire database from a headache-specializing clinic.

BACKGROUND

Misdiagnosis of headache disorders is a serious issue and AI-based headache diagnosis models are scarce.

METHODS

We developed an AI-based headache diagnosis model and conducted internal validation based on a retrospective investigation of 6058 patients (4240 training dataset for model development and 1818 test dataset for internal validation) diagnosed by a headache specialist. The ground truth was the diagnosis by the headache specialist. The diagnostic performance of the AI model was evaluated.

RESULTS

The dataset included 4829/6058 (79.7%) patients with migraine, 834/6058 (13.8%) with tension-type headache, 78/6058 (1.3%) with trigeminal autonomic cephalalgias, 38/6058 (0.6%) with other primary headache disorders, and 279/6058 (4.6%) with other headaches. The mean (standard deviation) age was 34.7 (14.5) years, and 3986/6058 (65.8%) were female. The model's micro-average accuracy, sensitivity (recall), specificity, precision, and F-values for the test dataset were 93.7%, 84.2%, 84.2%, 96.1%, and 84.2%, respectively. The diagnostic performance for migraine was high, with a sensitivity of 88.8% and c-statistics of 0.89 (95% confidence interval 0.87-0.91).

CONCLUSIONS

Our AI model demonstrated high diagnostic performance for migraine. If secondary headaches can be ruled out, the model can be a powerful tool for diagnosing migraine; however, further data collection and external validation are required to strengthen the performance, ensure the generalizability in other outpatients, and demonstrate its utility in real-world settings.

Which headache disorders can be diagnosed concurrently? An analysis of ICHD3 criteria using prime encoding system

Introduction

Real-life headache presentations may fit more than one ICHD3 diagnosis. This project seeks to exhaustively list all logically consistent "co-diagnoses" according to the ICHD3 criteria. We limited our project to cases of two concurrent diagnoses.

Methods

We included the criteria for "Migraine" (1.1, 1.2, 1.3), "Tension-type headache" (2.1, 2.2, 2.3, 2.4), "Trigeminal autonomic cephalalgias" (3.1, 3.2, 3.3, 3.4, 3.5), and "Other primary headache disorders." We also excluded "probable" diagnosis criteria. Each characteristic in the above criteria is assigned a unique prime number. We then encoded each ICHD3 criteria into integers through multiplication in a list format; we called these criteria representations. "Codiagnoses representations" were generated by multiplying all possible pairings of criteria representations. We then manually encoded a list of logically inconsistent characteristics through multiplication. All co-diagnoses representations divisible by any inconsistency representations were filtered out, generating a list of co-diagnoses representations that were logically consistent. This list was then translated back into ICHD3 diagnoses.

Results

We used a total of 103 prime numbers to encode 578 ICHD3 criteria. Once illogical characteristics were excluded, we obtained 145 dual diagnoses. Of the dual diagnoses, two contained intersecting characteristics due to subset relationships, 14 contained intersecting characteristics without subset relationships, and 129 contained dual diagnoses as a result of non-intersecting characteristics.

Conclusion

Analysis of dual diagnosis in headaches offers insight into "loopholes" in the ICHD3 as well as a potential explanation for the source of a number of controversies regarding headache disorders. The existence of dual diagnoses and their identification may carry implications for future developments and testing of machine-learning diagnostic algorithms for headaches.

The utility of ChatGPT in the assessment of literature on the prevention of migraine: an observational, qualitative study

Background

It is not known how large language models, such as ChatGPT, can be applied toward the assessment of the efficacy of medications, including in the prevention of migraine, and how it might support those claims with existing medical evidence.

Methods

We queried ChatGPT-3.5 on the efficacy of 47 medications for the prevention of migraine and then asked it to give citations in support of its assessment. ChatGPT's evaluations were then compared to their FDA approval status for this indication as well as the American Academy of Neurology 2012 evidence-based guidelines for the prevention of migraine. The citations ChatGPT generated for these evaluations were then assessed to see if they were real papers and if they were relevant to the query.

Results

ChatGPT affirmed that the 14 medications that have either received FDA approval for prevention of migraine or AAN Grade A/B evidence were effective for migraine. Its assessments of the other 33 medications were unreliable including suggesting possible efficacy for four medications that have never been used for the prevention of migraine. Critically, only 33/115 (29%) of the papers ChatGPT cited were real, while 76/115 (66%) were "hallucinated" not real papers and 6/115 (5%) shared the names of real papers but had not real citations.

Conclusion

While ChatGPT produced tailored answers on the efficacy of the queried medications, the results were unreliable and inaccurate because of the overwhelming volume of "hallucinated" articles it generated and cited.

Developing an Artificial Intelligence-Based Pediatric and Adolescent Migraine Diagnostic Model

Introduction Misdiagnosis of pediatric and adolescent migraine is a significant problem. The first artificial intelligence (AI)-based pediatric migraine diagnosis model was made utilizing a database of questionnaires obtained from a previous epidemiological study, the Itoigawa Benizuwaigani Study. Methods The AI-based headache diagnosis model was created based on the internal validation based on a retrospective investigation of 909 patients (636 training dataset for model development and 273 test dataset for internal validation) aged six to 17 years diagnosed based on the International Classification of Headache Disorders 3rd edition. The diagnostic performance of the AI model was evaluated. Results The dataset included 234/909 (25.7%) pediatric or adolescent patients with migraine. The mean age was 11.3 (standard deviation 3.17) years. The model's accuracy, sensitivity (recall), specificity, precision, and F-values for the test dataset were 94.5%, 88.7%, 96.5%, 90.0%, and 89.4%, respectively. Conclusions The AI model exhibited high diagnostic performance for pediatric and adolescent migraine. It holds great potential as a powerful tool for diagnosing these conditions, especially when secondary headaches are ruled out. Nonetheless, further data collection and external validation are necessary to enhance the model's performance and ensure its applicability in real-world settings.