ChatGPT in healthcare: A taxonomy and systematic review

PhenoDP: leveraging deep learning for phenotype-based case reporting, disease ranking, and symptom recommendation

BACKGROUND

Current phenotype-based diagnostic tools often struggle with accurate disease prioritization due to incomplete phenotypic data and the complexity of rare disease presentations. Additionally, they lack the ability to generate patient-centered clinical insights or recommend further symptoms for differential diagnosis.

METHODS

We developed PhenoDP, a deep learning-based toolkit with three modules: Summarizer, Ranker, and Recommender. The Summarizer fine-tuned a distilled large language model to create clinical summaries from a patient's Human Phenotype Ontology (HPO) terms. The Ranker prioritizes diseases by combining information content-based, phi-based, and semantic-based similarity measures. The Recommender employs contrastive learning to recommend additional HPO terms for enhanced diagnostic accuracy.

RESULTS

PhenoDP's Summarizer produces more clinically coherent and patient-centered summaries than the general-purpose language model FlanT5. The Ranker achieves state-of-the-art diagnostic performance, consistently outperforming existing phenotype-based methods across both simulated and real-world datasets. The Recommender also outperformed GPT-4o and PhenoTips in improving diagnostic accuracy when its suggested terms were incorporated into different ranking pipelines.

CONCLUSIONS

PhenoDP enhances Mendelian disease diagnosis through deep learning, offering precise summarization, ranking, and symptom recommendation. Its superior performance and open-source design make it a valuable clinical tool, with potential to accelerate diagnosis and improve patient outcomes. PhenoDP is freely available at https://github.com/TianLab-Bioinfo/PhenoDP .

Is there any room for ChatGPT AI bot in speech-language pathology?

PURPOSE

This study investigates the potential of the ChatGPT-4.0 artificial intelligence bot to assist speech-language pathologists (SLPs) by assessing its accuracy, comprehensiveness, and relevance in various tasks related to speech, language, and swallowing disorders.

METHOD

In this cross-sectional descriptive study, 15 practicing SLPs evaluated ChatGPT-4.0's responses to task-specific queries across six core areas: report writing, assessment material generation, clinical decision support, therapy stimulus generation, therapy planning, and client/family training material generation. English prompts were created in seven areas: speech sound disorders, motor speech disorders, aphasia, stuttering, childhood language disorders, voice disorders, and swallowing disorders. These prompts were entered into ChatGPT-4.0, and its responses were evaluated. Using a three-point Likert-type scale, participants rated each response for accuracy, relevance, and comprehensiveness based on clinical expectations and their professional judgment.

RESULTS

The study revealed that ChatGPT-4.0 performed with predominantly high accuracy, comprehensiveness, and relevance in tasks related to speech and language disorders. High accuracy, comprehensiveness, and relevance levels were observed in report writing, clinical decision support, and creating education material. However, tasks such as creating therapy stimuli and therapy planning showed more variation with medium and high accuracy levels.

CONCLUSIONS

ChatGPT-4.0 shows promise in assisting SLPs with various professional tasks, particularly report writing, clinical decision support, and education material creation. However, further research is needed to address its limitations in therapy stimulus generation and therapy planning to improve its usability in clinical practice. Integrating AI technologies such as ChatGPT could improve the efficiency and effectiveness of therapeutic processes in speech-language pathology.

A Comparison of Prostate Cancer Screening Information Quality on Standard and Advanced Versions of ChatGPT, Google Gemini, and Microsoft Copilot: A Cross-Sectional Study

PurposeArtificially Intelligent (AI) chatbots have the potential to produce information to support shared prostate cancer (PrCA) decision-making. Therefore, our purpose was to evaluate and compare the accuracy, completeness, readability, and credibility of responses from standard and advanced versions of popular chatbots: ChatGPT-3.5, ChatGPT-4.0, Microsoft Copilot, Microsoft Copilot Pro, Google Gemini, and Google Gemini Advanced. We also investigated whether prompting chatbots for low-literacy PrCA information would improve the readability of responses. Lastly, we determined if the responses were appropriate for African-American men, who have the worst PrCA outcomes.ApproachThe study used a cross-sectional approach to examine the quality of responses solicited from chatbots.ParticipantsThe study did not include human subjects.MethodEleven frequently asked PrCA questions, based on resources produced by the Centers for Disease Control and Prevention (CDC) and the American Cancer Society (ACS), were posed to each chatbot twice (once for low literacy populations). A coding/rating form containing questions with key points/answers from the ACS or CDC to facilitate the rating process. Accuracy and completeness were rated dichotomously (i.e., yes/no). Credibility was determined by whether a trustworthy medical or health-related organization was cited. Readability was determined using a Flesch-Kincaid readability score calculator that enabled chatbot responses to be entered individually. Average accuracy, completeness, credibility, and readability percentages or scores were calculated using Excel.ResultsAll chatbots were accurate, but the completeness, readability, and credibility of responses varied. Soliciting low-literacy responses significantly improved readability, but sometimes at the detriment of completeness. All chatbots recognized the higher PrCA risk in African-American men and tailored screening recommendations. Microsoft Copilot Pro had the best overall performance on standard screening questions. Microsoft Copilot outperformed other chatbots on responses for low literacy populations.ConclusionsAI chatbots are useful tools for learning about PrCA screening but should be combined with healthcare provider advice.

Application of AI-assisted multi-advisor system combined with BOPPPS teaching model in clinical pharmacy education

BACKGROUND

The development of clinical pharmacy in China has been relatively slow, and standardized, effective training for clinical pharmacists remains a major challenge. At present, traditional teaching methods are not conducive to cultivating clinical practice abilities or critical thinking, leading to a lack of enthusiasm among students. To address this issue, a hybrid teaching model combining an AI-assisted multi-advisor system with the BOPPPS (bridge, objective, pre-assessment, participatory learning, post-assessment, summary) teaching method was applied in clinical pharmacy education.

METHODS

In this study, a teaching model was developed that consists of five components: advisor system, teaching strategy, teaching method, teaching mode, and assessment methods. The model was implemented in the Pharmacy Department of Xiangya Hospital at Central South University in Changsha, China, from spring 2023 to spring 2024. After the spring 2024 training session, anonymous questionnaires were completed by students. The effects of the teaching reform were evaluated. The questionnaire focused on clinical practice effectiveness, course design rationality, improvement in professional ability, interest in learning, and perceptions of ChatGPT as a teaching tool.

RESULTS

A total of 81 students participated in the study, with 69 completing the questionnaire survey (response rate: 85.2%, n = 69). Overall, students gave positive comments regarding the instructional improvements. A higher proportion of interns and advanced students considered the courses difficult (32.4%, n = 12 and 31.6%, n = 6, respectively). Identified shortcomings include the frequent periodic assessments and increased pressure. Regarding the use of ChatGPT, 49 students (71.0%, n = 49) commented that it played a multidimensional role in teaching. The most valued asset was its ability to provide massive data information (43.5%, n = 30), while the lack of interactivity was a prominent issue (56.5%, n = 39).

CONCLUSION

This innovative teaching model integrating an AI-assisted BOPPPS framework with a supervised ChatGPT implementation may facilitate the comprehensive self-evaluation of abilities. This research offers a replicable model for AI-driven educational transformation in clinical pharmacy training.

DeepSeekTM and lacrimal drainage disorders: hype or is it performing better than ChatGPTTM?

PURPOSE

This study aimed to report the performance of the large language model DeepSeekTM (DeepSeek TM, Hangzhou, China) and perform a head-to-head comparison with ChatGPTTM (OpenAI, San Francisco, USA) in the context of lacrimal drainage disorders.

METHODS

Questions and statements were used to construct prompts to include common and uncommon aspects of lacrimal drainage disorders. Prompts avoided covering new knowledge beyond February 2024. Prompts were presented at least twice to the latest versions of DeepSeekTM and ChatGPTTM [Accessed February 15-18, 2025]. A set of assessed prompts for ChatGPTTM from 2023 (ChatGPT-2023) was utilized in this study. The responses of DeepSeekTM and ChatGPTTM were analyzed for evidence-based content, updated knowledge, specific responses, speed, and factual inaccuracies. The responses of the current ChatGPTTM were also compared with those of 2023 to assess the improvement of the artificial intelligence chatbot. Three lacrimal surgeons graded the responses into three categories: correct, partially correct, and factually incorrect. They also compared the overall quality of the response between DeepSeekTM and ChatGPTTM based on the overall content, organization, and clarity of the answers.

RESULTS

25 prompts were presented to the latest versions [February 2025] of DeepSeekTM and ChatGPTTM. There was no significant difference in the speed of response. The agreement among the three observers was high (96%) in grading the responses. In terms of the accuracy of the responses, both AI models were similar. DeepSeek's responses were graded as correct in 60% (15/25), partially correct in 36% (9/25), and factually incorrect in 4% (1/25). ChatGPT-2025 responses were graded as correct in 56% (14/25), partially correct in 40% (10/25), and factually incorrect in 4% (1/25). Compared to 2023, ChatGPT-2025 gave responses which were more specific, more accurate, less generic with lesser recycling of phrases. When confronted with inaccuracies, both admitted and corrected the mistakes in subsequent responses. Both the AI models demonstrated the capability of challenging incorrect prompts and premises.

CONCLUSION

DeepSeekTM was not superior but comparable to ChatGPTTM in the context of lacrimal drainage disorders. Each had unique advantages and could complement each other. They need to be specifically trained and re-trained for individual medical subspecialties.

Impact of large language model (ChatGPT) in healthcare: an umbrella review and evidence synthesis

BACKGROUND

The emergence of Artificial Intelligence (AI), particularly Chat Generative Pre-Trained Transformer (ChatGPT), a Large Language Model (LLM), in healthcare promises to reshape patient care, clinical decision-making, and medical education. This review aims to synthesise research findings to consolidate the implications of ChatGPT integration in healthcare and identify research gaps.

MAIN BODY

The umbrella review was conducted following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. The Cochrane Library, PubMed, Scopus, Web of Science, and Google Scholar were searched from inception until February 2024. Due to the heterogeneity of the included studies, no quantitative analysis was performed. Instead, information was extracted, summarised, synthesised, and presented in a narrative form. Two reviewers undertook title, abstract, and full text screening independently. The methodological quality and overall rating of the included reviews were assessed using the A Measurement Tool to Assess systematic Reviews (AMSTAR-2) checklist. The review examined 17 studies, comprising 15 systematic reviews and 2 meta-analyses, on ChatGPT in healthcare, revealing diverse focuses. The AMSTAR-2 assessment identified 5 moderate and 12 low-quality reviews, with deficiencies like study design justification and funding source reporting. The most reported theme that emerged was ChatGPT's use in disease diagnosis or clinical decision-making. While 82.4% of studies focused on its general usage, 17.6% explored unique topics like its role in medical examinations and conducting systematic reviews. Among these, 52.9% targeted general healthcare, with 41.2% focusing on specific domains like radiology, neurosurgery, gastroenterology, public health dentistry, and ophthalmology. ChatGPT's use for manuscript review or writing was mentioned in 17.6% of reviews. Promising applications include enhancing patient care and clinical decision-making, though ethical, legal, and accuracy concerns require cautious integration.

CONCLUSION

We summarise the identified areas in reviews regarding ChatGPT's transformative impact in healthcare, highlighting patient care, decision-making, and medical education. Emphasising the importance of ethical regulations and the involvement of policymakers, we urge further investigation to ensure the reliability of ChatGPT and to promote trust in healthcare and research.

Fine-tuning large language models for improved health communication in low-resource languages

BACKGROUND

The reported study illustrates a methodology for compiling training datasets to fine-tune Large Language Models (LLMs) for healthcare information in Vietnamese, a low-resource language. The objective is to bridge the gap in medical information accessibility and enhance healthcare communication in developing countries by adapting LLMs to specific linguistic nuances and domain needs.

METHOD

The methodology involves selecting a base model, compiling a domain-specific dataset, and fine-tuning the model with this dataset. Three open-source models were selected. The dataset, comprising approximately 337,000 prompt-response pairs in Vietnamese, was compiled using existing datasets, data crawled from Vietnamese medical online forums, and distilled from Vietnamese medical textbooks. The three models were fine-tuned using the Low-Rank adaptation (LoRA) and Quantized Low-Rank adaptation (QLoRA) techniques. Models' performances were evaluated using BertScore score, Rouge-L score, and the "LLM-as-a-Judge" method.

RESULTS

The fine-tuned models showed enhancements in performance over their base versions across evaluation metrics in BertScore score, Rouge-L score and "LLM-as-a-Judge" method, confirming the effectiveness of the fine-tuning process. This study details the process of fine-tuning open-source LLMs for health information inquiries in Vietnamese, demonstrating its potential to improve healthcare communication in low-resource languages. Deploying the fine-tuned LLM on-premise enhances data privacy and security. However, the significant computing power and costs required pose challenges, especially for organizations in developing countries.

CONCLUSION

This case study highlights the unique challenges faced by developing countries using low-resource languages. Initiatives are needed to emphasize efforts to bridge healthcare gaps in underserved areas and contribute to global health equity.

Evaluating and mitigating bias in AI-based medical text generation

Artificial intelligence (AI) systems, particularly those based on deep learning models, have increasingly achieved expert-level performance in medical applications. However, there is growing concern that such AI systems may reflect and amplify human bias, reducing the quality of their performance in historically underserved populations. The fairness issue has attracted considerable research interest in the medical imaging classification field, yet it remains understudied in the text-generation domain. In this study, we investigate the fairness problem in text generation within the medical field and observe substantial performance discrepancies across different races, sexes and age groups, including intersectional groups, various model scales and different evaluation metrics. To mitigate this fairness issue, we propose an algorithm that selectively optimizes those underserved groups to reduce bias. Our evaluations across multiple backbones, datasets and modalities demonstrate that our proposed algorithm enhances fairness in text generation without compromising overall performance.

Natural Language Processing to Extract Head and Neck Cancer Data From Unstructured Electronic Health Records

AIMS

Patient data is frequently stored as unstructured data within Electronic Health Records (EHRs), requiring manual curation. AI tools using Natural Language Processing (NLP) may rapidly curate accurate real-world unstructured EHRs to enrich datasets. We evaluated this approach for Head and Neck Cancer (HNC) patient data extraction using an open-source general-purpose healthcare NLP tool (CogStack).

MATERIALS AND METHODS

CogStack was applied to extract relevant SNOMED-CT concepts from HNC patients' documents, generating outputs denoting the identifications of each concept for each patient. Outputs were compared to manually curated ground truth HNC datasets to calculate pre-training performance. Supervised model training was then performed using SNOMED-CT concept annotation on clinical documents, and the updated model was re-evaluated. A second training cycle was performed before the final evaluation. A thresholding approach (multiple detections needed to qualify a concept as 'present') was used to increase precision. The final model was evaluated on an unseen test cohort. F1 score (harmonic mean of precision and recall) was used for evaluation.

RESULTS

Pre-training, the F1 score was incalculable for 19.5% of concepts due to insufficient recall. Following one training cycle, F1 score became calculable for all concepts (median 0.692). After further training, the final model demonstrated improvement in the median F1 score (0.708). Test cohort median F1 score was 0.750. Thresholding analysis developed a concept-specific best threshold approach, resulting in a median F1 score of 0.778 in the test cohort, where 50 out of 109 SNOMED-CT concepts met pre-set criteria to be considered adequately fine-tuned.

CONCLUSIONS

NLP can mine unstructured cancer data following limited training. Certain concepts such as histopathology terms remained poorly retrieved. Model performance is maintained when applied to a test cohort, demonstrating good generalisability. Concept-specific thresholding strategy improved performance. Fine-tuning annotations were incorporated into the NLP parent model for future performance. CogStack has been applied to extract data for 50 concepts with validated performance for our entire retrospective HNC cohort.

Evaluating the Performance of Large Language Models (LLMs) in Answering and Analysing the Chinese Dental Licensing Examination

BACKGROUND

This study aimed to simulate diverse scenarios of students employing LLMs for CDLE examination preparation, providing a detailed evaluation of their performance in medical education.

METHODS

A stratified random sampling strategy was implemented to select and subsequently revise 200 questions from the CDLE. Seven LLMs, recognised for their exceptional performance in the Chinese domain, were selected as test subjects. Three distinct testing scenarios were constructed: answering questions, explaining questions and adversarial testing. The evaluation metrics included accuracy, agreement rate and teaching effectiveness score. Wald χ2 tests and Kruskal-Wallis tests were employed to determine whether the differences among the LLMs across various scenarios and before and after adversarial testing were statistically significant.

RESULTS

The majority of the tested LLMs met the passing threshold on the CDLE benchmark, with Doubao-pro 32k and Qwen2-72b (81%) achieving the highest accuracy rates. Doubao-pro 32k demonstrated the highest 98% agreement rate with the reference answers when providing explanations. Although statistically significant differences existed among various LLMs in their teaching effectiveness scores based on the Likert scale, all these models demonstrated a commendable ability to deliver comprehensible and effective instructional content. In adversarial testing, GPT-4 exhibited the smallest decline in accuracy (2%, p = 0.623), while ChatGLM-4 demonstrated the least reduction in agreement rate (14.6%, p = 0.001).

CONCLUSIONS

LLMs trained on Chinese corpora, such as Doubao-pro 32k, demonstrated superior performance compared to GPT-4 in answering and explaining questions, with no statistically significant difference. However, during adversarial testing, all models exhibited diminished performance, with GPT-4 displaying comparatively greater robustness. Future research should further investigate the interpretability of LLM outputs and develop strategies to mitigate hallucinations generated in medical education.

Enhancing patient-centered information on implant dentistry through prompt engineering: a comparison of four large language models

Background

Patients frequently seek dental information online, and generative pre-trained transformers (GPTs) may be a valuable resource. However, the quality of responses based on varying prompt designs has not been evaluated. As dental implant treatment is widely performed, this study aimed to investigate the influence of prompt design on GPT performance in answering commonly asked questions related to dental implants.

Materials and methods

Thirty commonly asked questions about implant dentistry - covering patient selection, associated risks, peri-implant disease symptoms, treatment for missing teeth, prevention, and prognosis - were posed to four different GPT models with different prompt designs. Responses were recorded and independently appraised by two periodontists across six quality domains.

Results

All models performed well, with responses classified as good quality. The contextualized model performed worse on treatment-related questions (21.5 ± 3.4, p < 0.05), but outperformed the input-output, zero-shot chain of thought, and instruction-tuned models in citing appropriate sources in its responses (4.1 ± 1.0, p < 0.001). However, responses had less clarity and relevance compared to the other models.

Conclusion

GPTs can provide accurate, complete, and useful information for questions related to dental implants. While prompt designs can enhance response quality, further refinement is necessary to optimize its performance.

Evaluation of Rhinoplasty Information from ChatGPT, Gemini, and Claude for Readability and Accuracy

OBJECTIVE

Assessment of the readability, accuracy, quality, and completeness of ChatGPT (Open AI, San Francisco, CA), Gemini (Google, Mountain View, CA), and Claude (Anthropic, San Francisco, CA) responses to common questions about rhinoplasty.

METHODS

Ten questions commonly encountered in the senior author's (SPM) rhinoplasty practice were presented to ChatGPT-4, Gemini and Claude. Seven Facial Plastic and Reconstructive Surgeons with experience in rhinoplasty were asked to evaluate these responses for accuracy, quality, completeness, relevance, and use of medical jargon on a Likert scale. The responses were also evaluated using several readability indices.

RESULTS

ChatGPT achieved significantly higher evaluator scores for accuracy, and overall quality but scored significantly lower on completeness compared to Gemini and Claude. All three chatbot responses to the ten questions were rated as neutral to incomplete. All three chatbots were found to use medical jargon and scored at a college reading level for readability scores.

CONCLUSIONS

Rhinoplasty surgeons should be aware that the medical information found on chatbot platforms is incomplete and still needs to be scrutinized for accuracy. However, the technology does have potential for use in healthcare education by training it on evidence-based recommendations and improving readability.

LEVEL OF EVIDENCE V

This journal requires that authors assign a level of evidence to each article. For a full description of these Evidence-Based Medicine ratings, please refer to the Table of Contents or the online Instructions to Authors www.springer.com/00266 .

Challenges and applications in generative AI for clinical tabular data in physiology

Recent advancements in generative approaches in AI have opened up the prospect of synthetic tabular clinical data generation. From filling in missing values in real-world data, these approaches have now advanced to creating complex multi-tables. This review explores the development of techniques capable of synthesizing patient data and modeling multiple tables. We highlight the challenges and opportunities of these methods for analyzing patient data in physiology. Additionally, it discusses the challenges and potential of these approaches in improving clinical research, personalized medicine, and healthcare policy. The integration of these generative models into physiological settings may represent both a theoretical advancement and a practical tool that has the potential to improve mechanistic understanding and patient care. By providing a reliable source of synthetic data, these models can also help mitigate privacy concerns and facilitate large-scale data sharing.

Nurses' assessment of artificial ıntelligence chatbots for health literacy education

BACKGROUND

Artificial intelligence (AI)-powered chatbots are emerging as a new tool in healthcare, offering the potential to provide patients with information and support. Despite their growing presence, there are concerns regarding the medical reliability of the information they provide and the potential risks to patient safety.

MATERIAL AND METHODS

The aim of this study is to assess the medical reliability of responses to health-related questions provided by an AI-powered chatbot and to evaluate the risks to patient safety. The study is designed using a mixed-methods phenomenology approach. The participants are 44 nurses working at a private hospital in Cyprus. Data collection was conducted via survey forms and focus group discussions. Quantitative data were analyzed using descriptive statistics, while qualitative data were examined using content analysis.

RESULTS

The results indicate that according to the nurses' evaluations, the medical reliability of the AI chatbot's responses is generally high. However, instances of incorrect or incomplete information were also noted. Specifically, the quantitative analysis showed that a majority of the nurses found the chatbot's responses to be accurate and useful. The qualitative analysis revealed concerns about the potential for the chatbot to misdirect patients or contribute to diagnostic errors. These risks highlight the importance of monitoring and improving the AI systems to minimize errors and enhance reliability.

CONCLUSION

AI chatbots can provide valuable information and support to patients, improving accessibility and engagement in healthcare. However, concerns about medical reliability and patient safety remain. Continuous evaluation and improvement of these systems are necessary, alongside efforts to enhance patients' health literacy to help them accurately assess information from AI chatbots.

ChatGPT-4o-Generated Exercise Plans for Patients with Type 2 Diabetes Mellitus-Assessment of Their Safety and Other Quality Criteria by Coaching Experts

In this discussion paper based on preliminary data, the safety and other quality criteria of ChatGPT-4o-generated exercise plans for patients with type 2 diabetes mellitus (T2DM) are evaluated. The study team created three fictional patient profiles varying in sex, age, body mass index, secondary diseases/complications, medication, self-rated physical fitness, weekly exercise routine and personal exercise preferences. Three distinct prompts were used to generate three exercise plans for each fictional patient. While Prompt 1 was very simple, Prompt 2 and Prompt 3 included more detailed requests. Prompt 3 was optimized by ChatGPT itself. Three coaching experts reviewed the exercise plans for safety and other quality criteria and discussed their evaluations. Some of the exercise plans showed serious safety issues, especially for patients with secondary diseases/complications. While most exercise plans incorporated key training principles, they showed some deficits, e.g., insufficient feasibility. The use of more detailed prompts (Prompt 2 and Prompt 3) tended to result in more elaborate exercise plans with better ratings. ChatGPT-4o-generated exercise plans may have safety issues for patients with T2DM, indicating the need to consult a professional coach for feedback before starting a training program.

Exploring Biases of Large Language Models in the Field of Mental Health: Comparative Questionnaire Study of the Effect of Gender and Sexual Orientation in Anorexia Nervosa and Bulimia Nervosa Case Vignettes

Background

Large language models (LLMs) are increasingly used in mental health, showing promise in assessing disorders. However, concerns exist regarding their accuracy, reliability, and fairness. Societal biases and underrepresentation of certain populations may impact LLMs. Because LLMs are already used for clinical practice, including decision support, it is important to investigate potential biases to ensure a responsible use of LLMs. Anorexia nervosa (AN) and bulimia nervosa (BN) show a lifetime prevalence of 1%-2%, affecting more women than men. Among men, homosexual men face a higher risk of eating disorders (EDs) than heterosexual men. However, men are underrepresented in ED research, and studies on gender, sexual orientation, and their impact on AN and BN prevalence, symptoms, and treatment outcomes remain limited.

objectives

We aimed to estimate the presence and size of bias related to gender and sexual orientation produced by a common LLM as well as a smaller LLM specifically trained for mental health analyses, exemplified in the context of ED symptomatology and health-related quality of life (HRQoL) of patients with AN or BN.

Methods

We extracted 30 case vignettes (22 AN and 8 BN) from scientific papers. We adapted each vignette to create 4 versions, describing a female versus male patient living with their female versus male partner (2 × 2 design), yielding 120 vignettes. We then fed each vignette into ChatGPT-4 and to "MentaLLaMA" based on the Large Language Model Meta AI (LLaMA) architecture thrice with the instruction to evaluate them by providing responses to 2 psychometric instruments, the RAND-36 questionnaire assessing HRQoL and the eating disorder examination questionnaire. With the resulting LLM-generated scores, we calculated multilevel models with a random intercept for gender and sexual orientation (accounting for within-vignette variance), nested in vignettes (accounting for between-vignette variance).

Results

In ChatGPT-4, the multilevel model with 360 observations indicated a significant association with gender for the RAND-36 mental composite summary (conditional means: 12.8 for male and 15.1 for female cases; 95% CI of the effect -6.15 to -0.35; P=.04) but neither with sexual orientation (P=.71) nor with an interaction effect (P=.37). We found no indications for main effects of gender (conditional means: 5.65 for male and 5.61 for female cases; 95% CI -0.10 to 0.14; P=.88), sexual orientation (conditional means: 5.63 for heterosexual and 5.62 for homosexual cases; 95% CI -0.14 to 0.09; P=.67), or for an interaction effect (P=.61, 95% CI -0.11 to 0.19) for the eating disorder examination questionnaire overall score (conditional means 5.59-5.65 95% CIs 5.45 to 5.7). MentaLLaMA did not yield reliable results.

Conclusions

LLM-generated mental HRQoL estimates for AN and BN case vignettes may be biased by gender, with male cases scoring lower despite no real-world evidence supporting this pattern. This highlights the risk of bias in generative artificial intelligence in the field of mental health. Understanding and mitigating biases related to gender and other factors, such as ethnicity, and socioeconomic status are crucial for responsible use in diagnostics and treatment recommendations.

Prospects and perils of ChatGPT in diabetes

ChatGPT, a popular large language model developed by OpenAI, has the potential to transform the management of diabetes mellitus. It is a conversational artificial intelligence model trained on extensive datasets, although not specifically health-related. The development and core components of ChatGPT include neural networks and machine learning. Since the current model is not yet developed on diabetes-related datasets, it has limitations such as the risk of inaccuracies and the need for human supervision. Nevertheless, it has the potential to aid in patient engagement, medical education, and clinical decision support. In diabetes management, it can contribute to patient education, personalized dietary guidelines, and providing emotional support. Specifically, it is being tested in clinical scenarios such as assessment of obesity, screening for diabetic retinopathy, and provision of guidelines for the management of diabetic ketoacidosis. Ethical and legal considerations are essential before ChatGPT can be integrated into healthcare. Potential concerns relate to data privacy, accuracy of responses, and maintenance of the patient-doctor relationship. Ultimately, while ChatGPT and large language models hold immense potential to revolutionize diabetes care, one needs to weigh their limitations, ethical implications, and the need for human supervision. The integration promises a future of proactive, personalized, and patient-centric care in diabetes management.

Studying the Potential Effects of Artificial Intelligence on Physician Autonomy: Scoping Review

BACKGROUND

Physician autonomy has been found to play a role in physician acceptance and adoption of artificial intelligence (AI) in medicine. However, there is still no consensus in the literature on how to define and assess physician autonomy. Furthermore, there is a lack of research focusing specifically on the potential effects of AI on physician autonomy.

OBJECTIVE

This scoping review addresses the following research questions: (1) How do qualitative studies conceptualize and assess physician autonomy? (2) Which aspects of physician autonomy are addressed by these studies? (3) What are the potential benefits and harms of AI for physician autonomy identified by these studies?

METHODS

We performed a scoping review of qualitative studies on AI and physician autonomy published before November 6, 2023, by searching MEDLINE and Web of Science. To answer research question 1, we determined whether the included studies explicitly include physician autonomy as a research focus and whether their interview, survey, and focus group questions explicitly name or implicitly include aspects of physician autonomy. To answer research question 2, we extracted the qualitative results of the studies, categorizing them into the 7 components of physician autonomy introduced by Schulz and Harrison. We then inductively formed subcomponents based on the results of the included studies in each component. To answer research question 3, we summarized the potentially harmful and beneficial effects of AI on physician autonomy in each of the inductively formed subcomponents.

RESULTS

The search yielded 369 studies after duplicates were removed. Of these, 27 studies remained after titles and abstracts were screened. After full texts were screened, we included a total of 7 qualitative studies. Most studies did not explicitly name physician autonomy as a research focus or explicitly address physician autonomy in their interview, survey, and focus group questions. No studies addressed a complete set of components of physician autonomy; while 3 components were addressed by all included studies, 2 components were addressed by none. We identified a total of 11 subcomponents for the 5 components of physician autonomy that were addressed by at least 1 study. For most of these subcomponents, studies reported both potential harms and potential benefits of AI for physician autonomy.

CONCLUSIONS

Little research to date has explicitly addressed the potential effects of AI on physician autonomy and existing results on these potential effects are mixed. Further qualitative and quantitative research is needed that focuses explicitly on physician autonomy and addresses all relevant components of physician autonomy.

Medical Misinformation in AI-Assisted Self-Diagnosis: Development of a Method (EvalPrompt) for Analyzing Large Language Models

Background

Rapid integration of large language models (LLMs) in health care is sparking global discussion about their potential to revolutionize health care quality and accessibility. At a time when improving health care quality and access remains a critical concern for countries worldwide, the ability of these models to pass medical examinations is often cited as a reason to use them for medical training and diagnosis. However, the impact of their inevitable use as a self-diagnostic tool and their role in spreading health care misinformation has not been evaluated.

Objective

This study aims to assess the effectiveness of LLMs, particularly ChatGPT, from the perspective of an individual self-diagnosing to better understand the clarity, correctness, and robustness of the models.

Methods

We propose the comprehensive testing methodology evaluation of LLM prompts (EvalPrompt). This evaluation methodology uses multiple-choice medical licensing examination questions to evaluate LLM responses. Experiment 1 prompts ChatGPT with open-ended questions to mimic real-world self-diagnosis use cases, and experiment 2 performs sentence dropout on the correct responses from experiment 1 to mimic self-diagnosis with missing information. Humans then assess the responses returned by ChatGPT for both experiments to evaluate the clarity, correctness, and robustness of ChatGPT.

Results

In experiment 1, we found that ChatGPT-4.0 was deemed correct for 31% (29/94) of the questions by both nonexperts and experts, with only 34% (32/94) agreement between the 2 groups. Similarly, in experiment 2, which assessed robustness, 61% (92/152) of the responses continued to be categorized as correct by all assessors. As a result, in comparison to a passing threshold of 60%, ChatGPT-4.0 is considered incorrect and unclear, though robust. This indicates that sole reliance on ChatGPT-4.0 for self-diagnosis could increase the risk of individuals being misinformed.

Conclusions

The results highlight the modest capabilities of LLMs, as their responses are often unclear and inaccurate. Any medical advice provided by LLMs should be cautiously approached due to the significant risk of misinformation. However, evidence suggests that LLMs are steadily improving and could potentially play a role in health care systems in the future. To address the issue of medical misinformation, there is a pressing need for the development of a comprehensive self-diagnosis dataset. This dataset could enhance the reliability of LLMs in medical applications by featuring more realistic prompt styles with minimal information across a broader range of medical fields.

Precision in Parsing: Evaluation of an Open-Source Named Entity Recognizer (NER) in Veterinary Oncology

Integrating Artificial Intelligence (AI) through Natural Language Processing (NLP) can improve veterinary medical oncology clinical record analytics. Named Entity Recognition (NER), a critical component of NLP, can facilitate efficient data extraction and automated labelling for research and clinical decision-making. This study assesses the efficacy of the Bio-Epidemiology-NER (BioEN), an open-source NER developed using human epidemiological and medical data, on veterinary medical oncology records. The NER's performance was compared with manual annotations by a veterinary medical oncologist and a veterinary intern. Evaluation metrics included Jaccard similarity, intra-rater reliability, ROUGE scores, and standard NER performance metrics (precision, recall, F1-score). Results indicate poor direct translatability to veterinary medical oncology record text and room for improvement in the NER's performance, with precision, recall, and F1-score suggesting a marginally better alignment with the oncologist than the intern. While challenges remain, these insights contribute to the ongoing development of AI tools tailored for veterinary healthcare and highlight the need for veterinary-specific models.

ChatGPT's role in alleviating anxiety in total knee arthroplasty consent process: a randomized controlled trial pilot study

BACKGROUND

Recent advancements in artificial intelligence (AI) like ChatGPT have expanded possibilities for patient education, yet its impact on perioperative anxiety in total knee arthroplasty (TKA) patients remains unexplored.

METHODS

In this single-blind, randomized controlled pilot study from April to July 2023, 60 patients were randomly allocated using sealed envelopes to either ChatGPT-assisted or traditional surgeon-led informed consent groups. In the ChatGPT group, physicians used ChatGPT 4.0 to provide standardized, comprehensive responses to patient queries during the consent process, while maintaining their role in interpreting and contextualizing the information. Outcomes were measured using Hospital Anxiety and Depression Scales (HADS), Perioperative Apprehension Scale-7 (PAS-7), Visual Analogue Scales for Anxiety and Pain (VAS-A, VAS-P), Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC), and satisfaction questionnaires.

RESULTS

Of 55 patients completing the study, the ChatGPT group showed significantly lower anxiety scores after informed consent (HADS-A: 10.48 ± 3.84 vs 12.75 ± 4.12, P = .04, Power = .67; PAS-7: 12.44 ± 3.70 vs 14.64 ± 2.11, P = .01, Power = .85; VAS-A: 5.40 ± 1.89 vs 6.71 ± 2.27, P = .02, Power = .75) and on the fifth postoperative day (HADS-A: 8.33 ± 3.20 vs 10.71 ± 3.83, P = .01, Power = .79; VAS-A: 3.41 ± 1.58 vs 4.64 ± 1.70, P = .008, Power = .85). The ChatGPT group also reported higher satisfaction with preoperative education (4.22 ± 0.51 vs 3.43 ± 0.84, P <.001, Power = .99) and overall hospitalization experience (4.11 ± 0.65 vs 3.46 ± 0.69, P = .001, Power = .97). No significant differences were found in depression scores, knee function, or pain levels.

CONCLUSIONS

ChatGPT-assisted informed consent effectively reduced perioperative anxiety and improved patient satisfaction in TKA patients. While these preliminary findings are promising, larger studies are needed to validate these results and explore broader applications of AI in preoperative patient education.

Use of ChatGPT to Generate Informed Consent for Surgery in Urogynecology

IMPORTANCE

Use of the publicly available Large Language Model, Chat Generative Pre-trained Transformer (ChatGPT 3.5; OpenAI, 2022), is growing in health care despite varying accuracies.

OBJECTIVE

The aim of this study was to assess the accuracy and readability of ChatGPT's responses to questions encompassing surgical informed consent in urogynecology.

STUDY DESIGN

Five fellowship-trained urogynecology attending physicians and 1 reconstructive female urologist evaluated ChatGPT's responses to questions about 4 surgical procedures: (1) retropubic midurethral sling, (2) total vaginal hysterectomy, (3) uterosacral ligament suspension, and (4) sacrocolpopexy. Questions involved procedure descriptions, risks/benefits/alternatives, and additional resources. Responses were rated using the DISCERN tool, a 4-point accuracy scale, and the Flesch-Kinkaid Grade Level score.

RESULTS

The median DISCERN tool overall rating was 3 (interquartile range [IQR], 3-4), indicating a moderate rating ("potentially important but not serious shortcomings"). Retropubic midurethral sling received the highest overall score (median, 4; IQR, 3-4), and uterosacral ligament suspension received the lowest (median, 3; IQR, 3-3). Using the 4-point accuracy scale, 44.0% of responses received a score of 4 ("correct and adequate"), 22.6% received a score of 3 ("correct but insufficient"), 29.8% received a score of 2 ("accurate and misleading information together"), and 3.6% received a score of 1 ("wrong or irrelevant answer"). ChatGPT performance was poor for discussion of benefits and alternatives for all surgical procedures, with some responses being inaccurate. The mean Flesch-Kinkaid Grade Level score for all responses was 17.5 (SD, 2.1), corresponding to a postgraduate reading level.

CONCLUSIONS

Overall, ChatGPT generated accurate responses to questions about surgical informed consent. However, it produced clearly false portions of responses, highlighting the need for a careful review of responses by qualified health care professionals.

Evaluation of the impact Of ChatGPT support on acromegaly management and patient education

PURPOSE

ChatGPT is a widely used artificial intelligence modeling tool. Healthcare is one potential area of use of ChatGPT. This study aimed to test the usability and reliability of ChatGPT in acromegaly, which is less known in society and should be evaluated by a group of specialized physicians.

METHODS

The study is designed in two parts. For the first part, 35 questions regarding acromegaly that patients frequently ask were identified, and these questions were asked to ChatGPT. In the second part, four patient examples were presented to ChatGPT using medical terminology. Three experts evaluated ChatGPT's answers to the questions and approaches in case management using 7-point scales in terms of safety, reliability, correctness, and usability.

RESULTS

When the ChatGPT answers to the patient's questions were evaluated, a mean score of 6.78 ± 0.55 was given for correctness and 6.69 ± 0.60 for reliability. The mean scores given by the raters for correctness, safety and usability in the evaluation of the cases were as follows: 6.33 ± 0.88, 6.16 ± 0. 71 and 6.08 ± 0.79 points for case 1; 5.35 ± 1.88, 5.29 ± 1.80 and 5.20 ± 1.86 points for case 2; 6.08 ± 0.97, 6.00 ± 0.93 and 5.91 ± 0.82 points for case 3; 6.10 ± 1.29, 6.13 ± 1.30 and 6.16 ± 1.14 points for case 4.

CONCLUSION

ChatGPT can actively answer the questions of acromegaly patients. Although it is not a reliable source alone in managing patients with acromegaly, it can be a supportive tool for physicians.

Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection

Background

The International Classification of Diseases (ICD), developed by the World Health Organization, standardizes health condition coding to support health care policy, research, and billing, but artificial intelligence automation, while promising, still underperforms compared with human accuracy and lacks the explainability needed for adoption in medical settings.

Objective

The potential of large language models for assisting medical coders in the ICD-10 coding was explored through the development of a computer-assisted coding system. This study aimed to augment human coding by initially identifying lead terms and using retrieval-augmented generation (RAG)-based methods for computer-assisted coding enhancement.

Methods

The explainability dataset from the CodiEsp challenge (CodiEsp-X) was used, featuring 1000 Spanish clinical cases annotated with ICD-10 codes. A new dataset, CodiEsp-X-lead, was generated using GPT-4 to replace full-textual evidence annotations with lead term annotations. A Robustly Optimized BERT (Bidirectional Encoder Representations from Transformers) Pretraining Approach transformer model was fine-tuned for named entity recognition to extract lead terms. GPT-4 was subsequently employed to generate code descriptions from the extracted textual evidence. Using a RAG approach, ICD codes were assigned to the lead terms by querying a vector database of ICD code descriptions with OpenAI's text-embedding-ada-002 model.

Results

The fine-tuned Robustly Optimized BERT Pretraining Approach achieved an overall F1-score of 0.80 for ICD lead term extraction on the new CodiEsp-X-lead dataset. GPT-4-generated code descriptions reduced retrieval failures in the RAG approach by approximately 5% for both diagnoses and procedures. However, the overall explainability F1-score for the CodiEsp-X task was limited to 0.305, significantly lower than the state-of-the-art F1-score of 0.633. The diminished performance was partly due to the reliance on code descriptions, as some ICD codes lacked descriptions, and the approach did not fully align with the medical coder's workflow.

Conclusions

While lead term extraction showed promising results, the subsequent RAG-based code assignment using GPT-4 and code descriptions was less effective. Future research should focus on refining the approach to more closely mimic the medical coder's workflow, potentially integrating the alphabetic index and official coding guidelines, rather than relying solely on code descriptions. This alignment may enhance system accuracy and better support medical coders in practice.

Large Language Models for Chatbot Health Advice Studies: A Systematic Review

Importance

There is much interest in the clinical integration of large language models (LLMs) in health care. Many studies have assessed the ability of LLMs to provide health advice, but the quality of their reporting is uncertain.

Objective

To perform a systematic review to examine the reporting variability among peer-reviewed studies evaluating the performance of generative artificial intelligence (AI)-driven chatbots for summarizing evidence and providing health advice to inform the development of the Chatbot Assessment Reporting Tool (CHART).

Evidence Review

A search of MEDLINE via Ovid, Embase via Elsevier, and Web of Science from inception to October 27, 2023, was conducted with the help of a health sciences librarian to yield 7752 articles. Two reviewers screened articles by title and abstract followed by full-text review to identify primary studies evaluating the clinical accuracy of generative AI-driven chatbots in providing health advice (chatbot health advice studies). Two reviewers then performed data extraction for 137 eligible studies.

Findings

A total of 137 studies were included. Studies examined topics in surgery (55 [40.1%]), medicine (51 [37.2%]), and primary care (13 [9.5%]). Many studies focused on treatment (91 [66.4%]), diagnosis (60 [43.8%]), or disease prevention (29 [21.2%]). Most studies (136 [99.3%]) evaluated inaccessible, closed-source LLMs and did not provide enough information to identify the version of the LLM under evaluation. All studies lacked a sufficient description of LLM characteristics, including temperature, token length, fine-tuning availability, layers, and other details. Most studies (136 [99.3%]) did not describe a prompt engineering phase in their study. The date of LLM querying was reported in 54 (39.4%) studies. Most studies (89 [65.0%]) used subjective means to define the successful performance of the chatbot, while less than one-third addressed the ethical, regulatory, and patient safety implications of the clinical integration of LLMs.

Conclusions and Relevance

In this systematic review of 137 chatbot health advice studies, the reporting quality was heterogeneous and may inform the development of the CHART reporting standards. Ethical, regulatory, and patient safety considerations are crucial as interest grows in the clinical integration of LLMs.

ChatGPT 35 Better Improves Comprehensibility of English, Than Spanish, Generated Responses to Osteosarcoma Questions

BACKGROUND

Despite adequate discussion and counseling in the office, inadequate health literacy or language barriers may make it difficult to follow instructions from a physician and access necessary resources. This may negatively impact survival outcomes. Most healthcare materials are written at a 10th grade level, while many patients read at an 8th grade level. Hispanic Americans comprise about 25% of the US patient population, while only 6% of physicians identify as bilingual.

QUESTIONS/PURPOSE

(1) Does ChatGPT 3.5 provide appropriate responses to frequently asked patient questions that are sufficient for clinical practice and accurate in English and Spanish? (2) What is the comprehensibility of the responses provided by ChatGPT 3.5 and are these modifiable?

METHODS

Twenty frequently asked osteosarcoma patient questions, evaluated by two fellowship-trained musculoskeletal oncologists were input into ChatGPT 3.5. Responses were evaluated by two independent reviewers to assess appropriateness for clinical practice, and accuracy. Responses were graded using the Flesch Reading Ease Score (FRES) and the Flesch-Kincaid Grade Level test (FKGL). The responses were then input into ChatGPT 3.5 for a second time with the following command "Make text easier to understand". The same method was done in Spanish.

RESULTS

All responses generated were appropriate for a patient-facing informational platform. There was no difference in the Flesch Reading Ease Score between English and Spanish responses before the modification (p = 0.307) and with the Flesch-Kincaid grade level (p = 0.294). After modification, there was a statistically significant difference in comprehensibility between English and Spanish responses (p = 0.003 and p = 0.011).

CONCLUSION

In both English and Spanish, none of the ChatGPT generated responses were found to be factually inaccurate. ChatGPT was able to modify responses upon follow-up with a simplified command. However, it was shown to be better at improving English responses than equivalent Spanish responses.

Approaches for extracting daily dosage from free-text prescription signatures in heart failure with reduced ejection fraction: a comparative study

Objective

To compare various methods for extracting daily dosage information from prescription signatures (sigs) and identify the best performers.

Materials and Methods

In this study, 5 daily dosage extraction methods were identified. Parsigs, RxSig, Sig2db, a large language model (LLM), and a bidirectional long short-term memory (BiLSTM) model were selected. The methods were analyzed with regard to positive predictive value (PPV), sensitivity, F1-score, cost to compute, and time to finish on a sig dataset in the context of heart failure with reduced ejection fraction.

Results

The dataset consisted of 29 896 free-text sigs, which were split into training and validation sets of 70% and 30%, respectively. The BiLSTM model scored lowest with an F1-score of 0.71. The LLM GPT-4o and regular expression-based RxSig achieved the highest F1-scores with 0.98 and 0.95, respectively. The LLM outperformed RxSig in sensitivity. RxSig outperformed the LLM in PPV. Additionally, RxSig had a lower run time and no costs compared to a cost of 25 dollars.

Discussion

In practical usage, it would be preferable for an algorithm to score high on PPV and F1-score, to reduce false positive assertions of daily dosage. Additionally, long running times and high costs are not scalable for larger datasets. Thus, RxSig is likely the most scalable approach. Further research is needed to investigate the generalizability of the findings.

Conclusion

This study demonstrates that both the LLM and RxSig models excel in daily dose extraction from free-text sigs, with the RxSig model appearing to be the more scalable approach.

Navigating vaccination in pregnancy: Qualitative study in 21 ethnically diverse pregnant women

BACKGROUND

Vaccination during pregnancy is crucial for safeguarding maternal and neonatal health, but vaccination rates remain suboptimal, especially in women from Black and Asian ethnic minorities. We explored the perspectives and decision-making processes of pregnant women regarding uptake of the three recommended vaccines in pregnancy: Influenza, Pertussis (whooping cough) and COVID-19. We also explored women's attitudes to taking part in vaccine trials during pregnancy and the use of artificial intelligence (AI) to obtain information on vaccines.

METHODS

In 2023, we conducted in-depth telephone interviews with ethnically diverse pregnant women in the Greater London area using convenience and snowball sampling. The interviews focused on participants' views on vaccination during pregnancy, participation in vaccine trials, information-seeking behaviours, and attitudes to emerging technologies for health information. Interviews were transcribed verbatim and thematically analysed. The data collection and analysis were conducted alongside the iterative development of the topic guide and coding framework, with key themes emerging through collaborative team discussions.

RESULTS

Twenty one pregnant women aged 20-39 were interviewed of whom 67% were from ethnic minorities and 29% were from migrant backgrounds. Half the participants (53%) reported hesitancy towards at least one of the vaccines. The analysis revealed several themes: concerns about vaccine safety, particularly regarding newer vaccines due to lack of long-term data; reliance on healthcare professionals for guidance, balanced with personal research; and a strong desire for clear and comprehensive information specifically tailored to pregnant women. Pregnant women reported insufficient information, explanation, or recommendation by midwives. Additionally, there was widespread refusal regarding participation in vaccine trials; and mixed responses to the use of AI (such as chatbots) for obtaining vaccine information.

CONCLUSIONS

Pregnant women's vaccination decisions are complex and require clear, unambiguous communication from healthcare providers, especially midwives, to address their specific concerns. Although information obtained via AI can be useful, responses were mixed.

Evaluating ChatGPT's diagnostic potential for pathology images

Background

Chat Generative Pretrained Transformer (ChatGPT) is a type of large language model (LLM) developed by OpenAI, known for its extensive knowledge base and interactive capabilities. These attributes make it a valuable tool in the medical field, particularly for tasks such as answering medical questions, drafting clinical notes, and optimizing the generation of radiology reports. However, keeping accuracy in medical contexts is the biggest challenge to employing GPT-4 in a clinical setting. This study aims to investigate the accuracy of GPT-4, which can process both text and image inputs, in generating diagnoses from pathological images.

Methods

This study analyzed 44 histopathological images from 16 organs and 100 colorectal biopsy photomicrographs. The initial evaluation was conducted using the standard GPT-4 model in January 2024, with a subsequent re-evaluation performed in July 2024. The diagnostic accuracy of GPT-4 was assessed by comparing its outputs to a reference standard using statistical measures. Additionally, four pathologists independently reviewed the same images to compare their diagnoses with the model's outputs. Both scanned and photographed images were tested to evaluate GPT-4's generalization ability across different image types.

Results

GPT-4 achieved an overall accuracy of 0.64 in identifying tumor imaging and tissue origins. For colon polyp classification, accuracy varied from 0.57 to 0.75 in different subtypes. The model achieved 0.88 accuracy in distinguishing low-grade from high-grade dysplasia and 0.75 in distinguishing high-grade dysplasia from adenocarcinoma, with a high sensitivity in detecting adenocarcinoma. Consistency between initial and follow-up evaluations showed slight to moderate agreement, with Kappa values ranging from 0.204 to 0.375.

Conclusion

GPT-4 demonstrates the ability to diagnose pathological images, showing improved performance over earlier versions. Its diagnostic accuracy in cancer is comparable to that of pathology residents. These findings suggest that GPT-4 holds promise as a supportive tool in pathology diagnostics, offering the potential to assist pathologists in routine diagnostic workflows.

Current applications and challenges in large language models for patient care: a systematic review

BACKGROUND

The introduction of large language models (LLMs) into clinical practice promises to improve patient education and empowerment, thereby personalizing medical care and broadening access to medical knowledge. Despite the popularity of LLMs, there is a significant gap in systematized information on their use in patient care. Therefore, this systematic review aims to synthesize current applications and limitations of LLMs in patient care.

METHODS

We systematically searched 5 databases for qualitative, quantitative, and mixed methods articles on LLMs in patient care published between 2022 and 2023. From 4349 initial records, 89 studies across 29 medical specialties were included. Quality assessment was performed using the Mixed Methods Appraisal Tool 2018. A data-driven convergent synthesis approach was applied for thematic syntheses of LLM applications and limitations using free line-by-line coding in Dedoose.

RESULTS

We show that most studies investigate Generative Pre-trained Transformers (GPT)-3.5 (53.2%, n = 66 of 124 different LLMs examined) and GPT-4 (26.6%, n = 33/124) in answering medical questions, followed by patient information generation, including medical text summarization or translation, and clinical documentation. Our analysis delineates two primary domains of LLM limitations: design and output. Design limitations include 6 second-order and 12 third-order codes, such as lack of medical domain optimization, data transparency, and accessibility issues, while output limitations include 9 second-order and 32 third-order codes, for example, non-reproducibility, non-comprehensiveness, incorrectness, unsafety, and bias.

CONCLUSIONS

This review systematically maps LLM applications and limitations in patient care, providing a foundational framework and taxonomy for their implementation and evaluation in healthcare settings.

Artificial Intelligence-Driven Translation Tools in Intensive Care Units for Enhancing Communication and Research

There is a need to improve communication for patients and relatives who belong to cultural minority communities in intensive care units (ICUs). As a matter of fact, language barriers negatively impact patient safety and family participation in the care of critically ill patients, as well as recruitment to clinical trials. Recent studies indicate that Google Translate and ChatGPT are not accurate enough for advanced medical terminology. Therefore, developing and implementing an ad hoc machine translation tool is essential for bridging language barriers. This tool would enable language minority communities to access advanced healthcare facilities and innovative research in a timely and effective manner, ensuring they receive the comprehensive care and information they need.

METHOD

Key factors that facilitate access to advanced health services, in particular ICUs, for language minority communities are reviewed.

RESULTS

The existing digital communication tools in emergency departments and ICUs are reviewed. To the best of our knowledge, no AI English/French translation app has been developed for deployment in ICUs. Patient privacy and data confidentiality are other important issues that should be addressed.

CONCLUSIONS

Developing an artificial intelligence-driven translation tool for intensive care units (AITIC) which uses language models trained with medical/ICU terminology datasets could offer fast and accurate real-time translation. An AITIC could support communication, and consolidate and expand original research involving language minority communities.

Transforming dental diagnostics with artificial intelligence: advanced integration of ChatGPT and large language models for patient care

Artificial intelligence has dramatically reshaped our interaction with digital technologies, ushering in an era where advancements in AI algorithms and Large Language Models (LLMs) have natural language processing (NLP) systems like ChatGPT. This study delves into the impact of cutting-edge LLMs, notably OpenAI's ChatGPT, on medical diagnostics, with a keen focus on the dental sector. Leveraging publicly accessible datasets, these models augment the diagnostic capabilities of medical professionals, streamline communication between patients and healthcare providers, and enhance the efficiency of clinical procedures. The advent of ChatGPT-4 is poised to make substantial inroads into dental practices, especially in the realm of oral surgery. This paper sheds light on the current landscape and explores potential future research directions in the burgeoning field of LLMs, offering valuable insights for both practitioners and developers. Furthermore, it critically assesses the broad implications and challenges within various sectors, including academia and healthcare, thus mapping out an overview of AI's role in transforming dental diagnostics for enhanced patient care.

ChatGPT's Attitude, Knowledge, and Clinical Application in Geriatrics Practice and Education: Exploratory Observational Study

BACKGROUND

The increasing use of ChatGPT in clinical practice and medical education necessitates the evaluation of its reliability, particularly in geriatrics.

OBJECTIVE

This study aimed to evaluate ChatGPT's trustworthiness in geriatrics through 3 distinct approaches: evaluating ChatGPT's geriatrics attitude, knowledge, and clinical application with 2 vignettes of geriatric syndromes (polypharmacy and falls).

METHODS

We used the validated University of California, Los Angeles, geriatrics attitude and knowledge instruments to evaluate ChatGPT's geriatrics attitude and knowledge and compare its performance with that of medical students, residents, and geriatrics fellows from reported results in the literature. We also evaluated ChatGPT's application to 2 vignettes of geriatric syndromes (polypharmacy and falls).

RESULTS

The mean total score on geriatrics attitude of ChatGPT was significantly lower than that of trainees (medical students, internal medicine residents, and geriatric medicine fellows; 2.7 vs 3.7 on a scale from 1-5; 1=strongly disagree; 5=strongly agree). The mean subscore on positive geriatrics attitude of ChatGPT was higher than that of the trainees (medical students, internal medicine residents, and neurologists; 4.1 vs 3.7 on a scale from 1 to 5 where a higher score means a more positive attitude toward older adults). The mean subscore on negative geriatrics attitude of ChatGPT was lower than that of the trainees and neurologists (1.8 vs 2.8 on a scale from 1 to 5 where a lower subscore means a less negative attitude toward aging). On the University of California, Los Angeles geriatrics knowledge test, ChatGPT outperformed all medical students, internal medicine residents, and geriatric medicine fellows from validated studies (14.7 vs 11.3 with a score range of -18 to +18 where +18 means that all questions were answered correctly). Regarding the polypharmacy vignette, ChatGPT not only demonstrated solid knowledge of potentially inappropriate medications but also accurately identified 7 common potentially inappropriate medications and 5 drug-drug and 3 drug-disease interactions. However, ChatGPT missed 5 drug-disease and 1 drug-drug interaction and produced 2 hallucinations. Regarding the fall vignette, ChatGPT answered 3 of 5 pretests correctly and 2 of 5 pretests partially correctly, identified 6 categories of fall risks, followed fall guidelines correctly, listed 6 key physical examinations, and recommended 6 categories of fall prevention methods.

CONCLUSIONS

This study suggests that ChatGPT can be a valuable supplemental tool in geriatrics, offering reliable information with less age bias, robust geriatrics knowledge, and comprehensive recommendations for managing 2 common geriatric syndromes (polypharmacy and falls) that are consistent with evidence from guidelines, systematic reviews, and other types of studies. ChatGPT's potential as an educational and clinical resource could significantly benefit trainees, health care providers, and laypeople. Further research using GPT-4o, larger geriatrics question sets, and more geriatric syndromes is needed to expand and confirm these findings before adopting ChatGPT widely for geriatrics education and practice.

Clustering of clinical symptoms using large language models reveals low diagnostic specificity of proposed alternatives to consensus mast cell activation syndrome criteria

BACKGROUND

The rate of diagnosis of mast cell activation syndrome (MCAS) has increased since the disorder's original description as a mastocytosis-like phenotype. While a set of consortium MCAS criteria is well described and widely accepted, this increase occurs in the setting of a broader set of proposed alternative MCAS criteria.

OBJECTIVE

Effective diagnostic criteria must minimize the range of unrelated diagnoses that can be erroneously classified as the condition of interest. We sought to determine if the symptoms associated with alternative MCAS criteria result in less concise or consistent diagnostic alternatives, reducing diagnostic specificity.

METHODS

We used multiple large language models, including ChatGPT, Claude, and Gemini, to bootstrap the probabilities of diagnoses that are compatible with consortium or alternative MCAS criteria. We utilized diversity and network analyses to quantify diagnostic precision and specificity compared to control diagnostic criteria including systemic lupus erythematosus, Kawasaki disease, and migraines.

RESULTS

Compared to consortium MCAS criteria, alternative MCAS criteria are associated with more variable (Shannon diversity 5.8 vs 4.6, respectively; P = .004) and less precise (mean Bray-Curtis similarity 0.07 vs 0.19, respectively; P = .004) diagnoses. The diagnosis networks derived from consortium and alternative MCAS criteria had lower between-network similarity compared to the similarity between diagnosis networks derived from 2 distinct systemic lupus erythematosus criteria (cosine similarity 0.55 vs 0.86, respectively; P = .0022).

CONCLUSION

Alternative MCAS criteria are associated with a distinct set of diagnoses compared to consortium MCAS criteria and have lower diagnostic consistency. This lack of specificity is pronounced in relation to multiple control criteria, raising the concern that alternative criteria could disproportionately contribute to MCAS overdiagnosis, to the exclusion of more appropriate diagnoses.

Assessment of Large Language Models in Cataract Care Information Provision: A Quantitative Comparison

INTRODUCTION

Cataracts are a significant cause of blindness. While individuals frequently turn to the Internet for medical advice, distinguishing reliable information can be challenging. Large language models (LLMs) have attracted attention for generating accurate, human-like responses that may be used for medical consultation. However, a comprehensive assessment of LLMs' accuracy within specific medical domains is still lacking.

METHODS

We compiled 46 commonly inquired questions related to cataract care, categorized into six domains. Each question was presented to the LLMs, and three consultant-level ophthalmologists independently assessed the accuracy of their responses on a three-point scale (poor, borderline, good) and their comprehensiveness on a five-point scale. A majority consensus approach established the final rating for each response. Responses rated as 'Poor' were prompted for self-correction and reassessed.

RESULTS

For accuracy, ChatGPT-4o and Google Bard both achieved average sum scores of 8.7 (out of 9), followed by ChatGPT-3.5, Bing Chat, Llama 2, and Wenxin Yiyan. In consensus-based ratings, ChatGPT-4o outperformed Google Bard in the 'Good' rating. For completeness, ChatGPT-4o had the highest average sum score of 13.22 (out of 15), followed by Google Bard, ChatGPT-3.5, Llama 2, Bing Chat, and Wenxin Yiyan. Detailed performance data reveal nuanced differences in model capabilities. In the 'Prevention' domain, apart from Wenxin Yiyan, all other models were rated as 'Good'. All models showed improvement in self-correction. Bard and Bing improved 1/1 from 'Poor' to better, Llama improved 3/4, and Wenxin Yiyan improved 4/5.

CONCLUSIONS

Our findings emphasize the potential of LLMs, particularly ChatGPT-4o, to deliver accurate and comprehensive responses to cataract-related queries, especially in prevention, indicating potential for medical consultation. Continuous efforts to enhance LLMs' accuracy through ongoing strategies and evaluations are essential.

Applications and Future Prospects of Medical LLMs: A Survey Based on the M-KAT Conceptual Framework

The success of large language models (LLMs) in general areas have sparked a wave of research into their applications in the medical field. However, enhancing the medical professionalism of these models remains a major challenge. This study proposed a novel model training theoretical framework, the M-KAT framework, which integrated domain-specific training methods for LLMs with the unique characteristics of the medical discipline. This framework aimed to improve the medical professionalism of the models from three perspectives: general knowledge acquisition, specialized skill development, and alignment with clinical thinking. This study summarized the outcomes of medical LLMs across four tasks: clinical diagnosis and treatment, medical question answering, medical research, and health management. Using the M-KAT framework, we analyzed the contribution to enhancement of professionalism of models through different training stages. At the same time, for some of the potential risks associated with medical LLMs, targeted solutions can be achieved through pre-training, SFT, and model alignment based on cultivated professional capabilities. Additionally, this study identified main directions for future research on medical LLMs: advancing professional evaluation datasets and metrics tailored to the needs of medical tasks, conducting in-depth studies on medical multimodal large language models (MLLMs) capable of integrating diverse data types, and exploring the forms of medical agents and multi-agent frameworks that can interact with real healthcare environments and support clinical decision-making. It is hoped that predictions of work can provide a reference for subsequent research.

Artificial Intelligence and ChatGPT in Medical Education: A Cross-Sectional Questionnaire on students' Competence

Artificial intelligence is rapidly transforming the field of health science and medical education, but less is known about the students´ competencies related to knowledge, skills and attitudes towards the application of AI tools like ChatGPT. Therefore, a unicentric questionnaire-based cross-sectional study was applied to students in the medical field (n = 207). The data revealed that while most students were familiar with ChatGPT (66.7%), other AI tools were significantly less known or utilised for study purposes. Students approached AI tools rather informally, often preferring to use them as a simple search engine. More than half of the students admitted that they were not sufficiently informed about the underlying technology of AI. They applied ChatGPT in a self-directed manner but expressed considerable uncertainty regarding effective prompt engineering and ChatGPT's legal implications. Overall, the majority of respondents showed interest in and positivity towards the introduction of AI. However, they did not feel adequately prepared to handle AI confidently, leading many to express interest in further training. This training should be directly related to students' professional roles, e.g. as a physician. The three most favoured AI-topics for voluntary learning formats were AI in their studies (62.5%), AI in general (58.0%), and the use of AI in scientific writing (57.0%). Notable subgroup differences related to the students" gender or self-assessed study performance were observed and should be considered in future research.

Real-world utility of ChatGPT in pre-vasectomy counselling, a safe and efficient practice: a prospective single-centre clinical study

PURPOSE

This study sought to assess if pre-vasectomy counselling with ChatGPT can safely streamline the consultation process by reducing visit times and increasing patient satisfaction.

METHODS

A single-institution randomized pilot study was conducted to evaluate the safety and efficacy of ChatGPT for pre-vasectomy counselling. All adult patients interested in undergoing a vasectomy were included. Unwillingness to provide consent or not having internet access constituted exclusion. Patients were randomized 1:1 to ChatGPT with standard in-person or in-person consultation without ChatGPT. Length of visit, number of questions asked, and a Likert scale questionnaire (on a scale of 10, with 10 being defined as great and 0 being defined as poor), were collected. Descriptive statistics and a comparative analysis were performed.

RESULTS

18 patients were included with a mean age of 35.8 ± 5.4 (n = 9) in the intervention arm and 36.9 ± 7.4 (n = 9) in the control arm. Pre-vasectomy counselling with ChatGPT was associated with a higher provider perception of patient understanding of the procedure (8.8 ± 1.0 vs. 6.7 ± 2.8; p = 0.047) and a decreased length of in-person consultation (7.7 ± 2.3 min vs. 10.6 ± 3.4 min; p = 0.05). Quality of information provided by ChatGPT, ease of use, and overall experience were rated highly at 8.3 ± 1.9, 9.1 ± 1.5, and 8.6 ± 1.7, respectively.

CONCLUSIONS

ChatGPT for pre-vasectomy counselling improved the efficiency of consultations and the provider's perception of the patient's understanding of the procedure.

NLP-Driven Analysis of Pneumothorax Incidence Following Central Venous Catheter Procedures: A Data-Driven Re-Evaluation of Routine Imaging in Value-Based Medicine

Background: This study presents a systematic approach using a natural language processing (NLP) algorithm to assess the necessity of routine imaging after central venous catheter (CVC) placement and removal. With pneumothorax being a key complication of CVC procedures, this research aims to provide evidence-based recommendations for optimizing imaging protocols and minimizing unnecessary imaging risks. Methods: We analyzed electronic health records from four university hospitals in Salzburg, Austria, focusing on X-rays performed between 2012 and 2021 following CVC procedures. A custom-built NLP algorithm identified cases of pneumothorax from radiologists' reports and clinician requests, while excluding cases with contraindications such as chest injuries, prior pneumothorax, or missing data. Chi-square tests were used to compare pneumothorax rates between CVC insertion and removal, and multivariate logistic regression identified risk factors, with a focus on age and gender. Results: This study analyzed 17,175 cases of patients aged 18 and older, with 95.4% involving CVC insertion and 4.6% involving CVC removal. Pneumothorax was observed in 106 cases post-insertion (1.3%) and in 3 cases post-removal (0.02%), with no statistically significant difference between procedures (p = 0.5025). The NLP algorithm achieved an accuracy of 93%, with a sensitivity of 97.9%, a specificity of 87.9%, and an area under the ROC curve (AUC) of 0.9283. Conclusions: The findings indicate no significant difference in pneumothorax incidence between CVC insertion and removal, supporting existing recommendations against routine imaging post-removal for asymptomatic patients and suggesting that routine imaging after CVC insertion may also be unnecessary in similar cases. This study demonstrates how advanced NLP techniques can support value-based medicine by enhancing clinical decision making and optimizing resources.

The role of large language models in self-care: a study and benchmark on medicines and supplement guidance accuracy

BACKGROUND

The recent surge in the capabilities of artificial intelligence systems, particularly large language models, is also impacting the medical and pharmaceutical field in a major way. Beyond specialized uses in diagnostics and data discovery, these tools have now become accessible to the general public.

AIM

The study aimed to critically analyse the current performance of large language models in answering patient's self-care questions regarding medications and supplements.

METHOD

Answers from six major language models were analysed for correctness, language-independence, context-sensitivity, and reproducibility using a newly developed reference set of questions and a scoring matrix.

RESULTS

The investigated large language models are capable of answering a clear majority of self-care questions accurately, providing relevant health information. However, substantial variability in the responses, including potentially unsafe advice, was observed, influenced by language, question structure, user context and time. GPT 4.0 scored highest on average, while GPT 3.5, Gemini, and Gemini Advanced had varied scores. Responses were context and language sensitive. In terms of consistency over time, Perplexity had the worst performance.

CONCLUSION

Given the high-quality output of large language models, their potential in self-care applications is undeniable. The newly created benchmark can facilitate further validation and guide the establishment of strict safeguards to combat the sizable risk of misinformation in order to reach a more favourable risk/benefit ratio when this cutting-edge technology is used by patients.

Large Language Models in Biomedical and Health Informatics: A Review with Bibliometric Analysis

Large language models (LLMs) have rapidly become important tools in Biomedical and Health Informatics (BHI), potentially enabling new ways to analyze data, treat patients, and conduct research. This study aims to provide a comprehensive overview of LLM applications in BHI, highlighting their transformative potential and addressing the associated ethical and practical challenges. We reviewed 1698 research articles from January 2022 to December 2023, categorizing them by research themes and diagnostic categories. Additionally, we conducted network analysis to map scholarly collaborations and research dynamics. Our findings reveal a substantial increase in the potential applications of LLMs to a variety of BHI tasks, including clinical decision support, patient interaction, and medical document analysis. Notably, LLMs are expected to be instrumental in enhancing the accuracy of diagnostic tools and patient care protocols. The network analysis highlights dense and dynamically evolving collaborations across institutions, underscoring the interdisciplinary nature of LLM research in BHI. A significant trend was the application of LLMs in managing specific disease categories, such as mental health and neurological disorders, demonstrating their potential to influence personalized medicine and public health strategies. LLMs hold promising potential to further transform biomedical research and healthcare delivery. While promising, the ethical implications and challenges of model validation call for rigorous scrutiny to optimize their benefits in clinical settings. This survey serves as a resource for stakeholders in healthcare, including researchers, clinicians, and policymakers, to understand the current state and future potential of LLMs in BHI.

Automated Identification of Breast Cancer Relapse in Computed Tomography Reports Using Natural Language Processing

PURPOSE

Breast cancer relapses are rarely collected by cancer registries because of logistical and financial constraints. Hence, we investigated natural language processing (NLP), enhanced with state-of-the-art deep learning transformer tools and large language models, to automate relapse identification in the text of computed tomography (CT) reports.

METHODS

We analyzed follow-up CT reports from patients diagnosed with breast cancer between January 1, 2005, and December 31, 2014. The reports were curated and annotated for the presence or absence of local, regional, and distant breast cancer relapses. We performed 10-fold cross-validation to evaluate models identifying different types of relapses in CT reports. Model performance was assessed with classification metrics, reported with 95% confidence intervals.

RESULTS

In our data set of 1,445 CT reports, 799 (55.3%) described any relapse, 72 (5.0%) local relapses, 97 (6.7%) regional relapses, and 743 (51.4%) distant relapses. The any-relapse model achieved an accuracy of 89.6% (87.8-91.1), with a sensitivity of 93.2% (91.4-94.9) and a specificity of 84.2% (80.9-87.1). The local relapse model achieved an accuracy of 94.6% (93.3-95.7), a sensitivity of 44.4% (32.8-56.3), and a specificity of 97.2% (96.2-98.0). The regional relapse model showed an accuracy of 93.6% (92.3-94.9), a sensitivity of 70.1% (60.0-79.1), and a specificity of 95.3% (94.2-96.5). Finally, the distant relapse model demonstrated an accuracy of 88.1% (86.2-89.7), a sensitivity of 91.8% (89.9-93.8), and a specificity of 83.7% (80.5-86.4).

CONCLUSION

We developed NLP models to identify local, regional, and distant breast cancer relapses from CT reports. Automating the identification of breast cancer relapses can enhance data collection about patient outcomes.

Understanding Occlusion and Temporomandibular Joint Function Using Deep Learning and Predictive Modeling

OBJECTIVES

Advancements in artificial intelligence (AI)-driven predictive modeling in dentistry are outpacing the clinical translation of research findings. Predictive modeling uses statistical methods to anticipate norms related to TMJ dynamics, complementing imaging modalities like cone beam computed tomography (CBCT) and magnetic resonance imaging (MRI). Deep learning, a subset of AI, helps quantify and analyze complex hierarchical relationships in occlusion and TMJ function. This narrative review explores the application of predictive modeling and deep learning to identify clinical trends and associations related to occlusion and TMJ function.

RESULTS

Debates persist regarding best practices for managing occlusal factors in temporomandibular joint (TMJ) function analysis while interpreting and quantifying findings related to the TMJ and occlusion and mitigating biases remain challenging. Data generated from noninvasive chairside tools such as jaw trackers, video tracking, and 3D scanners with virtual articulators offer unique insights by predicting variations in dynamic jaw movement, TMJ, and occlusion. The predictions help us understand the highly individualized norms surrounding TMJ function that are often required to address temporomandibular disorders (TMDs) in general practice.

CONCLUSIONS

Normal TMJ function, occlusion, and the appropriate management of TMDs are complex and continue to attract ongoing debate. This review examines how predictive modeling and artificial intelligence aid in understanding occlusion and TMJ function and provides insights into complex dental conditions such as TMDs that may improve diagnosis and treatment outcomes with noninvasive techniques.

Utilizing large language models in infectious disease transmission modelling for public health preparedness

Introduction

OpenAI's ChatGPT, a Large Language Model (LLM), is a powerful tool across domains, designed for text and code generation, fostering collaboration, especially in public health. Investigating the role of this advanced LLM chatbot in assisting public health practitioners in shaping disease transmission models to inform infection control strategies, marks a new era in infectious disease epidemiology research. This study used a case study to illustrate how ChatGPT collaborates with a public health practitioner in co-designing a mathematical transmission model.

Methods

Using natural conversation, the practitioner initiated a dialogue involving an iterative process of code generation, refinement, and debugging with ChatGPT to develop a model to fit 10 days of prevalence data to estimate two key epidemiological parameters: i) basic reproductive number (Ro) and ii) final epidemic size. Verification and validation processes are conducted to ensure the accuracy and functionality of the final model.

Results

ChatGPT developed a validated transmission model which replicated the epidemic curve and gave estimates of Ro of 4.19 (95 % CI: 4.13- 4.26) and a final epidemic size of 98.3 % of the population within 60 days. It highlighted the advantages of using maximum likelihood estimation with Poisson distribution over least squares method.

Conclusion

Integration of LLM in medical research accelerates model development, reducing technical barriers for health practitioners, democratizing access to advanced modeling and potentially enhancing pandemic preparedness globally, particularly in resource-constrained populations.

Exploring ChatGPT in clinical inquiry: a scoping review of characteristics, applications, challenges, and evaluation

Introduction

Recent advancements in generative AI, exemplified by ChatGPT, hold promise for healthcare applications such as decision-making support, education, and patient engagement. However, rigorous evaluation is crucial to ensure reliability and safety in clinical contexts. This scoping review explores ChatGPT's role in clinical inquiry, focusing on its characteristics, applications, challenges, and evaluation.

Methods

This review, conducted in 2023, followed PRISMA-ScR guidelines (Supplemental Digital Content 1, http://links.lww.com/MS9/A636). Searches were performed across PubMed, Scopus, IEEE, Web of Science, Cochrane, and Google Scholar using relevant keywords. The review explored ChatGPT's effectiveness in various medical domains, evaluation methods, target users, and comparisons with other AI models. Data synthesis and analysis incorporated both quantitative and qualitative approaches.

Results

Analysis of 41 academic studies highlights ChatGPT's potential in medical education, patient care, and decision support, though performance varies by medical specialty and linguistic context. GPT-3.5, frequently referenced in 26 studies, demonstrated adaptability across diverse scenarios. Challenges include limited access to official answer keys and inconsistent performance, underscoring the need for ongoing refinement. Evaluation methods, including expert comparisons and statistical analyses, provided significant insights into ChatGPT's efficacy. The identification of target users, such as medical educators and nonexpert clinicians, illustrates its broad applicability.

Conclusion

ChatGPT shows significant potential in enhancing clinical practice and medical education. Nevertheless, continuous refinement is essential for its successful integration into healthcare, aiming to improve patient care outcomes, and address the evolving needs of the medical community.

ChatGPT for parents' education about early childhood caries: A friend or foe?

BACKGROUND

With the increasing popularity of online sources for health information, parents may seek information related to early childhood caries (ECC) from artificial intelligence-based chatbots.

AIM

The aim of this article was to evaluate the usefulness, quality, reliability, and readability of ChatGPT answers to parents' questions about ECC.

DESIGN

Eighty questions commonly asked about ECC were compiled from experts and keyword research tools. ChatGPT 3.5 was asked these questions independently. The answers were evaluated by experts in paediatric dentistry.

RESULTS

ChatGPT provided "very useful" and "useful" responses to 82.5% of the questions. The mean global quality score was 4.3 ± 1 (good quality). The mean reliability score was 18.5 ± 8.9 (average to very good). The mean understandability score was 59.5% ± 13.8 (not highly understandable), and the mean actionability score was 40.5% ± 12.8 (low actionability). The mean Flesch-Kincaid reading ease score was 32% ± 25.7, and the mean Simple Measure of Gobbledygook index readability score was 15.3 ± 9.1(indicating poor readability for the lay person). Misleading and false information were detected in some answers.

CONCLUSION

ChatGPT has significant potential as a tool for answering parent's questions about ECC. Concerns, however, do exist about the readability and actionability of the answers. The presence of false information should not be overlooked.

Feasibility of GPT-3.5 versus Machine Learning for Automated Surgical Decision-Making Determination: A Multicenter Study on Suspected Appendicitis

Background: Nonsurgical treatment of uncomplicated appendicitis is a reasonable option in many cases despite the sparsity of robust, easy access, externally validated, and multimodally informed clinical decision support systems (CDSSs). Developed by OpenAI, the Generative Pre-trained Transformer 3.5 model (GPT-3) may provide enhanced decision support for surgeons in less certain appendicitis cases or those posing a higher risk for (relative) operative contra-indications. Our objective was to determine whether GPT-3.5, when provided high-throughput clinical, laboratory, and radiological text-based information, will come to clinical decisions similar to those of a machine learning model and a board-certified surgeon (reference standard) in decision-making for appendectomy versus conservative treatment. Methods: In this cohort study, we randomly collected patients presenting at the emergency department (ED) of two German hospitals (GFO, Troisdorf, and University Hospital Cologne) with right abdominal pain between October 2022 and October 2023. Statistical analysis was performed using R, version 3.6.2, on RStudio, version 2023.03.0 + 386. Overall agreement between the GPT-3.5 output and the reference standard was assessed by means of inter-observer kappa values as well as accuracy, sensitivity, specificity, and positive and negative predictive values with the “Caret” and “irr” packages. Statistical significance was defined as p < 0.05. Results: There was agreement between the surgeon’s decision and GPT-3.5 in 102 of 113 cases, and all cases where the surgeon decided upon conservative treatment were correctly classified by GPT-3.5. The estimated model training accuracy was 83.3% (95% CI: 74.0, 90.4), while the validation accuracy for the model was 87.0% (95% CI: 66.4, 97.2). This is in comparison to the GPT-3.5 accuracy of 90.3% (95% CI: 83.2, 95.0), which did not perform significantly better in comparison to the machine learning model (p = 0.21). Conclusions: This study, the first study of the “intended use” of GPT-3.5 for surgical treatment to our knowledge, comparing surgical decision-making versus an algorithm found a high degree of agreement between board-certified surgeons and GPT-3.5 for surgical decision-making in patients presenting to the emergency department with lower abdominal pain.

The Potential of Using Generative AI/NLP to Identify and Analyse Critical Incidents in a Critical Incident Reporting System (CIRS): A Feasibility Case-Control Study

BACKGROUND

To enhance patient safety in healthcare, it is crucial to address the underreporting of issues in Critical Incident Reporting Systems (CIRSs). This study aims to evaluate the effectiveness of generative Artificial Intelligence and Natural Language Processing (AI/NLP) in reviewing CIRS cases by comparing its performance with human reviewers and categorising these cases into relevant topics.

METHODS

A case-control feasibility study was conducted using CIRS cases from the German CIRS-Anaesthesiology subsystem. Each case was reviewed by a human expert and by an AI/NLP model (ChatGPT-3.5). Two CIRS experts blindly assessed these reviews, rating them on linguistic quality, recognisable expertise, logical derivability, and overall quality using six-point Likert scales.

RESULTS

On average, the CIRS experts correctly classified 80% of human CIRS reviews as created by a human and misclassified 45.8% of AI reviews as written by a human. Ratings on a scale of 1 (very good) to 6 (failed) revealed a comparable performance between human- and AI-generated reviews across the dimensions of linguistic expression (p = 0.39), recognisable expertise (p = 0.89), logical derivability (p = 0.84), and overall quality (p = 0.87). The AI model was able to categorise the cases into relevant topics independently.

CONCLUSIONS

This feasibility study demonstrates the potential of generative AI/NLP in analysing and categorising cases from the CIRS. This could have implications for improving incident reporting in healthcare. Therefore, additional research is required to verify and expand upon these discoveries.