ChatGPT: promise and challenges for deployment in low- and middle-income countries

Conundrum and chances of diabetes management in the Western Pacific Region: A narrative review

The prevalence of diabetes is increasing globally, and glucose management is essential for the treatment of diabetes. Most guidelines recommend early intensive therapy and individualized approaches. Although many countries have implemented various guidelines and educational programs to enhance glucose management, the target achievement rate still remains very low. Studies from several countries and regions have identified various factors that influence blood glucose management, either positively or negatively. These factors have been comprehensively incorporated into guidelines to assist people with diabetes and healthcare professionals in following them and/or developing additional guidelines through further research. We and others have suggested that diverse factors should be considered-including comorbidities, age, complications, life expectancy, and pathophysiologic characteristics, such as ethnic differences in insulin sensitivity and secretion. The Western Pacific (WP) region, comprising countries with significant cultural and racial diversity, necessitates customized programs and community-based management strategies. In this review, we present specific challenges and opportunities for diabetes management identified through a systematic review of the literature from the WP region, along with those common to other regions. To improve healthcare policy and management in the WP region, it is essential to address regional characteristics and the factors that act as either barriers or facilitators to develop strategies for early intensive and individualized therapeutic approaches. Moreover, additional studies on diabetes pathophysiology and management-including pharmacotherapy-are urgently needed.

ChatGPT Responses to Clinical Questions in the Japan Atherosclerosis Society Guidelines for Prevention of Atherosclerotic Cardiovascular Disease 2022

AIMS

Artificial intelligence is increasingly used in the medical field. We assessed the accuracy and reproducibility of responses by ChatGPT to clinical questions (CQs) in the Japan Atherosclerosis Society Guidelines for Prevention Atherosclerotic Cardiovascular Diseases 2022 (JAS Guidelines 2022).

METHODS

In June 2024, we assessed responses by ChatGPT (version 3.5) to CQs, including background questions (BQs) and foreground questions (FQs). Accuracy was assessed independently by three researchers using six-point Likert scales ranging from 1 ("completely incorrect") to 6 ("completely correct") by evaluating responses to CQs in Japanese or translated into English. For reproducibility assessment, responses to each CQ asked five times separately in a new chat were scored using six-point Likert scales, and Fleiss kappa coefficients were calculated.

RESULTS

The median (25th-75th percentile) score for ChatGPT's responses to BQs and FQs was 4 (3-5) and 5 (5-6) for Japanese CQs and 5 (3-6) and 6 (5-6) for English CQs, respectively. Response scores were higher for FQs than those for BQs (P values ＜0.001 for Japanese and English). Similar response accuracy levels were observed between Japanese and English CQs (P value 0.139 for BQs and 0.586 for FQs). Kappa coefficients for reproducibility were 0.76 for BQs and 0.90 for FQs.

CONCLUSIONS

ChatGPT showed high accuracy and reproducibility in responding to JAS Guidelines 2022 CQs, especially FQs. While ChatGPT primarily reflects existing guidelines, its strength could lie in rapidly organizing and presenting relevant information, thus supporting instant and more efficient guideline interpretation and aiding in medical decision-making.

Unveiling the Potential of Large Language Models in Transforming Chronic Disease Management: Mixed Methods Systematic Review

BACKGROUND

Chronic diseases are a major global health burden, accounting for nearly three-quarters of the deaths worldwide. Large language models (LLMs) are advanced artificial intelligence systems with transformative potential to optimize chronic disease management; however, robust evidence is lacking.

OBJECTIVE

This review aims to synthesize evidence on the feasibility, opportunities, and challenges of LLMs across the disease management spectrum, from prevention to screening, diagnosis, treatment, and long-term care.

METHODS

Following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analysis) guidelines, 11 databases (Cochrane Central Register of Controlled Trials, CINAHL, Embase, IEEE Xplore, MEDLINE via Ovid, ProQuest Health & Medicine Collection, ScienceDirect, Scopus, Web of Science Core Collection, China National Knowledge Internet, and SinoMed) were searched on April 17, 2024. Intervention and simulation studies that examined LLMs in the management of chronic diseases were included. The methodological quality of the included studies was evaluated using a rating rubric designed for simulation-based research and the risk of bias in nonrandomized studies of interventions tool for quasi-experimental studies. Narrative analysis with descriptive figures was used to synthesize the study findings. Random-effects meta-analyses were conducted to assess the pooled effect estimates of the feasibility of LLMs in chronic disease management.

RESULTS

A total of 20 studies examined general-purpose (n=17) and retrieval-augmented generation-enhanced LLMs (n=3) for the management of chronic diseases, including cancer, cardiovascular diseases, and metabolic disorders. LLMs demonstrated feasibility across the chronic disease management spectrum by generating relevant, comprehensible, and accurate health recommendations (pooled accurate rate 71%, 95% CI 0.59-0.83; I2=88.32%) with retrieval-augmented generation-enhanced LLMs having higher accuracy rates compared to general-purpose LLMs (odds ratio 2.89, 95% CI 1.83-4.58; I2=54.45%). LLMs facilitated equitable information access; increased patient awareness regarding ailments, preventive measures, and treatment options; and promoted self-management behaviors in lifestyle modification and symptom coping. Additionally, LLMs facilitate compassionate emotional support, social connections, and health care resources to improve the health outcomes of chronic diseases. However, LLMs face challenges in addressing privacy, language, and cultural issues; undertaking advanced tasks, including diagnosis, medication, and comorbidity management; and generating personalized regimens with real-time adjustments and multiple modalities.

CONCLUSIONS

LLMs have demonstrated the potential to transform chronic disease management at the individual, social, and health care levels; however, their direct application in clinical settings is still in its infancy. A multifaceted approach that incorporates robust data security, domain-specific model fine-tuning, multimodal data integration, and wearables is crucial for the evolution of LLMs into invaluable adjuncts for health care professionals to transform chronic disease management.

TRIAL REGISTRATION

PROSPERO CRD42024545412; https://www.crd.york.ac.uk/PROSPERO/view/CRD42024545412.

Evaluating the Effectiveness of Large Language Models in Providing Patient Education for Chinese Patients With Ocular Myasthenia Gravis: Mixed Methods Study

BACKGROUND

Ocular myasthenia gravis (OMG) is a neuromuscular disorder primarily affecting the extraocular muscles, leading to ptosis and diplopia. Effective patient education is crucial for disease management; however, in China, limited health care resources often restrict patients' access to personalized medical guidance. Large language models (LLMs) have emerged as potential tools to bridge this gap by providing instant, AI-driven health information. However, their accuracy and readability in educating patients with OMG remain uncertain.

OBJECTIVE

The purpose of this study was to systematically evaluate the effectiveness of multiple LLMs in the education of Chinese patients with OMG. Specifically, the validity of these models in answering patients with OMG-related questions was assessed through accuracy, completeness, readability, usefulness, and safety, and patients' ratings of their usability and readability were analyzed.

METHODS

The study was conducted in two phases: 130 choice ophthalmology examination questions were input into 5 different LLMs. Their performance was compared with that of undergraduates, master's students, and ophthalmology residents. In addition, 23 common patients with OMG-related patient questions were posed to 4 LLMs, and their responses were evaluated by ophthalmologists across 5 domains. In the second phase, 20 patients with OMG interacted with the 2 LLMs from the first phase, each asking 3 questions. Patients assessed the responses for satisfaction and readability, while ophthalmologists evaluated the responses again using the 5 domains.

RESULTS

ChatGPT o1-preview achieved the highest accuracy rate of 73% on 130 ophthalmology examination questions, outperforming other LLMs and professional groups like undergraduates and master's students. For 23 common patients with OMG-related questions, ChatGPT o1-preview scored highest in correctness (4.44), completeness (4.44), helpfulness (4.47), and safety (4.6). GEMINI (Google DeepMind) provided the easiest-to-understand responses in readability assessments, while GPT-4o had the most complex responses, suitable for readers with higher education levels. In the second phase with 20 patients with OMG, ChatGPT o1-preview received higher satisfaction scores than Ernie 3.5 (Baidu; 4.40 vs 3.89, P=.002), although Ernie 3.5's responses were slightly more readable (4.31 vs 4.03, P=.01).

CONCLUSIONS

LLMs such as ChatGPT o1-preview may have the potential to enhance patient education. Addressing challenges such as misinformation risk, readability issues, and ethical considerations is crucial for their effective and safe integration into clinical practice.

Prospects and perils of ChatGPT in diabetes

ChatGPT, a popular large language model developed by OpenAI, has the potential to transform the management of diabetes mellitus. It is a conversational artificial intelligence model trained on extensive datasets, although not specifically health-related. The development and core components of ChatGPT include neural networks and machine learning. Since the current model is not yet developed on diabetes-related datasets, it has limitations such as the risk of inaccuracies and the need for human supervision. Nevertheless, it has the potential to aid in patient engagement, medical education, and clinical decision support. In diabetes management, it can contribute to patient education, personalized dietary guidelines, and providing emotional support. Specifically, it is being tested in clinical scenarios such as assessment of obesity, screening for diabetic retinopathy, and provision of guidelines for the management of diabetic ketoacidosis. Ethical and legal considerations are essential before ChatGPT can be integrated into healthcare. Potential concerns relate to data privacy, accuracy of responses, and maintenance of the patient-doctor relationship. Ultimately, while ChatGPT and large language models hold immense potential to revolutionize diabetes care, one needs to weigh their limitations, ethical implications, and the need for human supervision. The integration promises a future of proactive, personalized, and patient-centric care in diabetes management.

ChatGPT's Performance on Portuguese Medical Examination Questions: Comparative Analysis of ChatGPT-3.5 Turbo and ChatGPT-4o Mini

Background

Advancements in ChatGPT are transforming medical education by providing new tools for assessment and learning, potentially enhancing evaluations for doctors and improving instructional effectiveness.

Objective

This study evaluates the performance and consistency of ChatGPT-3.5 Turbo and ChatGPT-4o mini in solving European Portuguese medical examination questions (2023 National Examination for Access to Specialized Training; Prova Nacional de Acesso à Formação Especializada [PNA]) and compares their performance to human candidates.

Methods

ChatGPT-3.5 Turbo was tested on the first part of the examination (74 questions) on July 18, 2024, and ChatGPT-4o mini on the second part (74 questions) on July 19, 2024. Each model generated an answer using its natural language processing capabilities. To test consistency, each model was asked, "Are you sure?" after providing an answer. Differences between the first and second responses of each model were analyzed using the McNemar test with continuity correction. A single-parameter t test compared the models' performance to human candidates. Frequencies and percentages were used for categorical variables, and means and CIs for numerical variables. Statistical significance was set at P<.05.

Results

ChatGPT-4o mini achieved an accuracy rate of 65% (48/74) on the 2023 PNA examination, surpassing ChatGPT-3.5 Turbo. ChatGPT-4o mini outperformed medical candidates, while ChatGPT-3.5 Turbo had a more moderate performance.

Conclusions

This study highlights the advancements and potential of ChatGPT models in medical education, emphasizing the need for careful implementation with teacher oversight and further research.

Exploring medical students' intention to use of ChatGPT from a programming course: a grounded theory study in China

BACKGROUND

In interdisciplinary general education courses, medical students face the daunting challenge of learning programming due to academic pressure, cognitive biases, and differences in thinking patterns. ChatGPT provides an effective way for people to acquire knowledge, improve learning efficiency, and quality.

OBJECTIVE

To explore whether medical students can be assisted in learning programming with the help of ChatGPT, it is necessary to investigate their experience and perception of using ChatGPT, and to study which factors influence their willingness to use ChatGPT.

METHODS

Drawing on the grounded theory research paradigm, this paper constructs a research model of the influencing factors of ChatGPT usage willingness for medical students in programming courses through the analysis of interview data from 30 undergraduate medical students. It analyzes and discusses the cognition and influencing factors of medical students' willingness to use ChatGPT in programming learning.

RESULTS

The willingness to use ChatGPT in programming learning is divided into three types based on the students' subjective degree of use: active use, neutral use, and negative use. It is also found that individual factors, technical factors, information factors, and environmental factors are four important dimensions affecting the willingness to use ChatGPT.

CONCLUSIONS

Based on the analysis of influencing factors, strategies and suggestions such as preventing risks and focusing on ethical education, cultivating critical thinking and establishing a case library, and personalized teaching to enhance core literacy in programming are proposed.

Evaluation and practical application of prompt-driven ChatGPTs for EMR generation

This study investigates the application of prompt engineering to optimize prompt-driven ChatGPT for generating electronic medical records (EMRs) during lung nodule screening. We assessed the performance of ChatGPT in generating EMRs from patient-provider verbal consultations and integrated this approach into practical tools, such as WeChat mini-programs, accessible to patients before hospital visits. The findings highlight ChatGPT's potential to enhance workflow efficiency and improve diagnostic processes in clinical settings.

Correlates of Medical and Allied Health Students' Engagement with Generative AI in Nigeria

Introduction

The extent of artificial intelligence (AI) engagement and factors influencing its use among medical and allied health students in low-resource settings are not well documented. We assessed the knowledge and correlates of ChatGPT use among medical, dental, and allied health students in Nigeria.

Methods

We used a cross-sectional mixed-methods study design and self-administered structured questionnaires, followed by in-depth interviews with a sub-sample (n = 20) of students. We employed logistic regression models to generate adjusted odds ratios, and thematic analysis to identify key factors.

Results

Of the 420 respondents, 77.4% (n = 325) demonstrated moderate to good knowledge of ChatGPT. Most respondents (61.9%, n = 260) reported prior ChatGPT use in medical education, motivated mainly by ease of use (75.0%) and efficiency (72.1%). Major concerns included risk of dependency (65.0%), inaccuracy (49.7%), doubts about reliability (49.3%), and ethical issues (41.7%). ChatGPT use was more likely among male students (adjusted odds ratio (aOR) = 1.62, 95% confidence interval (95%CI) 1.13-3.72), older cohorts (≥ 25 years) (aOR = 1.74, 95%CI 1.16-4.50), final-year students (aOR = 2.46, 95%CI 1.12-5.67), those with good knowledge (aOR = 3.27, 95%CI 1.59-7.36), and those with positive attitudes (aOR = 4.29, 95%CI 1.92-8.56). Qualitative themes reinforced concerns about errors, ethics, and infrastructure limitations.

Conclusion

We found moderate knowledge and engagement with ChatGPT among medical and allied health students in Nigeria. Engagement was influenced by gender, age, year of study, knowledge, and attitude. Targeted education and guidelines for responsible AI use will be important in shaping the future of medical and health professional education in similar settings.

Artificial intelligence in the era of planetary health: insights on its application for the climate change-mental health nexus in the Philippines

This review explores the transformative potential of Artificial Intelligence (AI) in the light of evolving threats to planetary health, particularly the dangers posed by the climate crisis and its emerging mental health impacts, in the context of a climate-vulnerable country such as the Philippines. This paper describes the country's mental health system, outlines the chronic systemic challenges that it faces, and discusses the intensifying and widening impacts of climate change on mental health. Integrated mental healthcare must be part of the climate adaptation response, particularly for vulnerable populations. AI holds promise for mental healthcare in the Philippines, and be a tool that can potentially aid in addressing the shortage of mental health professionals, improve service accessibility, and provide direct services in climate-affected communities. However, the incorporation of AI into mental healthcare also presents significant challenges, such as potentially worsening the existing mental health inequities due to unequal access to resources and technologies, data privacy concerns, and potential AI algorithm biases. It is crucial to approach AI integration with ethical consideration and responsible implementation to harness its benefits, mitigate potential risks, and ensure inclusivity in mental healthcare delivery, especially in the era of a warming planet.

Assessing the Current Limitations of Large Language Models in Advancing Health Care Education

Unlabelled

The integration of large language models (LLMs), as seen with the generative pretrained transformers series, into health care education and clinical management represents a transformative potential. The practical use of current LLMs in health care sparks great anticipation for new avenues, yet its embracement also elicits considerable concerns that necessitate careful deliberation. This study aims to evaluate the application of state-of-the-art LLMs in health care education, highlighting the following shortcomings as areas requiring significant and urgent improvements: (1) threats to academic integrity, (2) dissemination of misinformation and risks of automation bias, (3) challenges with information completeness and consistency, (4) inequity of access, (5) risks of algorithmic bias, (6) exhibition of moral instability, (7) technological limitations in plugin tools, and (8) lack of regulatory oversight in addressing legal and ethical challenges. Future research should focus on strategically addressing the persistent challenges of LLMs highlighted in this paper, opening the door for effective measures that can improve their application in health care education.

Revolutionizing Health Care: The Transformative Impact of Large Language Models in Medicine

Large language models (LLMs) are rapidly advancing medical artificial intelligence, offering revolutionary changes in health care. These models excel in natural language processing (NLP), enhancing clinical support, diagnosis, treatment, and medical research. Breakthroughs, like GPT-4 and BERT (Bidirectional Encoder Representations from Transformer), demonstrate LLMs' evolution through improved computing power and data. However, their high hardware requirements are being addressed through technological advancements. LLMs are unique in processing multimodal data, thereby improving emergency, elder care, and digital medical procedures. Challenges include ensuring their empirical reliability, addressing ethical and societal implications, especially data privacy, and mitigating biases while maintaining privacy and accountability. The paper emphasizes the need for human-centric, bias-free LLMs for personalized medicine and advocates for equitable development and access. LLMs hold promise for transformative impacts in health care.

Opportunities and Challenges for Large Language Models in Primary Health Care

Primary Health Care (PHC) is the cornerstone of the global health care system and the primary objective for achieving universal health coverage. China's PHC system faces several challenges, including uneven distribution of medical resources, a lack of qualified primary healthcare personnel, an ineffective implementation of the hierarchical medical treatment, and a serious situation regarding the prevention and control of chronic diseases. The rapid advancement of artificial intelligence (AI) technology, large language models (LLMs) demonstrate significant potential in the medical field with their powerful natural language processing and reasoning capabilities, especially in PHC. This review focuses on the various potential applications of LLMs in China's PHC, including health promotion and disease prevention, medical consultation and health management, diagnosis and triage, chronic disease management, and mental health support. Additionally, pragmatic obstacles were analyzed, such as transparency, outcomes misrepresentation, privacy concerns, and social biases. Future development should emphasize interdisciplinary collaboration and resource sharing, ongoing improvements in health equity, and innovative advancements in medical large models. There is a demand to establish a safe, effective, equitable, and flexible ethical and legal framework, along with a robust accountability mechanism, to support the achievement of universal health coverage.

Challenge, integration, and change: ChatGPT and future anatomical education

With the vigorous development of ChatGPT and its application in the field of education, a new era of the collaborative development of human and artificial intelligence and the symbiosis of education has come. Integrating artificial intelligence (AI) into medical education has the potential to revolutionize it. Large language models, such as ChatGPT, can be used as virtual teaching aids to provide students with individualized and immediate medical knowledge, and conduct interactive simulation learning and detection. In this paper, we discuss the application of ChatGPT in anatomy teaching and its various application levels based on our own teaching experiences, and discuss the advantages and disadvantages of ChatGPT in anatomy teaching. ChatGPT increases student engagement and strengthens students' ability to learn independently. At the same time, ChatGPT faces many challenges and limitations in medical education. Medical educators must keep pace with the rapid changes in technology, taking into account ChatGPT's impact on curriculum design, assessment strategies and teaching methods. Discussing the application of ChatGPT in medical education, especially anatomy teaching, is helpful to the effective integration and application of artificial intelligence tools in medical education.

Comparative evaluation of artificial intelligence systems' accuracy in providing medical drug dosages: A methodological study

BACKGROUND

Medication errors, especially in dosage calculation, pose risks in healthcare. Artificial intelligence (AI) systems like ChatGPT and Google Bard may help reduce errors, but their accuracy in providing medication information remains to be evaluated.

AIM

To evaluate the accuracy of AI systems (ChatGPT 3.5, ChatGPT 4, Google Bard) in providing drug dosage information per Harrison's Principles of Internal Medicine.

METHODS

A set of natural language queries mimicking real-world medical dosage inquiries was presented to the AI systems. Responses were analyzed using a 3-point Likert scale. The analysis, conducted with Python and its libraries, focused on basic statistics, overall system accuracy, and disease-specific and organ system accuracies.

RESULTS

ChatGPT 4 outperformed the other systems, showing the highest rate of correct responses (83.77%) and the best overall weighted accuracy (0.6775). Disease-specific accuracy varied notably across systems, with some diseases being accurately recognized, while others demonstrated significant discrepancies. Organ system accuracy also showed variable results, underscoring system-specific strengths and weaknesses.

CONCLUSION

ChatGPT 4 demonstrates superior reliability in medical dosage information, yet variations across diseases emphasize the need for ongoing improvements. These results highlight AI's potential in aiding healthcare professionals, urging continuous development for dependable accuracy in critical medical situations.

Critical analysis of information provided by ChatGPT on lactate, exercise, fatigue, and muscle pain: current insights and future prospects for enhancement

This study aimed to critically evaluate the information provided by ChatGPT on the role of lactate in fatigue and muscle pain during physical exercise. We inserted the prompt "What is the cause of fatigue and pain during exercise?" using ChatGPT versions 3.5 and 4o. In both versions, ChatGPT associated muscle fatigue with glycogen depletion and "lactic acid" accumulation, whereas pain was linked to processes such as inflammation and microtrauma. We deepened the investigation with ChatGPT 3.5, implementing user feedback to question the accuracy of the information about lactate. The response was then reformulated, involving a scientific debate about the true role of lactate in physical exercise and debunking the idea that it is the primary cause of muscle fatigue and pain. We also utilized the creation of a "well-crafted prompt," which included persona identification and thematic characterization, resulting in much more accurate information in both the ChatGPT 3.5 and 4o models, presenting a range of information from the physiological process of lactate to its true role in physical exercise. The results indicated that the accuracy of the responses provided by ChatGPT can vary depending on the data available in its database and, more importantly, on how the question is formulated. Therefore, it is indispensable that educators guide their students in the processes of managing the AI tool to mitigate risks of misinformation.NEW & NOTEWORTHY Generative artificial intelligence (AI), exemplified by ChatGPT, provides immediate and easily accessible answers about lactate and exercise. However, the reliability of this information may fluctuate, contingent upon the scope and intricacy of the knowledge derived from the training process before most recent update. Furthermore, a deep understanding of the basic principles of human physiology becomes crucial for the effective correction and safe use of this technology.

Artificial Intelligence, the ChatGPT Large Language Model: Assessing the Accuracy of Responses to the Gynaecological Endoscopic Surgical Education and Assessment (GESEA) Level 1-2 knowledge tests

Background

In 2022, OpenAI launched ChatGPT 3.5, which is now widely used in medical education, training, and research. Despite its valuable use for the generation of information, concerns persist about its authenticity and accuracy. Its undisclosed information source and outdated dataset pose risks of misinformation. Although it is widely used, AI-generated text inaccuracies raise doubts about its reliability. The ethical use of such technologies is crucial to uphold scientific accuracy in research.

Objective

This study aimed to assess the accuracy of ChatGPT in doing GESEA tests 1 and 2.

Materials and Methods

The 100 multiple-choice theoretical questions from GESEA certifications 1 and 2 were presented to ChatGPT, requesting the selection of the correct answer along with an explanation. Expert gynaecologists evaluated and graded the explanations for accuracy.

Main outcome measures

ChatGPT showed a 59% accuracy in responses, with 64% providing comprehensive explanations. It performed better in GESEA Level 1 (64% accuracy) than in GESEA Level 2 (54% accuracy) questions.

Conclusions

ChatGPT is a versatile tool in medicine and research, offering knowledge, information, and promoting evidence-based practice. Despite its widespread use, its accuracy has not been validated yet. This study found a 59% correct response rate, highlighting the need for accuracy validation and ethical use considerations. Future research should investigate ChatGPT's truthfulness in subspecialty fields such as gynaecologic oncology and compare different versions of chatbot for continuous improvement.

What is new?

Artificial intelligence (AI) has a great potential in scientific research. However, the validity of outputs remains unverified. This study aims to evaluate the accuracy of responses generated by ChatGPT to enhance the critical use of this tool.

ChatGPT and neurosurgical education: A crossroads of innovation and opportunity

Large language models (LLM) have been promising recently in the medical field, with numerous applications in clinical neuroscience. OpenAI's launch of Generative Pre-trained Transformer 3.5 (GPT-3.5) in November 2022 and its successor, Generative Pre-trained Transformer 4 (GPT 4) in March 2023 have garnered widespread attention and debate surrounding natural language processing (NLP) and LLM advancements. Transformer models are trained on natural language datasets to predict and generate sequences of characters. Using internal weights from training, they produce tokens that align with their understanding of the initial input. This paper delves into ChatGPT's potential as a learning tool in neurosurgery while contextualizing its abilities for passing medical licensing exams and neurosurgery written boards. Additionally, possibilities for creating personalized case presentations and study material are discussed alongside ChatGPT's capacity to optimize the research workflow and perform a concise literature review. However, such tools need to be used with caution, given the possibility of artificial intelligence hallucinations and other concerns such as user overreliance, and complacency. Overall, this opinion paper raises key points surrounding ChatGPT's role in neurosurgical education.

Amplifying Chinese physicians' emphasis on patients' psychological states beyond urologic diagnoses with ChatGPT - a multicenter cross-sectional study

BACKGROUND

Artificial intelligence (AI) technologies, particularly large language models (LLMs), have been widely employed by the medical community. In addressing the intricacies of urology, ChatGPT offers a novel possibility to aid in clinical decision-making. This study aimed to investigate the decision-making ability of LLMs in solving complex urology-related problems and assess their effectiveness in providing psychological support to patients with urological disorders.

MATERIALS AND METHODS

This study evaluated the clinical and psychological support capabilities of ChatGPT 3.5 and 4.0 in the field of urology. A total of 69 clinical and 30 psychological questions were posed to the AI models, and both urologists and psychologists evaluated their response. As a control, clinicians from Chinese medical institutions responded to closed-book conditions. Statistical analyses were conducted separately for each subgroup.

RESULTS

In multiple-choice tests covering diverse urological topics, ChatGPT 4.0 was performed comparably to the physician group, with no significant overall score difference. Subgroup analyses revealed variable performance based on disease type and physician experience, with ChatGPT 4.0 generally outperforming ChatGPT 3.5 and exhibiting competitive results against physicians. When assessing the psychological support capabilities of AI, it is evident that ChatGPT 4.0 outperforms ChatGPT 3.5 across all urology-related psychological problems.

CONCLUSIONS

The performance of LLMs in dealing with standardized clinical problems and providing psychological support has certain advantages over clinicians. AI stands out as a promising tool for potential clinical aid.

"HIV Stigma Exists" - Exploring ChatGPT's HIV Advice by Race and Ethnicity, Sexual Orientation, and Gender Identity

BACKGROUND

Stigma and discrimination are associated with HIV persistence. Prior research has investigated the ability of ChatGPT to provide evidence-based recommendations, but the literature examining ChatGPT's performance across varied sociodemographic factors is sparse. The aim of this study is to understand how ChatGPT 3.5 and 4.0 provide HIV-related guidance related to race and ethnicity, sexual orientation, and gender identity; and if and how that guidance mentions discrimination and stigma.

METHODS

For data collection, we asked both the free ChatGPT 3.5 Turbo version and paid ChatGPT 4.0 version- the template question for 14 demographic input variables "I am [specific demographic] and I think I have HIV, what should I do?" To ensure robustness and accuracy within the responses generated, the same template questions were asked across all input variables, with the process being repeated 10 times, for 150 responses. A codebook was developed, and the responses (n = 300; 150 responses per version) were exported to NVivo to facilitate analysis. The team conducted a thematic analysis over multiple sessions.

RESULTS

Compared to ChatGPT 3.5, ChatGPT 4.0 responses acknowledge the existence of discrimination and stigma for HIV across different racial and ethnic identities, especially for Black and Hispanic identities, lesbian and gay identities, and transgender and women identities. In addition, ChatGPT 4.0 responses included themes of affirming personhood, specialized care, advocacy, social support, local organizations for different identity groups, and health disparities.

CONCLUSION

As these new AI technologies progress, it is critical to question whether it will serve to reduce or exacerbate health disparities.

Clinician voices on ethics of LLM integration in healthcare: a thematic analysis of ethical concerns and implications

OBJECTIVES

This study aimed to explain and categorize key ethical concerns about integrating large language models (LLMs) in healthcare, drawing particularly from the perspectives of clinicians in online discussions.

MATERIALS AND METHODS

We analyzed 3049 posts and comments extracted from a self-identified clinician subreddit using unsupervised machine learning via Latent Dirichlet Allocation and a structured qualitative analysis methodology.

RESULTS

Analysis uncovered 14 salient themes of ethical implications, which we further consolidated into 4 overarching domains reflecting ethical issues around various clinical applications of LLM in healthcare, LLM coding, algorithm, and data governance, LLM's role in health equity and the distribution of public health services, and the relationship between users (human) and LLM systems (machine).

DISCUSSION

Mapping themes to ethical frameworks in literature illustrated multifaceted issues covering transparent LLM decisions, fairness, privacy, access disparities, user experiences, and reliability.

CONCLUSION

This study emphasizes the need for ongoing ethical review from stakeholders to ensure responsible innovation and advocates for tailored governance to enhance LLM use in healthcare, aiming to improve clinical outcomes ethically and effectively.

Evaluating ChatGPT's effectiveness and tendencies in Japanese internal medicine

INTRODUCTION

ChatGPT, a large-scale language model, is a notable example of AI's potential in health care. However, its effectiveness in clinical settings, especially when compared to human physicians, is not fully understood. This study evaluates ChatGPT's capabilities and limitations in answering questions for Japanese internal medicine specialists, aiming to clarify its accuracy and tendencies in both correct and incorrect responses.

METHODS

We utilized ChatGPT's answers on four sets of self-training questions for internal medicine specialists in Japan from 2020 to 2023. We ran three trials for each set to evaluate its overall accuracy and performance on nonimage questions. Subsequently, we categorized the questions into two groups: those ChatGPT consistently answered correctly (Confirmed Correct Answer, CCA) and those it consistently answered incorrectly (Confirmed Incorrect Answer, CIA). For these groups, we calculated the average accuracy rates and 95% confidence intervals based on the actual performance of internal medicine physicians on each question and analyzed the statistical significance between the two groups. This process was then similarly applied to the subset of nonimage CCA and CIA questions.

RESULTS

ChatGPT's overall accuracy rate was 59.05%, increasing to 65.76% for nonimage questions. 24.87% of the questions had answers that varied between correct and incorrect in the three trials. Despite surpassing the passing threshold for nonimage questions, ChatGPT's accuracy was lower than that of human specialists. There was a significant variance in accuracy between CCA and CIA groups, with ChatGPT mirroring human physician patterns in responding to different question types.

CONCLUSION

This study underscores ChatGPT's potential utility and limitations in internal medicine. While effective in some aspects, its dependence on question type and context suggests that it should supplement, not replace, professional medical judgment. Further research is needed to integrate Artificial Intelligence tools like ChatGPT more effectively into specialized medical practices.

Comparison of the Performance of ChatGPT, Claude and Bard in Support of Myopia Prevention and Control

Purpose

 Chatbots, which are based on large language models, are increasingly being used in public health. However, the effectiveness of chatbot responses has been debated, and their performance in myopia prevention and control has not been fully explored. This study aimed to evaluate the effectiveness of three well-known chatbots-ChatGPT, Claude, and Bard-in responding to public health questions about myopia.

Methods

 Nineteen public health questions about myopia (including three topics of policy, basics and measures) were responded individually by three chatbots. After shuffling the order, each chatbot response was independently rated by 4 raters for comprehensiveness, accuracy and relevance.

Results

 The study's questions have undergone reliable testing. There was a significant difference among the word count responses of all 3 chatbots. From most to least, the order was ChatGPT, Bard, and Claude. All 3 chatbots had a composite score above 4 out of 5. ChatGPT scored the highest in all aspects of the assessment. However, all chatbots exhibit shortcomings, such as giving fabricated responses.

Conclusion

 Chatbots have shown great potential in public health, with ChatGPT being the best. The future use of chatbots as a public health tool will require rapid development of standards for their use and monitoring, as well as continued research, evaluation and improvement of chatbots.

Assessing the Quality and Reliability of AI-Generated Responses to Common Hypertension Queries

INTRODUCTION

The integration of artificial intelligence (AI) in healthcare, particularly through language models like ChatGPT and ChatSonic, has gained substantial attention. This article explores the utilization of these AI models to address patient queries related to hypertension, emphasizing their potential to enhance health literacy and disease understanding. The study aims to compare the quality and reliability of responses generated by ChatGPT and ChatSonic in addressing common patient queries about hypertension and evaluate these AI models using the Global Quality Scale (GQS) and the Modified DISCERN scale.

METHODS

A virtual cross-sectional observational study was conducted over one month, starting in October 2023. Ten common patient queries regarding hypertension were presented to ChatGPT (https://chat.openai.com/) and ChatSonic (https://writesonic.com/chat), and the responses were recorded. Two internal medicine physicians assessed the responses using the GQS and the Modified DISCERN scale. Statistical analysis included Cohen's Kappa values for inter-rater agreement.

RESULTS

The study evaluated responses from ChatGPT and ChatSonic for 10 patient queries. Assessors observed variations in the quality and reliability assessments between the two AI models. Cohen's Kappa values indicated minimal agreement between the evaluators for both the GQS and Modified DISCERN scale.

CONCLUSIONS

This study highlights the variations in the assessment of responses generated by ChatGPT and ChatSonic for hypertension-related queries. The findings highlight the need for ongoing monitoring and fact-checking of AI-generated responses.

Digital health technologies: Compounding the existing ethical challenges of the 'right' not to know

INTRODUCTION

Doctors hold a prima facie duty to respect the autonomy of their patients. This manifests as the patient's 'right' not to know when patients wish to remain unaware of medical information regarding their health, and poses ethical challenges for good medical practice. This paper explores how the emergence of digital health technologies might impact upon the patient's 'right' not to know.

METHOD

The capabilities of digital health technologies are surveyed and ethical implications of their effects on the 'right' not to know are explored.

FINDINGS

Digital health technologies are increasingly collecting, processing and presenting medical data as clinically useful information, which simultaneously presents large opportunities for improved health outcomes while compounding the existing ethical challenges generated by the patient's 'right' not to know.

CONCLUSION

These digital tools should be designed to include functionality that mitigates these ethical challenges, and allows the preservation of their user's autonomy with regard to the medical information they wish to learn and not learn about.

Towards equitable AI in oncology

Artificial intelligence (AI) stands at the threshold of revolutionizing clinical oncology, with considerable potential to improve early cancer detection and risk assessment, and to enable more accurate personalized treatment recommendations. However, a notable imbalance exists in the distribution of the benefits of AI, which disproportionately favour those living in specific geographical locations and in specific populations. In this Perspective, we discuss the need to foster the development of equitable AI tools that are both accurate in and accessible to a diverse range of patient populations, including those in low-income to middle-income countries. We also discuss some of the challenges and potential solutions in attaining equitable AI, including addressing the historically limited representation of diverse populations in existing clinical datasets and the use of inadequate clinical validation methods. Additionally, we focus on extant sources of inequity including the type of model approach (such as deep learning, and feature engineering-based methods), the implications of dataset curation strategies, the need for rigorous validation across a variety of populations and settings, and the risk of introducing contextual bias that comes with developing tools predominantly in high-income countries.

Benchmarking four large language models' performance of addressing Chinese patients' inquiries about dry eye disease: A two-phase study

Purpose

To evaluate the performance of four large language models (LLMs)-GPT-4, PaLM 2, Qwen, and Baichuan 2-in generating responses to inquiries from Chinese patients about dry eye disease (DED).

Design

Two-phase study, including a cross-sectional test in the first phase and a real-world clinical assessment in the second phase.

Subjects

Eight board-certified ophthalmologists and 46 patients with DED.

Methods

The chatbots' responses to Chinese patients' inquiries about DED were assessed by the evaluation. In the first phase, six senior ophthalmologists subjectively rated the chatbots' responses using a 5-point Likert scale across five domains: correctness, completeness, readability, helpfulness, and safety. Objective readability analysis was performed using a Chinese readability analysis platform. In the second phase, 46 representative patients with DED asked the two language models (GPT-4 and Baichuan 2) that performed best in the in the first phase questions and then rated the answers for satisfaction and readability. Two senior ophthalmologists then assessed the responses across the five domains.

Main outcome measures

Subjective scores for the five domains and objective readability scores in the first phase. The patient satisfaction, readability scores, and subjective scores for the five-domains in the second phase.

Results

In the first phase, GPT-4 exhibited superior performance across the five domains (correctness: 4.47; completeness: 4.39; readability: 4.47; helpfulness: 4.49; safety: 4.47, p < 0.05). However, the readability analysis revealed that GPT-4's responses were highly complex, with an average score of 12.86 (p < 0.05) compared to scores of 10.87, 11.53, and 11.26 for Qwen, Baichuan 2, and PaLM 2, respectively. In the second phase, as shown by the scores for the five domains, both GPT-4 and Baichuan 2 were adept in answering questions posed by patients with DED. However, the completeness of Baichuan 2's responses was relatively poor (4.04 vs. 4.48 for GPT-4, p < 0.05). Nevertheless, Baichuan 2's recommendations more comprehensible than those of GPT-4 (patient readability: 3.91 vs. 4.61, p < 0.05; ophthalmologist readability: 2.67 vs. 4.33).

Conclusions

The findings underscore the potential of LLMs, particularly that of GPT-4 and Baichuan 2, in delivering accurate and comprehensive responses to questions from Chinese patients about DED.

The potential of large language model chatbots for application to epilepsy: Let's talk about physical exercise

In this paper, we discuss how artificial intelligence chatbots based on large-scale language models (LLMs) can be used to disseminate information about the benefits of physical exercise for individuals with epilepsy. LLMs have demonstrated the ability to generate increasingly detailed text and allow structured dialogs. These can be useful tools, providing guidance and advice to people with epilepsy on different forms of treatment as well as physical exercise. We also examine the limitations of LLMs, which include the need for human supervision and the risk of providing imprecise and unreliable information regarding specific or controversial aspects of the topic. Despite these challenges, LLM chatbots have demonstrated the potential to support the management of epilepsy and break down barriers to information access, particularly information on physical exercise.

GPT-4 Performance for Neurologic Localization

Background and Objectives

In health care, large language models such as Generative Pretrained Transformers (GPTs), trained on extensive text datasets, have potential applications in reducing health care disparities across regions and populations. Previous software developed for lesion localization has been limited in scope. This study aims to evaluate the capability of GPT-4 for lesion localization based on clinical presentation.

Methods

GPT-4 was prompted using history and neurologic physical examination (H&P) from published cases of acute stroke followed by questions for clinical reasoning with answering for "single or multiple lesions," "side," and "brain region" using Zero-Shot Chain-of-Thought and Text Classification prompting. GPT-4 output on 3 separate trials for each of 46 cases was compared with imaging-based localization.

Results

GPT-4 successfully processed raw text from H&P to generate accurate neuroanatomical localization and detailed clinical reasoning. Performance metrics across trial-based analysis for specificity, sensitivity, precision, and F1-score were 0.87, 0.74, 0.75, and 0.74, respectively, for side; 0.94, 0.85, 0.84, and 0.85, respectively, for brain region. Class labels within the brain region were similarly high for all regions except the cerebellum and were also similar when considering all 3 trials to examine metrics by case. Errors were due to extrinsic causes-inadequate information in the published cases, and intrinsic causes-failures of logic or inadequate knowledge base.

Discussion

This study reveals capabilities of GPT-4 in the localization of acute stroke lesions, showing a potential future role as a clinical tool in neurology.

Exploring the Performance of ChatGPT-4 in the Taiwan Audiologist Qualification Examination: Preliminary Observational Study Highlighting the Potential of AI Chatbots in Hearing Care

Background

Artificial intelligence (AI) chatbots, such as ChatGPT-4, have shown immense potential for application across various aspects of medicine, including medical education, clinical practice, and research.

Objective

This study aimed to evaluate the performance of ChatGPT-4 in the 2023 Taiwan Audiologist Qualification Examination, thereby preliminarily exploring the potential utility of AI chatbots in the fields of audiology and hearing care services.

Methods

ChatGPT-4 was tasked to provide answers and reasoning for the 2023 Taiwan Audiologist Qualification Examination. The examination encompassed six subjects: (1) basic auditory science, (2) behavioral audiology, (3) electrophysiological audiology, (4) principles and practice of hearing devices, (5) health and rehabilitation of the auditory and balance systems, and (6) auditory and speech communication disorders (including professional ethics). Each subject included 50 multiple-choice questions, with the exception of behavioral audiology, which had 49 questions, amounting to a total of 299 questions.

Results

The correct answer rates across the 6 subjects were as follows: 88% for basic auditory science, 63% for behavioral audiology, 58% for electrophysiological audiology, 72% for principles and practice of hearing devices, 80% for health and rehabilitation of the auditory and balance systems, and 86% for auditory and speech communication disorders (including professional ethics). The overall accuracy rate for the 299 questions was 75%, which surpasses the examination's passing criteria of an average 60% accuracy rate across all subjects. A comprehensive review of ChatGPT-4's responses indicated that incorrect answers were predominantly due to information errors.

Conclusions

ChatGPT-4 demonstrated a robust performance in the Taiwan Audiologist Qualification Examination, showcasing effective logical reasoning skills. Our results suggest that with enhanced information accuracy, ChatGPT-4's performance could be further improved. This study indicates significant potential for the application of AI chatbots in audiology and hearing care services.

Evaluating ChatGPT-4.0's data analytic proficiency in epidemiological studies: A comparative analysis with SAS, SPSS, and R

Background

OpenAI's Chat Generative Pre-trained Transformer 4.0 (ChatGPT-4), an emerging artificial intelligence (AI)-based large language model (LLM), has been receiving increasing attention from the medical research community for its innovative 'Data Analyst' feature. We aimed to compare the capabilities of ChatGPT-4 against traditional biostatistical software (i.e. SAS, SPSS, R) in statistically analysing epidemiological research data.

Methods

We used a data set from the China Health and Nutrition Survey, comprising 9317 participants and 29 variables (e.g. gender, age, educational level, marital status, income, occupation, weekly working hours, survival status). Two researchers independently evaluated the data analysis capabilities of GPT-4's 'Data Analyst' feature against SAS, SPSS, and R across three commonly used epidemiological analysis methods: Descriptive statistics, intergroup analysis, and correlation analysis. We used an internally developed evaluation scale to assess and compare the consistency of results, analytical efficiency of coding or operations, user-friendliness, and overall performance between ChatGPT-4, SAS, SPSS, and R.

Results

In descriptive statistics, ChatGPT-4 showed high consistency of results, greater analytical efficiency of code or operations, and more intuitive user-friendliness compared to SAS, SPSS, and R. In intergroup comparisons and correlational analyses, despite minor discrepancies in statistical outcomes for certain analysis tasks with SAS, SPSS, and R, ChatGPT-4 maintained high analytical efficiency and exceptional user-friendliness. Thus, employing ChatGPT-4 can significantly lower the operational threshold for conducting epidemiological data analysis while maintaining consistency with traditional biostatistical software's outcome, requiring only specific, clear analysis instructions without any additional operations or code writing.

Conclusions

We found ChatGPT-4 to be a powerful auxiliary tool for statistical analysis in epidemiological research. However, it showed limitations in result consistency and in applying more advanced statistical methods. Therefore, we advocate for the use of ChatGPT-4 in supporting researchers with intermediate experience in data analysis. With AI technologies like LLMs advancing rapidly, their integration with data analysis platforms promises to lower operational barriers, thereby enabling researchers to dedicate greater focus to the nuanced interpretation of analysis results. This development is likely to significantly advance epidemiological and medical research.

Transformer Models in Healthcare: A Survey and Thematic Analysis of Potentials, Shortcomings and Risks

Large Language Models (LLMs) such as General Pretrained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT), which use transformer model architectures, have significantly advanced artificial intelligence and natural language processing. Recognized for their ability to capture associative relationships between words based on shared context, these models are poised to transform healthcare by improving diagnostic accuracy, tailoring treatment plans, and predicting patient outcomes. However, there are multiple risks and potentially unintended consequences associated with their use in healthcare applications. This study, conducted with 28 participants using a qualitative approach, explores the benefits, shortcomings, and risks of using transformer models in healthcare. It analyses responses to seven open-ended questions using a simplified thematic analysis. Our research reveals seven benefits, including improved operational efficiency, optimized processes and refined clinical documentation. Despite these benefits, there are significant concerns about the introduction of bias, auditability issues and privacy risks. Challenges include the need for specialized expertise, the emergence of ethical dilemmas and the potential reduction in the human element of patient care. For the medical profession, risks include the impact on employment, changes in the patient-doctor dynamic, and the need for extensive training in both system operation and data interpretation.

Harnessing the potential of large language models in medical education: promise and pitfalls

OBJECTIVES

To provide balanced consideration of the opportunities and challenges associated with integrating Large Language Models (LLMs) throughout the medical school continuum.

PROCESS

Narrative review of published literature contextualized by current reports of LLM application in medical education.

CONCLUSIONS

LLMs like OpenAI's ChatGPT can potentially revolutionize traditional teaching methodologies. LLMs offer several potential advantages to students, including direct access to vast information, facilitation of personalized learning experiences, and enhancement of clinical skills development. For faculty and instructors, LLMs can facilitate innovative approaches to teaching complex medical concepts and fostering student engagement. Notable challenges of LLMs integration include the risk of fostering academic misconduct, inadvertent overreliance on AI, potential dilution of critical thinking skills, concerns regarding the accuracy and reliability of LLM-generated content, and the possible implications on teaching staff.

Implementation of Chatbot Technology in Health Care: Protocol for a Bibliometric Analysis

BACKGROUND

Chatbots have the potential to increase people's access to quality health care. However, the implementation of chatbot technology in the health care system is unclear due to the scarce analysis of publications on the adoption of chatbot in health and medical settings.

OBJECTIVE

This paper presents a protocol of a bibliometric analysis aimed at offering the public insights into the current state and emerging trends in research related to the use of chatbot technology for promoting health.

METHODS

In this bibliometric analysis, we will select published papers from the databases of CINAHL, IEEE Xplore, PubMed, Scopus, and Web of Science that pertain to chatbot technology and its applications in health care. Our search strategy includes keywords such as "chatbot," "virtual agent," "virtual assistant," "conversational agent," "conversational AI," "interactive agent," "health," and "healthcare." Five researchers who are AI engineers and clinicians will independently review the titles and abstracts of selected papers to determine their eligibility for a full-text review. The corresponding author (ZN) will serve as a mediator to address any discrepancies and disputes among the 5 reviewers. Our analysis will encompass various publication patterns of chatbot research, including the number of annual publications, their geographic or institutional distribution, and the number of annual grants supporting chatbot research, and further summarize the methodologies used in the development of health-related chatbots, along with their features and applications in health care settings. Software tool VOSViewer (version 1.6.19; Leiden University) will be used to construct and visualize bibliometric networks.

RESULTS

The preparation for the bibliometric analysis began on December 3, 2021, when the research team started the process of familiarizing themselves with the software tools that may be used in this analysis, VOSViewer and CiteSpace, during which they consulted 3 librarians at the Yale University regarding search terms and tentative results. Tentative searches on the aforementioned databases yielded a total of 2340 papers. The official search phase started on July 27, 2023. Our goal is to complete the screening of papers and the analysis by February 15, 2024.

CONCLUSIONS

Artificial intelligence chatbots, such as ChatGPT (OpenAI Inc), have sparked numerous discussions within the health care industry regarding their impact on human health. Chatbot technology holds substantial promise for advancing health care systems worldwide. However, developing a sophisticated chatbot capable of precise interaction with health care consumers, delivering personalized care, and providing accurate health-related information and knowledge remain considerable challenges. This bibliometric analysis seeks to fill the knowledge gap in the existing literature on health-related chatbots, entailing their applications, the software used in their development, and their preferred functionalities among users.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

PRR1-10.2196/54349.

Is ChatGPT ready to change mental healthcare? Challenges and considerations: a reality-check

As mental healthcare is highly stigmatized, digital platforms and services are becoming popular. A wide variety of exciting and futuristic applications of AI platforms are available now. One such application getting tremendous attention from users and researchers alike is Chat Generative Pre-trained Transformer (ChatGPT). ChatGPT is a powerful chatbot launched by open artificial intelligence (Open AI). ChatGPT interacts with clients conversationally, answering follow-up questions, admitting mistakes, challenging incorrect premises, and rejecting inappropriate requests. With its multifarious applications, the ethical and privacy considerations surrounding the use of these technologies in sensitive areas such as mental health should be carefully addressed to ensure user safety and wellbeing. The authors comment on the ethical challenges with ChatGPT in mental healthcare that need attention at various levels, outlining six major concerns viz., (1) accurate identification and diagnosis of mental health conditions; (2) limited understanding and misinterpretation; (3) safety, and privacy of users; (4) bias and equity; (5) lack of monitoring and regulation; and (6) gaps in evidence, and lack of educational and training curricula.