A scoping review of ChatGPT's role in healthcare education and research

"Having providers who are trained and have empathy is life-saving": Improving primary care communication through thematic analysis with ChatGPT and human expertise

In the rapidly evolving field of healthcare research, Artificial Intelligence (AI) and conversational models like ChatGPT (Conversational Generative Pre-trained Transformer) offer promising tools for data analysis. The aim of this study was to: 1) apply ChatGPT methodology alongside human coding to analyze qualitative health services feedback, and 2) examine healthcare experiences among lesbian, gay, bisexual, transgender, and queer (LGBTQ+) patients (N = 41) to inform future intervention. The hybrid approach facilitated the identification of themes related to affirming care practices, provider education, communicative challenges and successes, and environmental cues. While ChatGPT accelerated the coding process, human oversight remained crucial for ensuring data integrity and context accuracy. This hybrid method promises significant improvements in analyzing patient feedback, providing actionable insights that could enhance patient-provider interactions and care for diverse populations. Innovation: This study is the first to combine ChatGPT with human coding for rapid thematic analysis of LGBTQ+ patient primary care experiences.

Attitude and utilization of ChatGPT among registered nurses: A cross-sectional study

AIM

This study explores the influencing factors of attitudes and behaviors toward use of ChatGPT based on the Technology Acceptance Model among registered nurses in Taiwan.

BACKGROUND

The complexity of medical services and nursing shortages increases workloads. ChatGPT swiftly answers medical questions, provides clinical guidelines, and assists with patient information management, thereby improving nursing efficiency.

INTRODUCTION

To facilitate the development of effective ChatGPT training programs, it is essential to examine registered nurses' attitudes toward and utilization of ChatGPT across diverse workplace settings.

METHODS

An anonymous online survey was used to collect data from over 1000 registered nurses recruited through social media platforms between November 2023 and January 2024. Descriptive statistics and multiple linear regression analyses were conducted for data analysis.

RESULTS

Among respondents, some were unfamiliar with ChatGPT, while others had used it before, with higher usage among males, higher-educated individuals, experienced nurses, and supervisors. Gender and work settings influenced perceived risks, and those familiar with ChatGPT recognized its social impact. Perceived risk and usefulness significantly influenced its adoption.

DISCUSSION

Nurse attitudes to ChatGPT vary based on gender, education, experience, and role. Positive perceptions emphasize its usefulness, while risk concerns affect adoption. The insignificant role of perceived ease of use highlights ChatGPT's user-friendly nature.

CONCLUSION

Over half of the surveyed nurses had used or were familiar with ChatGPT and showed positive attitudes toward its use. Establishing rigorous guidelines to enhance their interaction with ChatGPT is crucial for future training.

IMPLICATIONS FOR NURSING AND HEALTH POLICY

Nurse managers should understand registered nurses' attitudes toward ChatGPT and integrate it into in-service education with tailored support and training, including appropriate prompt formulation and advanced decision-making, to prevent misuse.

Development of Evidence-Based Guidelines for the Integration of Generative AI in University Education Through a Multidisciplinary, Consensus-Based Approach

INTRODUCTION

The introduction highlights the transformative impact of generative artificial intelligence (GenAI) on higher education (HE), emphasising its potential to enhance student learning and instructor efficiency while also addressing significant challenges such as accuracy, privacy, and ethical concerns. By exploring the benefits and risks of AI integration, the introduction underscores the urgent need for evidence-based, inclusive, and adaptable frameworks to guide universities in leveraging GenAI responsibly and effectively in academic environments.

AIMS

This paper presents a comprehensive process for developing cross-disciplinary and consensus-based guidelines, based on the latest evidence for the integration of GenAI at European University Cyprus (EUC). In response to the rapid adoption of AI tools such as LLMs in HE, a task group at EUC created a structured framework to guide the ethical and effective use of GenAI in academia, one that was intended to be flexible enough to incorporate new developments and not infringe on instructors' academic freedoms, while also addressing ethical and practical concerns.

RESULTS

The framework development was informed by extensive literature reviews and consultations. Key pillars of the framework include: addressing the risks and opportunities presented by GenAI; promoting transparent communication; ensuring responsible use by students and educators; safeguarding academic integrity. The guidelines emphasise the balance between, on the one hand, leveraging AI to enhance educational experiences, and, on the other maintaining critical thinking and originality. The framework also includes practical recommendations for AI usage, classroom integration, and policy formulation, ensuring that AI augments rather than replaces human judgement in educational settings.

CONCLUSIONS

The iterative development process, including the use of GenAI tools for refining the guidelines, illustrates a hands-on approach to AI adoption in HE, and the resulting guidelines may serve as a model for other higher education institutions (HEIs) aiming to integrate AI tools while upholding educational quality and ethical standards.

Evaluating the application of ChatGPT in China's residency training education: An exploratory study

OBJECTIVE

The purpose of this study was to assess the utility of information generated by ChatGPT for residency education in China.

METHODS

We designed a three-step survey to evaluate the performance of ChatGPT in China's residency training education including residency final examination questions, patient cases, and resident satisfaction scores. First, 204 questions from the residency final exam were input into ChatGPT's interface to obtain the percentage of correct answers. Next, ChatGPT was asked to generate 20 clinical cases, which were subsequently evaluated by three instructors using a pre-designed Likert scale with 5 points. The quality of the cases was assessed based on criteria including clarity, relevance, logicality, credibility, and comprehensiveness. Finally, interaction sessions between 31 third-year residents and ChatGPT were conducted. Residents' perceptions of ChatGPT's feedback were assessed using a Likert scale, focusing on aspects such as ease of use, accuracy and completeness of responses, and its effectiveness in enhancing understanding of medical knowledge.

RESULTS

Our results showed ChatGPT-3.5 correctly answered 45.1% of exam questions. In the virtual patient cases, ChatGPT received mean ratings of 4.57 ± 0.50, 4.68 ± 0.47, 4.77 ± 0.46, 4.60 ± 0.53, and 3.95 ± 0.59 points for clarity, relevance, logicality, credibility, and comprehensiveness from clinical instructors, respectively. Among training residents, ChatGPT scored 4.48 ± 0.70, 4.00 ± 0.82 and 4.61 ± 0.50 points for ease of use, accuracy and completeness, and usefulness, respectively.

CONCLUSION

Our findings demonstrate ChatGPT's immense potential for personalized Chinese medical education.

Quantum leap in medical mentorship: exploring ChatGPT's transition from textbooks to terabytes

ChatGPT, an advanced AI language model, presents a transformative opportunity in several fields including the medical education. This article examines the integration of ChatGPT into healthcare learning environments, exploring its potential to revolutionize knowledge acquisition, personalize education, support curriculum development, and enhance clinical reasoning. The AI's ability to swiftly access and synthesize medical information across various specialties offers significant value to students and professionals alike. It provides rapid answers to queries on medical theories, treatment guidelines, and diagnostic methods, potentially accelerating the learning curve. The paper emphasizes the necessity of verifying ChatGPT's outputs against authoritative medical sources. A key advantage highlighted is the AI's capacity to tailor learning experiences by assessing individual needs, accommodating diverse learning styles, and offering personalized feedback. The article also considers ChatGPT's role in shaping curricula and assessment techniques, suggesting that educators may need to adapt their methods to incorporate AI-driven learning tools. Additionally, it explores how ChatGPT could bolster clinical problem-solving through AI-powered simulations, fostering critical thinking and diagnostic acumen among students. While recognizing ChatGPT's transformative potential in medical education, the article stresses the importance of thoughtful implementation, continuous validation, and the establishment of protocols to ensure its responsible and effective application in healthcare education settings.

Knowledge and use, perceptions of benefits and limitations of artificial intelligence chatbots among Italian physiotherapy students: a cross-sectional national study

BACKGROUND

Artificial Intelligence (AI) Chatbots (e.g., ChatGPT, Microsoft Bing, and Google Bard) can emulate human interaction and may support physiotherapy education. Despite growing interest, physiotherapy students' perspectives remain unexplored. This study investigated Italian physiotherapy students' knowledge, use, and perception of the benefits and limitations of AI Chatbots.

METHODS

A cross-sectional study was conducted through Survey Monkey from February to June 2024. One thousand five hundred and thirty-one physiotherapy students from 10 universities were involved. The survey consisted of 23 questions investigating: (a) respondent characteristics, (b) AI Chatbot knowledge and use, (c) perceived benefits, and (d) limitations. Multiple-choice and Likert-scale-based questions were adopted. Factors associated with knowledge, use, and perceptions of AI were explored using logistic regression models.

RESULTS

Of 589 students (38%) that completed the survey, most were male (n = 317; 53.8%) with a mean age of 22 years (SD = 3.88). Nearly all (n = 561; 95.3%) had heard of AI Chatbots, but 53.7% (n = 316) never used these tools for academic purposes. Among users, learning support was the most common purpose (n = 187; 31.8%), while only 9.9% (n = 58) declared Chatbot use during internships. Students agreed that Chatbots have limitations in performing complex tasks and may generate inaccurate results (median = 3 out of 4). However, they neither agreed nor disagreed about Chatbots' impact on academic performance, emotional intelligence, bias, and fairness (median = 2 out of 4). The students agreed to identify the risk of misinformation as a primary barrier (median = 3 out of 4). In contrast, they neither agreed nor disagreed on content validity, plagiarism, privacy, and impacts on critical thinking and creativity (median = 2 out of 4). Young students had 11% more odds of being familiar with Chatbots than older students (OR = 0.89; 95%CI 0.84-0.95; p = < 0.01), whereas female students had 39% lesser odds than males to have used Chatbots for academic purposes (OR = 0.61; 95%CI 0.44-0.85; p = < 0.01).

CONCLUSIONS

While most students recognize the potential of AI Chatbots, they express caution about their use in academia. Targeted training for students and faculty, supported by institutional and national guidelines, could guarantee a responsible integration of these technologies into physiotherapy education.

TRIAL REGISTRATION

Not applicable.

[What is the potential of ChatGPT for qualified patient information? : Attempt of a structured analysis on the basis of a survey regarding complementary and alternative medicine (CAM) in rheumatology]

INTRODUCTION

The chatbot ChatGPT represents a milestone in the interaction between humans and large databases that are accessible via the internet. It facilitates the answering of complex questions by enabling a communication in everyday language. Therefore, it is a potential source of information for those who are affected by rheumatic diseases. The aim of our investigation was to find out whether ChatGPT (version 3.5) is capable of giving qualified answers regarding the application of specific methods of complementary and alternative medicine (CAM) in three rheumatic diseases: rheumatoid arthritis (RA), systemic lupus erythematosus (SLE) and granulomatosis with polyangiitis (GPA). In addition, it was investigated how the answers of the chatbot were influenced by the wording of the question.

METHODS

The questioning of ChatGPT was performed in three parts. Part A consisted of an open question regarding the best way of treatment of the respective disease. In part B, the questions were directed towards possible indications for the application of CAM in general in one of the three disorders. In part C, the chatbot was asked for specific recommendations regarding one of three CAM methods: homeopathy, ayurvedic medicine and herbal medicine. Questions in parts B and C were expressed in two modifications: firstly, it was asked whether the specific CAM was applicable at all in certain rheumatic diseases. The second question asked which procedure of the respective CAM method worked best in the specific disease. The validity of the answers was checked by using the ChatGPT reliability score, a Likert scale ranging from 1 (lowest validity) to 7 (highest validity).

RESULTS

The answers to the open questions of part A had the highest validity. In parts B and C, ChatGPT suggested a variety of CAM applications that lacked scientific evidence. The validity of the answers depended on the wording of the questions. If the question suggested the inclination to apply a certain CAM, the answers often lacked the information of missing evidence and were graded with lower score values.

CONCLUSION

The answers of ChatGPT (version 3.5) regarding the applicability of CAM in selected rheumatic diseases are not convincingly based on scientific evidence. In addition, the wording of the questions affects the validity of the information. Currently, an uncritical application of ChatGPT as an instrument for patient information cannot be recommended.

Perspectives and experiences of health sciences academics regarding ChatGPT: A qualitative study

PURPOSE

This study aimed to explore the perspectives and experiences of healthcare academics regarding the impact of ChatGPT, an artificial intelligence (AI)-supported language model, on education and research.

SAMPLE AND METHODS

This qualitative study employed a phenomenological analysis approach. The study sample consisted of nine academics from the Faculty of Health Sciences at a university in Türkiye, selected through purposive sampling method. Data were collected through semi-structured interviews, coded using the MAXQDA software, and analyzed using content analysis.

RESULTS

The participants highlighted that while ChatGPT offers rapid access to information, it occasionally fails to provide current and accurate data. They also noted that the students' misuse of ChatGPT for assignments and exams has a negative effect on their critical thinking and information retrieval skills. The academics reported that there is a need for expert oversight and verification of the data generated by ChatGPT.

CONCLUSION

While ChatGPT offers significant benefits such as enhanced efficiency in academic research and education, it also presents challenges, including accuracy and ethical concerns. Institutions should integrate ChatGPT with clear guidelines to maximize its benefits while maintaining academic integrity. Future studies should explore the long-term impacts of AI tools, such as ChatGPT, on educational outcomes and their application across various disciplines.

Policies on artificial intelligence chatbots among academic publishers: a cross-sectional audit

BACKGROUND

Artificial intelligence (AI) chatbots are novel computer programs that can generate text or content in a natural language format. Academic publishers are adapting to the transformative role of AI chatbots in producing or facilitating scientific research. This study aimed to examine the policies established by scientific, technical, and medical academic publishers for defining and regulating the authors' responsible use of AI chatbots.

METHODS

This study performed a cross-sectional audit on the publicly available policies of 162 academic publishers, indexed as members of the International Association of the Scientific, Technical, and Medical Publishers (STM). Data extraction of publicly available policies on the webpages of all STM academic publishers was performed independently, in duplicate, with content analysis reviewed by a third contributor (September 2023-December 2023). Data was categorized into policy elements, such as 'proofreading' and 'image generation'. Counts and percentages of 'yes' (i.e., permitted), 'no', and 'no available information' (NAI) were established for each policy element.

RESULTS

A total of 56/162 (34.6%) STM academic publishers had a publicly available policy guiding the authors' use of AI chatbots. No policy allowed authorship for AI chatbots (or other AI tool). Most (49/56 or 87.5%) required specific disclosure of AI chatbot use. Four policies/publishers placed a complete ban on the use of AI chatbots by authors.

CONCLUSIONS

Only a third of STM academic publishers had publicly available policies as of December 2023. A re-examination of all STM members in 12-18 months may uncover evolving approaches toward AI chatbot use with more academic publishers having a policy.

Correlates of Medical and Allied Health Students' Engagement with Generative AI in Nigeria

Introduction

The extent of artificial intelligence (AI) engagement and factors influencing its use among medical and allied health students in low-resource settings are not well documented. We assessed the knowledge and correlates of ChatGPT use among medical, dental, and allied health students in Nigeria.

Methods

We used a cross-sectional mixed-methods study design and self-administered structured questionnaires, followed by in-depth interviews with a sub-sample (n = 20) of students. We employed logistic regression models to generate adjusted odds ratios, and thematic analysis to identify key factors.

Results

Of the 420 respondents, 77.4% (n = 325) demonstrated moderate to good knowledge of ChatGPT. Most respondents (61.9%, n = 260) reported prior ChatGPT use in medical education, motivated mainly by ease of use (75.0%) and efficiency (72.1%). Major concerns included risk of dependency (65.0%), inaccuracy (49.7%), doubts about reliability (49.3%), and ethical issues (41.7%). ChatGPT use was more likely among male students (adjusted odds ratio (aOR) = 1.62, 95% confidence interval (95%CI) 1.13-3.72), older cohorts (≥ 25 years) (aOR = 1.74, 95%CI 1.16-4.50), final-year students (aOR = 2.46, 95%CI 1.12-5.67), those with good knowledge (aOR = 3.27, 95%CI 1.59-7.36), and those with positive attitudes (aOR = 4.29, 95%CI 1.92-8.56). Qualitative themes reinforced concerns about errors, ethics, and infrastructure limitations.

Conclusion

We found moderate knowledge and engagement with ChatGPT among medical and allied health students in Nigeria. Engagement was influenced by gender, age, year of study, knowledge, and attitude. Targeted education and guidelines for responsible AI use will be important in shaping the future of medical and health professional education in similar settings.

Facilitators and Barriers of Large Language Model Adoption Among Nursing Students: A Qualitative Descriptive Study

AIM

To explore nursing students' perceptions and experiences of using large language models and identify the facilitators and barriers by applying the Theory of Planned Behaviour.

DESIGN

A qualitative descriptive design.

METHOD

Between January and June 2024, we conducted individual semi-structured online interviews with 24 nursing students from 13 medical universities across China. Participants were recruited using purposive and snowball sampling methods. Interviews were conducted in Mandarin. Data were analysed through directed content analysis.

RESULTS

Analysis revealed 10 themes according to 3 constructs of the Theory of Planned Behaviour: (a) attitude: perceived value and expectations were facilitators, while perceived caution posed barriers; (b) subjective norm: media effects and role model effectiveness were described as facilitators, whereas organisational pressure exerted by medical universities, research institutions and hospitals acted as a barrier to usage; (c) perceived behavioural control: the design of models and free access were strong incentives for students to use, while geographic access restrictions and digital literacy deficiencies were key factors hindering adoption.

CONCLUSION

This study explored nursing students' attitudes, subjective norms and perceived behavioural control regarding the use of large language models. The findings provided valuable insights into the factors that hindered or facilitated nursing students' adoption.

IMPLICATIONS FOR THE PROFESSION

Through the lens of this study, we have enhanced knowledge of the journey of nursing students using large language models, which contributes to the implementation and management of these tools in nursing education.

IMPACT

There is a gap in the literature regarding nursing students' views and perceptions of large language models and the factors that influence their usage, which this study addresses. These findings could provide evidence-based support for nurse educators to formulate management strategies and guidelines.

REPORTING METHOD

Reporting adheres to the consolidated criteria for reporting qualitative research (COREQ) checklist.

PUBLIC CONTRIBUTION

No patient or public contribution.

Large Language Models in Worldwide Medical Exams: Platform Development and Comprehensive Analysis

BACKGROUND

Large language models (LLMs) are increasingly integrated into medical education, with transformative potential for learning and assessment. However, their performance across diverse medical exams globally has remained underexplored.

OBJECTIVE

This study aims to introduce MedExamLLM, a comprehensive platform designed to systematically evaluate the performance of LLMs on medical exams worldwide. Specifically, the platform seeks to (1) compile and curate performance data for diverse LLMs on worldwide medical exams; (2) analyze trends and disparities in LLM capabilities across geographic regions, languages, and contexts; and (3) provide a resource for researchers, educators, and developers to explore and advance the integration of artificial intelligence in medical education.

METHODS

A systematic search was conducted on April 25, 2024, in the PubMed database to identify relevant publications. Inclusion criteria encompassed peer-reviewed, English-language, original research articles that evaluated at least one LLM on medical exams. Exclusion criteria included review articles, non-English publications, preprints, and studies without relevant data on LLM performance. The screening process for candidate publications was independently conducted by 2 researchers to ensure accuracy and reliability. Data, including exam information, data process information, model performance, data availability, and references, were manually curated, standardized, and organized. These curated data were integrated into the MedExamLLM platform, enabling its functionality to visualize and analyze LLM performance across geographic, linguistic, and exam characteristics. The web platform was developed with a focus on accessibility, interactivity, and scalability to support continuous data updates and user engagement.

RESULTS

A total of 193 articles were included for final analysis. MedExamLLM comprised information for 16 LLMs on 198 medical exams conducted in 28 countries across 15 languages from the year 2009 to the year 2023. The United States accounted for the highest number of medical exams and related publications, with English being the dominant language used in these exams. The Generative Pretrained Transformer (GPT) series models, especially GPT-4, demonstrated superior performance, achieving pass rates significantly higher than other LLMs. The analysis revealed significant variability in the capabilities of LLMs across different geographic and linguistic contexts.

CONCLUSIONS

MedExamLLM is an open-source, freely accessible, and publicly available online platform providing comprehensive performance evaluation information and evidence knowledge about LLMs on medical exams around the world. The MedExamLLM platform serves as a valuable resource for educators, researchers, and developers in the fields of clinical medicine and artificial intelligence. By synthesizing evidence on LLM capabilities, the platform provides valuable insights to support the integration of artificial intelligence into medical education. Limitations include potential biases in the data source and the exclusion of non-English literature. Future research should address these gaps and explore methods to enhance LLM performance in diverse contexts.

Current trends and future implications in the utilization of ChatGPT in nursing: A rapid review

Background

The past decade has witnessed a surge in the development of artificial intelligence (AI)-based technology systems for healthcare. Launched in November 2022, ChatGPT (Generative Pre-trained Transformer), an AI-based Chatbot, is being utilized in nursing education, research and practice. However, little is known about its pattern of usage, which prompted this study.

Objective

To provide a concise overview of the existing literature on the application of ChatGPT in nursing education, practice and research.

Methods

A rapid review based on the Cochrane methodology was applied to synthesize existing literature. We conducted systematic searches in several databases, including CINAHL, Ovid Medline, Embase, Web of Science, Scopus, Education Search Complete, ERIC, and Cochrane CENTRAL, to ensure no publications were missed. All types of primary and secondary research studies, including qualitative, quantitative, mixed methods, and literature reviews published in the English language focused on the use of ChatGPT in nursing education, research, and practice, were included. Dissertations or theses, conference proceedings, government and other organizational reports, white papers, discussion papers, opinion pieces, editorials, commentaries, and published review protocols were excluded. Studies involving other healthcare professionals and/or students without including nursing participants were excluded. Studies exploring other language models without comparison to ChatGPT and those examining the technical specifications of ChatGPT were excluded. Data screening was completed in two stages: titles and abstract and full-text review, followed by data extraction and quality appraisal. Descriptive analysis and narrative synthesis were applied to summarize and categorize the findings.

Results

Seventeen studies were included: 15 (88.2 %) focused on nursing education and one each on nursing practice and research. Of the 17 included studies, 5 (29.4 %) were evaluation studies, 3 (17.6 %) were narrative reviews, 3 (17.6 %) were cross-sectional studies, 2 (11.8 %) were descriptive studies, and 1 (5.9 %) was a randomized controlled trial, quasi-experimental study, case study, and qualitative study, respectively.

Conclusion

This study has provided a snapshot of ChatGPT usage in nursing education, research, and practice. Although evidence is inconclusive, integration of ChatGPT should consider addressing ethical concerns and ongoing education on ChatGPT usage. Further research, specifically interventional studies, is recommended to ascertain and track the impact of ChatGPT in different contexts.

Using ChatGPT in Nursing: Scoping Review of Current Opinions

Background

Since the release of ChatGPT in November 2022, this emerging technology has garnered a lot of attention in various fields, and nursing is no exception. However, to date, no study has comprehensively summarized the status and opinions of using ChatGPT across different nursing fields.

Objective

We aim to synthesize the status and opinions of using ChatGPT according to different nursing fields, as well as assess ChatGPT's strengths, weaknesses, and the potential impacts it may cause.

Methods

This scoping review was conducted following the framework of Arksey and O'Malley and guided by the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews). A comprehensive literature research was conducted in 4 web-based databases (PubMed, Embase, Web of Science, and CINHAL) to identify studies reporting the opinions of using ChatGPT in nursing fields from 2022 to September 3, 2023. The references of the included studies were screened manually to further identify relevant studies. Two authors conducted studies screening, eligibility assessments, and data extraction independently.

Results

A total of 30 studies were included. The United States (7 studies), Canada (5 studies), and China (4 studies) were countries with the most publications. In terms of fields of concern, studies mainly focused on "ChatGPT and nursing education" (20 studies), "ChatGPT and nursing practice" (10 studies), and "ChatGPT and nursing research, writing, and examination" (6 studies). Six studies addressed the use of ChatGPT in multiple nursing fields.

Conclusions

As an emerging artificial intelligence technology, ChatGPT has great potential to revolutionize nursing education, nursing practice, and nursing research. However, researchers, institutions, and administrations still need to critically examine its accuracy, safety, and privacy, as well as academic misconduct and potential ethical issues that it may lead to before applying ChatGPT to practice.

A systematic review of large language models and their implications in medical education

INTRODUCTION

In the past year, the use of large language models (LLMs) has generated significant interest and excitement because of their potential to revolutionise various fields, including medical education for aspiring physicians. Although medical students undergo a demanding educational process to become competent health care professionals, the emergence of LLMs presents a promising solution to challenges like information overload, time constraints and pressure on clinical educators. However, integrating LLMs into medical education raises critical concerns and challenges for educators, professionals and students. This systematic review aims to explore LLM applications in medical education, specifically their impact on medical students' learning experiences.

METHODS

A systematic search was performed in PubMed, Web of Science and Embase for articles discussing the applications of LLMs in medical education using selected keywords related to LLMs and medical education, from the time of ChatGPT's debut until February 2024. Only articles available in full text or English were reviewed. The credibility of each study was critically appraised by two independent reviewers.

RESULTS

The systematic review identified 166 studies, of which 40 were found by review to be relevant to the study. Among the 40 relevant studies, key themes included LLM capabilities, benefits such as personalised learning and challenges regarding content accuracy. Importantly, 42.5% of these studies specifically evaluated LLMs in a novel way, including ChatGPT, in contexts such as medical exams and clinical/biomedical information, highlighting their potential in replicating human-level performance in medical knowledge. The remaining studies broadly discussed the prospective role of LLMs in medical education, reflecting a keen interest in their future potential despite current constraints.

CONCLUSIONS

The responsible implementation of LLMs in medical education offers a promising opportunity to enhance learning experiences. However, ensuring information accuracy, emphasising skill-building and maintaining ethical safeguards are crucial. Continuous critical evaluation and interdisciplinary collaboration are essential for the appropriate integration of LLMs in medical education.

Can ChatGPT-4 Diagnose and Treat Like an Orthopaedic Surgeon? Testing Clinical Decision Making and Diagnostic Ability in Soft-Tissue Pathologies of the Foot and Ankle

INTRODUCTION

ChatGPT-4, a chatbot with an ability to carry human-like conversation, has attracted attention after demonstrating aptitude to pass professional licensure examinations. The purpose of this study was to explore the diagnostic and decision-making capacities of ChatGPT-4 in clinical management specifically assessing for accuracy in the identification and treatment of soft-tissue foot and ankle pathologies.

METHODS

This study presented eight soft-tissue-related foot and ankle cases to ChatGPT-4, with each case assessed by three fellowship-trained foot and ankle orthopaedic surgeons. The evaluation system included five criteria within a Likert scale, scoring from 5 (lowest) to 25 (highest possible).

RESULTS

The average sum score of all cases was 22.0. The Morton neuroma case received the highest score (24.7), and the peroneal tendon tear case received the lowest score (16.3). Subgroup analyses of each of the 5 criterion using showed no notable differences in surgeon grading. Criteria 3 (provide alternative treatments) and 4 (provide comprehensive information) were graded markedly lower than criteria 1 (diagnose), 2 (treat), and 5 (provide accurate information) (for both criteria 3 and 4: P = 0.007; P = 0.032; P < 0.0001). Criteria 5 was graded markedly higher than criteria 2, 3, and 4 (P = 0.02; P < 0.0001; P < 0.0001).

CONCLUSION

This study demonstrates that ChatGPT-4 effectively diagnosed and provided reliable treatment options for most soft-tissue foot and ankle cases presented, noting consistency among surgeon evaluators. Individual criterion assessment revealed that ChatGPT-4 was most effective in diagnosing and suggesting appropriate treatment, but limitations were seen in the chatbot's ability to provide comprehensive information and alternative treatment options. In addition, the chatbot successfully did not suggest fabricated treatment options, a common concern in prior literature. This resource could be useful for clinicians seeking reliable patient education materials without the fear of inconsistencies, although comprehensive information beyond treatment may be limited.

The potential of ChatGPT in medicine: an example analysis of nephrology specialty exams in Poland

Background

In November 2022, OpenAI released a chatbot named ChatGPT, a product capable of processing natural language to create human-like conversational dialogue. It has generated a lot of interest, including from the scientific community and the medical science community. Recent publications have shown that ChatGPT can correctly answer questions from medical exams such as the United States Medical Licensing Examination and other specialty exams. To date, there have been no studies in which ChatGPT has been tested on specialty questions in the field of nephrology anywhere in the world.

Methods

Using the ChatGPT-3.5 and -4.0 algorithms in this comparative cross-sectional study, we analysed 1560 single-answer questions from the national specialty exam in nephrology from 2017 to 2023 that were available in the Polish Medical Examination Center's question database along with answer keys.

Results

Of the 1556 questions posed to ChatGPT-4.0, correct answers were obtained with an accuracy of 69.84%, compared with ChatGPT-3.5 (45.70%, P = .0001) and with the top results of medical doctors (85.73%, P = .0001). Of the 13 tests, ChatGPT-4.0 exceeded the required ≥60% pass rate in 11 tests passed, and scored higher than the average of the human exam results.

Conclusion

ChatGPT-3.5 was not spectacularly successful in nephrology exams. The ChatGPT-4.0 algorithm was able to pass most of the analysed nephrology specialty exams. New generations of ChatGPT achieve similar results to humans. The best results of humans are better than those of ChatGPT-4.0.

Toward Clinical Generative AI: Conceptual Framework

Clinical decision-making is a crucial aspect of health care, involving the balanced integration of scientific evidence, clinical judgment, ethical considerations, and patient involvement. This process is dynamic and multifaceted, relying on clinicians' knowledge, experience, and intuitive understanding to achieve optimal patient outcomes through informed, evidence-based choices. The advent of generative artificial intelligence (AI) presents a revolutionary opportunity in clinical decision-making. AI's advanced data analysis and pattern recognition capabilities can significantly enhance the diagnosis and treatment of diseases, processing vast medical data to identify patterns, tailor treatments, predict disease progression, and aid in proactive patient management. However, the incorporation of AI into clinical decision-making raises concerns regarding the reliability and accuracy of AI-generated insights. To address these concerns, 11 "verification paradigms" are proposed in this paper, with each paradigm being a unique method to verify the evidence-based nature of AI in clinical decision-making. This paper also frames the concept of "clinically explainable, fair, and responsible, clinician-, expert-, and patient-in-the-loop AI." This model focuses on ensuring AI's comprehensibility, collaborative nature, and ethical grounding, advocating for AI to serve as an augmentative tool, with its decision-making processes being transparent and understandable to clinicians and patients. The integration of AI should enhance, not replace, the clinician's judgment and should involve continuous learning and adaptation based on real-world outcomes and ethical and legal compliance. In conclusion, while generative AI holds immense promise in enhancing clinical decision-making, it is essential to ensure that it produces evidence-based, reliable, and impactful knowledge. Using the outlined paradigms and approaches can help the medical and patient communities harness AI's potential while maintaining high patient care standards.