Chatbot-based serious games: A useful tool for training medical students? A randomized controlled trial

Class in Session: Analysis of GPT-4-created Plastic Surgery In-service Examination Questions

Background

The Plastic Surgery In-Service Training Examination (PSITE) remains a critical milestone in residency training. Successful preparation requires extensive studying during an individual's residency. This study focuses on the capacity of Generative Pre-trained Transformer 4 (GPT-4) to generate PSITE practice questions.

Methods

GPT-4 was prompted to generate multiple choice questions for each PSITE section and provide answer choices with detailed rationale. Question composition via readability metrics were analyzed, along with quality. Descriptive statistics compared GPT-4 and the 2022 PSITE.

Results

The overall median Flesch-Kincaid reading ease for GPT-4-generated questions was 43.90 (versus 50.35 PSITE, P = 0.036). GPT-4 provided questions that contained significantly fewer mean sentences (1 versus 4), words (16 versus 56), and percentage of complex words (3 versus 13) than 2022 PSITE questions (P < 0.001). When evaluating GPT-4 generated questions for each examination section, the highest median Flesch-Kincaid reading ease was on the core surgical principles section (median: 63.30, interquartile range [54.45-68.28]) and the lowest was on the craniomaxillofacial section (median: 36.25, interquartile range [12.57-58.40]). Most readability metrics were higher for the 2022 PSITE compared with GPT-4 generated questions. Overall question quality was poor for the chatbot.

Conclusions

Our study found that GPT-4 can be adapted to generate practice questions for the 2022 PSITE, but its questions are of poor quality. The program can offer general explanations for both the correct and incorrect answer options but was observed to generate false information and poor-quality explanations. Although trainees should navigate with caution as the technology develops, GPT-4 has the potential to serve as an effective educational adjunct under the supervision of trained plastic surgeons.

Performance of ChatGPT on Stage 1 of the Taiwanese medical licensing exam

Introduction

Since its release by OpenAI in November 2022, numerous studies have subjected ChatGPT to various tests to evaluate its performance in medical exams. The objective of this study is to evaluate ChatGPT's accuracy and logical reasoning across all 10 subjects featured in Stage 1 of Senior Professional and Technical Examinations for Medical Doctors (SPTEMD) in Taiwan, with questions that encompass both Chinese and English.

Methods

In this study, we tested ChatGPT-4 to complete SPTEMD Stage 1. The model was presented with multiple-choice questions extracted from three separate tests conducted in February 2022, July 2022, and February 2023. These questions encompass 10 subjects, namely biochemistry and molecular biology, anatomy, embryology and developmental biology, histology, physiology, microbiology and immunology, parasitology, pharmacology, pathology, and public health. Subsequently, we analyzed the model's accuracy for each subject.

Result

In all three tests, ChatGPT achieved scores surpassing the 60% passing threshold, resulting in an overall average score of 87.8%. Notably, its best performance was in biochemistry, where it garnered an average score of 93.8%. Conversely, the performance of the generative pre-trained transformer (GPT)-4 assistant on anatomy, parasitology, and embryology was not as good. In addition, its scores were highly variable in embryology and parasitology.

Conclusion

ChatGPT has the potential to facilitate not only exam preparation but also improve the accessibility of medical education and support continuous education for medical professionals. In conclusion, this study has demonstrated ChatGPT's potential competence across various subjects within the SPTEMD Stage 1 and suggests that it could be a helpful tool for learning and exam preparation for medical students and professionals.

Artificial Intelligence Revolutionizing the Field of Medical Education

Medical education has ventured into a new arena of computer-assisted teaching powered by artificial intelligence (AI). In medical institutions, AI can serve as an intelligent tool facilitating the decision-making process effectively. AI can enhance teaching by assisting in developing new strategies for educators. Similarly, students also benefit from intelligent systems playing the role of competent teachers. Thus, AI-integrated medical education paves new opportunities for advanced teaching and learning experiences and improved outcomes. On the other hand, optical mark recognition and automated scoring are ways AI can also transform into a real-time assessor and evaluator in medical education. This review summarizes the AI tools and their application in medical teaching or learning, assessment, and administrative support. This article can aid medical institutes in planning and implementing AI according to the needs of the educators.