Does developing multiple-choice Questions Improve Medical Students' Learning? A Systematic Review

ChatGPT as a teaching tool: Preparing pathology residents for board examination with AI-generated digestive system pathology tests

OBJECTIVES

To evaluate the effectiveness of ChatGPT 4 in generating multiple-choice questions (MCQs) with explanations for pathology board examinations, specifically for digestive system pathology.

METHODS

The customized ChatGPT 4 model was developed for MCQ and explanation generation. Expert pathologists evaluated content accuracy and relevance. These MCQs were then administered to pathology residents, followed by an analysis focusing on question difficulty, accuracy, item discrimination, and internal consistency.

RESULTS

The customized ChatGPT 4 generated 80 MCQs covering various gastrointestinal and hepatobiliary topics. While the MCQs demonstrated moderate to high agreement in evaluation parameters such as content accuracy, clinical relevance, and overall quality, there were issues in cognitive level and distractor quality. The explanations were generally acceptable. Involving 9 residents with a median experience of 1 year, the average score was 57.4 (71.8%). Pairwise comparisons revealed a significant difference in performance between each year group (P < .01). The test analysis showed moderate difficulty, effective item discrimination (index = 0.15), and good internal consistency (Cronbach's α = 0.74).

CONCLUSIONS

ChatGPT 4 demonstrated significant potential as a supplementary educational tool in medical education, especially in generating MCQs with explanations similar to those seen in board examinations. While artificial intelligence-generated content was of high quality, it necessitated refinement and expert review.

Gender differences in learning and study strategies impact medical students' preclinical and USMLE step 1 examination performance

BACKGROUND

Evaluation of students' learning strategies can enhance academic support. Few studies have investigated differences in learning strategies between male and female students as well as their impact on United States Medical Licensing Examination® (USMLE) Step 1 and preclinical performance.

METHODS

The Learning and Study Strategies Inventory (LASSI) was administered to the classes of 2019-2024 (female (n = 350) and male (n = 262)). Students' performance on preclinical first-year (M1) courses, preclinical second-year (M2) courses, and USMLE Step 1 was recorded. An independent t-test evaluated differences between females and males on each LASSI scale. A Pearson product moment correlation determined which LASSI scales correlated with preclinical performance and USMLE Step 1 examinations.

RESULTS

Of the 10 LASSI scales, Anxiety, Attention, Information Processing, Selecting Main Idea, Test Strategies and Using Academic Resources showed significant differences between genders. Females reported higher levels of Anxiety (p < 0.001), which significantly influenced their performance. While males and females scored similarly in Concentration, Motivation, and Time Management, these scales were significant predictors of performance variation in females. Test Strategies was the largest contributor to performance variation for all students, regardless of gender.

CONCLUSION

Gender differences in learning influence performance on STEP1. Consideration of this study's results will allow for targeted interventions for academic success.

The equation for medical multiple-choice question testing time estimation

Introduction and importance

Multiple-choice questions (MCQs) offer a suitable means to assess the cognitive domain of learners with a high degree of objectivity. The study's objective is to formulate an equation for determining the ideal timing for MCQ examinations, thereby inspiring the development of a model to estimate the duration of these examinations.

Methods

The authors generated a specific computer program that integrated with the operating system of the examination. Technical-specific features included the ability to calculate the speed of students taking examinations with images or videos in the questions or options. This bespoke computer program was designed specifically for assessing individual students' MCQ test-taking pace and generating a proctor report in a computer-readable format. Subsequently, data derived from this program underwent regression analysis to determine the speed at which students completed MCQ examinations.

Outcomes

The data were collected from a total of 1035 examinees, all of whom were non-native English speakers. The average reading rate was 62.38±20.4 words/min. It was found that the rate decreased significantly in difficult (50.65±6.9 words/min) items compared to easy (82.29±21.3 words/min) and intermediate (60.56±19.1 words/min) items (p<0.001), respectively. The linear regression analysis predicted option selection (words/min) as; 33.92+1.93(%tables/figures)+0.14(%recall)-0.37(%application), r2=0.45, p<0.001.

Conclusion

It is not advisable to base the decision solely on reading time or time allocation. Examination administrators are advised to proactively plan ahead, with particular emphasis on establishing a well-defined taxonomy, as it constitutes a fundamental cornerstone in the utilization of the estimation equation.

ChatGPT 3.5 fails to write appropriate multiple choice practice exam questions

Artificial intelligence (AI) may have a profound impact on traditional teaching in academic settings. Multiple concerns have been raised, especially related to using ChatGPT for creating de novo essays. However, AI programs such as ChatGPT may augment teaching techniques. In this article, we used ChatGPT 3.5 to create 60 multiple choice questions. Author written text was uploaded and ChatGPT asked to create multiple choice questions with an explanation for the correct answer and explanations for the incorrect answers. Unfortunately, ChatGPT only generated correct questions and answers with explanations in 32 % of the questions (19 out of 60). In many instances, ChatGPT failed to provide an explanation for the incorrect answers. An additional 25 % of the questions had answers that were either wrong or misleading. A grade of 32 % would be considered failing in most courses. Despite these issues, instructors may still find ChatGPT useful for creating practice exams with explanations-with the caveat that extensive editing may be required.

Twelve tips to leverage AI for efficient and effective medical question generation: A guide for educators using Chat GPT

BACKGROUND

Crafting quality assessment questions in medical education is a crucial yet time-consuming, expertise-driven undertaking that calls for innovative solutions. Large language models (LLMs), such as ChatGPT (Chat Generative Pre-Trained Transformer), present a promising yet underexplored avenue for such innovations.

AIMS

This study explores the utility of ChatGPT to generate diverse, high-quality medical questions, focusing on multiple-choice questions (MCQs) as an illustrative example, to increase educator's productivity and enable self-directed learning for students.

DESCRIPTION

Leveraging 12 strategies, we demonstrate how ChatGPT can be effectively used to generate assessment questions aligned with Bloom's taxonomy and core knowledge domains while promoting best practices in assessment design.

CONCLUSION

Integrating LLM tools like ChatGPT into generating medical assessment questions like MCQs augments but does not replace human expertise. With continual instruction refinement, AI can produce high-standard questions. Yet, the onus of ensuring ultimate quality and accuracy remains with subject matter experts, affirming the irreplaceable value of human involvement in the artificial intelligence-driven education paradigm.

Feedback, fairness, and validity: effects of disclosing and reusing multiple-choice questions in medical schools

BACKGROUND

Disclosure of items used in multiple-choice-question (MCQ) exams may decrease student anxiety and improve transparency, feedback, and test-enhanced learning but potentially compromises the reliability and fairness of exams if items are eventually reused. Evidence regarding whether disclosure and reuse of test items change item psychometrics is scarce and inconclusive.

METHODS

We retrospectively analysed difficulty and discrimination coefficients of 10,148 MCQ items used between fall 2017 and fall 2019 in a large European medical school in which items were disclosed from fall 2017 onwards. We categorised items as 'new'; 'reused, not disclosed'; or 'reused, disclosed'. For reused items, we calculated the difference from their first ever use, that is, when they were new. Differences between categories and terms were analysed with one-way analyses of variance and independent-samples t tests.

RESULTS

The proportion of reused, disclosed items grew from 0% to 48.4%; mean difficulty coefficients increased from 0.70 to 0.76; that is, items became easier, P < .001, η_p² = 0.011. On average, reused, disclosed items were significantly easier (M = 0.83) than reused, not disclosed items (M = 0.71) and entirely new items (M = 0.66), P < .001, η_p² = 0.087. Mean discrimination coefficients increased from 0.21 to 0.23; that is, item became slightly more discriminating, P = .002, η_p² = 0.002.

CONCLUSIONS

Disclosing test items provides the opportunity to enhance feedback and transparency in MCQ exams but potentially at the expense of decreased item reliability. Discrimination was positively affected. Our study may help weigh advantages and disadvantages of using previously disclosed items.

A National Medical Student Diagnostic Radiology Competition Initiative: Survey and Outcomes of the RadiOlympics

RATIONALE AND OBJECTIVES

Diagnostic radiology residents may participate in an annual diagnostic imaging tournament that enables residents to engage in friendly competition, network with peers, and practice for board examinations. Medical students would likely enjoy a similar activity, which could increase their interest and knowledge in radiology. Given the lack of initiatives designed to promote competition and learning in medical school radiology education, we designed and implemented the RadiOlympics, the first known national medical student radiology competition in the United States.

MATERIAL AND METHODS

A draft version of the competition was emailed to many medical schools in the United States. Medical students interested in assisting with implementation of the competition were invited to a meeting to refine the layout. Ultimately, the format of seven rounds of five questions each and a final round of ten questions all over four months was decided. Questions were written by students and approved by faculty. At the conclusion of the competition, surveys were sent out to gather feedback and gauge how this competition has influenced their interest in radiology.

RESULTS

Out of 89 schools that were successfully contacted, 16 schools' radiology clubs agreed to participate, which made up 187 medical students on average per round. At the conclusion of the competition, feedback from students was very positive. Students' confidence in interpreting imaging studies increased after the competition (p < 0.001), although there was not an increased interest in radiology as a career (p = 0.77).

CONCLUSION

The RadiOlympics is a national competition that can be successfully organized by medical students for medical students and is an engaging opportunity for medical students to be exposed to radiology.

Interactive Learning: Online Audience Response System and Multiple Choice Questions Improve Student Participation in Lectures

Background and Objective Multiple Choice Questions (MCQs) are commonly used in medical education for formative and summative assessment. However, the use of MCQs as a pedagogical tool in lectures is an area that is yet to be fully explored. This study aimed to gather feedback on including MCQs in lectures using an Online Audience Response System (OARS). Methods This quasi-experimental study involved 114 first professional MBBS students. A lecture with strategically integrated MCQs was delivered in a traditional classroom setting. Students answered the ten MCQs and provided feedback via OARS using their mobile phones. The feedback included eleven questions about student engagement, attentiveness, and critical thinking skills. Students' perception of the advantages and limitations of OARS in lectures was also collected. Data are presented as percentages and median with an interquartile range. Results Higher levels of engagement were reported by 80% of students; 81% felt improvement in understanding of the learning objectives and lecture content; 5% felt that mobile phone usage caused distractions; 79% reported increased focus and concentration; 84% reported that MCQs promoted their critical thinking skills; 75% reported enhanced overall learning experience without causing any discomfort; 69% believed that asking questions individually by the instructor is less effective than MCQs with OARS; 83% felt motivated to come prepared with study material when MCQs were included in the lecture; 67% preferred an even distribution of MCQs throughout the lecture; 53% preferred six to ten MCQs per lecture. Conclusion This study suggests that integrating MCQs in lectures using OARS can be a valuable pedagogical strategy in medical education and can potentially improve the learning experience by enhancing student engagement, attentiveness, and critical thinking skills. However, educators should also be aware of the potential limitations and take steps to mitigate them.

Evidence for commonly used teaching, learning and assessment methods in contact lens clinical skills education

INTRODUCTION

Evidence based practice is now an important part of healthcare education. The aim of this narrative literature review was to determine what evidence exists on the efficacy of commonly used teaching and learning and assessment methods in the realm of contact lens skills education (CLE) in order to provide insights into best practice. A summary of the global regulation and provision of postgraduate learning and continuing professional development in CLE is included.

METHOD

An expert panel of educators was recruited and completed a literature review of current evidence of teaching and learning and assessment methods in healthcare training, with an emphasis on health care, general optometry and CLE.

RESULTS

No direct evidence of benefit of teaching and learning and assessment methods in CLE were found. There was evidence for the benefit of some teaching and learning and assessment methods in other disciplines that could be transferable to CLE and could help students meet the intended learning outcomes. There was evidence that the following teaching and learning methods helped health-care and general optometry students meet the intended learning outcomes; clinical teaching and learning, flipped classrooms, clinical skills videos and clerkships. For assessment these methods were; essays, case presentations, objective structured clinical examinations, self-assessment and formative assessment. There was no evidence that the following teaching and learning methods helped health-care and general optometry students meet the intended learning outcomes; journal clubs and case discussions. Nor was any evidence found for the following assessment methods; multiple-choice questions, oral examinations, objective structured practical examinations, holistic assessment, and summative assessment.

CONCLUSION

Investigation into the efficacy of common teaching and learning and assessment methods in CLE are required and would be beneficial for the entire community of contact lens educators, and other disciplines that wish to adapt this approach of evidence-based teaching.