Factors Leading to Successful Performance on U.S. National Licensure Exams for Medical Students: A Scoping Review

GPT-4 generated answer rationales to multiple choice assessment questions in undergraduate medical education

BACKGROUND

Pre-clerkship medical students benefit from practice questions that provide rationales for answer choices. Creating these rationales is a time-intensive endeavor. Therefore, not all practice multiple choice questions (MCQ) have corresponding explanations to aid learning. The authors examined artificial intelligence's (AI) potential to create high-quality answer rationales for clinical vignette-style MCQs.

METHODS

The authors conducted a single-center pre-post intervention survey study in August 2023 assessing 8 pre-clerkship course director (CD) attitudes towards GPT-4 generated answer rationales to clinical vignette style MCQs. Ten MCQs from each course's question bank were selected and input into GPT-4 with instructions to select the best answer and generate rationales for each answer choice. CDs were provided their unmodified GPT-4 interactions to assess the accuracy, clarity, appropriateness, and likelihood of implementation of the rationales. CDs were asked about time spent reviewing and making necessary modifications, satisfaction, and receptiveness in using GPT-4 for this purpose.

RESULTS

GPT-4 correctly answered 75/80 (93.8%) questions on the first attempt. CDs were receptive to using GPT-4 for rationale generation and all were satisfied with the generated rationales. CDs determined that the majority of rationales were very accurate (77.5%), very clear (83.8%) and very appropriate (93.8%). Most rationales could be implemented with little or no modification (88.3%). All CDs would implement AI-generated answer rationales with CD editorial insights. Most CDs (75%) took ≤ 4 min to review a set of generated rationales for a question.

CONCLUSION

GPT-4 is an acceptable and feasible tool for generating accurate, clear and appropriate answer rationales for MCQs in medical education. Future studies should examine students' feedback to generated rationales and further explore generating rationales for question with media. The authors plan to explore the implementation of this technological application at their medical school, including logistics and training to create a streamlined process that benefits both learners and educators.

CLINICAL TRIAL

Not applicable; not a clinical trial.

Can all roads lead to competency? School levels effects in Licensing examinations scores

At the foundation of research concerned with professional training is the idea of an assumed causal chain between the policies and practices of education and the eventual behaviours of those that graduate these programs. In medicine, given the social accountability to ensure that teaching and learning gives way to a health human resource that is willing and able to provide the healthcare that patients and communities need, it is of critical importance to generate evidence regarding this causal relationship. One question that medical education scholars ask regularly is the degree to which the unique features of training programs and learning environments impact trainee achievement of the intended learning outcomes. To date, this evidence has been difficult to generate because data pertaining to learners is only rarely systematically brought together across institutions or periods of training. We describe new research which leverages an inter-institutional data-driven approach to investigate the influence of school-level factors on the licensing outcomes of medical students. Specifically, we bring together sociodemographic, admissions, and in-training assessment variables pertaining to medical trainee graduates at each of the six medical schools in Ontario, Canada into multilevel stepwise regression models that determine the degree of association between these variables and graduate performances on the Medical Council of Canada Qualifying Examinations (Part 1, n = 1097 observations; Part 2, n = 616 observations), established predictors of downstream physician performance. As part of this analysis, we include an anonymized school-level (School 1, School 2) independent variable in each of these models. Our results demonstrate that the largest variable associated with performance on both the first and second parts of the licensing examinations is prior academic achievement, notably clerkship performance. Ratings of biomedical knowledge were also significantly associated with the first examination, while clerkship OSCE scores and enrollment in a family medicine residency were significantly associated with the Part 2. Small significant school effects were realized in both models accounting for 4% and 2% of the variance realized in the first and second examinations, respectively. These findings highlight that school enrollment plays a minor role relative to individual student performance in influencing examination outcomes.

A Strategic Approach to Succeed on Clinical Case-Based Multiple-Choice Exams

Despite the importance of case-based multiple-choice question (CBMCQ) exams, medical educators rarely discuss strategies to systematically approach these questions, and literature on the topic is limited. Through trial-and-error, many students develop more refined and efficient approaches to answering CBMCQs that help them maximize the application of their knowledge base. In this article we present a structured six-step approach to answering CBMCQs, grounded in dual process theory. We provide strategies for success on CBMCQ-based exams and approaches to challenging question types. We also present tips for helping neurodiverse students. Medical educators can use this structured approach and the related tips to coach students on improving performance on CBMCQ-based exams.

Unlocking Medical Student Success: A Systematic Review and Meta-Analysis of Third-Party Resources Used for Medical Education and USMLE Board Preparation

An increase in third-party commercial resources targeted towards medical students has led to substantial changes in the way students prepare for medical assessments. A systematic review and meta-analysis was conducted to identify the third-party resources most used by medical students. A total of 1056 citations were screened with 19 full-text studies included. Twenty-two unique third-party resources were examined, with meta-analysis revealing the three most used resources to be First Aid, UWorld, and Pathoma. Medical faculty should consider curriculum adaptation to better cater to the learning needs of students and institute systems that ensure equitable access to verified commercial resources.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-024-02116-7.

Performance of respiratory therapy programs in the Saudi Respiratory Care Licensure Examination: Cross-sectional national results

Background: Recently, there has been an increase in the number of respiratory therapy programs. However, a national consensus is lacking in intended learning objectives, appropriate teaching methods, and suitable assessment tools. Consequently, variations in outcomes among these programs are expected. Aim: To evaluate the performance of respiratory therapy programs in the Saudi Respiratory Care Licensure Examination (SRCLE). Methods: The SRCLE data were retrieved from the Saudi Commission for Health Specialties (SCFHS) database as of 18 March 2024. The datasets included the number of applicants, overall passing rates, maximum scores, and average scores. Data were categorized based on academic institution, including the type of university (governmental or private), nationality, gender, passing status, number of exam attempts, and year of examinations. Performance comparisons were conducted based on gender and year of examinations. Results: The database from the SCFHS shows that 1305 examinees underwent the SRCLEs from the second quarter of 2021 to the first quarter of 2024. Females accounted for 46% of the total, while Saudi examinees made up 97% of all applicants. The overall passing rate stood at 96%. The average score was 613, with the highest score recorded being 740. Notably, there was no significant difference in performance between males and females (p = 0.299). However, there was a considerable variance in performance based on the year of examination (p = 0.024). Conclusion: The existing data demonstrates that most respiratory therapy programs perform well in SRCLE. We found no significant differences based on gender or the type of school attended. Additionally, the performance of these programs has remained consistent over the years.

Exploring the usage of learning resources by medical students in the basic science stage and their effect on academic performance

BACKGROUND

The United States Medical Licensing Examination (USMLE) step 1 is one of the two examinations written after completion of the first two years (basic science stage) of medical school to be eligible to apply for residency training in the USA. A huge number and types of study materials are available to prepare for the exam which might confuse students choosing a resource. We investigated learning resources being used by the third and fifth-semester medical students and their association with academic performance. We also compared learning resources and exam scores of high-performing and low-performing students.

METHODS

Data collection was done using structured (quantitative study) and semi-structured (qualitative study) questionnaires during a face-to-face interview. This article is about the quantitative part which was designed as a correlational study. Single factor one-way analysis of variance (ANOVA), Pearson correlation coefficient test, T-test, and Fisher's exact test were used to analyze the data.

RESULTS

About half of all students used three or more commercial resources dealing with the same content. A weak negative correlation was observed between the number of commercial resources and the exam scores, especially when the number of these resources was three or more (r = -0.26). The mean exam score of textbook users was statistically significantly higher than the mean score of textbook non-users (p = 0.01). The usage of textbooks was statistically significantly higher in the cohort of top performers in comparison to the rest of the students (p = 0.006). In addition to less usage of textbooks, the mean number of review books was higher in the group of weakest students (2.84 versus 3.7; p = 0.75).

CONCLUSIONS

Most students did not use professional textbooks and about half used too many commercial review resources. While the former fact was significantly associated with poor academic performance, the later fact had weak negative correlation with exam score. Pedagogical interventions are urgently needed to make the right type of learning resources available by making professional textbooks more USMLE-oriented and helping the students choose the best and right number of resources for optimum academic performance. By fulfilling the observed needs of the students in this way, they might feel empowered because of self-determination which will motivate studies.

PRIME-nurse practitioner competency model validation and criterion based OSCE rubric interrater reliability

The PRIME-NP (Professional-Reporter-Interpreter-Manager-Educator/Evaluation-Nurse Practitioner) Model is adapted from the RIME (Reporter-Interpreter-Manager-Educator) model used in medical education to guide medical student and resident education. The Delphi technique was used to validate the PRIME-NP Model. After two rounds of review by a group of experts in NP curriculum, the model was determined to be valid based on expert consensus. Agreement percent increase from the first round to the second round in all categories. Interrater reliability (IRR) was assessed using interclass correlation after instrument validation was completed for each of the five levels of the PRIME-NP model. Overall, the IRR of the instrument was found to be acceptable with some notable exceptions. No variance was noted in professional behaviors at any level. Variance was increased in management and educator/evaluator behaviors in higher/later course levels. The PRIME-NP Model and PRIME-NP OSCE Rubric is a valid and reliable instrument to assess NP student progression in objective structured clinical examinations. This instrument has the potential for adaptation for use in other types of health sciences education and settings.