Can USMLE and COMLEX-USA Scores Predict At-Risk Emergency Medicine Residents' Performance on In-Training Examinations?

PURPOSE

The United States Medical Licensing Examination (USMLE) and Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores are standard methods used to determine residency candidates' medical knowledge. The authors were interested in using the USMLE and COMLEX part 2 scores in our emergency medicine (EM) residency program to identify at-risk residents who may have difficulty on the in-training exam (ITE) and to determine the cutoff values under which an intern could be given an individualized study plan to ensure medical knowledge competency.

METHODS

The authors abstracted the USMLE and COMLEX part 2 scores and the American Board of Emergency Medicine (ABEM) ITE scores for a cohort of first-year EM residents graduating years 2010-2022, converting raw scores to percentiles, and compared part 2 and ABEM ITE scores with Pearson's correlation, a Bland-Altman analysis of bias and 95% limits of agreement, and ROC analysis to determine optimal the cut-off values for predicting ABEM ITE < 50th percentile and the estimated test characteristics.

RESULTS

Scores were available for 152 residents, including 93 USMLE and 88 COMLEX exams. The correlations between part 2 scores and ABEM ITE were r = 0.36 (95%CI: 0.17, 0.52; p < 0.001) for USMLE and r = 0.50 (95%CI: 0.33, 0.64; p < 0.001) for COMLEX. Bias and limits of agreement for both part 2 scores were -14 ± 63% for USMLE and 13 ± 50% for COMLEX in predicting the ABEM ITE scores. USMLE < 37th percentile and COMLEX < 53rd percentile identified 42% (N = 39) and 27% (N = 24) of EM residents, respectively, as at risk, with a sensitivity of 61% and 49% and specificity of 71% and 92%, respectively.

CONCLUSION

USMLE and COMLEX part 2 scores have a very limited role in identifying those at risk of low ITE performance, suggesting that other factors should be considered to identify interns in need of medical knowledge remediation.

Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study

BACKGROUND

The potential of artificial intelligence (AI)-based large language models, such as ChatGPT, has gained significant attention in the medical field. This enthusiasm is driven not only by recent breakthroughs and improved accessibility, but also by the prospect of democratizing medical knowledge and promoting equitable health care. However, the performance of ChatGPT is substantially influenced by the input language, and given the growing public trust in this AI tool compared to that in traditional sources of information, investigating its medical accuracy across different languages is of particular importance.

OBJECTIVE

This study aimed to compare the performance of GPT-3.5 and GPT-4 with that of medical students on the written German medical licensing examination.

METHODS

To assess GPT-3.5's and GPT-4's medical proficiency, we used 937 original multiple-choice questions from 3 written German medical licensing examinations in October 2021, April 2022, and October 2022.

RESULTS

GPT-4 achieved an average score of 85% and ranked in the 92.8th, 99.5th, and 92.6th percentiles among medical students who took the same examinations in October 2021, April 2022, and October 2022, respectively. This represents a substantial improvement of 27% compared to GPT-3.5, which only passed 1 out of the 3 examinations. While GPT-3.5 performed well in psychiatry questions, GPT-4 exhibited strengths in internal medicine and surgery but showed weakness in academic research.

CONCLUSIONS

The study results highlight ChatGPT's remarkable improvement from moderate (GPT-3.5) to high competency (GPT-4) in answering medical licensing examination questions in German. While GPT-4's predecessor (GPT-3.5) was imprecise and inconsistent, it demonstrates considerable potential to improve medical education and patient care, provided that medically trained users critically evaluate its results. As the replacement of search engines by AI tools seems possible in the future, further studies with nonprofessional questions are needed to assess the safety and accuracy of ChatGPT for the general population.

The effectiveness of badging systems in engaging, motivating, and incentivizing students in the mastery of nursing licensure materials

INTRODUCTION

Integrating gamification methods into undergraduate nursing programs has prepared students to pass the nursing comprehensive predictor test and the licensure examination.

LITERATURE REVIEW

Research demonstrates that the motivational factors of game elements like badging and leaderboards are of great value and utility to student engagement and motivation. The badges symbolize achievement, authority, and belonging, whereas leaderboards rank students based on different levels of course engagement.

DISCUSSION

This paper explores the benefits, challenges, and strategies of incorporating digital badges and leaderboards in the undergraduate program. Immersion courses preparing students for professional practice are usually one of the last didactic courses offered in the undergraduate curriculum, designed to integrate all knowledge gained from the nursing program.

IMPLICATIONS FOR AN INTERNATIONAL AUDIENCE

The digital badging system can encourage nurse educators globally to engage, motivate, and power students to achieve professional goals. Furthermore, nursing programs worldwide can benefit from adding digital badges and leaderboards to final semester preparatory courses or any nursing course with the same emphasis.

CONCLUSIONS

Digital badges and leaderboards engage and motivate students to integrate knowledge and skills learned in the nursing program and successfully master nurse licensure materials.

Pre-clerkship National Board of Medical Examiners Subject Examinations Versus End-of-Semester Final Examinations: How Well Do They Assess Preparedness for the United States Medical Licensing Examination Step 1?

Background The ability to provide performance insights of various United States Medical Licensing Examination (USMLE) Step 1 assessments is of great importance to medical educators. Two custom pre-clerkship assessments used at the Kirk Kerkorian School of Medicine at the University of Nevada, Las Vegas (KSOM) are National Board of Medical Examiners (NBME)-derived end-of-semester final examinations and subject examinations. The authors sought to determine if performance on these custom assessments can provide feedback on a medical student's readiness to undertake the USMLE Step 1 examination. Methodology Deidentified student performance data were provided by institutional databases for the KSOM graduating class of 2023 (N = 60). Pearson correlation analyses were utilized to evaluate the strength of the correlation between USMLE Step 1 performance and NBME subject examinations versus NBME end-of-semester final examinations. Results The results indicated that the NBME end-of-semester final examinations have a statistically higher correlation to the USMLE Step 1 score than the majority of the individual NBME subject examinations. However, the mean NBME subject examination score (Semester 1: r = 0.53, p < 0.05; Semester 2: r = 0.58, p < 0.05) demonstrated significantly higher correlation to the USMLE Step 1 performance than the NBME end-of-semester final examination score for both Semesters 1 and 2 (Semester 1: r = 0.50, p < 0.05; Semester 2: r = 0.48, p < 0.05). Conclusions These results showed that the mean of the NBME subject examination score was a better metric to assess readiness for the USMLE Step 1 than the NBME end-of-semester final examinations. However, each NBME end-of-semester final examination score showed a better correlation than the majority of the NBME subject examinations.

NCLEX-RN© preparation resources available online in French: An integrative review

AIM

This review describes the availability of online French NCLEX-RN^© preparation resources for candidates BACKGROUND: One entry to practice requirement for Canadian nurses is to successfully pass a licensing exam upon graduation from their educational program. In 2015, the American NCLEX-RN^© replaced the Canadian entry to practice licensing examination which was offered in Canada's two official languages: English and French. The NCLEX-RN^© was developed in English and later translated to French. Since its implementation, Francophone candidates and educators in Canada have reported a lack of preparatory resources available in their language and have had substantial lower NCLEX-RN^© pass rates, consistently below 50% METHODS: An integrative review using Whittemore and Knafl's framework was conducted between February and May 2019, and updated in September 2020, through online searches of CINAHL, PubMed, Science Direct and Google Scholar databases. Grey literature was included from 2012 onwards. Results are presented narratively.

RESULTS

A total of 17 French language preparatory resources were found. These resources were categorised into four main groups: (1) What is the NCLEX-RN^© ?; (2) What do I need to do prior to writing the NCLEX-RN^© ?; (3) What is assessed through the NCLEX-RN^© ? and finally, (4) How can I practice before taking the NCLEX-RN^© ?

CONCLUSION

Limited French-language NCLEX-RN^© preparatory resources exist for Francophone candidates. Furthermore, practice questions in French are few compared to what is available in English IMPLICATIONS FOR NURSING AND NURSING POLICY: Other countries may consider implementing an entry to practice exam such as the NCLEX-RN^© because of its availability in both the French and English language, as well as the possibility of translating the exam to other languages, creating a potential market for this test around the globe. The lack of preparatory resources in French is a major concern to Francophone candidates undertaking such a high-stakes examination in their language. Nursing stakeholders and policy leaders should acknowledge that such gaps place Francophone writers in a disadvantaged position in comparison to their Anglophone counterparts.

The Saudi Medical Licensure Examination-Clinical Skills (SMLE-CS): A Call for Implementation

The Saudi Commission for Heath Specialties (SCFHS) was founded in 1992 to nationally regulate healthcare-related practices and accreditation. Specifically, SCFHS is the official organization that principally oversees the postgraduate residency training programs (RTPs). A crucial aim of SCFHS is to warrant that medical graduates, prior to their enrollment into RTPs as first-year resident physicians, have national minimum entry standards of learning competencies to practice safe and effective healthcare. Generally, there are three primary domains of learning competencies that should be assessed, namely: theoretical clinical knowledge, practical clinical skills and professional attitudes. SCFHS primarily evaluates the theoretical clinical knowledge of applicants through the administration of the Saudi Medical Licensure Examination (SMLE) and we call on SCFHS to rename the conventional SMLE to SMLE-Clinical Knowledge (CK), or shortly abbreviated as SMLE-CK. On the other hand, to date, there is no examination administered by the SCFHS that assesses the applicants' competencies of practical clinical skills and professional attitudes prior to admission to RTPs. Herein, we call on SCFHS to formally incorporate a mandatory national practical licensure examination. The suggested name is the SMLE-Clinical Skills (CS), or shortly abbreviated as SMLE-CS. The purpose, structure, content, rationale, potential counteractive views and future research directions regarding the SMLE-CS are presented. This proposal is not limited to Saudi Arabia only, and it may be contemplated by the other countries, too.

Non-Patient-Based Clinical Licensure Examination for Dentistry in Minnesota: Significance of Decision and Description of Process

In recent years in the United States, there has been heightened interest in offering clinical licensure examination (CLE) alternatives to the live patient-based method in dentistry. Fueled by ethical concerns of faculty members at the University of Minnesota School of Dentistry, the state of Minnesota's Board of Dentistry approved a motion in 2009 to provide two CLE options to the school's future predoctoral graduates: a patient-based one, administered by the Central Regional Dental Testing Service, and a non-patient-based one administered by the National Dental Examining Board of Canada (NDEB). The validity of the NDEB written exam and objective structured clinical exam (OSCE) has been verified in a multi-year study. Via five-option, one-best-answer, multiple-choice questions in the written exam and extended match questions with up to 15 answer options in the station-based OSCE, competent candidates are distinguished from those who are incompetent in their didactic knowledge and clinical critical thinking and judgment across all dental disciplines. The action had the additional effects of furthering participation of Minnesota Board of Dentistry members in the University of Minnesota School of Dentistry's competency-based curriculum, of involving the school's faculty in NDEB item development workshops, and, beginning in 2018, of no longer permitting the patient-based CLE option on site. The aim of this article is to describe how this change came about and its effects.

Constructing licensure exams: a reliability study of case-based questions on the National Board Dental Hygiene Examination

Patient cases with associated questions are a method for increasing the clinical relevance of licensure exams. This study used generalizability theory to assess changes in score reliability when the number of questions per case varied in the National Board Dental Hygiene Examination (NBDHE). The experimental design maintained the same total number of case-based items, while varying the number of cases and items within cases to assess changes in score reliability. Using generalizability theory, the amounts of error variance within cases and between cases on the NBDHE were assessed. Impact on score reliability (generalizability) was computed. The data were from the responses of 4,528 candidates who took the paper-pencil version of the NBDHE in spring 2009. Results showed that the minimum value of generalizability occurred when fourteen cases with ten items each were used in the examination. The maximum value of generalizability occurred when fifty cases with three items each were used in the examination. The research findings support the development of more cases with fewer items per case on the NBDHE in order to enhance test score reliability and validity. Practical constraints should be considered if more cases with fewer items per case are developed for future examinations.

The NAPLEX: evolution, purpose, scope, and educational implications

Since 2004, passing the North American Pharmacist Licensure Examination (NAPLEX) has been a requirement for earning initial pharmacy licensure in all 50 United States. The creation and evolution from 1952-2005 of the particular pharmacy competency testing areas and quantities of questions are described for the former paper-and-pencil National Association of Boards of Pharmacy Licensure Examination (NABPLEX) and the current candidate-specific computer adaptive NAPLEX pharmacy licensure examinations. A 40% increase in the weighting of NAPLEX Blueprint Area 2 in May 2005, compared to that in the preceding 1997-2005 Blueprint, has implications for candidates' NAPLEX performance and associated curricular content and instruction. New pharmacy graduates' scores on the NAPLEX are neither intended nor validated to serve as a criterion for assessing or judging the quality or effectiveness of pharmacy curricula and instruction. The newest cycle of NAPLEX Blueprint revision, a continual process to ensure representation of nationwide contemporary practice, began in early 2008. It may take up to 2 years, including surveying several thousand national pharmacists, to complete.