Factors Associated With Surgery Clerkship Performance and Subsequent USMLE Step Scores

Impact of Obstetrics and Gynecology Clerkship Order on NBME Exam Scores After a Curriculum Change

OBJECTIVE

Our Institution instituted curriculum reform in 2013. We sought to examine the impact of rotation order on the end of clerkship National Board of Medical Examiners (NBME) Obstetrics and Gynecology (Ob/Gyn) Subject Exam scores after curriculum restructuring.

DESIGN

This is a retrospective analysis of Ob/Gyn NBME scores over 2 years after curriculum reform. At our university, a 15-week block is divided into 5-week intervals of General Surgery, Ob/Gyn, and surgical subspecialties, in any order. During the 16 weeks, students take the NBME Subject Examinations for Ob/Gyn and Surgery. We defined rotation groups by proximity to the shelf exam. Group 1 completed Ob/Gyn first, furthest away from the exam, Group 2 completed Ob/Gyn second, and Group 3 completed Ob/Gyn last, closest to the test. We compared average shelf exam scores between Groups.

SETTING

Uniformed Services University of Health Sciences during the clerkship year.

PARTICIPANTS

Medical students at the Uniformed Services University.

RESULTS

We obtained data from 331 students (118 students in Group 1, 106 in Group 2, and 107 in Group 3). Scores ranged from 55 to 99 (mean 72.5, SD 7.3). Mean (SD) NBME score was 71.9 (6.9) in Group 1, 73.2 (7.2) in Group 2, and 72.6 (7.7) in Group 3 (p = 0.415).

CONCLUSION

Rotation order does not affect NBME Ob/Gyn Subject exam scores in a fifteen-week integrated clerkship block.

Factors Leading to Successful Performance on U.S. National Licensure Exams for Medical Students: A Scoping Review

PURPOSE

To synthesize the evidence of the factors leading to successful performance on knowledge-based national licensure exams (NLEs) for medical students.

METHOD

The authors conducted a scoping review to summarize the peer-reviewed empiric literature that used United States Medical Licensing Examination (USMLE) Step 1 or Step 2 Clinical Knowledge or Comprehensive Osteopathic Medical Licensing Examination (COMLEX) Level 1 or Level 2 Cognitive Evaluation scores as outcomes. The authors searched PubMed and Scopus without date restrictions through April 30, 2021. Two reviewers independently screened and selected studies for inclusion. Data were summarized narratively and with descriptive statistics.

RESULTS

The authors screened 1,185 unique citations and included 233 full-text studies in their review. Of these, 201 (86%) were studies of USMLE exams, 31 (13%) were studies of COMLEX exams, and 1 (0.4%) reported on both. The authors classified 29 studies (12%) as informing NLE preparation, 163 (70%) as attempting to identify predictive variables, and 76 (33%) as using NLE scores for program evaluation. Preparation studies found that the number of practice test items, practice exam scores, and less time in dedicated preparation correlated with higher NLE scores. Use of other commercial resources or study strategies was not consistently associated with higher scores. Predictive studies found the strongest relationships between individuals' performance on past assessments and their NLE scores.

CONCLUSIONS

The factors leading to successful performance on knowledge-based NLEs align with well-known principles from the cognitive sciences. Learners build on existing foundations of knowledge (reflected in their prior academic performance) and are likely to learn more efficiently with testing and spaced learning over time. While commercial test preparation resources are ubiquitous, there is no evidence that a single resource gives students a competitive advantage on NLEs. Developing habits of regular and continuous learning is necessary for clinical practice and successful NLE performance.

Measuring what matters: identifying assessments that reflect learning on the core surgical clerkship

Purpose

There are various assessments used during the core surgical clerkship (CSC), each of which may be influenced by factors external to the CSC or have inherent biases from an equity lens. In particular, the National Board of Medical Examiners' Clinical Subject Exams ("Shelf") is used heavily and may not reflect clerkship curriculum or clinical learning.

Methods

This is a retrospective review of medical student characteristics and assessments during the CSC from July 2017-June 2021. Assessment methods included: subjective Clinical Performance Assessments (CPA), Shelf, Objective Structured Clinical Examinations, and a short-answer in-house examination (IHE) culminating in a Final Grade (FG) of Honors/Pass/Fail. A Shelf score threshold for Honors was added in academic years 2020-2021. Descriptive, univariate, and multivariable logistic and linear regression statistics were utilized.

Results

We reviewed records of 192 students. Of these, 107 (55.7%) were female, median age was 24 [IQR: 23-26] years, and most were White/Caucasian (N = 106, 55.2%). Univariate analysis showed the number of Exceeds Expectations obtained on CPA to be influenced by surgical subspecialty taken (p = 0.013) and academic year (p < 0.001). Shelf was influenced by students' race (p = 0.009), timing of CSC before or after Internal Medicine (67.9 ± 7.3 vs 72.9 ± 7.1, p < 0.001), and Term taken (increasing from 66.0 ± 8.7 to 73.4 ± 7.5, p < 0.001). IHE scores did not have any external associations. After adjustment with multivariable logistic and linear regressions, CPA and IHE did not have external associations, but higher scores were obtained on Shelf exam in Terms 3, 5, and 6 (by 4.62 [95% CI 0.86-8.37], 4.92 [95% CI 0.53-9.31], and 7.56 [95% CI 2.81-12.31] points, respectively. Odds of FG honors were lower when Shelf threshold was implemented (OR 0.17 [95% CI 0.06-0.50]), and increased as students got older (OR 1.14 [95% CI 1.01-1.30]) or on specific subspecialties, such as vascular surgery (OR 7.06 [95% CI 1.21-41.26]).

Conclusions

The Shelf is substantially influenced by temporal associations across Terms and timing in relation to other clerkships, such as Internal Medicine. An IHE reflective of a clerkship's specified curriculum may be a more equitable summative assessment of the learning that occurs from the CSC curriculum, with fewer biases or influences external to the CSC.

Supplementary Information

The online version contains supplementary material available at 10.1007/s44186-022-00047-8.

The Impact of Internal Medicine Clerkship Characteristics and NBME Subject Exams on USMLE Step 2 Clinical Knowledge Exam Performance

BACKGROUND

Residency program directors will likely emphasize the United States Medical Licensing Exam (USMLE) Step 2 clinical knowledge (CK) exam more during residency application given the recent USMLE Step 1 transition to pass/fail scoring. We examined how internal medicine clerkship characteristics and NBME subject exam scores affect USMLE Step 2 CK performance.

DESIGN

The authors used univariable and multivariable generalized estimating equations to determine associations between Step 2 CK performance and internal medicine clerkship characteristics and NBME subject exams. The sample had 21,280 examinees' first Step 2 CK scores for analysis.

RESULTS

On multivariable analysis, Step 1 performance (standardized β = 0.45, p < .001) and NBME medicine subject exam performance (standardized β = 0.40, p < .001) accounted for approximately 60% of the variance in Step 2 CK performance. Students who completed the internal medicine clerkship last in the academic year scored lower on Step 2 CK (M_diff = -3.17 p < .001). Students who had a criterion score for passing the NBME medicine subject exam scored higher on Step 2 CK (M_diff = 1.10, p = .03). There was no association between Step 2 CK performance and other internal medicine clerkship characteristics (all p > 0.05) nor with the total NBME subject exams completed (β=0.05, p = .78).

CONCLUSION

Despite similarities between NBME subject exams and Step 2 CK, the authors did not identify improved Step 2 CK performance for students who had more NBME subject exams. The lack of association of Step 2 CK performance with many internal medicine clerkship characteristics and more NBME subject exams has implications for future clerkship structure and summative assessment. The improved Step 2 CK performance in students that completed their internal medicine clerkship earlier warrants further study given the anticipated increase in emphasis on Step 2 CK.

Surgical clerkship: Do examination scores correlate with clinical performance?

BACKGROUND

This study aims to determine if there are correlations between clinical performance and objective grading parameters for medical students in the third-year surgery clerkship.

METHODS

Clerkship grades were compiled from 2016 to 2020. Performance on clinical rotations, NBME shelf exam, oral exam, and weekly quizzes were reviewed. Students were divided into quartiles (Q1-Q4) based on clinical performance. Standard statistical analysis was performed.

RESULTS

There were 625 students included in the study. Students in Q1+Q2 were more likely than those in Q3+Q4 to score in the top quartile on the shelf exam (29% vs. 19%, p = 0.002), oral exam (24% vs. 17%, p = 0.032), and quizzes (22% vs. 15%, p = 0.024). However, there was negligible correlation between clinical performance and performance on objective measures: shelf exam (R² = 0.027, p < 0.001), oral exam (R² = 0.021, p < 0.001), and weekly quizzes (R² = 0.053, p = 0.092).

CONCLUSIONS

Clinical performance does not correlate with objective grading parameters for medical students in the third-year surgery clerkship.

The Impact of Shortened Clinical Clerkships on Medical Student Performance and Clerkship Assessment

INTRO

Medical schools sometimes need to adjust the length of third-year clinical clerkships. The literature surrounding the effects of shortened clerkships on student experience and performance is mixed.

METHODS

Our medical school shortened the third year by an average of 20% per clerkship to accommodate a curricular re-design in 2018-2019. We examined test scores and measures of clinical performance as well as student experience in order to understand the impact of this change.

RESULTS

Two hundred and eight students were included in the analysis, 104 in each cohort. No statistically significant differences were noted between cohorts on NBME subject examination results. There were no significant differences on Step 2 CK scores between the traditional curriculum cohort (M = 249.4, SD = 13.7) and shortened curriculum cohort (M = 248.7, SD = 15.8). Student performance on OSCE cases was similar. Similar percentages of students rated each clerkship either "good" or "excellent" in the traditional (77%) and shortened (78%) curriculum.

CONCLUSION

There was no significant impact on student test scores after shortening the curriculum. Measures of student satisfaction and experience also remained stable, likely related to emphasis on retaining patient care experiences and streamlining of didactics. Curricular shortening during the third year of medical school was feasible and safe from the student perspective in our experience.

Effect of a required online graded curriculum in the clerkship years on medical student national standardized examination performance

CONTEXT

Many medical schools have a distributed model for clinical clerkship education, challenging our ability to determine student gaps during clinical education. With the graduating class of 2017, A.T. Still University's School of Osteopathic Medicine in Arizona (ATSU-SOMA) began requiring additional online curricula for all clerkship courses.

OBJECTIVES

To determine whether third year and fourth year students receiving ATSU-SOMA's online curricula during core clerkships performed better overall on national standardized examinations than students from previous years who had not received the curricula, and whether scores from online coursework correlated with outcomes on standardized examinations as possible early predictors of success.

METHODS

This retrospective cohort study analyzed existing data (demographics and assessments) from ATSU-SOMA classes of 2017-2020 (curriculum group) and 2014-2016 (precurriculum group). The effect of the curriculum on national standardized examinations (Comprehensive Osteopathic Medical Achievement Test [COMAT] and Comprehensive Osteopathic Medical Licensing Examination of the United States [COMLEX-USA]) was estimated using augmented inverse probability weighting (AIPW). Correlations between assignment scores and national standardized examinations were estimated using linear regression models.

RESULTS

The curriculum group had 405 students with a mean (standard deviation [SD]) age of 25.7 (±3.1) years. Two hundred and fifteen (53.1%) students in the curriculum group were female and 190 (46.9%) were male. The precurriculum group had 308 students (mean ± SD age, 26.4 ± 4.2 years; 157 [51.0%] male; 151 [49.0%] female). The online curriculum group had higher COMAT clinical subject exam scores in obstetrics and gynecology, osteopathic principles and practice (OPP), psychiatry, and surgery (all p≤0.04), as well as higher COMLEX-USA Level 2-Cognitive Evaluation (CE) family medicine and OPP subscores (both p≤0.03). The curriculum group had a 9.4 point increase in mean total COMLEX-USA Level 2-CE score (p=0.08). No effect was found for the curriculum overall on COMAT mean or COMLEX-USA Level 2-Performance Evaluation scores (all p≥0.11). Total coursework scores in each core clerkship, excluding pediatrics, were correlated with COMAT mean score (all adjusted p≤0.03). Mean scores for five of the seven assignment types in core clerkships, excluding evidence based medicine types, were positively correlated with COMAT mean scores (all adjusted p≤0.049). All assignment types correlated with COMLEX-USA Level 2-CE total score (all adjusted p≤0.04), except interprofessional education (IPE).

CONCLUSIONS

Results from this study of 713 students from ATSU-SOMA suggested that our online curriculum supplemented clinic based learning during clerkship courses and improved student outcomes on national standardized examinations.

Clerkship Grading and the U.S. Economy: What Medical Education Can Learn From America's Economic History

Clerkship grades (like money) are a social construct that function as the currency through which value exchanges in medical education are negotiated between the system's various stakeholders. They provide a widely recognizable and efficient medium through which learner development can be assessed, tracked, compared, and demonstrated and are commonly used to make decisions regarding progression, distinction, and selection for residency. However, substantial literature has demonstrated how grades imprecisely and unreliably reflect the value of learners. In this article, the authors suggest that challenges with clerkship grades are fundamentally tied to their role as currency in the medical education system. Associations are drawn between clerkship grades and the history of the U.S. economy; 2 major concepts are highlighted: regulation and stock prices. The authors describe the history of these economic concepts and how they relate to challenges in clerkship grading. Using lessons learned from the history of the U.S. economy, the authors then propose a 2-step solution to improve upon grading for future generations of medical students: (1) transition from grades to a federally regulated competency-based assessment model and (2) development of a departmental competency letter that incorporates competency-based assessments rather than letter grades and meets the needs of program directors.

Sustained Clinical Performance During Surgical Rotations Predicts NBME Shelf Exam Outcomes

OBJECTIVE

To determine if longitudinal, excellent clinical performance reflected in subjective evaluations during a surgery clerkship would be associated with a greater likelihood of National Board of Medical Examiners Surgery Shelf Exam ("shelf exam") success.

DESIGN

We retrospectively reviewed medical students' surgical clerkship performance from 2014 to 2019. Clinical evaluations for each rotation were abstracted and students were stratified by performance: excellent performers and non-excellent performers. The rotation performance grades were then combined to classify overall clerkship performance: sustained excellent performers, improved performers, worsened performers, and sustained non-excellent performers. We compared the shelf exam scores between performer class for each clinical rotation and the overall clerkship. Using logistic regression, we also sought to determine if clinical performance predicted passing the shelf exam.

SETTING

Emory University School of Medicine in Atlanta, Georgia.

PARTICIPANTS

Third-year medical students (N = 674) who completed a surgery clerkship.

RESULTS

Excellent performers scored higher than non-excellent performers on the shelf exam during both clinical rotations (all p < 0.01). Sustained excellent performers had the highest exam scores out of all the clerkship performance groups (p < 0.0001). Excellent performers for both rotations were associated with increased odds of passing the shelf exam. Sustained excellent performers had the greatest odds (OR 3, 95% confidence interval 1.5-6.3, p = 0.003) of passing the exam.

CONCLUSIONS

Clinical performance during the surgical clerkship and individual rotations correlates with shelf exam scores. Students should be encouraged to excel on the wards to maximize the educational experience and improve their odds of passing the exam.

Best Study Strategy for the NBME Clinical Science Surgery Exam

INTRODUCTION

Providing medical students with effective study strategies is paramount in fostering their success on the NBME Clinical Science Surgery exam. As of yet, there has not been a comprehensive inquiry into a study strategy for this exam. We aim to identify if the following are associated with higher NBME raw exam scores: (1) the use of popular study materials, (2) the number of study materials used, and (3) the amount of time spent studying throughout the clerkship.

METHODS

Over the period of 1 academic year, third-year medical students at our institution were administered a survey during their surgical clerkship inquiring about study materials used and hours spent studying per week. The data were then matched to students' raw NBME scores and then depersonalized. A total of 82 of 193 (42%) students responded with an overall average raw score of 76.74 on the NBME Clinical Science Surgery exam. By comparing our data to the NBME national average of 70 with a standard deviation of 8, a z test was used to compare the population mean to our sample means.

RESULTS

When investigating resources used, the combination of using an online question bank and a high yield review book yielded a high z score (6.23) and using 4 resources yielded the highest z score (6.28). Regarding study hours, the highest z scores were seen when students studied for 6 to 10 hours per week during the first half of the clerkship, and 11 to 15 hours per week during the second half of the clerkship (5.76 and 6.02, respectively).

CONCLUSIONS

In conclusion, higher NBME Clinical Science Surgery exam scores were correlated with the use of multiple and varied types of resources and increasing study time closer to the exam date. The resources and methods utilized by students achieving the highest exam scores can be recommended by surgical educators and employed by other medical students to foster academic success.

A Validity Study of COMLEX-USA Level 2-CE and COMAT Clinical Subjects: Concurrent and Predictive Evidence

BACKGROUND

The Comprehensive Osteopathic Medical Licensure Examination (COMLEX-USA) Level 2-Cognitive Examination (CE) and the Comprehensive Osteopathic Medical Achievement Test (COMAT) are administered to similar populations (third- and fourth-year osteopathic students) at similar points in time. Examining the relationship between scores on the 2 assessments that measure similar constructs ultimately supports the validity of both.

OBJECTIVE

The purpose of this study is to provide empirical evidence of the concurrent and predictive validity of COMAT and COMLEX-USA Level 2-CE.

METHODS

In 2018, first-attempt scores on Level 2-CE were aggregated from June 2015 to May 2018 and matched with first-attempt scores on each COMAT clinical subject. We conducted correlational analyses between performance on COMAT and Level 2-CE, and COMAT scores and Level 2-CE discipline subscores. Additionally, we used multivariate regression to analyze the predictive relationship between performance on all COMAT clinical subjects and Level 2-CE.

RESULTS

The results from correlational analyses indicated statistically significant, positive associations between COMAT and Level 2-CE scores (r = 0.49-0.68, P < .0001), and statistically significant, but slightly weaker relationships between COMAT scores and Level 2-CE discipline subscores (r = 0.31-0.60, P < .0001). Furthermore, results from the multiple regression indicated that scores on COMAT explained 68% of the variance in Level 2-CE scores, and that COMAT internal medicine and emergency medicine were weighted more heavily than other specialties.

CONCLUSIONS

The findings from this study can inform assessment practices by supporting the use of COMAT for osteopathic medical schools that do not administer COMAT.