Internal medicine residency program director perceptions of USMLE Step 1 pass/fail scoring: A cross-sectional survey

Perceptions and Expectations Regarding the Step 1 Score Change Among Surgical and Non-Surgical Program Directors: A Systematic Review and Meta-Analysis

There are currently no studies examining differences in perceptions and expected impact of the Step 1 score change to pass/fail between surgical and non-surgical program directors (PDs). We conducted a systematic review in May 2023 of PubMed, Scopus, Web of Science, and PSYCInfo to evaluate studies examining PDs' perspectives regarding the Step 1 score change. We performed random-effects meta-analyses to determine differences in perspectives among surgical and non-surgical PDs. Surgical PDs (76.8% [95% CI, 72.1%-82.0%], I2 = 52%) reported significantly greater rates of disagreement with the score change compared to non-surgical (65.1% [95% CI, 57.9%-73.1%], I2 = 69.7%) (P = .01). Surgical PDs also reported significantly greater rates of agreement that the score change will increase the difficulty in objectively comparing applicants (88.1% [95% CI, 84.6%-91.7%], I2 = 16.4%), compared to non-surgical (81.0% [95% CI, 75.6%-86.8%], I2 = 72.6%) (P = .04). There was less heterogeneity among non-surgical PDs (88.7% [95% CI, 86.2%-91.2%], I2 = 0%), compared to surgical (84.7% [95% CI, 79.0%-90.8%], I2 = 67.3%), regarding expected increases in emphasis on Step 2, although the difference in rates of agreement was not statistically significant. Overall, there is significant heterogeneity in the literature regarding expected changes in the residency application review process. Most PDs reported significant disagreement with the score change, greater expected difficulty in objectively evaluating applicants, and greater emphasis on Step 2, with surgical PDs reporting greater rates of disagreement, greater expected difficulty, and heterogeneity regarding expected increases in emphasis on Step 2, compared to non-surgical. Additionally, there is significant heterogeneity in the overall literature regarding expected changes in the residency application review process. Further research is needed to establish evidence-based guidelines that improve the overall residency application process for all stakeholders.

Clinical Assessment Specialists: A New Faculty Role to Improve Clerkship Assessment

INTRODUCTION

Ambulatory clerkships, including longitudinal integrated clerkships (LICs), face challenges to assessment, including time pressure and clinical demands on preceptors. High-quality clinical assessment is critical to implementing competency-based medical education, generating valid grades, and supporting learning. This importance is further heightened with the new pass/fail scoring for US Medical Licensing Exam Step 1, discontinuation of US Medical Licensing Exam Step 2 Clinical Skills, and the growing concern for bias in assessment.

METHODS

The Kaiser Permanente Bernard J. Tyson School of Medicine's LIC spans the first 2 years with 50 students per class. In 2021-2022, the authors created a new faculty role, the clinical assessment specialist (CASp). CASps are highly trained clinical teachers who directly observe clerkship students in the ambulatory setting, provide feedback, and complete competency-based assessment forms.

RESULTS

CASps completed 186 assessments of first-year (Y1) LIC students and 333 assessments of second-year (Y2) LIC students. Y2 students achieved average higher milestones and were rated as requiring less supervision compared to Y1 students. Y1 students rated CASps more favorably than Y2 students. Preceptors rated the contribution of CASps similarly across both years. Clerkship directors described benefits including identification of at-risk students and value of augmenting preceptor assessments.

DISCUSSION

The CASp role may offer an innovative way to generate valid assessment of student performance, offset clinical pressures faced by preceptors, identify at-risk students, and mitigate bias, especially in an LIC. Future studies may examine assessment validity, including use in summative assessment.

CONCLUSION

CASps are an innovative approach to clinical clerkship assessment.

Residency Program Directors' Views on Research Conducted During Medical School: A National Survey

PURPOSE

With the United States Medical Licensing Examination Step 1 transition to pass/fail in 2022, uncertainty exists regarding how other residency application components, including research conducted during medical school, will inform interview and ranking decisions. The authors explore program director (PD) views on medical student research, the importance of disseminating that work, and the translatable skill set of research participation.

METHOD

Surveys were distributed to all U.S. residency PDs and remained open from August to November 2021 to query the importance of research participation in assessing applicants, whether certain types of research were more valued, productivity measures that reflect meaningful research participation, and traits for which research serves as a proxy. The survey also queried whether research would be more important without a numeric Step 1 score and the importance of research vs other application components.

RESULTS

A total of 885 responses from 393 institutions were received. Ten PDs indicated that research is not considered when reviewing applicants, leaving 875 responses for analysis. Among 873 PDs (2 nonrespondents), 358 (41.0%) replied that meaningful research participation will be more important in offering interviews. A total of 164 of 304 most competitive specialties (53.9%) reported increased research importance compared with 99 of 282 competitive (35.1%) and 95 of 287 least competitive (33.1%) specialties. PDs reported that meaningful research participation demonstrated intellectual curiosity (545 [62.3%]), critical and analytical thinking skills (482 [55.1%]), and self-directed learning skills (455 [52.0%]). PDs from the most competitive specialties were significantly more likely to indicate that they value basic science research vs PDs from the least competitive specialties.

CONCLUSIONS

This study demonstrates how PDs value research in their review of applicants, what they perceive research represents in an applicant, and how these views are shifting as the Step 1 exam transitions to pass/fail.

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

Clerkship grading is a core feature of evaluation for medical students' skills as physicians and is considered by most residency program directors to be an indicator of future performance and success. With the transition of the U.S. Medical Licensing Examination Step 1 score to pass/fail, there will likely be even greater reliance on clerkship grades, which raises several important issues that need to be urgently addressed. This article details the current landscape of clerkship grading and the systemic discrepancies in assessment and allocation of honors. The authors examine not only objectivity and fairness in clerkship grading but also the reliability of clerkship grading in predicting residency performance and the potential benefits and drawbacks to adoption of a pass/fail clinical clerkship grading system. In the promotion of a more fair and equitable residency selection process, there must be standardization of grading systems with consideration of explicit grading criteria, grading committees, and/or structured education of evaluators and assessors regarding implicit bias. In addition, greater adherence and enforcement of transparency in grade distributions in the Medical Student Performance Evaluation is needed. These changes have the potential to level the playing field, foster equitable comparisons, and ultimately add more fairness to the residency selection process.

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

PURPOSE

With the change in Step 1 score reporting, Step 2 Clinical Knowledge (CK) may become a pivotal factor in resident selection. This systematic review and meta-analysis seeks to synthesize existing observational studies that assess the relationship between Step 2 CK scores and measures of resident performance.

METHOD

The authors searched MEDLINE, Web of Science, and Scopus databases using terms related to Step 2 CK in 2021. Two researchers identified studies investigating the association between Step 2 CK and measures of resident performance and included studies if they contained a bivariate analysis examining Step 2 CK scores' association with an outcome of interest: in-training examination (ITE) scores, board certification examination scores, select Accreditation Council for Graduate Medical Education core competency assessments, overall resident performance evaluations, or other subjective measures of performance. For outcomes that were investigated by 3 or more studies, pooled effect sizes were estimated with random-effects models.

RESULTS

Among 1,355 potential studies, 68 met inclusion criteria and 43 were able to be pooled. There was a moderate positive correlation between Step 2 CK and ITE scores (0.52, 95% CI 0.45-0.59, P < .01). There was a moderate positive correlation between Step 2 CK and ITE scores for both nonsurgical (0.59, 95% CI 0.51-0.66, P < .01) and surgical specialties (0.41, 95% CI 0.33-0.48, P < .01). There was a very weak positive correlation between Step 2 CK scores and subjective measures of resident performance (0.19, 95% CI 0.13-0.25, P < .01).

CONCLUSIONS

This study found Step 2 CK scores have a statistically significant moderate positive association with future examination scores and a statistically significant weak positive correlation with subjective measures of resident performance. These findings are increasingly relevant as Step 2 CK scores will likely become more important in resident selection.

The US Residency Selection Process After the United States Medical Licensing Examination Step 1 Pass/Fail Change: Overview for Applicants and Educators

The United States Medical Licensing Examination (USMLE) Step 1, arguably the most significant assessment in the USMLE examination series, changed from a 3-digit score to a pass/fail outcome in January 2022. Given the rapidly evolving body of literature on this subject, this paper aims to provide a comprehensive review of the historical context and impact of this change on various stakeholders involved in residency selection. For this, relevant keyword-based searches were performed in PubMed, Google Scholar, and Scopus to identify relevant literature. Given the unique history of USMLE Step 1 in the US residency selection process and the score's correlation with future performance in board-certifying examinations in different specialties, this scoring change is predicted to significantly impact US Doctor of Medicine students, US Doctor of Osteopathic Medicine students, international medical graduates, and residency program directors, among others. The significance and the rationale of the pass/fail change along with the implications for both residency applicants and educators are also summarized in this paper. Although medical programs, academic institutions, and residency organizing bodies across the United States have swiftly stepped up to ensure a seamless transition and have attempted to ensure equity for all, the conversion process carries considerable uncertainty for residency applicants. For educators, the increasing number of applications conflicts with holistic application screening, leading to the expected greater use of objective measures, with USMLE Step 2 Clinical Knowledge likely becoming the preferred screening tool in lieu of Step 1.

A comparison of orthopaedic surgery and internal medicine perceptions of USMLE Step 1 pass/fail scoring

BACKGROUND

United States Medical Licensing Examination Step 1 will transition from numeric grading to pass/fail, sometime after January 2022. The aim of this study was to compare how program directors in orthopaedics and internal medicine perceive a pass/fail Step 1 will impact the residency application process.

METHODS

A 27-item survey was distributed through REDCap to 161 U.S. orthopaedic residency program directors and 548 U.S. internal medicine residency program directors. Program director emails were obtained from the American Medical Association's Fellowship and Residency Electronic Interactive Database.

RESULTS

We received 58 (36.0%) orthopaedic and 125 (22.8%) internal medicine program director responses. The majority of both groups disagree with the change to pass/fail, and felt that the decision was not transparent. Both groups believe that the Step 2 Clinical Knowledge exam and clerkship grades will take on more importance. Compared to internal medicine PDs, orthopaedic PDs were significantly more likely to emphasize research, letters of recommendation from known faculty, Alpha Omega Alpha membership, leadership/extracurricular activities, audition elective rotations, and personal knowledge of the applicant. Both groups believe that allopathic students from less prestigious medical schools, osteopathic students, and international medical graduates will be disadvantaged. Orthopaedic and internal medicine program directors agree that medical schools should adopt a graded pre-clinical curriculum, and that there should be a cap on the number of residency applications a student can submit.

CONCLUSION

Orthopaedic and internal medicine program directors disagree with the change of Step 1 to pass/fail. They also believe that this transition will make the match process more difficult, and disadvantage students from less highly-regarded medical schools. Both groups will rely more heavily on the Step 2 clinical knowledge exam score, but orthopaedics will place more importance on research, letters of recommendation, Alpha Omega Alpha membership, leadership/extracurricular activities, personal knowledge of the applicant, and audition electives.

Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training

AIM

The study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination.

METHODS

In this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05.

RESULTS

Mean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1.

CONCLUSIONS

The change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant's future performance in residency.