USMLE Step 1 Is Pass/Fail - Should the ABSITE Follow Suit?

The United States Medical Licensing Examination (USMLE) Step 1 was recently made pass/fail. This decision was controversial largely because of the reliance on USMLE Step 1 scores in resident selection. However, these scores do not correlate with resident ability. In this manuscript, we consider if the American Board of Surgery In-Training Examination (ABSITE) should be pass/fail as well. The ABSITE has been used for "high-stakes" purposes, such as preliminary resident advancement and prospective fellow evaluation, for which it was not intended. Moreover, similar to the USMLE Step 1 exam, ABSITE scores have demonstrated no correlation with clinical ability. A pass/fail ABSITE would return the exam to its original purpose and minimize an over-reliance on scores. Moving forward, new objective measures will need to be developed to assess surgical trainees in a more holistic manner.

Can Deficiencies in Performance Be Identified Earlier in Surgical Residency? An Initial Report of a Surgical Trainee Assessment of Readiness Exam

OBJECTIVE

Identifying gaps in medical knowledge, patient management, and procedural competence is difficult early in surgical residency. We designed and implemented an end-of-year examination for our postgraduate year 1 residents, entitled Surgical Trainee Assessment of Readiness (STAR). Our objective in this study was to determine whether STAR scores correlated with other available indicators of resident performance, such as the American Board of Surgery in-training exam (ABSITE) and Milestone scores, and if they provided evidence of additional discriminatory value.

STUDY DESIGN

Overall and component scores of the STAR exam were compared to the ABSITE and Milestone assessment scores for the 17 categorical residents that took the exam in 2016 and 2017.

SETTING

Harbor-UCLA Medical Center, a university-affiliated academic medical center.

PARTICIPANTS

Seventeen categorical general surgery residents.

RESULTS

The STAR Total Test Score (β = 2.77, p = 0.006) was an independent predictor of the ABSITE taken the same year, and components of the STAR were independent predictors of ABSITE taken the following year. The STAR Total Test Score was lowest in the 3 residents who had at least 1 low Milestone score assessed in the same year; and 2 of these 3 residents had at least 1 low Milestone score assigned the next year after STAR. Lastly, the Patient Care 1 and 2 Milestones assessed in the same year as STAR were uniformly scored as appropriate for level of training, yet the corresponding STAR component for those milestones demonstrated 3 residents as having deficiencies.

CONCLUSIONS

We have created a multifaceted standardized STAR exam, which correlates with performance on the ABSITE and early milestone scores. It also appears to discriminate resident performance where milestone assessments do not. Further evaluation of the STAR exam with longer term follow-up is needed to confirm these initial findings.

A Feedback and Evaluation System That Provokes Minimal Retaliation by Trainees

BACKGROUND

Grade inflation is pervasive in educational settings in the United States. One driver of grade inflation may be faculty concern that assigning lower clinical performance scores to trainees will cause them to retaliate and assign lower teaching scores to the faculty member. The finding of near-zero retaliation would be important to faculty members who evaluate trainees.

METHODS

The authors used a bidirectional confidential evaluation and feedback system to test the hypothesis that faculty members who assign lower clinical performance scores to residents subsequently receive lower clinical teaching scores. From September 1, 2008, to February 15, 2013, 177 faculty members evaluated 188 anesthesia residents (n = 27,561 evaluations), and 188 anesthesia residents evaluated 204 faculty members (n = 25,058 evaluations). The authors analyzed the relationship between clinical performance scores assigned by faculty members and the clinical teaching scores received using linear regression. The authors used complete dyads between faculty members and resident pairs to conduct a mixed effects model analysis. All analyses were repeated for three different epochs, each with different administrative attributes that might influence retaliation.

RESULTS

There was no relationship between mean clinical performance scores assigned by faculty members and mean clinical teaching scores received in any epoch (P ≥ 0.45). Using only complete dyads, the authors' mixed effects model analysis demonstrated a very small retaliation effect in each epoch (effect sizes of 0.10, 0.06, and 0.12; P ≤ 0.01).

CONCLUSIONS

These results imply that faculty members can provide confidential evaluations and written feedback to trainees with near-zero impact on their mean teaching scores.