Challenges Associated With Moving the United States Medical Licensing Examination (USMLE) Step 1 to After the Core Clerkships and How to Approach Them

How a Team Effectiveness Approach to Health Systems Science Can Illuminate Undergraduate Medical Education Outcomes

ABSTRACT

Health care delivery requires physicians to operate in teams to successfully navigate complexity in caring for patients and communities. The importance of training physicians early in core concepts of working in teams (i.e., "teaming") has long been established. Over the past decade, however, little evidence of team effectiveness training for medical students has been available. The recent introduction of health systems science as a third pillar of medical education provides an opportunity to teach and prepare students to work in teams and achieve related core competencies across the medical education continuum and health care delivery settings. Although educators and health care system leaders have emphasized the teaching and learning of team-based care, conceptual models and evidence that inform effective teaming within all aspects of undergraduate medical education (including classroom, clinical, and community settings) are needed to advance the science regarding learning and working in teams. Anchoring teaming through the core foundational theory of team effectiveness and its operational components could catalyze the empirical study of medical student teams, uncover modifiable factors that lead to the evidence for improved student learning, and improve the link among competency-based assessments between undergraduate medical education and graduate medical education. In this article, authors articulate several implications for medical schools through 5 conceptual areas: admissions, the design and teaching of team effectiveness in health systems science curricula, the related competency-based assessments, and course and program evaluations. The authors then discuss the relevance of the measurable components and intended outcomes to team effectiveness in undergraduate medical education as critical to successfully prepare students for teaming in clerkships and eventually residency and clinical practice.

Designing a Shortened Preclinical Basic Science Curriculum: Expert-Derived Recommendations

PURPOSE

To generate an expert-derived list of recommendations for how medical schools should approach decisions about the placement of basic science topics within shortened preclinical curricula, which allow for early clinical immersion.

METHOD

A modified Delphi process was used to develop consensus on recommendations (March-November 2021). The authors performed semistructured interviews with national undergraduate medical education (UME) experts from institutions that previously underwent curricular reforms involving shortened preclinical curricula to elicit perspectives on how decisions were made at their institutions. The authors condensed the findings into a preliminary list of recommendations and distributed this list in 2 survey rounds to a larger group of national UME experts (from institutions that previously underwent curricular reforms or held positions of authority within national UME organizations) to gauge their level of agreement with each recommendation. Recommendations were revised based on participant comments, and those with at least 70% somewhat or strong agreement after the second survey were included in the final comprehensive list of recommendations.

RESULTS

Interviews were conducted with 9 participants and resulted in 31 preliminary recommendations that were then sent via survey to the 40 recruited participants. Seventeen/40 (42.5%) participants completed the first survey, after which 3 recommendations were removed, 5 were added, and 5 were revised based on comments-resulting in 33 recommendations. Twenty-two/38 (57.9%) participants responded to the second survey, after which all 33 recommendations met inclusion criteria. The authors removed 3 recommendations that did not directly address the curriculum reform process and consolidated the final 30 recommendations into 5 succinct, actionable takeaways.

CONCLUSIONS

This study generated 30 recommendations (summarized by the authors in 5 succinct takeaways) for medical schools designing a shortened preclinical basic science curriculum. These recommendations reinforce the importance of vertically integrating basic science instruction with explicit clinical relevance into all curricular phases.

Improving Student Experiences During USMLE Step 1 "Dedicated Preparation Period" via a Course Designed to Holistically Support Academic and Wellness Needs

Common medical school practice includes an independent "dedicated preparation period" for the USMLE Step 1 exam during which students have reported considerable stress and anxiety. Carle Illinois College of Medicine sought to improve their students' experience during the dedicated preparation period via a unique course, Phase 1 Synthesis and Summary, designed to provide support for academics and wellness. The strengths of the course were elements that maximized student flexibility, autonomy, wellness, and targeted self-study. The course design reported here may provide a model for other schools to design courses to holistically improve students' preparation for Step 1 and other board exams.

Supplementary Information

The online version contains supplementary material available at 10.1007/s40670-023-01791-2.

Pilot Study: Increasing Confidence in Obstetrics and Gynecology Applicants Through a Prep Course Prior to Audition Rotations

Introduction: As more medical schools schedule the United States Medical Licensing Examination (USMLE) Step 1 after the completion of clinical rotations in their curriculum, students are finding themselves increasingly removed from the clinical setting prior to their audition rotations. In the military, these audition rotations are important for matching into competitive specialties, such as Obstetrics and Gynecology (Ob-Gyn). This pilot study explores the confidence and audition readiness of prospective candidates through their participation in an Ob-Gyn preparatory course.

Methods: Rising fourth year medical students applying for an Ob-Gyn residency attended an in-person review session. It consisted of four interactive presentations addressing rotation resources, labor and delivery triage, postpartum care, laparoscopic anatomy review, and concluded with a resident question and answer panel. These attendees completed a pre- and post-course survey on Google Forms. Categorical answers were recorded as response frequency and Likert scales were converted into a 5-point system for analysis.

Results: A 100% response rate from attendees revealed pre-course 81.8% confidence in performing well on an audition rotation with only 27% feeling prepared in terms of their medical knowledge. After completion of the course, all participants reported increased medical knowledge and recommended the course to other students.

Conclusions: Preparatory courses for Ob-Gyn residency candidates can increase confidence and preparedness for audition rotations and, ultimately, internship. As the residency application process becomes more competitive, departments can take steps such as hosting a preparatory course to best assist their students into matching and provide skills that they can practice heading into internship.

Impact of USMLE Step-1 accommodation denial on US medical schools: A national survey

INTRODUCTION

In 2019, 4.6% of US-MD students self-identified as students with disabilities (SWD); many of these students will require accommodations on the USMLE Step-1 examination. Given the high-stakes nature of Step-1 for medical school advancement and residency match, SWD denied accommodations on Step-1 face considerable consequences. To date no study has investigated the rate of accommodation denial and its impact on medical school operations.

METHODS

To investigate the rate of accommodation denial and evaluate whether Step-1 accommodation denial impacts medical school operations, a 10-question survey was sent to Student Affairs Deans and disability resource professionals at all fully-accredited US-MD granting programs. Two open-ended questions were analyzed using qualitative content analysis.

RESULTS

Seventy-three of the 141 schools responded (52%). In the 2018-2019 academic year, 276 students from 73 schools applied for Step-1 accommodations. Of these, 144 (52%) were denied. Of those denied, 74/144 (51%) were delayed entry into the next phase of curriculum and 110/144 (76%) took the Step-1 exam unaccommodated. Of the 110 who took Step-1 without accommodations, 35/110 (32%) failed the exam, and 4/110 (3%) withdrew or were dismissed following exam failure. Schools reported varied investments of time and financial support for students denied accommodations, with most schools investing less than 20 hours (67%) and less than $1,000.00 (69%). Open-responses revealed details regarding the impact of denial on schools and students including frustration with process; financial and human resources allocation; delay in student progression; lack of resourcing and expertise; and emotional and financial burdens on students.

DISCUSSION

Step-1 accommodation denial has non-trivial financial, operational, and career impacts on medical schools and students alike. The cause of accommodation denial in this population requires further exploration.

Does Delaying the United States Medical Licensing Examination Step 1 to After Clerkships Affect Student Performance on Clerkship Subject Examinations?

Phenomenon: Schools are considering the optimal timing of Step 1 of the United States Medical Licensing Examination (USMLE). Two primary reasons for moving Step 1 after the core clerkships are to promote deeper, more integrated basic science learning in clinical contexts and to better prepare students for the increasingly clinical focus of Step 1. Positioning Step 1 after the core clerkships leverages a major national assessment to drive learning, encouraging students to deepen their basic science knowledge while in the clinical setting. Previous studies demonstrated small increases in Step 1 scores, reductions in failure rates, and similar Step 2 Clinical Knowledge scores when Step 1 was after the clerkships. Some schools that have moved Step 1 reported declines in clinical subject examination (CSE) performance. This may be due to shortened pre-clerkship curricula, the absence of the Step 1 study period for knowledge consolidation, or exposure to fewer National Board of Medical Examiners type questions prior to taking CSEs. This multi-institutional study aimed to determine whether student performance on CSEs was affected by moving Step 1 after the core clerkships. Approach: CSE scores for students from eight schools that moved Step 1 after core clerkships between 2012 and 2016 were analyzed in a pre-post format. Hierarchical linear modeling was used to quantify the effect of the curriculum on CSE performance. Additional analysis determined if clerkship order impacted clinical subject exam performance and whether the curriculum change resulted in more students scoring in the lowest percentiles (as defined as below the national fifth percentile) before and after the curricular change. Findings: After moving Step 1 to after the clerkships, collectively these eight schools demonstrated statistically significant lower performance on four CSEs (Medicine, Neurology, Pediatrics, and Surgery) but not Obstetrics/Gynecology or Psychiatry. Comparing performance within the three years pre and post Step 1 change, differences across all clerkships ranged from 0.3 to -2.0 points, with an average difference of -1.1. CSE performance in clerkships taken early in the sequence was more affected by the curricular change, and differences gradually disappeared with subsequent examinations. Medicine and Neurology showed the largest average differences between curricular-group when taken early in the clinical year. Finally, there was a slightly higher chance of scoring below the national fifth percentile in four of the clinical subject exams (Medicine, Neurology, Pediatrics, and Psychiatry) for the cohort with Step 1 after the clerkships. Insights: Moving Step 1 after core clerkships had a small impact on CSE scores overall, with decreased scores for exams early in the clerkship sequence and an increased number of students below the fifth percentile. Score differences have minor effects on clerkship grades, but overall the size of the effect is unlikely to be educationally meaningful. Schools can use a variety of mitigation strategies to address CSE performance and Step 1 preparation in the clerkship phase.

The Optimal Timing of Step 1 in Medical Education Following the Transition to Pass/Fail: A Unique Perspective from Post-clerkship Step 1 Schools

The National Board of Medical Examiners' decision to change Step 1 of the United States Medical Licensing Examination (USMLE) from a three-digit score to Pass/Fail (P/F) represents a disruptive change for students, faculty, and leaders in the academic community. In the context of this change, some schools may re-consider the optimal timing of Step 1 as they strive to align their assessment practices with sound educational principles. Currently, over 20 schools administer USMLE Step 1 after the core clerkships. In this commentary, we review the educational rationale for a post-clerkship Step 1, highlighting how adult learning theories support this placement. We discuss some short-term challenges post-clerkship Step 1 schools may encounter due to the proposed timing of the change in scoring, which creates three unique scenarios for learners that can introduce inequity in the system and provoke anxiety. We review outcomes of potentially heightened importance when Step 1 is P/F, including lower clinical subject exam scores in some clerkships, lower failure rates on Step 1 and stable Step 2 Clinical Knowledge scores with implications for the residency match. We outline the future potential for performance-based time-variable Step 1 study periods that are facilitated by post-clerkship placement of the exam. Finally, we discuss opportunities to achieve the goal of enhancing student well-being, which was a major rationale for eliminating the three-digit score.

Identifying and Supporting Students to Prevent USMLE Step 1 Failures When Testing Follows Clerkships: Insights From 9 Schools

Several schools have moved the United States Medical Licensing Examination Step 1 exam after core clerkships, and others are considering this change. Delaying Step 1 may improve Step 1 performance and lower Step 1 failure rates. Schools considering moving Step 1 are particularly concerned about late identification of struggling students and late Step failures, which can be particularly problematic due to reduced time to remediate and accumulated debt if remediation is ultimately unsuccessful. In the literature published to date, little attention has been given to these students. In this article, authors from 9 medical schools with a postclerkship Step 1 exam share their experiences. The authors describe curricular policies, early warning and identification strategies, and interventions to enhance success for all students and struggling students in particular. Such learners can be identified by understanding challenges that place them "at risk" and by tracking performance outcomes, particularly on other standardized assessments. All learners can benefit from early coaching and advising, mechanisms to ensure early feedback on performance, commercial study tools, learning specialists or resources to enhance learning skills, and wellness programs. Some students may need intensive tutoring, neuropsychological testing and exam accommodations, board preparation courses, deceleration pathways, and options to postpone Step 1. In rare instances, a student may need a compassionate off-ramp from medical school. With the National Board of Medical Examiner's announcement that Step 1 scoring will change to pass/fail as early as January 2022, residency program directors might use failing Step 1 scores to screen out candidates. Institutions altering the timing of Step 1 can benefit from practical guidance by those who have made the change, to both prevent Step 1 failures and minimize adverse effects on those who fail.

Beyond High-Stakes Testing: Learner Trust, Educational Commodification, and the Loss of Medical School Professionalism

With ever-growing emphasis on high-stakes testing in medical education, such as the Medical College Admission Test and the United States Medical Licensing Examination Step 1, there has been a recent surge of concerns on the rise of a "Step 1 climate" within U.S. medical schools. The authors propose an alternative source of the "climate problem" in current institutions of medical education. Drawing on the intertwined concepts of trust and professionalism as organizational constructs, the authors propose that the core problem is not hijacking-by-exam but rather a hijackable learning environment weakened by a pernicious and under-recognized tide of commodification within the U.S. medical education system. The authors discuss several factors contributing to this weakening of medicine's control over its learning environments, including erosion of trust in medical school curricula as adequate preparation for entry into the profession, increasing reliance on external profit-driven sources of medical education, and the emergence of an internal medical education marketplace. They call attention to breaches in the core tenets of a profession-namely a logic that differentiates its work from market and managerial forces, along with related slippages in discretionary decision making. The authors suggest reducing reliance on external performance metrics (high-stakes exams and corporate rankings), identifying and investing in alternative metrics that matter, abandoning the marketization of medical education "products," and attending to the language of educational praxis and its potential corruption by market and managerial lexicons. These steps might salvage some self-governing independence implied in the term "profession" and make possible (if not probable) a recovery of a public trust becoming of the term and its training institutions.

Effects of Moving the United States Medical Licensing Examination Step 1 After Core Clerkships on Step 2 Clinical Knowledge Performance

PURPOSE

To investigate the effect of a change in the United States Medical Licensing Examination Step 1 timing on Step 2 Clinical Knowledge (CK) scores, the effect of lag time on Step 2 CK performance, and the relationship of incoming Medical College Admission Test (MCAT) score to Step 2 CK performance pre and post change.

METHOD

Four schools that moved Step 1 after core clerkships between academic years 2008-2009 and 2017-2018 were analyzed. Standard t tests were used to examine the change in Step 2 CK scores pre and post change. Tests of differences in proportions were used to evaluate whether Step 2 CK failure rates differed between curricular change groups. Linear regressions were used to examine the relationships between Step 2 CK performance, lag time and incoming MCAT score, and curricular change group.

RESULTS

Step 2 CK performance did not change significantly (P = .20). Failure rates remained highly consistent (pre change: 1.83%; post change: 1.79%). The regression indicated that lag time had a significant effect on Step 2 CK performance, with scores declining with increasing lag time, with small but significant interaction effects between MCAT and Step 2 CK scores. Students with lower incoming MCAT scores tended to perform better on Step 2 CK when Step 1 was after clerkships.

CONCLUSIONS

Moving Step 1 after core clerkships appears to have had no significant impact on Step 2 CK scores or failure rates, supporting the argument that such a change is noninferior to the traditional model. Students with lower MCAT scores benefit most from the change.

Fundamental Curriculum Change with 1-Year Pre-Clerkship Phase and Effect on Stress Associated with Residency Specialty Selection

Vanderbilt University School of Medicine recently changed from 2 pre-clerkship years (Traditional curriculum) to a 1.5-year pre-clerkship phase for one class (Hybrid curriculum) to a 1-year pre-clerkship phase (Curriculum 2.0). This study investigated the relationship between shortened pre-clerkship training and stress associated with selecting a residency field. The surveyed graduating medical student population included one cohort from the Traditional and Hybrid curricula, and the first two cohorts from Curriculum 2.0. The authors modeled recollected stress levels using a Zero-Inflated Linear Mixed Model with additional covariate and random effects adjustments. Specialty decision-related stress levels increased in the Hybrid curriculum by 10.208 points [p = 0.0115, 95% CI 2.293, 18.122] on a 0-100 point scale. Curriculum 2.0 students had an insignificant increase in stress of 4.062 points [p = 0.304, 95% CI - 3.690, 11.814] relative to the Traditional curriculum. Time since starting medical school and time when a specialty was chosen were the largest factors associated with stress. While this study only evaluated a single facet of the potential downstream effects of curricular change, these data should inspire confidence for reform efforts as a significant increase in specialty decision-related stress present in Hybrid curriculum resolved in both cohorts of Curriculum 2.0.

Post-Carnegie II curricular reform: a north American survey of emerging trends & challenges

BACKGROUND

In 2010, coincident with the 100th anniversary of Flexner's sentinel report, the Carnegie Foundation published an updated review of North American medical education and challenged medical schools to initiate further educational reforms. Specific recommendations pertained to a) ensuring standardized outcomes while allowing for individualized processes, b) integrating foundational knowledge with clinical experience, c) cultivating habits of inquiry and innovation and d) professional identity formation. As we approach the 10-year anniversary of this latest report, we sought to determine what type of curricular revisions have been emerging within the past decade and what types of challenges have been encountered along the way?

METHODS

In 2018, an electronic survey was sent to all 166 Liaison Committee on Medical Education (LCME) accredited North American Medical Schools, using the points of contact (educational deans) that were listed in a publicly available, Association of American Medical Colleges database. Free text comments were grouped into themes using the constant-comparative technique.

RESULTS

Sixty unique responses yielding a 36.14% response rate. The distribution of responses was proportionally representative of the distribution of public vs. private, old vs. new vs. established North American medical schools. Self-reported curricular changes aggregated into five main themes: Changes in curricular structure/organization, changes in curricular content, changes in curricular delivery, changes in assessment, and changes involving increased use of technology/informatics. Challenges were predominantly focused on overcoming faculty resistance, faculty development, securing adequate resourcing, change management, and competition for limited amounts of curricular time.

CONCLUSIONS

Changes in curricular organization, content, delivery, assessment and the use of technology reflect reforms that are broad and deep. Empowering faculty to "let go" of familiar constructs/processes requires strong leadership, particularly when initiating particularly disruptive curricular changes, such as relocating the Step 1 examination or shifting to a competency-based curriculum. While North American medical schools are responding to the calls for action described in the second (2010) Carnegie Foundation report, the full vision has yet to be achieved.