The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence From a Three-Year National Study

Simulation to support transitions in the careers of surgical trainees and surgeons

Transitions during the careers of surgical trainees and surgeons may be associated with risks that have the potential to negatively impact delivery of safe and effective patient care and the professional standing of individuals involved in delivering care. Simulation-based education interventions that address specific needs during the transitions can be very helpful in mitigating the risks. These interventions should be based on contemporary educational frameworks and strategies relating to teaching, learning, and assessment. Simulation-based education programs should be used to address the broad range of cognitive, technical, and nontechnical skills, and highly individualized educational interventions, such as preceptoring, proctoring, mentoring, and coaching, employed to yield the best outcomes. The programs should include competency-based or mastery-based methods to ensure that surgical trainees achieve proficiency and surgeons achieve expertise leading to mastery, which should be the aspirational goal for all surgeons. Faculty development is key to supporting these educational endeavors. The American College of Surgeons Division of Education has developed and launched a spectrum of innovative simulation-based education programs that address transitions in the careers of surgical trainees and surgeons. These programs focus specifically on the transition from medical school to surgery residency, transitions during the core period of surgery training, the transition from surgery training to surgical practice, and transitions during the core period of surgical practice. An innovative program to accredit simulation centers has been established and is aimed at developing and implementing effective simulation-based education models, promoting collaboration among simulation educators and across institutions, advancing the field of surgical simulation through scholarship, and training future leaders in surgical simulation. These accredited simulation centers are called American College of Surgeons-accredited Education Institutes. An innovative American College of Surgeons Academy of Master Surgeon Educators has been established to recognize preeminent surgeon educators and engage them in advancing surgical education and providing mentorship to mid-career surgeon educators. Surgical simulation is a major domain of focus for the Academy. The simulation-based education programs of the Division of Education aim to promote excellence in surgery and continue to establish new standards and benchmarks.

A study of internal structure validity for the American board of surgery in training examination

BACKGROUND

As principles of competency-based medical education are implemented into graduate medical education [GME] programs, it is imperative that the assessments employed are reliable and valid. Internal structure is a core component of validity evidence that has been under studied. In this study, we examined elements of the internal structure for the American Board of Surgery In-Training Examination [ABSITE].

METHODS

This national retrospective cohort study utilized performance data for general surgery residents from 2018 to 2023 to determine aspects of the internal structure for the ABSITE. The item difficulty, item discrimination, internal consistency, and dimensionality of the exam were calculated.

RESULTS

55,986 unique test results were obtained in the study. The ABSITE exhibits strong reliability, (Cronbach's alpha >0.9), and appropriate item difficulty (0.66) and item discrimination index (>0.2) for a formative examination. Results of the exploratory factor analysis reveals that the ABSITE is unidimensional.

CONCLUSIONS

The ABSITE is a unidimensional examination with strong reliability. Furthermore, the item difficulty and item discrimination levels of the exam are appropriate for a formative test. Future studies utilizing consensus methods could be employed to determine what specific construct is measured by the ABSITE.

Piloting a National Curricular Resource for the Transition to Surgical Residency: Characteristics of Participating Schools and Their Students

OBJECTIVE

To determine the extent to which U.S. Liaison Committee of Medical Education (LCME)-accredited medical schools chose to participate in piloting a national curricular resource, the American College of Surgeons/Association of Program Directors in Surgery/Association of Surgical Education Resident Prep Curriculum ("ACS-surgery-prep curriculum"), and implications of such participation for student access nationally to this resource.

DESIGN

We examined the significance of school-level differences in ACS-surgery-prep curriculum pilot participation and student-level differences in curriculum access based on medical school attended in bivariate analysis.

SETTING

U.S. medical schools choosing to participate in the ACS-surgery-prep curriculum through 2021. Students graduating from U.S. LCME-accredited medical schools in 2020-2021 were invited to complete the Association of American Medical Colleges 2021 Graduation Questionnaire (GQ).

PARTICIPANTS

Our study included data for 2569 students intending surgery specialties (16% of 16,353 2021 GQ respondents) from ACS-surgery-prep curriculum pilot and non-pilot schools.

RESULTS

Of 148 medical schools attended by 2021 GQ respondents, 93 (63%) were identified as ACS-surgery-prep curriculum pilot schools. Pilot participation varied by school region, community-based designation, and research intensity (each p < 0.05) but not by ownership or transition to residency (TTR) course requirements (each p > 0.05). Of 2569 GQ respondents nationally intending surgery specialties, 1697 (66%) attended an ACS-surgery-prep curriculum pilot school; this proportion did not vary by gender or race/ethnicity (each p > 0.05) but varied by students' school TTR course requirements (p < 0.001). Findings were similar among the 1059 students intending general surgery specialties specifically (41% of all 2569 students intending surgery specialties).

CONCLUSIONS

Many U.S. LCME-accredited medical schools piloted this national TTR surgery curriculum. School-level characteristics associated with pilot participation can inform outreach efforts to encourage the participation of interested schools in piloting this TTR resource. With this curriculum distribution model, we observed no gender or racial/ethnic disparities in curriculum access nationally among students intending surgery specialties.

Transition to Residency: National Study of Factors Contributing to Variability in Learner Milestones Ratings in Emergency Medicine and Family Medicine

PURPOSE

The developmental trajectory of learning during residency may be attributed to multiple factors, including variation in individual trainee performance, program-level factors, graduating medical school effects, and the learning environment. Understanding the relationship between medical school and learner performance during residency is important in prioritizing undergraduate curricular strategies and educational approaches for effective transition to residency and postgraduate training. This study explores factors contributing to longitudinal and developmental variability in resident Milestones ratings, focusing on variability due to graduating medical school, training program, and learners using national cohort data from emergency medicine (EM) and family medicine (FM).

METHOD

Data from programs with residents entering training in July 2016 were used (EM: n=1,645 residents, 178 residency programs; FM: n=3,997 residents, 487 residency programs). Descriptive statistics were used to examine data trends. Cross-classified mixed-effects regression were used to decompose variance components in Milestones ratings.

RESULTS

During postgraduate year (PGY)-1, graduating medical school accounted for 5% and 6% of the variability in Milestones ratings, decreasing to 2% and 5% by PGY-3 for EM and FM, respectively. Residency program accounted for substantial variability during PGY-1 (EM=70%, FM=53%) but decreased during PGY-3 (EM=62%, FM=44%), with greater variability across training period in patient care (PC), medical knowledge (MK), and systems-based practice (SBP). Learner variance increased significantly between PGY-1 (EM=23%, FM=34%) and PGY-3 (EM=34%, FM=44%), with greater variability in practice-based learning and improvement (PBLI), professionalism (PROF), and interpersonal communication skills (ICS).

CONCLUSIONS

The greatest variance in Milestone ratings can be attributed to the residency program and to a lesser degree, learners, and medical school. The dynamic impact of program-level factors on learners shifts during the first year and across the duration of residency training, highlighting the influence of curricular, instructional, and programmatic factors on resident performance throughout residency.

Bias in Surgical Residency Evaluations: A Scoping Review

OBJECTIVE

Given widespread disparities in the surgical workforce and the advent of competency-based training models that rely on objective evaluations of resident performance, this review aims to describe the landscape of bias in the evaluation methods of residents in surgical training programs in the United States.

DESIGN

A scoping review was conducted within PubMed, Embase, Web of Science, and ERIC in May 2022, without a date restriction. Studies were screened and reviewed in duplicate by 3 reviewers. Data were described descriptively.

SETTING/PARTICIPANTS

English-language studies conducted in the United States that assessed bias in the evaluation of surgical residents were included.

RESULTS

The search yielded 1641 studies, of which 53 met inclusion criteria. Of the included studies, 26 (49.1%) were retrospective cohort studies, 25 (47.2%) were cross-sectional studies, and 2 (3.8%) were prospective cohort studies. The majority included general surgery residents (n = 30, 56.6%) and nonstandardized examination modalities (n = 38, 71.7%), such as video-based skills evaluations (n = 5, 13.2%). The most common performance metric evaluated was operative skill (n = 22, 41.5%). Overall, the majority of studies demonstrated bias (n = 38, 73.6%) and most investigated gender bias (n = 46, 86.8%). Most studies reported disadvantages for female trainees regarding standardized examinations (80.0%), self-evaluations (73.7%), and program-level evaluations (71.4%). Four studies (7.6%) assessed racial bias, of which all reported disadvantages for trainees underrepresented in surgery.

CONCLUSIONS

Evaluation methods for surgery residents may be prone to bias, particularly with regard to female trainees. Research is warranted regarding other implicit and explicit biases, such as racial bias, as well as for nongeneral surgery subspecialties.