The Feasibility of Real-Time Intraoperative Performance Assessment With SIMPL (System for Improving and Measuring Procedural Learning): Early Experience From a Multi-institutional Trial

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Decision-Making Confidence of Clinical Competency Committees for Entrustable Professional Activities

Importance

A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs' clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known.

Objective

To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions.

Design, Setting, and Participants

This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023.

Exposure

Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle.

Main Outcomes and Measures

The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence.

Results

Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association.

Conclusions and Relevance

In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.

Entrustable Professional Activities in Surgery: A Review

Importance

Entrustable professional activities (EPAs) compose a competency-based education (CBE) assessment framework that has been increasingly adopted across medical specialties as a workplace-based assessment tool. EPAs focus on directly observed behaviors to determine the level of entrustment a trainee has for a given activity of that specialty. In this narrative review, we highlight the rationale for EPAs in general surgery, describe current evidence supporting their use, and outline some of the practical considerations for EPAs among residency programs, faculty, and trainees.

Observations

An expanding evidence base for EPAs in general surgery has provided moderate validity evidence for their use as well as practical recommendations for implementation across residency programs. Challenges to EPA use include garnering buy-in from individual faculty and residents to complete EPA microassessments and engage in timely, specific feedback after a case or clinical encounter. When successfully integrated into a program's workflow, EPAs can provide a more accurate picture of residents' competence for a fundamental surgical task or activity compared with other assessment methods.

Conclusions and Relevance

EPAs represent the next significant shift in the evaluation of general surgery residents as part of the overarching progression toward CBE among all US residency programs. While pragmatic challenges to the implementation of EPAs remain, the best practices from EPA and other CBE assessment literature summarized in this review may assist individuals and programs in implementing EPAs. As EPAs become more widely used in general surgery resident training, further analysis of barriers and facilitators to successful and sustainable EPA implementation will be needed to continue to optimize and advance this new assessment framework.

Impact of Intermittent Intraoperative Neuromonitoring (IONM) on the Learning Curve for Total Thyroidectomy by Residents in General Surgery

INTRODUCTION

Recurrent laryngeal nerve (RNL) identification constitutes the standard in thyroidectomy. Intraoperative nerve monitoring (IONM) has been introduced as a complementary tool for RLN functionality evaluation. The aim of this study is to establish how routine use of IONM can affect the learning curve (LC) in thyroidectomy.

METHODS

Patients undergoing total thyroidectomy performed by surgery residents in their learning curve course in 2 academic hospitals, were divided into 2 groups: Group A, including 150 thyroidectomies performed without IONM by 3 different residents, and Group B, including 150 procedures with routine use of intermittent IONM, by other 3 different residents. LC was measured by comparing operative time (OT), its stabilization during the development of the LC, perioperative complication rate.

RESULTS

As previously demonstrated, the LC was achieved after 30 procedures, in both groups, with no differences due to the use of IONM. Similarly, there were no significant differences among the 2 groups, and between subgroups independently matched, for both OT and complications, even when comparing RLN palsy. Direct nerve visualization and IONM assessment rates were comparable in all groups, and no bilateral RLN palsy (transient or permanent) were reported. No case of interrupted procedure to unilateral lobectomy, due to evidence of RLN injury, was reported.

CONCLUSIONS

The study demonstrates that the use of IONM thyroid surgery, despite requiring a specific training with experienced surgeons, does not particularly affect the learning curve of residents approaching this kind of surgery, and for this reason its routine use should be encouraged even for trainees.

Mortality and Severe Complications Among Newly Graduated Surgeons in the United States

OBJECTIVE

To evaluate severe complications and mortality over years of independent practice among general surgeons.

BACKGROUND

Despite concerns that newly graduated general surgeons may be unprepared for independent practice, it is unclear whether patient outcomes differ between early and later career surgeons.

METHODS

We used Medicare claims for patients discharged between July 1, 2007 and December 31, 2019 to evaluate 30-day severe complications and mortality for 26 operations defined as core procedures by the American Board of Surgery. Generalized additive mixed models were used to assess the association between surgeon years in practice and 30-day outcomes while adjusting for differences in patient, hospital, and surgeon characteristics.

RESULTS

The cohort included 1,329,358 operations performed by 14,399 surgeons. In generalized mixed models, the relative risk (RR) of mortality was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [5.5% (95% CI: 4.1%-7.3%) vs 4.7% (95% CI: 3.5%-6.3%), RR: 1.17 (95% CI: 1.11-1.22)]. Similarly, the RR of severe complications was higher among surgeons in their first year of practice compared with surgeons in their 15th year of practice [7.5% (95% CI: 6.6%-8.5%) versus 6.9% (95% CI: 6.1%-7.9%), RR: 1.08 (95% CI: 1.03-1.14)]. When stratified by individual operation, 21 operations had a significantly higher RR of mortality and all 26 operations had a significantly higher RR of severe complications in the first compared with the 15th year of practice.

CONCLUSIONS

Among general surgeons performing common operations, rates of mortality and severe complications were higher among newly graduated surgeons compared with later career surgeons.

National pilot of entrustable professional activities in pathology residency training

Entrustable professional activities (EPAs) are observable clinical skills and/or procedures that have been introduced into medical education at the student and resident levels in most specialties to determine readiness to advance into residency or independent practice, respectively. This publication describes the process and outcomes of a pilot study looking at the feasibility of using two anatomic pathology and two clinical pathology EPAs in pathology residency in 6 pathology residency programs that volunteered for the study. Faculty development on EPAs and their assessment was provided to pilot program faculty, and EPA assessment tools were developed and used by the pilot programs. Pre- and post-study surveys were given to participating residents, faculty, and program directors to gauge baseline practices and to gather feedback on the EPA implementation experience. Results demonstrated overall good feasibility in implementing EPAs. Faculty acceptance of EPAs varied and was less than that of program directors. Residents reported a significant increase in the frequency with which faculty provided formative assessments that included specific examples of performance and specific ways to improve, as well as increased frequency with which faculty provided summative assessments that included specific ways to improve. EPAs offered the most benefit in setting clear expectations for performance of each task, for providing more specific feedback to residents, and in increasing Program director's understanding of resident strengths abilities and weaknesses.

Evaluation Tools for Assessing Autonomy of Surgical Residents in the Operating Room and Factors Influencing Access to Autonomy: A Systematic Literature Review

BACKGROUND

Surgical residents in France lack a clear pedagogical framework for achieving autonomy in the operating room. The progressive acquisition of surgical autonomy is a determining factor in the confidence of operators for their future independent practice. Currently, there is no autonomy scale commonly used in Europe. The objective of this study is to identify existing tools for quantifying the autonomy of residents and the factors that influence it.

MATERIALS AND METHODS

We conducted a qualitative systematic review following the recommendations of the Systematic Review Without Meta-Analysis (SWiM) guidelines. Publications were extracted from the MEDLINE (PubMed), EMBASE, and PSYCINFO databases. All publications without date restrictions up to July 2022 were identified.

RESULTS

Among the 231 identified publications, 21 met the inclusion criteria. Seventeen publications used a graded autonomy assessment tool by the student and/or the teacher, while 4 used evaluations by an observing third party. We found 8 different autonomy scales, with the Zwisch Scale representing 57.1% of the cases. Factors influencing autonomy were diverse, including the work context, experience, and gender of the resident and their teacher.

DISCUSSION

We found heterogeneity in the tools used to "measure" the autonomy of a resident in the operating room. The SIMPL tool or the Zwisch Scale appear to be the most frequently used tools. The relationship between autonomy, performance, confidence, and knowledge may require multidimensional tools that encompass various areas of competence, but this could make their daily application more challenging. The factors influencing autonomy are numerous; and understanding them would improve teaching in the operating room. There is a significant lack of data on surgical autonomy in France, as well as a lack of evaluation in the field of gynecology-obstetrics worldwide.

Racial and Ethnic Disparities in Operative Experience Among General Surgery Residents: A Multi-Institutional Study from the US ROPE Consortium

OBJECTIVE

To determine the relationship between race/ethnicity and case volume among graduating surgical residents.

BACKGROUND

Racial/ethnic minority individuals face barriers to entry and advancement in surgery; however, no large-scale investigations of the operative experience of racial/ethnic minority residents have been performed.

METHODS

A multi-institutional retrospective analysis of the Accreditation Council for Graduate Medical Education case logs of categorical general surgery residents at 20 programs in the US Resident OPerative Experience Consortium database was performed. All residents graduating between 2010 and 2020 were included. The total, surgeon chief, surgeon junior, and teaching assistant case volumes were compared between racial/ethnic groups.

RESULTS

The cohort included 1343 residents. There were 211 (15.7%) Asian, 65 (4.8%) Black, 73 (5.4%) Hispanic, 71 (5.3%) "Other" (Native American or Multiple Race), and 923 (68.7%) White residents. On adjusted analysis, Black residents performed 76 fewer total cases (95% CI, -109 to -43, P <0.001) and 69 fewer surgeon junior cases (-98 to -40, P <0.001) than White residents. Comparing adjusted total case volume by graduation year, both Black residents and White residents performed more cases over time; however, there was no difference in the rates of annual increase (10 versus 12 cases per year increase, respectively, P =0.769). Thus, differences in total case volume persisted over the study period.

CONCLUSIONS

In this multi-institutional study, Black residents graduated with lower case volume than non-minority residents throughout the previous decade. Reduced operative learning opportunities may negatively impact professional advancement. Systemic interventions are needed to promote equitable operative experience and positive culture change.

How Many Operative Performance Ratings Does a Pediatric Surgery Fellow Need to Be Deemed Practice Ready?

PURPOSE

Identifying the number of cases required for a fellow to achieve competence has been challenging. Workplace-based assessment (WBA) systems make collecting performance data practical and create the opportunity to translate WBA ratings into probabilistic statements about a fellow's likelihood of performing to a given standard on a subsequent assessment opportunity.

METHODS

We compared data from two pediatric surgery training programs that used the performance rating scale from the Society for Improving Medical Professional Learning (SIMPL). We used a Bayesian generalized linear mixed effects model to examine the relationship past and future performance for three procedures: Laparoscopic Inguinal Hernia Repair, Laparoscopic Gastrostomy Tube Placement, and Pyloromyotomy.

RESULTS

For site one, 26 faculty assessed 9 fellows on 16 procedures yielding 1094 ratings, of which 778 (71%) earned practice-ready ratings. For site two, 25 faculty rated 3 fellows on 4 unique procedures yielding 234 ratings of which 151 (65%) were deemed practice-ready. We identified similar model-based future performance expectations, with prior practice-ready ratings having a similar average effect across both sites (Site one, B = 0.25; Site two, B = 0.25). Similar prior practice-ready ratings were needed for Laparoscopic G-Tube Placement (Site one = 13; Site two = 14), while greater differences were observed for Laparoscopic Inguinal Hernia Repair (Site one = 10; Site two = 15) and Pyloromyotomy (Site one = 10; Site two = 15).

CONCLUSION

Our approach to modeling operative performance data is effective at determining future practice readiness of pediatric surgery fellows across multiple faculty and fellow groups. This method could be used to establish minimum case number requirements.

TYPE OF STUDY

Original manuscript, Study of Diagnostic Test.

LEVEL OF EVIDENCE

II.

Variation in Competence of Graduating General Surgery Trainees

OBJECTIVE

To examine the readiness of general surgery residents in their final year of training to perform 5 common surgical procedures based on their documented performance during training.

DESIGN

Intraoperative performance ratings were analyzed using a Bayesian mixed effects approach, adjusting for rater, trainee, procedure, case complexity, and postgraduate year (PGY) as random effects as well as month in academic year and cumulative, procedure-specific performance per trainee as fixed effects. This model was then used to estimate each PGY 5 trainee's final probability of being able to independently perform each procedure. The actual, documented competency rates for individual trainees were then identified across each of the 5 most common general surgery procedures: appendectomy, cholecystectomy, ventral hernia repair, groin hernia repair, and partial colectomy.

SETTING

This study was conducted using data from members of the SIMPL collaborative.

PARTICIPANTS

A total of 17,248 evaluations of 927 PGY5 general surgery residents were analyzed from 2015 to 2021.

RESULTS

The percentage of residents who requested a SIMPL rating during their PGY5 year and achieved a ≥90% probability of being rated as independent, or "Practice-Ready," was 97.4% for appendectomy, 82.4% for cholecystectomy, 43.5% for ventral hernia repair, 24% for groin hernia repair, and 5.3% for partial colectomy.

CONCLUSIONS

There is substantial variation in the demonstrated competency of general surgery residents to perform several common surgical procedures at the end of their training. This variation in readiness calls for careful study of how surgical residents can become more adequately prepared to enter independent practice.

Implementing Workplace-Based Assessments at Scale: The SIMPL Approach

Over the past decade, medical education has shifted from a time-based approach to a competency-based approach for surgical training. This transition presents many new systemic challenges. The Society for Improving Medical Professional Learning (SIMPL) was created to respond to these challenges through coordinated collaboration across an international network of medical educators. The primary goal of the SIMPL network was to implement a workplace-based assessment and feedback platform. To date, SIMPL has developed, implemented, and sustained a platform that represents the earliest and largest effort to support workplace-based assessment at scale. The SIMPL model for collaborative improvement demonstrates a potential approach to addressing other complex systemic problems in medical education.

Assessing operative competence in core surgical training: A reliability analysis

BACKGROUND

This study quantifies the number of observations required to reliably assess the operative competence of Core Surgical Trainees (CSTs) in Ireland, using the Supervised Structured Assessment of Operative Performance (SSAOP) tool.

METHODS

SSAOPs (April 2016-February 2021) were analysed across a mix of undifferentiated procedures, as well as for three commonly performed general surgery procedures in CST: appendicectomy, abdominal wall hernia repair, and skin/subcutaneous lesion excision. Generalizability and Decision studies determined the number of observations required to achieve dependability indices ≥0.8, appropriate for use in high-stakes assessment.

RESULTS

A total of 2,294 SSAOPs were analysed. Four assessors, each observing 10 cases, can generate scores sufficiently reliable for use in high-stakes assessments. Focusing on a selection of core procedures yields more favourable reliability indices.

CONCLUSION

Trainers should conduct repeated assessments across a smaller number of procedures to improve reliability. Programs should increase the assessor mix to yield sufficient dependability indices for high-stakes assessment.

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

OBJECTIVE

Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment.

DESIGN

The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures.

SETTING

This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative.

PARTICIPANTS

A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded.

RESULTS

The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures.

CONCLUSIONS

Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees

INTRODUCTION

Assessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance.

MATERIALS AND METHODS

A web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments.

RESULTS

The Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority.

CONCLUSIONS

This novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.

Implementation of a Smartphone-Based Platform for Operative Feedback at Tenwek Hospital in Kenya: A Mixed Methods Study

BACKGROUND

The SIMPL operative feedback tool is used in many U.S. surgical residency programs. However, the challenges of implementation and benefits of the web-based platform in low- and middle-income countries are unknown. The aim of this study was to evaluate implementation of SIMPL in a general surgery residency training program in Kenya.

METHODS

SIMPL was pilot tested at Tenwek Hospital from January through December 2021. Participant perspectives of SIMPL were elicited through a survey and semi-structured interviews. Descriptive statistics were used to analyze survey data. Inductive qualitative content analysis of interview responses was performed by two independent researchers.

RESULTS

Fourteen residents and six faculty (100% response rate) were included in the study and completed over 600 operative assessments. All respondents reported numerical evaluations and dictated feedback were useful. Respondents felt that SIMPL was easy to use, improved quality and frequency of feedback, helped refine surgical skills, and increased resident autonomy. Barriers to use included participants forgetting to complete evaluations, junior residents not submitting evaluations when minimally involved in cases, and technological challenges. Suggestions for improvement included expansion of SIMPL to surgical subspecialties and allowing senior residents to provide feedback to juniors. All respondents wanted to continue using SIMPL, and 90% recommended use at other programs.

CONCLUSION

Residents and faculty at Tenwek Hospital believed SIMPL were a positive addition to their training program. There were a few barriers to use and suggestions for improvement specific to the training environment in Kenya, but this study demonstrates it is feasible to use SIMPL in settings outside the U.S. with the appropriate resources.

Autonomy Maps: Building a Shared Mental Model for Progressive Resident Operative Autonomy on the Road to Entrustable Professional Activities (EPAs)

Our residents expressed dissatisfaction with operative autonomy and faculty feedback regarding technical skills. They reported variability among faculty regarding allowed operative autonomy. Our goals were to establish a shared mental model among residents and faculty regarding intraoperative performance expectations. We asked faculty to assign a level of expected autonomy (Zwisch scale) for various steps of common procedures according to the resident post-graduate year. Through an iterative process, the maps were standardized across service lines. The resulting "Autonomy Maps" were distributed to the faculty and residents. We held educational sessions and set expectations for use. Selected benchmarks were incorporated into resident end-of-rotation assessment forms. Initial operative case mapping identified variability in faculty expectations for a given post-graduate year and procedure. Residents reported improved satisfaction with understanding expectations regarding operative performance. Establishing autonomy benchmarks facilitated more specific feedback regarding residents' technical skills. Faculty expectations for resident operative autonomy are variable. Autonomy Maps provide structure for a shared mental model between faculty and residents for progressive operative autonomy and serve as a framework for expectations that improve resident satisfaction. Case-specific technical benchmarks are useful tools for assessing residents' technical milestones.

Active control time: an objective performance metric for trainee participation in robotic surgery

Trainee participation and progression in robotic general surgery remain poorly defined. Computer-assisted technology offers the potential to provide and track objective performance metrics. In this study, we aimed to validate the use of a novel metric-active control time (ACT)-for assessing trainee participation in robotic-assisted cases. Performance data from da Vinci Surgical Systems was retrospectively analyzed for all robotic cases involving trainees with a single minimally invasive surgeon over 10 months. The primary outcome metric was percent ACT-the amount of trainee console time spent in active system manipulations over total active time from both consoles. Kruskal-Wallis and Mann-Whitney U statistical tests were applied in analyses. A total of 123 robotic cases with 18 general surgery residents and 1 fellow were included. Of these, 56 were categorized as complex. Median %ACT was statistically different between trainee levels for all case types taken in aggregate (PGY1s 3.0% [IQR 2-14%], PGY3s 32% [IQR 27-66%], PGY4s 42% [IQR 26-52%], PGY5s 50% [IQR 28-70%], and fellow 61% [IQR 41-85%], p =  < 0.0001). When stratified by complexity, median %ACT was higher in standard versus complex cases for PGY5 (60% vs. 36%, p = 0.0002) and fellow groups (74% vs. 47%, p = 0.0045). In this study, we demonstrated an increase in %ACT with trainee level and with standard versus complex robotic cases. These findings are consistent with hypotheses, providing validity evidence for ACT as an objective measurement of trainee participation in robotic-assisted cases. Future studies will aim to define task-specific ACT to guide further robotic training and performance assessments.

Early findings and strategies for successful implementation of SIMPL workplace-based assessments within vascular surgery residency and fellowship programs

OBJECTIVE

As medical education systems increasingly move toward competency-based training, it is important to understand the tools available to assess competency and how these tools are utilized. The Society for Improving Medical Professional Learning (SIMPL) offers a smart phone-based assessment system that supports workplace-based assessment of residents' and fellows' operative autonomy, performance, and case complexity. The purpose of this study was to characterize implementation of the SIMPL app within vascular surgery integrated residency (0+5) and fellowship (5+2) training programs.

METHODS

SIMPL operative ratings recorded between 2018 and 2022 were collected from all participating vascular surgery training institutions (n = 9 institutions with 5+2 and 0+5 programs; n = 4 institutions with 5+2 program only). The characteristics of programs, trainees, faculty, and SIMPL operative assessments were evaluated using descriptive statistics.

RESULTS

Operative assessments were completed for 2457 cases by 85 attendings and 86 trainees, totaling 4615 unique operative assessment ratings. Attendings included dictated feedback in 52% of assessments. Senior-level residents received more assessments than junior-level residents (postgraduate year [PGY]1-3, n = 439; PGY4-5, n = 551). Performance ratings demonstrated increases from junior to senior trainees for both resident and fellow cohorts with "performance-ready" or "exceptional performance" ratings increasing by nearly two-fold for PGY1 to PGY5 residents (28.1% vs 40.6%), and from first- to second-year fellows (PGY6, 46.7%; PGY7, 60.3%). Similar gains in autonomy were demonstrated as trainees progressed through training. Senior residents were more frequently granted autonomy with "supervision only" than junior residents (PGY1, 8.7%; PGY5, 21.6%). "Supervision only" autonomy ratings were granted to 21.8% of graduating fellows. Assessment data included a greater proportion of complex cases for senior compared with junior fellows (PGY6, 20.9% vs PGY7, 26.5%). Program Directors felt that faculty and trainee buy-in were the main barriers to implementation of the SIMPL assessment app.

CONCLUSIONS

This is the first description of the SIMPL app as an operative assessment tool within vascular surgery that has been successfully implemented in both residency and fellowship programs. The assessment data demonstrates expected progressive gains in trainees' autonomy and performance, as well as increasing case complexity, across PGY years. Given the selection of SIMPL as the assessment platform for required American Board of Surgery and Vascular Surgery Board Entrustable Professional Activities assessments, understanding facilitators and barriers to implementation of workplace-based assessments using this app is imperative, particularly as we move toward competency-based medical education.

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs

Assessment is essential to professional development. Assessment provides the information needed to give feedback, support coaching and the creation of individualized learning plans, inform progress decisions, determine appropriate supervision levels, and, most importantly, help ensure patients and families receive high-quality, safe care in the training environment. While the introduction of competency-based medical education has catalyzed advances in assessment, much work remains to be done. First, becoming a physician (or other health professional) is primarily a developmental process, and assessment programs must be designed using a developmental and growth mindset. Second, medical education programs must have integrated programs of assessment that address the interconnected domains of implicit, explicit and structural bias. Third, improving programs of assessment will require a systems-thinking approach. In this paper, the authors first address these overarching issues as key principles that must be embraced so that training programs may optimize assessment to ensure all learners achieve desired medical education outcomes. The authors then explore specific needs in assessment and provide suggestions to improve assessment practices. This paper is by no means inclusive of all medical education assessment challenges or possible solutions. However, there is a wealth of current assessment research and practice that medical education programs can use to improve educational outcomes and help reduce the harmful effects of bias. The authors' goal is to help improve and guide innovation in assessment by catalyzing further conversations.

Preparing for and passing the fundamentals of laparoscopic surgery (FLS) exam improves general surgery resident operative performance and autonomy

BACKGROUND

The American Board of Surgery made the Fundamentals of Laparoscopic Surgery (FLS) exam a prerequisite for board certification in 2009. Some residency programs have questioned the need for a continued FLS testing mandate given limited evidence that supports the impact of FLS on intraoperative skills. The Society for Improving Medical Professional Learning (SIMPL) app is a tool to evaluate resident intraoperative performance. We hypothesized that general surgery resident operative performance would improve immediately after preparing for the FLS exam.

METHODS

The national public FLS data registry was matched with SIMPL resident evaluations from 2015 to 2021 and de-identified. SIMPL evaluations are scored in three categories: supervision required (Zwisch scale 1-4, 1 = show and tell and 4 = supervision only), performance (scale 1-5, 1 = exceptional and 5 = unprepared), and case complexity (scale 1-3, 1 = easiest and 3 = hardest). Statistical analyses compared pre and post-FLS exam resident average operative evaluation scores.

RESULTS

There were a total of 76 general surgery residents, and 573 resident SIMPL evaluations included in this study. Residents required more supervision in laparoscopic cases performed before compared to after the FLS exam (2.84 vs. 3.03, respectively, p = 0.007). Residents performance scores improved from cases before compared to after the FLS exam (2.70 vs. 2.43, respectively, p = 0.001). Case complexity did not differ before versus after the FLS exam (2.13 vs. 2.18, respectively, p = 0.202). PGY level significantly predicted evaluation scores with a moderate correlation. A sub analysis grouped by PGY level revealed a significant improvement after the FLS exam in supervision among PGY-2 residents (2.33 vs. 2.58, respectively, p = 0.04) and performance among PGY-4 residents (2.67 vs 2.04, respectively, p < 0.001).

CONCLUSIONS

Preparation for, and passing, the FLS exam improves resident intraoperative laparoscopic performance and independence. We recommend taking the exam in the first two years of residency to enhance the laparoscopic experience for the remainder of training.

Model-Based Operative Performance Expectations for Quantifying Competency in General Surgery

Importance

Understanding how to translate workplace-based assessment (WBA) ratings into metrics that communicate the ability of a surgeon to perform a procedure would represent a critical advancement in graduate medical education.

Objective

To evaluate the association between past and future performance in a comprehensive assessment system for the purpose of assessing point-in-time competence among general surgery trainees.

Design, Setting, and Participants

This case series included WBA ratings from September 2015 to September 2021 from the WBA system of the Society for Improving Medical Professional Learning (SIMPL) for all general surgery residents who were provided a rating following an operative performance across 70 programs in the US. The study included ratings for 2605 trainees from 1884 attending surgeon raters. Analyses were conducted between September 2021 and December 2021 using bayesian generalized linear mixed-effects models and marginal predicted probabilities.

Exposures

Longitudinal SIMPL ratings.

Main Outcomes and Measures

Performance expectations for 193 unique general surgery procedures based on an individual trainee's prior successful ratings for a procedure, clinical year of training, and month of the academic year.

Results

Using 63 248 SIMPL ratings, the association between prior and future performance was positive (β, 0.13; 95% credible interval [CrI], 0.12-0.15). The largest source of variation was postgraduate year (α, 3.15; 95% CrI, 1.66-6.03), with rater (α, 1.69; 95% CrI, 1.60-1.78), procedure (α, 1.35; 95% CrI, 1.22-1.51), case complexity (α, 1.30; 95% CrI, 0.42-3.66), and trainee (α, 0.99; 95% CrI, 0.94-1.04) accounting for significant variation in practice ready ratings. After marginalizing overcomplexity and trainee and holding rater constant, mean predicted probabilities had strong overall discrimination (area under the receiver operating characteristic curve, 0.81) and were well calibrated.

Conclusions and Relevance

In this study, prior performance was associated with future performance. This association, combined with an overall modeling strategy that accounted for various facets of an assessment task, may offer a strategy for quantifying competence as performance expectations.

A simulation curriculum for laparoscopic common bile duct exploration, balloon sphincterotomy, and endobiliary stenting: Associations with resident performance and autonomy in the operating room

BACKGROUND

Laparoscopic common bile duct exploration is safe and effective for managing choledocholithiasis, but laparoscopic common bile duct exploration is rarely performed, which threatens surgical trainee proficiency. This study tests the hypothesis that prior operative or simulation experience with laparoscopic common bile duct exploration is associated with greater resident operative performance and autonomy without adversely affecting patient outcomes.

METHODS

This longitudinal cohort study included 33 consecutive patients undergoing laparoscopic common bile duct exploration in cases involving postgraduate years 3, 4, and 5 general surgery residents at a single institution during the implementation of a laparoscopic common bile duct exploration simulation curriculum. For each of the 33 cases, resident performance and autonomy were rated by residents and attendings, the resident's prior operative and simulation experience were recorded, and patient outcomes were ascertained from electronic health records for comparison among 3 cohorts: prior operative experience, prior simulation experience, and no prior experience.

RESULTS

Operative approach was similar among cohorts. Overall morbidity was 6.1% and similar across cohorts. The operative performance scores were higher in prior experience cohorts according to both residents (3.0 [2.8-3.0] vs 2.0 [2.0-3.0]; P = .01) and attendings (3.0 [3.0-4.0]; P < .001). The autonomy scores were higher in prior experience cohorts according to both residents (2.0 [2.0-3.0] vs 2.0 [2.0-2.0]; P = .005) and attendings (2.5 [2.0-3.0] vs 2.0 [1.0-2.0]; P = .001). Prior simulation and prior operative experience had similar associations with performance and autonomy.

CONCLUSION

Simulation experience with laparoscopic common bile duct exploration was associated with greater resident operative performance and autonomy, with effects that mimic prior operative experience. This illustrates the potential for simulation-based training to improve resident operative performance and autonomy for laparoscopic common bile duct exploration.

Resident Operative Autonomy and Attending Verbal Feedback Differ by Resident and Attending Gender

Objectives

This study tests the null hypotheses that overall sentiment and gendered words in verbal feedback and resident operative autonomy relative to performance are similar for female and male residents.

Background

Female and male surgical residents may experience training differently, affecting the quality of learning and graduated autonomy.

Methods

A longitudinal, observational study using a Society for Improving Medical Professional Learning collaborative dataset describing resident and attending evaluations of resident operative performance and autonomy and recordings of verbal feedback from attendings from surgical procedures performed at 54 US general surgery residency training programs from 2016 to 2021. Overall sentiment, adjectives, and gendered words in verbal feedback were quantified by natural language processing. Resident operative autonomy and performance, as evaluated by attendings, were reported on 5-point ordinal scales. Performance-adjusted autonomy was calculated as autonomy minus performance.

Results

The final dataset included objective assessments and dictated feedback for 2683 surgical procedures. Sentiment scores were higher for female residents (95 [interquartile range (IQR), 4-100] vs 86 [IQR 2-100]; P < 0.001). Gendered words were present in a greater proportion of dictations for female residents (29% vs 25%; P = 0.04) due to male attendings disproportionately using male-associated words in feedback for female residents (28% vs 23%; P = 0.01). Overall, attendings reported that male residents received greater performance-adjusted autonomy compared with female residents (P < 0.001).

Conclusions

Sentiment and gendered words in verbal feedback and performance-adjusted operative autonomy differed for female and male general surgery residents. These findings suggest a need to ensure that trainees are given appropriate and equitable operative autonomy and feedback.

Surgical resident experience with common bile duct exploration and assessment of performance and autonomy with formative feedback

BACKGROUND

Common bile duct exploration (CBDE) is safe and effective for managing choledocholithiasis, but most US general surgeons have limited experience with CBDE and are uncomfortable performing this procedure in practice. Surgical trainee exposure to CBDE is limited, and their learning curve for achieving autonomous, practice-ready performance has not been previously described. This study tests the hypothesis that receipt of one or more prior CBDE operative performance assessments, combined with formative feedback, is associated with greater resident operative performance and autonomy.

METHODS

Resident and attending assessments of resident operative performance and autonomy were obtained for 189 laparoscopic or open CBDEs performed at 28 institutions. Performance and autonomy were graded along validated ordinal scales. Cases in which the resident had one or more prior CBDE case evaluations (n = 48) were compared with cases in which the resident had no prior evaluations (n = 141).

RESULTS

Compared with cases in which the resident had no prior CBDE case evaluations, cases with a prior evaluation had greater proportions of practice-ready or exceptional performance ratings according to both residents (27% vs. 11%, p = .009) and attendings (58% vs. 19%, p < .001) and had greater proportions of passive help or supervision only autonomy ratings according to both residents (17% vs. 4%, p = .009) and attendings (69% vs. 32%, p < .01).

CONCLUSIONS

Residents with at least one prior CBDE evaluation and formative feedback demonstrated better operative performance and received greater autonomy than residents without prior evaluations, underscoring the propensity of feedback to help residents achieve autonomous, practice-ready performance for rare operations.

Implementation of the SIMPL (Society for Improving Medical Professional Learning) performance assessment tool in the emergency department: A pilot study

Background

Feedback and assessment are difficult to provide in the emergency department (ED) setting despite their critical importance for competency-based education, and traditional end-of-shift evaluations (ESEs) alone may be inadequate. The SIMPL (Society for Improving Medical Professional Learning) mobile application has been successfully implemented and studied in the operative setting for surgical training programs as a point-of-care tool that incorporates three assessment scales in addition to dictated feedback. SIMPL may represent a viable tool for enhancing workplace-based feedback and assessment in emergency medicine (EM).

Methods

We implemented SIMPL at a 4-year EM residency program during a pilot study from March to June 2021 for observable activities such as medical resuscitations and related procedures. Faculty and residents underwent formal rater training prior to launch and were asked to complete surveys regarding the SIMPL app's content, usability, and future directions at the end of the pilot.

Results

A total of 36/58 (62%) of faculty completed at least one evaluation, for a total of 190 evaluations and an average of three evaluations per faculty. Faculty initiated 130/190 (68%) and residents initiated 60/190 (32%) evaluations. Ninety-one percent included dictated feedback. A total of 45/54 (83%) residents received at least one evaluation, with an average of 3.5 evaluations per resident. Residents generally agreed that SIMPL increased the quality of feedback received and that they valued dictated feedback. Residents generally did not value the numerical feedback provided from SIMPL. Relative to the residents, faculty overall responded more positively toward SIMPL. The pilot generated several suggestions to inform the optimization of the next version of SIMPL for EM training programs.

Conclusions

The SIMPL app, originally developed for use in surgical training programs, can be implemented for use in EM residency programs, has positive support from faculty, and may provide important adjunct information beyond current ESEs.

Benefits of an Orthopedic Education Research Collaborative: An Innovative Approach

Background Graduate Medical Education (GME) research in orthopedic surgery is an important but underrepresented subject in the medical literature. It was unknown if orthopedic residency leaders were interested in a surgical education research collaborative (orthopedic collaborative). The objectives of this study were to assess the potential benefit of an orthopedic collaborative from orthopedic residency leaders and investigate the factors associated with the support of a research collaborative within a surgical subspecialty. Methodology An anonymous 19-question survey-based study was distributed through REDCap (Nashville, TN, USA) to orthopedic residency leaders in the United States, from July to October 2020. The main outcome was perceived benefit. Additional aspects included program characteristics, challenges in performing resident education research, and organizational issues such as authorship, frequency of study requests, and governance. Results Almost all orthopedic faculty leadership (99%, 73/74) stated that resident education and faculty development research projects would benefit from an orthopedic education research collaborative. In comparison to unsupportive respondents, younger age (P = 0.006), 15 or fewer years in practice (P = 0.04), and having 0 to 100 peer-reviewed publications (P = 0.047) were associated with support for an orthopedic collaborative. Conclusions Challenges related to survey-based study quality and generalizability at single institutions can benefit from multi-institutional collaboration to develop high-quality studies that capture a representative sample to support orthopedic surgery program development.

Competency-Based Education: Will This be the New Training Paradigm in Plastic Surgery?

The Accreditation Council for Graduate Medical Education created the "Next Accreditation System" in 2013 requiring residents to meet educational milestones based on core competencies over the course of their training. The 6 core competencies include patient care and technical skills, medical knowledge, practice-based learning and improvement, interpersonal and communication skills, professionalism, and systems-based practice. Since the traditional time-based model requires a predetermined length of training irrespective of learning style, pace, or activity, a competency-based model is appealing because it refocuses education on deliberate and relevant skills acquisition and retention. Plastic surgery has been slowly transitioning to competency-based education (CBE), thereby permitting residents to learn at their own pace to master each competency. We performed a nonsystematic literature review of the efficacy of CBE and implementation efforts, particularly within plastic surgery. The literature revealed perceived barriers to implementation, as well as the nuts and bolts of implementation. We highlighted possible solutions and training tools with practical applications in plastic surgery. Success of CBE in plastic surgery requires instituting a transparent process that involves continuously piloting multiple assessment tools and a discussion of related costs. CBE may be particularly appealing for trainees focused on further training in craniofacial or pediatric plastic surgery after completion of an integrated or independent training program in plastic surgery to allow them to focus on their career interests once competence is achieved in the core skills required of a plastic surgeon.

A Resident-Driven Mobile Evaluation System Can Be Used to Augment Traditional Surgery Rotation Evaluations

BACKGROUND

The Accreditation Council for Graduate Medical Education requires residents to receive milestone-based evaluations in key areas. Shortcomings of the traditional evaluation system (TES) are a low completion rate and delay in completion. We hypothesized that adoption of a mobile evaluation system (MES) would increase the number of evaluations completed and improve their timeliness.

METHODS

Traditional evaluations for a general surgery residency program were converted into a web-based form via a widely available, free, and secure application and implemented in August 2017. After 8 months, MES data were analyzed and compared to that of our TES.

RESULTS

122 mobile evaluations were completed; 20% were solicited by residents. Introduction of the MES resulted in an increased number of evaluations per resident (P = .0028) and proportion of faculty completing evaluations (P = .0220). Timeliness also improved, with 71% of evaluations being completed during one's clinical rotation.

CONCLUSIONS

A resident-driven MES is an inexpensive and effective method to augment traditional end-of-rotation evaluations.

The use of an educational time-out in thyroid and parathyroid surgery to move the needle in periprocedural education

BACKGROUND

As surgical training shifts toward a competency-based paradigm, deliberate practice for procedures must be a point of focus. The purpose of this study was to assess the impact of an educational time-out intervention on educational experience and operative performance in endocrine surgery.

METHODS

For 12 months, third-year general surgery residents used the educational time-out to establish an operative step of focus for thyroidectomy and parathyroidectomy procedures. Data were collected using the System for Improving and Measuring Procedural Learning application and post-rotation surveys. The Zwisch scale was used to classify supervision, with meaningful autonomy defined as passive help or supervision only.

RESULTS

Eight residents and 3 attending surgeons performed the educational time-out for a total of 211 operations (93% completion rate). At the end of each rotation, there was improvement in the frequency of goal setting. There was strong agreement (90%) that the intervention strengthened the educational experience. For most cases (52%), the residents were rated at active help. Residents performed a median of 3/6 thyroidectomy steps at meaningful autonomy and a median of 2/5 parathyroidectomy steps at meaningful autonomy. Review of the qualitative data revealed that optimal feedback was provided in 46% of cases.

CONCLUSION

The educational time-out strengthened educational experiences. Stepwise procedural data revealed the varying levels of supervision that exist within an operation. Broader implementation of this intervention could facilitate competency-based procedural education.

Identifying Intraoperative Behaviors Associated With Surgical Resident Teachability

OBJECTIVE

The resident-attending dyad influences the intraoperative training of surgery residents. To better understand the role of trainees within the dyad, we hypothesized there is a measurable concept of "teachability," a combination of the trainee's observed skills and behaviors with their performance. This study aims to define teachability and identify discrete intraoperative behaviors that contribute to this concept. We posit that residents who are active learners as demonstrated by asking questions, proposing next steps, and initiating purposeful actions have higher teachability.

DESIGN, SETTING, PARTICIPANTS

Previously recorded videos from 26 laparoscopic inguinal hernia repairs performed by two PGY-5 general surgery residents at a Midwest tertiary care center were qualitatively reviewed for intraoperative behaviors. A summative content analysis identified behaviors associated with increased teachability and improved operative performance assessment scores.

RESULTS

Average frequencies of intraoperative behaviors for resident 1 and 2 (R1 and R2) were not significantly different, although R2 asked more medical knowledge and technical questions. While the rate of attending feedback was similar for both residents (x=3.82 vs 3.40, p=0.646), R1 consistently incorporated feedback (x=2.27 vs 0.40, p=0.001) whereas R2 needed frequent prompting (x=2.45 vs 1.55, p=0.239). R1 scored higher in all but one operative performance assessment category, including overall performance (x=4.17 vs 2.93, p=0.007), but R2 had a larger magnitude of overall improvement (+1 vs +2).

CONCLUSIONS

Teachability is a dynamic component of the resident-attending dyad. While intraoperative active learning behaviors do not appear to be associated with teachability, asking questions may increase the magnitude of improvement in performance. Most importantly, the ability to incorporate intraoperative feedback in real time seems to be a critical aspect of teachability and warrants further research.

Advancements in Surgical Performance Evaluation and Feedback

PURPOSE OF REVIEW

How today's urology trainees acquire surgical skills has changed dramatically due to multiple forces placing strain on the graduate medical education mission. The development of workplace-based assessments that deliver feedback while capturing performance data has led to a paradigm shift toward individualized learning.

RECENT FINDINGS

Delivering feedback that drives surgical skill development requires the educator to provide a meaningful assessment of the learner after an operative experience. Workplace-based assessment involves direct observation of routine clinical practice and has become a central component of competency-based medical education. Urology has the chance to fully embrace competency-based medical education, employing robust feedback mechanisms and workplace-based assessments. We must first define what it means to be a proficient urologist and design an assessment system that captures this collective sentiment. This can only be done through effective engagement and collaboration with stakeholders across our specialty.

Video Analysis of Otologic Instrument Movement During Resident Mastoidectomies

OBJECTIVE

To measure surgical instrument movement during resident mastoidectomies and identify metrics that correlate with experience.

STUDY DESIGN

Retrospective case series.

SETTING

Tertiary care center.

SUBJECTS

Ten postgraduate year (PGY) 2, 6 PGY3, 7 PGY4, and 19 PGY5 recordings of mastoidectomy performed by otolaryngology residents.

INTERVENTIONS

One-minute intraoperative recordings of mastoidectomies performed during cochlear implantation were collected. Drill and suction-irrigator motion were analyzed with sports motion tracking software.

MAIN OUTCOME MEASURES

Mean instrument speed, angle, and angular velocity were calculated. Mann-Whitney U tests compared mean instrument metrics between PGY levels. Change in drill speed for seven residents between their PGY2 to PGY5 years was individually analyzed.

RESULTS

Mean drill speed was significantly greater for PGY5 residents compared with PGY2s (2.9 versus 1.8 cm/s, p = 0.001). Compared with PGY2 residents, suction speed was greater as a PGY5 (1.2 versus 0.9 cm/s; p = 0.201) and significantly greater as a PGY4 (1.5 versus 0.9 cm/s, p = 0.039). Of the seven residents individually analyzed, group mean drill speed increased by 0.4 cm/s, yearly.

CONCLUSIONS

Drill and suction-irrigator movement during the second minute of drilling of a cortical mastoidectomy seems to increase with resident level. Objective video analysis is a potential adjunct for differentiating novices from more experienced surgeons and monitoring surgical skills progress.

Development of Competencies in Oral and Maxillofacial Surgery Training

The goal of graduate surgical education is to ensure that the graduate is competent to practice in his or her chosen specialty. Traditionally, surgical learning has been based on an apprenticeship model; that is, the long-term observation and assessment of the trainee over a prolonged period of time. Patient expectations, work hour restrictions, and expectations of increased faculty oversight have led to decreased resident autonomy and independence. Graduates completing surgical training with less surgical autonomy may have lower clinical competence, which may affect patient safety, patient outcomes, and career satisfaction. This will require the modification of current assessment and training methods.

Opportunities to Increase Workplace Feedback: A Multi-institutional Examination

OBJECTIVE

While feedback is an essential component of resident education, there are few large-scale studies examining when and under what conditions formative feedback is provided. Workplace-based assessment systems offer an opportunity to identify factors influencing when faculty provides feedback to trainees. Influential factors affecting feedback may provide targets for increasing and improving feedback in resident education.

DESIGN

Data on whether dictated feedback was provided were obtained from the Society for Improving Medical Professional Learning (SIMPL) mobile application. We used generalized linear mixed effects models to identify the degree to which faculty members, procedures, surgical case characteristics, and trainee performance were associated with whether narrative feedback was provided using SIMPL.

SETTING

This study was conducted using data from members of the SIMPL collaborative.

PARTICIPANTS

67,434 evaluations from 70 general surgery programs were included from 2015 to 2021. Of these, 25,355 evaluations included dictated feedback.

RESULTS

Approximately 61% of the variation in whether dictated feedback was provided was attributable to the individual faculty member. Compared to residents who achieved autonomy ratings of "Active Help," residents who achieved ratings of "Supervision Only" (odds ratio (OR) = 0.80, 95% confidence interval (CI) = 0.72, 0.88) had a lower likelihood of receiving dictated feedback. Residents who achieved ratings of "Intermediate" (OR = 0.81, CI = 0.74, 0.89), "Practice-Ready" (OR = 0.50, CI = 0.45, 0.57), or "Exceptional (OR = 0.64, CI = 0.54, 0.76) showed a lower likelihood of receiving dictated feedback compared to those rated as "Inexperienced." Cases rated as "High" in terms of complexity were associated with an increased likelihood of having dictation (OR = 1.35, CI = 1.26, 1.44).

CONCLUSIONS

The largest contributing factor for whether dictated feedback is included in a SIMPL evaluation are factors specific to the attending surgeon. Resident performance, resident autonomy, and case complexity had only modest associations with feedback decisions. Efforts to improve the amount of formative feedback for trainees should be directed towards reducing the variation in which attending surgeons elect to provide feedback.

Validity Evidence for ENTRUST as an Assessment of Surgical Decision-Making for the Inguinal Hernia Entrustable Professional Activity (EPA)

OBJECTIVE

As the American Board of Surgery (ABS) moves toward implementation of Entrustable Professional Activities (EPAs), there is a growing need for objective evaluation of readiness for entrustment of residents. This requires not only assessment of technical skills and knowledge, but also surgical decision-making in preoperative, intraoperative, and postoperative settings. We developed and piloted an Inguinal Hernia EPA Assessment on ENTRUST, a serious game-based online virtual patient simulation platform to assess trainees' decision-making competence.

DESIGN

This is a prospective analysis of resident performance on the ENTRUST Inguinal Hernia EPA Assessment using bivariate analyses.

SETTING

This study was conducted at an academic institution in a proctored exam setting.

PARTICIPANTS

Forty-three surgical residents completed the ENTRUST Inguinal Hernia EPA Assessment.

RESULTS

Four case scenarios for the Inguinal Hernia EPA and corresponding scoring algorithms were iteratively developed by expert consensus aligned with ABS EPA descriptions and functions. ENTRUST Inguinal Hernia Grand Total Score was positively correlated with PGY-level (p < 0.0001). Preoperative, Intraoperative, and Postoperative Total Scores were also positively correlated with PGY-level (p = 0.001, p = 0.006, and p = 0.038, respectively). Total Case Scores were positively correlated with PGY-level for cases representing elective unilateral inguinal hernia (p = 0.0004), strangulated inguinal hernia (p < 0.0001), and elective bilateral inguinal hernia (p = 0.0003). Preoperative Sub-Scores were positively correlated with PGY-level for all cases (p < 0.01). Intraoperative Sub-Scores were positively correlated with PGY-level for strangulated inguinal hernia and bilateral inguinal hernia (p = 0.0007 and p = 0.0002, respectively). Grand Total Score and Intraoperative Sub-Score were correlated with prior operative experience (p < 0.0001). Prior video game experience did not correlate with performance on ENTRUST (p = 0.56).

CONCLUSIONS

Performance on the ENTRUST Inguinal Hernia EPA Assessment was positively correlated to PGY-level and prior inguinal hernia operative performance, providing initial validity evidence for its use as an objective assessment for surgical decision-making. The ENTRUST platform holds potential as tool for assessment of ABS EPAs in surgical residency programs.

Empowering the Resident Learner

Surgical resident autonomy is an essential element of the transition from supervised training to independent practice. However, in an age of duty hour restrictions, legislative constraints, increased litigation, and heightened societal expectations, training an autonomous resident proves increasingly difficult. To tackle these barriers to training successful surgeons, the American College of Surgeons Committee on Resident Education initiated the Resident Mentored Autonomy Project. As a subdivision of this project, the Empowered Learner research team here presents a framework for training the self-directed and empowered surgical resident learner. There are many strategies by which surgical faculty, program directors and chairs, and residents themselves may engage to improve resident operative autonomy.

Gender and Racial/Ethnic Disparities in Operative Volumes of Graduating General Surgery Residents

INTRODUCTION

Gender disparities in resident operative experience have been described; however, their etiology is poorly understood, and racial/ethnic disparities have not been explored. This study investigated the relationship between gender, race/ethnicity, and surgery resident case volumes.

MATERIALS AND METHODS

A retrospective analysis of graduating general surgery resident case logs (2010-2020) at an academic medical center was performed. Self-reported gender and race/ethnicity data were collected from program records. Residents were categorized as underrepresented in medicine (URM) (Black, Hispanic, Native American) or non-URM (White, Asian). Associations between gender and URM status and major, chief, and teaching assistant (TA) mean case volumes were analyzed using t-tests.

RESULTS

The cohort included 80 residents: 39 female (48.8%) and 17 URM (21.3%). Compared to male residents, female residents performed fewer TA cases (33 versus 47, P < 0.001). Compared to non-URM residents, URM residents graduated with fewer major (948 versus 1043, P = 0.008) and TA cases (32 versus 42, P = 0.038). Male URM residents performed fewer TA cases than male non-URM residents (32 versus 50, P = 0.031). Subanalysis stratified by graduation year demonstrated that from 2010 to 2015, female residents performed fewer chief (218 versus 248, P = 0.039) and TA cases (29 versus 50, P = 0.001) than male residents. However, from 2016 to 2020, when gender parity was achieved, no significant associations were observed between gender and case volumes.

CONCLUSIONS

Female and URM residents perform fewer TA and major cases than male non-URM residents, which may contribute to reduced operative autonomy, confidence, and entrustment. Prioritizing gender and URM parity may help decrease case volume gaps among underrepresented residents.

Nontechnical skills for the surgery clerkship in the operating room based on adult learning principles in Taiwan

Medical education has traditionally been rooted in the teaching of medical knowledge and surgical technique in the operating room, with little attention to the nontechnical skills (NTS) that include situational awareness, decision-making, teamwork and communication, and leadership. Under the guidance of adult learning principles, the aim of this study is to evaluate the learning outcomes of the NTS program for the surgery clerkship in the operation room in Taiwan. Self-directed learning principles were practiced during this 2-week subspecialty rotation. The learners were randomly divided into two groups: the control group (the learners and teachers deciding together what should be learnt) and the intervention group (the same as the control group and the formal NTS program implemented during the second week). A 12-item quantitative questionnaire including both the 6-item Accreditation Council for Graduate Medical Education (ACGME) Core Competencies and the 6-item adult learning characteristics was completed twice: before and after rotation. From Oct 2020 to May 2021, 16 medical students received NTS training and 16 were training-naïve. For those undergoing NTS training, six core competencies significantly improved on a 6-point scale. Moreover, total score differences (post-training - pre-training) in core competency had significant correlations to the differences in adult learning characteristics (r = 0.648, p = 0.007). The NTS program could be feasibly integrated into core objectives of the basic surgical clerkships, with the improvement in adult learning motivation in the learners helping to facilitate NTS training under the competency-based surgical training program.

Representativeness of Workplace-Based Operative Performance Assessments for Resident Operative Experience

OBJECTIVE

Workplace-based assessment is increasingly prevalent in surgical education, especially for assessing operative skill. With current implementations, not all observed clinical performances are assessed, in part because trainees often have discretion about when they seek assessment. As a result, these samples of observed operative performances may not be representative of the full breadth of experience of surgical trainees. Therefore, analyses of these samples may be biased. We aimed to benchmark patterns of procedures logged in the SIMPL operative performance assessment system against records of trainee experience in Accreditation Council for Graduate Medical Education (ACGME) case logs.

DESIGN

We analyzed SIMPL longitudinal intraoperative performance assessments from categorical trainees in US general surgery residency programs. We compared overall patterns of how procedures are logged in SIMPL and in ACGME case logs using a Pearson correlation, and we examined differences in how individual procedures are logged in each system using Fisher's exact test.

RESULTS

Total procedure frequency from the SIMPL dataset was strongly correlated with total procedure frequency from ACGME case logs (r = 0.86, 95% CI 0.80-0.90). A subset of these procedures (10 of 116 procedures) was logged more frequently in the SIMPL dataset. These 10 procedures accounted for 56% of SIMPL observations and 30% of ACGME logged cases. Case complexity was comparable for assessments initiated by residents and faculty.

CONCLUSIONS

Samples of intraoperative performance ratings gathered using the SIMPL application largely resemble ACGME case logs. There is no evidence to indicate that residents preferentially select fewer complex cases for assessment.

Assessing Operative Skill in the Competency-based Education Era: Lessons From the UK and Ireland

INTRODUCTION

Decisions regarding the operative competence of surgical residents in the United Kingdom and Ireland are informed by operative workplace-based assessments (WBAs) and operative number targets for index procedures. This review seeks to outline the validity evidence of these assessment methods.

METHODS

A review of the MEDLINE (Pubmed), EMBASE and Cochrane Library databases was undertaken in accordance with the Joanna Briggs Institute Protocol for Scoping Reviews (2020). Articles were included if they provided evidence of the validity of procedure-based assessments, direct observation of procedural skills, or indicative operative number targets. The educational impact of each article was evaluated using a modified Kirkpatrick model.

RESULTS

Twenty-eight articles outlining validity evidence of WBAs and operative number targets were synthesised by narrative review. Five studies documented users' views on current assessment methods (Kirkpatrick level 1). Two articles recorded changes in attitudes towards current operative assessments (level 2a). Ten studies documented the ability of current assessments to record improvements in operative competence (level 2b). Ten studies measured a change in behaviour as a result of the introduction of these assessments (level 3). One article studied the ability of operative assessments to predict clinical outcomes (level 4b).

CONCLUSIONS

Operative WBAs are reliable. Scores achieved correlate with both time spent in training and recorded operative experience. Trainers and residents have concerns regarding the subjectivity of these assessments and the opportunistic nature in which they are used. Operative number targets are not criterion-referenced, lack validity evidence, and may be set too low to ensure operative competence.

Keep it SIMPL: Improved Feedback After Implementation of an App-Based Feedback Tool

BACKGROUND

Data defining the utility of the system for improving and measuring procedural learning (SIMPL) in surgical education is limited. The aim of this pilot study is to describe the impact of SIMPL on resident and faculty perspectives regarding operative feedback.

METHODS

Residents and faculty were surveyed prior to and 6 months after SIMPL implementation. Associations were analyzed using χ2 for categorical and Student's t-test for continuous variables. Statistical significance was defined as P-value < .05.

RESULTS

The proportion of residents receiving intraoperative feedback at least once/day increased significantly (35% to 73%, P = .025); there was a trend toward increased postoperative feedback (15% to 33%, P = .201). Faculty reported an increase in intraoperative (55% to 91%, P = .041) and postoperative feedback (21% to 64%, P = .020). Satisfaction with intraoperative feedback improved from a score of 3.50 ± 1.05 to 3.93 ± .62, although not statistically significant (P = .181). Satisfaction with postoperative feedback improved significantly from 2.85 ± .93 to 3.50 ± .65 (P = .032). The proportion of faculty who felt they were providing effective feedback increased (53% to 91%, P = .032). The proportion of residents who perceived that feedback enhanced their surgical skill increased as well (65% to 93%, P = .048).

DISCUSSION

Despite a modest increase in frequency of postoperative feedback, perceived quality of feedback improved substantially after implementation of SIMPL. Introduction of SIMPL also increased the amount of feedback provided by faculty intraoperatively. SIMPL, via direct and indirect effects, has a positive impact on the resident operative learning environment. Further work is necessary to examine the influence this may have on resident operative skill and patient outcomes.

Closing the Gap: Evaluation of Gender Disparities in Urology Resident Operative Autonomy and Performance

OBJECTIVE

Gender disparities have demonstrated influence on several areas of medical trainee academic performance and surgeon professional attainment. The impact of gender on perceived operative autonomy and performance of urology residents is not well understood. This single-institution pilot study explores this relationship by evaluating urology faculty and resident assessment of resident operative autonomy and performance using the Society for Improving Medical Professional Learning app.

DESIGN

Using Society for Improving Medical Professional Learning, trainees in a single urology residency program were assessed in operative cases on three scales (autonomy, performance, and case complexity). Intraoperative assessments were completed by both faculty and residents (self-evaluation). Respective evaluations were compared to explore differences in ratings by gender.

SETTING

University of Michigan Health, Ann Arbor, MI.

PARTICIPANTS

University of Michigan Urology Residents and Faculty.

RESULTS

A total of 516 evaluations were submitted from 18 urology residents and 20 urology faculty. Self-reported ratings among female and male residents did not differ significantly for autonomy (p = 0.20) or performance (p = 0.82). Female and male residents received overall similar autonomy ratings that were not significantly different from female faculty (p = 0.66) and male faculty (p = 0.81). For female residents, there was no significant difference in performance ratings by faculty gender (p = 0.20). This finding was consistent when the resident was male (p = 0.70).

CONCLUSIONS

At our institution, there is no overall gender-based difference in self-rated or faculty-rated operative autonomy or performance among urology trainees. Understanding relevant facets of institutional culture as well as educational strategies between faculty and residents may identify factors contributing to this outcome.

The Quality of Operative Performance Narrative Feedback: A Retrospective Data Comparison Between End of Rotation Evaluations and Workplace-based Assessments

OBJECTIVE

To describe the quality of operative performance feedback using evaluation tools commonly used by general surgery residency training programs.

SUMMARY OF BACKGROUND DATA

The majority of surgical training programs administer an evaluation through which faculty members may rate and comment on trainee operative performance at the end of the rotation (EOR). Many programs have also implemented the system for improving and measuring procedural learning (SIMPL), a workplace-based assessment tool with which faculty can rate and comment on a trainee's operative performance immediately after a case. It is unknown how the quality of narrative operative performance feedback delivered with these tools compares.

METHODS

The authors collected EOR evaluations and SIMPL narrative comments on trainees' operative performance from 3 university-based surgery training programs during the 2016-2017 academic year. Two surgeon raters categorized comments relating to operative skills as being specific or general and as encouraging and/or corrective. Comments were then classified as effective, mediocre, ineffective, or irrelevant. The frequencies with which comments were rated as effective were compared using Chi-square analysis.

RESULTS

The authors analyzed a total of 600 comments. 10.7% of EOR and 58.3% of SIMPL operative performance evaluation comments were deemed effective (P < 0.0001).

CONCLUSIONS

Evaluators give significantly higher quality operative performance feedback when using workplace-based assessment tools rather than EOR evaluations.

Feedback-Seeking Behavior and Practice Readiness for General Surgery

OBJECTIVE

To analyze the relationship between feedback-seeking behavior, operationalized as the number of trainee-requested evaluations, with ratings of surgical trainees' operative autonomy and performance.

DESIGN

We analyzed operative assessment data using the System for Improving and Measuring Procedural Learning's smartphone-based assessment app called Society for Improving Medical Professional Learning (SIMPL) OR. Using cross-classified mixed effects models, we analyzed the association between trainee-requested SIMPL OR app evaluations and both trainee performance and autonomy ratings. Models included covariates for requested evaluations, PGY-year, month of the academic year, and patient-related case complexity. Random effects for program, procedure, rater, and trainee were also included to account for correlations among evaluations. Only ratings for procedures deemed Core to general surgery were included.

SETTING

Operative assessment data using the SIMPL OR app requested by categorical U.S. general surgery residents between September 2015 to April 2021.

PARTICIPANTS

A total of 61 general surgery residency programs, encompassing 2190 categorical general surgery residents.

RESULTS

A total of 58,104 SIMPL app operative assessments were analyzed. Autonomy scores were weakly but positively associated with number of trainee-requested evaluations (B = 0.002, p < 0.001). Trainee-requested evaluations were also statistically associated with operative performance scores ( B = 0.002, p < 0.001).

CONCLUSIONS

The propensity of a resident to seek feedback using the SIMPL app was weakly associated with higher operative autonomy ratings and higher operative performance ratings. While regular feedback is important for monitoring performance over time, more direct approaches related to the quality of feedback that trainees receive may be needed to better assess the relationships between feedback-seeking behavior and operative autonomy as well as performance.

Are You Thinking What I'm Thinking? Exploring Response Process Validity Evidence for a Workplace-based Assessment for Operative Feedback

OBJECTIVE

Workplace-based assessments (WBAs) are used in multiple surgical specialties to facilitate feedback to residents as a form of formative assessment. The validity evidence to support this purpose is limited and has yet to include investigations of how users interpret the assessment and make rating decisions (response processes). This study aimed to explore the validity evidence based on response processes for a WBA in surgery.

DESIGN

Semi-structured interviews explored the reasonings and strategies used when answering questions in a surgical WBA, the System for Improving and Measuring Procedural Learning (SIMPL). Interview questions probed the interpretation of the three assessment questions and their respective answer categories (level of autonomy, operative performance, case complexity). Researchers analyzed transcripts using directed qualitative content analysis to generate themes.

SETTING

Single tertiary academic medical center.

PARTICIPANTS

Eight residents and 13 faculty within the Department of Otolaryngology-Head and Neck Surgery participating in a 6-month pilot of SIMPL.

RESULTS

We identified four overarching themes that that characterized faculty and resident response processes while completing SIMPL: (1) Faculty and resident users had similar content-level interpretations of the questions and corresponding answer choices; (2) Users employed a variety of cognitive, behavioral, and emotional processes to make rating decisions; (3) Contextual factors influenced ratings; and (4) Tensions during interpretation contributed to rating uncertainty.

CONCLUSIONS

Response processes are a key source of evidence to support the validity for the formative use of WBAs. Evaluating response process evidence should go beyond basic content-level analysis as contextual factors and tensions that arise during interpretation also play a large role in rating decisions. Additional work and a continued critical lens are needed to ensure that WBAs can truly meet the needs for formative assessment.

How Do Resident Surgeons Identify Operative Case Complexity? An Analysis of Resident versus Attending Perceptions

OBJECTIVE

Accurate recognition of patient-related complexity of an operation is critical for appropriate surgical decision making. It is not yet understood whether general surgery residents are able to accurately assess the relative complexity of a given operative case. This study investigates the agreement of case complexity ratings between residents and attending surgeons and explores whether resident-related factors correlate with any discordance in perception of patient-related operative complexity.

DESIGN

Residents and attending surgeons rated the relative complexity of completed cases on a 3 point scale via the SIMPL (Society for Improving Medical Professional Learning) operative assessment smartphone app. Additional trainee demographic data, autonomy ratings, and performance ratings were also obtained from the SIMPL registry for each rated case. Complexity agreement was defined as an equal rating between the resident and attending and assigned a value of zero. Over-estimate ratings were assigned a positive value and under-estimate ratings were assigned a negative value. Trends in complexity agreement were analyzed using descriptive statistics and mixed-effects models.

RESULTS

A total of 43,179 general surgery cases were rated by 1946 categorical general surgery residents and 1520 attending surgeons between 2015 and 2020. Residents and attendings agreed on case complexity in 63.23% of cases, while the residents overestimated complexity in 13.37% of cases and underestimated complexity in 23.40% of cases. Every level of resident except post-graduate year 2 had similar rates of agreement about the complexity of a procedure, while residents who received a higher autonomy rating were more likely to be in agreement with the faculty raters (OR 1.12, 95% CI 1.06-1.19).

CONCLUSIONS

The results of this study suggest that general surgery residents inaccurately perceive the patient-related complexity of a given case approximately one third of the time. Greater experience and operative autonomy appear to be associated with higher complexity agreement. Future research into factors influencing perceived case complexity may provide insight into how to best implement new teaching for surgical residents regarding the concept of case complexity.

Feasibility of Smartphone Application in Plastic Surgery Operative Assessments

Background:

The evaluation model of operative competence is based on aggregate tabulations of procedures and end-of-rotation feedback from faculty members. Procedural tabulations do not detail the level of resident involvement in the case, and end of rotation feedback is infrequent and inaccurate due to the necessity of long-term recall. Smart phone-based evaluation systems provide residents with immediate and permanent feedback for surgical encounters. In this study, we examine the feasibility of smart phone-based evaluations in plastic surgery residency.

Methods:

This was a 6-month prospective, single institution pilot study at three teaching hospitals, assessing all PGY levels. We utilized our department mobile application (Wayne State University Surgery Department application), which includes intraoperative evaluations based on the Zwisch scale. Prestudy and poststudy surveys were conducted. An unstructured interview of the Clinical Competency Committee provided feedback for the new evaluation tool against the previous evaluation forms.

Results:

Eleven physicians participated in the study, resulting in 126 encounters and 184 procedures. A 10-question prestudy survey was given with answers ranging from 1 (strongly disagree) to 5 (strongly agree). The Clinical Competency Committee faculty ranked the prestudy resident assessment tool 2.82 of 5, whereas the poststudy survey scored 4.64 of 5.

Conclusions:

Residents and faculty both rated the smartphone application as a useful tool for evaluating residents. The success of the application proves its feasibility within plastic surgery residency and may play an important role in rating resident operative competency in the future.

Factors affecting operative autonomy and performance during otolaryngology training: A multicenter trial

Objective

Surgical education is changing in an era of new regulations and evolving training cultures. We sought to understand the factors that affect operative experiences during otolaryngology residency.

Methods

From December 2019 to December 2020, five otolaryngology training programs used the SIMPL OR smartphone application to evaluate residents after each operation. Residents and attendings rated the trainee's autonomy on a 4‐level Zwisch scale, performance on a 5‐level scale, and case complexity on a 3‐level scale. We examined associations between ratings of autonomy and performance with variables including postgraduate year (PGY), case complexity, gender, week of the academic year (AY), and whether multiple procedures were logged.

Results

78 attendings and 92 residents logged 2984 evaluations. PGY level and week of the AY were positively associated with attending ratings of autonomy and performance (PGY3 vs. PGY2: B = 0.63, p < .001 for autonomy and B = 1.05, p < .001 for performance; week of the AY: B = 0.013, p = .002 for autonomy; B = 0.025, p < .001 for performance). Multiple procedures logged and increasing case complexity were negatively associated with attending ratings (multiple procedures: B = −0.19, p = .04 for autonomy and B = −0.48, p < .001 for performance; hardest vs. easiest 1/3 of cases: B = −1.01, p < .001 for autonomy and B = −0.59, p < .001 for performance). Attending and trainee genders were not associated with attending ratings of autonomy or performance.

Conclusion

Resident autonomy and performance were positively associated with PGY level and week of the academic year, and negatively associated with case complexity and multiple procedures. These findings highlight the need to align training level with case complexity to promote quality operative experiences.

Level of Evidence

2.