Implementation of Entrustable Professional Activities into a General Surgery Residency

A realist synthesis of prospective entrustment decision making by entrustment or clinical competency committees

INTRODUCTION

The real-world mechanisms underlying prospective entrustment decision making (PEDM) by entrustment or clinical competency committees (E/CCCs) are poorly understood. To advance understanding in this area, the authors conducted a realist synthesis of the published literature to address the following research question: In E/CCC efforts to make defensible prospective entrustment decisions (PEDs), what works, for whom, under what circumstances and why?

METHODS

Realist work seeks to understand the contexts (C), mechanisms (M) and outcomes (O) that explain how and why things work (or do not). In the authors' study, contexts included individual E/CCC members, E/CCC structures and processes, and training programmes. The outcome (i.e. desired outcome) was a PED. Mechanisms were a substantial focus of the analysis and informed the core findings. To define a final corpus of 52 included papers, the authors searched four databases, screened all results from those searches and performed a full-text review of a subset of screened papers. Data extraction focused on developing context-mechanism-outcome configurations from the papers, which were used to create a theory for how PEDM leads to PEDs.

RESULTS

PEDM is often driven by default (non-deliberate) decision making rather than a deliberate process of deciding whether a trainee should be entrusted or not. When defaulting, some E/CCCs find red flags that sometimes lead to being more deliberate with decision making. E/CCCs that seek to be deliberate describe PEDM that can be effortful (when data are insufficient or incongruent) or effortless (when data are robust and tell a congruent story about a trainee). Both information about trainee trustworthiness and the sufficiency of data about trainee performance influence PEDM. Several moderators influence what is considered to be sufficient data, how trustworthiness data are viewed and how PEDM is carried out. These include perceived consequences and associated risks, E/CCC member trust propensity, E/CCC member personal knowledge of and experience with trainees and E/CCC structures and processes.

DISCUSSION

PEDM is rarely deliberate but should be. Data about trainee trustworthiness are foundational to making PEDs. Bias, equity and fairness are nearly absent from the papers in this synthesis, and future efforts must seek to advance understanding and practice regarding the roles of bias, equity and fairness in PEDM.

A Scoping Review of the Literature on Entrustable Professional Activities in Surgery Residency Programs

OBJECTIVE

Entrustable professional activities (EPAs) are a crucial component of contemporary postgraduate medical education with many surgery residency programs having implemented EPAs as a competency assessment framework to assess and provide feedback on the performance of their residents. Despite broad implementation of EPAs, there is a paucity of evidence regarding the impact of EPAs on the learners and learning environments. A first step in improving understanding of the use and impact of EPAs is by mapping the rising number of EPA-related publications from the field of surgery. The primary objective of this scoping review is to examine the nature, extent, and range of articles on the development, implementation, and assessment of EPAs. The second objective is to identify the experiences and factors that influence EPA implementation and use in practice in surgical specialties.

DESIGN

Scoping review. Four electronic databases (Medline, Embase, Education Source, and ERIC) were searched on January 20, 2022, and then again on July 19, 2023. A quasi-statistical content analysis was employed to quantify and draw meaning from the information related to the development, implementation, assessment, validity, reliability, and experiences with EPAs in the workplace.

PARTICIPANTS

A total of 42 empirical and nonempirical articles were included.

RESULTS

Four thematic categories describe the topic areas in included articles related to: 1) the development and refinement of EPAs, including the multiple steps taken to develop and refine unique EPAs for surgery residency programs; 2) the methods for implementing EPAs; 3) outcomes of EPA use in practice; 4) barriers, facilitators, and areas for improvement for the implementation and use of EPAs in surgical education.

CONCLUSIONS

This scoping review highlights the key trends and gaps from the rapidly increasing number of publications on EPAs in surgery residency, from development to their use in the workplace. Existing EPA studies lack a theoretical and/or conceptual basis; future development and implementation studies should adopt implementation science frameworks to better structure and operationalize EPAs within surgery residency programs.

Faculty and Resident Perspectives on the Implementation of Entrustable Professional Activities in General Surgery Residency

OBJECTIVE

This study aimed to identify what best practices facilitate implementation of Entrustable Professional Activities (EPAs) into surgical training programs.

DESIGN

This is a mixed methods study utilizing both survey data as well as semi-structured interviews of faculty and residents involved in the American Board of Surgery (ABS) EPA pilot study.

SETTING

From 2018 to 2020, the ABS conducted a pilot that introduced five EPAs across 28 general surgery training programs.

PARTICIPANTS

All faculty members and residents at the 28 pilot programs were invited to participate in the study.

RESULTS

About 117 faculty members and 79 residents responded to the survey. The majority of faculty (81%) and residents (66%) felt that EPAs were useful and were a valuable addition to training. While neither group felt that EPAs were overly time consuming to complete, residents did report difficulty incorporating them into their daily workflow (44%). Semi-structured interviews found that programs that focused on faculty and resident -development and utilized frequent reminders about the importance and necessity of EPAs tended to perform better.

CONCLUSIONS

EPA implementation is feasible in general surgery training programs but requires significant effort and engagement from all levels of program personnel. As EPAs are implemented by the ABS nationally a focus on resident and faculty development will be critical to success.

Implementing Entrustable Professional Activities in Pediatric Fellowships: Facilitating the Process

BACKGROUND AND OBJECTIVES

Entrustable professional activities (EPAs) will be used for initial certification by the American Board of Pediatrics by 2028. Less than half of pediatric fellowships currently use EPAs for assessment, yet all will need to adopt them. Our objectives were to identify facilitators and barriers to the implementation of EPAs to assess pediatric fellows and to determine fellowship program directors' (FPD) perceptions of EPAs and Milestones.

METHODS

We conducted a survey of FPDs from 15 pediatric subspecialties. EPA users were asked about their implementation of EPAs, barriers encountered, and perceptions of EPAs. Nonusers were queried about deterrents to using EPAs. Both groups were asked about potential facilitators of implementation and their perceptions of Milestones.

RESULTS

The response rate was 65% (575/883). Of these, 344 (59.8%) were EPA users and 231 (40.2%) were nonusers. Both groups indicated work burden as a barrier to implementation. Nonusers reported more barriers than users (mean [SD]: 7 [3.8] vs 5.8 [3.4], P < .001). Both groups identified training materials and premade assessment forms as facilitators to implementation. Users felt that EPAs were easier to understand than Milestones (89%) and better reflected what it meant to be a practicing subspecialty physician (90%). In contrast, nonusers felt that Milestones were easy to understand (57%) and reflected what it meant to be a practicing subspecialist (58%).

CONCLUSIONS

Implementing EPA-based assessment will require a substantial investment by FPDs, facilitated by guidance and easily accessible resources provided by multiple organizations. Perceived barriers to be addressed include FPD time constraints, a need for additional assessment tools, and outcomes data.

Decision-Making Confidence of Clinical Competency Committees for Entrustable Professional Activities

Importance

A competency-based assessment framework using entrustable professional activities (EPAs) was endorsed by the American Board of Surgery following a 2-year feasibility pilot study. Pilot study programs' clinical competency committees (CCCs) rated residents on EPA entrustment semiannually using this newly developed assessment tool, but factors associated with their decision-making are not yet known.

Objective

To identify factors associated with variation in decision-making confidence of CCCs in EPA summative entrustment decisions.

Design, Setting, and Participants

This cohort study used deidentified data from the EPA Pilot Study, with participating sites at 28 general surgery residency programs, prospectively collected from July 1, 2018, to June 30, 2020. Data were analyzed from September 27, 2022, to February 15, 2023.

Exposure

Microassessments of resident entrustment for pilot EPAs (gallbladder disease, inguinal hernia, right lower quadrant pain, trauma, and consultation) collected within the course of routine clinical care across four 6-month study cycles. Summative entrustment ratings were then determined by program CCCs for each study cycle.

Main Outcomes and Measures

The primary outcome was CCC decision-making confidence rating (high, moderate, slight, or no confidence) for summative entrustment decisions, with a secondary outcome of number of EPA microassessments received per summative entrustment decision. Bivariate tests and mixed-effects regression modeling were used to evaluate factors associated with CCC confidence.

Results

Among 565 residents receiving at least 1 EPA microassessment, 1765 summative entrustment decisions were reported. Overall, 72.5% (1279 of 1765) of summative entrustment decisions were made with moderate or high confidence. Confidence ratings increased with increasing mean number of EPA microassessments, with 1.7 (95% CI, 1.4-2.0) at no confidence, 1.9 (95% CI, 1.7-2.1) at slight confidence, 2.9 (95% CI, 2.6-3.2) at moderate confidence, and 4.1 (95% CI, 3.8-4.4) at high confidence. Increasing number of EPA microassessments was associated with increased likelihood of higher CCC confidence for all except 1 EPA phase after controlling for program effects (odds ratio range: 1.21 [95% CI, 1.07-1.37] for intraoperative EPA-4 to 2.93 [95% CI, 1.64-5.85] for postoperative EPA-2); for preoperative EPA-3, there was no association.

Conclusions and Relevance

In this cohort study, the CCC confidence in EPA summative entrustment decisions increased as the number of EPA microassessments increased, and CCCs endorsed moderate to high confidence in most entrustment decisions. These findings provide early validity evidence for this novel assessment framework and may inform program practices as EPAs are implemented nationally.

Creation and Implementation of an Online Tool for Feedback on Resident Teaching: A Pilot Study

OBJECTIVE

There are few assessments of the competence and growth of surgical residents as educators. We developed and piloted an observation-based feedback tool (FT) to provide residents direct feedback during a specific teaching session, as perceived by medical students (MS). We hypothesized that residents' performance would improve with frequent, low stakes, observation-based feedback.

SETTING

This prospective study took place at an academic general surgery program.

PARTICIPANTS

Focus groups of MS, surgical residents, and faculty informed FT development. MS completed the FT regarding resident teaching.

DESIGN

The FT utilized 5 slider-bar ratings (0 to 100) about the teaching encounter and a checklist of 16 desirable teaching behaviors. QR codes and weekly email links were distributed for 12 months (6 clerkship blocks) to promote use. Residents were sent their results after each block. A survey after each block assessed motivation for use and gathered feedback on the FT. Descriptive statistics were used for analysis (medians, IQRs). Primary measures of performance were median of the slider-bar scores and the number of teaching behaviors.

RESULTS

The FT was used 111 times; 37 of 46 residents were rated by up to 65 MS. The median rating on the slider-bars was 100 and the median number of desirable teaching behaviors was 12; there were no differences based on gender or PGY level. 10 residents had 5 or more FT observations during the year. Four residents had evaluations completed in 4 or more blocks and 19 residents had evaluations completed in at least 2 blocks. Over time, 13 residents had consistent slider-bar scores, 1 resident had higher scores, and 5 residents had lower scores (defined as a more than 5-point change from initial rating). Frequency of use of the FT decreased over time (38, 32, 9, 21, 7, 5 uses per block). The post-use survey was completed by 24 MS and 19 residents. Most common reasons for usage were interest in improving surgical learning environment, giving positive feedback (MS), and improving teaching skills (residents). Most common reasons for lack of usage from residents were "I did not think I taught enough to ask for feedback," "I forgot it existed," and "I did not know it existed."

CONCLUSIONS

The FT did not lead to any meaningful improvement in resident scores over the course of the year. This may be due to overall high scores, suggesting that the components of the FT may require reevaluation. Additionally, decreased utilization of the instrument over time made it challenging to assess change in performance of specific residents, likely due to lack of awareness of the FT despite frequent reminders. Successful implementation of observation-based teaching assessments may require better integration with residency or clerkship objectives.

Entrustable Professional Activities in Surgery: A Review

Importance

Entrustable professional activities (EPAs) compose a competency-based education (CBE) assessment framework that has been increasingly adopted across medical specialties as a workplace-based assessment tool. EPAs focus on directly observed behaviors to determine the level of entrustment a trainee has for a given activity of that specialty. In this narrative review, we highlight the rationale for EPAs in general surgery, describe current evidence supporting their use, and outline some of the practical considerations for EPAs among residency programs, faculty, and trainees.

Observations

An expanding evidence base for EPAs in general surgery has provided moderate validity evidence for their use as well as practical recommendations for implementation across residency programs. Challenges to EPA use include garnering buy-in from individual faculty and residents to complete EPA microassessments and engage in timely, specific feedback after a case or clinical encounter. When successfully integrated into a program's workflow, EPAs can provide a more accurate picture of residents' competence for a fundamental surgical task or activity compared with other assessment methods.

Conclusions and Relevance

EPAs represent the next significant shift in the evaluation of general surgery residents as part of the overarching progression toward CBE among all US residency programs. While pragmatic challenges to the implementation of EPAs remain, the best practices from EPA and other CBE assessment literature summarized in this review may assist individuals and programs in implementing EPAs. As EPAs become more widely used in general surgery resident training, further analysis of barriers and facilitators to successful and sustainable EPA implementation will be needed to continue to optimize and advance this new assessment framework.

Leveraging faculty development to support validation of entrustable professional activities assessment tools in anatomic and clinical pathology training

Entrustable professional activities (EPAs) are observable activities that define the practice of medicine and provide a framework of evaluation that has been incorporated into US medical school curricula in both undergraduate and graduate medical education. This manuscript describes the development of an entrustment scale and formative and summative evaluations for pathology EPAs, outlines a process for faculty development that was employed in a pilot study implementing two Anatomic Pathology and two Clinical Pathology EPAs in volunteer pathology residency programs, and provides initial validation data for the proposed pathology entrustment scales. Prior to implementation, faculty development was necessary to train faculty on the entrustment scale for each given activity. A "train the trainer" model used performance dimension training and frame of reference training to train key faculty at each institution. The session utilized vignettes to practice determination of entrustment ratings and development of feedback for trainees as to strengths and weaknesses in the performance of these activities. Validity of the entrustment scale is discussed using the Messick framework, based on concepts of content, response process, and internal structure. This model of entrustment scales, formative and summative assessments, and faculty development can be utilized for any pathology EPA and provides a roadmap for programs to design and implement EPA assessments into pathology residency training.

Analysis of fellowship program director opinions of entrustable professional activities in adolescent medicine fellowship

OBJECTIVES

This study aimed to explore the minimum entrustable professional activity (EPA) supervision levels at which pediatric fellowship program directors (FPDs) would be willing to graduate fellows and the levels deemed necessary for safe and effective practice for each of the common pediatric subspecialty and the four adolescent medicine-specific EPAs.

METHODS

This cross-sectional study utilized survey data from pediatric FPDs in 2017. FPDs indicated the minimum level of supervision (LOS) for fellows at graduation and for safe and effective practice.

RESULTS

82 percent (23/28) of adolescent medicine FPDs completed the survey. For each EPA, there were differences (p<0.05) between LOS expected for graduation and for safe and effective practice. There was also variability in the level at which FPDs would graduate fellows.

CONCLUSIONS

This study summarizes pediatric FPD opinions regarding the minimum levels of supervision required for fellows at the time of graduation as well as the levels deemed necessary for safe and effective practice. The difference between the minimum LOS at which FPDs would graduate a fellow and that deemed appropriate for safe and effective practice, along with variability in minimum LOS for graduation, highlight the need for clearer standards for fellowship graduation as well as more structured early career support for ongoing learning. These data highlight variability in FPD opinion regarding such expectations and both the need to better define desired training outcomes and potential need for post-graduation supervision in clinical practice.

Applying augmented reality in teaching of surgical residents-telementoring, a "stress-free" way to surgical autonomy?

BACKGROUND

Achieving surgical autonomy can be considered the ultimate goal of surgical training. Innovative head-mounted augmented reality (AR) devices enable visualization of the operating field and teaching from remote. Therefore, utilization of AR glasses may be a novel approach to achieve autonomy. The aim of this pilot study is to analyze the feasibility of AR application in surgical training and to assess its impact on intraoperative stress.

METHODS

A head-mounted RealWear Navigator® 500 glasses and the TeamViewer software were used. Initial "dry lab" testing of AR glasses was performed in combination with the Symbionix LAP Mentor™. Subsequently, residents performed various stage-adapted surgical procedures semi-autonomously (SA) (on-demand consultation of senior surgeon, who is in theatre but not scrubbed) versus permanent remote supervision (senior surgeon not present) via augmented reality (AR) glasses, worn by the resident in theatre. Stress was measured by intraoperative heart rate (Polar® pulse belt) and State-Trait Anxiety Inventory (STAI) questionnaire.

RESULTS

After "dry lab" testing, N = 5 senior residents performed equally N = 25 procedures SA and with AR glasses. For both, open and laparoscopic procedure AR remote assistance showed satisfactory applicability. Utilization of AR significantly reduced intraoperative peak pulse rate from 131 to 119 bpm (p = 0.004), as compared with the semi-autonomous group. Likewise, subjectively perceived stress according to STAI was significantly lower in the AR group (p = 0.011).

CONCLUSION

AR can be applied in surgical training and may help to reduce stress in theatre. In the future, AR has a huge potential to become a stepping stone to surgical autonomy.

Why Not This Case? Differences Between Resident and Attending Operative Cases at Teaching Hospitals

INTRODUCTION

Previous studies have focused on outcomes pertaining to resident operative autonomy, but there has been little academic work examining the types of patients and cases where autonomy is afforded. We sought to describe the differences between surgical patient populations in teaching cases where residents are and are not afforded autonomy.

METHODS

We examined all general and vascular operations at Veterans Affairs teaching hospitals from 2004 to 2019 using Veterans Affairs Surgical Quality Improvement Program. Level of resident supervision is prospectively recorded by the operating room nurse at the time of surgery: attending primary (AP): the attending performs the case with or without a resident; attending resident (AR): the resident performs the case with the attending scrubbed; resident primary (RP): resident operating with supervising attending not scrubbed. Resident (R) cases refer to AR + RP. Patient demographics, comorbidities, level of supervision, and top cases within each group were evaluated.

RESULTS

A total of 618,578 cases were analyzed; 154,217 (24.9%) were AP, 425,933 (68.9%) AR, and 38,428 (6.2%) RP. Using work relative value unit as a surrogate for complexity, RP was the least complex compared to AP and AR (10.4/14.4/14.8, P < 0.001). RP also had a lower proportion of American Society of Anesthesiologists 3 and 4 + 5 patients (P < 0.001), were younger (P < 0.001), and generally had lower comorbidities. The most common RP cases made up a higher proportion of all RP cases than they did for AP/AR and demonstrated several core competencies (hernia, cholecystectomy, appendectomy, amputation). R cases, however, were generally sicker than AP cases.

CONCLUSIONS

In the small proportion of cases where residents were afforded autonomy, we found they were more focused on the core general surgery cases on lower risk patients. This selection bias likely demonstrates appropriate attending judgment in affording autonomy. However, this cohort consisted of many "sicker" patients and those factors alone should not disqualify resident involvement.

Faculty Entrustment and Resident Entrustability

Importance

As the surgical education paradigm transitions to entrustable professional activities, a better understanding of the factors associated with resident entrustability are needed. Previous work has demonstrated intraoperative faculty entrustment to be associated with resident entrustability. However, larger studies are needed to understand if this association is present across various surgical training programs.

Objective

To assess intraoperative faculty-resident behaviors and determine if faculty entrustment is associated with resident entrustability across 4 university-based surgical training programs.

Design, Setting, and Participants

This cross-sectional study was conducted at 4 university-based surgical training programs from October 2018 to May 2022. OpTrust, a validated tool designed to assess both intraoperative faculty entrustment and resident entrustability behaviors independently, was used to assess faculty-resident interactions. A total of 94 faculty and 129 residents were observed. Purposeful sampling was used to create variation in type of operation performed, case difficulty, faculty-resident pairings, faculty experience, and resident training level.

Main Outcomes and Measures

Observed resident entrustability scores (scale 1-4, with 4 indicating full entrustability) were compared with reported measures (faculty level, case difficulty, resident postgraduate year [PGY], resident gender, observation month) and observed faculty entrustment scores (scale 1-4, with 4 indicating full entrustment). Path analysis was used to explore direct and indirect effects of the predictors. Associations between resident entrustability and faculty entrustment scores were assessed by pairwise Pearson correlation coefficients.

Results

A total of 338 cases were observed. Cases observed were evenly distributed by faculty experience (1-5 years' experience: 67 [20.9%]; 6-14 years' experience: 186 [58%]; ≥15 years' experience: 67 [20.9%]), resident PGY (PGY 1: 28 [8%]; PGY 2: 74 [22%]; PGY 3: 64 [19%]; PGY 4: 40 [12%]; PGY 5: 97 [29%]; ≥PGY 6: 36 [11%]), and resident gender (female: 183 [54%]; male: 154 [46%]). At the univariate level, PGY (mean [SD] resident entrustability score range, 1.44 [0.46] for PGY 1 to 3.24 [0.65] for PGY 6; F = 38.92; P < .001) and faculty entrustment (2.55 [0.86]; R2 = 0.94; P < .001) were significantly associated with resident entrustablity. Path analysis demonstrated that faculty entrustment was associated with resident entrustability and that the association of PGY with resident entrustability was mediated by faculty entrustment at all 4 institutions.

Conclusions and Relevance

Faculty entrustment remained associated with resident entrustability across various surgical training programs. These findings suggest that efforts to develop faculty entrustment behaviors may enhance intraoperative teaching and resident progression by promoting resident entrustability.

Surgical Specialty Impacts Quality of Operative Training in Carotid Endarterectomy

BACKGROUND

Carotid endarterectomy (CEA) is currently performed by multiple surgical specialties. The impact of surgical specialty and operative volume on post-CEA outcomes has been well described. However, it is unclear whether trainees of different surgical specialties have similar quality of operative training.

METHODS

Data from Accreditation Council for Graduate Medical Education annual reports were collected and compared between graduating vascular surgery (VS) residents, VS fellows, and neurological surgery (NS) residents. Only cases reported as chief/senior/lead resident, surgeon junior, or surgeon fellow were included in analysis. Linear regression analysis was utilized to evaluate trends in case-mix and volume.

RESULTS

From 2013 to 2022, total CEA case volume was higher among VS residents and fellows, compared to NS residents (52.8 ± 0.8 vs. 44.3 ± 1.4 vs. 12.9 ± 0.6, P < 0.0001). Additionally, VS residents and fellows performed other carotid operations including transfemoral or transcarotid artery stenting (11.1 ± 0.9 vs. 11.2 ± 0.8 vs. 0), carotid body tumor resection (0.7 ± 0.1 vs. 0.7 ± 0.0 vs. 0), and extracranial cervical bypass (6.7 ± 0.3 vs. 6.3 ± 0.3 vs. 0) that were not reported by the NS resident cohort (P < 0.0001 each). On linear regression analysis, total CEA procedures did not change for VS residents (R2 = 0.03, P = 0.62), decreased for VS fellows (-1.29 cases/yr, R2 = 0.75, P < 0.0001), and decreased among NS residents (-0.41 cases/yr, R2 = 0.44, P = 0.01) over the study period.

CONCLUSIONS

Although residents of multiple surgical specialties are trained in CEA, vascular training offers significantly greater numbers and diversity of extracranial carotid cases. It also appears that CEA volume is decreasing among neurosurgical trainees. In light of recent reports on the volume-outcome effect in carotid surgery, these data may have implications for future practice patterns in the domain of extracranial carotid artery disease.

Laparoscopic cholecystectomy critical view of safety (LC-CVS): a multi-national validation study of an objective, procedure-specific assessment using video-based assessment (VBA)

BACKGROUND

A novel 6-item objective, procedure-specific assessment for laparoscopic cholecystectomy incorporating the critical view of safety (LC-CVS OPSA) was developed to support trainee formative and summative assessments. The LC-CVS OPSA included two retraction items (fundus and infundibulum retraction) and four CVS items (hepatocystic triangle visualization, gallbladder-liver separation, cystic artery identification, and cystic duct identification). The scoring rubric for retraction consisted of poor (frequently outside of defined range), adequate (minimally outside of defined range) and excellent (consistently inside defined range) and for CVS items were "poor-unsafe", "adequate-safe", or "excellent-safe".

METHODS

A multi-national consortium of 12 expert LC surgeons applied the OPSA-LC CVS to 35 unique LC videos and one duplicate video. Primary outcome measure was inter-rater reliability as measured by Gwet's AC2, a weighted measure that adjusts for scales with high probability of random agreement. Analysis of the inter-rater reliability was conducted on a collapsed dichotomous scoring rubric of "poor-unsafe" vs. "adequate/excellent-safe".

RESULTS

Inter-rater reliability was high for all six items ranging from 0.76 (hepatocystic triangle visualization) to 0.86 (cystic duct identification). Intra-rater reliability for the single duplicate video was substantially higher across the six items ranging from 0.91 to 1.00.

CONCLUSIONS

The novel 6-item OPSA LC CVS demonstrated high inter-rater reliability when tested with a multi-national consortium of LC expert surgeons. This brief instrument focused on safe surgical practice was designed to support the implementation of entrustable professional activities into busy surgical training programs. Instrument use coupled with video-based assessments creates novel datasets with the potential for artificial intelligence development including computer vision to drive assessment automation.

Patient Surgical Outcomes When Surgery Residents Are the Primary Surgeon by Intensity of Surgical Attending Supervision in Veterans Affairs Medical Centers

Objective

Using health records from the Department of Veterans Affairs (VA), the largest healthcare training platform in the United States, we estimated independent associations between the intensity of attending supervision of surgical residents and 30-day postoperation patient outcomes.

Background

Academic leaders do not agree on the level of autonomy from supervision to grant surgery residents to best prepare them to enter independent practice without risking patient outcomes.

Methods

Secondary data came from a national, systematic 1:8 sample of n = 862,425 teaching encounters where residents were listed as primary surgeon at 122 VA medical centers from July 1, 2004, through September 30, 2019. Independent associations between whether attendings had scrubbed or not scrubbed on patient 30-day all-cause mortality, complications, and 30-day readmission were estimated using generalized linear-mixed models. Estimates were tested for any residual confounding biases, robustness to different regression models, stability over time, and validated using moderator and secondary factors analyses.

Results

After accounting for potential confounding factors, residents supervised by scrubbed attendings in 733,997 nonemergency surgery encounters had fewer deaths within 30 days of the operation by 14.2% [0.3%, 29.9%], fewer case complications by 7.9% [2.0%, 14.0%], and fewer readmissions by 17.5% [11.2%, 24.2%] than had attendings not scrubbed. Over the 15 study years, scrubbed surgery attendings may have averted an estimated 13,700 deaths, 43,600 cases with complications, and 73,800 readmissions.

Conclusions

VA policies on attending surgeon supervision have protected patient safety while allowing residents in selected teaching encounters to have limited autonomy from supervision.

Implementation of entrustable professional activities in multiple surgical residencies: A quality improvement approach

BACKGROUND

The COVID-19 pandemic decreased the operative case volume for surgical residents. Our institution implemented Entrustable Professional Activities (EPAs) in all core surgical training programs to document the competency of graduating residents. Continuation of this project aimed to improve implementation.

METHODS

This project occurred at a large academic center with eight surgical specialties during the 2020-21 (Year 1) and 2021-22 (Year 2) academic years. Each specialty chose five EPAs, and residents were asked to obtain three micro-assessments per EPA. After the initial pilot year, program directors were surveyed regarding perceptions of EPA utility and barriers to implementation.

RESULTS

Seventy senior residents completed 732/906 (80.8%) micro-assessments. Of these, 99.6% were deemed practice ready. Total micro-assessment completion rates in four specialties, four specific EPAs (including one EPA identified "at risk" due to the COVID-19 pandemic), and overall were significantly higher in Year 2 than Year 1 (p ​< ​0.05) CONCLUSIONS: Implementing EPAs in all core surgical specialties at an institution is achievable, though expectedly initially imperfect. An ongoing quality collaborative initiative focused on barriers to implementation can improve completion rates.

Right Case, Right Time: Which Procedures Best Differentiate General Surgery Trainees' Operative Performance?

OBJECTIVE

Assessing surgical trainee operative performance is time- and resource-intensive. To maximize the utility of each assessment, it is important to understand which assessment activities provide the most information about a trainee's performance. The objective of this study is to identify the procedures that best differentiate performance for each general surgery postgraduate year (PGY)-level, leading to recommendations for targeted assessment.

DESIGN

The Society for Improving Medical Professional Learning (SIMPL) operative performance ratings were modeled using a multilevel Rasch model which identified the highest and lowest performing trainees for each PGY-level. For each procedure within each PGY-level, a procedural performance discrimination index was calculated by subtracting the proportion of "practice-ready" ratings of the lowest performing trainees from the proportion of "practice-ready" ratings of the highest performing trainees. Four-quadrant plots were created using the median procedure volume and median discrimination index for each PGY-level. All procedures within the upper right quadrant were considered "highly differentiating, high volume" procedures.

SETTING

This study was conducted across 70 general surgical residency programs who are members of the SIMPL collaborative.

PARTICIPANTS

A total of 54,790 operative performance evaluations of categorical general surgery trainees were collected between 2015 and 2021. Trainees who had at least 1 procedure in common were included. Procedures with less than 25 evaluations per training year were excluded.

RESULTS

The total number of evaluations per procedure ranged from 25 to 2,131. Discrimination values were generated for 51 (PGY1), 54 (PGY2), 92 (PGY3), 105 (PGY4), and 103 (PGY5) procedures. Using the above criteria, a total of 12 (PGY1), 15 (PGY2), 22 (PGY3), 21 (PGY4), and 28 (PGY5) procedures were identified as highly differentiating, high volume procedures.

CONCLUSIONS

Our study draws on national data to identify procedures which are most useful in differentiating trainee operative performance at each PGY-level. This list of procedures can be used to guide targeted assessment and improve assessment efficiency.

Surgical Education

This paper summarizes the proceedings of the joint European Surgical Association ESA/American Surgical Association symposium on Surgical Education that took place in Bordeaux, France, as part of the celebrations for 30 years of ESA scientific meetings. Three presentations on the use of quantitative metrics to understand technical decisions, coaching during training and beyond, and entrustable professional activities were presented by American Surgical Association members and discussed by ESA members in a symposium attended by members of both associations.

Enhancing the Value of Surgical Entrustable Professional Activities through Integrative Learning Analytics

OBJECTIVE

To demonstrate the value of integrating surgical resident Entrustable Professional Activity (EPA) data into a learning analytics platform that provides meaningful feedback for formative and summative decision-making.

DESIGN

Description of the Surgical Entrustable Professional Activities (SEPA) analytics dashboard, and examples of summary analytics and intuitive display features.

SETTING

Department of Surgery, University of Wisconsin Hospital and Clinics.

PARTICIPANTS

Surgery residents, faculty, and residency program administrators.

RESULTS

We outline the major functionalities of the SEPA dashboard and offer concrete examples of how these features are utilized by various stakeholders to support progressive entrustment decisions for surgical residents.

CONCLUSIONS

Our intuitive analytics platform allows for seamless integration of SEPA microassessment data to support Clinical Competency Committee (CCC) decisions for resident evaluation and provides point of training feedback to faculty and trainees in support of progressive autonomy.

Equal but Separate: The Slow Assimilation of Osteopathic Surgery Residents Two Years After the Unified Match

OBJECTIVES

The purpose of this study was to monitor the integration of general surgery residency programs before and after the 2020 unified match. We hypothesized that integration of osteopathic (DO) surgery residents would increase.

DESIGN

We performed a retrospective cohort study of surgery residency programs between 2019 and 2021 utilizing data provided by the Association of American Medical Colleges. Program composition (2021) and changes in composition (2019-2021) were compared by program type. Multivariable logistic regression models assessed variables associated with DO presence (2021) and integration (2019-2021).

SETTING

General surgery residency programs across the United States.

PARTICIPANTS

Civilian surgery residencies that completed the 2019-2021 program survey.

RESULTS

Out of 320 programs, DO residents were integrated at 69% (221/320), including 52% (63/122) university programs, 78% (101/129) university-affiliated programs and 83% (57/69) community programs (p < 0.01). Overall, 23 (8%) programs integrated DO residents from 2019 to 2021, and 9 (21%) ex-American Osteopathic Association programs integrated MD residents (both p < 0.01). The median number of DO residents was 1 (interquartile range, IQR 0-2) at university programs, 2 (IQR 1-7) at university-affiliated programs, and 5 (IQR 2-12) at community programs (p < 0.01). The median number of DO residents at all programs increased from 1 (IQR 0-5) to 2 (IQR 0-6) since 2019 (p < 0.01). Community (OR 2.6, p = 0.04), university-affiliated (OR 2.3, p = 0.02), and programs with DOs in 2019 (OR 19.0, p < 0.01) were associated with increased odds of DOs present in 2021, while DO faculty (OR 2.6, p = 0.02) was the only factor independently associated with integrating DOs after 2019.

CONCLUSIONS

While some programs have integrated DO residents, progress is slow, median numbers of DO residents remain low, and familiarity with DOs is most associated with integration. We explore barriers to integration, and advance recommendations to eliminate potential disparities.

Recovery of Skill Decay After COVID-19 Redeployments and Implications for Competency Attainment

INTRODUCTION

The COVID-19 pandemic immediately interrupted procedural training. The lasting impact of reduced caseloads and service redeployments on procedural-resident training has been underexplored. This longitudinal study investigated the long-term perspectives of skill decay after short breaks in training and implications for ensuring resident competency attainment.

METHODS

Web-based cross-sectional surveys distributed immediately after (June 2020) compared to 1 y after (July 2021) COVID-19 redeployments at two tertiary academic medical centers of an integrated health system in New York. Participants included general surgery, surgical subspecialty, and anesthesiology residents and faculty.

RESULTS

Fifty-five residents and 33 faculty completed the survey. Ninety-point nine percent of residents and 36.4% of faculty were redeployed to COVID-ICUs. Sixty-three-point seven percent of residents and 75.0% of faculty reported a reduction in resident technical skills in the short-term, with significantly less (45.5% of residents and 21.2% of faculty) reporting persistent reduction in technical skill after 1 y (P = 0.001, P < 0.001). Seventy-five percent of residents and 100% of faculty were confident residents would be able to practice independently at the conclusion of their training. Sixty-five-point five percent of residents and 63.6% of faculty felt that residents experienced a durable improvement in critical care skills. Residents also reported a positive long-term impact on professional core competencies at 1 y.

CONCLUSIONS

Longitudinal surveillance of residents after COVID-19 redeployments suggests washout of temporary skill decay and return of resident confidence upon resumption of traditional training. This may provide insight into the impact of other short-term training interruptions on resident skill and promote greater resident support upon training resumption to ensure competency attainment.

Advancing Urology Resident Surgical Autonomy

PURPOSE OF REVIEW

This paper aims to survey current literature on urologic graduate medical education focusing on surgical autonomy.

RECENT FINDINGS

Affording appropriate levels of surgical autonomy has a key role in the education of urologic trainees and perceived preparedness for independent practice. Recent studies in surgical resident autonomy have demonstrated a reduction in autonomy for trainees in recent years. Efforts to advance the state of modern surgical training include creation of targeted curricula, enhanced with use of surgical simulation, and structured feedback. Decline in surgical autonomy for urology residents may influence confidence after completion of their residency. Further study is needed into the declining levels of urology resident autonomy, how it affects urologists entering independent practice, and what interventions can advance autonomy in modern urologic training.

Declining Resident Surgical Autonomy and Improving Surgical Outcomes: Correlation Does Not Equal Causality

OBJECTIVE

The volume of cases that residents perform independently have decreased leaving graduating chief residents less prepared for independent practice. Outcomes are not worse when residents are given autonomy with appropriate supervision, however it is unknown if outcomes are worsening with decreasing operative autonomy experience. We hypothesize that resident autonomous cases parallel the improving outcomes in surgical care over time, however, are less complex and on lower acuity patients.

DESIGN

Retrospective study utilizing the Veterans Affairs Surgical Quality Improvement Program (VASQIP) database.

SETTING

Operative cases performed on teaching services within the VASQIP database from July 1, 2004 to September 30, 2019, were included.

PARTICIPANTS

All adult patients who underwent a surgical procedure from July 1, 2004, to September 30, 2019, at a VA hospital on a service that included residents were initially included. After inclusions and exclusions, there were 1,346,461 cases. Cases were divided into 3 sequential 5 year eras (ERA 1: 2004-2008 n = 415,614, ERA 2: 2009-2013 n = 478,528, and ERA 3: 2014-2019 n = 452,319). The main exposure of interest was level of resident supervision, coded at the time of procedure as: attending primary surgeon (AP); attending and resident (AR), or resident primary with the attending supervising but not scrubbed (RP). We compared 30 day all-cause mortality, composite morbidity, work relative value unit (wRVU), hospital length of stay, and operative time between each ERA for RP cases, as well as within each ERA for RP cases compared to AR and AP cases.

RESULTS

There was a progressive decline in the rate of RP cases in each successive ERA (ERA 1: 58,249 (14.0%) vs ERA 2: 47,891 (10.0%) vs ERA 3: 35,352 (7.8%), p < 0.001). For RP cases, patients were progressively getting older (60 yrs [53-71] vs 63 yrs [54-69] vs 66 yrs [57-72], p < 0.001) and sicker (ASA 3 58.7% vs 62.5% vs 66.2% and ASA 4/5 8.4% vs 9.6% vs10.0%, p < 0.001). Odds of mortality decreased in each ERA compared to the previous (aOR 0.71 [0.62-0.80] ERA 2 vs ERA 1 and 0.82 [0.70-0.97] ERA 3 vs ERA 2) as did morbidity (0.77 [0.73-0.82] ERA 2 vs ERA 1 and 0.72 [0.68-0.77] ERA 3 vs ERA 2). Operative and length of stay also decreased while wRVU stayed unchanged. When comparing RP cases to AP and AR within each ERA, RP cases tended to be on younger and healthier patients with a lower wRVU, particularly compared to AR cases. Mortality and morbidity were no different or better in RP compared to AR and AP.

CONCLUSIONS

Despite resident autonomy decreasing, outcomes in cases where they are afforded autonomy are improving over time. This despite RP cases being on sicker and older patients and performing roughly the same complexity of cases. They also continue to perform no worse than cases with higher levels of supervision. Efforts to increase surgical resident operative autonomy are still needed to improve readiness for independent practice.

Postgraduate Surgical Education in East, Central, and Southern Africa: A Needs Assessment Survey

BACKGROUND

The Lancet Commission on Global Surgery has identified workforce development as an important component of National Surgical Plans to advance the treatment of surgical disease in low- and middle-income countries. The goal of our study is to identify priorities of surgeon educators in the region so that collaboration and intervention may be appropriately targeted.

STUDY DESIGN

The American College of Surgeons Operation Giving Back, in collaboration with leaders of the College of Surgeons of Eastern, Central and Southern Africa (COSECSA), developed a survey to assess the needs and limitations of surgical educators working under their organizational purview. COSECSA members were invited to complete an online survey to identify and prioritize factors within 5 domains: (1) Curriculum Development, (2) Faculty Development, (3) Structured Educational Content, (4) Skills and Simulation Training, and (5) Trainee Assessment and Feedback.

RESULTS

One-hundred sixty-six responses were received after 3 calls for participation, representing all countries in which COSECSA operates. The majority of respondents (78%) work in tertiary referral centers. Areas of greatest perceived need were identified in the Faculty Development and Skills and Simulation domains. Although responses differed between domains, clinical responsibilities, cost, and technical support were commonly cited as barriers to development.

CONCLUSIONS

This needs assessment identified educational needs and priorities of COSECSA surgeons. Our study will serve as a foundation for interventions aimed at further improving graduate surgical education and ultimately patient care in the region.

Implementation of entrustable professional activities into fellowship council accredited programs: a pilot project

OBJECTIVE

The Fellowship Council (FC) is transitioning to a competency-based medical education (CBME) model, including the introduction of Entrustable Professional Activities (EPAs) for training and assessment of Fellows. This study describes the implementation process employed by the FC during a ten-month pilot project and presents data regarding feasibility and perceived value.

METHODS

The FC coordinated the development of EPAs in collaboration with the sponsoring societies for Advanced GI/MIS, Bariatrics, Foregut, Endoscopy and Hepatopancreaticobiliary (HPB) fellowships encompassing the preoperative, intraoperative, and postoperative phases of care for key competencies. Fifteen accredited fellowship programs participated in this project. The assessments were collected through a unique platform on the FC website. Programs were asked to convene a Clinical Competency Committee (CCC) on a quarterly basis. The pilot group met monthly to support and improve the process. An exit survey evaluated the perceived value of EPAs.

RESULTS

The 15 participating programs included 18 fellows and 106 faculty. A total of 655 assessments were initiated with 429 (65%) completed. The average (SD) number of EPAs completed for each fellow was 24(18); range 0-72. Intraoperative EPAs were preferentially completed (71%). The average(SD) time for both the fellow and faculty to complete an EPA was 27(78) hours. Engagement increased from 39% of fellows completing at least one EPA in September to 72% in December and declining to 50% in May. Entrustment level increased from 6% of EPAs evaluated as "Practice Ready" in September to 75% in June. The exit survey was returned by 63% of faculty and 72% of fellows. Overall, 46% of fellows and 74% of program directors recommended full-scale implementation of the EPA framework.

CONCLUSION

A competency-based assessment framework was developed by the FC and piloted in several programs. Participation was variable and required ongoing strategies to address barriers. The pilot project has prepared the FC to introduce CBME across all FC training programs.

Implementing an entrustable professional activities programmatic assessments for nurse practitioner training in emergency care: A pilot study

BACKGROUND

Entrustable professional activities (EPAs), as new methods used to operationalize competency-based education, are gaining acceptance in medical professions. However, no EPA-based framework exists to evaluate emergency care nurse practitioners' competencies in clinical practice.

OBJECTIVES

To develop, implement, and evaluate an EPA-based framework for emergency care nurse practitioners.

DESIGN

Prospective observational study.

SETTING

An accredited nurse practitioner training program in the emergency department of a University-affiliated Hospital.

PARTICIPANTS

Eight novice nurse practitioners and eighteen clinical instructors.

METHODS

A working group of academic educators, nurse practitioner leaders, and clinical instructors developed a set of EPAs using a participatory design method. We scheduled weekly ad hoc evaluations for the nurse practitioners in emergency care, and we collected several observations from multiple assessors for analysis.

RESULTS

Four nested EPAs were developed that adopted a 1-5 supervision-entrustment scale to measure performance. The results of the 20-month assessments illustrated that the average scores of EPA1, which measured the assessment and management of patients with acute medical presentations in the emergency department, differed significantly but were not positively correlated with the nurse practitioners' increasing subspecialty and department seniority levels (Kruskal-Wallis test P = .011 and 0.006, respectively). In addition, the most time required for the nurse practitioners to achieve a stable level 5 entrustability score for all EPAs for at least 3 consecutive months was 18 months. Clinical instructors noted that recognizing limits is a skill that is essential for a nurse practitioner to earn clinical instructors' trust.

CONCLUSIONS

Entrustability scores may not correlate with nurse practitioners' increasing seniority, and patient care requires the longest training time (18 months) for emergency care nurse practitioners to achieve autonomy. Program directors should adjust the supervision-entrustment level expected at various stages of training according to the complexity of individual EPAs. An EPA-based assessment framework in a nurse practitioner training program may assist in overseeing the development of competencies for emergency care nurse practitioners.

Multi-disciplinary assessment of the entrustable professional activities of surgery residents

Purpose

Medicine is practiced in a collaborative and interdisciplinary manner. However, medical training and assessment remain largely isolated in traditional departmental silos. Two Entrustable Professional Activities (EPAs) developed by the American Board of Surgery are multidisciplinary in nature and offer a unique opportunity to study interdisciplinary assessment.

Methods

EPA microassessments were collected from Surgery and Emergency Medicine (EM) faculty between July 2018 and May 2020. Differences in feedback provided by faculty were assessed using natural language processing (NLP) techniques, (1) automated algorithms; and (2) topic modeling. Summative content analysis was used to identify themes in text feedback. We developed automated coding algorithms for these themes using regular expressions. Topic modeling was performed using latent Dirichlet allocation.

Results

549 assessments were collected for two EPAs: 198 for GS Consultation and 351 for Trauma. 27 EM and 27 Surgery faculty provided assessments for 71 residents. EM faculty were significantly more likely than Surgery faculty to submit feedback coded as Communication, Demeanor, and Timeliness, (all chi-square test p-values < 0.01). No significant differences were found for Clinical Performance, Skill Level, or Areas for Improvement. Similarly, topic modeling indicated that assessments submitted by EM faculty focused on communication, timeliness, and interpersonal skills, while those submitted by Surgery faculty focused on the residents' abilities to effectively gather information and correctly diagnose the underlying pathology.

Conclusions

Feedback from EM and Surgery faculty differed significantly based on NLP analyses. EPA assessments should stem from multiple sources to avoid assessment gaps and represent a more holistic picture of performance.

TrustEd: A Tool for Developing Intraoperative Entrustment Skills

OBJECTIVE

Toolkits to assess progressive resident autonomy are integral to the movement toward competency-based surgical education. OpTrust is one such tool validated for intraoperative assessment of both faculty and resident entrustment behaviors. We developed a supplementary tool to OpTrust that would aid faculty and residents in making meaningful improvements in entrustment behavior by providing talking points and reflection items tailored to different motivational styles as defined by Regulatory Focus Theory (RFT).

DESIGN

Existing literature about surgical entrustment was used to build a list of sample dialogue and self-reflection items to use in the operating room. This list was distributed as a survey to individuals familiar with OpTrust and RFT, asking them to categorize each item as Promotion-oriented, Prevention-oriented, or Either. The respondents then met to discuss survey items that did not reach a consensus until the group agreed on their categorization.

SETTING

University of Wisconsin, School of Medicine and Public Health, Madison, Wisconsin Michigan Medicine, Ann Arbor, Michigan PARTICIPANTS: Clinician and education researchers familiar with intraoperative entrustment and RFT RESULTS: Eight respondents completed the survey categorizing the talking points and reflection items by RFT (100% response rate). Six of these respondents attended the additional meeting to discuss discordant items. The input from this panel was used to develop "TrustEd," the supplementary tool that faculty and residents can quickly reference before beginning a case.

CONCLUSION

Although tools such as OpTrust allow intraoperative entrustment behaviors to be quantified, TrustEd offers concrete strategies for faculty and residents who are interested in improving those behaviors over time. Further study is needed to assess whether the use of TrustEd does in fact lead to durable behavior change and improvement in OpTrust scores.

Concurrent Validity Evidence for Entrustable Professional Activities in General Surgery Residents

BACKGROUND

A subset of Entrustable Professional Activities (EPAs) has been developed for general surgery. We aim to contribute validity evidence for EPAs as an assessment framework for general surgery residents, including concurrent validity compared to ACGME milestones, the current gold standard for evaluating competency.

STUDY DESIGN

This is a cross-sectional study in a general surgery training program within a tertiary academic medical center. EPA assessments were submitted using a mobile app and scored on a numerical scale, mirroring milestones. EPA score distribution was analyzed with respect to post-graduate year (PGY) level and phase of care. Proportional odds logistic regression identified significant predictors. Spearman rank and Wilcoxon rank tests were used for comparisons with milestone ratings.

RESULTS

From August 2018 to December 2019, 320 assessments were collected. EPA scores increased by PGY level. Operative phase EPA scores were significantly lower than nonoperative phase scores. PGY level, operative phase, and case difficulty significantly influenced entrustment scoring. EPA scores demonstrated strong correlation with nonoperative milestones patient care-1, medical knowledge-1, interpersonal and communication skills-2, interpersonal and communication skills-3, professionalism-1, professionalism-3, and practice-based learning and improvement-2 (ρ > 0.5, p < 0.05) and a weaker correlation with operative milestones patient care-3 and medical knowledge-2 (ρ < 0.5, p < 0.05).

CONCLUSIONS

The influence of PGY level and operative phase on entrustment scoring supports the validity of EPAs as a formative evaluation framework for general surgery resident performance. In addition, evident correlations between EPA scores and respective milestone ratings provide concurrent validity evidence.

A Survey of the Impact of the COVID-19 Crisis on Skill Decay Among Surgery and Anesthesia Residents

OBJECTIVE

The SARS-CoV-2 (COVID-19) pandemic has profoundly impacted healthcare delivery and strained medical training. This study explores resident and faculty perceptions regarding the impact of the COVID-19 crisis on technical skill decay of surgical and anesthesia residents. We hypothesized that many residents perceived that their technical abilities diminished due to a short period of interruption in their training.

DESIGN

An IRB-exempt, web-based cross-sectional survey distributed to residents and faculty SETTING: Two large academic tertiary medical centers, North Shore University Hospital and Long Island Jewish Medical Center, of the Northwell Health System in New York.

PARTICIPANTS

General surgery, anesthesiology, plastic surgery, cardiothoracic surgery, orthopedic surgery, oral maxillofacial surgery, urology, podiatry residents and faculty.

RESULTS

All residents reported a significant impact on their training. Residents (82%) and faculty (94%) reported a significant reduction in case volumes due to the COVID-19 pandemic (p < 0.05). 64% of residents reported a reduction in technical skills, and 75% of faculty perceived a decrease in resident technical skills. Residents were concerned about fulfilling ACGME case requirements, however faculty were more optimistic that residents would achieve level-appropriate proficiency by the conclusion of their training. Both residents and faculty felt that resident critical care skills improved as a result of redeployment to COVID-19 intensive care units (66% and 94%). Additionally, residents reported increased confidence in their ability to care for critically ill patients and positive impact on professional competencies.

CONCLUSIONS

Effects of the COVID-19 pandemic on residency training are multi-dimensional. The majority of surgical and anesthesia residents perceived that their technical ability diminished as a result of skill decay, whereas other skillsets improved. Longitudinal surveillance of trainees is warranted to evaluate the effect of reduced operative volume and redeployment on professional competency.

Gender Differences in Entrustable Professional Activity Evaluations of General Surgery Residents

OBJECTIVE

To determine differences in entrustable professional activity (EPA) assessments between male and female general surgery residents.

SUMMARY BACKGROUND DATA

Evaluations play a critical role in career advancement for physicians. However, female physicians in training receive lower evaluations and underrate their own performance. Competency-based assessment frameworks, such as EPAs, may help address gender bias in surgery by linking evaluations to specific, observable behaviors.

METHODS

In this cohort study, EPA assessments were collected from July 2018 to May 2020. The effect of resident sex on EPA entrustment levels was analyzed using multiple linear and ordered logistic regressions. Narrative comments were analyzed using latent dirichlet allocation to identify topics correlated with resident sex.

RESULTS

Of the 2480 EPAs, 1230 EPAs were submitted by faculty and 1250 were submitted by residents. After controlling for confounding factors, faculty evaluations of residents were not impacted by resident sex (estimate = 0.09, P = 0.08). However, female residents rated themselves lower by 0.29 (on a 0-4 scale) compared to their male counterparts (P < 0.001). Within narrative assessments, topics associated with resident sex demonstrated that female residents focus on the "guidance" and "supervision" they received while performing an EPA, while male residents were more likely to report "independent" action.

CONCLUSIONS

Faculty assessments showed no difference in EPA levels between male and female residents. Female residents rate themselves lower by nearly an entire post graduate year (PGY) level compared to male residents. Latent dirichlet allocation -identified topics suggest this difference in self-assessment is related to differences in perception of autonomy.

The Use of Entrustable Professional Activities in Pediatric Postgraduate Medical Education: A Systematic Review

BACKGROUND

Entrustable Professional Activities (EPAs) provide a framework to make judgments of trainees' abilities in several settings including postgraduate medical education. No systematic review of the role of EPAs in pediatrics has yet been performed.

OBJECTIVES

In our systematic review, we sought to determine the use of EPAs in pediatrics to identify research gaps, summarize and discuss evidence relating to the development, implementation, and assessment.

DATA SOURCES

Medline, Scopus, PsycINFO, MedEdPortal, and Web of Science.

STUDY ELIGIBILITY CRITERIA

Two independent reviewers used a structured screening protocol in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and Association for Medical Education in Europe's guide for systematic reviews to include all articles reporting on EPAs in postgraduate medical education and pediatrics in particular. There were no restrictions due to language, study design, or participants.

STUDY APPRAISAL AND SYNTHESIS METHODS

Data on development, implementation, feasibility, acceptance, and assessment of EPAs were extracted and analyzed by the 2 independent researchers.

RESULTS

Twenty-eight articles published between 2014 and 2020 were included in the review. We found an increase in publications and a notable shift from descriptions of development processes toward aspects beyond development, ie, implementation, feasibility, acceptance/perception, and assessment.

LIMITATIONS

Studies from non-English-speaking countries are scarce which might lead to an inaccurate representation of actual international practice.

CONCLUSIONS AND IMPLICATIONS OF KEY FINDINGS

We provide a comprehensive overview of EPAs in pediatrics to guide future curriculum developers in collaborative development, implementation and assessment of EPAs in pediatric postgraduate medical education.

An update on current EPAs in graduate medical education: A scoping review

The purpose of this scoping review is to update the recent progress of EPAs research in GME, focusing on the topical concern of EPAs effectiveness, and to provide a reference for medical researchers in countries/regions interested in introducing EPAs. Guided by Arksey and O'Malley's framework regarding scoping reviews, the researchers, in January 2021, conducted a search in five databases to ensure the comprehensiveness of the literature. After the predetermined process, 29 articles in total were included in this study. The most common areas for the implementation and evaluation of EPAs were Surgery (n = 7,24.1%), Pediatric (n = 5,17.2%) and Internal medicine (n = 4,13.8%), a result that shows a relatively large change in the research trend of EPAs in the last two years. Prior to 2018, EPAs research focused on internal medicine, psychiatry, family medicine, and primary care. The articles in the category of EPAs implementation and evaluation had four main themes: (1) validation of EPAs (n = 16,55.2%); (2) describing the experience of implementing EPAs (n = 11,37.9%); (3) examining the factors and barriers that influence the implementation and evaluation of EPAs (n = 6,20.6%); and (4) researching the experiences of faculty, interns, and other relevant personnel in using EPAs. Training programs were the most common EPAs implementation setting (n = 26,89.6%); direct observation and evaluation (n = 12,41.4%), and evaluation by scoring reports (n = 5,17.2%) were the two most common means of assessing physicians' EPA levels; 19 papers (65.5%) used faculty evaluation, and nine of these papers also used self-assessment (31.0%); the most frequently used tools in the evaluation of EPAs were mainly researcher-made instruments (n = 37.9%), assessment form (n = 7,24.1%), and mobile application (n = 6,20.7%). Although EPAs occupy an increasingly important place in international medical education, this study concludes that the implementation and diffusion of EPAs on a larger scale is still difficult.

Surgical Education and Training in the US: a Collaborative Effort to Deliver the Next Generation of Surgeons

The foundational principles of surgical training in the USA are based on didactic education, structured skill training, and experiential learning in surgical patient care with the supervision of surgical faculty. A consortium of professional organizations, academic institutions, and teaching hospitals with surgical faculty provide the structural framework, policies, and curriculum to train and evaluate surgeons capable of independent practice. This manuscript describes the roles of the organizations responsible for surgical training in the USA and highlights areas in evolution in the modern surgical education landscape.

Changing Medical Education When Change Is Hard: Implementing an Interdepartmental Entrustable Professional Activity

BACKGROUND

Entrustable professional activities (EPAs) are a new tool for assessing learners that represents a significant movement in graduate medical education (GME) toward competency-based assessment and serves as a bridge between milestones and clinical practice. Whenever a major change is implemented to any system, resistance to change is expected. Many change management models have been proposed to overcome this resistance; a newer model is outlined in the book Switch. The objective was to describe the change management principles used to institute an EPA.

METHODS

The model introduced in Switch was used as a framework for implementing a trauma resuscitation EPA in a joint effort between the departments of surgery and emergency medicine (EM) at the University of Wisconsin Hospitals and Clinics. The Department of Emergency Medicine used the principles of change management, completing 295 faculty evaluations of trauma resuscitations compared to the Department of Surgery, whose faculty completed 50 evaluations between the study period of July 2018 through October 2019. A survey completed winter 2019 of EM faculty was used to determine the most important principles toward successful implementation.

RESULTS

Twenty-seven of 35 (78% response rate) of EM faculty identified key concepts from Switch as being instrumental in the successful implementation. Internal discussion of the implementation approach used by the Department of Surgery identified several limitations that would have been overcome by using these same change management principles.

CONCLUSION

We conclude that the principles of change management provide a useful framework for successfully implementing EPAs into GME.

Fundamentos para un currículo nacional en Cirugía General basado en competencia: Consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía

Frente a los desafíos de la Educación Basada en Competencia, este consenso Delphi de la División de Educación de la Asociación Colombiana de Cirugía informa sobre las mínimas competencias profesionales esperadas del cirujano egresado de los veinte programas de Especialización en Cirugía General en Colombia. Un total de 105 profesores de los programas de especialización evaluaron tres áreas de competencia profesional: 1) atributos profesionales generales del residente durante su formación, 2) competencias prácticas (procedimientos quirúrgicos) que los residentes deben realizar al final de su entrenamiento y 3) Actividades Profesionales Confiables (APC) que los residentes deben ejecutar sin supervisión al final de su entrenamiento. Los resultados informan un alto nivel de consenso en el 100 % los atributos profesionales generales y APC, y del 75 % en diferentes procedimientos quirúrgicos. El consenso abre la puerta para el desarrollo de un currículo nacional de la especialidad y tiene implicaciones para la práctica educativa e investigación futura.

Entrustable Professional Activities in Orthopaedics

Background

An entrustable professional activity (EPA) is defined as a core task of a specialty that is entrusted to a trainee once sufficient competence has been reached. A group of EPAs reflects the activities that clinicians commonly do on a day-to-day basis. Lists of EPAs have been created for most medical subspecialties, but not orthopaedic surgery. The aim of this study was to create a peer-reviewed list of essential EPAs that a resident must perform independently before completing orthopaedic residency training.

Methods

A focus group of 7 orthopaedic surgeons from the University of Toronto developed a comprehensive list of 285 EPAs. For each subspecialty group, the opinions of at least 15 academic and nonacademic surgeons, as well as subspecialty-trained and non-subspecialty-trained surgeons, were used. The modified Delphi method was used to rank EPAs on a five-point scale from not important to mandatory for a resident to competent before exiting training. Two Delphi rounds were used, using a threshold of >50% of surgeons considering the EPA as mandatory before being considered for the next round. A final list of EPAs was ratified using the focus group of academic surgeons involved in the study.

Results

Seventy-five (75) of 107 (70%) surgeons invited responded to the survey. Nearly half (129) of the 285 EPAs were discarded after the first round of Delphi. A further 118 EPAs were discarded after the second Delphi round, leaving 49 final EPAs, across 9 subspecialties in orthopaedic surgery.

Conclusions

Expert consensus was used to create a list of EPAs considered mandatory for completion of resident training in orthopaedics in our province. The final 49 peer-reviewed EPAs will be a valuable benchmark in curriculum design and assessment in orthopaedic surgery in the competency-based era for other programs.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy

BACKGROUND

Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice.

METHODS

All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters.

RESULTS

Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics).

CONCLUSIONS

LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

A longitudinal cadaver-based simulation curriculum creates sustainable increases in resident confidence and exposure to fundamental techniques: Results of a 5-year program evaluation

INTRODUCTION

For the past five years, our surgical residency program has led a cadaver-based simulation course focused on fundamental surgical maneuvers. This study aimed to quantify the impact of this course on resident exposure to surgical skills and longitudinal impact on resident education.

METHODS

General surgery residents participated in an annual cadaver-based simulation curriculum. Participants completed surveys regarding improvements in knowledge and confidence; these results were stratified between course iterations (P1: 2 years, 2014-15; P2: 3 years, 2016-2018).

RESULTS

Residents reported a sustained increase in knowledge of anatomy and technical dissection, confidence in performing operative skills independently, and exposure to operative skills that were otherwise not encountered in clinical rotations. Junior residents demonstrated an increase in gaining skills they would otherwise not achieve (87% vs. 98%, p = 0.028) and confidence to safely perform these procedures in the clinical setting (94% vs. 100%, p = 0.077).

CONCLUSION

This annual, longitudinal cadaver-based skills course focused on fundamental maneuvers demonstrates a sustained impact in resident and faculty surgical confidence in resident's operative skills as a component of a longitudinal simulation curriculum to enhance competency-based promotion.

Public Perception of General Surgery Resident Autonomy and Supervision

BACKGROUND

Despite patients being important stakeholders in surgical training, little is known about the public's perception of trainee participation in surgical care. This study evaluates the public's perception of surgical resident autonomy and supervision.

STUDY DESIGN

An anonymous electronic survey was sent to adult panelists older than 18 years in the US using SurveyGizmo. The design of the survey used Dillman's Tailored Design Method to optimize response rate. Participants completed surveys including demographic characteristics and perceptions toward general surgery resident autonomy. Univariable and multivariable analyses were used as appropriate.

RESULTS

Survey response rate was 93% (2,005 of 2,148). Demographic characteristics including age, gender, race or ethnicity, and highest level of education were nationally representative. Most respondents (87%) had health insurance. On multivariable logistic regression analysis, factors associated with participants who would never allow a resident to perform any portion of the operation include: female gender (odds ratio [OR] 1.58; 95% CI, 1.28 to 1.95), no health insurance (OR 1.38; 95% CI, 1.03 to 1.84), Black race (OR 1.82; 95% CI, 1.38 to 2.41), and Hispanic ethnicity (OR 1.49; 95% CI, 1.03 to 2.15). Participants who were younger than 50 years (OR 1.57; 95% CI, 1.24 to 1.98), male (OR 1.90; 95% CI, 1.56 to 2.32), of Black race (OR 1.45; 95% CI, 0.10 to 1.91), Hispanic ethnicity (OR 1.49; 95% CI, 1.05 to 2.11), working in healthcare (OR 2.18; 95% CI, 1.67 to 2.86), or insured (OR 1.46; 95% CI, 1.07 to 1.99) were more likely to believe that resident involvement increases complications.

CONCLUSIONS

Among survey participants broadly representing the US population, resident participation in operations is not universally accepted. Public perception of surgical resident autonomy and supervision is important, as GME continues to evolve to address readiness for independent practice.

Entrustable professional activities for apheresis medicine education

BACKGROUND

Entrustable professional activities (EPAs) are well-defined, executable, observable, and measurable activities that are performed by a trainee and can be performed independently as training progresses. The purpose of this study is to develop EPAs specific for the practice of apheresis medicine (AM).

METHODS

Members of the American Society for Apheresis Graduate Medical Education subcommittee developed a list of 28 apheresis medical activities linked to Accreditation Council for Graduate Medical Education milestones and competencies in five areas: (a) consultation, (b) clinical care for therapeutic apheresis, (c) clinical care for donor collections, (d) test optimization, and (e) vascular access. Ten AM experts using a validated tool to measure the quality of the EPAs (QUEPA) evaluated these activities with use of a Likert scale. Per group consensus, an activity was considered acceptable for each domain if it had received an average score greater than 3.7, and it was rated 4 or 5 (agree or strongly agree) by at least 70% of experts.

RESULTS

Of the 28 activities, 11 did not have acceptable QUEPA scores: 7 activities were rated as unobservable, 4 were rated unfocused, 2 were rated unrealistic and not generalizable, and 2 were rated as not addressing multiple competencies. Four activities had unacceptable scores in more than one domain. Subcommittee members edited these 11 activities over two review cycles to produce a final list of 26 activities.

CONCLUSION

A set of practical, focused, and observable EPAs in AM were systematically developed. These EPAs can be used to assess and support trainee performance in AM.