Greater faculty familiarity with residents improves intraoperative entrustment

Faculty Entrustment and Resident Entrustability

Importance

As the surgical education paradigm transitions to entrustable professional activities, a better understanding of the factors associated with resident entrustability are needed. Previous work has demonstrated intraoperative faculty entrustment to be associated with resident entrustability. However, larger studies are needed to understand if this association is present across various surgical training programs.

Objective

To assess intraoperative faculty-resident behaviors and determine if faculty entrustment is associated with resident entrustability across 4 university-based surgical training programs.

Design, Setting, and Participants

This cross-sectional study was conducted at 4 university-based surgical training programs from October 2018 to May 2022. OpTrust, a validated tool designed to assess both intraoperative faculty entrustment and resident entrustability behaviors independently, was used to assess faculty-resident interactions. A total of 94 faculty and 129 residents were observed. Purposeful sampling was used to create variation in type of operation performed, case difficulty, faculty-resident pairings, faculty experience, and resident training level.

Main Outcomes and Measures

Observed resident entrustability scores (scale 1-4, with 4 indicating full entrustability) were compared with reported measures (faculty level, case difficulty, resident postgraduate year [PGY], resident gender, observation month) and observed faculty entrustment scores (scale 1-4, with 4 indicating full entrustment). Path analysis was used to explore direct and indirect effects of the predictors. Associations between resident entrustability and faculty entrustment scores were assessed by pairwise Pearson correlation coefficients.

Results

A total of 338 cases were observed. Cases observed were evenly distributed by faculty experience (1-5 years' experience: 67 [20.9%]; 6-14 years' experience: 186 [58%]; ≥15 years' experience: 67 [20.9%]), resident PGY (PGY 1: 28 [8%]; PGY 2: 74 [22%]; PGY 3: 64 [19%]; PGY 4: 40 [12%]; PGY 5: 97 [29%]; ≥PGY 6: 36 [11%]), and resident gender (female: 183 [54%]; male: 154 [46%]). At the univariate level, PGY (mean [SD] resident entrustability score range, 1.44 [0.46] for PGY 1 to 3.24 [0.65] for PGY 6; F = 38.92; P < .001) and faculty entrustment (2.55 [0.86]; R2 = 0.94; P < .001) were significantly associated with resident entrustablity. Path analysis demonstrated that faculty entrustment was associated with resident entrustability and that the association of PGY with resident entrustability was mediated by faculty entrustment at all 4 institutions.

Conclusions and Relevance

Faculty entrustment remained associated with resident entrustability across various surgical training programs. These findings suggest that efforts to develop faculty entrustment behaviors may enhance intraoperative teaching and resident progression by promoting resident entrustability.

Faculty perception of resident promotion and prevention associated behaviors in the operating room to facilitate intra-operative learning

BACKGROUND

Previous work has demonstrated that residents are able to accurately perceive the intraoperative motivational style of faculty. Additionally, alignment of motivational style between residents and faculty has been demonstrated to enhance resident intraoperative autonomy. This study evaluated if faculty perception of resident behaviors aligned with resident self-perception in order to identify ways of enhancing intraoperative learning.

METHODS

General surgery residents were asked to complete a self-assessment evaluating their own intraoperative behaviors. Conversely, faculty rated how strongly the residents exhibited these behaviors in the operating room.

RESULTS

Of the 10 intraoperative behaviors that were evaluated, eight demonstrated no correlation between resident self-perception and faculty perception of resident behavior. Similarly, inconsistent correlations emerged when behaviors were assessed according to the self-reported gender and race of the resident.

CONCLUSION

Faculty are not able to accurately perceive the motivational style of residents. Strategies to improve faculty perception of resident motivational style may enhance intraoperative learning.

Perceptions and Guiding Strategies to Regulate Entrusted Autonomy of Residents in the Operating Room: A Systematic Literature Review

OBJECTIVE

To provide a systematic literature review of intraoperative entrusted autonomy for surgical residents. Specifically, perceptions from residents and supervising surgeons, supervising behavior and influencing factors on intraoperative teaching and learning are analyzed.

BACKGROUND

Increasing demands on surgical training and the need for effective development of technical skills, amplify the importance of making the most of intraoperative teaching and learning opportunities in the operating room. It is critical for residents to gain the greatest benefit from every surgical case and to achieve operative competence.

METHODS

A systematic literature search identified 921 articles from 2000 to 2022 that addressed surgical education/training, intraoperative supervision/teaching, autonomy and entrustment. 40 studies with heterogeneous designs and methodologies were included.

RESULTS

Four themes were established in the analysis: patient safety, learner, learning environment and supervising surgeon. The patient is identified as the primary responsibility during intraoperative teaching and learning. Supervisors continuously guard patient safety as well as the resident's learning process. Ideal intraoperative learning occurs when the resident has optimal entrusted autonomy during the procedure matching with the current surgical skills level. A safe learning environment with dedicated time for learning are prerequisites for both supervising surgeons and residents. Supervising surgeons' own preferences and confidence levels also play an important role.

CONCLUSIONS

This systematic literature review identifies patient safety as the overriding principle for supervising surgeons when regulating residents' entrusted autonomy. When the supervisor's responsibility toward the patient has been met, there is room for intraoperative teaching and learning. In this process the learner, the learning environment and the supervising surgeon's own preferences all intertwine, creating a triangular responsibility. This review outlines the challenge of establishing an equilibrium in this triangle and the broad arsenal of strategies supervising surgeons use to keep it in balance.

Cumulative Sum Chart as Complement to Objective Assessment of Graduating Surgical Resident Competency: An Exploratory Study

BACKGROUND

Rater-based assessment and objective assessment play an important role in evaluating residents' clinical competencies. We hypothesize that a cumulative sum (CUSUM) chart of operative time is a complement to the assessment of chief general surgery residents' competencies with ACGME Milestones, aiding residency programs' determination of graduating residents' practice readiness.

STUDY DESIGN

We extracted ACGME Milestone evaluations of performance of operations and procedures (POP) and 3 objective metrics (operative time, case type, and case complexity) from 3 procedures (cholecystectomy, colectomy, and inguinal hernia) performed by 3 cohorts of residents (N = 15) during their PGY4-5. CUSUM charts were computed for each resident on each procedure type. A learning plateau was defined as at least 4 cases consistently locating around the centerline (target performance) at the end of a CUSUM chart with minimal deviations (range 0 to 1).

RESULTS

All residents reached the ACGME graduation targets for the overall POP by the end of chief year. A total of 2,446 cases were included (cholecystectomy N = 1234, colectomy N = 507, and inguinal hernia N = 705), and 3 CUSUM chart patterns emerged: skewed distribution, bimodal distribution, and peaks and valleys distribution. Analysis of CUSUM charts revealed surgery residents' development processes in the operating room towards a learning plateau vary, and only 46.7% residents reach a learning plateau in all 3 procedures upon graduation.

CONCLUSIONS

CUSUM charts of operative time complement the ACGME Milestones evaluations. The use of both may enable residency programs to holistically determine graduating residents' practice readiness and provide recommendations for their upcoming career/practice transition.

Trends in Resident Supervision and Patient Outcomes of Laparoscopic Cholecystectomies Within the Veterans Affairs Healthcare System

OBJECTIVE

We sought to use the Veterans Affairs Surgical Quality Improvement Program (VASQIP) database to determine if there is an increase in morbidity or mortality when resident physicians independently perform laparoscopic cholecystectomy compared to when an attending surgeon is scrubbed.

DESIGN

We performed a retrospective review of 54,144 cases of laparoscopic cholecystectomy performed within the Veterans Affairs (VA) Healthcare system from 2000 to 2020. Cases were divided into groups based on if the attending was scrubbed or not scrubbed. We then performed 1:1 case matching without replacement based on sex, race, and major comorbidities.

PARTICIPANTS

Veterans over age 18 undergoing laparoscopic cholecystectomy within the VA healthcare system between 2000 and 2020. Cases were excluded if a resident was not involved in the surgery or if the level of autonomy was not defined.

RESULTS

Significantly more operative cases were performed without the attending scrubbed before 2003 than after (14.6% vs 1.60%, p < 0.01). After matching, in 1464 (48.6%) cases the attending physician was scrubbed, and in 1549 (51.4%) the attending physician was not scrubbed. Patients were statistically similar in all measured comorbidities between the groups. Operative time was noted to be slightly longer when the attending was scrubbed (1.86 hours ± 0.79 vs 1.72 ± 0.67, p < 0.01) as well as increased complication rates (9.0% vs 6.1%, p < 0.01). No differences existed for 30-day mortality (0.8% vs 0.5%, p = 0.416), postoperative length of stay (2.7 days vs 2.96 days, p = 0.43), or superficial infection (1.9% vs 1.7%, p = 0.73).

CONCLUSIONS

Our analysis of the VASQIP database indicates that decreased resident supervision during laparoscopic cholecystectomy has minimal impact on patient outcomes. Rates of resident independent operating have declined 10-fold since the early 2000's. Further research is required to better define the changes in resident surgical education and their impact on patient outcomes.

The use of an educational time-out in thyroid and parathyroid surgery to move the needle in periprocedural education

BACKGROUND

As surgical training shifts toward a competency-based paradigm, deliberate practice for procedures must be a point of focus. The purpose of this study was to assess the impact of an educational time-out intervention on educational experience and operative performance in endocrine surgery.

METHODS

For 12 months, third-year general surgery residents used the educational time-out to establish an operative step of focus for thyroidectomy and parathyroidectomy procedures. Data were collected using the System for Improving and Measuring Procedural Learning application and post-rotation surveys. The Zwisch scale was used to classify supervision, with meaningful autonomy defined as passive help or supervision only.

RESULTS

Eight residents and 3 attending surgeons performed the educational time-out for a total of 211 operations (93% completion rate). At the end of each rotation, there was improvement in the frequency of goal setting. There was strong agreement (90%) that the intervention strengthened the educational experience. For most cases (52%), the residents were rated at active help. Residents performed a median of 3/6 thyroidectomy steps at meaningful autonomy and a median of 2/5 parathyroidectomy steps at meaningful autonomy. Review of the qualitative data revealed that optimal feedback was provided in 46% of cases.

CONCLUSION

The educational time-out strengthened educational experiences. Stepwise procedural data revealed the varying levels of supervision that exist within an operation. Broader implementation of this intervention could facilitate competency-based procedural education.

How We Do It: Remediation Pathways in a Surgical Simulation Curriculum for Competency Improvement

OBJECTIVE

Validated assessment of procedural knowledge and skills with formative remediation is a foundational part of achieving surgical competency. High-fidelity simulation programs provide a unique area to assess resident proficiency and independence, as well as to assist in identifying residents in need of further practice. While several studies have validated the use of simulation to attain proficiency of specific technical skills, few have validated remediation pathways for their trainees objectively. In this descriptive analysis, we review 2 remediation pathways within our simulation training curricula and how these are used in assessments of resident proficiency.

MATERIALS AND METHODS

Two methods of remediation were formulated for use in high-fidelity simulation labs. One remediation pathway was a summative process, where ultimate judgment of resident competency was assessed through intra-operative assessments of a holistic skill set. The second remediation pathway was a formative "coaching" process, where feedback is given at several intervals along the pathway towards a specific technical skills competence. All general surgery residents are enrolled in the longitudinal, simulation curricula.

RESULTS

Approximately one-third of surgical residents entered into a remediation pathway for either of the high-fidelity simulation curricula. Both residents and faculty expressed support for the summative and formative remediation pathways as constructed. Residents who entered remediation pathways believed it was a beneficial exercise, and the most common feedback was that remediation principles should be expanded to all residents. Interestingly, faculty demonstrated stronger support for the formative coaching feedback model than the summative assessment model.

CONCLUSIONS

Through the complementary use of both formative and summative remediation pathways, resident competence can be enriched in a constructive, nonpunitive method for self-directed performance improvement. Both trainees and faculty express high satisfaction with programs explicitly organized to ensure that skills are rated through a standardized process.

Differences in progression by surgical specialty: a national cohort study

The aim of surgical training across the 10 surgical specialties is to produce competent day 1 consultants. Progression through training in the UK is assessed by the Annual Review of Competency Progression (ARCP).

OBJECTIVE

This study aimed to examine variation in ARCP outcomes within surgical training and identify differences in outcomes between specialties.

DESIGN

A national cohort study using data from the UK Medical Education Database was performed. ARCP outcome was the primary outcome measure. Multilevel ordinal regression analyses were performed, with ARCP outcomes nested within trainees.

PARTICIPANTS

Higher surgical trainees (ST3-ST8) from nine UK surgical specialties were included (vascular surgery was excluded due to insufficient data). All surgical trainees across the UK with an ARCP outcome between 2010 and 2017 were included.

RESULTS

Eight thousand two hundred and twenty trainees with an ARCP outcome awarded between 2010 and 2017 were included, comprising 31 788 ARCP outcomes. There was substantial variation in the proportion of non-standard outcomes recorded across specialties with general surgery trainees having the highest proportion of non-standard outcomes (22.5%) and urology trainees the fewest (12.4%). After adjustment, general surgery trainees were 1.3 times more likely to receive a non-standard ARCP outcome compared with trainees in trauma and orthopaedics (T&O) (OR 1.33, 95% CI 1.21 to 1.45, p=0.001). Urology trainees were 36% less likely to receive a non-standard outcome compared with T&O trainees (OR 0.64, 95% CI 0.54 to 0.75, p<0.001). Female trainees and older age were associated with non-standard outcomes (OR 1.11, 95% CI 1.02 to 1.22, p=0.020; OR 1.04, 95% CI 1.03 to 1.05, p<0.001).

CONCLUSION

There is wide variation in the training outcome assessments across surgical specialties. General surgery has higher rates of non-standard outcomes compared with other surgical specialties. Across all specialties, female sex and older age were associated with non-standard outcomes.

The long game: Evolution of clinical decision making throughout residency and fellowship

BACKGROUND

The purpose of this study was to explore the trajectory of autonomy in clinical decision making.

METHODS

We conducted a qualitative secondary analysis of interviews with 45 residents and fellows from the General Surgery and Obstetrics & Gynecology departments across all clinical postgraduate years (PGY) using convenience sampling. Each interview was recorded, transcribed and iteratively analyzed using a framework method.

RESULTS

A total of 16 junior residents, 22 senior residents and 7 fellows participated in 12 original interviews. Early in training residents take their abstract ideas about disease processes and make them concrete in their applications to patient care. A transitional stage follows in which residents apply concepts to concrete patient care. Chief residents re-abstract their concrete technical and clinical knowledge to prepare for future surgical practice.

CONCLUSIONS

Understanding where each learner is on this pathway will assist development of curriculum that fosters resident readiness for practice at each PGY level.

Validity evidence for the Anesthesia Clinical Encounter Assessment (ACEA) tool to support competency-based medical education

BACKGROUND

Workplace-based assessment (WBA) is key to a competency-based assessment strategy. Concomitantly with our programme's launch of competency-based medical education, we developed an entrustment-based WBA, the Anesthesia Clinical Encounter Assessment (ACEA), to assess readiness for independent practice of competencies essential to perioperative patient care. This study aimed to examine validity evidence of the ACEA during postgraduate anaesthesiology training.

METHODS

The ACEA comprises an eight-item global rating scale (GRS), an overall independence rating, an eight-item checklist, and case details. ACEA data were extracted for University of Toronto anaesthesia residents from July 2017 to January 2020 from the programme's online assessment portal. Validity evidence was generated following Messick's validity framework, including response process, internal structure, relations with other variables, and consequences.

RESULTS

We analysed 8664 assessments for 137 residents completed by 342 assessors. From generalisability analysis, 10 independent observations (two assessments each from five assessors) were sufficient to achieve a reliability threshold of ≥0.70 for in-training assessments. A composite GRS score of 3.65/5 provided optimal sensitivity (93.6%) and specificity (90.8%) for determining entrustment on receiver operator characteristic curve analysis. Test-retest reliability was high (intraclass correlation coefficient [ICC_2,1]=0.81) for matched assessments within 14 days of each other. Composite GRS scores differed significantly between residents based on their training level (P<0.0001) and correlated highly with overall independence (0.91, P<0.001). The internal consistency of the GRS (α=0.96) was excellent.

CONCLUSIONS

This study supports the validity of the ACEA for assessing the competence of residents performing perioperative care and supports its use in competency-based anaesthesiology training.

Entrustment Unpacked: Aligning Purposes, Stakes, and Processes to Enhance Learner Assessment

Educators use entrustment, a common framework in competency-based medical education, in multiple ways, including frontline assessment instruments, learner feedback tools, and group decision making within promotions or competence committees. Within these multiple contexts, entrustment decisions can vary in purpose (i.e., intended use), stakes (i.e., perceived risk or consequences), and process (i.e., how entrustment is rendered). Each of these characteristics can be conceptualized as having 2 distinct poles: (1) purpose has formative and summative, (2) stakes has low and high, and (3) process has ad hoc and structured. For each characteristic, entrustment decisions often do not fall squarely at one pole or the other, but rather lie somewhere along a spectrum. While distinct, these continua can, and sometimes should, influence one another, and can be manipulated to optimally integrate entrustment within a program of assessment. In this article, the authors describe each of these continua and depict how key alignments between them can help optimize value when using entrustment in programmatic assessment within competency-based medical education. As they think through these continua, the authors will begin and end with a case study to demonstrate the practical application as it might occur in the clinical learning environment.

Examining the Development of Operative Autonomy in Vascular Surgery Training and When Trainees and Program Directors Agree and Disagree

BACKGROUND

Lack of autonomy in the operating room (OR) during general surgery residency is a major contributing factor to low confidence operating independently after graduation. Although attempts to address decreased autonomy and development of entrustment in the OR are being made in general surgery programs, this issue has not been examined thoroughly in vascular surgery. We sought to determine barriers and opportunities for developing operative autonomy during vascular surgery training by surveying program directors (PDs) and trainees (integrated residents and fellows) in U.S. vascular surgery training programs.

METHODS

An anonymous electronic survey was sent via email to all PDs (n = 155) and trainees (n = 516) in United States vascular surgery training programs. Demographics, academic characteristics, and responses regarding factors impacting the development of entrustment were collected.

RESULTS

Thirty-five PDs and 100 trainees completed the survey (22.5% and 19.4% response rate, respectively). Sixty percent of trainees were integrated residents and 40% were fellows. Twenty percent of PDs and 33% of trainees were female, and 5% of all PDs and trainees were from underrepresented minorities. The single most positive factor affecting the development of autonomy according to trainees and PDs is familiarity of the faculty with the trainee. Both PDs and trainees thought the trainee's preparation for the case positively affected development of autonomy; however, more PDs believed that involvement with preoperative preparation in particular (marking the patient, consenting the patient, filling out a history and physical, prepping and draping the patient) was important (P < 0.05). PDs believed that duty-hour limitations negatively affected the trainee's ability to develop autonomy in the OR, whereas more trainees believed that hospital or OR efficiency policies played a negative role (P < 0.05). Finally, compared with trainees, PDs believed that the appropriate amount of time for safe struggle before the attending should take over the case was when OR efficiency was compromised or at any moment the trainee is unsure of themselves (P < 0.05); trainees believed that the attending should take over the case after the limit of their skill set or troubleshooting ability was reached (P < 0.05).

CONCLUSIONS

Familiarity of the attending physician with the trainee is an important positive factor for development of entrustment and autonomy in vascular surgery trainees. Duty-hour limitations and belief of the need for hospital efficiency may negatively impact operative independence of trainees. An open discussion about balancing OR efficiency and trainees' safe struggle is essential to address the growth of independent operative skills in vascular surgery trainees.

Repeat Attending Exposure Influences Operative Autonomy in Endocrine Surgical Procedures

PURPOSE

There is concern that graduating surgery residents are not prepared for independent practice. This study aimed to identify predictors of performance, autonomy, and readiness for independence ratings of trainees by attendings for thyroidectomy and parathyroidectomy with respect to repeated resident-attending exposure. We hypothesized that increased exposure with a particular attending increases resident autonomy.

METHODS

All residents and faculty at a single institution performing parathyroidectomy or thyroidectomy were invited to complete an operative performance evaluation at case competition using the Zwisch scale to measure performance and autonomy for individual operative steps. In addition, each survey evaluated the trainee's readiness for practice in a straightforward procedure as a binary variable. Categorical variables were evaluated via Chi-squared or Fisher's exact tests and ordinal variables were evaluated with Wilcoxon or Kruskal-Wallis tests. Multivariable analysis was conducted with random effects logistic regression, and learning curves were generated for each procedure.

RESULTS

Operative performance evaluations were obtained from 36 individual learners and 6 faculty members, with a total of 145 evaluations for parathyroidectomy and 116 for thyroidectomy. On bivariate analysis, readiness for practice ratings was significantly associated with increasing chronologic procedure number, but not resident gender or case difficulty. The multivariable model demonstrated that increasing chronologic procedure number, while a significant predictor without accounting for exposure, did not remain a significant predictor of practice-readiness for parathyroidectomy when accounting for resident-attending exposure. Bivariate analysis comparing resident and attending ratings showed no difference between the 2, but there were significant differences in autonomy and performance scores for both groups of raters. Trainees rated by attendings as independence ready completed a median of 7 parathyroidectomies and 5 thyroidectomies. Descriptive learning curves generated serve as a model of the multistate nature that residents undergo when moving from novice to proficiency.

CONCLUSIONS

Not surprisingly, the more operations residents perform with a single attending, the higher their ratings for performance and autonomy from that individual, with increased exposure allowing improved performance with less attending autonomy. By contrast, our data also show that repeated exposure between resident and attending may confound the use of procedural numbers alone when predicting resident ability in the operating room.

High vs. low entrustment behaviors in the operating room

BACKGROUND

Operative experience with an appropriate degree of supervised autonomy is critical to resident training. Progressively greater intraoperative entrustment has been associated with gradually higher levels of resident autonomy. This study attempts to identify consistently observed intraoperative behaviors that are linked with higher resident entrustment.

METHODS

This qualitative study analyzed observational notes recorded by trained raters who provided entrustment scores for 204 surgical cases at Michigan Medicine from 2015 to 2017. Notes were coded in NVivo12. Thematic analysis was used to identify themes and patterns within the data.

RESULTS

The analysis generated 144 codes. Codes were clustered into 10 themes. These themes manifested differently in intraoperative behaviors strongly associated with high entrustment versus low entrustment.

CONCLUSION

This study demonstrates key differences in intraoperative behaviors exhibited by residents and faculty in high and low entrustment interactions. Awareness of behaviors that enhance entrustment can help faculty augment resident learning and enable higher resident operative autonomy.