Predicting Resident Competence for Otolaryngology Key Indicator Procedures

OBJECTIVE

Competency-based surgical education requires practical assessments and meaningful benchmarks. In otolaryngology, key indicator procedure (KIP) minima are indicators of surgical exposure during training, yet it remains unknown how many times trainees must be evaluated on KIPs to ensure operative competence. Herein, we used Bayesian mixed effects models to compute predicted performance expectations for KIPs.

METHODS

From November 2017 to September 2021, a smartphone application (SIMPL OR) was used by attendings at five otolaryngology training programs to rate resident operative performance after each case on a five-level scale. Bayesian mixed effects models were used to estimate the probability that postgraduate year (PGY) 3, 4, or 5 trainees would earn a "practice-ready" (PR) rating on a subsequent evaluation based on their previously earned PR ratings for each KIP. Probabilities of earning a subsequent PR rating were examined for interpretability, and cross-validation was used to assess predictive validity.

RESULTS

A total of 842 assessments of KIPs were submitted by 72 attendings for 92 residents PGY 2-5. The predictive model had an average Area Under the Receiver Operating Curve of 0.77. The number of prior PR ratings that senior residents needed to attain a 95% probability of earning a PR rating on a subsequent evaluation was estimated for each KIP. For example, for mastoidectomies, PGY4 residents needed to earn 10 PR ratings whereas PGY5 residents needed 4 PR ratings on average to have a 95% probability of attaining a PR rating on a subsequent evaluation.

CONCLUSION

Predictive modeling can inform assessment benchmarks for competency-based surgical education.

LEVEL OF EVIDENCE

NA Laryngoscope, 133:3341-3345, 2023.

Unintended consequences of technology in competency-based education: a qualitative study of lessons learned in an OtoHNS program

BACKGROUND

Formative feedback and entrustment ratings on assessments of entrustable professional activities (EPAs) are intended to support learner self-regulation and inform entrustment decisions in competency-based medical education. Technology platforms have been developed to facilitate these goals, but little is known about their effects on these new assessment practices. This study investigates how users interacted with an e-portfolio in an OtoHNS surgery program transitioning to a Canadian approach to competency-based assessment, Competence by Design.

METHODS

We employed a sociomaterial perspective on technology and grounded theory methods of iterative data collection and analysis to study this OtoHNS program's use of an e-portfolio for assessment purposes. All residents (n = 14) and competency committee members (n = 7) participated in the study; data included feedback in resident portfolios, observation of use of the e-portfolio in a competency committee meeting, and a focus group with residents to explore how they used the e-portfolio and visualize interfaces that would better meet their needs.

RESULTS

Use of the e-portfolio to document, access, and interpret assessment data was problematic for both residents and faculty, but the residents faced more challenges. While faculty were slowed in making entrustment decisions, formative assessments were not actionable for residents. Workarounds to these barriers resulted in a "numbers game" residents played to acquire EPAs. Themes prioritized needs for searchable, contextual, visual, and mobile aspects of technology design to support use of assessment data for resident learning.

CONCLUSION

Best practices of technology design begin by understanding user needs. Insights from this study support recommendations for improved technology design centred on learner needs to provide OtoHNS residents a more formative experience of competency-based training.

Factors affecting operative autonomy and performance during otolaryngology training: A multicenter trial

Objective

Surgical education is changing in an era of new regulations and evolving training cultures. We sought to understand the factors that affect operative experiences during otolaryngology residency.

Methods

From December 2019 to December 2020, five otolaryngology training programs used the SIMPL OR smartphone application to evaluate residents after each operation. Residents and attendings rated the trainee's autonomy on a 4‐level Zwisch scale, performance on a 5‐level scale, and case complexity on a 3‐level scale. We examined associations between ratings of autonomy and performance with variables including postgraduate year (PGY), case complexity, gender, week of the academic year (AY), and whether multiple procedures were logged.

Results

78 attendings and 92 residents logged 2984 evaluations. PGY level and week of the AY were positively associated with attending ratings of autonomy and performance (PGY3 vs. PGY2: B = 0.63, p < .001 for autonomy and B = 1.05, p < .001 for performance; week of the AY: B = 0.013, p = .002 for autonomy; B = 0.025, p < .001 for performance). Multiple procedures logged and increasing case complexity were negatively associated with attending ratings (multiple procedures: B = −0.19, p = .04 for autonomy and B = −0.48, p < .001 for performance; hardest vs. easiest 1/3 of cases: B = −1.01, p < .001 for autonomy and B = −0.59, p < .001 for performance). Attending and trainee genders were not associated with attending ratings of autonomy or performance.

Conclusion

Resident autonomy and performance were positively associated with PGY level and week of the academic year, and negatively associated with case complexity and multiple procedures. These findings highlight the need to align training level with case complexity to promote quality operative experiences.

Level of Evidence

2.

Multi-institutional Study of Otolaryngology Resident Intraoperative Experiences for Key Indicator Procedures

OBJECTIVE

There is concern that current otolaryngology residents may not receive adequate surgical training. We aimed to characterize residents' surgical experiences at 5 academic centers performing the 14 key indicator procedures (KIPs) outlined by the Accreditation Council for Graduate Medical Education.

STUDY DESIGN

Prospective study.

SETTING

Five otolaryngology training programs.

METHODS

Data were gathered from December 2019 to December 2020 with a smartphone application from the Society for Improving Medical Professional Learning. After each operation, residents and faculty rated trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale.

RESULTS

Residents and attendings (n = 92 and 78, respectively) logged 2984 evaluations. Attending ratings of resident autonomy and performance increased with training level (P < .001). Resident self-assessments of autonomy and performance were lower than paired attending assessments (P < .001). Among attending evaluations of KIPs performed by senior residents (postgraduate year 4 or 5), 55% of cases were performed with meaningful autonomy (passive help or supervision only). Similarly, attendings rated 55% of these cases as a practice-ready or exceptional performance. Senior residents had meaningful autonomy for ≥50% of cases for most KIPs, with the exception of flaps and grafts (40%), pediatric/adult airway (39%), and stapedectomy/ossiculoplasty (33%). Similarly, senior residents received practice-ready or exceptional performance ratings for ≥50% of cases across all KIPs other than pediatric/adult airway (42%) and stapedectomy/ossiculoplasty (33%).

CONCLUSION

In this multicenter study, resident surgical autonomy and performance varied across otolaryngology KIPs. The development of nationwide benchmarks will help programs and residents set educational goals.

LEVEL OF EVIDENCE

2.

Autonomy in the Operating Room: A Multicenter Study of Gender Disparities During Surgical Training

BACKGROUND

Gender disparities are prevalent in medicine, but their impact on surgical training is not well studied.

OBJECTIVE

To quantify gender disparities in trainee intraoperative experiences and explore the variables associated with ratings of surgical autonomy and performance.

METHODS

From September 2015 to May 2019, attending surgeons and trainees from 71 programs assessed trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale after surgical procedures. Multivariable regression models were used to examine the association of trainee gender with autonomy and performance evaluations.

RESULTS

A total of 3255 trainees and attending surgeons completed 94 619 evaluations. Attendings gave lower ratings of operative autonomy to female trainees than male trainees when controlling for training level, attending, and surgical procedure (effect size B = -0.0199, P = .008). There was no difference in ratings of autonomy at the beginning of training (P = .32); the gap emerged as trainees advanced in years (B = -0.0163, P = .020). The gender difference in autonomy was largest for the most complex cases (B = -0.0502, P = .002). However, there was no difference in attending ratings of surgical performance for female trainees compared to male trainees (B = -0.0124, P = .066). Female trainees rated themselves as having less autonomy and worse performance than males when controlling for training level, attending, procedure, case complexity, and attending ratings (autonomy B = -0.0669, P < .001; performance B = -0.0704, P < .001).

CONCLUSIONS

While there was no significant difference in ratings of operative performance, a small difference between ratings of operative autonomy for female and male surgical trainees was identified.

A Systematic Review of Educational Mobile-Applications (Apps) for Surgery Residents: Simulation and Beyond

INTRODUCTION

The number of mobile-applications (Apps) increases daily. The regulation of App content is minimal yet surgical residents use these in daily educational practice. Surgical educators must be aware of the quality, efficacy, and validity of Apps available to effectively educate residents. The aim of this review was to determine the quality, efficacy, and validity evidence for educational Apps used by surgical residents.

MATERIALS AND METHODS

We searched PubMed, Embase, and ERIC for articles published before September 1, 2019. Controlled vocabulary and natural language describing Apps/surgical residents were used. Two reviewers evaluated abstracts for inclusion.

INCLUSION CRITERIA

studies measuring the quality, efficacy, or validity of educational Apps for surgical residents. Data was extracted from full text of included articles: study design, participants, App investigated, App development, evidence for efficacy, or validity of App.

RESULTS

Initial search identified 278 articles. 64 articles were duplicates and 214 articles were screened. A further 156 were excluded with 58 full text articles assessed for eligibility. Forty-five were included in analysis. Simulation (9/45) and Feedback (15/45) Apps were the most commonly studied in surgical residents. These were the main Apps that provided validity evidence for their use in education.

CONCLUSION

Surgical education is evolving as educational technology becomes more prevalent. To be effective as surgical educators we must understand and appropriately use available tools. Of the educational Apps studied (21 Apps in 8 categories), only 3 categories reported validity evidence. Future studies should take care to measure validity and efficacy of educational Apps for surgical education to ensure quality control.

Tracking operative autonomy and performance in otolaryngology training using smartphone technology: A single institution pilot study

BACKGROUND

In the era of duty hour restrictions, otolaryngology residents may not gain the operative experience necessary to function autonomously by the end of training. This study quantifies residents' autonomy during key indicator cases, defined by the Accreditation Council for Graduate Medical Education.

STUDY DESIGN

Prospective cohort study.

METHODS

Faculty and residents at a large academic institution were surveyed on the surgical autonomy trainees should achieve for otolaryngology key indicator surgeries at each training level. Residents and faculty used the mobile application "System for Improving and Measuring Procedural Learning" (SIMPL) between December 2017 and July 2018 to log trainees' operative autonomy during cases on a validated four-level Zwisch scale, from "show and tell" to "supervision only."

RESULTS

The study included 40 participants (23 residents and 17 attendings). The survey response rate was 83%. In surveys, residents overestimated the autonomy PGY5 residents should achieve for parotidectomy, rhinoplasty, thyroid/parathyroidectomy, and airway procedures compared with faculty (P < .05). Using SIMPL, 833 evaluations were logged of which 253 were paired evaluations for key indicator cases. Comparing survey predictions with actual cases logged in SIMPL, residents and faculty overestimated the autonomy achieved by senior trainees performing mastoidectomy (PGY5, P < .05) and ethmoidectomy (PGY4/5, P < .05); both felt that senior residents should operate with between "passive help" and "supervision only" whereas residents actually had "passive help." Residents overestimated their autonomy during rhinoplasty (PGY5, P = .017) and parotidectomy (PGY5, P = .007) while attendings accurately expected chief residents to have "passive help."

CONCLUSIONS

Resident surgical autonomy varies across otolaryngology procedures. Multicenter studies are needed to elucidate national trends.

LEVEL OF EVIDENCE

2.