The measured effect of delay in completing operative performance ratings on clarity and detail of ratings assigned

Is time really of the essence? Timeliness of narrative feedback in ophthalmology CBME assessments

PURPOSE

Competency-based medical education relies on a strong program of assessment, and quality comments play a vital role in ensuring its success. The goal of this study is to determine the effect of the timeliness of assessment completion on the quality of the feedback.

MATERIALS AND METHODS

Using the Quality of Assessment for Learning (QuAL) score 2478 assessments were evaluated. The assessments included those completed between July 2017 and December 2020 for 18 ophthalmology residents. Spearman correlation, Mann-Whitney U and Kruskal-Wallis tests were used to assess variations in QuAL scores based on the timeliness of assessment completion.

RESULTS

The timeliness of assessment completion ranged from 0 to 299 d with the mean time for completion being 3 d. As the delay increased, the QuAL score decreased. Feedback provided 4, 5, and 14 d post-encounter demonstrated statistically significant differences in the QuAL score. Additionally, there was a significant difference in the timeliness of feedback when there is no written comment.

CONCLUSIONS

This study demonstrates that the timeliness of assessment completion might have an effect on the quality of written feedback. Written feedback should be completed within 14 d of the encounter to optimize quantity and quality.

Entrustable Professional Activities in Surgery: A Review

Importance

Entrustable professional activities (EPAs) compose a competency-based education (CBE) assessment framework that has been increasingly adopted across medical specialties as a workplace-based assessment tool. EPAs focus on directly observed behaviors to determine the level of entrustment a trainee has for a given activity of that specialty. In this narrative review, we highlight the rationale for EPAs in general surgery, describe current evidence supporting their use, and outline some of the practical considerations for EPAs among residency programs, faculty, and trainees.

Observations

An expanding evidence base for EPAs in general surgery has provided moderate validity evidence for their use as well as practical recommendations for implementation across residency programs. Challenges to EPA use include garnering buy-in from individual faculty and residents to complete EPA microassessments and engage in timely, specific feedback after a case or clinical encounter. When successfully integrated into a program's workflow, EPAs can provide a more accurate picture of residents' competence for a fundamental surgical task or activity compared with other assessment methods.

Conclusions and Relevance

EPAs represent the next significant shift in the evaluation of general surgery residents as part of the overarching progression toward CBE among all US residency programs. While pragmatic challenges to the implementation of EPAs remain, the best practices from EPA and other CBE assessment literature summarized in this review may assist individuals and programs in implementing EPAs. As EPAs become more widely used in general surgery resident training, further analysis of barriers and facilitators to successful and sustainable EPA implementation will be needed to continue to optimize and advance this new assessment framework.

Mobile technologies to support workplace-based assessment for entrustment decisions: Guidelines for programs and educators: AMEE Guide No. 154

With the rise of competency-based medical education and workplace-based assessment (WBA) since the turn of the century, much has been written about methods of assessment. Direct observation and other sources of information have become standard in many clinical programs. Entrustable professional activities (EPAs) have also become a central focus of assessment in the clinical workplace. Paper and pencil (one of the earliest mobile technologies!) to document observations have become almost obsolete with the advent of digital technology. Typically, clinical supervisors are asked to document assessment ratings using forms on computers. However, accessing these forms can be cumbersome and is not easily integrated into existing clinical workflows. With a call for more frequent documentation, this practice is hardly sustainable, and mobile technology is quickly becoming indispensable. Documentation of learner performance at the point of care merges WBA with patient care and WBA increasingly uses smartphone applications for this purpose.This AMEE Guide was developed to support institutions and programs who wish to use mobile technology to implement EPA-based assessment and, more generally, any type of workplace-based assessment. It covers backgrounds of WBA, EPAs and entrustment decision-making, provides guidance for choosing or developing mobile technology, discusses challenges and describes best practices.

Implementation of an Online Intraoperative Assessment of Technical Performance for Surgical Trainees

INTRODUCTION

Assessment of surgical resident technical performance is an integral component of any surgical training program. Timely assessment delivered in a structured format is a critical step to enhance technical skills, but residents often report that the quality and quantity of timely feedback received is lacking. Moreover, the absence of written feedback with specificity can allow residents to seemingly progress in their operative milestones as a junior resident, but struggle as they progress into their postgraduate year 3 and above. We therefore designed and implemented a web-based intraoperative assessment tool and corresponding summary "dashboard" to facilitate real-time assessment and documentation of technical performance.

MATERIALS AND METHODS

A web form was designed leveraging a cloud computing platform and implementing a modified Ottawa Surgical Competency Operating Room Evaluation instrument; this included additional, procedure-specific criteria for select operations. A link to this was provided to residents via email and to all surgical faculty as a Quick Response code. Residents open and complete a portion of the form on a smartphone, then relinquish the device to an attending surgeon who then completes and submits the assessment. The data are then transferred to a secure web-based reporting interface; each resident (together with a faculty advisor) can then access and review all completed assessments.

RESULTS

The Assessment form was activated in June 2021 and formally introduced to all residents in July 2021, with residents required to complete at least one assessment per month. Residents with less predictable access to operative procedures (night float or Intensive Care Unit) were exempted from the requirement on those months. To date a total of 559 assessments have been completed for operations performed by 56 trainees, supervised by 122 surgical faculty and senior trainees. The mean number of procedures assessed per resident was 10.0 and the mean number per assessor was 4.6. Resident initiation of Intraoperative Assessments has increased since the tool was introduced and scores for technical and nontechnical performance reliably differentiate residents by seniority.

CONCLUSIONS

This novel system demonstrates that an online, resident-initiated technical assessment tool is feasible to implement and scale. This model's requirement that the attending enter performance ratings into the trainee's electronic device ensures that feedback is delivered directly to the trainee. Whether this aspect of our assessment ensures more direct and specific (and therefore potentially actionable) feedback is a focus for future study. Our use of commercial cloud computing services should permit cost-effective adoption of similar systems at other training programs.

Early findings and strategies for successful implementation of SIMPL workplace-based assessments within vascular surgery residency and fellowship programs

OBJECTIVE

As medical education systems increasingly move toward competency-based training, it is important to understand the tools available to assess competency and how these tools are utilized. The Society for Improving Medical Professional Learning (SIMPL) offers a smart phone-based assessment system that supports workplace-based assessment of residents' and fellows' operative autonomy, performance, and case complexity. The purpose of this study was to characterize implementation of the SIMPL app within vascular surgery integrated residency (0+5) and fellowship (5+2) training programs.

METHODS

SIMPL operative ratings recorded between 2018 and 2022 were collected from all participating vascular surgery training institutions (n = 9 institutions with 5+2 and 0+5 programs; n = 4 institutions with 5+2 program only). The characteristics of programs, trainees, faculty, and SIMPL operative assessments were evaluated using descriptive statistics.

RESULTS

Operative assessments were completed for 2457 cases by 85 attendings and 86 trainees, totaling 4615 unique operative assessment ratings. Attendings included dictated feedback in 52% of assessments. Senior-level residents received more assessments than junior-level residents (postgraduate year [PGY]1-3, n = 439; PGY4-5, n = 551). Performance ratings demonstrated increases from junior to senior trainees for both resident and fellow cohorts with "performance-ready" or "exceptional performance" ratings increasing by nearly two-fold for PGY1 to PGY5 residents (28.1% vs 40.6%), and from first- to second-year fellows (PGY6, 46.7%; PGY7, 60.3%). Similar gains in autonomy were demonstrated as trainees progressed through training. Senior residents were more frequently granted autonomy with "supervision only" than junior residents (PGY1, 8.7%; PGY5, 21.6%). "Supervision only" autonomy ratings were granted to 21.8% of graduating fellows. Assessment data included a greater proportion of complex cases for senior compared with junior fellows (PGY6, 20.9% vs PGY7, 26.5%). Program Directors felt that faculty and trainee buy-in were the main barriers to implementation of the SIMPL assessment app.

CONCLUSIONS

This is the first description of the SIMPL app as an operative assessment tool within vascular surgery that has been successfully implemented in both residency and fellowship programs. The assessment data demonstrates expected progressive gains in trainees' autonomy and performance, as well as increasing case complexity, across PGY years. Given the selection of SIMPL as the assessment platform for required American Board of Surgery and Vascular Surgery Board Entrustable Professional Activities assessments, understanding facilitators and barriers to implementation of workplace-based assessments using this app is imperative, particularly as we move toward competency-based medical education.

Real-Time Student Feedback on the Surgical Learning Environment: Use of a Mobile Application

OBJECTIVE

Improvements to the medical student surgical learning environment are limited by lack of granular data and recall bias on end-of-clerkship evaluations. The purpose of this study was to identify specific areas for intervention using a novel real-time mobile application.

DESIGN

An application was designed to obtain real-time feedback from medical students regarding the learning environment on their surgical clerkship. Thematic analysis of student experiences was performed at the conclusion of 4 consecutive 12-week rotation blocks.

SETTING

Brigham and Women's Hospital/Harvard Medical School, Boston, Massachusetts.

RESULTS

Fifty-four medical students at a single institution were asked to participate during their primary clerkship experience. Students submitted 365 responses over 48 weeks. Multiple themes emerged which were dichotomized into positive and negative emotions centered on specific student priorities. Approximately half of responses were associated with positive emotions (52.9%) and half with negative emotions (47.1%). Student priorities included the desire to feel included in the surgical team (resulting in feeling engaged/ignored), to have a positive relationship with members of the team (perceiving kind/rude interactions), to witness compassionate patient care (observing empathy/disrespect for patients), to have a well-planned surgical rotation (experiencing organization/disorganization within teams), and to feel that student well-being is prioritized (reporting opportunities/disregard for student wellness).

CONCLUSION

A novel, user-friendly mobile application identified several areas to improve the experience and engagement of students on their surgery clerkship. Allowing clerkship directors and other educational leaders to collect longitudinal data in real time may allow for more targeted, timely improvements to the medical student surgical learning environment.

From Their Eyes: What Constitutes Quality Formative Written Feedback for Neurosurgery Residents

BACKGROUND

The characteristics of quality feedback from the neurosurgery resident's perspective are not fully elucidated. The Surgical Autonomy Program is an intraoperative assessment tool based on Vygotsky's Zone of Proximal Development (ZPD). SAP facilitates assessment of a resident's operative performance accompanied by written feedback.

OBJECTIVE

The goal of this study was twofold: to identify themes from the written feedback of SAP operative assessments and to examine if these themes influenced the neurosurgery residents' perception of feedback quality.

METHODS

In 2021, SAP data from 2019-2021 at two neurosurgery programs were reviewed. Feedback quality from the SAP was determined by the resident at the time of their assessment. Using a constant comparative technique, the feedback was coded using a thematic analysis. The quality of feedback within each code was analyzed.

RESULTS

There were 2968 SAP entries evaluated. When the ZPD concept was fully used, residents reported high quality feedback 91.4% of the time compared to 58.6% when ZPD was not used (p < 0.001). Qualitative analysis of the written feedback revealed five themes: Non-Specific, Specific General Observations, Key Points, Next Steps, and Independent Practice. Feedback in the Specific General Observations, Key Points, and Independent Practice categories were associated with higher level feedback than leaving the space blank (p < 0.001) or writing Non-Specific comments (p < 0.001).

CONCLUSIONS

Providing comments that discuss the resident's specific performance in the case, key learning points, or their progress towards independence, results in high quality feedback. Utilizing a theory-based tool such as the SAP can provide meaningful feedback to neurosurgical residents.

A Resident-Driven Mobile Evaluation System Can Be Used to Augment Traditional Surgery Rotation Evaluations

BACKGROUND

The Accreditation Council for Graduate Medical Education requires residents to receive milestone-based evaluations in key areas. Shortcomings of the traditional evaluation system (TES) are a low completion rate and delay in completion. We hypothesized that adoption of a mobile evaluation system (MES) would increase the number of evaluations completed and improve their timeliness.

METHODS

Traditional evaluations for a general surgery residency program were converted into a web-based form via a widely available, free, and secure application and implemented in August 2017. After 8 months, MES data were analyzed and compared to that of our TES.

RESULTS

122 mobile evaluations were completed; 20% were solicited by residents. Introduction of the MES resulted in an increased number of evaluations per resident (P = .0028) and proportion of faculty completing evaluations (P = .0220). Timeliness also improved, with 71% of evaluations being completed during one's clinical rotation.

CONCLUSIONS

A resident-driven MES is an inexpensive and effective method to augment traditional end-of-rotation evaluations.

The use of an educational time-out in thyroid and parathyroid surgery to move the needle in periprocedural education

BACKGROUND

As surgical training shifts toward a competency-based paradigm, deliberate practice for procedures must be a point of focus. The purpose of this study was to assess the impact of an educational time-out intervention on educational experience and operative performance in endocrine surgery.

METHODS

For 12 months, third-year general surgery residents used the educational time-out to establish an operative step of focus for thyroidectomy and parathyroidectomy procedures. Data were collected using the System for Improving and Measuring Procedural Learning application and post-rotation surveys. The Zwisch scale was used to classify supervision, with meaningful autonomy defined as passive help or supervision only.

RESULTS

Eight residents and 3 attending surgeons performed the educational time-out for a total of 211 operations (93% completion rate). At the end of each rotation, there was improvement in the frequency of goal setting. There was strong agreement (90%) that the intervention strengthened the educational experience. For most cases (52%), the residents were rated at active help. Residents performed a median of 3/6 thyroidectomy steps at meaningful autonomy and a median of 2/5 parathyroidectomy steps at meaningful autonomy. Review of the qualitative data revealed that optimal feedback was provided in 46% of cases.

CONCLUSION

The educational time-out strengthened educational experiences. Stepwise procedural data revealed the varying levels of supervision that exist within an operation. Broader implementation of this intervention could facilitate competency-based procedural education.

Best Practices for Remediation in Pulmonary and Critical Care Medicine Fellowship Training

Background

Remediation of struggling learners in pulmonary and critical care fellowship programs is a challenge, even for experienced medical educators.

Objective

This evidence-based narrative review provides a framework program leaders may use to address fellows having difficulty achieving competency during fellowship training.

Methods

The relevant evidence for approaches on the basis of each learner’s needs is reviewed and interpreted in the context of fellowship training in pulmonary medicine and critical care. Issues addressed include bias in fellow assessments and remediation, the impacts of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) pandemic, the specific challenges of pulmonary and critical care fellowship programs, a brief review of relevant legal issues, guidance on building and leveraging program resources, and a discussion of learner outcomes.

Results

This results in a concise, evidence-based toolkit for program leaders based around four pillars: early identification, fellow assessment, collaborative intervention, and reassessment. Important concepts also include the need for documentation, clear and written communication, and fellow-directed approaches to the creation of achievable goals.

Conclusion

Evidence-based remediation helps struggling learners in pulmonary and critical care fellowship to improve their ability to meet Accreditation Council for Graduate Medical Education (ACGME) milestones.

Empowering the Resident Learner

Surgical resident autonomy is an essential element of the transition from supervised training to independent practice. However, in an age of duty hour restrictions, legislative constraints, increased litigation, and heightened societal expectations, training an autonomous resident proves increasingly difficult. To tackle these barriers to training successful surgeons, the American College of Surgeons Committee on Resident Education initiated the Resident Mentored Autonomy Project. As a subdivision of this project, the Empowered Learner research team here presents a framework for training the self-directed and empowered surgical resident learner. There are many strategies by which surgical faculty, program directors and chairs, and residents themselves may engage to improve resident operative autonomy.

The Quality of Operative Performance Narrative Feedback: A Retrospective Data Comparison Between End of Rotation Evaluations and Workplace-based Assessments

OBJECTIVE

To describe the quality of operative performance feedback using evaluation tools commonly used by general surgery residency training programs.

SUMMARY OF BACKGROUND DATA

The majority of surgical training programs administer an evaluation through which faculty members may rate and comment on trainee operative performance at the end of the rotation (EOR). Many programs have also implemented the system for improving and measuring procedural learning (SIMPL), a workplace-based assessment tool with which faculty can rate and comment on a trainee's operative performance immediately after a case. It is unknown how the quality of narrative operative performance feedback delivered with these tools compares.

METHODS

The authors collected EOR evaluations and SIMPL narrative comments on trainees' operative performance from 3 university-based surgery training programs during the 2016-2017 academic year. Two surgeon raters categorized comments relating to operative skills as being specific or general and as encouraging and/or corrective. Comments were then classified as effective, mediocre, ineffective, or irrelevant. The frequencies with which comments were rated as effective were compared using Chi-square analysis.

RESULTS

The authors analyzed a total of 600 comments. 10.7% of EOR and 58.3% of SIMPL operative performance evaluation comments were deemed effective (P < 0.0001).

CONCLUSIONS

Evaluators give significantly higher quality operative performance feedback when using workplace-based assessment tools rather than EOR evaluations.

Factors affecting operative autonomy and performance during otolaryngology training: A multicenter trial

Objective

Surgical education is changing in an era of new regulations and evolving training cultures. We sought to understand the factors that affect operative experiences during otolaryngology residency.

Methods

From December 2019 to December 2020, five otolaryngology training programs used the SIMPL OR smartphone application to evaluate residents after each operation. Residents and attendings rated the trainee's autonomy on a 4‐level Zwisch scale, performance on a 5‐level scale, and case complexity on a 3‐level scale. We examined associations between ratings of autonomy and performance with variables including postgraduate year (PGY), case complexity, gender, week of the academic year (AY), and whether multiple procedures were logged.

Results

78 attendings and 92 residents logged 2984 evaluations. PGY level and week of the AY were positively associated with attending ratings of autonomy and performance (PGY3 vs. PGY2: B = 0.63, p < .001 for autonomy and B = 1.05, p < .001 for performance; week of the AY: B = 0.013, p = .002 for autonomy; B = 0.025, p < .001 for performance). Multiple procedures logged and increasing case complexity were negatively associated with attending ratings (multiple procedures: B = −0.19, p = .04 for autonomy and B = −0.48, p < .001 for performance; hardest vs. easiest 1/3 of cases: B = −1.01, p < .001 for autonomy and B = −0.59, p < .001 for performance). Attending and trainee genders were not associated with attending ratings of autonomy or performance.

Conclusion

Resident autonomy and performance were positively associated with PGY level and week of the academic year, and negatively associated with case complexity and multiple procedures. These findings highlight the need to align training level with case complexity to promote quality operative experiences.

Level of Evidence

2.

Multi-institutional Study of Otolaryngology Resident Intraoperative Experiences for Key Indicator Procedures

OBJECTIVE

There is concern that current otolaryngology residents may not receive adequate surgical training. We aimed to characterize residents' surgical experiences at 5 academic centers performing the 14 key indicator procedures (KIPs) outlined by the Accreditation Council for Graduate Medical Education.

STUDY DESIGN

Prospective study.

SETTING

Five otolaryngology training programs.

METHODS

Data were gathered from December 2019 to December 2020 with a smartphone application from the Society for Improving Medical Professional Learning. After each operation, residents and faculty rated trainee autonomy on a 4-level Zwisch scale and performance on a 5-level modified Dreyfus scale.

RESULTS

Residents and attendings (n = 92 and 78, respectively) logged 2984 evaluations. Attending ratings of resident autonomy and performance increased with training level (P < .001). Resident self-assessments of autonomy and performance were lower than paired attending assessments (P < .001). Among attending evaluations of KIPs performed by senior residents (postgraduate year 4 or 5), 55% of cases were performed with meaningful autonomy (passive help or supervision only). Similarly, attendings rated 55% of these cases as a practice-ready or exceptional performance. Senior residents had meaningful autonomy for ≥50% of cases for most KIPs, with the exception of flaps and grafts (40%), pediatric/adult airway (39%), and stapedectomy/ossiculoplasty (33%). Similarly, senior residents received practice-ready or exceptional performance ratings for ≥50% of cases across all KIPs other than pediatric/adult airway (42%) and stapedectomy/ossiculoplasty (33%).

CONCLUSION

In this multicenter study, resident surgical autonomy and performance varied across otolaryngology KIPs. The development of nationwide benchmarks will help programs and residents set educational goals.

LEVEL OF EVIDENCE

2.

Evaluating chief resident readiness for the teaching assistant role: The Teaching Evaluation assessment of the chief resident (TEACh-R) instrument

BACKGROUND

The American Board of Surgery has mandated chief residents complete 25 cases in the teaching assistant (TA) role. We developed a structured instrument, the Teaching Evaluation and Assessment of the Chief Resident (TEACh-R), to determine readiness and provide feedback for residents in this role.

METHODS

Senior (PGY3-5) residents were scored on technical and teaching performance by faculty observers using the TEACh-R instrument in the simulation lab. Residents were provided with their TEACh-R scores and surveyed on their experience.

RESULTS

Scores in technical (p < 0.01) and teaching (p < 0.01) domains increased with PGY. Higher technical, but not teaching, scores correlated with attending-rated readiness for operative independence (p 0.02). Autonomy mismatch was inversely correlated with teaching competence (p < 0.01). Residents reported satisfaction with TEACh-R feedback and desire for use of this instrument in operating room settings.

CONCLUSION

Our TEACh-R instrument is an effective way to assess technical and teaching performance in the TA role.

Perspectives of Recent Graduates on Clerkship Procedural Skill Training at a Canadian Medical School: an Exploratory Study

The implementation of competency-based medical education in Canada has presented both unique opportunities and challenges for improving undergraduate procedural skills curricula. Despite the recognized importance of procedural skills, there remains a lack of national congruency in procedural training across medical schools that must be addressed. When undertaking such curricular development, obtaining learner feedback is a crucial step that can facilitate practical changes and address disparities. The purpose of the current study is to explore the perspectives and insights of recent medical graduates surrounding the clerkship procedural skills curriculum at a Canadian medical school. Six residents from a variety of program specialties participated in a semi-structured focus group interview discussing key aspects of procedural skill training. The focus group was later transcribed and qualitatively analyzed for themes. The results highlight barriers to competency-based procedural skill training involving time constraints and obtaining required evaluations, and the ability of students to self-advocate for learning opportunities. Participants note few opportunities to practice nasogastric tube insertion and casting in particular. Recommendations for curricular improvement are discussed, including options for curricular remediation and resident perspectives on which procedural skills undergraduate trainees should achieve competency in by graduation.

US general surgical trainee performance for representative global surgery procedures

BACKGROUND

Many US general surgery residents are interested in global surgery, but their competence with key procedures is unknown.

METHODS

Using a registry managed by the Society for Improving Medical Professional Learning (SIMPL), we extracted longitudinal operative performance ratings data for a national cohort of US general surgery residents. Operative performance at the time of graduation was estimated via a Bayesian generalized linear mixed model.

RESULTS

Operative performance ratings for 12,976 procedures performed by 1584 residents in 52 general surgery programs were analyzed. These spanned 17 of 31 (55%) procedures deemed important for global surgical practice. For these procedures, the probability of a graduating resident being deemed competent to perform a procedure was 0.95 (95% confidence interval 0.86-1.00) but was less than 0.9 for 3 observed procedures.

CONCLUSION

Our results highlight gaps in the preparedness of US general surgery trainees to perform procedures deemed most important for global surgery settings.

A Proposed Blueprint for Operative Performance Training, Assessment, and Certification

OBJECTIVE

The aim of this study was to propose an evidence-based blueprint for training, assessment, and certification of operative performance for surgical trainees.

SUMMARY BACKGROUND DATA

Operative skill is a critical aspect of surgical performance. High-quality assessment of operative skill therefore has profound implications for training, accreditation, certification, and the public trust of the profession. Current methods of operative skill assessment for surgeons rely heavily on global assessment strategies across a very broad domain of procedures. There is no mechanism to assure technical competence for individual procedures. The science and scalability of operative skill assessment has progressed significantly in recent decades, and can inform a much more meaningful strategy for competency-based assessment of operative skill than has been previously achieved.

METHODS

The present article reviews the current status and science of operative skill assessment and proposes a template for competency-based assessment which could be used to update training, accreditation, and certification processes. The proposal is made in reference to general surgery but is more generally applicable to other procedural specialties.

RESULTS

Streamlined, routine assessment of every procedure performed by surgical trainees is feasible and would enable a more competency-based educational paradigm. In light of the constraints imposed by both clinical volume and assessment bias, trainees should be expected to become proficient and be measured against a mastery learning standard only for the most important and highest-frequency procedures. For less frequently observed procedures, performance can be compared to a norm-referenced standard and, to provide an overall trajectory of performance, analyzed in aggregate. Key factors in implementing this approach are the number of evaluations, the number of raters, the timeliness of evaluation, and evaluation items.

CONCLUSIONS

A competency-based operative skill assessment can be incorporated into surgical training, assessment, and certification. The time has come to develop a systematic approach to this issue as a means of demonstrating professional standards worthy of the public trust.

Predicting prospective resident entrustment: From evaluation to action

OBJECTIVE

We aimed to identify potential variables predictive of a resident achieving faculty future entrustment as a way to enhance attending surgeons' planning of teaching in the operating room leading to improved resident operative autonomy in practice.

METHODS

We reviewed 273 resident performance evaluations from 91 surgical cases that were collected from 11 general surgery chief residents and 16 attending surgeons between April 2018 and June 2019 using a validated evaluation instrument. The primary outcome measure was prospective resident entrustment estimated by the rater for future similar cases. We used descriptive statistics and the boosted tree analysis model to find potential predictors for the outcome measure and examine test-retest reliability by procedure.

RESULTS

Step-specific guidance (r = 0.77, p < 0.0001) was the variable most highly associated with prospective resident entrustment in bivariate linear analysis. The boosted tree analysis demonstrated step-specific guidance was the strongest predictor for prospective resident entrustment in the OR, and its predictive importance was much higher than the overall guidance (0.64 > 0.18). Test-retest reliability was from 0.93 to 0.98 across procedures, indicating the likelihood that attending surgeons granted future autonomy complied with their evaluation of prospective resident entrustment was high.

CONCLUSIONS

By assessing step-specific guidance, attending surgeons can reliably judge residents' future entrustment and potentially better plan for operative teaching/supervision that may lead to granting a surgical resident operative autonomy on similar cases in the future. Our findings provide insight into prospective faculty development of surgical teaching aimed at improving resident readiness for independent practice.

Utilization of a quality reporting system to increase faculty participation in resident operative assessment

BACKGROUND

A quality collaborative across our hospital system was initiated to track surgical outcomes. We sought to determine whether incorporating a resident operative performance assessment into this quality collaborative would increase the quantity and quality of these assessments and impact relevant milestones.

METHODS

A resident operative assessment was added to a quality reporting system required to be completed by faculty at the completion of 2 operations. Three milestones directly related to operative performance were analyzed-Patient Care 3, Medical Knowledge 2, and Interpersonal and Communication Skills 3. Residents were divided in 2 groups: quality collaborative (≥10 operative assessments) and no quality collaborative (<10 operative assessments). Milestones from Spring 2019 and Fall 2019 were analyzed.

RESULTS

Faculty participation was 86% with 407 assessments completed from February to October 2019. A difference in the rate of change in resident performance for Patient Care 3 (+0.95 vs +0.55; P = .04) and Interpersonal and Communication Skills 3 (+1.05 vs +0.52; P = .02) was observed for those residents in the quality collaborative group (n = 20) compared with baseline data.

CONCLUSION

Addition of an operative assessment to a mandatory quality collaborative increases faculty participation and impacts resident milestone determination. These findings highlight opportunities to find innovative and efficient methods to improve faculty engagement.

Association Between Entrustable Professional Activities and Milestones Evaluations: Real-time Assessments Correlate With Semiannual Reviews

OBJECTIVE

Entrustable professional activities (EPAs) have been developed to refine competency-based education. The American Board of Surgery has initiated a 2-year pilot study to evaluate the impact of EPAs on the evaluation and feedback of surgical residents. The ACGME Milestones in Surgery is a semiannual competency-based evaluation program to measure resident progression through 16 professional attributes across 8 practice domains. The correlation between these 2 evaluation tools remains unclear. The purpose of this study is to evaluate this correlation through comparison of an EPA with the corresponding elements of the ACGME Milestones.

DESIGN

From July, 2018 to October, 2019, all residents submitting EPA evaluations for gall bladder disease were evaluated for preoperative, intraoperative, and/or postoperative entrustability. The ratings were converted to a numerical rank from 0 to 4. Milestones scores from May 2019 and November 2019 were obtained for each resident, with scores ranging from 0 to 4. The gall bladder EPA incorporates the operative PC3 and MK2 and nonoperative PC1, PC2, and ICS3 components. Spearman rank correlation was conducted to evaluate the association between each resident's median EPA ranking and his/her milestones scores.

SETTING

SUNY Upstate Medical University, Syracuse, NY, a university-based hospital.

PARTICIPANTS

General surgery residents.

RESULTS

Among 24 residents, 106 intraoperative EPA evaluations were. For both the May and November milestones, significant positive correlations were noted for PC3 (correlation coefficient ρ = 0.690, p < 0.001; ρ = 0.876, p < 0.001). Similarly, for MK2, a significant positive correlation was noted (ρ = 0.882, p < 0.001; ρ = 0.759, p < 0.001). Interestingly, significant positive correlations were also identified between the 3 nonoperative milestones and the intraoperative entrustability ranking.

CONCLUSIONS

We observed significant correlations between EPAs for cholecystectomy and associated milestones evaluation scores. These findings indicate that EPAs may provide more timely and specific feedback than existing tools and, on aggregate, may improve upon existing formative feedback practices provided through the biannual evaluation of surgical residents.

Constructing Learning Curves to Benchmark Operative Performance of General Surgery Residents Against a National Cohort of Peers

OBJECTIVE

No method or data exist to allow surgical trainees or their programs to contextualize their technical progress. The objective of this study was to create peer benchmarks for Cumulative Sum (CUSUM) charts based upon operative evaluations from a national cohort of general surgery residents.

DESIGN, SETTING, PARTICIPANTS

In 2016-2018, faculty from 26 general surgery residency programs nationwide rated 328 residents' operative performance on a case-by-case basis using a validated 5-point Likert scale. An individual case was considered a "misstep" if scoring below the national median score for that procedure in that postgraduate year (PGY). We constructed 2-sided observed-expected CUSUM charts to capture each resident's cumulative performance over time relative to the national medians. Upper (failure) and lower (positive outlier) benchmarks were established based on the PGY-specific 75th percentile and median misstep rates; consistent/repeated missteps are reflected by crossing of the upper boundary. Procedures with ≤10 observations and residents who were evaluated <10 times for each PGY were excluded.

RESULTS

Around 8,161 evaluations on 76 procedure types were analyzed. The individual misstep rate was lowest among PGY-3s at 13.3% and highest among PGY-4s at 28.6%. No interns had curves that crossed the failure boundary. 8.7% of PGY-2s and 8.9% of PGY-3s finished the year past the failure boundary. PGY-2s had the most positive outliers, with 28.3% of them demonstrating an outlying success performance beyond the lower boundary for at least once. PGY-5s most frequently failed, with 16.7% ever crossing the upper boundary and 11.1% remaining above it at graduation.

CONCLUSIONS

CUSUM is a valid statistical approach for benchmarking individual residents' operative performance against national peers as they progress through the year in real-time. With further validation, CUSUM could be used to set progression and/or graduation standards and objectively identify residents who might benefit from remediation.

A Preoperative Educational Time-Out is Associated with Improved Resident Goal Setting and Strengthens Educational Experiences

OBJECTIVE

The purpose of this study was to assess the impact of a preoperative Educational Time-Out (ETO) with structured postoperative feedback on resident preoperative goal-setting and the educational experience of a clinical rotation.

DESIGN

A preoperative ETO was developed during which trainees and faculty jointly identified an operative goal and discussed the trainee's operative autonomy. Postoperative feedback with a smartphone application was encouraged. From November 2016 to October 2017, the intervention was piloted with 1 surgical service. Outcomes included ETO completion rate, goal setting rate, and subjects' perception of the impact of the ETO on identification of performance deficits, trainee autonomy, and receipt of feedback. Data were analyzed using descriptive statistics.

SETTING

This study was performed in an institutional hospital setting.

PARTICIPANTS

Third-year general surgery residents and surgical faculty in the Department of Hepatobiliary Surgery and Liver Transplantation at Vanderbilt University Medical Center took part in the intervention.

RESULTS

Seven residents and 7 attending surgeons participated in this study. Residents performed a median of 15 procurements during an average of 6.5 weeks each on service. The ETO completion rate was 83%. Resident-reported preoperative goal setting increased after the intervention (from 36% to 78%, p = 0.015). Subjects reported a positive impact of the intervention, with high resident agreement that the ETO helped identify deficits (82% median agreement), increased autonomy (82% median agreement), and increased receipt of feedback (84% median agreement). Residents and attendings agreed that the educational experience was stronger due to the ETO (median 81% and 77%, respectively).

CONCLUSIONS

The ETO intervention improved rates of resident preoperative goal setting and strengthened perceived educational experiences. Resident participants also reported improvements in autonomy and rates of postoperative feedback. Broader implementation of this brief preoperative pause is an easy way to emphasize procedural education in the operating room.

Assessment of Operative Autonomy and Readiness for Independent Practice Among Pediatric Surgery Fellows

PURPOSE

The purpose of this study was to evaluate the implementation of the System for Improving and Measuring Procedural Learning (SIMPL) "app" in a pediatric surgery training program. SIMPL is a novel workplace-based operative performance assessment platform which allows case by case assessment of trainees using three scales (autonomy, performance, and case complexity) and takes seconds to complete.

METHODS

A pediatric surgery-specific procedural taxonomy for SIMPL was developed from the ACGME procedural library. SIMPL was piloted and implemented in a single pediatric surgery training program. A descriptive analysis of the operative assessments was undertaken.

RESULTS

In the year following SIMPL implementation, 565 (39% of cases) assessments were completed by 21 faculty for 3 fellows for 148 unique procedures within a median of 8.5 h. Recorded feedback accompanied 61% of assessments. 2nd year fellows were more likely to be deemed autonomous (81% ratings) and practice ready (84%) vs. 1st year fellows (50%; p < 0.001 both), with improvements over time.

CONCLUSION

Within pediatric surgery, this is the first implementation of a workplace-based operative performance assessment "app". With its ease of use, SIMPL drastically increased the volume of operative evaluations and diversified the mix of cases evaluated while SIMPL ratings demonstrated expected and graduated performance improvements over time.

TYPE OF STUDY

Education.

LEVEL OF EVIDENCE

N/A.

Association of intraoperative entrustment with clinical competency amongst general surgery residents

BACKGROUND

Lack of transparency and meaningful assessment in surgical residency has led to inconsistent intraoperative entrustment and highly variable trainee competence at graduation. The relationship between faculty entrustment and resident entrustability on clinical competency remains unclear. We sought to evaluate the dynamic between entrustment/entrustability and clinical competency in general surgery residency.

METHODS

Intraoperative observations were conducted across a 22-month period at an academic tertiary center. Entrustment/entrustability were measured using OpTrust. Clinical competencies were appraised via ACGME Milestones and Objective Structured Assessment of Technical Skill (OSATS) scores. Mixed effects linear regression was used to investigate the relationship among overall ACGME Milestone scores, OSATS domain scores, and overall OpTrust scores.

RESULTS

Overall OpTrust scores significantly correlated with overall Milestone scores and multiple OSATS score domains.

CONCLUSIONS

OpTrust demonstrated a positive association between ACGME general surgery Milestones and OSATS scores. Overall, OpTrust may help optimize intraoperative faculty entrustment and resident entrustability, facilitating surgical trainee success during residency.

Do End-of-Rotation Evaluations Adequately Assess Readiness to Operate?

PURPOSE

Medical educators have developed no standard way to assess the operative performance of surgical residents. Most residency programs use end-of-rotation (EOR) evaluations for this purpose. Recently, some programs have implemented workplace-based "microassessment" tools that faculty use to immediately rate observed operative performance. The authors sought to determine (1) the degree to which EOR evaluations correspond to workplace-based microassessments and (2) which factors most influence EOR evaluations and directly observed workplace-based performance ratings and how the influence of those factors differs for each assessment method.

METHOD

In 2017, the authors retrospectively analyzed EOR evaluations and immediate postoperative assessment ratings of surgical trainees from a university-based training program from the 2015-2016 academic year. A Bayesian multivariate mixed model was constructed to predict operative performance ratings for each type of assessment.

RESULTS

Ratings of operative performance from EOR evaluations vs workplace-based microassessment ratings had a Pearson correlation of 0.55. Postgraduate year (PGY) of training was the most important predictor of operative performance ratings on EOR evaluations: Model estimates ranged from 0.62 to 1.75 and increased with PGY. For workplace-based assessment, operative autonomy rating was the most important predictor of operative performance (coefficient = 0.74).

CONCLUSIONS

EOR evaluations are perhaps most useful in assessing the ability of a resident to become a surgeon compared with other trainees in the same PGY of training. Workplace-based microassessments may be better for assessing a trainee's ability to perform specific procedures autonomously, thus perhaps providing more insight into a trainee's true readiness for operative independence.

Tracking operative autonomy and performance in otolaryngology training using smartphone technology: A single institution pilot study

BACKGROUND

In the era of duty hour restrictions, otolaryngology residents may not gain the operative experience necessary to function autonomously by the end of training. This study quantifies residents' autonomy during key indicator cases, defined by the Accreditation Council for Graduate Medical Education.

STUDY DESIGN

Prospective cohort study.

METHODS

Faculty and residents at a large academic institution were surveyed on the surgical autonomy trainees should achieve for otolaryngology key indicator surgeries at each training level. Residents and faculty used the mobile application "System for Improving and Measuring Procedural Learning" (SIMPL) between December 2017 and July 2018 to log trainees' operative autonomy during cases on a validated four-level Zwisch scale, from "show and tell" to "supervision only."

RESULTS

The study included 40 participants (23 residents and 17 attendings). The survey response rate was 83%. In surveys, residents overestimated the autonomy PGY5 residents should achieve for parotidectomy, rhinoplasty, thyroid/parathyroidectomy, and airway procedures compared with faculty (P < .05). Using SIMPL, 833 evaluations were logged of which 253 were paired evaluations for key indicator cases. Comparing survey predictions with actual cases logged in SIMPL, residents and faculty overestimated the autonomy achieved by senior trainees performing mastoidectomy (PGY5, P < .05) and ethmoidectomy (PGY4/5, P < .05); both felt that senior residents should operate with between "passive help" and "supervision only" whereas residents actually had "passive help." Residents overestimated their autonomy during rhinoplasty (PGY5, P = .017) and parotidectomy (PGY5, P = .007) while attendings accurately expected chief residents to have "passive help."

CONCLUSIONS

Resident surgical autonomy varies across otolaryngology procedures. Multicenter studies are needed to elucidate national trends.

LEVEL OF EVIDENCE

2.

Using smartphones for trainee performance assessment: A SIMPL case study

Workplace-based assessments are used by raters to evaluate observed performance of trainees in actual clinical practice. These types of assessments are of growing interest, especially because observed performance is prioritized within the larger competency-based medical educational movement. Implementation of workplace-based assessments has, however, been challenging. This article describes the motivations and implications for workplace-based assessments that leverage smartphone technology. It does so in reference to an app called SIMPL (System for Improving and Measuring Procedural Learning) in order to highlight some of the challenges and benefits one might encounter during implementation of similar systems.

Assessments of Otolaryngology Resident Operative Experiences Using Mobile Technology: A Pilot Study

Objectives

Surgical education has shifted from the Halstedian model of “see one, do one, teach one” to a competency-based model of training. Otolaryngology residency programs can benefit from a fast and simple system to assess residents’ surgical skills. In this quality initiative, we hypothesize that a novel smartphone application called System for Improving and Measuring Procedural Learning (SIMPL) could be applied in an otolaryngology residency to facilitate the assessment of resident operative experiences.

Methods

The Plan Do Study Act method of quality improvement was used. After researching tools of surgical assessment and trialing SIMPL in a resident-attending pair, we piloted SIMPL across an otolaryngology residency program. Faculty and residents were trained to use SIMPL to rate resident operative performance and autonomy with a previously validated Zwisch Scale.

Results

Residents (n = 23) and faculty (n = 17) were trained to use SIMPL using a standardized curriculum. A total of 833 assessments were completed from December 1, 2017, to June 30, 2018. Attendings completed a median 20 assessments, and residents completed a median 14 self-assessments. All evaluations were resident initiated, and attendings had a 78% median response rate. Evaluations took residents a median 22 seconds to complete; 126 unique procedures were logged, representing all 14 key indicator cases for otolaryngology.

Discussion

This is the first residency-wide application of a mobile platform to track the operative experiences of otolaryngology residents.

Implications for Practice

We adapted and implemented a novel assessment tool in a large otolaryngology program. Future multicenter studies will benchmark resident operative experiences nationwide.

Obtaining Meaningful Assessment in Thoracic Surgery Education

Training in thoracic surgery has evolved immensely over the past decade due to the advent of integrated programs, technological innovations, and regulations on resident duty hours, decreasing the time trainees have to learn. These changes have made assessment of thoracic surgical trainees even more important. Shifts in medical education have increasingly emphasized competency, which has led to novel competency-based assessment tools for clinical and operative assessment. These novel tools take advantage of simulation and modern technology to provide more frequent and comprehensive assessment of the surgical trainee to ensure competence.

Value and Barriers to Use of the SIMPL Tool for Resident Feedback

OBJECTIVE

The System for Improving and Measuring Procedural Learning (SIMPL) is a smart-phone application used to provide residents with an evaluation of operative autonomy and feedback. This study investigated the perceived benefits and barriers to app use.

DESIGN

A database of previously performed SIMPL evaluations was analyzed to identify high, low, and never users. Potential predisposing factors to use were explored. A survey investigating key areas of value and barriers to use for the SIMPL application was sent to resident and faculty users. Respondents were asked to self-identify how often they used the app. The perceived benefits and barriers were correlated with the level of usage. Qualitative analysis of free text responses was used to determine strategies to increase usage.

SETTING

General surgery training programs who are members of the Procedural Learning and Safety Collaborative.

PARTICIPANTS

Surgical residents and faculty.

RESULTS

At least 1 SIMPL evaluation was created for 411 residents and 524 faculty. Thirty percent of both faculty and residents were high-frequency users. Thirty percent of faculty were never users. One hundred eighty-eight residents and 207 faculty (response rate 46%) completed the survey. High-frequency resident users were more likely to perceive a benefit for both numerical evaluations (76% vs 30%) and dictated feedback (92% vs 30%). Faculty and residents commonly blamed each other for not creating and completing evaluations regularly (87% of residents, 81% of faculty). Suggested strategies to increase usage included reminders and integration with existing data systems.

CONTRIBUTIONS

Frequent users perceive value from the application, particularly from dictated feedback and see a positive impact on feedback in their programs. Faculty engagement represents a major barrier to adoption. Mechanisms which automatically remind residents to initiate an evaluation will help improve utilization but programs must work to enhance faculty willingness to respond and dictate feedback.

Comparison of the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) to a Single-Item Performance Score

Construct: We compared a single-item performance score with the Ottawa Surgical Competency Operating Room Evaluation (O-SCORE) for their ability in assessing surgical competency.

BACKGROUND

Surgical programs are adopting competency-based frameworks. The adoption of these frameworks for assessment requires tools that produce accurate and valid assessments of knowledge and technical performance. An assessment tool that is quick to complete could improve feasibility, reduce delays, and result in a higher volume of assessments of learners. Previous work demonstrated that the 9-item O-SCORE can produce valid results; the goal of this study was to determine if a single-item performance rating (Is candidate competent to independently complete procedure: yes or no) completed at a separate viewing would correlate to the O-SCORE, thus increasing feasibility of procedural competence assessment.

APPROACH

Nineteen residents and 2 staff orthopedic surgeons from the University of Ottawa volunteered for a 2-part OSCE-style station including a written questionnaire and videotaped simulated open reduction and internal fixation midshaft radius fracture. Each performance was rated independently by 3 orthopedic surgeons using a single-item performance score (Time 1). The performances were assessed again 6 weeks later by the 3 raters using the O-SCORE (Time 2). Correlation between the single-item performance score and the O-SCORE were evaluated.

RESULTS

Three orthopedic surgeons completed 21 ratings each resulting in 63 orthopedic ratings. There was a high level of correlation and agreement between the single-item performance score at Time 1 and Time 2 (κ correlation =0.72-1.00; p < .001; percentage agreement =90%-100%). The reliability of the O-SCORE at Time 2 with three raters was 0.83 and the internal consistency was 0.89. There was a tendency for each rater to assign more yes responses to the more senior trainees.

CONCLUSIONS

A single-item performance score correlated highly with the O-SCORE in an orthopedic setting. A single-item score could be used to supplement a multi-item score with similar results in orthopedics. There is still benefit in completing multi-item scores such as the O-SCORE evaluations to guide specific areas of improvement and direct feedback.

Correlation of Objective Assessment Data With General Surgery Resident In-Training Evaluation Reports and Operative Volumes

OBJECTIVE

Faculty evaluations, ABSITE scores, and operative case volumes often tell little about true resident performance. We developed an objective structured clinical examination called the Surgical X-Games (5 rooms, 15 minutes each, 12-15 tests total, different for each postgraduate [PGY] level). We hypothesized that performance in X-Games will prove more useful in identifying areas of strength or weakness among general surgery (GS) residents than faculty evaluations, ABSITE scores, or operative cases volumes.

DESIGN

PGY 2 to 5 GS residents (n = 35) were tested in a semiannual X-Games assessment using multiple simulation tasks: laparoscopic skills, bowel anastomosis, CT/CXR analysis, chest tube placement, etc. over 1 academic year. Resident scores were compared to their ABSITE, in-training evaluation reports, and operating room case numbers.

SETTING

Academic medical center.

PARTICIPANTS

PGY-2, 3, 4, and 5 GS residents at Mayo Clinic in Rochester, MN.

RESULTS

Results varied greatly within each class except for staff evaluations: in-training evaluation reports medians for PGY-2s were 5.3 (range: 5.0-6.0), PGY-3s 5.9 (5.5-6.3), PGY-4s 5.6 (5.0-6.0), and PGY-5s were 6.1 (5.6-6.9). Although ABSITE and operating room case volumes fluctated greatly with each PGY class, only X-Games scores (median: PGY-2 = 82, PGY-3 = 61, PGY-4 = 76, and PGY-5 = 60) correlated positively (p < 0.05) with operative case volume and negatively (p < 0.05) with staff evaluations.

CONCLUSIONS

X-Games assessment generated wide differentiation of resident performance quickly, inexpensively, and objectively. Although "Minnesota-nice" surgical staff may feel all GS trainees are "above average," objective assessment tells us otherwise.

A SIMPL application to assess oral medicine residents' performance and level of autonomy

OBJECTIVES

The aim of this study was to evaluate a smartphone application, SIMPL, to optimize faculty guidance and oral medicine residents' performance, and to measure resident-faculty agreement for performance and supervision levels.

STUDY DESIGN

Raters used the 5-level "Performance" scale to assess resident's readiness for independent practice and the 4-level Zwisch scale to assess faculty guidance, between June 2016 and June 2017. Junior residents were trainees with less than 18 months of training, and senior residents had more than 18 months of training. Descriptive statistics were used to analyze supervision and autonomy. Performance and supervision agreement was estimated as percentage-agreement and measured using κ and 95% confidence intervals.

RESULTS

A total of 660 evaluations were performed by 6 residents and 6 faculty members. Senior residents received higher performance scores compared with junior residents (P for trend < .01). In terms of complexity of cases and level of supervision, there were no significant differences between juniors and seniors (P = .69 and P = .39, respectively). Residents were "practice ready" or greater than for 80% cases. Residents received meaningful autonomy for 64.8% cases. Faculty-resident concordance was 86.1% for performance (with κ of 0.77 [95% CI 66.1%-87.6%]) and 92.4% for supervision (with κ of 0.84 [95% CI 80%-88%]).

CONCLUSIONS

SIMPL can feasibly be used for real-time assessment of residents' performance and autonomy.

Assessment of Technical Skills Competence in the Operating Room: A Systematic and Scoping Review

PURPOSE

While academic accreditation bodies continue to promote competency-based medical education (CBME), the feasibility of conducting regular CBME assessments remains challenging. The purpose of this study was to identify evidence pertaining to the practical application of assessments that aim to measure technical competence for surgical trainees in a nonsimulated, operative setting.

METHOD

In August 2016, the authors systematically searched Medline, Embase, and the Cochrane Database of Systematic Reviews for English-language, peer-reviewed articles published in or after 1996. The title, abstract, and full text of identified articles were screened. Data regarding study characteristics, psychometric and measurement properties, implementation of assessment, competency definitions, and faculty training were extracted. The findings from the systematic review were supplemented by a scoping review to identify key strategies related to faculty uptake and implementation of CBME assessments.

RESULTS

A total of 32 studies were included. The majority of studies reported reasonable scores of interrater reliability and internal consistency. Seven articles identified minimum scores required to establish competence. Twenty-five articles mentioned faculty training. Many of the faculty training interventions focused on timely completion of assessments or scale calibration.

CONCLUSIONS

There are a number of diverse tools used to assess competence for intraoperative technical skills and a lack of consensus regarding the definition of technical competence within and across surgical specialties. Further work is required to identify when and how often trainees should be assessed and to identify strategies to train faculty to ensure timely and accurate assessment.

Assessment Considerations for Core Entrustable Professional Activities for Entering Residency

Context

In the process of analyzing entrustable professional activities (EPAs) for use in medical education, ten Cate and others identified challenges, including the need for valid and reliable EPA assessment strategies.

Objective

To provide osteopathic medical schools with a database of assessment tools compiled from the literature to assist them with the development and implementation of robust, evidence-based assessment methods.

Methods

MEDLINE, ERIC, PubMed, and other relevant databases were searched using MeSH keywords for articles outlining robust, evidence-based assessment tools that could be used in designing assessments for EPAs 1 through 6.

Results

A total of 55 publications were included in content analysis and reporting. All but 2 of the assessment articles were conducted in an undergraduate or graduate medical education setting. The majority of the 55 articles related to assessment of competencies affiliated with EPA 2 (16 articles) and EPA 4 (15 articles). Four articles focused on EPA 3.

Conclusion

Osteopathic medical schools can use this database of assessment tools to support the development of EPA-specific assessment plans that match the unique context and needs of their institution.

Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment

OBJECTIVE

We have previously demonstrated the feasibility and validity of a smartphone-based system called Procedural Autonomy and Supervision System (PASS), which uses the Zwisch autonomy scale to facilitate assessment of the operative performances of surgical residents and promote progressive autonomy. To determine whether the use of PASS in a general surgery residency program is associated with any negative consequences, we tested the null hypothesis that PASS implementation at our institution would not negatively affect resident or faculty satisfaction in the operating room (OR) nor increase mean OR times for cases performed together by residents and faculty.

METHODS

Mean OR times were obtained from the electronic medical record at Northwestern Memorial Hospital for the 20 procedures most commonly performed by faculty members with residents before and after PASS implementation. OR times were compared via two-sample t-test. The OR Educational Environment Measure tool was used to assess OR satisfaction with all clinically active general surgery residents (n = 31) and full-time general surgery faculty members (n = 27) before and after PASS implementation. Results were compared using the Mann-Whitney rank sum test.

RESULTS

A significant prolongation in mean OR time between control and study period was found for only 1 of the 20 operative procedures performed at least 20 times by participating faculty members with residents. Based on the overall survey score, no significant differences were found between resident and faculty responses to the OR Educational Environment Measure survey before and after PASS implementation. When individual survey items were compared, while no differences were found with resident responses, differences were noted with faculty responses for 7 of the 35 items addressed although after Bonferroni correction none of these differences remained significant.

CONCLUSIONS

Our data suggest that PASS does not increase mean OR times for the most commonly performed procedures. Resident OR satisfaction did not significantly change during PASS implementation, whereas some changes in faculty satisfaction were noted suggesting that PASS implementation may have had some negative effect with them. Although the effect on faculty satisfaction clearly requires further investigation, our findings support that use of an autonomy-based OR performance assessment system such as PASS does not appear to have a major negative influence on OR times nor OR satisfaction.

Relationship of procedural numbers with meaningful procedural autonomy in general surgery residents

BACKGROUND

Concerns exist regarding the competency of general surgery graduates with performing core general surgery procedures. Current competence assessment incorporates minimal procedural numbers requirements.

METHODS

Based on the Zwisch scale we evaluated the level of autonomy achieved by categorical PGY1-5 general surgery residents at 14 U.S. general surgery resident training programs between September 1, 2015 and December 31, 2016. With 5 of the most commonly performed core general surgery procedures, we correlated the level of autonomy achieved by each resident with the number of procedures they had performed before the evaluation period, with the intent of identifying specific target numbers that would correlate with the achievement of meaningful autonomy for each procedure with most residents.

RESULTS

Whereas a definitive target number was identified for laparoscopic appendectomy (i.e. 25), for the other 4 procedures studied (i.e. laparoscopic cholecystectomy, 52; open inguinal hernia repair, 42; ventral hernia repair, 35; and partial colectomy, 60), target numbers identified were less definitive and/or were higher than many residents will experience during their surgical residency training.

CONCLUSIONS

We conclude that procedural target numbers are generally not effective in predicting procedural competence and should not be used as the basis for determining residents' readiness for independent practice.

Novel Mobile App Allows for Fast and Validated Intraoperative Assessment of Otolaryngology Residents

Evaluation of resident operative skills is challenging in the fast-paced operating room environment and limited by lack of validated assessment metrics. We describe a smartphone-based app that enables rapid assessment of operative skills. Accreditation Council for Graduate Medical Education (ACGME) otolaryngology taxonomy surgical procedures (n = 593) were uploaded to the software platform. The app was piloted over 1 month. Outcomes included (1) completion of evaluation, (2) time spent completing the evaluation, and (3) quantification of case complexity, operative autonomy, and performance. During the study, 12 of 12 procedures, corresponding to 3 paired evaluated by the resident/attending dyad. Mean ± SD time of evaluation completion was 98.0 ± 24.2 and 123.0 ± 14.0 seconds for the resident and attending, respectively. Mean time between resident and attending evaluation completion was 27.9 ± 26.8 seconds. Resident and attending scores for case complexity, operative autonomy, and performance were strongly correlated (P < .0001). Rapid evaluation of resident intraoperative performance is feasible using smartphone-based technology.

A taxonomy of perioperative surgical learning: Trending resident skill acquisition

BACKGROUND

Resident and curriculum evaluation require tracking surgical resident operative performance, yet what and when to measure remains unclear.

METHODS

From a multi-institutional database, we reviewed 611 resident/surgeon-paired assessments of ACGME Milestones and modified OPRS ratings for different cases and postgraduate years.

RESULTS

Faculty Milestone ratings increased with each PGY (p=<0.001) and correlated with resident self-ratings (ICC = 0.83). Mean OPRS scores increased in small increments with substantial intra-year variability. Progression among individual OPRS subcategories was not apparent from more global analyses. Interestingly, male faculty offered lower ratings than female faculty.

CONCLUSIONS

Milestones and modified mean OPRS ratings suggest residents are learning, yet lack sufficient discrimination for promotion or curricular analysis. Differential progression through OPRS subcategories suggests a taxonomy of surgical learning that can be tailored to focus on different skills at each point in the training continuum. The effect of faculty gender on resident ratings awaits further study.

Twelve tips for assessing surgical performance and use of technical assessment scales

Using validated assessment scales for technical competence can help structure and standardize assessment and feedback for both the trainee and the supervisor and thereby avoid bias and drive learning. Correct assessment of operative skills can establish learning curves and allow adequate monitoring. However, the assessment of surgical performance is not an easy task, since it includes many proxy parameters, which are hard to measure. Although numerous technical assessment scales exist, both within laparoscopic and open surgery, the validity evidence is often sparse, and this can raise doubts about reliability and educational outcome. Furthermore, the implementation of technical assessment scales varies due to several obstacles and doubts about accurate use. In this 12-tips article, we aim to give the readers a critical and useful appraisal of some of the common questions and misunderstandings regarding the use of surgical assessment scales and provide tips to ease and overcome potential pitfalls.