Resident self-assessment of operative performance

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis

BACKGROUND

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments.

OBJECTIVES

We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills.

METHODS

The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study.

RESULTS

We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains.

CONCLUSIONS

Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.

Validation of a simplified evaluation grid applied to the microsurgery on inert material via smartphone

OBJECTIVES

Microsurgery is usually performed in experimental research models and clinical surgery. It requires meticulous technical skills and continuous training. Inert materials such as Shirataki noodles are readily available and low-cost consumables regularly used for practice. The objective of this study was to evaluate the repeatability and reproducibility of a simplified evaluation grid of suture on inert material (Shirataki Konnyaku noodle) under smartphone magnification.

MATERIAL AND METHODS

Ten students performed end-to-end suture on inert material with magnification via their smartphone. Each suture was filmed, and the videos were randomized. Each student was evaluated on each video three times over three consecutive days, using a simplified evaluation grid. Intra- and inter-observer agreement was evaluated on Concordance Correlation Coefficients. Values were assessed on Pearson's correlation coefficient.

RESULTS

Intra-observer correlation was weak for 2 items (0.288 and 0.246) and moderate for the other 2 (0.419 and 0.529). Inter-observer correlation was weak for 3 items (0.344, 0.358, and 0.276) and close to zero for the other (0.034).

CONCLUSION

This simplified evaluation grid for microsurgery training on inert material via smartphone was poorly repeatable and reproducible. Loss of certain items in the grid due to the use of inert material probably impaired relevance.

Are Plastic Surgery Trainees Accurate Assessors of Their Own Microsurgical Skill?

Background

Microsurgery is a technically demanding surgical discipline with a steep learning curve. Trainees have faced several difficulties due to less hands-on theater time and pandemic-related limits on access to technical training. To overcome this, trainees engaged in self-directed training, which requires an accurate self-assessment of skill. This study aimed to assess the ability of trainees to accurately self-assess their performance while performing a simulated microvascular anastomosis.

Methods

Novice and specialist plastic surgery trainees performed a simulated microvascular anastomosis on a high-fidelity chicken femoral vessel model. Each participant objectively rated the quality of their anastomosis using the Anastomosis Lapse Index (ALI). Two expert microsurgeons subsequently blindly rated each anastomosis. To determine the accuracy of self-evaluation, self-scores and expert-scores were compared using a Wilcoxon signed-rank test.

Results

Twenty-seven surgical trainees completed the simulation, with a mean time to completion (TTC) of 40.3 minutes (range 14.2-106.0 minutes). For the entire cohort, the median ALI self-score was 4 (range 3-10), while the median ALI expert-score was 5.5 (range 2.5-9.5). There was a significant difference between the ALI self-score and the expert-score (p<0.001). When grouped by experience level, there was no significant difference between self-score and expert-score within the specialist group, while there was a significant difference within the novice group (p=0.001).

Conclusion

These findings suggest that specialist trainees are accurate assessors of their own microsurgical skills, but novice trainees tend to overestimate their technical skills. While novice trainees can engage in independent self-directed microsurgical training, they should seek expert feedback to ensure targeted training.

Role of video self-assessment in laparoscopic simulation training: a randomized pilot trial

BACKGROUND

Residency programs have implemented simulation training to compensate for reduced operating room exposure. Video recording is an educational tool that can be utilized for coaching, telepresence, and self-assessment during simulation training. Data is limited on the utility of video recording and self-assessment for laparoscopic training in Ob/Gyn residency programs..

OBJECTIVE

This study aimed to determine the role of video self-assessment as an educational tool in laparoscopic simulation training and to establish the feasibility of our study design for a larger randomized controlled trial.

STUDY DESIGN

This was a prospective pilot study with a parallel, randomized, trial design that occurred in the Department of Obstetrics and Gynecology at the Mount Sinai Hospital. Subject participation took place in a surgical simulation training room. A total of 23 subjects were recruited (7 medical students, 15 residents, 1 fellow) voluntarily. All participants completed the study. All the subjects completed a pretest survey. The surgical simulation room contained a single Fundamentals of Laparoscopic Surgery box trainer and video-recording station. For session #1, each participant performed 2 Fundamentals of Laparoscopic Surgery tasks (A, peg transfer; B, intracorporeal knot tie). Participants were video recorded during session #1 and were randomized to either receive or not receive their video recording. The video group (n=13) and control group (n=10) repeated the Fundamentals of Laparoscopic Surgery tasks 7 to 10 days later (session #2). The primary outcome was percentage change in completion time between sessions. Secondary outcomes were percentage change in peg and needle drops between sessions.

RESULTS

The participant characteristics (video vs control) were as follows: average training level (6.15 vs 4.90 years), self-assessment (1=poor, 10=excellent) of surgical skill (4.8 vs 3.7), and laparoscopic skill (4.4 vs 3.5). Training level was inversely correlated with completion time for tasks A and B (r, -0.79 and -0.87; P<.0001). Less experienced trainees required the maximum time allotted for each task in session #1 (A, 3; B, 13). Regarding the primary outcome, the video group improved less than the control group (A, 16.7% vs 28.3%; B, 14.4% vs 17.3%). After controlling for training level (residents only), the video group improved more in the primary outcome (A, 17% vs 7.4%; B, 20.9% vs 16.5%) and secondary outcomes (A, 0.0% vs -194.1%; B, 41.3% vs 37.6%).

CONCLUSION

Video self-assessment has a potential role in simulation training for obstetrics-gynecology residents. With key improvements, the feasibility of our study design was demonstrated in preparation for a future definitive trial.

Video-based interventions to improve self-assessment accuracy among physicians: A systematic review

PURPOSE

Self-assessment of a physician's performance in both procedure and non-procedural activities can be used to identify their deficiencies to allow for appropriate corrective measures. Physicians are inaccurate in their self-assessments, which may compromise opportunities for self- development. To improve this accuracy, video-based interventions of physicians watching their own performance, an experts' performance or both, have been proposed to inform their self-assessment. We conducted a systematic review of the effectiveness of video-based interventions targeting improved self-assessment accuracy among physicians.

MATERIALS AND METHODS

The authors performed a systematic search of MEDLINE, Embase, EBM reviews, and Scopus databases from inception to August 23, 2022, using combinations of terms for "self-assessment", "video-recording", and "physician". Eligible studies were empirical investigations assessing the effect of video-based interventions on physicians' self-assessment accuracy with a comparison of self-assessment accuracy pre- and post- video intervention. We defined self-assessment accuracy as a "direct comparison between an external evaluator and self-assessment that was quantified using formal statistical analysis". Two reviewers independently screened records, extracted data, assessed risk of bias, and evaluated quality of evidence. A narrative synthesis was conducted, as variable outcomes precluded a meta-analysis.

RESULTS

A total of 2,376 papers were initially retrieved. Of these, 22 papers were selected for full-text review; a final 9 studies met inclusion criteria for data extraction. Across studies, 240 participants from 5 specialties were represented. Video-based interventions included self-video review (8/9), benchmark video review (3/9), and/or a combination of both types (1/9). Five out of nine studies reported that participants had inaccurate self-assessment at baseline. After the intervention, 5 of 9 studies found a statistically significant improvement in self-assessment accuracy.

CONCLUSIONS

Overall, current data suggests video-based interventions can improve self-assessment accuracy. Benchmark video review may enable physicians to improve self-assessment accuracy, especially for those with limited experience performing a particular clinical skill. In contrast, self-video review may be able to provide improvement in self-assessment accuracy for more experience physicians. Future research should use standardized methods of comparison for self-assessment accuracy, such as the Bland-Altman analysis, to facilitate meta-analytic summation.

A Comparison of Checklist and Domain-Based Ratings in the Assessment of Objective Structured Clinical Examination (OSCE) Performance

INTRODUCTION

The Objective Structured Clinical Examination (OSCE) is the mainstay of clinical assessment in the final-year undergraduate Family Medicine clerkship at King Faisal Specialist Hospital and Research Centre (KFSHRC). The gold standard for OSCE assessment is the checklist rating, completed by physician examiners. Numerous studies have suggested that global or domain-based OSCE ratings may be a better indicator of competence than checklist ratings. The aim of this study was to examine the utility of domain-based OSCE ratings in the context of final-year, undergraduate, Family Medicine OSCE examinations in Riyadh, Saudi Arabia. This is akin to an exercise in quality improvement, as we continuously look for ways to improve our OSCE assessment processes.

METHODS

This study utilised a quantitative methodology. Three final year OSCE exams were chosen. Physicians rated each student using a checklist score and using a more holistic domain-based score. Physician checklist scores and physician domain-based scores were then compared, and correlation was assessed. We also looked at the internal consistency of the scoring methods.

RESULTS

A significant correlation was found between checklist and domain-based scores by physicians for all exams (r=0.858, p<0.01), with a good internal consistency for these methodologies for all exams.

CONCLUSION

The results demonstrate that both checklist and domain-based scores offer some benefit to the assessment, with a similar internal consistency and strong correlation. Domain-based ratings should be utilised for softer skills that are not easily assessed by checklists. There is clearly a need to rethink our OSCE assessment. The assessment should combine checklist and domain-based physician scores. As trainees become more experienced, checklist OSCE may penalise directness and efficiency, while domain-based ratings would offer a better appraisal of competence, and have been shown to be more sensitive to the level of training and expertise. Changing the assessment methods will lead to necessary changes in the student approach to the OSCE and improve authenticity and validity.

Video Coaching in Surgical Education: Utility, Opportunities, and Barriers to Implementation

OBJECTIVE

This review discusses the literature on Video-Based Coaching (VBC) and explores the barriers to widespread implementation.

DESIGN

A search was performed on Scopus and PubMed for the terms "operation," "operating room," "surgery," "resident," "house staff," "graduate medical education," "teaching," "coaching," "assessment," "reflection," "camera," and "video" on July 27, 2021, in English. This yielded 828 results. A single author reviewed the titles and abstracts and eliminated any results that did not pertain to operative VBC or assessment. All bibliographies were reviewed, and appropriate manuscripts were included in this study. This resulted in a total of 52 manuscripts included in this review.

SETTING/PARTICIPANTS

Original, peer-reviewed studies focused on VBC or assessment.

RESULTS

VBC has been both subjectively and objectively found to be a valuable educational tool. Nearly every study of video recording in the operating room found that subjects, including surgical residents and seasoned surgeons alike, overwhelmingly considered it a useful, non-redundant adjunct to their training. Most studies that evaluated skill acquisition via standardized assessment tools found that surgical residents who underwent a VBC program had significant improvements compared to their counterparts who did not undergo video review. Despite this evidence of effectiveness, fewer than 5% of residency programs employ video recording in the operating room. Barriers to implementation include significant time commitments for proposed coaching curricula and difficulty with integration of video cameras into the operating room.

CONCLUSIONS

VBC has significant educational benefits, but a scalable curriculum has not been developed. An optimal solution would ensure technical ease and expediency, simple, high-quality cameras, immediate review, and overcoming entrenched surgical norms and culture.

Comparing control room operators' and experts' assessment of team performance using structured task-specific observation protocols and scenario replay

Operators' self-assessment has received limited interest within process control or human-system evaluation. Research on self-assessment has been criticised for poor assessment methodology, and consequently, its status is unclear. This study hypothesised that, given adequate assessment methods (such as task-specific assessment items and scenario replay), we could observe relatively accurate self-assessment results. Eighteen licensed operators and two experts assessed team performance in six nuclear control room scenarios. The results reveal an overall agreement between operators and experts, measured by the intraclass correlation coefficient, ranging from 0.60 to 0.70, which lies close to the intraclass correlation coefficient of 0.75 for the experts. This demonstrates potential for achievement of relatively accurate operator self-assessment for complex work. The agreement varied in a similar manner for both expert agreement and operator-expert agreement across eight performance dimensions. In addition, the operators' self-assessment provided additional information beyond observer assessment in identifying non-acceptable performance items.

Performance of Residents During Laparoscopic Cholecystectomy: Is Self-assessment Reliable?

BACKGROUND

To compare the self-assessment of surgical residents and observers (faculty members and nurses) during laparoscopic cholecystectomy (LC).

MATERIALS AND METHODS

A prospective observational study was conducted between February 2020 and July 2020 at a medical school hospital. Seventy-four LC surgeries were performed by surgical residents in the presence of faculty members. A self-assessment of the technical and nontechnical performance of the residents was requested. The self-assessment of residents was compared with observer evaluations using the Kruskal-Wallis test. Gwet AC2 fit coefficient was used to determine the consistency between the observers' and residents' assessments. Bland-Altman plots were generated with 95% limits of agreement to describe the agreement between the total scores of the observers.

RESULTS

The self-assessment of residents had a statistically significant higher score when compared with observers (faculty and nurses) (P<0.001). However, no significant difference was observed between the total scores given by the observers (faculty members and nurses) (P>0.05). There was a moderate agreement between the resident versus faculty members [0.503; 95% confidence interval (CI), 0.430-0.576] and resident versus nurse (0.518; 95% CI, 0.432-0.605) when evaluating technical skills. However, there was substantial agreement between faculty members and nurses (0.736; 95% CI, 0.684-0.789). Postoperative pain was significantly correlated with resident self-assessment (P=0.022).

CONCLUSION

The self-assessment scores of surgical residents in LC operations were overestimated compared with observer assessments.

Performance Improvement With Implementation of a Surgical Skills Curriculum

OBJECTIVE

To assess the efficacy of an intern surgical skills curriculum involving a boot camp for core open and laparoscopic skills, self-guided practice with positive and negative incentives, and semiannual performance evaluations.

DESIGN

Longitudinal cohort study.

SETTING

Academic tertiary care center.

PARTICIPANTS

Intervention group (n = 15): residents who completed the intern surgical skills curriculum and had performance evaluations in fall of intern year, spring of intern year, and fall of second year. Control group (n = 8): second-year residents who were 1 year ahead of the intervention group in the same residency program, did not participate in the curriculum, and had performance evaluations in fall of second year.

RESULTS

In fall of second year of residency, the intervention group had better performance (presented as median values with interquartile ranges) than the control group on one-hand ties (left hand: 9.1 [6.3-10.1] vs 14.6 [13.5-15.4] seconds, p = 0.007; right hand: 8.7 [8.5-9.6] vs 11.5 [9.9-16.8] seconds, p = 0.039). The intervention group also had better performance on all open suturing skills, including mattress suturing (vertical: 33.4 [30.0-40.0] vs 55.8 [50.0-67.6] seconds, p = 0.001; horizontal: 28.7 [27.3-39.9] vs 52.7 [40.7-57.8] seconds, p = 0.003), and a water-filled glove clamp, divide, and ligate task (28.0 [25.0-31.0] vs 59.1 [53.0-93.0] seconds, p < 0.001). Finally, the intervention group had better performance on all laparoscopic skills, including peg transfer (66.0 [59.0-82.0] vs 95.2 [87.5-101.5] seconds, p = 0.018), circle cut (82.0 [69.0-124.0] seconds vs 191.8 [155.5-231.5] seconds, p = 0.002), and intracorporeal suturing (195.0 [117.0-200.0] seconds vs 359.5 [269.0-450.0] seconds, p = 0.002).

CONCLUSIONS

Implementation of a comprehensive surgical skills curriculum was associated with improved performance on core open and laparoscopic skills. Further research is needed to understand and optimize motivational factors for deliberate practice and surgical skill acquisition.

The reliability of resident self-evaluation of operative performance

BACKGROUND

Self-assessment is critical to professional self-regulation yet many trainees may not reliably self-evaluate. We examine the gap between resident and faculty perceptions of trainee operative performance and contributing factors.

METHODS

Surgery resident and faculty evaluations of trainee performance were collected from 14 academic institutions using smartphone-based performance assessments. Differences in resident/faculty ratings evaluating the same procedure were analyzed using descriptive statistics and Bayesian mixed models.

RESULTS

Of 7382 evaluations, 46% trainees and faculty performance ratings were discrepant (r = 0.47), with 80% residents rating themselves lower than faculty in those cases. This gap existed regardless of case complexity and widened as trainees gained experience. Trainees who overrated themselves had the lowest mean performance scores from faculty.

CONCLUSION

Half of residents perceived their performance differently from faculty, and this difference widened for senior residents. Future focus should be to provide opportunity for trainees to improve skills to reliably assess themselves before graduation.

Surgical coaching to achieve the ABMS vision for the future of continuing board certification

In February 2019, the American Board of Medical Specialties (ABMS) released the final report of the Continuing Board Certification: Vision for the Future initiative, issuing strong recommendations to replace ineffective, traditional mechanisms for physicians' maintenance of certification with meaningful strategies that strengthen professional self-regulation and simultaneously engender public trust. The Vision report charges ABMS Member Boards, including the American Board of Surgery (ABS), to develop and implement a more formative, less summative approach to continuing certification. To realize the ABMS's Vision in surgery, new programs must support the assessment of surgeons' performance in practice, identification of individualized performance gaps, tailored goals to address those gaps, and execution of personalized action plans with accountability and longitudinal support. Peer surgical coaching, especially when paired with video-based assessment, provides a structured approach that can meet this need. Surgical coaching was one of the approaches to continuing professional development that was discussed at an ABS-sponsored retreat in January 2020; this commentary review provides an overview of that discussion. The professional surgical societies, in partnership with the ABS, are uniquely positioned to implement surgical coaching programs to support the continuing certification of their membership. In this article, we provide historical context for board certification in surgery, interpret how the ABMS's Vision applies to surgical performance, and highlight recent developments in video-based assessment and peer surgical coaching. We propose surgical coaching as a foundational strategy for accomplishing the ABMS's Vision for continuing board certification in surgery.

Self-Assessment of Surgical Skills: A Systematic Review

OBJECTIVE

Self-assessment is fundamental in surgical training to enhance learning in the absence of trainer feedback. The primary objective of this review was to assess the factors that influence accuracy of self-assessment at technical skills across all surgical specialties. The secondary objective was to assess whether there are any innate factors or attributes to predict those that will carry out effective self-assessment.

DESIGN

A systematic review was carried out in accordance with PRISMA guidelines. A search strategy encompassing MEDLINE, EMBASE, ERIC, WHO, and the Cochrane database was conducted to identify studies investigating self-assessment at any surgical task. Quality was assessed using the Newcastle-Ottawa scale. A summary table was created to describe specialty, participants, task, setting, assessment tool, and correlation coefficient between self and expert assessment. The review protocol was registered in PROSPERO.

RESULTS

Of 24,638 citations, 40 met inclusion criteria. In total 1753 participants performed 68 procedures. Twenty-six studies investigated skills in general surgery with the remaining 14 in various other surgical specialties. Accuracy of self-assessment is superior in those with greater experience and age, and with use of retrospective video playback. Accuracy tends to be reflected by overestimation of performance. Stressful environments reduce accuracy. There is limited evidence in the literature regarding predicting traits for those who will carry out accurate self-assessment.

CONCLUSIONS

The ability to perform accurate self-assessment is an important skill in surgical training, with accuracy being influenced by a multitude of factors. The use of self-assessment from retrospective video playback may be of benefit in surgical training curricula to enhance learning of technical skills. Further studies are required to define predictors of good self-assessment, which will strengthen recruitment and mentoring to assist trainee learning.

Towards Development of an Open Surgery Competency Assessment for Residents (OSCAR) Tool - A Systematic Review of the Literature and Delphi Consensus

OBJECTIVE

Competency-based education has mandated accurate intra-operative assessment tools. We aimed to define consensus-based open surgical skills perceived by experts as critical for assessment.

DESIGN

A mixed-method design was employed: systematic review and e-Delphi methodology.

SETTING

The study was performed at McGill University-affiliated large tertiary academic centers in Montreal, Quebec, Canada.

PARTICIPANTS

Per PRISMA guidelines, a peer-reviewed search strategy was employed. Studies published in English and those describing technical skill assessment of open abdominal surgery were included; subspecialty-specific skills, conference abstracts, academic memoirs were excluded. Most-cited skills were subjected to e-Delphi methodology to identify those deemed essential by experts, based a 3-point Likert scale. Eighteen McGill University-affiliated general surgeons, representing a variety of subspecialties of General Surgery, were invited to answer the questionnaire.

RESULTS

Around 120 of 4285 references were retained for analysis. The 12 most cited skills included suturing, tissue and instrument handling, movement economy, instrument knowledge, knot tying, flow, knowledge of procedure, completion time, dissection technique, knowledge of anatomy and sterile technique; 6 of these achieved high or perfect scores and agreement after 2 rounds of survey: suturing, sterile technique, knot tying, knowledge of anatomy, knowledge of procedure, and tissue handling. Median standard deviation decreased (0.495 to 0.450) from first to second round, indicating improvement in consensus.

CONCLUSION

These results will help develop and validate the OSCAR (objective structured clinical assessment rubric) assessment tool for immediate intra-operative feedback of open technical skills for surgical trainees.

Video Modeling and Video Feedback to Reduce Time to Perform Intravenous Cannulation in Medical Students: A Randomized-Controlled Mixed-Methods Study

PURPOSE

Combined video modeling (VM) and video feedback (VF) may be more beneficial than traditional feedback when teaching procedural skills. This study examined whether repeated VM and VF compared with VM alone reduced the time required for medical students to perform peripheral intravenous (IV) cannulation.

METHODS

Twenty-five novice medical students were randomly assigned to groups in a one-way blinded embedded mixed-methods study to perform IV cannulation. Participants received standardized instruction and performed IV cannulation on each other while being audio-video recorded. They were assigned to review a video of an expert performing IV cannulation (VM alone), or both the expert video and a video of their own most recent IV cannulation (VM+VF), before returning to perform another IV cannulation. This was repeated for a total of four IV cannulation encounters and three video reviews. A post-test interview was also conducted and analyzed qualitatively using thematic content analysis.

RESULTS

The median [interquartile range] time required to perform IV cannulation in the final encounter was significantly different between the VM+VF group vs VM alone group (126 [93-226] sec vs 345 [131-537] sec, respectively; median difference, 111 sec; 95% confidence interval, 8 to 391; P = 0.02). There was no significant difference in IV cannulation success between VM alone and VM+VF in the final encounter (75% vs 85% respectively; P = 0.65). For the VM+VF group, the time to perform IV cannulation was reduced after the final encounter compared with the baseline encounter (P = 0.002), which was not true of the VM alone group (P = 0.35).

CONCLUSION

Video modeling and feedback shortened time to IV skill completion, reduced complications, and improved satisfaction in novice medical students.

Getting better all the time? Facilitating accurate team self-assessments through simulation

Introduction

Our study explores the extent to which teams are accurate assessors of their own performance and teamwork, and how simulation can help this critical skill develop over time.

Methods

Surgery residents in teams of three completed five daily simulations. After each scenario, each team reviewed their performance and jointly completed a scenario-specific team performance evaluation and a 17-item Communication and Teamwork Skills tool. Videos were rated to obtain discrepancy values. Paired-samples t-tests and mean comparisons were used to examine changes in team self-assessment accuracy and comparisons between high-performing and low-performing teams.

Results

Resident (n=30) teams rated team performance higher than faculty across the first 3 days (p<0.01), but provided similar ratings thereafter. Agreement of team performance from day 1 to 5 significantly improved (p<0.001). Teams rated their teamwork higher than faculty across all days (p<0.01). Top performing teams provided more accurate self-assessments for both teamwork (average discrepancy 8% vs 39%) and team performance (average discrepancy 12% vs 23%).

Conclusion

Teams that continue to work together over time may become more accurate judges of their own performance, but do not become more accurate assessors of teamwork competencies.

Pain medicine fellow neuromodulation surgical skill assessment tool: a pilot

OBJECTIVE

Spinal cord stimulation (SCS) is an evidence-based therapy for the treatment of refractory pain. Current American College of Graduate Medical Education requirements for pain medicine fellowship graduation include observation of five neuromodulation cases. Matriculating fellows have varying degrees of technical skill, training, and experience. The purpose of this study was to use an innovative skill-based assessment tool using the validated Zwisch scale to evaluate fellow surgical performance during SCS cases.

DESIGN

Cross-sectional survey SETTING: Mayo Clinic, Rochester, Minnesota.

SUBJECTS

Consultant faculty (10) in the Division of Pain Medicine and pain medicine fellows (5).

METHODS

A list of faculty-driven neuromodulation surgical objectives was formed and a rubric was created that focused on technical approach, imaging, epidural access and wound closure. Consultants who perform surgical cases were instructed to use the Zwisch scale as a guide to evaluate fellow surgical performance during a neuromodulation case. Faculty and fellows were surveyed before and after implementation of the tool to determine their satisfaction with use of the skill assessment tool.

RESULTS

The consultants reported they were more satisfied, consistent, and efficient with feedback provided to the trainee on every aspect of neuromodulation surgical cases. The improvement was most significant during the fellows' intraoperative skill assessment. The fellows indicated increased satisfaction, improved communication, and increased efficiency of feedback when the tool was utilized. The fellows reported greater intraoperative skill assessment and consistency of feedback provided.

CONCLUSIONS

The diverse nature of primary specialty backgrounds observed in pain medicine fellowship training offers a unique opportunity to assess and improve fellow skill and surgical competence in the field of neuromodulation. Both faculty and fellows reported improved satisfaction, consistency and efficiency with feedback provided. Importantly, this pilot project observed that implementation of a skill assessment tool was beneficial for both the faculty and fellow as the feedback received was viewed as strongly beneficial to the educational experience.

Self-assessment of Skills by Surgeons and Anesthesiologists After a Trauma Surgery Masterclass

BACKGROUND

In the Netherlands, each year a three-day international multidisciplinary trauma masterclass is organized to provide the knowledge and skills needed to care for critically injured trauma patients. This study was designed to longitudinally evaluate the effect of the course on participant's self-assessment of their own ability and confidence to perform general and specific skills.

METHODS

Between 2013 and 2016, all participants were invited to complete a questionnaire before and during follow-up. Participants were asked to self-assess their level of confidence to perform general skills (communication, teamwork, leadership) and specific skills. Mean scores were calculated, and mixed models were used to evaluate correlation.

RESULTS

We asked 265 participants to participate. Response rate was 64% for the pre-questionnaire, 63% for the post-questionnaire and for 3 months, 1 year and 2 years, respectively, 40%, 30%, 20%. The surgical group showed a statistically significant increase in self-assessed confidence for general skills (3.82-4.20) and specific technical skills (3.01-3.83; p < 0.001). In the anesthetic group, self-assessed confidence increased significantly in general skills (3.72-4.26) and specific technical skills (3.33-4.08; p < 0.001). For both groups statistical significance remained during follow-up.

CONCLUSIONS

This study demonstrated a sustained positive effect of a dedicated multidisciplinary trauma training curriculum on participant's self-assessed confidence to perform both general and specific technical skills necessary for the care of injured patients. Given the known association between confidence and competence, these findings provide evidence that dedicated trauma training curricula can provide positive lasting results.

LEVEL OF EVIDENCE

This is a basic science paper and therefore does not require a level of evidence.

Experience with key indicator cases among otolaryngology residents

Objective

To describe the resident experience with respect to key indicator cases for each year of training.

Study Design

Multi‐institution, cross‐sectional assessment.

Methods

Using an electronic survey, current otolaryngology residents were solicited to complete a survey regarding their experiences with the key indicator cases to that point. The survey was sent to this cohort in the winter of 2017–2018.

Results

Three hundred and three residents responded, with 293 completing the survey. Twenty‐three percent were PGY1, 19% PGY2, 21% PGY3, 18% PGY4, and 19% PGY5 or higher. The majority of residents progress from resident assistant as a PGY2, to resident surgeon as a PGY3 and self‐assessed competent surgeon as a PGY4 for the majority of the key indicator cases. Less than 50% of the surveyed PGY5 residents had reached independent practice in all the key indicator cases, with stapedectomy (16%), rhinoplasty (18%), and paramedian forehead flap (14.5%) being the cases least frequently performed independently. Ninety‐five percent of the respondent residents felt their program provided adequate training, but 20% of the respondents were either unsure or believed that they would be unable to perform all the key indicator cases by the completion of their training.

Conclusions

The majority of otolaryngology residents feel confident in their training, but experience with certain cases lags behind and may not currently be taught as resident level cases. These findings raise the question of whether the current key indicator cases are the best option for assessing breadth and depth of residency training.

Level of Evidence

NA

Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial

Background and study aims  Novice endoscopists are inaccurate in self-assessment of procedures. One means of improving self-assessment accuracy is through video-based feedback. We aimed to determine the comparative effectiveness of three video-based interventions on novice endoscopists' self-assessment accuracy of endoscopic competence. Materials and methods  Novice endoscopists (performed < 20 previous procedures) were recruited. Participants completed a simulated esophagogastroduodenoscopy (EGD) on a virtual reality simulator. They were then randomized to one of three groups: self-video review (SVR), which involved watching a recorded video of their own performance; benchmark review (BVR), which involved watching a video of a simulated EGD completed by an expert; and self- and benchmark video (SBVR), which involved both videos. Participants then completed two additional simulated EGD cases. Self-assessments were conducted immediately after the first procedure, after the video intervention and after the additional two procedures. External assessments were conducted by two experienced endoscopists, who were blinded to participant identity and group assignment through video recordings. External and self-assessments were completed using the global rating scale component of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT GRS). Results  Fifty-one participants completed the study. The BVR group had significantly improved self-assessment accuracy in the short-term, compared to the SBVR group ( P  = .005). The SBVR group demonstrated significantly improved self-assessment accuracy over time ( P  = .016). There were no significant effects of group or of time for the SVR group. Conclusions  Video-based interventions, particularly combined use of self- and benchmark video review, can improve accuracy of self-assessment of endoscopic competence among novices.

Use of a Novel Measure of Nontechnical Skills in Surgical Trainees: Is There an Association With Technical Skills Performance?

OBJECTIVE

To investigate whether scores on a psychological measure of concentration and interpersonal characteristics, The Attentional and Interpersonal Style Inventory (TAIS), are associated with performance of surgical skills.

DESIGN

Postgraduate surgical trainees completed an operative surgical skills assessment in the simulation laboratory and the psychological measure (TAIS). The surgical skills assessment consisted of 6 tasks (3 per trainee): laceration suturing; lipoma excision; incision and closure of a laparotomy wound; bowel anastomosis; saphenofemoral junction ligation and basic laparoscopic skills. The association between operative surgical skill performance and TAIS factors was investigated.

SETTING

The TAIS assessments and surgical skills assessments were conducted at the National Surgical Training Centre at the Royal College of Surgeons in Ireland (RCSI).

PARTICIPANTS

One hundred and two surgical trainees in years one and two (PGY 2-3 equivalent) participated in the study.

RESULTS

Performance on 2 of the 6 tasks assessed (bowel anastomosis and lipoma excision) were positively associated with multiple TAIS factors (energy, confidence, competitiveness, extroversion, self-criticism and performing under pressure). Another factor, focus over time, was significantly associated with scores on the lipoma excision task.

CONCLUSIONS

Trainees with high levels of energy, confidence, competitiveness, extroversion, and focus over time and low levels of self-criticism demonstrated better performance on specific technical skills tasks.

Variability and dimensionality of students' and supervisors' mini-CEX scores in undergraduate medical clerkships - a multilevel factor analysis

BACKGROUND

The mini clinical evaluation exercise (mini-CEX)-a tool used to assess student-patient encounters-is increasingly being applied as a learning device to foster clinical competencies. Although the importance of eliciting self-assessment for learning is widely acknowledged, little is known about the validity of self-assessed mini-CEX scores. The aims of this study were (1) to explore the variability of medical students' self-assessed mini-CEX scores, and to compare them with the scores obtained from their clinical supervisors, and (2) to ascertain whether learners' self-assessed mini-CEX scores represent a global dimension of clinical competence or discrete clinical skills.

METHODS

In year 4, medical students conducted one to three mini-CEX per clerkship in gynaecology, internal medicine, paediatrics, psychiatry and surgery. Students and clinical supervisors rated the students' performance on a 10-point scale (1 = great need for improvement; 10 = little need for improvement) in the six domains history taking, physical examination, counselling, clinical judgement, organisation/efficiency and professionalism as well as in overall performance. Correlations between students' self-ratings and ratings from clinical supervisors were calculated (Pearson's correlation coefficient) based on averaged scores per domain and overall. To investigate the dimensionality of the mini-CEX domain scores, we performed factor analyses using linear mixed models that accounted for the multilevel structure of the data.

RESULTS

A total of 1773 mini-CEX from 164 students were analysed. Mean scores for the six domains ranged from 7.5 to 8.3 (student ratings) and from 8.8 to 9.3 (supervisor ratings). Correlations between the ratings of students and supervisors for the different domains varied between r = 0.29 and 0.51 (all p < 0.0001). Mini-CEX domain scores revealed a single-factor solution for both students' and supervisors' ratings, with high loadings of all six domains between 0.58 and 0.83 (students) and 0.58 and 0.84 (supervisors).

CONCLUSIONS

These findings put a question mark on the validity of mini-CEX domain scores for formative purposes, as neither the scores obtained from students nor those obtained from clinical supervisors unravelled specific strengths and weaknesses of individual students' clinical competence.

Self-assessment of surgical ward crisis management using video replay augmented with stress biofeedback

Background

We aimed to explore the feasibility and attitudes towards using video replay augmented with real time stress quantification for the self-assessment of clinical skills during simulated surgical ward crisis management.

Methods

Twenty two clinicians participated in 3 different simulated ward based scenarios of deteriorating post-operative patients. Continuous ECG recordings were made for all participants to monitor stress levels using heart rate variability (HRV) indices. Video recordings of simulated scenarios augmented with real time stress biofeedback were replayed to participants. They were then asked to self-assess their performance using an objective assessment tool. Participants attitudes were explored using a post study questionnaire.

Results

Using HRV stress indices, we demonstrated higher stress levels in novice participants. Self-assessment scores were significantly higher in more experienced participants. Overall, participants felt that video replay and augmented stress biofeedback were useful in self-assessment.

Conclusion

Self-assessment using an objective self-assessment tool alongside video replay augmented with stress biofeedback is feasible in a simulated setting and well liked by participants.

Electronic supplementary material

The online version of this article (10.1186/s13037-018-0153-5) contains supplementary material, which is available to authorized users.

Initial Comparison of Resident and Attending Milestones Evaluations in Plastic Surgery

BACKGROUND

Graduate medical education has recently undergone a major archetypal shift toward competency-based evaluations of residents' performance. The implementation of the Milestones program by the Accreditation Council for Graduate Medical Education (ACGME) is a core component of the shift, designed to ensure uniformity in measuring residency knowledge using a series of specialty-specific achievements. This study evaluates the correlation between residents' self-evaluations and program directors' assessments of their performance.

METHODS

The study population comprised 12 plastic surgery residents, ranging from postgraduate year 1 to postgraduate year 6, enrolled in an integrated residency program at a single institution.

RESULTS

Overall, average attending scores were lower than average resident scores at all levels except postgraduate year 6. Correlation between resident and attending evaluations ranged from 0.417 to 0.957, with the correlation of average scores of Patient Care (0.854) and Medical Knowledge (0.816) Milestones significantly higher than those of professional skillsets (0.581). "Patient care, facial esthetics" was the Milestone with the lowest average scores from both groups. Residents scored themselves notably higher than their attendings' evaluations in Practice-based Learning and Improvement categories (+0.958) and notably lower in Medical Knowledge categories such as "Cosmetic Surgery, Trunk and Lower Extremities" (-0.375) and "Non-trauma hand" (-0.208). The total possible number of participants in this study was 12. The actual number of participants was 12 (100%).

CONCLUSIONS

The remarkable range of correlations suggests that expectations for performance standards may vary widely between residents and program directors. Understanding gaps between expectations and performance is vital to inform current and future residents as the restructuring of the accreditation process continues.

Utility of a Validated Rating Scale for Self-Assessment in Microsurgical Training

OBJECTIVE

The purpose of this study is to determine the utility of self-assessment in microsurgical training using a previously validated rating scale.

DESIGN

A prospective study of surgical residents taking a hands-on 5-day microsurgical training course. Learners completed multiple self-assessments of their technical skills using the University of Western Ontario Microsurgical Acquisition/Assessment instrument. Simultaneously, preceptors assessed the learners using the same scale. Self-assessment and preceptor scores were compared using the Pearson correlation coefficient (PCC).

RESULTS

There was a significant agreement noted between the 32 preceptor assessments and 36 self-assessments that were completed. Correlation between scores for the knot-tying (PCC = 0.62) and anastomosis modules (PCC = 0.77) was good and excellent, respectively. Preceptor scores and self-scores improved over the duration of the course: for preceptors, knot-tying scores increased from 58% on day 1 to 78% on day 5 (p = 0.02) and anastomosis scores improved from 56% to 82% (p = 0.004); for self-scores, knot-tying scores increased from 44% to 81% (p = 0.001) and anastomosis scores from 49% to 84% (p = 0.001). Learners with greater experience (higher postgraduate year level) tended to have higher self as well as preceptor ratings, albeit not statistically significant.

CONCLUSION

Self-assessment using the University of Western Ontario Microsurgical Acquisition/Assessment instrument has good to excellent agreement with preceptor-assessment scores suggesting good interrater reliability. Self-assessment using such tools may, therefore, be used along with preceptor supervision and assessment to potentially improve self-directed learning during these courses.

Video Coaching as an Efficient Teaching Method for Surgical Residents-A Randomized Controlled Trial

BACKGROUND

As surgical training is evolving and operative exposure is decreasing, new, effective, and experiential learning methods are needed to ensure surgical competency and patient safety. Video coaching is an emerging concept in surgery that needs further investigation.

DESIGN

In this randomized controlled trial conducted at a single teaching hospital, participating residents were filmed performing a side-to-side intestinal anastomosis on cadaveric dog bowel for baseline assessment. The Surgical Video Coaching (SVC) group then participated in a one-on-one video playback coaching and debriefing session with a surgeon, during which constructive feedback was given. The control group went on with their normal clinical duties without coaching or debriefing. All participants were filmed making a second intestinal anastomosis. This was compared to their first anastomosis using a 7-category-validated technical skill global rating scale, the Objective Structured Assessment of Technical Skills. A single independent surgeon who did not participate in coaching or debriefing to the SVC group reviewed all videos. A satisfaction survey was then sent to the residents in the coaching group.

SETTING

Department of Surgery, HôpitalMaisonneuve-Rosemont, tertiary teaching hospital affiliated to the University of Montreal, Canada.

PARTICIPANTS

General surgery residents from University of Montreal were recruited to take part in this trial. A total of 28 residents were randomized and completed the study.

RESULTS

After intervention, the SVC group (n = 14) significantly increased their Objective Structured Assessment of Technical Skills score (mean of differences 3.36, [1.09-5.63], p = 0.007) when compared to the control group (n = 14) (mean of differences 0.29, p = 0.759). All residents agreed or strongly agreed that video coaching was a time-efficient teaching method.

CONCLUSIONS

Video coaching is an effective and efficient teaching intervention to improve surgical residents' technical skills.

Are General Surgery Residents Accurate Assessors of Their Own Flexible Endoscopy Skills?

BACKGROUND

Surgeons in training must be able to accurately gauge their own ability and performance to better understand where additional practice is needed and can help inform self-directed learning endeavors. This study had the following 3 goals: (1) to examine the accuracy of residents' assessments of their endoscopic skills, (2) to investigate if accuracy improves over time and practice, and (3) to compare the efficacy of 3 interventions-practice only (PO), self-observation (SO), or expert observation (EO)-on self-assessment accuracy.

METHODS

Overall, 30 first-year general surgery residents completed a pretest on a colonoscopy simulator, which measured time to completion, time to reach the cecum, efficiency of screening, percentage of mucosal surface area examined, time the patient was in pain, and time with a clear view. Residents assigned to the SO and EO conditions reviewed a video of their own performances (SO) or an expert's performance (EO). Residents in all conditions engaged in practice trials using an abstract endoscopy training exercise. Residents then completed a posttest. Self-assessment was examined by calculating discrepancy scores by subtracting actual measurements from participant judgments.

RESULTS

Results indicated that performance for participants in the PO group significantly improved from pretest to posttest for 2 of the 6 metrics and participants in the SO and EO groups improved for 4 metrics. In terms of self-assessment discrepancy scores, only the EO group significantly improved for 2 of the 6 metrics (overall time and screening efficiency).

DISCUSSION

Novice trainees are inaccurate self-assessors of their endoscopic skills before training. Allowing trainees to watch videos of themselves or an expert performing an endoscopic task enhances performance. Participants assigned to PO exhibited decreased ability to accurately judge their own performance. Those in the EO group became significantly better at assessing their overall time and overall efficiency.

SUMMARY

Novice trainees are inaccurate self-assessors of their endoscopic skills before training. Allowing trainees to watch videos of themselves or an expert performing an endoscopic task enhances performance.

Factors affecting confidence and knowledge in spinal palpation among International Manual Physical Therapists

OBJECTIVES

We sought to find if there was a relationship between the confidence in use of static palpation, passive physiological intervertebral motion (PPIVM) and passive accessory intervertebral motion (PAIVM) and the manual therapist's (MTs) knowledge of the literature on these topics.

METHODS

We designed an international survey to achieve our objectives. Each skill was surveyed for the cervical, thoracic and lumbar spines. We also included several other factors that we believed might influence the use of these skills.

RESULTS

We concluded that familiarity of the literature was significantly associated with a MTs' confidence in the use of static palpation, PPIVM and PAIVM techniques. We also found a relationship with the country of practice of the MT and their confidence using these techniques.

DISCUSSION

Spinal palpation is an integral part of the MT's evaluation and treatment abilities. The vast majority of MTs use spinal palpation and nearly all entry-level education programmes include it in their training. Knowing what factors influence MTs' confidence assessing and treating the spine may allow for more effective teaching and training, as well as improved patient outcomes.

Strategies for increasing the feasibility of performance assessments during competency-based education: Subjective and objective evaluations correlate in the operating room

BACKGROUND

Competency-based education necessitates assessments that determine whether trainees have acquired specific competencies. The evidence on the ability of internal raters (staff surgeons) to provide accurate assessments is mixed; however, this has not yet been directly explored in the operating room. This study's objective is to compare the ratings given by internal raters vs an expert external rater (independent to the training process) in the operating room.

METHODS

Raters assessed general surgery residents during a laparoscopic cholecystectomy for their technical and nontechnical performance.

RESULTS

Fifteen cases were observed. There was a moderately positive correlation (r_s = .618, P = .014) for technical performance and a strong positive correlation (r_s = .731, P = .002) for nontechnical performance. The internal raters were less stringent for technical (mean rank 3.33 vs 8.64, P = .007) and nontechnical (mean rank 3.83 vs 8.50, P = .01) performances.

CONCLUSIONS

This study provides evidence to help operationalize competency-based assessments.

Evaluating Intraoperative Laparoscopic Skill: Direct Observation Versus Blinded Videotaped Performances

The Global Operative Assessment of Laparoscopic Skill (GOALS) has been shown to meet high standards for direct observation. The purpose of this study was to investigate the reliability and validity of GOALS when applied to blinded, videotaped performances. Five novice surgeons and 5 experienced surgeons were each evaluated by 2 observers during a laparoscopic cholecystectomy. Subsequently, 4 laparoscopists (V1 to V4) evaluated the videotaped procedures using GOALS. Two of the raters (V1 and V3) had prior experience using GOALS. The interrater reliabilities between video raters (VRs) and between VRs and direct raters (DRs) were calculated using the intraclass correlation coefficient. Construct validity was assessed using 2-way analysis of variance. Interrater reliability between the 4 VRs and the 2 DRs was 0.72. The intraclass correlation coefficient for the 4 VRs was 0.68 and for each VR compared with the mean DR was 0.86, 0.39, 0.94, and 0.76, respectively. All raters, except V2, differentiated between novice and experienced groups ( P values ranged from .01 to .05). These data suggest that GOALS can be used to assess laparoscopic skill based on videotaped performances but that rater training may play an important role in ensuring the reliability and validity of the instrument. Experience with the tool in the operating room may improve the reliability of video rating and could be of value in training evaluators.

Flawed Self-Assessment

Research from numerous corners of psychological inquiry suggests that self-assessments of skill and character are often flawed in substantive and systematic ways. We review empirical findings on the imperfect nature of self-assessment and discuss implications for three real-world domains: health, education, and the workplace.

In general, people's self-views hold only a tenuous to modest relationship with their actual behavior and performance. The correlation between self-ratings of skill and actual performance in many domains is moderate to meager—indeed, at times, other people's predictions of a person's outcomes prove more accurate than that person's self-predictions. In addition, people overrate themselves. On average, people say that they are “above average” in skill (a conclusion that defies statistical possibility), overestimate the likelihood that they will engage in desirable behaviors and achieve favorable outcomes, furnish overly optimistic estimates of when they will complete future projects, and reach judgments with too much confidence. Several psychological processes conspire to produce flawed self-assessments.

Research focusing on health echoes these findings. People are unrealistically optimistic about their own health risks compared with those of other people. They also overestimate how distinctive their opinions and preferences (e.g., discomfort with alcohol) are among their peers—a misperception that can have a deleterious impact on their health. Unable to anticipate how they would respond to emotion-laden situations, they mispredict the preferences of patients when asked to step in and make treatment decisions for them. Guided by mistaken but seemingly plausible theories of health and disease, people misdiagnose themselves—a phenomenon that can have severe consequences for their health and longevity.

Similarly, research in education finds that students' assessments of their performance tend to agree only moderately with those of their teachers and mentors. Students seem largely unable to assess how well or poorly they have comprehended material they have just read. They also tend to be overconfident in newly learned skills, at times because the common educational practice of massed training appears to promote rapid acquisition of skill—as well as self-confidence—but not necessarily the retention of skill. Several interventions, however, can be introduced to prompt students to evaluate their skill and learning more accurately.

In the workplace, flawed self-assessments arise all the way up the corporate ladder. Employees tend to overestimate their skill, making it difficult to give meaningful feedback. CEOs also display overconfidence in their judgments, particularly when stepping into new markets or novel projects—for example, proposing acquisitions that hurt, rather then help, the price of their company's stock. We discuss several interventions aimed at circumventing the consequences of such flawed assessments; these include training people to routinely make cognitive repairs correcting for biased self-assessments and requiring people to justify their decisions in front of their peers.

The act of self-assessment is an intrinsically difficult task, and we enumerate several obstacles that prevent people from reaching truthful self-impressions. We also propose that researchers and practitioners should recognize self-assessment as a coherent and unified area of study spanning many subdisciplines of psychology and beyond. Finally, we suggest that policymakers and other people who makes real-world assessments should be wary of self-assessments of skill, expertise, and knowledge, and should consider ways of repairing self-assessments that may be flawed.

Validation of a performance checklist for ultrasound-guided internal jugular central lines for use in procedural instruction and assessment

PURPOSE OF THE STUDY

Tools created to measure procedural competency must be tested in their intended environment against an established standard in order to be validated. We previously created a checklist for ultrasound-guided internal jugular central venous catheter (US IJ CVC) insertion using the modified Delphi method. We sought to further validate the checklist tool for use in an educational environment.

STUDY DESIGN

This is a cohort study involving 15 emergency medicine interns being evaluated on their skill in US IJ CVC placement. We compared the checklist tool with a modified version of a clinically validated global rating scale (GRS) for procedural performance.

RESULTS

The correlation between the GRS tool and the checklist tool was excellent, with a correlation coefficient (Pearson's r) of 0.90 (p<0.0001).

CONCLUSIONS

This checklist represents a useful tool for measuring procedural competency.

Arterial Catheter Use in the ICU: A National Survey of Antiseptic Technique and Perceived Infectious Risk

OBJECTIVES

Recent studies have shown that the occurrence rate of bloodstream infections associated with arterial catheters is 0.9-3.4/1,000 catheter-days, which is comparable to that of central venous catheters. In 2011, the Centers for Disease Control and Prevention published new guidelines recommending the use of limited barrier precautions during arterial catheter insertion, consisting of sterile gloves, a surgical cap, a surgical mask, and a small sterile drape. The goal of this study was to assess the attitudes and current infection prevention practices used by clinicians during insertion of arterial catheters in ICUs in the United States.

DESIGN

An anonymous, 22-question web-based survey of infection prevention practices during arterial catheter insertion.

SETTING

Clinician members of the Society of Critical Care Medicine.

SUBJECTS

Eleven thousand three hundred sixty-one physicians, nurse practitioners, physician assistants, respiratory therapists, and registered nurses who elect to receive e-mails from the Society of Critical Care Medicine.

INTERVENTIONS

None.

MEASUREMENTS AND MAIN RESULTS

There were 1,265 responses (11% response rate), with 1,029 eligible participants after exclusions were applied. Only 44% of participants reported using the Centers for Disease Control and Prevention-recommended barrier precautions during arterial catheter insertion, and only 15% reported using full barrier precautions. The mean and median estimates of the incidence density of bloodstream infections associated with arterial catheters were 0.3/1,000 catheter-days and 0.1/1,000 catheter-days, respectively. Thirty-nine percent of participants reported that they would support mandatory use of full barrier precautions during arterial catheter insertion.

CONCLUSIONS

Barrier precautions are used inconsistently by critical care clinicians during arterial catheter insertion in the ICU setting. Less than half of clinicians surveyed were in compliance with current Centers for Disease Control and Prevention guidelines. Clinicians significantly underestimated the infectious risk posed by arterial catheters, and support for mandatory use of full barrier precautions was low. Further studies are warranted to determine the optimal preventive strategies for reducing bloodstream infections associated with arterial catheters.

[Evaluation of residency training in general and digestive surgery in Tunisia]

Introduction

De nombreux moyens sont mis à disposition des résidents en chirurgie générale et digestive pour assurer leur formation théorique et pratique. Cependant, le niveau d'utilisation de ces différents outils et leur impact sur la formation des résidents n'ont jamais été évalués. L'objectif de notre étude était d’étudier l’état des lieux des moyens de formation utilisés par les résidents pour évaluer leurs degrés de satisfaction et leurs propositions en vue d'améliorer leur formation.

Méthodes

Un questionnaire anonyme a été distribué aux résidents de chirurgie générale et digestive de l'année 2012-2013. Ce questionnaire portait sur les caractéristiques démographiques, les ressources pédagogiques, ainsi que le cursus médical et universitaire. Une évaluation de la formation ainsi qu'un recueil des propositions faites en vue d'améliorer leurs formations étaient réalisées.

Résultats

Cinquante résidents sur 83 ont répondu au questionnaire. L'orientation de carrière la plus fréquente était l'hospitalo-universitaire dans 70% des cas. La pratique quotidienne et l'internet étaient les deux ressources pédagogiques les plus utilisées. La formation chirurgicale était jugée satisfaisante par seulement 10% des répondants. Parmi l'ensemble des propositions faites, l'apprentissage sur simulateur chirurgical, l'existence d'un ouvrage national de référence, et l'institution d'un tutorat par un chirurgien senior recueillaient plus de 80% d'avis favorable.

Conclusion

La majorité des résidents jugent leur formation non satisfaisante. Une meilleure information sur les ressources déjà existantes, un renforcement du compagnonnage et un accès plus large à un apprentissage sur simulateur chirurgical permettraient de diminuer ce sentiment d'insatisfaction.

Resident Surgeons Underrate Their Laparoscopic Skills and Comfort Level When Compared With the Rating by Attending Surgeons

OBJECTIVE

The development of operative skills during general surgery residency depends largely on the resident surgeons' (residents) ability to accurately self-assess and identify areas for improvement. We compared evaluations of laparoscopic skills and comfort level of residents from both the residents' and attending surgeons' (attendings') perspectives.

DESIGN

We prospectively observed 111 elective cholecystectomies at the University of Michigan as part of a larger quality improvement initiative. Immediately after the operation, both residents and attendings completed a survey in which they rated the residents' operative proficiency, comfort level, and the difficulty of the case using a previously validated instrument. Residents' and attendings' evaluations of residents' performance were compared using 2-sided t tests.

SETTING

The University of Michigan Health System in Ann Arbor, MI. Large academic, tertiary care institution.

PARTICIPANTS

All general surgery residents and faculty at the University of Michigan performing laparoscopic cholecystectomy between June 1 and August 31, 2013. Data were collected for 28 of the institution's 54 trainees.

RESULTS

Attendings rated residents higher than what residents rated themselves on a 5-point Likert-type scale regarding depth perception (3.86 vs. 3.38, p < 0.005), bimanual dexterity (3.75 vs. 3.36, p = 0.005), efficiency (3.58 vs. 3.18, p < 0.005), tissue handling (3.69 vs. 3.23, p < 0.005), and comfort while performing a case (3.86 vs. 3.38, p < 0.005). Attendings and residents were in agreement on the level of autonomy displayed by the resident during the case (3.31 vs. 3.34, p = 0.85), the level of difficulty of the case (2.98 vs. 2.85, p = 0.443), and the degree of teaching done by the attending during the case (3.61 vs. 3.54, p = 0.701).

CONCLUSIONS

A gap exists between residents' and attendings' perception of residents' laparoscopic skills and comfort level in performing laparoscopic cholecystectomy. These findings call for improved communication between residents and attendings to ensure that graduates are adequately prepared to operate independently. In the context of changing methods of resident evaluations that call for explicitly defined competencies in surgery, it is essential that residents are able to accurately self-assess and be in general agreement with attendings on their level of laparoscopic skills and comfort level while performing a case.

Are general surgeons able to accurately self-assess their level of technical skills?

INTRODUCTION

Self-assessment is a way of improving technical capabilities without the need for trainer feedback. It can identify areas for improvement and promote professional medical development. The aim of this review was to identify whether self-assessment is an accurate form of technical skills appraisal in general surgery.

METHODS

The PubMed, MEDLINE(®), Embase(™) and Cochrane databases were searched for studies assessing the reliability of self-assessment of technical skills in general surgery. For each study, we recorded the skills assessed and the evaluation methods used. Common endpoints between studies were compared to provide recommendations based on the levels of evidence.

RESULTS

Twelve studies met the inclusion criteria from 22,292 initial papers. There was no level 1 evidence published. All papers compared the correlation between self-appraisal versus an expert score but differed in the technical skills assessment and the evaluation tools used. The accuracy of self-assessment improved with increasing experience (level 2 recommendation), age (level 3 recommendation) and the use of video playback (level 3 recommendation). Accuracy was reduced by stressful learning environments (level 2 recommendation), lack of familiarity with assessment tools (level 3 recommendation) and in advanced surgical procedures (level 3 recommendation).

CONCLUSIONS

Evidence exists to support the reliability of self-assessment of technical skills in general surgery. Several variables have been shown to affect the accuracy of self-assessment of technical skills. Future work should focus on evaluating the reliability of self-assessment during live operating procedures.

Privileging Naturals Over Strivers

A preference for “naturals” over “strivers” in performance judgments was investigated to test whether the effect is generalizable across domains, as well as to ascertain any costs imposed on decision quality by favoring naturals. Despite being presented with entrepreneurs equal in achievement, participants judged the natural and his business proposal to be superior to the striver and his proposal on multiple dimensions of performance and success (Study 1a and Study 1b). These findings were extended in Study 2, which quantified the costs of the naturalness bias using conjoint analysis to measure specific decision tradeoffs. Together, these three studies show that people tend to pass over better-qualified individuals in favor of apparent naturals.

The effect of video review of resident laparoscopic surgical skills measured by self- and external assessment

BACKGROUND

Video review of surgical skills is an educational modality that allows trainees to reflect on self-performance. The purpose of this study was to determine whether resident and attending assessments of a resident's laparoscopic performance differ and whether video review changes assessments.

METHODS

Third-year surgery residents were invited to participate. Elective laparoscopic procedures were video recorded. The Global Operative Assessment of Laparoscopic Skills evaluation was completed immediately after the procedure and again 7 to 10 days later by both resident and attending. Scores were compared using t tests.

RESULTS

Nine residents participated and 76 video reviews were completed. Residents scored themselves significantly lower than the faculty scores both before and after video review. Resident scores did not change significantly after video review.

CONCLUSIONS

Attending and resident self-assessment of laparoscopic skills differs and subsequent video review does not significantly affect Global Operative Assessment of Laparoscopic Skills scores. Further studies should evaluate the impact of video review combined with verbal feedback on skill acquisition and assessment.

The adequacy of Hepato-Pancreato-Biliary training: how closely do perceptions of fellows and programme directors align?

BACKGROUND

Hepatopancreatobiliary fellowship programmes have recently undergone significant changes with regards to training standards, case-volume thresholds and multimodality educational platforms. The goals of this study were to compare the perspectives of fellows and programme directors (PDs) on perceptions of readiness to enter practice and identify core Hepato-Pancreato-Biliary (HPB) procedures that require increased emphasis during training.

METHODS

This survey targeted PDs and trainees participating in the Fellowship Council/AHPBA pathway. Data related to demographics, education and career plans were collected. Analysis of PD and fellow opinions regarding their confidence to perform core HPB procedures was completed.

RESULTS

The response rate was 88% for both fellows (21/24) and PDs (23/26). There was good agreement between PDs and fellows in the perception of case volumes. Select differences where PDs ranked higher perceptions included major hepatectomies (PDs: 87% versus fellows: 57%, P = 0.04), pancreaticoduodenectomies (100% versus 81%, P = 0.04) and laparoscopic distal pancreatectomies (78% versus 43%, P = 0.03). 'Good or excellent' case volumes translated into increased fellow readiness, except for some pancreatitis procedures, laparoscopic distal pancreatectomies and potentially major hepatectomies.

CONCLUSIONS

This study provides insight into content domains that may require additional attention to achieve an appropriate level of proficiency and confidence upon completion of training.