Self vs expert assessment of technical and non-technical skills in high fidelity simulation

Enhancing trauma care through innovative trauma and disaster team response training: A blended learning approach in Tanzania

BACKGROUND

In Tanzania, inadequate infrastructures and shortages of trauma-response training exacerbate trauma-related fatalities. McGill University's Centre for Global Surgery introduced the Trauma and Disaster Team Response course (TDTR) to address these challenges. This study assesses the impact of simulation-based TDTR training on care providers' knowledge/skills and healthcare processes to enhance patient outcomes.

METHODS

The study used a pre-post-interventional design. TDTR, led by Tanzanian instructors at Muhimbili Orthopedic Institute from August 16-18, 2023, involved 22 participants in blended online and in-person approaches with simulated skills sessions. Validated tools assessed participants' knowledge/skills and teamwork pre/post-interventions, alongside feedback surveys. Outcome measures included evaluating 24-h emergency department patient arrival-to-care time pre-/post-TDTR interventions, analyzed using parametric and non-parametric tests based on data distributions.

RESULTS

Participants' self-assessment skills significantly improved (median increase from 34 to 58, p < 0.001), along with teamwork (median increase from 44.5 to 87.5, p < 0.003). While 99% of participants expressed satisfaction with TDTR meeting their expectations, 97% were interested in teaching future sessions. The six-month post-intervention arrival-to-care time significantly decreased from 29 to 13 min, indicating a 55.17% improvement (p < 0.004). The intervention led to fewer ward admissions (35.26% from 51.67%) and more directed to operating theaters (29.83% from 16.85%), suggesting improved patient management (p < 0.018).

CONCLUSION

The study confirmed surgical skills training effectiveness in Tanzanian settings, highlighting TDTR's role in improving teamwork and healthcare processes that enhanced patient outcomes. To sustain progress and empower independent trauma educators, ongoing refresher sessions and expanding TDTR across low- and middle-income countries are recommended to align with global surgery goals.

Observation of behavioural skills by medical simulation facilitators: a cross-sectional analysis of self-reported importance, difficulties, observation strategies and expertise development

BACKGROUND

The association between team performance and patient care was an immense boost for team-based education in health care. Behavioural skills are an important focus in these sessions, often provided via a mannikin-based immersive simulation experience in a (near) authentic setting. Observation of these skills by the facilitator(s) is paramount for facilitated feedback with the team. Despite the acknowledgement that trained facilitators are important for optimal learning, insight into this observation process by facilitators is limited.

OBJECTIVES

What are the self-reported current practices and difficulties regarding the observation of behavioural skills amongst facilitators during team training and how have they been trained to observe behavioural skills?

METHODS

This cross-sectional study used a pilot-tested, content-validated, multi-linguistic online survey within Europe, distributed through a non-discriminative snowball sampling method. Inclusion was limited to facilitators observing behavioural skills within a medical team setting.

RESULTS

A total of 175 persons filled in the questionnaire. All aspects of behavioural skill were perceived as very important to observe. The self-perceived difficulty of the behavioural skill aspects ranged from slightly to moderately difficult. Qualitative analysis revealed three major themes elaborating on this perceived difficulty: (1) not everything can be observed, (2) not everything is observed and (3) interpretation of observed behavioural skills is difficult. Additionally, the number of team members health care facilitators have to observe, outnumbers their self-reported maximum. Strategies and tools used to facilitate their observation were a blank notepad, co-observers and predefined learning goals. The majority of facilitators acquired observational skills through self-study and personal experience and/or observing peers. Co-observation with either peers or experts was regarded as most learn some for their expertise development. Overall, participants perceived themselves as moderately competent in the observation of behavioural skills during team training.

CONCLUSIONS

Observation of behavioural skills by facilitators in health care remains a complex and challenging task. Facilitators' limitations with respect to attention, focus and (in)ability to perform concomitant tasks, need to be acknowledged. Although strategies and tools can help to facilitate the observation process, they all have their limitations and are used in different ways.

Are Plastic Surgery Trainees Accurate Assessors of Their Own Microsurgical Skill?

Background

Microsurgery is a technically demanding surgical discipline with a steep learning curve. Trainees have faced several difficulties due to less hands-on theater time and pandemic-related limits on access to technical training. To overcome this, trainees engaged in self-directed training, which requires an accurate self-assessment of skill. This study aimed to assess the ability of trainees to accurately self-assess their performance while performing a simulated microvascular anastomosis.

Methods

Novice and specialist plastic surgery trainees performed a simulated microvascular anastomosis on a high-fidelity chicken femoral vessel model. Each participant objectively rated the quality of their anastomosis using the Anastomosis Lapse Index (ALI). Two expert microsurgeons subsequently blindly rated each anastomosis. To determine the accuracy of self-evaluation, self-scores and expert-scores were compared using a Wilcoxon signed-rank test.

Results

Twenty-seven surgical trainees completed the simulation, with a mean time to completion (TTC) of 40.3 minutes (range 14.2-106.0 minutes). For the entire cohort, the median ALI self-score was 4 (range 3-10), while the median ALI expert-score was 5.5 (range 2.5-9.5). There was a significant difference between the ALI self-score and the expert-score (p<0.001). When grouped by experience level, there was no significant difference between self-score and expert-score within the specialist group, while there was a significant difference within the novice group (p=0.001).

Conclusion

These findings suggest that specialist trainees are accurate assessors of their own microsurgical skills, but novice trainees tend to overestimate their technical skills. While novice trainees can engage in independent self-directed microsurgical training, they should seek expert feedback to ensure targeted training.

Comparison Between Surgical Trainee Self-Assessment With Supervisor Scores in Technical and Nontechnical Skills to Identify Underperforming Trainees

OBJECTIVE

This study aims to compare trainee self-assessment with supervisor assessment to identify differences in correlations of the demographic data and evaluate whether the instrument can be utilized to identify underperforming trainees.

DESIGN

A novel instrument was designed based off the Royal Australasian College of Surgeons original 9 competencies utilizing the JDocs framework and covers 48-items across all surgical competencies. A multiple regression model using age, gender, postgraduate year, IMG status, and level of training as the variables was performed with backwards elimination, and pairwise comparisons made to identify the degree and direction of influence each variable contributed to trainee and supervisor ratings.

SETTING

Surgical trainees employed in tertiary centers within the Australian Capital Territory and South-East New South Wales health network in Australia.

PARTICIPANTS

A total of 36 of 59 (61%) trainees responded. Two grossly incomplete responses were excluded from further analysis resulting in 34 completed self-assessments for analysis. There was a total of 68 supervisor assessments provided by 25 different nominated supervisors. Of these assessments, 67 were fully completed and one was partially complete.

RESULTS

We identified that for both self-ratings and supervisor ratings, the most significant correlation is with the postgraduate year of the trainee, with correlation identified in 7 out of 9 competencies, although in different competency domains. International Medical Graduate status is associated in 2 of 9 self-ratings, and in 3 of 9 supervisor ratings. Underperforming trainees were able to be identified through supervisor assessment but not self-assessment.

CONCLUSION

The supervisor assessment form was able to identify underperforming trainees. Our findings resonate with existing literature in other specialty fields, and surgical units that employ assessment forms should feel more confident in the interpretation of the data and provision of feedback to trainees.

Simulation in Surgical Resident Education: A Porcine Hemostasis and Laparoscopic Model

The steep learning curve associated with learning laparoscopic techniques and limited training opportunities represents a challenge to general surgery resident training. The objective of this study was to use a live porcine model to improve surgical training in laparoscopic technique and management of bleeding. Nineteen general surgery residents (ranging from PGY 3 to 5) completed the porcine simulation and completed pre-lab and post-lab questionnaires. The institution's industry partner served as sponsors and educators on hemostatic agents and energy devices. Residents had a significant increase in confidence with laparoscopic techniques and the management of hemostasis (P = .01 and P = .008, respectively). Residents agreed and then strongly agreed that a porcine model was suitable to simulate laparoscopic and hemostatic techniques, but there was no significant change between pre- and post-lab opinions. This study demonstrates that a porcine lab is an effective model for surgical resident education and increases resident confidence.

Interprofessional simulation of acute care for nursing and medical students: interprofessional competencies and transfer to the workplace

BACKGROUND

Teamwork and communication are essential tools for doctors, nurses and other team members in the management of critically ill patients. Early interprofessional education during study, using acute care simulation, may improve teamwork and communication between interprofessional team members on the long run.

METHODS

A comparative sequential quantitative-qualitative study was used to understand interprofessional learning outcomes in nursing and medical students after simulation of acute care. Students were assigned to a uni- or interprofessional training. Questionnaires were used to measure short and long term differences in interprofessional collaboration and communication between the intervention and control group for nursing and medical students respectively. Semi-structured focus groups were conducted to gain a better understanding of IPE in acute simulation.

RESULTS

One hundred and ninety-one students participated in this study (131 medical, 60 nursing students). No differences were found between the intervention and control group in overall ICCAS scores for both medical and nursing students (p = 0.181 and p = 0.441). There were no differences in ICS scores between the intervention and control group. Focus groups revealed growing competence in interprofessional communication and collaboration for both medical and nursing students.

CONCLUSIONS

Interprofessional simulation training did show measurable growth of interprofessional competencies, but so did uniprofessional training. Both medical and nursing students reported increased awareness of perspective and expertise of own and other profession. Furthermore, they reported growing competence in interprofessional communication and collaboration in transfer to their workplace.

Radiography education with VR using head mounted display: proficiency evaluation by rubric method

BACKGROUND

The use of head mounted display (HMD)-based immersive virtual reality (VR) coaching systems (HMD-VRC) is expected to be effective for skill acquisition in radiography. The usefulness of HMD-VRC has been reported in many previous studies. However, previous studies have evaluated the effectiveness of HMD-VRC only through questionnaires. HMD-VRC has difficulties in palpation and patient interaction compared to real-world training. It is expected that these issues will have an impact on proficiency. The purpose of this study is to determine the impact of VR constraints in HMD-VRC, especially palpation and patient interaction, on radiographic skills proficiency in a real-world setting.

METHODS

First-year students (n = 30) at a training school for radiology technologists in Japan were randomly divided into two groups, one using HMD-VRC (HMD-VRC group) and the other practicing with conventional physical equipment (RP group) and trained for approximately one hour. The teachers then evaluated the students for proficiency using a rubric method.

RESULTS

In this study, it was found that some skills in the HMD-VRC group were equivalent to those of the RP group and some were significantly lower than those of the RP group. There was a significant decrease in proficiency in skills related to palpation and patient interaction.

CONCLUSIONS

This study suggests that HMD-VRC can be less effective than real-world training in radiographic techniques, which require palpation and patient interaction. For effective training, it is important to objectively evaluate proficiency in the real world, even for HMD-VRC with new technologies, such as haptic presentation and VR patient interaction.

TRIAL REGISTRATION

The study was conducted with the approval of the Ethics Committee of International University of Health and Welfare (Approval No.21-Im-035, Registration date: September 28, 2021).

Development of a novel behaviourally anchored instrument for the assessment of surgical trainees

Background

The Royal Australasian College of Surgeons (RACS) created its competency framework in 2003 which initially consisted of nine competencies each regarded as equally important for a practising surgeon. The JDocs Framework is aligned to these competencies and provides guidance for junior doctors working towards the Surgical Education and Training program.

Methods

A novel assessment instrument was designed around the JDocs framework using 48 behaviourally anchored questions. The study was completed in 2020 across five public hospitals in the ACT and NSW. Participants were invited to complete the self‐assessment form online.

Results

Thirty‐six of 59 (61%) trainees participated in the study, with 67 of 68 (98.5%) supervisors having completed the assessment form. Trainee self‐rating scores were lower than that of supervisor ratings across all competencies except communication. The self‐rating scores were negatively correlated with the seniority of a trainee's level in all nine competencies. The years of post‐graduate experience was positively correlated with seven of the nine competencies. For gender and International Medical Graduate status, correlation was only identified for health advocacy and medical expertise. There was no correlation identified with a trainee's age.

Conclusion

This pilot study has provided an opportunity to explore a new assessment instrument for surgical trainees that is aligned to the RACS competency framework using behaviourally anchored questions. Looking ahead, a better understanding of this instrument will potentially be helpful in early identification of underperforming trainees in order to facilitate early intervention, or its use as a selection tool for formal training programs.

Focused Bedside Ultrasound Training Program for Surgical Residents in the Intensive Care Unit of Tertiary hospital

Bedside ultrasound has become one of the most important noninvasive and readily available diagnostic tools for critically ill patients. However, the current ultrasound training program for surgical residents is not standardized and is mostly unavailable to all surgical residents equally. Herein, we evaluated the effectiveness of the new training program in bedside ultrasound for surgical residents. Postgraduate residents (years 1 to 4) from the department of general surgery in a tertiary hospital attended the newly designed, 8-week ultrasound training course at the surgical intensive care unit. Didactic and experimental lectures about basic ultrasound physics and machine usage were delivered, followed by daily hands-on training to actual patients. Each participant documented their ultrasound findings and completed a self-assessment survey of ultrasound skills using the Likert scale. A total of 44 residents were enrolled, and only 36.4% of them were previously exposed to bedside ultrasound experience. Following the completion of the training course, the proficiency levels and the objective structured assessment of ultrasound skill scores showed significant improvement in every element (P < 0.001). The mean differences in pre- and post-course scores between post-graduate years and post hoc analysis revealed that the post-graduate year 2 group showed a higher improvement in most elements. Whether or not residents had previous experience with ultrasound, the significant improvement was seen in post-course scores. The knowledge and confidence of surgical residents in bedside ultrasound could be improved after our short training curriculum. Such education should be encouraged for all surgical residents to enhance their competency in performing bedside ultrasounds and use in managing critically ill patients.

Comparing control room operators' and experts' assessment of team performance using structured task-specific observation protocols and scenario replay

Operators' self-assessment has received limited interest within process control or human-system evaluation. Research on self-assessment has been criticised for poor assessment methodology, and consequently, its status is unclear. This study hypothesised that, given adequate assessment methods (such as task-specific assessment items and scenario replay), we could observe relatively accurate self-assessment results. Eighteen licensed operators and two experts assessed team performance in six nuclear control room scenarios. The results reveal an overall agreement between operators and experts, measured by the intraclass correlation coefficient, ranging from 0.60 to 0.70, which lies close to the intraclass correlation coefficient of 0.75 for the experts. This demonstrates potential for achievement of relatively accurate operator self-assessment for complex work. The agreement varied in a similar manner for both expert agreement and operator-expert agreement across eight performance dimensions. In addition, the operators' self-assessment provided additional information beyond observer assessment in identifying non-acceptable performance items.

Performance of Residents During Laparoscopic Cholecystectomy: Is Self-assessment Reliable?

BACKGROUND

To compare the self-assessment of surgical residents and observers (faculty members and nurses) during laparoscopic cholecystectomy (LC).

MATERIALS AND METHODS

A prospective observational study was conducted between February 2020 and July 2020 at a medical school hospital. Seventy-four LC surgeries were performed by surgical residents in the presence of faculty members. A self-assessment of the technical and nontechnical performance of the residents was requested. The self-assessment of residents was compared with observer evaluations using the Kruskal-Wallis test. Gwet AC2 fit coefficient was used to determine the consistency between the observers' and residents' assessments. Bland-Altman plots were generated with 95% limits of agreement to describe the agreement between the total scores of the observers.

RESULTS

The self-assessment of residents had a statistically significant higher score when compared with observers (faculty and nurses) (P<0.001). However, no significant difference was observed between the total scores given by the observers (faculty members and nurses) (P>0.05). There was a moderate agreement between the resident versus faculty members [0.503; 95% confidence interval (CI), 0.430-0.576] and resident versus nurse (0.518; 95% CI, 0.432-0.605) when evaluating technical skills. However, there was substantial agreement between faculty members and nurses (0.736; 95% CI, 0.684-0.789). Postoperative pain was significantly correlated with resident self-assessment (P=0.022).

CONCLUSION

The self-assessment scores of surgical residents in LC operations were overestimated compared with observer assessments.

Development and Validation of a Virtual Reality Haptic Femoral Nailing Simulator

OBJECTIVE

To create a virtual reality (VR) femoral nailing simulator combining haptics and image intensifier functionality and then carry out validation studies to assess its educational value.

DESIGN

The simulator consisted of a 3D virtual environment, a haptic device and 3D printed drill handle and a VR headset. The environment was created using a video game development engine, interfaced with plugins to allow haptic feedback and image intensifier functionality. Two tasks were created within the simulator as part of an antegrade femoral intramedullary (IM) nail procedure: proximal guidewire entry and distal locking.For the validation study, participants performed the above tasks on the simulator. Metrics were collected including time taken, number of X-rays and tool distance travelled and used to assess construct validity. A questionnaire was then completed to assess authenticity and content validity.

SETTING

Simulator development in centre for simulation and engagement science laboratory. Validation study in a teaching hospital environment.

PARTICIPANTS

Orthopedic specialist trainees and consultants.

RESULTS

Surgeon experience (number of IM nails performed/postgraduate year) correlated with significantly improved task performance. More experienced surgeons took less time, used fewer X-rays and had greater economy of movement than less experienced surgeons. Authenticity and content validity were well rated, with criticisms primarily due to hardware limitations.

CONCLUSIONS

To our knowledge this is the first orthopedic simulator to combine immersive VR with haptics and full image intensifier functionality. By combining multiple aspects of surgical practice within a single device, we aimed to improve participant immersion and educational value. Our work so far has focused on technical skills, demonstrating good authenticity, content and construct validity, however our findings show promise in other applications such as nontechnical skill development and assessment.

Attending Surgeons Demonstrate Greater Correlations of Skill Assessment and Anticipation of Adverse Events Than Residents During Laparoscopic Cholecystectomy

BACKGROUND

Surgical training includes the development of technical and nontechnical skills. While technical skills are more easily quantified, nontechnical skills such as situation awareness (SA) are more difficult to measure and quantify. This study investigated the relationships between different SA elements and expertise.

METHODS

Twenty attending and resident surgeons rated their anticipation of an impending adverse event while watching 20 videos of laparoscopic cholecystectomies with and without adverse events. After watching each video, they assessed surgeon skills and self-assessed their anticipation ratings. All participants answered a general confidence questionnaire before and after the study.

RESULTS

Videos with adverse events led to significantly higher anticipation of adverse events (P < 0.001), lower surgeon skill rating (P < 0.001), and higher self-assessment in their anticipation ratings (P < 0.001) across both participant groups. General confidence was significantly lower for residents than that for attending surgeons (P < 0.001). Compared with the residents, attendings exhibited stronger and more stable correlations between measurements of SA. When viewing videos with adverse events, attendings showed significantly higher correlation between anticipation of an impending adverse event and skill assessment of the surgeon (P = 0.005).

CONCLUSIONS

This study investigated how different elements of SA and their relationships were influenced by experience. The results indicated that attendings had stronger and more stable correlations between SA elements than residents, demonstrating how measurement correlations could be meaningful and sensitive indicators of expertise and autonomy readiness.

Simulation-based team training in time-critical clinical presentations in emergency medicine and critical care: a review of the literature

BACKGROUND

The use of simulation-based team training has increased over the past decades. Simulation-based team training within emergency medicine and critical care contexts is best known for its use by trauma teams and teams involved in cardiac arrest. In the domain of emergency medicine, simulation-based team training is also used for other typical time-critical clinical presentations. We aimed to review the existing literature and current state of evidence pertaining to non-technical skills obtained via simulation-based team training in emergency medicine and critical care contexts, excluding trauma and cardiac arrest contexts.

METHODS

This systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement. Before the initiation of the study, the protocol was registered in the International Prospective Register of Systematic Reviews (PROSPERO) database. We conducted a systematic literature search of 10 years of publications, up to December 17, 2019, in the following databases: PubMed/MEDLINE, EMBASE, Cochrane Library, and CINAHL. Two authors independently reviewed all the studies and extracted data.

RESULTS

Of the 456 studies screened, 29 trials were subjected to full-text review, and 13 studies were included in the final review. None of the studies was randomized controlled trials, and no studies compared simulation training to different modalities of training. Studies were heterogeneous; they applied simulation-training concepts of different durations and intensities and used different outcome measures for non-technical skills. Two studies reached Kirkpatrick level 3. Out of the remaining 11 studies, nine reached Kirkpatrick level 2, and two reached Kirkpatrick level 1.

CONCLUSIONS

The literature on simulation-based team training in emergency medicine is heterogeneous and sparse, but somewhat supports the hypothesis that simulation-based team training is beneficial to teams' knowledge and attitudes toward non-technical skills (Kirkpatrick level 2). Randomized trials are called for to clarify the effect of simulation compared to other modalities of team training. Future research should focus on the transfer of skills and investigate improvements in patient outcomes (Kirkpatrick level 4).

Evaluation of a Novel Laparoscopic Cholecystectomy Curriculum With the Use of Animal Models and Live Operating

BACKGROUND

Despite wide recognition of the benefits of simulation training, there is no established model for the teaching of laparoscopic cholecystectomy. The authors developed a replicable, intensive 2-day proficiency-based curriculum, to include simulation and live operating, designed to be practical to both attend and organize. The primary objective of this study was to evaluate this curriculum for improved procedural confidence, measured by participant self-evaluation. Secondary outcomes were objective improvement in technical skills in accordance with the Objective Structured Assessment of Technical Skills (OSATS) scale and trainee self-assessment scores.

METHODS

The course consisted of lectures, operating on a sheep hepatobiliary model through a laparoscopic box trainer, and live operating on female patient volunteers. It was attended by eight junior registrars. Precourse data collated included demographic information, experience, and procedural confidence scores using a visual analog scale. Performance on an animal model and live patient was assessed by experts using the OSATS score. Procedural confidence was re-evaluated after each task, as well as self-assessment of speed, accuracy, and overall performance.

RESULTS

Procedural confidence scores improved by a mean of 12% (P < 0.001). All trainees demonstrated sustained objective improvement in technical skills (P < 0.001). The overall mean OSATS score increased by 18%. Significant improvement was observed after performing the procedure on an animal model (P < 0.001); however, no further significant improvement was observed with live operating. No significant difference was found on trainee self-assessment scores, in any category.

CONCLUSIONS

The study describes a successful curriculum model for the teaching of laparoscopic cholecystectomy, to include procedural and technical skill acquisition, in addition to the refinement and development of procedural confidence. Importantly, this was carried out in a safe environment with direct transferability to the operating theater.

Video-based self-assessment enhances laparoscopic skills on a virtual reality simulator: a randomized controlled trial

BACKGROUND

Hysterectomy rates are decreasing in many countries, and virtual reality (VR) simulators bring new training opportunities for residents. As coaching interventions while training on a simulated complex procedure represents a resource challenge, alternative strategies to improve surgical skills must be investigated. We sought to determine whether self-guided learning using a video-based self-assessment (SA) leads to improved surgical skills in laparoscopic hysterectomy (LH) on a VR simulator.

METHODS

Twenty-four gynecology residents from two university hospitals were randomized into an SA group (n = 12) and a Control group (n = 12). Each participant's baseline performance on a validated VR basic task was assessed. Both groups then performed three virtually simulated LHs during which the participants received no guidance nor feedback. Following each LH, the SA group participants rated the video of their own performance using a generic and a procedure-specific rating scale, while the Control group participants watched an LH video demonstration. The LH videos of both groups' participants were blindly reviewed and rated by expert surgeons, using modified Objective Structured Assessment of Technical Skills scores (OSATS). Objective metrics recorded by the VR simulator were also compared.

RESULTS

There was no difference between the groups' baseline performances on the VR basic task. For the first LH, the OSATS-derived scores did not differ between SA and Control groups (9 [7-13] versus 9 [8-14]; p = 0.728). For the third LH, the OSATS-derived scores were higher for the SA group than for the Control group (17 [15-21] versus 15 [11-17], p = 0.039). Between the two groups, the objective metrics did not differ from the first to the third LH.

CONCLUSIONS

The use of a structured video-based SA leads to improved procedural skills in LH on a VR simulator compared to watching benchmark expert performance, in a population of residents with moderate experience in the operating room.

Evaluating the Impact of Surgeon Self-Awareness by Comparing Self vs Peer Ratings of Surgical Skill and Outcomes for Bariatric Surgery

OBJECTIVE

To evaluate variation in self vs. peer-assessments of surgical skill using surgical videos and compare surgeon-specific outcomes with bariatric surgery.

SUMMARY BACKGROUND DATA

Prior studies have demonstrated that surgeons with lower peer-reviewed ratings of surgical skill had higher complication rates after bariatric surgery.

METHODS

This is a retrospective cohort study of 25 surgeons who voluntarily submitted a video of a typical laparoscopic sleeve gastrectomy (SG) between 2015-2016. Videos were self and peer-rated using a validated instrument based on a 5-point Likert scale (5="master surgeon" and 1="surgeon-in-training"). Risk adjusted 30-day complication rates were compared between surgeons who over-rated and under-rated their skill based on data from 24,186 SG cases as well as 12,888 gastric bypass (GBP) cases.

RESULTS

Individual overall self-rating of surgical skill varied between 2.5 to 5. Surgeons in the top quartile for self:peer ratings (n=6, ratio 1.58) had lower overall mean peer-scores (2.98 vs 3.79, p = 0.0150) than surgeons in the lowest quartile (n = 6, ratio 0.94). Complication rates between top and bottom quartiles were similar after SG, however leak rates were higher with GBP among surgeons who over-rated their skill with sleeve gastrectomy (0.65 vs 0.27, p = 0.0181). Surgeon experience was similar between comparison groups.

CONCLUSIONS AND RELEVANCE

Self-perceptions of surgical skill varied widely. Surgeons who over-rated their skill had higher leak rates for more complex procedures. Video assessments can help identify surgeons with poor self-awareness who may benefit from a surgical coaching program.

Simulation training for lobectomy: a review of current literature and future directions†

With growing work-time restrictions and public expectations, the Halstedian educational model of 'see one, do one, teach one' is unfit for the modern training of thoracic residents. With the cardiothoracic surgical workforce set to decline by 50% over the next 10 years, new models are desperately needed to help trainees bypass the early error-prone phase of the lobectomy learning curve. In this review, we detail the development and validation of numerous simulators designed to teach trainees an array of skills ranging from basic technical skills to more complex non-technical skills. Given the recent increases in minimally invasive lobectomies, we critique both open and thoracoscopic simulators. We elucidate that whilst there are a growing number of thoracic simulators of varying fidelity available, fundamentally, there is currently a significant lack of well-designed trials validating various simulators for teaching lobectomy despite an awareness of their potential to improve surgical education. Furthermore, there is a void in the simulation training of non-technical skills within thoracic surgery. Encouragingly, there is a definite awareness of the ability of simulation to aid with the training of future thoracic surgical trainees.

The effect of verbal feedback, video feedback, and self-assessment on laparoscopic intracorporeal suturing skills in novices: a randomized trial

BACKGROUND

Laparoscopic skill acquisition involves a steep learning curve and laparoscopic suturing is an exceptionally challenging task. By improving the way feedback is given, trainees can learn these skills more effectively. This study aims to establish the most effective form of structured feedback on laparoscopic suturing skill acquisition in novices, by comparing the effects of expert verbal feedback, video review with expert feedback (video feedback), and video review with self-assessment.

METHODS

A prospective randomized blinded trial comparing verbal feedback, video feedback, and self-assessment. Novices in laparoscopic surgery were tasked with performing laparoscopic suturing with intracorporeal knot tying. Time was given for practice, and pre- and post-feedback assessments were undertaken. Suturing performance was measured using a task-specific checklist and global ratings. A post-study questionnaire was used to measure participant-perceived confidence, knowledge, and experience levels.

RESULTS

Fifty-one participants were randomized and allocated equally into the three groups. Performance in all three groups improved significantly from baseline. Video feedback had the largest improvement margin with checklist and global score improvements of 17.1% (± 9.9%) and 14.7% (± 9.3%), respectively. Performance improvements between groups were statistically significant in the global components (p = 0.004) but not the checklist components (p = 0.186). Global score improvement was significantly better in the video feedback group but was statistically insignificant between the self-assessment and verbal feedback groups. Questionnaire responses demonstrated positive results in confidence, knowledge, and experience levels, across all three study groups, with no differences between the groups (p > 0.05).

CONCLUSION

Structured video feedback facilitates reflection and self-directed learning, which improves the ability to develop proficiency in surgical skills. Combining both self-assessment and video feedback may be beneficial over verbal feedback alone due to the advantages of video review. These techniques should therefore be considered for implementation into surgical education curricula.

Team performance training for medical students: Low vs high fidelity simulation

Objectives

The aim of this study is to evaluate a simulation-based team performance course for medical students and compare its low- and high-fidelity components.

Study design

This is a prospective crossover observational study. Groups participated in one low- and one high-fidelity session twice. Low-fidelity scenarios included management of an emergency case on a simulated-patient, whereas high-fidelity scenarios constituted of multiple-trauma cases where simulated-patients wore a hyper-realistic suit. Team performance was assessed objectively, using the TEAM™ tool, and subjectively using questionnaires. Questionnaires were also used to assess presence levels, stress levels and evaluate the course.

Results

Participants’ team performance was higher in the low-fidelity intervention as assessed by the TEAM™ tool. An overall mean increase in self-assessed confidence towards non-technical skills attitudes was noted after the course, however there was no difference in self-assessed performance between the two interventions. Both reported mean stress and presence levels were higher for the high-fidelity module. Evaluation scores for all individual items of the questionnaire were ≥4.60 in both NTS modules. Students have assessed the high-fidelity module higher (4.88 out of 5, SD = 0.29) compared to low-fidelity module (4.74 out of 5, SD = 0.67).

Conclusions

Both the low- and high-fidelity interventions demonstrated an improvement in team performance of the attending medical students. The high-fidelity intervention was more realistic, yet more stressful. Furthermore, it proved to be superior in harvesting leadership, teamwork and task management skills. Both modules were evaluated highly by the students, however, future research should address retention of the taught skills and adaptability of such interventions.

•

The high-fidelity intervention was superior in harvesting leadership, teamwork and task management skills, yet more stressful.

•

We set the group to develop a novel group of team-based OSCE performance.

•

The next step is to adapt such taught skills to different specialties and implement initiatives in the medical school curricula.

Simulation-based acquisition of non-technical skills to improve patient safety

Traditional surgical training has focused on the acquisition of technical skills and knowledge with minimal focus on teaching nontechnical skills. Patient safety depends on both technical and nontechnical skills, with a higher rate of non-technical skills failure leading to patient harm. Many surgical training and regulatory bodies have incorporated nontechnical skills in the required competencies of a surgeon, but few have introduced formal training in nontechnical skills. Emerging research shows simulation-based education to be a powerful tool to teach nontechnical skills to individual surgeons and surgeons in training, and to interprofessional surgical teams with subsequent improvement of patient safety outcomes.

Modern Plastic Surgical Practice: Technical Competence Alone Is Not Enough

Annually, an estimated 234 million major surgical operations occur worldwide, with concomitant seven million complications and one million deaths. It is now well established that technical competence is necessary, but not sufficient for modern surgical practice and outcomes. Breakdown in non-technical skills has been attributed as a key root cause for near misses and patient harm in the operating room. This article discusses the multi-faceted skills-set that is necessary for the modern surgeon to succeed and for optimal patient outcomes. This includes technical skills, non-technical skills, with a focus on key CanMEDS framework domains, including leadership, communication, evidence-based surgery and mentorship.

Simulations and simulators in head and neck endocrine surgery

Simulations and simulators have become an increasingly important tool in trainee education across many surgical disciplines, particularly for robotic and minimally invasive procedures. Thyroidectomy and parathyroidectomy are common procedures performed across multiple surgical disciplines, however, there is limited literature regarding training models/simulators for these operations. This is despite the advent and growing popularity of remote-access thyroidectomy techniques, where simulators may provide significant value in trainee education and safe implementation. Here we review the literature regarding available simulations/simulators in head and neck endocrine surgery for both conventional transcervical approaches and newer remote-access thyroidectomy techniques.

Self-Assessment of Surgical Skills: A Systematic Review

OBJECTIVE

Self-assessment is fundamental in surgical training to enhance learning in the absence of trainer feedback. The primary objective of this review was to assess the factors that influence accuracy of self-assessment at technical skills across all surgical specialties. The secondary objective was to assess whether there are any innate factors or attributes to predict those that will carry out effective self-assessment.

DESIGN

A systematic review was carried out in accordance with PRISMA guidelines. A search strategy encompassing MEDLINE, EMBASE, ERIC, WHO, and the Cochrane database was conducted to identify studies investigating self-assessment at any surgical task. Quality was assessed using the Newcastle-Ottawa scale. A summary table was created to describe specialty, participants, task, setting, assessment tool, and correlation coefficient between self and expert assessment. The review protocol was registered in PROSPERO.

RESULTS

Of 24,638 citations, 40 met inclusion criteria. In total 1753 participants performed 68 procedures. Twenty-six studies investigated skills in general surgery with the remaining 14 in various other surgical specialties. Accuracy of self-assessment is superior in those with greater experience and age, and with use of retrospective video playback. Accuracy tends to be reflected by overestimation of performance. Stressful environments reduce accuracy. There is limited evidence in the literature regarding predicting traits for those who will carry out accurate self-assessment.

CONCLUSIONS

The ability to perform accurate self-assessment is an important skill in surgical training, with accuracy being influenced by a multitude of factors. The use of self-assessment from retrospective video playback may be of benefit in surgical training curricula to enhance learning of technical skills. Further studies are required to define predictors of good self-assessment, which will strengthen recruitment and mentoring to assist trainee learning.

Getting better all the time? Facilitating accurate team self-assessments through simulation

Introduction

Our study explores the extent to which teams are accurate assessors of their own performance and teamwork, and how simulation can help this critical skill develop over time.

Methods

Surgery residents in teams of three completed five daily simulations. After each scenario, each team reviewed their performance and jointly completed a scenario-specific team performance evaluation and a 17-item Communication and Teamwork Skills tool. Videos were rated to obtain discrepancy values. Paired-samples t-tests and mean comparisons were used to examine changes in team self-assessment accuracy and comparisons between high-performing and low-performing teams.

Results

Resident (n=30) teams rated team performance higher than faculty across the first 3 days (p<0.01), but provided similar ratings thereafter. Agreement of team performance from day 1 to 5 significantly improved (p<0.001). Teams rated their teamwork higher than faculty across all days (p<0.01). Top performing teams provided more accurate self-assessments for both teamwork (average discrepancy 8% vs 39%) and team performance (average discrepancy 12% vs 23%).

Conclusion

Teams that continue to work together over time may become more accurate judges of their own performance, but do not become more accurate assessors of teamwork competencies.

Self-assessment of Skills by Surgeons and Anesthesiologists After a Trauma Surgery Masterclass

BACKGROUND

In the Netherlands, each year a three-day international multidisciplinary trauma masterclass is organized to provide the knowledge and skills needed to care for critically injured trauma patients. This study was designed to longitudinally evaluate the effect of the course on participant's self-assessment of their own ability and confidence to perform general and specific skills.

METHODS

Between 2013 and 2016, all participants were invited to complete a questionnaire before and during follow-up. Participants were asked to self-assess their level of confidence to perform general skills (communication, teamwork, leadership) and specific skills. Mean scores were calculated, and mixed models were used to evaluate correlation.

RESULTS

We asked 265 participants to participate. Response rate was 64% for the pre-questionnaire, 63% for the post-questionnaire and for 3 months, 1 year and 2 years, respectively, 40%, 30%, 20%. The surgical group showed a statistically significant increase in self-assessed confidence for general skills (3.82-4.20) and specific technical skills (3.01-3.83; p < 0.001). In the anesthetic group, self-assessed confidence increased significantly in general skills (3.72-4.26) and specific technical skills (3.33-4.08; p < 0.001). For both groups statistical significance remained during follow-up.

CONCLUSIONS

This study demonstrated a sustained positive effect of a dedicated multidisciplinary trauma training curriculum on participant's self-assessed confidence to perform both general and specific technical skills necessary for the care of injured patients. Given the known association between confidence and competence, these findings provide evidence that dedicated trauma training curricula can provide positive lasting results.

LEVEL OF EVIDENCE

This is a basic science paper and therefore does not require a level of evidence.

Influence of video-based feedback on self-assessment accuracy of endoscopic skills: a randomized controlled trial

Background and study aims  Novice endoscopists are inaccurate in self-assessment of procedures. One means of improving self-assessment accuracy is through video-based feedback. We aimed to determine the comparative effectiveness of three video-based interventions on novice endoscopists' self-assessment accuracy of endoscopic competence. Materials and methods  Novice endoscopists (performed < 20 previous procedures) were recruited. Participants completed a simulated esophagogastroduodenoscopy (EGD) on a virtual reality simulator. They were then randomized to one of three groups: self-video review (SVR), which involved watching a recorded video of their own performance; benchmark review (BVR), which involved watching a video of a simulated EGD completed by an expert; and self- and benchmark video (SBVR), which involved both videos. Participants then completed two additional simulated EGD cases. Self-assessments were conducted immediately after the first procedure, after the video intervention and after the additional two procedures. External assessments were conducted by two experienced endoscopists, who were blinded to participant identity and group assignment through video recordings. External and self-assessments were completed using the global rating scale component of the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT GRS). Results  Fifty-one participants completed the study. The BVR group had significantly improved self-assessment accuracy in the short-term, compared to the SBVR group ( P  = .005). The SBVR group demonstrated significantly improved self-assessment accuracy over time ( P  = .016). There were no significant effects of group or of time for the SVR group. Conclusions  Video-based interventions, particularly combined use of self- and benchmark video review, can improve accuracy of self-assessment of endoscopic competence among novices.

The relevance of low-fidelity virtual reality simulators compared with other learning methods in basic endovascular skills training

OBJECTIVE

The use of simulators has shown a profound impact on the development of both training and assessment of endovascular skills. Furthermore, there is evidence that simulator training is of great benefit for novice trainees. However, there are only a few simulators available geared specifically toward novice learners. Whereas research suggests that low-fidelity simulators could fill this gap, there are insufficient data available to determine the role of low-fidelity simulators in the training of endovascular skills.

METHODS

Medical students in their fifth year (N = 50) with no previous endovascular experience were randomized into three groups: conventional learning through a video podcast (group V; n = 12), low-fidelity simulation training with tablet-paired touch-gesture navigation (group A; n = 12), and low-fidelity simulation training with tablet-paired physical endovascular tool navigation (group S; n = 26). Within their respective groups, all students attended a 1-day class on basic endovascular skills. Questionnaire items for self-assessment before and after the class and assessment after the class of the participant's practical skills on a high-fidelity simulator were analyzed across all three groups as well as for each group separately using nonparametric tests.

RESULTS

All 50 participants completed the training. Participants in group S showed a significantly increased interest in working in interventional cardiology (P = .02) and vascular surgery (P = .03) after the class. Evaluation of the questionnaire items after the class showed that participants in group S rated their practical skills significantly higher after the class compared with those in group V and group A (P < .001 for pairwise comparison of all three groups, respectively), creating a significant trend across the three groups. However, analysis of the practical skills assessment for all three groups showed a significant difference between the groups only for choosing a guidewire (P = .045) and a significant trend in performance across the groups for choosing a guidewire and for positioning the guidewire in the vessel (P = .02 and P = .05, respectively). All other steps of the skills assessment showed no significant differences or a trend across the groups.

CONCLUSIONS

Low-fidelity simulation training, particularly with physical endovascular tool navigation, led to increased motivation in novice trainees. Whereas simulator training was associated with increased confidence of trainees in their skills, assessment of their practical skills showed no actual improvement in this study. Overall, low-fidelity simulation has the potential to benefit novice trainees, but possible risks of simulation training should be further evaluated.

Non-technical attributes and surgical experience: A cross-sectional study comparing communication styles and attitudes in surgical staff, trainees and applicants

BACKGROUND

This monocentric study aimed to explore whether key non-technical attributes can be reliably measured in a mixed population of candidates applying for surgical training, surgical trainees and staff and to identify any differences between these groups.

MATERIALS & METHODS

Candidates applying for surgical training, surgical trainees and staff from four surgical specialties (general surgery, orthopedics, plastic surgery or urology) at a tertiary academic teaching hospital were all sent an online self-report questionnaire. The Communication Styles Inventory (CSI, 96 items) was used to assess a six-dimensional behavioral model of participant communication styles (expressiveness, preciseness, verbal aggressiveness, questioningness, emotionality and impression manipulativeness). Attitudes toward uncertainty and risks were assessed with the Physicians' Reaction toward Uncertainty (PRU, 15 items) and Physician Risk Attitudes (PRA, 6 items) scales respectively. Data was encoded and analyzed using parametric testing.

RESULTS

The questionnaire was completed by 177 participants (110 candidates; 42 trainees; 25 staff). All scales had very good internal consistency (Cronbach's alpha >0.80). After controlling for gender-based differences, surgical candidates scored significantly higher on 'expressiveness' (P = 0.012) and were significantly less risk-averse (P = 0.006) than trainees and staff. Surgical trainees scored lowest on the CSI 'questioningness' subscale (P = 0.019) and had significantly more difficulties dealing with uncertainty, characterized by their highest scores on the 'concern about bad outcome' (P = 0.021) and reluctance to disclose uncertainty to patients' (P = 0.05) subscales. Multiple subscales revealed gender-based differences in candidate and trainee groups, which were not noted for surgical staff.

CONCLUSIONS

Meaningful differences in non-technical attributes of surgical staff, trainees and candidates have been identified, which may be explained by differences in clinical experience and learning and may suggest that these develop over time. Further research on assessment of non-technical attributes during surgical selections and the role of both technical and non-technical attributes in surgery at large is needed.

An interprofessional training program for intrahospital transport of critically ill patients: model build-up and assessment

Intrahospital transport of critically ill patients for diagnostic or therapeutic procedures can be compromised by patient instability, equipment problems or inexperienced teamworking. This quasi-experimental study aimed to assess the effectiveness of an in-situ interprofessional simulation-based training (IIST) model for junior member transport teams. Newly registered postgraduate physicians, nurses and respiratory therapists underwent the IIST. The technical skills (TS) of each participant and non-technical skills (NTS) of each interprofessional team were assessed using well-validated checklists. Thirty-six participants enrolled and were randomly assigned to six experimental and six control teams. Most participants achieved a significantly higher level of both TS and NTS. Both the control and experimental teams overvalued their NTS in the pretest, while the posttest self-assessment scores among the experimental groups more closely matched the expert assessments. Despite challenges in scheduling and the setting, the IIST was successfully conducted in a crowded hospital, which enabled trainees to optimize their learning in a real-life environment. In conclusion, the IIST model can facilitate the development of both TS and NTS for transport team members. Transport teams made up of newly registered staff from different disciplines may lack insight into their NTS in critical patient transfer management, but simulation training may cause improvements.

Nontechnical Skills in Surgery: A Systematic Review of Current Training Modalities

BACKGROUND

The complexity of the operating room requires a surgeon to have both technical ability and an array of nontechnical skills. The emphasis on technical skills during surgical training is well established, however it is deficiencies in nontechnical skills that have been identified as the main cause of errors in the operating room.

OBJECTIVE

This systematic review aims to identify current methods used to teach nontechnical skills and how these methods are assessed to determine their validity, evidence, and role in training.

METHODS

MEDLINE and Embase databases were searched for English language articles between 2000 and 2017 for nontechnical surgical skills training. Original research articles were included if they described non-technical surgical skills training modalities and their assessment. Results were assessed for the level of evidence and each modality was awarded a level of recommendation, using a modified educational Oxford Centre for Evidence-Based Medicine classification, as adapted by the European Association of Endoscopic Surgery.

RESULTS

A total of 19 studies were identified pertaining to high fidelity simulation (n = 8), low fidelity simulation (n = 6), didactic teaching (n = 2) and crisis resource management (n = 3). Of the included studies 1 was classified Level 1b, 1 level 2b, 7 level 2b, 2 level 2c, and 8 level 3.

CONCLUSION

With the importance of nontechnical skills being increasingly recognized, it is essential for surgeons to receive adequate training in nontechnical skills. Therefore the most valuable teaching modalities such as high and low fidelity simulation needs to be implemented into surgical training curricula.

Development and validation of an objective assessment scale for chest tube insertion under 'direct' and 'indirect' rating

BACKGROUND

There is an increasing need for objective and validated educational concepts. This holds especially true for surgical procedures like chest tube insertion (CTI). Thus, we developed an instrument for objectification of learning successes: the assessment scale based on Objective Structured Assessment of Technical Skill (OSATS) for chest tube insertion, which is evaluated in this study. Primary endpoint was the evaluation of intermethod reliability (IM). Secondary endpoints are 'indirect' interrater reliability (IR) and construct validity of the scale (CV).

METHODS

Every participant (N = 59) performed a CTI on a porcine thorax. Participants received three ratings (one 'direct' on site, two 'indirect' via video rating). IM compares 'direct' with 'indirect' ratings. IR was assessed between 'indirect' ratings. CV was investigated by subgroup analysis based on prior experience in CTI for 'direct' and 'indirect' rating.

RESULTS

We included 59 medical students to our study. IM showed moderate conformity ('direct' vs. 'indirect 1' ICC = 0.735, 95% CI: 0.554-0.843; 'direct' vs. 'indirect 2' ICC = 0.722, 95% CI 0.533-0.835) and good conformity between 'direct' vs. 'average indirect' rating (ICC = 0.764, 95% CI: 0.6-0.86). IR showed good conformity (ICC = 0.84, 95% CI: 0.707-0.91). CV was proven between subgroups in 'direct' (p = 0.037) and 'indirect' rating (p = 0.013).

CONCLUSION

Results for IM suggest equivalence for 'direct' and 'indirect' ratings, while both IR and CV was demonstrated in both rating methods. Thus, the assessment scale seems a reliable method for rating trainees' performances 'directly' as well as 'indirectly'. It may help to objectify and facilitate the assessment of training of chest tube insertion.

Comparing self-assessment of laparoscopic technical skills with expert opinion for gynecological surgeons in an operative setting

Background

Competence in laparoscopic skills is important for all gynaecological surgeons. Most residency programmes teach technical skills in the operating room and through lectures, where the evaluation of surgical skills is usually done through subjective evaluation. After graduating residency, most surgeons depend on themselves to decide if they are competent in performing a certain procedure. The objective of this study is to evaluate the accuracy of surgeon self-assessment compared with expert assessment of competence in laparoscopic surgical skills. A double-blind prospective cohort study was undertaken at Prince Hamza Hospital between January 2016 and April 2016 in Amman, Jordan. Eight practicing gynecologists and obstetricians performed and recorded 88 laparoscopic procedures including ovarian cystectomy, salpingectomy for ectopic pregnancy, salpingoophorectomy, resection of endometriosis, adhesiolysis and ovarian drilling. Participating gynecologists recorded the procedures and were asked to complete a Global Rating Index of Technical Skills (GRITS) evaluation after the surgery testing across multiple areas with a lowest score of 8 and a highest score of 40. Two well-versed laparoscopic experts in objective structured assessment of technical skills (OSATS) also independently scored all procedures using the same parameters. The correlation coefficient and internal consistency were calculated.

Results

The GRITS score was calculated for each participant with a mean assessment score of 3.47 for each parameter. Participants self-assessment scores were significantly higher than expert assessment scores (p<0.05). The correlation coefficient was calculated and it can be seen that there was high inter-expert correlation in assessment across all participants evaluations (ICC > 0.90).

Conclusion

Self-assessment of surgical laparoscopic skills is higher than expert evaluation of these technical skills. Quality assurance measures need to be revisited and restructured through more frequent assessments using peer and expert assessment alongside self-assessment. Gynecologists also need to undergo proper assessment prior to starting independently performing procedures that require new skills.

The Montreal Augmentation Mammaplasty Operation (MAMO) Simulator: An Alternative Method to Train and Assess Competence in Breast Augmentation Procedures

BACKGROUND

Surgical residents' exposure to aesthetic procedures remains limited in residency training. The development of the Montreal augmentation mammaplasty operation (MAMO) simulator aims to provide an adjunctive training method and assessment tool to complement the evolving competency-based surgical curriculum.

OBJECTIVES

To perform face, content, and construct validations of the MAMO simulator for subpectoral breast augmentation procedures and assess the reliability of the assessment scales used.

METHODS

Plastic surgery staff and residents were recruited to perform a subpectoral breast augmentation on the simulator. Video recordings of their performance were blindly evaluated using the objective structured assessment of technical skills (OSATS) system consisting of the global rating scale (GRS), mammaplasty objective assessment tool (MOAT), and a surgery-specific Checklist score.

RESULTS

Fourteen plastic surgery residents and seven expert plastic surgeons were enrolled. Experts' performance was significantly higher than residents' according to each of GRS, MOAT, and Checklist scores. Mean values of residents and experts were 23.4 (2.5) vs 36.9 (3.1) (P < 0.0001) for GRS score, 30.4 (2.2) vs 40 (3.2) (P < 0.0001) for MOAT scores, and 9.7 (1.5) vs 12 (1) (P < 0.001) for Checklist scores, respectively. Face and content validations showed excellent results among parameters evaluated, with an overall mean score of 4.8 (0.3) on 5. Cronbach's alpha was 0.96 and 0.83 for GRS and MOAT scores, respectively. Intraclass correlation coefficients for interrater reliability were excellent at 0.93, 0.92, and 0.89 for the GRS, MOAT, and Checklist scores, respectively.

CONCLUSIONS

This study proves the construct simulator to be valid and the assessment scales to be reliable.

"Reflection-Before-Practice" Improves Self-Assessment and End-Performance in Laparoscopic Surgical Skills Training

OBJECTIVE

To establish whether a systematized approach to self-assessment in a laparoscopic surgical skills course improves accordance between expert- and self-assessment.

DESIGN

A systematic training course in self-assessment using Competency Assessment Tool was introduced into the normal course of evaluation within a Laparoscopic Surgical Skills training course for the test group (n = 30). Differences between these and a control group (n = 30) who did not receive the additional training were assessed.

SETTING

Catharina Hospital, Eindhoven, The Netherlands (n = 27), and GSL Medical College, Rajahmundry, India (n = 33).

PARTICIPANTS

Sixty postgraduate year 2 and 3 surgical residents who attended the 2-day Laparoscopic Surgical Skills grade 1 level 1 curriculum were invited to participate.

RESULTS

The test group (n = 30) showed better accordance between expert- and self-assessment (difference of 1.5, standard deviation [SD] = 0.2 versus 3.83, SD = 0.6, p = 0.009) as well as half the number (7 versus 14) of cases of overreporting. Furthermore, the test group also showed higher overall mean performance (mean = 38.1, SD = 0.7 versus mean = 31.8, SD = 1.0, p < 0.001) than the control group (n = 30). The systematic approach to self-assessment can be viewed as responsible for this and can be seen as "reflection-before-practice" within the framework of reflective practice as defined by Donald Schon.

CONCLUSION

Our results suggest that "reflection-before-practice" in implementing self-assessment is an important step in the development of surgical skills, yielding both better understanding of one's strengths and weaknesses and also improving overall performance.

Impact of experience on self-assessment accuracy of clinical colonoscopy competence

BACKGROUND AND AIMS

Self-assessment is important for life-long learning and a recommended assessment method for endoscopy skills. Prior literature has not investigated self-assessment accuracy of colonoscopic competence in the clinical setting. This study aimed to determine the self-assessment accuracy of novice, intermediate, and experienced endoscopists.

METHODS

Novice (performed <50 previous colonoscopies), intermediate (50-500), and experienced (>1000) endoscopists from 5 hospitals each performed a clinical colonoscopy. Video recordings of procedures were independently assessed by 2 blinded expert endoscopists by using the Gastrointestinal Endoscopy Competency Assessment Tool (GiECAT). Externally assessed and self-assessed GiECAT scores were defined as the mean of the 2 video-based ratings and as participants' own assigned ratings, respectively. Self-assessment accuracy between the externally assessed and self-assessed scores was evaluated by using absolute difference scores, intraclass correlation coefficients, and the Bland-Altman analysis.

RESULTS

Twenty novice, 10 intermediate, and 10 experienced endoscopists participated. There was moderate agreement of externally assessed and self-assessed GiECAT scores, with an intraclass correlation coefficient of 0.65 (95% confidence interval, 0.44-0.80). The absolute difference scores among the 3 groups were significantly different (P = .002), with experienced endoscopists demonstrating a more accurate self-assessment ability compared with novices (P = .002). Bland-Altman plots suggest that novice and experienced endoscopists tend to overrate and underrate their clinical competence, respectively; no specific trends were associated with intermediates.

CONCLUSION

Participants demonstrated moderate self-assessment accuracy of clinical competence. Endoscopist experience was positively associated with self-assessment accuracy; novices demonstrated lower self-assessment accuracy compared with experienced endoscopists. Moreover, novices tended to overestimate their performances. Novice endoscopists may benefit from targeted interventions to improve self-assessment accuracy.