Correcting for Rater Effects in Operating Room Surgical Skills Assessment

OBJECTIVE

To estimate and adjust for rater effects in operating room surgical skills assessment performed using a structured rating scale for nasal septoplasty.

METHODS

We analyzed survey responses from attending surgeons (raters) who supervised residents and fellows (trainees) performing nasal septoplasty in a prospective cohort study. We fit a structural equation model with the rubric item scores regressed on a latent component of skill and then fit a second model including the rating surgeon as a random effect to model a rater-effects-adjusted latent surgical skill. We validated this model against conventional measures including the level of expertise and post-graduation year (PGY) commensurate with the trainee's performance, the actual PGY of the trainee, and whether the surgical goals were achieved.

RESULTS

Our dataset included 188 assessments by 7 raters and 41 trainees. The model with one latent construct for surgical skill and the rater as a random effect was the best. Rubric scores depended on how severe or lenient the rater was, sometimes almost as much as they depended on trainee skill. Rater-adjusted latent skill scores increased with attending-estimated skill levels and PGY of trainees, increased with the actual PGY, and appeared constant over different levels of achievement of surgical goals.

CONCLUSION

Our work provides a method to obtain rater effect adjusted surgical skill assessments in the operating room using structured rating scales. Our method allows for the creation of standardized (i.e., rater-effects-adjusted) quantitative surgical skill benchmarks using national-level databases on trainee assessments.

LEVEL OF EVIDENCE

N/A Laryngoscope, 2024.

Merits of simulation-based education: A systematic review and meta-analysis

BACKGROUND

The drive to improve surgical proficiency through advanced simulation-based training has gained momentum. This meta-analysis systematically evaluated evidence regarding the impact of plastic surgery-related simulation on the performance of residents.

METHODS

A systematic search of PubMed, Web of Science, and Cochrane Library and review of articles was performed following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis protocol. An inverse-variance random-effects model was used to combine study estimates to account for between-study variability. Objective structured assessment of technical skills (OSATS) scores and subjective confidence scores were used to assess the impact of the simulation with positive changes from the baseline indicating better outcomes.

RESULTS

Eighteen studies pooling 367 trainees who participated in various simulations were included. Completion of simulation training was associated with significant improvement in subjective confidence scores with a mean increase of 1.44 units (95% CI: 0.93 to 1.94, P < 0.001), and in OSATS scores, with a mean increase of 1.24 units (95% CI: 0.87 to 1.62, P < 0.001), both on a 1-5 scale. Participants reported high satisfaction scores (mean = 4.76 units, 95% CI = 4.61 to 4.91, P = 0.006), also on a 1-5 scale.

CONCLUSIONS

Participation in surgical simulation markedly improved objective and subjective scoring metrics for surgical trainees. Several simulation devices are available for honing surgical skills, with the potential for advancements. The evidence demonstrates the effectiveness of simulations; thus, incorporating simulation into training curricula should be a priority in the field of plastic surgery.

French translation and validation of the OSATS tool for the assessment of surgical skill

AIM OF THE STUDY

The OSATS rating scale is one of the tools most widely used in the scientific literature for assessing technical surgical skill. The aim of this study was to translate this scale into French (OSATS-FR) and validate it.

METHODS

OSATS was translated into French by two independent translators. The two translations were pooled and back-translated by a third translator into English to produce a consensus version (OSATS-FR). Three independent assessors then used it to assess surgical skill in tracheostomy videos. The reproducibility of the scoring was measured.

RESULTS

The OSATS-FR version was easily understood and used by all three assessors. In all, 18 procedures were assessed, performed by 14 interns and 4 senior surgeons. The mean OSATS-FR score was 18.6±4.08 for the interns and 31.6±1.62 for the seniors (p<0.0001). The Pearson correlation coefficient was 0.937 CI95% [0.867-1.269] between assessors 1 and 2, 0.977 CI95% [0.899-1.149] between assessors 1 and 3, and 0.965 CI95% [0.876-1.196] between assessors 2 and 3. Cronbach's alpha was greater than 0.9 for each assessor.

CONCLUSION

OSATS-FR can distinguish between two groups with different surgical expertise and assess surgical skill with an inter-assessor reproducibility and internal consistency comparable to those of the English version.

Assessment of suturing and scaling skills of periodontology and oral medicine residents by OSATS method: a pilot study

INTRODUCTION

Updating the method for evaluating suturing and scaling skills in dental education has attracted relatively little attention and there is no consensus to what should be assessed and how. The purpose of this study was to investigate the applicability of the Objective Structured Assessment of Technical Skill (OSATS) method for these two basic skills, the possible association between the scores and demographic factors, and the level of satisfaction of residents with this method.

METHODS

All six periodontics and three oral medicine residents were recruited by census method and video-recorded while performing a simple interrupted suture, a figure eight suture and scaling on a model. Then, the videos were evaluated independently via a checklist and a global rating scale (GRS) by two expert raters. Agreement between raters and residents' satisfaction were evaluated. Correlation between demographic factors of participants and scores was also assessed. T-test and linear regression analysis were used.

RESULTS

There was no significant difference between the scores based on the views of the two raters for each of the checklist (ICC = 0.99, CI = 0.96-0.99, P < 0.001) and GRS (ICC = 0.97, CI = 0.86-0.99, P < 0.001). Linear regression showed no correlation between gender and scores but periodontics major and higher year of education showed correlation with higher scores.

CONCLUSION

Considering the excellent agreement between raters in using both the checklist and GRS components of OSATS, and satisfaction of 88% the residents with this method, it seems to be able to provide a reliable assessment.

Influence of surgical performance on clinical outcome after osteosynthesis of distal radius fracture

INTRODUCTION

Clinical outcome after surgery depends on the surgeon's level of expertise or performance. The present study of minimally invasive plate osteosynthesis (MIPO) with anterior plate for distal radius fracture assessed whether clinical outcome correlated with surgeon performance.

METHODS

The series included 30 distal radius fractures: 15 operated on by 4 level III surgeons (Group 1) and 15 by 4 level V surgeons (Group 2), utilizing the MIPO technique. The surgical performance of all 8 surgeons was assessed using the OSATS global rating scale. Clinical outcomes were assessed at 3 months' follow-up using the modified Mayo score (MMS), in 4 grades: 0-64 (poor), 65-79 (moderate), 80-89 (good), and 90-100 (excellent). The QuickDASH score (QDASH) was also calculated, and complications were recorded.

RESULTS

Median MMS was better for level V (75 = fair result) than level III surgeons (62 = poor result). Median QDASH score likewise was better in group 2 (9.1) than group 1 (22.7). In group 1, there were 2 paresthesias in the median nerve territory, 1 type-1 complex regional pain syndrome, and 1 hypoesthesia in the scar area. Mean correlation between the 2 scores was -0.68. Group 1 patients were on average 7 years older. The number of patients, number of surgeons and distribution of OA A and C fractures were almost identical in the two groups. On MMS, the overall result of the two groups was moderate (70.5), which can be explained by short mean follow-up.

DISCUSSION

Quality of the clinical outcome on MMS and QDASH increased with surgical performance, with fewer complications. In the patients' interest, protocols for improving surgical performance should be implemented, for example, through deliberate practice.

Developing an Objective Structured Assessment of Technical Skills (OSATS) for Microlaryngoscopy

BACKGROUND

Microlaryngoscopy is a basic technical skill in Oto-HNS. It is essential for residency programs to have a competency-based assessment tool to evaluate residents' performance of this procedure. An Objective Structured Assessment of Technical Skills (OSATS) is a procedure-specific assessment, which consists of the following: (a) Operation-Specific Checklist and (b) Global Rating Scale (GRS).

OBJECTIVE

The objective of this study was to create an OSATS for adult microlaryngoscopy.

METHODS

This was a prospective study, with an initial qualitative phase for OSATS development (Phase I), and a clinical pilot phase (Phase II). In Phase I, interviews were conducted with three laryngologists to establish a stepwise description of adult microlaryngoscopy and review a previously validated GRS for relevance to microlaryngoscopy. Responses were used to create a framework for the OSATS. The OSATS was then presented to Oto-HNS residents and laryngologists in an alternating fashion, for review of clarity and relevance. A pilot study was then performed to evaluate the resident performance of adult microlaryngoscopy. Multiple regression analysis was carried out to investigate whether training level, case complexity, and previous OSATS exposure could predict participant scores.

RESULTS

Phase I of this study led to the creation of a 34-item OSATS. The pilot study (N = 28 procedures) revealed that training level was significantly correlated with increased OSATS scores. There was no statistically significant correlation between case complexity and resident scores. Assessors reported the perceived utility of the OSATS and intent for use in residency training.

CONCLUSION

Application of the proposed OSATS will allow for competency-based assessment of the resident performance of microlaryngoscopy.

LEVEL OF EVIDENCE

NA Laryngoscope, 133:2719-2724, 2023.

Learning surgical knot tying and suturing technique - effects of different forms of training in a controlled randomized trial with dental students

Objective

The acquisition of surgical skills requires motor learning. A special form of this is intermanual transfer by transferring motor skills from the nondominant hand (NDH) to the dominant hand (DH). The purpose of this study was to determine the learning gains that can be achieved for the DH by training with the DH, the NDH, and by non-surgical alternative training (AT).

Methods

124 preclinical (n=62) and clinical (n=62) dental students completed surgical knot tying and suturing technique training with the DH, with the NDH, and an AT in a controlled randomized trial.

Results

A statistically significant learning gain in knot tying and suture technique with the DH was evident only after training with the DH when compared to training with the NDH (p<0.001 and p=0.004, respectively) and an AT (p=0.001 and p=0.010, respectively). Of those students who achieved a learning gain ≥4 OSATS points, 46.4% (n=32) benefited in their knot tying technique with the DH from training with the DH, 29.0% (n=20) from training with the NDH, and 24.6% (n=17) from an AT while 45.7% (n=32) benefited in their suturing technique with the DH from training with the DH, 31.4% (n=22) from training with the NDH, and 22, 9% (n=16) from an AT.

Conclusions

Training with the DH enabled significantly better learning gains in the surgical knot tying and suturing techniques with the DH.

An Innovative Technique of Microsurgical Training on Fresh "Chicken Quarter" Model: Our Experience

Purpose  Regular practice, quality clinical exposure, and academic discussion are essential in any surgical specialty training. This study discusses and validates the option of using a fresh "chicken quarter" model with a measurable scoring system, as a standard training regimen in microvascular surgery. This can be a very effective, economical, and easily accessible model for residents. Materials and Methods  This study was conducted in the Department of Plastic surgery, from October 2020 to May 2021. Twenty-four fresh "chicken quarter" specimens were dissected and the ischial arteries and femoral veins' external diameter (ED) were measured. The microsurgical skills of the trainee were assessed in 6 months intervals using the Objective Structured Assessment of Technical Skills Scale (OSATS) as well as the time taken for anastomosis. All the data were analyzed using SPSS (statistical package for social sciences) version 21. Results  A task-specific score value of 50% on October 2020 improved to 85.7% by May 2021. This was found to be statistically significant ( p  = 0.043). The mean ED of the ischial artery and femoral vein was 2.07 and 2.26 mm, respectively. The mean width of the vein measured at the lower one-third of the tibia was 2.08 mm. A greater than 50% reduction in anastomosis time was observed after a period of 6 months. Conclusion  In our minimal experience, the "chicken quarter model" with OSATS scoring system seems to be effective, economical, very affordable, and easily accessible microsurgery training model for the residents. Our study is done only as a pilot project due to limited resources and we have the plan to introduce it as a proper training method in the near future with more residents.

Development and preliminary validation of an Objective Structured Assessment of Technical Skills (OSATS) for a partial pulpotomy procedure

AIM

To develop an Objective Structured Assessment of Technical Skills (OSATS) tool for partial pulpotomy and to preliminary assess the validity of this tool. Secondary objectives were to check internal consistency as well as the interrater reliability of this specific checklist and to compare it with the previously validated global rating score (GRS).

METHODOLOGY

The study assessed three groups of dental participants (mix of practitioners or students) possessing differing levels of clinical experience (7 novices, 7 intermediates, 7 experts), during a partial pulpotomy procedure on a simulated tooth with deep caries, using a bespoke procedure-specific checklist (PSC) and the GRS. Two independent examiners received the study participants in groups of two, in order to be able to directly observe their actions and grade the different steps of the procedure from 1 to 5 on the PSC and GRS. The internal validity of the specific checklist was assessed using a Cronbach's alpha test. As the construct validity of such a tool can be determined when the tool differentiates performance based on the level of experience, the total score of the PSC of the three groups (e.g. novice group, intermediate group, experienced group) were compared using Kruskal-Wallis and post-hoc Mann-Whitney pairwise U test. The total scores of the GRS were also analysed in the same manner. A correlation test (Correlation Matrices Test) was carried out for the entire sample between the specific checklist and the GRS (on total score). The intraclass correlation coefficient was used to measure the degree of agreement between the two evaluators. The statistical analysis was performed with XLSTAT® and Statistica® (significance p = 0.05).

RESULTS

The partial pulpotomy specific checklist graded from 1 to five demonstrated good internal consistency (0.86) and a good interrater reliability (0.91), a correct construct validity (p generally <0.05) and a good positive correlation with the validated GRS (r=0.92).

CONCLUSIONS

To address the need to develop education in vital pulp treatment, and more specifically the technical skills assessment of partial pulpotomy, an OSATS specific checklist was developed and preliminary validated. The checklist will need to be validated in larger cohorts.

Methods of assessment of the level of surgeons for minimally invasive fixation of distal radius fractures: "expertise" versus "performance"

INTRODUCTION

The results of surgical techniques vary depending on the level of the surgeon's experience. The level of expertise can be evaluated quantitatively or qualitatively. In this study, we evaluated the duration of the procedure and the size of the incision of minimally invasive fixation of distal radius fractures as a means of grading expertise and surgical performance. The null hypothesis of our study was that the level of expertise did not match the level of performance.

MATERIAL AND METHODS

The study included 52 surgeons split in five levels of expertise who had performed 467 minimally invasive fixations of distal radius fractures. The performance of the surgeons in terms of duration of the procedure and size of the incision were scored according to the five levels of surgical experience using four different methodologies: clinical (setting limits compared to data found in literature), statistical (setting statistical limits of comparison in percentiles (20th, 40th, 60th, 80th) for each of the two parameters measured), arithmetical (setting limits compared to equal intervals for each of the 2 variables), and success rate (setting a threshold for each of the 2 variables).

RESULTS

Our results showed a great disparity between levels of experience depending on the method used. The scores for levels 1 and 2 were 72% for expertise, 13% for performance according to the clinical method, 75% for the statistical method, 0% for the arithmetical method, and 57% according the success rate. The rate of level 3 was 23% for expertise, 41% for performance by the clinical method, 17% by the statistical method, 17% by the arithmetical method, and 15% by the success rate. The rates of levels 4 and 5 were 5% for expertise, 46% for performance clinical method, 8% by the statistical method, 83% by the arithmetical method, and 28% by the success rate.

DISCUSSION

The null hypothesis of our study was confirmed by the results which show that there was no correlation between levels of expertise and performance. The clinical method of assessment appears to reflect best the true level of performance of the surgeon.

CONCLUSION

Publications reporting the results of a surgical technique for a given pathology should always mention the level of performance as measured on a clinical scale.

Telestration in the Teaching of Basic Surgical Skills: A Randomized Trial

OBJECTIVE

To evaluate how an affordable course using telestration with augmented reality can be compared to the traditional teaching of basic surgical skills.

DESIGN

Prospective, randomized and blinded study.

SETTING

Faculty of Medicine of Porto University.

PARTICIPANTS AND METHODS

Twenty medical students without any experience in basic surgical skills were randomized into two different learning groups: telestration and traditional teaching (on-site mentoring) groups. Five different types of sutures were taught: the single interrupted, the cruciate mattress, the horizontal mattress, the vertical mattress and the simple continuous sutures. Data was obtained on the time taken to learn each of the techniques and to perform each exercise without any support from the faculty, tension of the suture, quality of the procedure using a modified Objective Structured Assessment of Technical Skills and participants' answers to a Likert questionnaire in terms of their learning experience, confidence, and self-evaluation.

RESULTS

Trainees in the telestration group were globally faster when performing independently (1393.40 [SD 288.89] vs 1679.00 [SD 328.22] seconds, p = 0.04) particularly during the cruciate mattress suture (235.50 [SD 61.81] vs 290.00 [SD 68.77] seconds, p = 0.05) and the simple continuous suture (492.40 [SD 87.49] vs 630.30 [SD 132.34] seconds, p = 0.01).Time needed for students to learn the procedures was similar between the groups. There were also no statistically significant differences in terms of the quality of the surgical gesture, tension of the suture, self-evaluation or confidence.

CONCLUSIONS

A basic surgical skills course using telestration through a head-mounted device with augmented reality capabilities can be a viable alternative to traditional teaching, considering time and quality of the gesture. Though costs can discourage from using this technology in basic procedures, the use of free software may turn it into an affordable option in the context of distant learning.

Anterior plating technique for distal radius: comparing performance after learning through naive versus deliberate practice

BACKGROUND

Surgical teaching is most often carried out in the operating theatre through mentorship, and the performance of surgical procedures is rarely measured. The objective of this article is to compare the progression in learning curves of junior surgeons trained in the anterior plating technique for the distal radius on a nonbiological model according to three different methods.

METHODS

The materials comprised 12 junior surgeons of level 1 or 2 (as per Tang and Giddins) divided into three groups: control (G1), naive practice (G2), and deliberate practice (G3). The three groups watched a demonstration video of a level 5 expert. The four G1 surgeons (two level 1 and two level 2) saw the video only once, and each inserted five plates. The four G2 surgeons (two level 1 and two level 2) inserted five plates and watched the video before each time. The four G3 surgeons (two level 1 and two level 2) saw the video before the first plate insertion. Before posing the subsequent four plates, the four G3 surgeons watched their own video, and the expert indicated their errors and how to avoid them next time. A 12-criteria OSATS defined on the basis of the 60 videos, each graded from one (min.) to five (max.), was used to measure the objective surgical performance per plating (min. 12; max. 60) and per series of five plate fixations (min. 60, max. 300).

RESULTS

The total average objective performance of G1 was 44.73, of G2 was 50.57 and of G3 was 54.35. Change in objective performance was better for G3 (13.25) than G2 (5) or G1 (3.75). For all groups, the progression in objective performance was better amongst level 1 surgeons (9) than level 2 surgeons (5.6).

CONCLUSION

Surgical teaching is based on mentorship and experience. However, since "see one, practice many, do one" has started to replace "see one, do one, teach one", learning techniques have increasingly relied on procedure simulators. Against this background, few studies have looked at measuring the performance of surgical procedures and improved learning curves. Our results appear to suggest that deliberate practice, when used in addition to mentorship, is the best option for shortening the growth phase of the learning curve and improving performance. Deliberate practice is a learning technique for surgical procedures that is complementary to mentorship and experience, which allows the growth phase of the learning curve to be shortened and the objective performance of junior surgeons to be improved.

Performance with robotic surgery versus 3D- and 2D-laparoscopy during pancreatic and biliary anastomoses in a biotissue model: pooled analysis of two randomized trials

BACKGROUND

Robotic surgery may improve surgical performance during minimally invasive pancreatoduodenectomy as compared to 3D- and 2D-laparoscopy but comparative studies are lacking. This study assessed the impact of robotic surgery versus 3D- and 2D-laparoscopy on surgical performance and operative time using a standardized biotissue model for pancreatico- and hepatico-jejunostomy using pooled data from two randomized controlled crossover trials (RCTs).

METHODS

Pooled analysis of data from two RCTs with 60 participants (36 surgeons, 24 residents) from 11 countries (December 2017-July 2019) was conducted. Each included participant completed two pancreatico- and two hepatico-jejunostomies in biotissue using 3D-robotic surgery, 3D-laparoscopy, or 2D-laparoscopy. Primary outcomes were the objective structured assessment of technical skills (OSATS: 12-60) rating, scored by observers blinded for 3D/2D and the operative time required to complete both anastomoses. Sensitivity analysis excluded participants with excess experience compared to others.

RESULTS

A total of 220 anastomoses were completed (robotic 80, 3D-laparoscopy 70, 2D-laparoscopy 70). Participants in the robotic group had less surgical experience [median 1 (0-2) versus 6 years (4-12), p < 0.001], as compared to the laparoscopic group. Robotic surgery resulted in higher OSATS ratings (50, 43, 39 points, p = .021 and p < .001) and shorter operative time (56.5, 65.0, 81.5 min, p = .055 and p < .001), as compared to 3D- and 2D-laparoscopy, respectively, which remained in the sensitivity analysis.

CONCLUSION

In a pooled analysis of two RCTs in a biotissue model, robotic surgery resulted in better surgical performance scores and shorter operative time for biotissue pancreatic and biliary anastomoses, as compared to 3D- and 2D-laparoscopy.

Development and validity evidence of an objective structured assessment of technical skills score for minimally invasive linear-stapled, hand-sewn intestinal anastomoses: the A-OSATS score

INTRODUCTION

The aim of this study was to develop a reliable objective structured assessment of technical skills (OSATS) score for linear-stapled, hand-sewn closure of enterotomy intestinal anastomoses (A-OSATS).

MATERIALS AND METHODS

The Delphi methodology was used to create a traditional and weighted A-OSATS score highlighting the more important steps for patient outcomes according to an international expert consensus. Minimally invasive novices, intermediates, and experts were asked to perform a minimally invasive linear-stapled intestinal anastomosis with hand-sewn closure of the enterotomy in a live animal model either laparoscopically or robot-assisted. Video recordings were scored by two blinded raters assessing intrarater and interrater reliability and discriminative abilities between novices (n = 8), intermediates (n = 24), and experts (n = 8).

RESULTS

The Delphi process included 18 international experts and was successfully completed after 4 rounds. A total of 4 relevant main steps as well as 15 substeps were identified and a definition of each substep was provided. A maximum of 75 points could be reached in the unweighted A-OSATS score and 170 points in the weighted A-OSATS score respectively. A total of 41 anastomoses were evaluated. Excellent intrarater (r = 0.807-0.988, p < 0.001) and interrater (intraclass correlation coefficient = 0.923-0.924, p < 0.001) reliability was demonstrated. Both versions of the A-OSATS correlated well with the general OSATS and discriminated between novices, intermediates, and experts defined by their OSATS global rating scale.

CONCLUSION

With the weighted and unweighted A-OSATS score, we propose a new reliable standard to assess the creation of minimally invasive linear-stapled, hand-sewn anastomoses based on an international expert consensus. Validity evidence in live animal models is provided in this study. Future research should focus on assessing whether the weighted A-OSATS exceeds the predictive capabilities of patient outcomes of the unweighted A-OSATS and provide further validity evidence on using the score on different anastomotic techniques in humans.

Efficacy of simulation training for beating heart coronary anastomosis using BEAT + YOUCAN simulator

BACKGROUND

We sought to evaluate our distributed practice program developed for training for beating heart anastomosis by employing a novel beating heart simulator.

METHODS

Eleven trainees watched and reviewed instructional video recordings of coronary anastomosis methods with a BEAT  +  YOUCAN training device, then performed coronary anastomosis procedures under a beating condition. Next, they participated in a four-hour training program developed by faculty surgeons. Ten different anastomosis components were assessed on a five-point rating scale (5, good; 3, average; 1, poor). After finishing the training program, each trainee again performed a coronary anastomosis procedure. Component scores were then compared before and after the training program.

RESULTS

The mean time to completion of the procedure improved from 1033  ±  424 to 795  ±  201 s (p < 0.05). Assessment scores improved from 1.88  ±  0.41 to 2.57  ±  0.30 (p < 0.05). Improvements in some technical components related to handling of instruments were noted (p < 0.05), whereas no significant improvement was seen with arteriotomy, graft orientation, suture management, or knot tying after finishing the training program.

CONCLUSION

Trainees who participated in our four-hour focused training program for coronary anastomosis with a novel beating heart simulator showed improved ability under the beating condition in regard to technical skills related to handling instruments.

Training programme in gasless laparoscopy for rural surgeons of India (TARGET study) - Observational feasibility study

Background

Benefits of laparoscopic surgery are well recognised but uptake in rural settings of low- and middle-income countries is limited due to implementation barriers. Gasless laparoscopy has been proposed as an alternative but requires a trained rural surgical workforce to upscale. This study evaluates a feasibility of implementing a structured laparoscopic training programme for rural surgeons of North-East India.

Methods

A 3-day training programme was held at Kolkata Medical College in March 2019. Laparoscopic knowledge and Fundamentals of Laparoscopic Skills (FLS) were assessed pre and post simulation training using multiple choice questions and the McGill Inanimate System for Training and Evaluation of Laparoscopic Skills (MISTELS), respectively. Competency with an abdominal lift device was assessed using the Objective Structured Assessment of Technical Skills (OSATS) and live operating performance via the Global Operative Assessment of Laparoscopic Skills (GOALS) scores during live surgery. Costs of the training programme and qualitative feedback were evaluated.

Results

Seven rural surgeons participated. There was an improvement in knowledge acquisition (mean difference in MCQ score 5.57 (SD = 4.47)). The overall normalised mean MISTELS score for the FLS tasks improved from 386.02 (SD 110.52) pre-to 524.40 (SD 94.98) post-training (p = 0.09). Mean OSATS score was 22.4 out of 35 (SD 3.31) indicating competency with the abdominal lift device whilst a mean GOALS score of 16.42 out of 25 (SD 2.07) indicates proficiency in performing diagnostic laparoscopy using the gasless technique during live operating. Costs of the course were estimated at 354 USD for trainees and 461 USD for trainers.

Conclusion

Structured training programme in gasless laparoscopy improves overall knowledge and skills acquisition in laparoscopic surgery for rural surgeons of North-East India. It is feasible to deliver a training programme in gasless laparoscopy for rural surgeons. Larger studies are needed to assess the benefits for wider adoption in a similar context.

Preoperative Briefings and Postoperative Debriefings to Increase Resident Operative Autonomy and Performance

OBJECTIVE

To implement the use of standardized preoperative briefings and postoperative debriefings for surgical cases involving residents in an effort to improve resident autonomy and skill acquisition.

DESIGN

Prospective longitudinal study.

SETTING

Johns Hopkins Department of Otolaryngology-Head and Neck Surgery.

PARTICIPANTS

Resident and attending physicians.

RESULTS

Joint Huddles for Improving Resident Education (JHFIRE) tool was created and successfully implemented by 19 residents and 17 faculty members. Over the course of three data collection periods spanning an academic year, overall scores improved though not statistically significantly in the metrics of Zwisch autonomy, Resident Performance, and Objective Structured Assessment of Technical Skills (OSATS) scores. Female residents were scored significantly higher by attendings than their male counterparts in the assessment of baseline Resident Performance.

CONCLUSIONS

(1) JHFIRE tool implemented a standardized preoperative briefing and postoperative debriefing to improve communication and resident skill acquisition; (2) The tool was accepted and utilized throughout an academic year; (3) Zwisch, Resident Performance, and OSATS scores improved though not significantly.

Added value of 3D-vision during robotic pancreatoduodenectomy anastomoses in biotissue (LAEBOT 3D2D): a randomized controlled cross-over trial

BACKGROUND

We tested the added value of 3D-vision on procedure time and surgical performance during robotic pancreatoduodenectomy anastomoses in biotissue. Robotic surgery has the advantage of articulating instruments and 3D-vision. Consensus is lacking on the added value of 3D-vision during laparoscopic surgery. Given the improved dexterity with robotic surgery, the added value of 3D-vision may be even less with robotic surgery.

METHODS

In this experimental randomized controlled cross-over trial, 20 surgeons and surgical residents from 5 countries performed robotic pancreaticojejunostomy and hepaticojejunostomy anastomoses in a biotissue organ model using the da Vinci® system and were randomized to start with either 3D- or 2D-vision. Primary endpoint was the time required to complete both anastomoses. Secondary endpoint was the objective structured assessment of technical skill (OSATS; range 12-60) rating; scored by two observers blinded to 3D/2D.

RESULTS

Robotic 3D-vision reduced the combined operative time from 78.1 to 57.3 min (24.6% reduction, p < 0.001; 20.8 min reduction, 95% confidence intervals 12.8-28.8 min). This reduction was consistent for both anastomoses and between surgeons and residents, p < 0.001. Robotic 3D-vision improved OSATS performance by 6.1 points (20.8% improvement, p = 0.003) compared to 2D (39.4 to 45.1 points, ± 5.5).

CONCLUSION

3D-vision has a considerable added value during robotic pancreatoduodenectomy anastomoses in biotissue in both time reduction and improved surgical performance as compared to 2D-vision.

Association of intraoperative entrustment with clinical competency amongst general surgery residents

BACKGROUND

Lack of transparency and meaningful assessment in surgical residency has led to inconsistent intraoperative entrustment and highly variable trainee competence at graduation. The relationship between faculty entrustment and resident entrustability on clinical competency remains unclear. We sought to evaluate the dynamic between entrustment/entrustability and clinical competency in general surgery residency.

METHODS

Intraoperative observations were conducted across a 22-month period at an academic tertiary center. Entrustment/entrustability were measured using OpTrust. Clinical competencies were appraised via ACGME Milestones and Objective Structured Assessment of Technical Skill (OSATS) scores. Mixed effects linear regression was used to investigate the relationship among overall ACGME Milestone scores, OSATS domain scores, and overall OpTrust scores.

RESULTS

Overall OpTrust scores significantly correlated with overall Milestone scores and multiple OSATS score domains.

CONCLUSIONS

OpTrust demonstrated a positive association between ACGME general surgery Milestones and OSATS scores. Overall, OpTrust may help optimize intraoperative faculty entrustment and resident entrustability, facilitating surgical trainee success during residency.

Simulator training and residents' first laparoscopic hysterectomy: a randomized controlled trial

BACKGROUND

Hysterectomy rates are decreasing in many countries, and virtual reality simulators bring new opportunities into residents' surgical education. The objective of this study was to evaluate the effect of training in laparoscopic hysterectomy module with virtual reality simulator on surgical outcomes among residents performing their first laparoscopic hysterectomy.

METHODS

This randomized study was carried out at the Department of Obstetrics and Gynecology in Helsinki University Hospital and Hyvinkää Hospital. We recruited twenty residents and randomly signed half of them to train ten times with the laparoscopic hysterectomy module on a virtual reality simulator, while the rest represented the control group. Their first laparoscopic hysterectomy was video recorded and assessed later by using the Objective Structured Assessment of Technical Skills (OSATS) forms and Visual Analog Scale (VAS). The scores and surgical outcomes were compared between the groups.

RESULTS

The mean OSATS score for the Global Rating Scale (GRS) was 17.0 (SD 3.1) in the intervention group and 11.2 (SD 2.4) in the control group (p = 0.002). The mean procedure-specific OSATS score was 20.0 (SD 3.3) and 16.0 (SD 2.8) (p = 0.012), and the mean VAS score was 55.0 (SD 14.8) and 29.9 (SD 14.9) (p = 0.001). Operative time was 144 min in the intervention group and 165 min in the control group, but the difference did not reach statistical significance (p = 0.205). There were no differences between the groups in blood loss or direct complications.

CONCLUSION

Residents training with a virtual reality simulator prior to the first laparoscopic hysterectomy seem to perform better in the actual live operation. Thus, a virtual reality simulator hysterectomy module could be considered as a part of laparoscopic training curriculum.

Video-based coaching in surgical education: a systematic review and meta-analysis

BACKGROUND

In the era of competency-based surgical education, VBC has gained increased attention and may enhance the efficacy of surgical education. The objective of this systematic review was to summarize the existing evidence of video-based coaching (VBC) and compare VBC to traditional master-apprentice-based surgical education.

METHODS

We performed a systematic review and meta-analysis of randomized controlled trials (RCT) assessing VBC according to the PRISMA and Cochrane guidelines. The MEDLINE, EMBASE, and COCHRANE and Researchgate databases were searched for eligible manuscripts. Standard mean difference (SMD) of performance scoring scales was used to assess the effect of VBC versus traditional training without VBC (control).

RESULTS

Of 627 studies identified, 24 RCTs were eligible and evaluated. The studies included 778 surgical trainees (n = 386 VBC vs. n = 392 control). 13 performance scoring scales were used to assess technical competence; OSATS-GRS was the most common (n = 15). VBC was provided preoperative (n = 11), intraoperative (n = 1), postoperative (n = 10), and perioperative (n = 2). The majority of studies were unstructured, where identified coaching frameworks were PRACTICE (n = 1), GROW (n = 2) and Wisconsin Coaching Framework (n = 1). There was an effect on performance scoring scales in favor of VBC coaching (SMD 0.87, p < 0.001). In subgroup analyses, the residents had a larger relative effect (SMD 1.13; 0.61-1.65, p < 0.001) of VBC compared to medical students (SMD 0.43, 0.06-0.81, p < 0.001). The greatest source of potential bias was absence of blinding of the participants and personnel (n = 20).

CONCLUSION

Video-based coaching increases technical performance of medical students and surgical residents. There exist significant study and intervention heterogeneity that warrants further exploration, showing the need to structure and standardize video-based coaching tools.

Simulation as tool for evaluating and improving technical skills in laparoscopic gynecological surgery

BACKGROUND

Simulation in laparoscopic surgery is nowadays recognized as a valid instrument for learning and training surgeons in different procedures. However, its role as evaluation test and self-assessment tool to verify basic surgical skills is still under discussion.

METHODS

Thirty-three residents in obstetrics and gynecology at University of Pisa, Italy were recruited, and they received a simulation program consisting of 5 tasks. They had to perform basic laparoscopic surgery maneuvers as creating pneumoperitoneum, positioning trocars under vision, demonstrating the appropriate use of dominant and non-dominant hand and making single stitch and knot. They were evaluated with a modified OSATs scale.

RESULTS

Senior trainees had better score than junior trainees (p value< 0,005) and after different sessions of simulation scores of both groups significantly improved (p < 0,001), especially for the junior group. All the trainees reported self-assessments that matched with the evaluation of external observers demonstrating the importance of simulation also as auto-evaluation test.

CONCLUSIONS

In this study, we demonstrated the role of simulation as powerful tool to evaluate and to self-assess surgical technical skills and to improve own capacities, with the use of a modified OSATs scale adapted to specific exercises.

Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees

OBJECTIVE

Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video.

DESIGN

Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed.

SETTING

The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital.

PARTICIPANTS

Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard.

RESULTS

The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute.

CONCLUSIONS

Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.

The importance of technical skills assessment during an airway foreign body removal course

PURPOSE

Surgical simulation has proven useful in training for high-acuity, low frequency events such as airway foreign body removal (AWFBR). Studies have supported the role of simulation to improve trainee confidence and technical proficiency, but rigorous methodology is often lacking. Through a pilot study, we sought to evaluate the feasibility and utility of two-view video capture for rigorous assessment of trainee skill in an educational course setting.

METHODS

Participants were asked to perform 1) self-assessment surveys using 5- point Likert scale questions and 2) rigid bronchoscopy with AWFBR on intubation trainers, at the beginning and end of a multi-institution airway course. Video was collected from endoscopic and wide-angle body cameras. The videos were blinded and evaluated in random order by two expert reviewers using a 5-point pediatric airway endoscopy specific objective structured assessment of technical skills (OSATS) instrument.

RESULTS

Fourteen trainees submitted pre- and post-course surveys, and eight of these also had complete video data. Faculty feedback indicated the importance of pre- and post-course AWFBR recordings for real-time trainee feedback and post-course curricular refinement. Survey data showed an increase in confidence for AWFBR from 2.0 to 3.3 (p = 0.05). Average OSATS scores increased from 1.84 to 2.58 but this did not reach significance (p = 0.51). Paired improvements in confidence exceeded improvements in OSATS scores (1.29 vs. 0.18, p = 0.058).

CONCLUSIONS

Rigorously evaluated two-view video capture was feasible in an educational course setting. The course resulted in improvements in confidence to a greater degree than OSATS scores. This supports the importance of assessing course impact and refining curricula using all available data including objectively assessing technical skills.

Development and validation of an objective assessment scale for chest tube insertion under 'direct' and 'indirect' rating

BACKGROUND

There is an increasing need for objective and validated educational concepts. This holds especially true for surgical procedures like chest tube insertion (CTI). Thus, we developed an instrument for objectification of learning successes: the assessment scale based on Objective Structured Assessment of Technical Skill (OSATS) for chest tube insertion, which is evaluated in this study. Primary endpoint was the evaluation of intermethod reliability (IM). Secondary endpoints are 'indirect' interrater reliability (IR) and construct validity of the scale (CV).

METHODS

Every participant (N = 59) performed a CTI on a porcine thorax. Participants received three ratings (one 'direct' on site, two 'indirect' via video rating). IM compares 'direct' with 'indirect' ratings. IR was assessed between 'indirect' ratings. CV was investigated by subgroup analysis based on prior experience in CTI for 'direct' and 'indirect' rating.

RESULTS

We included 59 medical students to our study. IM showed moderate conformity ('direct' vs. 'indirect 1' ICC = 0.735, 95% CI: 0.554-0.843; 'direct' vs. 'indirect 2' ICC = 0.722, 95% CI 0.533-0.835) and good conformity between 'direct' vs. 'average indirect' rating (ICC = 0.764, 95% CI: 0.6-0.86). IR showed good conformity (ICC = 0.84, 95% CI: 0.707-0.91). CV was proven between subgroups in 'direct' (p = 0.037) and 'indirect' rating (p = 0.013).

CONCLUSION

Results for IM suggest equivalence for 'direct' and 'indirect' ratings, while both IR and CV was demonstrated in both rating methods. Thus, the assessment scale seems a reliable method for rating trainees' performances 'directly' as well as 'indirectly'. It may help to objectify and facilitate the assessment of training of chest tube insertion.

Assessment of Surgical Skills and Competency

Evaluation of surgical skills and competency are important aspects of the medical education process. Measurable and reproducible methods of assessment with objective feedback are essential components of surgical training. Objective Structured Assessment of Technical Skills (OSATS) is widely used across the medical specialties and otolaryngology-specific tools have been developed and validated for sinus and mastoid surgery. Although assessment of surgical skills can be time-consuming and requires human and financial resources, new evaluation methods and emerging technology may alleviate these barriers while also improving data collection practices.

Integration of Hands-On Team Training into Existing Curriculum Improves Both Technical and Nontechnical Skills in Laparoscopic Cholecystectomy

OBJECTIVE

Nontechnical skills are an essential component of surgical education and a major competency assessed by the ACGME milestones project. However, the optimal way to integrate nontechnical skills training into existing curricula and then objectively evaluate the outcome is still unknown. The aim of this study was to determine the effect laparoscopic team-based task training would have on the nontechnical skills needed for laparoscopic surgery.

DESIGN

9 PGY-1 residents underwent an established training curriculum for teaching the knowledge and technical skills involved in laparoscopic cholecystectomy. Initial training involved a didactic session, expert-led practice on a porcine model in a simulated operating room and laparoscopic skills practice on a virtual reality trainer. Residents then performed a laparoscopic cholecystectomy on the same porcine model as a preintervention test. Three to four months following this, residents were subjected to specific nontechnical skills training involving 2 simple team-based laparoscopic tasks. They then practiced a further 4 to 6 hours on the virtual reality trainer. A repeat postintervention laparoscopic cholecystectomy was then performed 3 to 4 months after nontechnical skills training. Both the preintervention and postintervention laparoscopic cholecystectomies were audiovisually recorded and then evaluated by 2 independent surgeons in a blinded fashion. Technical skills were assessed using objective structured assessment of technical skills (OSATS) and a technique specific rating scale (TRS) that we developed for laparoscopic cholecystectomy. Nontechnical skills were assessed using nontechnical skills for surgeons (NOTSS). Residents also completed a survey at the beginning and end of the training.

SETTING

Tertiary care, university based teaching institution.

PARTICIPANTS

A total of 9 general surgery residents at the intern level.

RESULTS

The mean OSATS score improved from 13.7 ± 1.24 to 26.7 ± 0.31 (p < 0.001), the mean TRS score improved from 6 ± 0.46 to 13.1 ± 0.36 (p < 0.001) and the mean NOTSS score improved from 21.7 ± 1.83 to 36.3 ± 0.87 (p < 0.001) following the training. There was a strong correlation between OSATS and NOTSS scores (Pearson's R = 0.98) and TRS and NOTSS (R = 0.94). The inter-rater agreement was 0.79 for NOTSS, 0.9 for OSATS, and 0.82 for TRS. Following completion of the training, residents self-reported improvements in exchanging information (p < 0.01), coordinating activities (p < 0.01) and coping with pressure in the operating room (p < 0.001).

CONCLUSION

Simple, team-based nontechnical skills training for laparoscopic cholecystectomy that was separate from technical skills training led to a sustained increase in residents' nontechnical skills 3 to 4 months after training. This was associated with a self-reported improvement in many nontechnical skills based on resident survey. Based on these results, we recommend that such designated nontechnical skills training is a valid alternative to other methods such as coaching and debriefing. We, therefore, plan to continue our efforts to develop team-based simulation tasks aimed at improving nontechnical skills for multiple surgical modalities.

Training or non-surgical factors-what determines a good surgical performance? A randomised controlled trial

BACKGROUND

Acquiring laparoscopic skills is a necessity for every young surgeon. Whether it is a talent or a non-surgical skill that determines the surgical performance of an endoscopic operation has been discussed for years. In other disciplines aptitude testing has become the norm. Airlines, for example, have implemented assessments to test the natural aptitude of future pilots to predict their performance later on. In the medical field, especially surgery, there are no similar comparable tests implemented or even available. This study investigates the influence of potential factors that may predict the successful performance of a complex laparoscopic operation, such as the surgeon's age, gender or learning method.

METHODS

This study focussed 70 surgical trainees. It was designed as a secondary analysis of data derived from a 2 × 2 factorial randomised controlled trial of practical training and/or multimedia training (four groups) in an experimental exercise. Both before and then after the training sessions, the participating trainees performed a laparoscopic cholecystectomy in a pelvitrainer. Surgical performance was then evaluated using a modified objective structured assessment of technical skills (OSATS). Participants were classified as 'Skilled' (high score in the pre-test), 'Good Learner' (increase from pre- to post-test) or 'Others' based on the OSATS results. Based on the results of the recorded performance, the training methods as well as non-surgical skills were eventually evaluated in a univariate and in a multivariate analysis.

RESULTS

In the pre-training performance 11 candidates were categorised as 'Skilled' (15.7%), 35 participants as 'Good Learners' (50.0%) and 24 participants were classified as 'Others'. The univariate analysis showed that the age, a residency in visceral surgery, and participation in a multimedia training were significantly associated with this grouping. Multivariate analyses revealed that residency in visceral surgery was the most predictive factor for the 'Skilled' participants (p = 0.059), and multimedia training was most predictive for the 'Good Learner' (p = 0.006). Participants in the group of 'Others' who were neither 'Skilled' nor improved in the training phase were younger (p = 0.011) and did not receive multimedia (p < 0.001) or practical (p = 0.025) training.

CONCLUSION

The type of learning method has been shown to be the most effective factor to improve laparoscopic skills, with multimedia training proving to be more effective than practical training.

Constructing a validity argument for the Objective Structured Assessment of Technical Skills (OSATS): a systematic review of validity evidence

In order to construct and evaluate the validity argument for the Objective Structured Assessment of Technical Skills (OSATS), based on Kane's framework, we conducted a systematic review. We searched MEDLINE, EMBASE, CINAHL, PsycINFO, ERIC, Web of Science, Scopus, and selected reference lists through February 2013. Working in duplicate, we selected original research articles in any language evaluating the OSATS as an assessment tool for any health professional. We iteratively and collaboratively extracted validity evidence from included articles to construct and evaluate the validity argument for varied uses of the OSATS. Twenty-nine articles met the inclusion criteria, all focussed on surgical technical skills assessment. We identified three intended uses for the OSATS, namely formative feedback, high-stakes assessment and program evaluation. Following Kane's framework, four inferences in the validity argument were examined (scoring, generalization, extrapolation, decision). For formative feedback and high-stakes assessment, there was reasonable evidence for scoring and extrapolation. However, for high-stakes assessment there was a dearth of evidence for generalization aside from inter-rater reliability data and an absence of evidence linking multi-station OSATS scores to performance in real clinical settings. For program evaluation, the OSATS validity argument was supported by reasonable generalization and extrapolation evidence. There was a complete lack of evidence regarding implications and decisions based on OSATS scores. In general, validity evidence supported the use of the OSATS for formative feedback. Research to provide support for decisions based on OSATS scores is required if the OSATS is to be used for higher-stakes decisions and program evaluation.

Validity, reliability and support for implementation of independence-scaled procedural assessment in laparoscopic surgery

Background

There is no widely used method to evaluate procedure-specific laparoscopic skills. The first aim of this study was to develop a procedure-based assessment method. The second aim was to compare its validity, reliability and feasibility with currently available global rating scales (GRSs).

Methods

An independence-scaled procedural assessment was created by linking the procedural key steps of the laparoscopic cholecystectomy to an independence scale. Subtitled and blinded videos of a novice, an intermediate and an almost competent trainee, were evaluated with GRSs (OSATS and GOALS) and the independence-scaled procedural assessment by seven surgeons, three senior trainees and six scrub nurses. Participants received a short introduction to the GRSs and independence-scaled procedural assessment before assessment. The validity was estimated with the Friedman and Wilcoxon test and the reliability with the intra-class correlation coefficient (ICC). A questionnaire was used to evaluate user opinion.

Results

Independence-scaled procedural assessment and GRS scores improved significantly with surgical experience (OSATS p = 0.001, GOALS p < 0.001, independence-scaled procedural assessment p < 0.001). The ICCs of the OSATS, GOALS and independence-scaled procedural assessment were 0.78, 0.74 and 0.84, respectively, among surgeons. The ICCs increased when the ratings of scrub nurses were added to those of the surgeons. The independence-scaled procedural assessment was not considered more of an administrative burden than the GRSs (p = 0.692).

Discussion/conclusion

A procedural assessment created by combining procedural key steps to an independence scale is a valid, reliable and acceptable assessment instrument in surgery. In contrast to the GRSs, the reliability of the independence-scaled procedural assessment exceeded the threshold of 0.8, indicating that it can also be used for summative assessment. It furthermore seems that scrub nurses can assess the operative competence of surgical trainees.