Objective Structured Assessment of technical skill (OSATS) in the Surgical Skills and Technology Elective Program (SSTEP): Comparison of peer and expert raters

Nonphysician Evaluators and Recording-Based Tools in Surgical Skill Assessment: A Feasibility Study

OBJECTIVES

To investigate the feasibility of nonmedically trained evaluators and image- and video-based tools in the assessment of surgical skills in a key orthopedic procedure.

DESIGN

Orthopedic surgeons at varying skill levels were evaluated by their ability to repair a cadaveric bi-malleolar ankle fracture. Nonphysician viewers and expert orthopedic surgeons independently scored video recordings and fluoroscopy images of the procedure through Global Rating Scales (GRS) and procedure-specific checklist tools. Statistical analysis was used to determine if the evaluators and assessment tools were able to differentiate skill level.

SETTING

An academic tertiary care hospital.

PARTICIPANTS

The surgical procedure was completed by 3 orthopedic residents, 3 orthopedic trauma fellows, and 4 orthopedic trauma attending surgeons. The procedure was independently evaluated by 2 orthopedic surgeons and 2 nonphysicians.

RESULTS

Operating participants were stratified by ≤ or >10 bimalleolar ankle fracture cases performed alone (inexperienced, n = 5 vs experienced, n = 5). Expert surgeon viewers could effectively stratify skill group through the GRS for video and fluoroscopy analysis (p < 0.05), and the video procedure-specific checklist (p < 0.05), but not the fluoroscopy procedure-specific checklist. Nonphysician viewers generally recognized skill groupings, although with less separation than surgeon viewers. These evaluators performed the best when aided by video and fluoroscopy procedure-specific checklists. Meanwhile, breakdowns of each tool into critical zones for improvement and evaluator-independent metrics such as case experience, self-reported confidence, and surgical time also indicated some skill differentiation.

CONCLUSIONS

The feasibility of using video recordings and fluoroscopic imaging based surgical skills assessment tools in orthopedic trauma was demonstrated. The tools highlighted in this study are applicable to both cadaver laboratory settings and live surgeries. The degree of training that is required by the evaluators and the utility of measuring surgical times of specific tasks should be the subject of future studies.

The Role of Interprofessional Education in Optimizing Laparoscopic Skills Training: A Randomized Controlled Trial

Background: This investigation evaluates the utility and benefits of integrating interprofessional education (IPE) into laparoscopic training, aiming to enrich medical education and skill acquisition methodologies. Methods: The study randomly allocated 36 participants of a 2023 laparoscopic training course into experimental and control groups, each comprising 18 individuals. The control group underwent traditional theory and practical training, whereas the experimental group additionally engaged in interdisciplinary instruction with nursing educators and participated in simulated laparoscopic surgery exercises. The effectiveness of this interdisciplinary approach was assessed by comparing laparoscopic theory and simulation performance, Objective Structured Assessment of Technical Skills (OSATS) scores in animal-based training, and course satisfaction between the groups. Moreover, the impact on interdisciplinary collaborative competencies was measured through pre- and post-training self-evaluations using the Interprofessional Collaborative Competency Attainment Survey (ICCAS) in the experimental group. Results: The experimental group demonstrated superior performance in laparoscopic theory and simulation, as well as higher OSATS scores, compared with the control group. Satisfaction ratings regarding the skills practice mode, effects, and instructional quality were also significantly better in the experimental group (P < .05, P < .01). Furthermore, participants in the experimental group reported significant pre-to-post training enhancements in interprofessional communication, team collaboration, role perception, conflict management, and learning and feedback marked by statistically significant differences (P < .05, P < .01). Conclusion: The introduction of an IPE framework significantly boosts laparoscopic training efficiency and promotes team collaboration awareness. This model effectively bridges gaps between disciplines, illustrating substantial applicative value and expansion potential within medical education and skill training arenas.

Comparing Faculty and Trainee Evaluators of First-Year Resident Skills

OBJECTIVE

To determine if senior residents are comparable to faculty in assessing first-year resident skills on their overall assessment.

BACKGROUND

As resident training moves towards a competency-based model, innovative approaches to evaluation and feedback through simulation need to be developed for both procedural as well as interpersonal and communication skills. In most areas of simulation, the faculty assess resident performance however; in clinical practice, first-year residents are often overseen and taught by senior residents. We aim to explore the agreement between faculty and senior resident assessors to determine if senior residents can be incorporated into a competency-based curriculum as appropriate evaluators of first-year resident skills.

DESIGN

Annual surgical first year resident training for central line placement, obtaining informed consent and breaking bad news at a single institution is assessed through an overall assessment (OA). In previous years, only faculty have been the evaluators for the OA. In this study, select senior residents were asked to participate as evaluators and agreement between groups of evaluators was assessed across the 3 tasks taught during surgical first-year resident training.

SETTING

Vanderbilt University Medical Center, tertiary hospital, Simulation Center.

PARTICIPANTS

Anesthesia and surgery interns, chief residents, anesthesia and surgical faculty.

RESULTS

Agreement between faculty and senior resident assessors was strongest for the central line placement simulation with a faculty average competency score of 10.71 and 9.59 from senior residents (κ = 0.43; 95% CI: -0.2, 0.34). Agreement was less substantial for simulated informed consent (κ = 0.08; 95% CI: -0.19, 0.36) and the breaking bad news simulation (κ = 0.07; 95% CI: -0.2, 0.34).

CONCLUSION

Select senior residents are comparable to faculty evaluators for procedural competency; however, there was less agreement between evaluator groups for interpersonal and communication-based competencies.

Assessment of suturing and scaling skills of periodontology and oral medicine residents by OSATS method: a pilot study

INTRODUCTION

Updating the method for evaluating suturing and scaling skills in dental education has attracted relatively little attention and there is no consensus to what should be assessed and how. The purpose of this study was to investigate the applicability of the Objective Structured Assessment of Technical Skill (OSATS) method for these two basic skills, the possible association between the scores and demographic factors, and the level of satisfaction of residents with this method.

METHODS

All six periodontics and three oral medicine residents were recruited by census method and video-recorded while performing a simple interrupted suture, a figure eight suture and scaling on a model. Then, the videos were evaluated independently via a checklist and a global rating scale (GRS) by two expert raters. Agreement between raters and residents' satisfaction were evaluated. Correlation between demographic factors of participants and scores was also assessed. T-test and linear regression analysis were used.

RESULTS

There was no significant difference between the scores based on the views of the two raters for each of the checklist (ICC = 0.99, CI = 0.96-0.99, P < 0.001) and GRS (ICC = 0.97, CI = 0.86-0.99, P < 0.001). Linear regression showed no correlation between gender and scores but periodontics major and higher year of education showed correlation with higher scores.

CONCLUSION

Considering the excellent agreement between raters in using both the checklist and GRS components of OSATS, and satisfaction of 88% the residents with this method, it seems to be able to provide a reliable assessment.

A Study of Surgical Residents' Self-Assessment of Open Surgery Skills Using Gap Analysis

BACKGROUND

Surgical residents need structured and objective feedback to develop their skills and become capable of performing surgical procedures autonomously. A shortage of experienced surgical staff has prompted residents to seek feedback from self-assessment and peer assessments.

OBJECTIVES

We investigated whether surgical residents can reliably rate their own and their peers' basic surgical skills using the Global Rating Scale (GRS) from the Objective Structured Assessment of Technical Skills.

METHODS

The study was a prospective and descriptive study conducted using gap analysis at the Copenhagen Academy for Medical Education and Simulation (Copenhagen, Denmark) from 2016 to 2017. Surgical residents were recruited during a course in basic open surgical skills. Among 102 course participants, 53 met the inclusion criteria and 22 participated in the study.

RESULTS

We recruited surgical residents based in the Capital Region and Zealand Region of Demark, and 42% of eligible residents participated in the study. Surgical residents underestimated their own surgical performance (median, 17 [range, 15-18] vs. 20 [range, 19.75-22]; P < 0.001). They also rated their peers lower than an experienced rater did (median 10 [range, 8.75-14] vs. 15 and median 20.5 [range, 19-22] vs. 23; both P < 0.001). Gap analysis revealed residents had unrecognized strengths (ie, self-underappraisal) in most GRS domains.

CONCLUSIONS

Ratings are unreliable when surgical residents assess their own and their peers' performances using GRS. A gap analysis revealed unrecognized strengths in time and motion, instrument handling, knowledge of instruments and sutures, and knowledge of specific procedure as well as unrecognized weaknesses in flow of operation and forward planning.

Deep neural network architecture for automated soft surgical skills evaluation using objective structured assessment of technical skills criteria

PURPOSE

Classic methods of surgery skills evaluation tend to classify the surgeon performance in multi-categorical discrete classes. If this classification scheme has proven to be effective, it does not provide in-between evaluation levels. If these intermediate scoring levels were available, they would provide more accurate evaluation of the surgeon trainee.

METHODS

We propose a novel approach to assess surgery skills on a continuous scale ranging from 1 to 5. We show that the proposed approach is flexible enough to be used either for scores of global performance or several sub-scores based on a surgical criteria set called Objective Structured Assessment of Technical Skills (OSATS). We established a combined CNN+BiLSTM architecture to take advantage of both temporal and spatial features of kinematic data. Our experimental validation relies on real-world data obtained from JIGSAWS database. The surgeons are evaluated on three tasks: Knot-Tying, Needle-Passing and Suturing. The proposed framework of neural networks takes as inputs a sequence of 76 kinematic variables and produces an output float score ranging from 1 to 5, reflecting the quality of the performed surgical task.

RESULTS

Our proposed model achieves high-quality OSATS scores predictions with means of Spearman correlation coefficients between the predicted outputs and the ground-truth outputs of 0.82, 0.60 and 0.65 for Knot-Tying, Needle-Passing and Suturing, respectively. To our knowledge, we are the first to achieve this regression performance using the OSATS criteria and the JIGSAWS kinematic data.

CONCLUSION

An effective deep learning tool was created for the purpose of surgical skills assessment. It was shown that our method could be a promising surgical skills evaluation tool for surgical training programs.

Boot camp approach to surgical residency preparation: feedback from a French university hospital

INTRODUCTION

The transition from medical student to surgical resident is not a simple one. The aim of this study was to report the experience of a university hospital in the organization of the induction course for future surgical residents and the contribution of a video support in the learning of the suture.

MATERIAL AND METHOD

We were able to study two consecutive years of students (October 2020 and 2021). Concerning the practical and technical workshops (learning suture) we carried out a comparative study between two groups of students. A group that had video support for learning suture (video group) and a group without video (control group). The evaluation of the suture was performed in a blinded manner by two supervising surgeons. The other practical workshop was drain fixation; the students did not have a video for this workshop. A comparative study was also performed for the drain fixation workshop between the two groups (video group and control group). A program of theoretical courses was also set up. This program is established according to the different future functions of the residents by integrating medico-legal notions and teamwork. Satisfaction questionnaires were given to the students and the answers were given two months after taking up their duties in the hospital (6 questions with Likert scale and 4 free questions).

RESULTS

The cohort consisted of 58 students (29 each in 2020 and 29 in 2021). Comparative analyses of the evaluation of the suture workshops showed better performance in the video group compared with the group without video. The comparison of these two groups did not show significant differences in the drain fixation workshop. The theoretical teaching was broken down according to the students' future tasks and each speaker was a specialist in his or her field of expertise. The results of the questionnaires showed a desire on the part of the students to increase the time spent on practical workshops and theoretical forensic teaching.

CONCLUSION

We were able to show through these two years of a program that we were able to offer a surgical resident preparation course. In addition, we have highlighted the contribution of a video support in the learning curve of the suture.

The "chicken-leg anastomosis": Low-cost tissue-realistic simulation model for esophageal atresia training in pediatric surgery

INTRODUCTION

Shifting the training from the operating room (OR) to simulation models has been proven effective in enhancing patient safety and reducing the learning time to achieve competency and increase the operative efficiency. Currently the field of pediatric surgery only offers few low-cost trainers for specialized training and these feature predominantly artificial and often unrealistic tissue. The aim of this study was to develop an easy access low-cost tissue-realistic simulation model for open training of esophageal atresia and to evaluate the acceptance in trainees and junior pediatric surgeons.

MATERIALS AND METHODS

The model is fashioned using reconfigured chicken skin from a chicken leg. To create a model of esophageal atresia, the chicken skin is dissected off the muscle and reconfigured around a foley catheter balloon to recreate the proximal pouch and a feeding tube to recreate the distal pouch. Surrounding structures such as the tracheo-esophageal fistula and the azygos vein can be easily added, obtaining a realistic esophageal atresia (Type C) prototype. Evaluation of model construction, usage and impact on user were performed by both a self-assessment questionnaire with pre- and post-training questions as well as observer-based variables and a revised Objective Structured Assessment of Technical Skills (OSATS) score.

RESULTS

A total of 10 participants were constructing and using the model at two different timepoints. OSATS score for overall performance was significantly higher (p = 0.005, z = -2.78) during the second observational period [median (MD): 4,95% confidence interval CI: 3.4, 5.1] compared to the first (MD: 3, 95% CI 2.4, 4.1). Self-reported boost in confidence after model usage for performing future esophageal atresia (EA) repair and bowel anastomosis (BA) in general was significantly higher (EA: U = 1, z = -2.3, p = 0.021, BA: U = 1, z = -2.41, p = 0.016) in participants with more years in training/attending status (EA MD:5, BA MD: 5.5) compared to less experienced participants (EA MD: 1.5, BA: 1).

CONCLUSION

Our easy access low-cost simulation model represents a feasible and tissue realistic training option to increase surgical performance of pediatric surgical trainees outside the OR.

Surgical coaching applied to laparoscopic TME for continuous professional development in rectal surgery: proof of concept

BACKGROUND

Coaching is maturing as a strategy for surgeons' continuous professional development in different types of surgery. Laparoscopic total mesorectal excision (LAP TME) is one of the recognized difficult procedures in colorectal surgery. Aim of this trial is to introduce the surgical coaching as a tool for a continuous technical education of LAP TME for cancer in a consultant surgeon carrier.

METHODS

Twelve Italian colorectal surgeons were enrolled as trainees in the AIMS Academy rectal cancer surgical coaching project and attended a face-to-face 90-min surgical coaching on a pre-edited 45-min-long video of a laparoscopic proctectomy according to pre-determined guidelines. At the end of the coaching, all mentors were asked to fill a questionnaire evaluating the trainee's skills. All trainees had to fill a post-coaching questionnaire addressing the appropriateness of the coaching with respect to their actual level.

RESULTS

Trainees were more confident in performing the extra-pelvic part of the surgical procedures compared to the intra-pelvic dissection. The most challenging steps according to the trainees were the seminal vesicles identification and the pelvic floor dissection. Mentors found the trainees quite confident in the approach to the vascular structures, lymphadenectomy, stapler utilization and bleeding control. The sharpness and the efficacy of the dissection, the dissection of the surgical planes and the anastomosis fashioning were reported at a lower level of confidence. The higher grade of satisfaction reported by the trainee came from the attention that the mentors demonstrated towards them, from the availability of the mentors to take into consideration the surgical issues raised and from the willingness to apply the suggestions received during their next proctectomies.

CONCLUSIONS

The surgical coaching applied to LAP TME should be considered as an innovative tool for continuous professional development.