Simulation platforms to assess laparoscopic suturing skills: a scoping review

EASIER: A new model for online learning of minimally invasive surgery skills

INTRODUCTION

Technology Enhanced Learning (TEL) can provide the tools to safely master minimally invasive surgery (MIS) skills in patient-free environments and receive immediate objective feedback without the constant presence of an instructor. However, TEL-based systems tend to work isolated from one another, focus on different skills, and fail to provide contents without a sound pedagogical background.

OBJECTIVE

The objective of this descriptive study is to present in detail EASIER, an innovative TEL platform for surgical and interventional training, as well as the results of its validation.

METHODS

EASIER provides a Learning Management System (LMS) for institutions and content creators that can connect and integrate TEL "external assets" (virtual reality simulators, augmented box trainers, augmented videos, etc.) addressing different skills. The platform integrates all skills under an Assessment Module that measures skills' progress in different courses. Finally, it provides content creators with a pedagogical model to scaffold contents while retaining flexibility to approach course design with different training philosophies in mind. Three courses were developed and hosted in the platform to validate it with end-users in terms of usability, performance, learning results in the courses and student self-perception on learning.

RESULTS

In total 111 volunteers completed the validation. The study was limited due to the COVID-19 pandemic, which limited access to external assets (virtual reality simulators). Nevertheless, usability was rated with 73.1 in the System Usability Scale. Most positive aspects on performance were easiness to access the platform, easiness to change the configuration and not requiring additional plug-ins to use the platform. The platform was rated above average in the six scales of the User Experience Questionnaire. Overall, student results improved significantly across the three courses (p < 0.05).

CONCLUSIONS

This study provides, within its limitations, evidence on the usefulness of the EASIER platform for distance learning of MIS skills. Results show the potential impact of the platform and are an encouraging boost for the future, especially in the aftermath of the COVID-19 pandemic.

A Video-Based Assessment Tool to Measure Intraoperative Laparoscopic Suturing Using a Modified Script Concordance Methodology

OBJECTIVES

Laparoscopic suturing (LS) is a challenging laparoscopic skill to teach. Its complexity and nuances are not modeled or measured in current simulation and assessment platforms.The script concordance test (SCT) is used to assess clinical reasoning.The purpose of this study is to provide evidence for validity of this novel SCT based online assessment for LS skills.

DESIGN

We designed a video-based online SCT for LS using a cognitive task analysis and expert panelists.The CTA yielded 4 LS domains: needle handling (NH), tissue handling (TH), knot tying techniques (KT) and operative ergonomics (OE). Five-point scales with anchoring descriptors from -2 to +2 were used. Scoring was based on a modified SCT methodology.

SETTING AND PARTICIPANTS

The test was administrated to 37 subjects (18 experts and 19 novices). There was no time limit given. A different expert group from the minimal invasive surgery (MIS) panelist were recruited. Experts were defined as surgeons and fellows with LS experience of >25 cases annually. Validity was assessed by comparing SCT scores of experienced and inexperienced surgeons. Cronbach's alpha was used to assess the internal consistency of the test.

RESULTS

The survey started off with 47 questions in each of the following domains: 13 NH, 4 TH, 20 KT and 10 OE. Thirty-seven surgeons (18 experts and 19 inexperienced surgeons). Questions that demonstrated a large discrepancy among experts and panelists with a weighted score difference more than 40 were discarded (n = 20). One question was discarded because it received a 100% score from all participants. This yielded 26 remaining questions in the following domains: 8 NH, 2 TH, 11 KT and 5 OE. The test reliability level (Cronbach a) was 0.80. The mean score was 72 ± 9% and 63 ± 15% (p = 0.02) for experts and inexperienced surgeons, respectively. The mean time to complete the test was 21 minutes.

CONCLUSION

This study provides validity evidence for a novel intraoperative LS assessment. The variability of responses between experts and panelists suggests that SCT may capture the clinical differences/surgeon preferences in performing LS intraoperatively.

Construct Validity of a Novel Assessment System for Laparoscopic Suture Accuracy Based on Stereoscopy

BACKGROUND

An accurate and objective measuring tool is lacking for laparoscopic suture accuracy assessment in simulation training. We designed and developed the suture accuracy testing system (SATS) and aimed to determine its construct validity in this study.

METHODS

Twenty laparoscopic experts and 20 novices were recruited to perform a suturing task in three sessions using traditional laparoscopic instruments (Tra. session), a handheld multi-degree-of-freedom (MDoF) laparoscopic instrument (MDoF session) and a surgical robot (Rob. session), respectively. The needle entry and exit errors were calculated using the SATS and compared between the two groups.

RESULTS

No significant difference of the needle entry error was found in all comparisons. As for the needle exit error, the value of the novice group was significantly higher than that of the expert group in Tra. session (3.48 ± 0.61 mm vs. 0.85 ± 0.14 mm; p = 1.451e-11) and MDoF session (2.65 ± 0.41 mm vs. 1.06 ± 0.17 mm; p = 1.451e-11) but not in Rob. session (0.51 ± 0.12 mm vs. 0.45 ± 0.08 mm; p = 0.091).

CONCLUSIONS

The SATS demonstrates construct validity. Surgeons' experience in conventional laparoscopic instruments could be transferred to the MDoF instrument. Surgical robot helps to improve suture accuracy and may bridge the experience gap between laparoscopic experts and novices in basic exercises.

Application of a highly simulated and adaptable training system in the laparoscopic training course for surgical residents: Experience from a high-volume teaching hospital in China

Objective

To explore the effectiveness, feasibility, and training effect of a highly simulated and adaptable laparoscopic training system in the advanced integrated two-stage laparoscopic simulation training course for surgical residents.

Methods

This study prospectively took the surgical residents who received the advanced integrated two-stage laparoscopic simulation training course in our hospital from December 2019 to December 2021 as the research objects. In the stage one course, the trainees are randomly distributed into the dry simulation system group and Darwin laparoscopic training system group. The subjective assessment results of the trainees from the two groups are collected by questionnaires, and the simulation assessment results of the two groups are evaluated in a unified, objective, and standardized assessment form. The pre-course and post-course questionnaires were used to evaluate the feasibility and effectiveness of the Darwin system in the stage two course.

Results

A total of 62 trainees completed the stage one and stage two courses. In the stage one course, the trainees were randomly distributed into the dry simulation trainer group (N = 19) and the Darwin group (N = 43). The results of the subjective assessment questionnaire showed that compared with the dry simulator group, the students in the Darwin group had higher subjective scores (P < 0.05). The objective assessment results for the 3 modules of "One Track Transfer", "One Tunnel Pass" and "High and Low Pillars" in the Darwin group were significantly better than those in the dry simulator group (P < 0.05). The trainees who received the stage two course completed the questionnaires before and after the course. The results showed that compared with pre-course evaluation, "basic theoretical knowledge of laparoscopy", "basic skills of laparoscopy", "laparoscopic suture technique" and "camera-holding technique" were significantly improved after training (P < 0.05).

Conclusion

The highly simulated and adaptable laparoscopic training system is effective and feasible in the advanced integrated two-stage laparoscopic simulation training course for surgical residents.

Assessment of Minimally Invasive Suturing Skills: Is Instrument Tracking an Accurate Prediction?

Introduction: Minimally invasive surgery (MIS) suturing demands advanced surgical skills. Therefore, it is important these skills are adequately trained and assessed. Assessment and feedback can consist of judgments and scores of expert observers or objective parameters using instrument tracking. The aim of this study was to determine to what extent objective parameters correspond to expert assessment. Methods: Participants performed an intracorporeal suturing task on the EoSim simulator repeatedly (maximum 20 repetitions) during training. The best discriminating parameters, which previously shown construct validation, were combined into a composite score, using regression analysis. All videos were blinded and assessed by 2 independent reviewers using the validated laparoscopic suturing competency assessment tool (LS-CAT). These scores were compared with the composite score. Results: A 100 videos of 16 trainees, during separate points on their learning curve, and 8 experts were used. The parameters "time" and "distance" were statistically significantly correlated with all LS-CAT domains. The composite score (calculated from "time" and "distance") showed improvement between the first and the last knot (57% versus 94%, P < .001). Also the LS-CAT score improved (28 versus 17, P < .001). However, the correlation of the composite score with the LS-CAT score was weak (R: 0.351), with an accuracy of 55/100 when pooling the outcomes based on inadequate, adequate, or good performance. Conclusion: Instrument tracking parameters (using Surgtrac) could give an indication of the skill level, however, it missed important elements, essential for reliable assessment. Therefore, expert assessment remains superior to determine the skill level in MIS suturing skills.

Establishing validity evidence for device-assisted advanced laparoscopic suturing tasks using simulation

BACKGROUND

Laparoscopic suturing (LS) is an essential technique required for a wide range of procedures, and it is one of the most challenging for surgical trainees to master. We designed and collected validity evidence for advanced LS tasks using an automated suturing device and evaluated the perceived educational value of the tasks.

METHODS

This project was a multicentre prospective study involving McGill University, University of Toronto (UofT), and Louisiana State University (LSU) Health New Orleans. Novice (NS) and experienced (ES) surgeons performed suturing under tension (UT) and continuous suturing (CS) tasks. ES performed the tasks twice to establish proficiency benchmarks, and they were interviewed to develop formative feedback tools (FFT). Participants were assessed on completion time, error, Global Operative Assessment of Laparoscopic Skills (GOALS), and FFT. Data were analyzed using descriptive and inferential statistical methods.

RESULTS

Twenty-seven participants (13 ES, 14 NS, median age 34 years; 85% male) completed the study. Eight were attending surgeons, 7 fellows, 4 PGY5, 5 PGY4, and 3 PGY3 (18 from McGill, 5 UofT, and 4 LSU). Comparing ES and NS, for UT task, ES had significantly greater task scores (570 [563-648] vs 323 [130-464], p value 0.00036) and GOALS scores (14 [13-16] vs 10 [8-12], p value 0.0038). Similarly, for CS, ES had significantly greater task scores (976 [959-1010] vs 785 [626-856], p value 0.00009) and GOALS scores (16 [12-17] vs 12.5 [8.25-15], p value 0.028). After FFTs were developed, comparing ES and NS, for both UT and CS tasks, ES had significantly greater FFT scores (UT 25 [24-26] vs 17 [14-20], p value 0.0016 and CS 30 [27-32] vs 22[17.2-25.8], p value 0.00061).

CONCLUSION

In conclusion, preliminary validity evidence was provided for the tasks. Once further validity evidence is established, incorporating the tasks into the training curricula could improve trainee skills and help to meet the need for better advanced suturing models.

Pilot Evaluation of a Novel, Low-Cost, Simulation Model for Training and Assessment of Laparoscopic Intracorporeal Continuous Suturing

Background: Laparoscopic intracorporeal continuous suturing is being employed in a growing number of minimally invasive procedures. However, there is a lack of adequate bench models for gaining proficiency in this complex task. The purpose of this study was to assess a novel simulation model for running suture. Methods: Participants were grouped as novice (LSN) or expert (LSE) at laparoscopic suturing based on prior experience and training level. A novel low-cost bench model was developed to simulate laparoscopic intracorporeal continuous closure of a defect. The primary outcome measured was time taken to complete the task. Videos were scored by independent raters for Global Operative Assessment of Laparoscopic Skills (GOALS). Results: Sixteen subjects (7 LSE and 9 LSN) participated in this study. LSE completed the task significantly faster than LSN (430 ± 107 vs 637 ± 164 seconds, P ≤ .05). LSN scored higher on accuracy penalties than LSE (Median 30 vs 0, P ≤ .05). Mean GOALS score was significantly different between the 2 groups (LSE 20.64 ± 2.64 vs LSN 14.28 ± 1.94, P < .001) with good inter-rater reliability (ICC ≥ .823). An aggregate score using the formula: Performance Score = 1200-time(sec)-(accuracy penalties x 10) was significantly different between groups with a mean score of 741 ± 141 for LSE vs 285 ± 167 for LSN ( P < .001). Conclusion A novel bench model for laparoscopic continuous suturing was able to significantly discriminate between laparoscopic experts and novices. This low-cost model may be useful for both training and assessment of laparoscopic continuous suturing proficiency.

Standard Setting in Simulation-Based Training of Surgical Procedures: A Systematic Review

OBJECTIVE

This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures.

SUMMARY OF BACKGROUND

Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods.

METHODS

Systematic literature search was done in four databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with application of a standard setting were included. Quality of evidence was appraised using GRADE.

RESULTS

Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, set-up, and expert level classification. The studies were appraised as having low and moderate evidence.

CONCLUSION

Surgical education is shifting towards competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid and reliable assessment.

Distinguishing Intermediate and Novice Surgeons by Eye Movements

Surgical skill-level assessment is key to collecting the required feedback and adapting the educational programs accordingly. Currently, these assessments for the minimal invasive surgery programs are primarily based on subjective methods, and there is no consensus on skill level classifications. One of the most detailed of these classifications categorize skill levels as beginner, novice, intermediate, sub-expert, and expert. To properly integrate skill assessment into minimal invasive surgical education programs and provide skill-based training alternatives, it is necessary to classify the skill levels in as detailed a way as possible and identify the differences between all skill levels in an objective manner. Yet, despite the existence of very encouraging results in the literature, most of the studies have been conducted to better understand the differences between novice and expert surgical skill levels leaving out the other crucial skill levels between them. Additionally, there are very limited studies by considering the eye-movement behaviors of surgical residents. To this end, the present study attempted to distinguish novice- and intermediate-level surgical residents based on their eye movements. The eye-movement data was recorded from 23 volunteer surgical residents while they were performing four computer-based simulated surgical tasks under different hand conditions. The data was analyzed using logistic regression to estimate the skill levels of both groups. The best results of the estimation revealing a 91.3% recognition rate of predicting novice and intermediate surgical residents on one scenario were selected from four under the dominant hand condition. These results show that the eye-movements can be potentially used to identify surgeons with intermediate and novice skills. However, the results also indicate that the order in which the scenarios are provided, and the design of the scenario, the tasks, and their appropriateness with the skill levels of the participants are all critical factors to be considered in improving the estimation ratio, and hence require thorough assessment for future research.