A scoping review of assessment tools for laparoscopic suturing

New 3D printed simulator for training of endoscopic transsphenoidal surgery used in a dedicated pituitary course: A French cross-sectional study

CONTEXT

The objective was to evaluate the feasibility and usefulness of a new 3D printed simulator for endoscopic pituitary surgery. This simulator was used at the Pituitary Workshop, a French theoretical and practical course designed to teach the basics of pituitary tumour management.

METHODS

We conducted a cross-sectional study. The pituitary workshop had two components: (i) hands-on skills lab on the simulator, and (ii) lecture-based learning. The simulator was assessed by scoring its realism (face validity, score out of 80), its effectiveness in improving participants' surgical technique (content validity, score out of 60), and its ability to differentiate levels of skill competence using performance metrics (construct validity, score out of 12). In a subgroup of 9 participants, the impact of the theoretical component of the course was also assessed (score out of 33), using a knowledge questionnaire before and after the course.

RESULTS

Participants were neurosurgery residents (n = 24) and board-certified neurosurgeons (n = 6), supervised by expert neurosurgeons (n = 3). Face and content validity scores were high, reaching 63.3 ± 8.4 and 55.7 ± 4.6 respectively. Performance scores were higher in the group of experts and board-certified neurosurgeons than in the group of residents (12 ± 0 and 11.3 ± 1.2 vs. 9.2 ± 2.3, P = 0.019). The knowledge score improved significantly after the workshop (25.5 ± 4.8 vs. 10.9 ± 5.8, P = 7.1e-05).

CONCLUSIONS

Based on the face, content and construct validity scores, simulation training using the new model can be considered feasible and useful for neurosurgery residents. The pituitary workshop has a positive practical and theoretical impact on the majority of participants.

Construct Validity of a Novel Assessment System for Laparoscopic Suture Accuracy Based on Stereoscopy

BACKGROUND

An accurate and objective measuring tool is lacking for laparoscopic suture accuracy assessment in simulation training. We designed and developed the suture accuracy testing system (SATS) and aimed to determine its construct validity in this study.

METHODS

Twenty laparoscopic experts and 20 novices were recruited to perform a suturing task in three sessions using traditional laparoscopic instruments (Tra. session), a handheld multi-degree-of-freedom (MDoF) laparoscopic instrument (MDoF session) and a surgical robot (Rob. session), respectively. The needle entry and exit errors were calculated using the SATS and compared between the two groups.

RESULTS

No significant difference of the needle entry error was found in all comparisons. As for the needle exit error, the value of the novice group was significantly higher than that of the expert group in Tra. session (3.48 ± 0.61 mm vs. 0.85 ± 0.14 mm; p = 1.451e-11) and MDoF session (2.65 ± 0.41 mm vs. 1.06 ± 0.17 mm; p = 1.451e-11) but not in Rob. session (0.51 ± 0.12 mm vs. 0.45 ± 0.08 mm; p = 0.091).

CONCLUSIONS

The SATS demonstrates construct validity. Surgeons' experience in conventional laparoscopic instruments could be transferred to the MDoF instrument. Surgical robot helps to improve suture accuracy and may bridge the experience gap between laparoscopic experts and novices in basic exercises.

Video-based assessment (VBA) of an open, simulated orthopedic surgical procedure: a pilot study using a single-angle camera to assess surgical skill and decision making

BACKGROUND

Videos have been used in many settings including medical simulation. Limited information currently exists on video-based assessment in surgical training. Effective assessment tools have substantial impact on the future of training. The objectives of this study were as follows: to evaluate the inter-rater reliability of video-based assessment of orthopedic surgery residents performing open cadaveric simulation procedures and to explore the benefits and limitations of video-based assessment.

METHODS

A multi-method technique was used. In the quantitative portion, four residents participated in a Surgical Objective Structured Clinical Examination in 2017 at a quaternary care training center. A single camera bird's-eye view was used to videotape the procedures. Five orthopedic surgeons evaluated the surgical videos using the Ottawa Surgical Competency Operating Room Evaluation. Interclass correlation coefficient was used to calculate inter-rater reliability. In the qualitative section, semi-structured interviews were used to explore the perceived strengths and limitations of video-based assessment.

RESULTS AND DISCUSSION

The scores using video-based assessment demonstrated good inter-rater reliability (ICC = 0.832, p = 0.014) in assessing open orthopedic procedures on cadavers. Qualitatively, the strengths of video-based assessment in this study are its ability to assess global performance and/or specific skills, ability to reassess missed points during live assessment, and potential use for less common procedures. It also allows for detailed constructive feedback, flexible assessment time, anonymous assessment, multiple assessors and serves as a good coaching tool. The main limitations of video-based assessment are poor audio-video quality, and questionable feasibility for assessing readiness for practice.

CONCLUSION

Video-based assessment is a potential adjunct to live assessment in orthopedic open procedures with good inter-rater reliability. Improving audio-video quality will enhance the quality of the assessment and improve the effectiveness of using this tool in surgical training.

Comparing Simulator Metrics and Rater Assessment of Laparoscopic Suturing Skills

BACKGROUND

Laparoscopic intracorporeal suturing is important to master and competence should be ensured using an optimal method in a simulated environment before proceeding to real operations. The objectives of this study were to gather validity evidence for two tools for assessing laparoscopic intracorporeal knot tying and compare the rater-based assessment of laparoscopic intracorporeal suturing with the assessment based on simulator metrics.

METHODS

Twenty-eight novices and 19 experienced surgeons performed four laparoscopic sutures on a Simball Box simulator twice. Two surgeons used the Intracorporeal Suturing Assessment Tool (ISAT) for blinded video rating.

RESULTS

Composite Simulator Score (CSS) had higher test-retest reliability than the ISAT. The correlation between the number performed procedures including suturing and ISAT score was 0.51, p<0.001, and 0.59 p<0.001 for CSS. We found an inter-rater reliability (0.72, p<0.001 for test 1 and 0.53 p<0.001 for test 2). The pass/fail rates for ISAT and CSS were similar.

CONCLUSION

CSS and ISAT provide similar results for assessing laparoscopic suturing but assess different aspects of performance. Using simulator metrics and raters' assessments in combination should be considered for a more comprehensive evaluation of laparoscopic knot-tying competency.

Establishing validity evidence for device-assisted advanced laparoscopic suturing tasks using simulation

BACKGROUND

Laparoscopic suturing (LS) is an essential technique required for a wide range of procedures, and it is one of the most challenging for surgical trainees to master. We designed and collected validity evidence for advanced LS tasks using an automated suturing device and evaluated the perceived educational value of the tasks.

METHODS

This project was a multicentre prospective study involving McGill University, University of Toronto (UofT), and Louisiana State University (LSU) Health New Orleans. Novice (NS) and experienced (ES) surgeons performed suturing under tension (UT) and continuous suturing (CS) tasks. ES performed the tasks twice to establish proficiency benchmarks, and they were interviewed to develop formative feedback tools (FFT). Participants were assessed on completion time, error, Global Operative Assessment of Laparoscopic Skills (GOALS), and FFT. Data were analyzed using descriptive and inferential statistical methods.

RESULTS

Twenty-seven participants (13 ES, 14 NS, median age 34 years; 85% male) completed the study. Eight were attending surgeons, 7 fellows, 4 PGY5, 5 PGY4, and 3 PGY3 (18 from McGill, 5 UofT, and 4 LSU). Comparing ES and NS, for UT task, ES had significantly greater task scores (570 [563-648] vs 323 [130-464], p value 0.00036) and GOALS scores (14 [13-16] vs 10 [8-12], p value 0.0038). Similarly, for CS, ES had significantly greater task scores (976 [959-1010] vs 785 [626-856], p value 0.00009) and GOALS scores (16 [12-17] vs 12.5 [8.25-15], p value 0.028). After FFTs were developed, comparing ES and NS, for both UT and CS tasks, ES had significantly greater FFT scores (UT 25 [24-26] vs 17 [14-20], p value 0.0016 and CS 30 [27-32] vs 22[17.2-25.8], p value 0.00061).

CONCLUSION

In conclusion, preliminary validity evidence was provided for the tasks. Once further validity evidence is established, incorporating the tasks into the training curricula could improve trainee skills and help to meet the need for better advanced suturing models.

Pilot Evaluation of a Novel, Low-Cost, Simulation Model for Training and Assessment of Laparoscopic Intracorporeal Continuous Suturing

Background: Laparoscopic intracorporeal continuous suturing is being employed in a growing number of minimally invasive procedures. However, there is a lack of adequate bench models for gaining proficiency in this complex task. The purpose of this study was to assess a novel simulation model for running suture. Methods: Participants were grouped as novice (LSN) or expert (LSE) at laparoscopic suturing based on prior experience and training level. A novel low-cost bench model was developed to simulate laparoscopic intracorporeal continuous closure of a defect. The primary outcome measured was time taken to complete the task. Videos were scored by independent raters for Global Operative Assessment of Laparoscopic Skills (GOALS). Results: Sixteen subjects (7 LSE and 9 LSN) participated in this study. LSE completed the task significantly faster than LSN (430 ± 107 vs 637 ± 164 seconds, P ≤ .05). LSN scored higher on accuracy penalties than LSE (Median 30 vs 0, P ≤ .05). Mean GOALS score was significantly different between the 2 groups (LSE 20.64 ± 2.64 vs LSN 14.28 ± 1.94, P < .001) with good inter-rater reliability (ICC ≥ .823). An aggregate score using the formula: Performance Score = 1200-time(sec)-(accuracy penalties x 10) was significantly different between groups with a mean score of 741 ± 141 for LSE vs 285 ± 167 for LSN ( P < .001). Conclusion A novel bench model for laparoscopic continuous suturing was able to significantly discriminate between laparoscopic experts and novices. This low-cost model may be useful for both training and assessment of laparoscopic continuous suturing proficiency.

How can surgical skills in laparoscopic colon surgery be objectively assessed?-a scoping review

BACKGROUND

In laparoscopic colorectal surgery, higher technical skills have been associated with improved patient outcome. With the growing interest in laparoscopic techniques, pressure on surgeons and certifying bodies is mounting to ensure that operative procedures are performed safely and efficiently. The aim of the present review was to comprehensively identify tools for skill assessment in laparoscopic colon surgery and to assess their validity as reported in the literature.

METHODS

A systematic search was conducted in EMBASE and PubMed/MEDLINE in May 2021 to identify studies examining technical skills assessment tools in laparoscopic colon surgery. Available information on validity evidence (content, response process, internal structure, relation to other variables, and consequences) was evaluated for all included tools.

RESULTS

Fourteen assessment tools were identified, of which most were procedure-specific and video-based. Most tools reported moderate validity evidence. Commonly not reported were rater training, assessment correlation with variables other than training level, and validity reproducibility and reliability in external educational settings.

CONCLUSION

The results of this review show that several tools are available for evaluation of laparoscopic colon cancer surgery, but few authors present substantial validity for tool development and use. As we move towards the implementation of new techniques in laparoscopic colon surgery, it is imperative to establish validity before surgical skill assessment tools can be applied to new procedures and settings. Therefore, future studies ought to examine different aspects of tool validity, especially correlation with other variables, such as patient morbidity and pathological reports, which impact patient survival.

Development of a simulation curriculum to teach and assess advanced laparoscopic suturing skills using telesimulation: a feasibility study

Background

Telesimulation helps overcome limitations in time and local expertise by eliminating the need for the learner and educator to be physically co-located, especially important during COVID-19. We investigated whether teaching advanced laparoscopic suturing (ALS) through telesimulation is feasible, effective, and leads to improved suturing in the operating room (OR).

Methods

In this prospective feasibility study, three previously developed 3D-printed ALS tasks were used: needle handling (NH), suturing under tension (UT), and continuous suturing (CS). General surgery residents (PGY4-5) underwent 1-month of telesimulation training, during which an expert educator at one site remotely trained residents at the other site over 2–3 teaching sessions. Trainees were assessed in the three tasks and in the OR at three time points: baseline(A1), control period(A2), and post-intervention(A3) and completed questionnaires regarding educational value and usability of telesimulation. Paired t-test was used to compare scores between the three assessment points.

Results

Six residents were included. Scores for UT improved significantly post-intervention A3(568 ± 60) when compared to baseline A1(416 ± 133) (p < 0.019). Similarly, scores for CS improved significantly post-intervention A3(756 ± 113) vs. baseline A1(539 ± 211) (p < 0.02). For intraoperative assessments, scores improved significantly post-intervention A3(21 ± 3) when compared to both A1(17 ± 4) (p < 0.018) and A2(18 ± 4) (p < 0.0008). All residents agreed that tasks were relevant to practice, helped improve technical competence, and adequately measured suturing skill. All residents found telesimulation easy to use, had strong educational value, and want the system to be incorporated into their training.

Conclusion

The use of telesimulation for remotely training residents using ALS tasks was feasible and effective. Residents found value in training using the tasks and telesimulation system, and improved ALS skills in the OR. As the pandemic has caused a major structural shift in resident education, telesimulation can be an effective alternative to on-site simulation programs. Future research should focus on how telesimulation can be effectively incorporated into training programs.

Characterization of technical skill progress in a standardized rabbit model for training in laparoscopic duodenal atresia repair

BACKGROUND

Laboratory skills training is an essential step before conducting minimally invasive surgery in clinical practice. Our main aim was to develop an animal model for training in clinically highly challenging laparoscopic duodenal atresia repair that could be useful in establishing a minimum number of repetitions to indicate safe performance of similar interventions on humans.

MATERIALS AND METHODS

A rabbit model of laparoscopic duodenum atresia surgery involving a diamond-shaped duodeno-duodenostomy was designed. This approach was tested in two groups of surgeons: in a beginner group without any previous clinical laparoscopic experience (but having undergone previous standardized dry-lab training, n = 8) and in an advanced group comprising pediatric surgery fellows with previous clinical experience of laparoscopy (n = 7). Each participant performed eight interventions. Surgical time, expert assessment using the Global Operative Assessment of Laparoscopic Skills (GOALS) score, anastomosis quality (leakage) and results from participant feedback questionnaires were analyzed.

RESULTS

Participants in both groups successfully completed all eight surgeries. The surgical time gradually improved in both groups, but it was typically shorter in the advanced group than in the beginner group. The leakage rate was significantly lower in the advanced group in the first two interventions, and it reached its optimal level after five operations in both groups. The GOALS and participant feedback scores showed gradual increases, evident even after the fifth surgery.

CONCLUSIONS

Our data confirm the feasibility of this advanced pediatric laparoscopic model. Surgical time, anastomosis quality, GOALS score and self-assessment parameters adequately quantify technical improvement among the participants. Anastomosis quality reaches its optimal value after the fifth operation even in novice, but uniformly trained surgeons. A minimum number of wet-lab operations can be determined before surgery can be safely conducted in a clinical setting, where the development of further non-technical skills is also required.

Coordinating assessment of spiritual needs: a cross-walk of narrative and psychometric assessment tools used in palliative care

Addressing spiritual needs of patients in healthcare settings improves patient experiences and clinical outcomes; however, non-chaplain providers typically assess spiritual needs differently (quantitative psychometric) than healthcare chaplains (long form narrative) and thus there is little shared language or cross-disciplinary evaluation frameworks across disciplines. This discrepancy impedes the provision of both team-based and patient-centered care. This paper used scoping review methodology to illustrate the overlap between narrative and psychometric assessment tools, comparing four narrative tools against eight psychometric tools. The SpNQ-120 and Brief RCOPE demonstrated consistent domain coverage across the four chaplain narrative tools. This work provides preliminary resources to aid clinicians and researchers in choosing an appropriate tool. Additionally, for those who do not work closely with chaplains, it provides a sense of what domains chaplains prioritize, from their professional and lived experience, in assessing the spiritual life of the patient. This improves interdisciplinary communication, and therefore, patient care.

Validation of a Novel Needle Holder to Train Advanced Laparoscopy Skills to Novices in a Simulator Environment

Background. Our aim was to determine if a newly designed Najar needle holder (NNH) shortens the time for novices to improve advanced laparoscopy (AL) techniques (suturing/knot tying), compared with a conventional macro needle holder (MNH) in a simulator. Furthermore, we aimed to validate a new video scoring system determining AL skills. Methods. Forty-six medical students performed identical surgical tasks in a prospective, crossover study evaluating AL skills (NNH vs MNH). All subjects performed a double-throw knot, 2 single-throw knots following 3 running sutures in the Simball Box (SB) simulator. After resting, subjects switched needle holders. All tasks were videotaped and analyzed using SB software and by 2 independent reviewers using the Objective Video Evaluation Scoring Table (OVEST). Trial performance expressed as SB Overall Score (SBOS) and OVEST. Results. In the group starting with NNH (followed by MNH) OVEST was consistently high during both trials (median = 12.5, range = 6.5-18.0, and median = 13.5, range = 6.5-21.0; P = .2360). However, in the group starting with MNH, OVEST improved significantly when the participants changed to NNH (median = 10.0, range = 2.5-19.5, vs median = 14.5, range = 4.5-18.0; P = .0003); an improvement was also found with SBOS (median = 37%, range = 27% to 92%, vs median = 48%, range = 34% to 70%; P = .0289). In both trials, both independent reviewers' OVEST measures correlated well: Trial 1: β = 0.97, P < .0001; and Trial 2: β = 0.95, P < .0001. A correlation also existed between SBOS and OVEST in both trials (β = 2.1, P < .0001; and β = 1.9, P = .0002). Conclusions. This study indicates a significantly higher improvement in laparoscopic suturing skills in novices training AL skills using NNH compared with MNH. Starting early, AL training in novices using NNH is a feasible option. Furthermore, OVEST used in experimental settings as an evaluation tool is comparable with the validated SBOS.