Objective assessment of gynecologic laparoscopic skills using the LapSimGyn virtual reality simulator

Objective assessment tools in laparoscopic or robotic-assisted gynecological surgery: A systematic review

INTRODUCTION

There is a growing emphasis on proficiency-based progression within surgical training. To enable this, clearly defined metrics for those newly acquired surgical skills are needed. These can be formulated in objective assessment tools. The aim of the present study was to systematically review the literature reporting on available tools for objective assessment of minimally invasive gynecological surgery (simulated) performance and evaluate their reliability and validity.

MATERIAL AND METHODS

A systematic search (1989-2022) was conducted in MEDLINE, Embase, PubMed, Web of Science in accordance with PRISMA. The trial was registered with the Prospective Register of Systematic Reviews (PROSPERO) ID: CRD42022376552. Randomized controlled trials, prospective comparative studies, prospective single-group (with pre- and post-training assessment) or consensus studies that reported on the development, validation or usage of assessment tools of surgical performance in minimally invasive gynecological surgery, were included. Three independent assessors assessed study setting and validity evidence according to a contemporary framework of validity, which was adapted from Messick's validity framework. Methodological quality of included studies was assessed using the modified medical education research study quality instrument (MERSQI) checklist. Heterogeneity in data reporting on types of tools, data collection, study design, definition of expertise (novice vs. experts) and statistical values prevented a meaningful meta-analysis.

RESULTS

A total of 19 746 titles and abstracts were screened of which 72 articles met the inclusion criteria. A total of 37 different assessment tools were identified of which 13 represented manual global assessment tools, 13 manual procedure-specific assessment tools and 11 automated performance metrices. Only two tools showed substantive evidence of validity. Reliability and validity per tool were provided. No assessment tools showed direct correlation between tool scores and patient related outcomes.

CONCLUSIONS

Existing objective assessment tools lack evidence on predicting patient outcomes and suffer from limitations in transferability outside of the research environment, particularly for automated performance metrics. Future research should prioritize filling these gaps while integrating advanced technologies like kinematic data and AI for robust, objective surgical skill assessment within gynecological advanced surgical training programs.

Variable practice is superior to self-directed training for laparoscopic simulator training: a randomized trial

BACKGROUND

Mastering laparoscopy is challenging-it requires specific psychomotor skills which are difficult to obtain in the operating room without potentially compromising patient safety. Proficiency-based training programs using virtual reality simulators allow novices to practice and develop their skills in a patient-safe learning environment. Variable practice leads to stronger retention and skills transfer in a non-surgical setting. The objective of this trial was to investigate if variable practice was superior to self-directed training.

METHODS

A randomized trial where participants (n = 36) were randomized to proficiency-based laparoscopic simulator training of basic skills using either variable practice or self-directed training, followed by a transfer test with proficiency-based training on a procedural task (a salpingectomy). All participants returned after a period of 3-5 weeks to perform a retention test.

RESULTS

The mean time to proficiency for the basic skills tasks were 119 min (SD: 93) for the variable practice group versus 182 min (SD: 46) for the self-directed training group (p = 0.015). The time to reach proficiency during the transfer test was 103 min (SD: 57) versus 183 min (SD: 64) for the variable practice group versus the self-directed training group, respectively (p < 0.001). The mean time to proficiency for the retention test was 51 min (SD: 26) and 109 min (SD: 53) for the variable practice group and self-directed training group, respectively (p < 0.001).

CONCLUSION

Variable practice is superior to self-directed training for proficiency-based laparoscopic training. With variable time to practice proficiency is reduced, there is higher transfer to a procedural task, and retention is improved.

Training simulator efficacy in developing thoracic and general surgical skills in a residency programme: a pilot study

OBJECTIVES

Virtual training simulators have been introduced in several surgical disciplines to improve residents' abilities. Through the use of the LapSim® virtual training simulator (Surgical Science, Göteborg, Sweden), this study aims to plan an effective learning path in minimally invasive thoracic and general surgery.

METHODS

All thoracic and general surgery trainees in their 1st and 2nd year of residency at the University of Insubria were enrolled and randomized into 2 groups: residents undergoing an intensive twice-a-week virtual training programme (group A: n = 8) and those undergoing a once-weekly non-intensive virtual training programme (group B: n = 9). The virtual training programme was divided into 4 modules, each of 12 weeks. In the 1st module, trainees repeated grasping, cutting, clip application, lifting and grasping, and fine dissection exercises during each training session. Seal-and-cut exercise was performed as the initial and final test. Data on surgical manoeuvres (time and on mistakes) were collected; intra- and inter-group comparisons were planned.

RESULTS

No significant differences were observed between groups A and B at the 1st session, confirming that the 2 groups had similar skills at the beginning. After 12 weeks, both groups showed improvements, but comparing data between initial and final test, only Group A registered a significant reduction in total time (P-value = 0.0015), left (P-value = 0.0017) and right (P-value = 0.0186) instrument path lengths, and in left (P-value = 0.0010) and right (P-value = 0.0073) instrument angular path lengths, demonstrating that group A acquired greater precision in surgical manoeuvres.

CONCLUSIONS

Virtual simulator training programme performed at least twice a week was effective for implementing basic surgical skills required for the trainee's professional growth. Additional virtual training modules focused on more complex exercises are planned to confirm these preliminary results.

No difference between using short and long intervals for distributed proficiency-based laparoscopy simulator training: a randomized trial

BACKGROUND

Simulation-based training is increasingly used to acquire basic laparoscopic skills. Multiple factors can influence training, e.g., distributed practice is superior to massed practice in terms of efficiency. However, the optimal interval between training sessions is unclear. The objective of this trial was to investigate if shorter intervals between sessions are more efficient than longer intervals during proficiency-based laparoscopy simulator training.

METHODS

A randomized simulation-based trial where medical students (n = 39) were randomized to proficiency-based training with either 1-2 days (intervention group) or 6-8 days (control group) between training sessions. Both groups practiced a series of basic tasks and a procedural module until proficiency level on the LapSim® simulator. Both groups were given instructor feedback upon request. After reaching proficiency, participants were invited back for a retention test 3-5 weeks later and practiced the same tasks to proficiency again.

RESULTS

The mean time to reach proficiency during training was 291 (SD 89) and 299 (SD 89) min in the intervention and control group, respectively (p = 0.81). During the retention test, the mean time to reach proficiency was 94 (SD 53) and 96 (SD 39) minutes in the intervention and control groups, respectively (p = 0.91).

CONCLUSION

We found no difference whether practicing with shorter intervals or longer intervals between training sessions when examining time to proficiency or retention.

Training and assessment using the LapSim laparoscopic simulator: a scoping review of validity evidence

INTRODUCTION

The LapSim (Surgical Science, Sweden) laparoscopic simulator is a high-fidelity virtual reality simulator for use in endoscopic surgical training. This review critiques the current validity evidence for the LapSim laparoscopic simulator, specifically with respect to its potential use as a tool and method of training and assessment in surgery.

METHODS

A scoping review of the MEDLINE (PubMed), EMBASE, Cochrane and Web of Science databases was conducted in accordance with PRISMA guidelines (2020)-scoping review extension. Articles were included if they presented validity evidence for the use of the LapSim in operative skill training or assessment, in accordance with Messick's validity framework. European Association of Endoscopic Surgeons (EAES) guidelines (2005) were used to provide recommendations for the use of the LapSim in operative performance training and assessments.

RESULTS

Forty-nine articles were included. An EAES level 2 recommendation was provided with regard to the internal consistency reliability of automated performance metrics in assessing performance. An EAES recommendation of 2 was awarded with respect to the ability of the LapSim to discriminate based on case volume and overall laparoscopic experience (relationships with other variables). Performance assessment metrics on the LapSim correlate with improved performance in the operating room (EAES level of recommendation 1-consequential validity).

CONCLUSION

The LapSim has accumulated substantial evidence supporting the validity of its use in surgical training and assessment. Future studies should explore the relationship between the achievement of performance benchmarks on the LapSim and subsequent patient outcomes, and interrogate the benefits of implementing virtual reality simulation training and assessment curricula in post-graduate surgical training programmes.

Haptic simulators accelerate laparoscopic simulator training, but skills are not transferable to a non-haptic simulator: a randomized trial

BACKGROUND

Laparoscopy requires specific psychomotor skills and can be challenging to learn. Most proficiency-based laparoscopic training programs have used non-haptic virtual reality simulators; however, haptic simulators can provide the tactile sensations that the surgeon would experience in the operating room. The objective was to investigate the effect of adding haptic simulators to a proficiency-based laparoscopy training program.

METHODS

A randomized controlled trial was designed where residents (n = 36) were randomized to proficiency-based laparoscopic simulator training using haptic or non-haptic simulators. Subsequently, participants from the haptic group completed a follow-up test, where they had to reach proficiency again using the non-haptic simulator. Participants from the non-haptic group returned to train until reaching proficiency again using the non-haptic simulator.

RESULTS

Mean completion times during the intervention were 120 min (SD 38.7 min) and 183 min (SD 66.3 min) for the haptic group and the non-haptic group, respectively (p = 0.001). The mean times to proficiency during the follow-up test were 107 min (SD 41.0 min) and 58 min (SD 23.7 min) for the haptic and the non-haptic group, respectively (p < 0.001). The haptic group was not faster to reach proficiency in the follow-up test than during the intervention (p = 0.22). In contrast, the non-haptic group reached the required proficiency level significantly faster in the follow-up test (p < 0.001).

CONCLUSION

Haptic virtual reality simulators reduce the time to reach proficiency compared to non-haptic simulators. However, the acquired skills are not transferable to the conventional non-haptic setting.

Correcting the Fundamentals of Laparoscopic Surgery "Illusion of Validity" in Laparoscopic Vaginal Cuff Suturing

STUDY OBJECTIVE

The "illusion of validity" is a cognitive bias in which the ability to interpret and predict surgical performance accurately is overestimated. To address this bias, we assessed participants comparing fundamentals of laparoscopic surgery (FLS) and non-FLS tasks with cadaveric vaginal cuff suturing to determine the most representative simulation task for laparoscopic vaginal cuff suturing.

DESIGN

Validity (Messick framework) study comparing FLS and non-FLS tasks with cadaveric vaginal cuff suturing.

SETTING

Simulation center cadaver laboratory.

PARTICIPANTS

Obstetrics and gynecology residents (n = 21), minimally invasive gynecologic surgery fellows (n = 3), gynecologic surgical subspecialists (n = 4), general obstetrician/gynecologists (n = 10).

INTERVENTIONS

Tasks included a simulated vaginal cuff (ipsilateral port placement), needle passage through a metal eyelet loop (contralateral and ipsilateral), and intracorporeal knot tying (contralateral and ipsilateral). Simulation task times were compared with the placement of the first cadaveric vaginal cuff suture time, as well as the in-person and blinded Global Operative Assessment of Laparoscopic Skills (GOALS) score ("relations to other variables" validity evidence). Statistical analyses included Spearman's test of correlation (continuous and ordinal variables) or Wilcoxon rank sum test (categoric variables).

MEASUREMENTS AND MAIN RESULTS

There was a stronger association with cadaver cuff suturing time for simulated vaginal cuff suturing time (r = 0.73, p <.001) compared with FLS intracorporeal contralateral suturing time (r = 0.54, p <.001). Additional measures associated with cadaveric performance included subspecialty training (median: 82 vs 185 seconds, p = .002), number of total laparoscopic hysterectomies (r = -0.53, p <.001), number of laparoscopic cuff closures (r = -0.61, p <.001), number of simulated laparoscopic suturing experiences (r = -0.51, p <.001), and eyelet contralateral time (r = 0.52, p <.001). Strong agreement between the in-person and blinded GOALS (intraclass correlation coefficient = 0.80) supports response process evidence. Correlations of cadaver cuff time with in-person (Spearman's r = -0.84, p <.001) and blinded GOALS (r = -0.76, p <.001) supports relations to other variables evidence CONCLUSION: The weaker correlation between FLS suturing and cadaver cuff suturing compared with a simulated vaginal cuff model may lead to an "illusion of validity" for assessment in gynecology. Since gynecology specific validity evidence has not been well established for FLS, we recommend prioritizing the use of a simulated vaginal cuff suturing assessment in addition to FLS.

Using Low-Cost Virtual Reality Simulation to Build Surgical Capacity for Cervical Cancer Treatment

PURPOSE

Worldwide, more than 80% of people diagnosed with cancer will require surgery during their disease course, but only 5% to 20% of low- and middle-income countries have access to safe, affordable, and timely surgery. Developing surgical oncology skills requires significant time and mentoring. Virtual reality (VR) simulators can reduce the time required to master surgical procedures but are prohibitively expensive. We sought to determine whether a VR simulator using low-cost computer gaming equipment could train novice surgeons in Africa to perform a virtual radical abdominal (open) hysterectomy (RAH).

METHODS

Our RAH VR simulator used the Oculus Rift (Oculus VR, Menlo Park, CA), a VR headset with hand controllers that costs less than $1,500. Surgical novices learned to perform five key steps of a virtual RAH. We measured and identified predictors of movement and time efficiency for the simulation.

RESULTS

Ten novice surgeons in Lusaka, Zambia, enrolled in the study. Movement and time efficiency greatly improved over time. Independent predictors of movement efficiency were number of simulations, surgical experience, and time since college graduation. Independent predictors of time efficiency were number of simulations, surgical experience, days between simulation sessions, age, sex, and an interaction between number of simulations and surgical experience.

CONCLUSION

Low-cost VR may be an effective tool to help surgical novices learn complex surgical oncology procedures. If learning to perform VR surgical procedures with low-cost hardware leads to faster mastery of surgical procedures in the operating room, low-cost VR may represent one of the solutions to increasing access to surgical cancer care globally.

Development and validation of a general and easy assessable scoring system for laparoscopic skills using a virtual reality simulator

Objectives

To develop and validate a scoring system for laparoscopic skills for five specific tasks on a virtual reality simulator.

Study design

A longitudinal, experimental, non-randomised study including 30 gynecologists and gynecological trainees at three hospitals. The participants were categorized as inexperienced (Group 1), moderately experienced (Group 2), and experienced (Group 3).

The study participants performed ten repetitions of three basic skill tasks, a salpingectomy and a laparoscopic supracervical hysterectomy on a virtual reality simulator. Assessment of skills was based on time, error parameters and economy of movements measured by the simulator. We used the results (mean and SD for each parameter in all tasks) of the four last repetitions performed by the experienced gynecologists as the basic for the scoring system. Performance equal to, and higher than, this mean score gave 2 points. A decrease of 1 SD from the mean gave 1 point. Every score below gave 0 points. The mean score for the inexperienced, moderately experienced and experienced study participants was compared.

Results

The mean scores in Task 1 were 3.4 (SD 0.6) in Group 1, 3.4 (SD 0.6) in Group 2 and 5.1 (SD 1.1) in Group 3, respectively. There was a statistically significant difference in score between Group 1 and 3 (p = 0.01), and group 2 and 3 (p = 0.01). In Task 2 no statistical significant differences were found. In Task 3, the total mean scores were 1.7 (SD 0.7) in Group 1, 1.9 (SD 0.9) in Group 2 and 2.8 (SD 0.5) in Group 3, respectively. The difference in score between study groups was statistically significant when comparing Group 1 and Group 3 (p < 0.01) and Group 2 and Group 3 (p = 0.02).

In Task 4, the difference in used time between group 1 and 3 was statistically significant (p = 0.03). In task 5 there was a significant difference in performance score between group 1 and 3 (p = 0.01).

Conclusions

There was significant difference in scores between the experienced and the inexperienced gynecologist in four out of five tasks.

The scoring system is easy assessable and can be used for summative and formative feedback in proficiency-based assessment.

Development of a structured virtual reality curriculum for laparoscopic appendicectomy

BACKGROUND

Laparoscopic appendicectomy (LA) is a common surgical emergency procedure mainly performed by trainees. The aim was to develop a step-wise structured virtual reality (VR) curriculum for LA to allow junior surgeons to hone their skills in a safe and controlled environment.

METHODS

A prospective randomized study was designed using a high-fidelity VR simulator. Thirty-five novices and 25 experts participated in the assessment and their performances were compared to assess construct validity. Learning curve analysis was performed.

RESULTS

Five of the psychomotor tasks and all appendicectomy tasks showed construct validity. Learning was demonstrated in the majority of construct-valid tasks. A novel goal-directed VR curriculum for LA was constructed.

CONCLUSIONS

A step-wise structured VR curriculum for LA is proposed with a framework which includes computer generated metrics and supports deliberate practice, spacing intervals, human instruction/feedback and assessment. Future study should test the feasibility of its implementation and transferability of acquired skill.

How do visual-spatial and psychomotor abilities influence clinical performance in periodontal plastic surgery?

AIM

We want to evaluate the relationship of self-assessed experience and proficiency, manual dexterity and visual-spatial ability with surgical performance.

MATERIAL AND METHODS

A total of 26 professionals were included in the study which consisted of four parts: (a) self-assessment by a questionnaire regarding proficiency and experience, (b) evaluation of visual-spatial ability, (c) testing of manual dexterity assessed by validated psychomotor tests and (d) evaluation of surgical performance by Objective Structured Assessment of Technical Skills (OSATS).

RESULTS

Self-assessed proficiency and experience levels did not correlate with objectively evaluated surgical performances (OSATS). However, low-level visual-spatial ability tests strongly correlated with OSATS while intermediate- and high-level tests did not. No correlation was found between psychomotor ability and clinical performance.

CONCLUSIONS

Self-assessed proficiency is not a good predictor for surgical performance as experts tend to be overconfident. To evaluate and predict surgical performance, visual-spatial ability tests seem to be more appropriate than measuring manual dexterity which failed to correlate with the surgical outcome.

The Heidelberg VR Score: development and validation of a composite score for laparoscopic virtual reality training

INTRODUCTION

Virtual reality (VR-)trainers are well integrated in laparoscopic surgical training. However, objective feedback is often provided in the form of single parameters, e.g., time or number of movements, making comparisons and evaluation of trainees' overall performance difficult. Therefore, a new standard for reporting outcome data is highly needed. The aim of this study was to create a weighted, expert-based composite score, to offer simple and direct evaluation of laparoscopic performance on common VR-trainers.

MATERIALS AND METHODS

An integrated analytic hierarchy process-Delphi survey was conducted with 14 international experts to achieve a consensus on the importance of different skill categories and parameters in evaluation of laparoscopic performance. A scoring algorithm was established to allow comparability between tasks and VR-trainers. A weighted composite score was calculated for basic skills tasks and peg transfer on the LapMentor™ II and III and validated for both VR-trainers.

RESULTS

Five major skill categories (time, efficiency, safety, dexterity, and outcome) were identified and weighted in two Delphi rounds. Safety, with a weight of 67%, was determined the most important category, followed by efficiency with 17%. The LapMentor™-specific score was validated using 15 (14) novices and 9 experts; the score was able to differentiate between both groups for basic skills tasks and peg transfer (LapMentor™ II: Exp: 86.5 ± 12.7, Nov. 52.8 ± 18.3; p < 0.001; LapMentor™ III: Exp: 80.8 ± 7.1, Nov: 50.6 ± 16.9; p < 0.001).

CONCLUSION

An effective and simple performance measurement was established to propose a new standard in analyzing and reporting VR outcome data-the Heidelberg virtual reality (VR) score. The scoring algorithm and the consensus results on the importance of different skill aspects in laparoscopic surgery are universally applicable and can be transferred to any simulator or task. By incorporating specific expert baseline data for the respective task, comparability between tasks, studies, and simulators can be achieved.

Validity evidence for procedural competency in virtual reality robotic simulation, establishing a credible pass/fail standard for the vaginal cuff closure procedure

BACKGROUND

The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum.

METHODS

Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module 'Guided Vaginal Cuff Closure' six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach's alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups' method.

RESULTS

The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach's alpha). The experienced surgeons' mean composite score for all six repetitions were significantly better than the novice surgeons' (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives).

CONCLUSION

Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

Cognitive and Technical Skill Assessment in Surgical Education: a Changing Horizon

Assessment is an integral component of training and credentialing surgeons for practice. Traditional methods of cognitive and technical appraisal are well established but have clear shortcomings. This review outlines the components of the surgical care assessment model, identifies the deficits of current evaluation techniques, and discusses novel and emerging technologies that attempt to ameliorate this educational void.

Preoperative Practice Paired With Instructor Feedback May Not Improve Obstetrics-Gynecology Residents' Operative Performance

BACKGROUND

There is evidence that preoperative practice prior to surgery can improve trainee performance, but the optimal approach has not been studied.

OBJECTIVE

We sought to determine if preoperative practice by surgical trainees paired with instructor feedback improved surgical technique, compared to preoperative practice or feedback alone.

METHODS

We conducted a randomized controlled trial of obstetrics-gynecology trainees, stratified on a simulator-assessed surgical skill. Participants were randomized to preoperative practice on a simulator with instructor feedback (PPF), preoperative practice alone (PP), or feedback alone (F). Trainees then completed a laparoscopic salpingectomy, and the operative performance was evaluated using an assessment tool.

RESULTS

A total of 18 residents were randomized and completed the study, 6 in each arm. The mean baseline score on the simulator was comparable in each group (67% for PPF, 68% for PP, and 70% for F). While the median score on the assessment tool for laparoscopic salpingectomy in the PPF group was the highest, there was no statistically significant difference in assessment scores for the PPF group (32.75; range, 15-36) compared to the PP group (14.5; range, 10-34) and the F group (21.25; range, 10.5-32). The interrater correlation between the video reviewers was 0.87 (95% confidence interval 0.70-0.95) using the intraclass correlation coefficient.

CONCLUSIONS

This study suggests that a surgical preoperative practice with instructor feedback may not improve operative technique compared to either preoperative practice or feedback alone.

Evaluation of Procedural Simulation as a Training and Assessment Tool in General Surgery-Simulating a Laparoscopic Appendectomy

BACKGROUND

Laparoscopic appendectomy is a commonly performed surgical procedure, but few training models have been described for it. We examined a virtual reality module for practising a laparoscopic appendectomy.

METHODS

A prospective cohort study with the following 3 groups of surgeons (n = 45): novices (0 procedures), intermediates (10-50 procedures), and experienced (>100 procedures). After being introduced to the simulator and 1 familiarization attempt on the procedural module, the participants practiced the module 20 times. Movements, task time, and procedure-specific parameters were compared over time.

RESULTS

The time and movement parameters were significantly different during the first attempt, and more experienced surgeons used fewer movements and less time than novices (p < 0.01), although only 2 parameters were significantly different between novices and intermediates. All 3 groups improved significantly over 20 attempts (p < 0.0001). The intraclass correlation coefficient varied between 0.55 and 0.68 and did not differ significantly between the 3 groups (p > 0.05). When comparing novices with experienced surgeons, novices had a higher risk of burn damage to cecum (odds ratio [OR] = 3.0 [95% CI: 1.3; 7.0] p = 0.03), pressure damage to appendix (OR = 3.1 [95% CI: 2.0; 4.9] p < 0.0001), and grasping of the appendix (OR = 2.9 [95% CI: 1.8; 4.7] p < 0.0001). The risk of causing a perforation was not significantly different among the different experience levels (OR = 1.9 [95% CI: 0.9; 3.8] p = 0.14). Only 3 out of 5 error parameters differed significantly when comparing novices and experienced surgeons. Similarly, when comparing intermediates and novices, it was only 2 of the parameters that differed.

DISCUSSION

The simulator module for practising laparoscopic appendectomy may be useful as a training tool, but further development is required before it can be used for assessment purposes. Procedural simulation may demonstrate more variation for movement parameters, and future research should focus on developing better procedure-specific parameters.

Using virtual reality simulation to assess competence in video-assisted thoracoscopic surgery (VATS) lobectomy

BACKGROUND

The societies of thoracic surgery are working to incorporate simulation and competency-based assessment into specialty training. One challenge is the development of a simulation-based test, which can be used as an assessment tool. The study objective was to establish validity evidence for a virtual reality simulator test of a video-assisted thoracoscopic surgery (VATS) lobectomy of a right upper lobe.

METHODS

Participants with varying experience in VATS lobectomy were included. They were familiarized with a virtual reality simulator (LapSim^®) and introduced to the steps of the procedure for a VATS right upper lobe lobectomy. The participants performed two VATS lobectomies on the simulator with a 5-min break between attempts. Nineteen pre-defined simulator metrics were recorded.

RESULTS

Fifty-three participants from nine different countries were included. High internal consistency was found for the metrics with Cronbach's alpha coefficient for standardized items of 0.91. Significant test-retest reliability was found for 15 of the metrics (p-values <0.05). Significant correlations between the metrics and the participants VATS lobectomy experience were identified for seven metrics (p-values <0.001), and 10 metrics showed significant differences between novices (0 VATS lobectomies performed) and experienced surgeons (>50 VATS lobectomies performed). A pass/fail level defined as approximately one standard deviation from the mean metric scores for experienced surgeons passed none of the novices (0 % false positives) and failed four of the experienced surgeons (29 % false negatives).

CONCLUSION

This study is the first to establish validity evidence for a VATS right upper lobe lobectomy virtual reality simulator test. Several simulator metrics demonstrated significant differences between novices and experienced surgeons and pass/fail criteria for the test were set with acceptable consequences. This test can be used as a first step in assessing thoracic surgery trainees' VATS lobectomy competency.

Take-home training in a simulation-based laparoscopy course

BACKGROUND

Simulation training can prepare trainees for clinical practice in laparoscopic surgery. Training on box trainers allows for simulation training at home, which studies have shown to be a feasible method of training. However, little research has been conducted into how to make it a more efficient method of training. Our aim was to investigate how box trainers are used in take-home training to help guide the design of take-home training courses.

METHODS

This study was designed using a mixed methods approach. Junior doctors participating in a laparoscopy curriculum, which included practising at home on box trainers, were invited. Quantitative data on training patterns was collected from logbooks. Qualitative data on the use of box trainers was retrieved from focus groups and individual interviews.

RESULTS

From logbooks, we found that 14 out of 18 junior doctors mixed their training modalities, and four practised first on box trainers then on virtual reality simulators. Twelve practised only at home, while five practised at both places and one practised solely at the simulation centre. After a delayed start, most practised for some time, then had a period without training and then started training again towards the end of the course. We found that the themes of the interviews were: training method, training pattern, feedback and self-regulation. Participants identified the lack of feedback as challenging but described how self-rating provided direction during unsupervised training. Mandatory training elements affected when and how much participants practised.

CONCLUSIONS

When participants practised at home, they took an individualised approach to training. They mixed their training at home with training at the simulation centre. Participants practised at the beginning and towards the end of the course. Self-rating helped to guide unsupervised training where feedback was not accessible. Curricular requirements and testing determined when and how much participants practised.

Validating a standardized laparoscopy curriculum for gynecology residents: a randomized controlled trial

BACKGROUND

Residency programs struggle with integrating simulation training into curricula, despite evidence that simulation leads to improved operating room performance and patient outcomes. Currently, there is no standardized laparoscopic training program available for gynecology residents.

OBJECTIVE

The purpose of this study was to develop and validate a comprehensive ex vivo training curriculum for gynecologic laparoscopy.

STUDY DESIGN

In a prospective, single-blinded randomized controlled trial (Canadian Task Force Classification I) postgraduate year 1 and 2 gynecology residents were allocated randomly to receive either conventional residency training or an evidence-based laparoscopy curriculum. The 7-week curriculum consisted of cognitive didactic and interactive sessions, low-fidelity box trainer and high-fidelity virtual reality simulator technical skills, and high-fidelity team simulation. The primary outcome measure was the technical procedure score at laparoscopic salpingectomy with the use of the objective structured assessment of laparoscopic salpingectomy tool. Secondary outcome measures related to performance in multiple-choice questions and technical performance at box trainer and virtual reality simulator tasks. A sample size of 10 residents per group was planned (n = 20). Results are reported as medians (interquartile ranges), and data were compared between groups with the Mann-Whitney U, chi-square, and Fisher's exact tests (P ≤ .05).

RESULTS

In July 2013, 27 residents were assigned randomly (14 curriculum, 13 conventional). Both groups were similar at baseline. Twenty-one residents (10 curriculum, 11 conventional) completed the surgical procedure-based assessment in the operating room (September to December 2013). Our primary outcome indicated that curriculum-trained residents displayed superior performance at laparoscopic salpingectomy (P = .043). Secondary outcomes demonstrated that curriculum-trained residents had higher performance scores on the cognitive multiple-choice questions (P < .001), the nontechnical skills multiple-choice questions (P = .016), box trainer task time (P < .001), and all virtual reality simulator parameters.

CONCLUSION

Participation in a comprehensive simulation-based training curriculum for gynecologic laparoscopy leads to a superior improvement in knowledge and technical performance in the operating room compared with conventional residency training.

Individualized feedback during simulated laparoscopic training:a mixed methods study

OBJECTIVE

This study aimed to explore the value of individualized feedback on performance, flow and self-efficacy during simulated laparoscopy. Furthermore, we wished to explore attitudes towards feedback and simulator training among medical students.

METHODS

Sixteen medical students were included in the study and randomized to laparoscopic simulator training with or without feedback. A teacher provided individualized feedback continuously throughout the procedures to the target group. Validated questionnaires and scales were used to evaluate self-efficacy and flow. The Mann-Whitney U test was used to evaluate differences between groups regarding laparoscopic performance (instrument path length), self-efficacy and flow. Qualitative data was collected by group interviews and interpreted using inductive thematic analyses.

RESULTS

Sixteen students completed the simulator training and questionnaires. Instrument path length was shorter in the feedback group (median 3.9 m; IQR: 3.3-4.9) as compared to the control group (median 5.9 m; IQR: 5.0-8.1), p<0.05. Self-efficacy improved in both groups. Eleven students participated in the focus interviews. Participants in the control group expressed that they had fun, whereas participants in the feedback group were more concentrated on the task and also more anxious. Both groups had high ambitions to succeed and also expressed the importance of getting feedback. The authenticity of the training scenario was important for the learning process.

CONCLUSIONS

This study highlights the importance of individualized feedback during simulated laparoscopy training. The next step is to further optimize feedback and to transfer standardized and individualized feedback from the simulated setting to the operating room.

Designing a Standardized Laparoscopy Curriculum for Gynecology Residents: A Delphi Approach

BACKGROUND

Evidence suggests that simulation leads to improved operative skill, shorter operating room time, and better patient outcomes. Currently, no standardized laparoscopy curriculum exists for gynecology residents.

OBJECTIVE

To design a structured laparoscopy curriculum for gynecology residents using Delphi consensus methodology.

METHODS

This study began with Delphi methodology to determine expert consensus on the components of a gynecology laparoscopic skills curriculum. We generated a list of cognitive content, technical skills, and nontechnical skills for training in laparoscopic surgery, and asked 39 experts in gynecologic education to rate the items on a Likert scale (1-5) for inclusion in the curriculum. Consensus was predefined as Cronbach α of ≥0.80. We then conducted another Delphi survey with 9 experienced users of laparoscopic virtual reality simulators to delineate relevant curricular tasks. Finally, a cross-sectional design defined benchmark scores for all identified tasks, with 10 experienced gynecologic surgeons performing the identified tasks at basic, intermediate, and advanced levels.

RESULTS

Consensus (Cronbach α=0.85) was achieved in the first round of the curriculum Delphi, and after 2 rounds (Cronbach α=0.80) in the virtual reality curriculum Delphi. Consensus was reached for cognitive, technical, and nontechnical skills as well as for 6 virtual reality tasks. Median time and economy of movement scores defined benchmarks for all tasks.

CONCLUSIONS

This study used Delphi consensus to develop a comprehensive curriculum for teaching gynecologic laparoscopy. The curriculum conforms to current educational standards of proficiency-based training, and is suggested as a standard in residency programs.

Medical students as facilitators for laparoscopic simulator training

BACKGROUND

Teaching basic clinical skills to student peers and residents by medical students has previously been shown effective. This study examines if medical students can facilitate laparoscopic procedural tasks to residents using a virtual reality simulator.

METHODS

This was a retrospective study comparing 2 groups of gynecology residents. One group was instructed by 2 student facilitators, and a resident facilitator instructed the other group. Facilitators in both the groups were experienced in laparoscopic simulator training. The outcome measures were time and repetitions to complete a laparoscopic simulator training program.

RESULTS

A total of 51 residents participated: 15 in the student-facilitated group and 36 in the resident-facilitated group after 18 and 7 residents dropped out, respectively. The student-facilitated group averaged 230 minutes, while the resident-facilitated group averaged 200 minutes (p = 0.724). To complete the final examination module, the student-facilitated group required 19.5 repetitions vs 39.2 in the resident-facilitated group (p = 0.007).

CONCLUSION

Our findings suggest that students can be used as facilitators as an alternative to residents when practicing on a laparoscopic virtual reality simulator.

Gynecology resident laparoscopy training: present and future

Simulator education is essential to surgical training and it should be a requirement at all training programs across North America. Yet, in a survey of North American obstetrics and gynecology program directors (response rate 52%), we found that while 73% (n = 98) of programs teach laparoscopic skills, only 59% (n = 81) were satisfied with their curriculum. Most programs lacked standard setting in the form of theoretical examinations (94%, n = 127) or skills assessments (91%, n = 123) prior to residents performing surgery on patients in the operating room. Most programs (97%, n = 131) were interested in standardizing laparoscopy education by implementing a common curriculum. We present 3 core recommendations to ensure that gynecologists across North America are receiving adequate training in gynecologic laparoscopic surgery as residents: (1) uniform simulator education should be implemented at all training programs across North American residency programs; (2) a standardized curriculum should be developed using evidence-based techniques; and (3) standardized assessments should take place prior to operating room performance and specialty certification. Future collaborative research initiatives should focus on establishing the content of a standardized laparoscopy curriculum for gynecology residents utilizing a consensus method approach.

Effect of instructor feedback on skills retention after laparoscopic simulator training: follow-up of a randomized trial

BACKGROUND

Instructor feedback reduces the number of repetitions and time to reach proficiency during laparoscopic simulator training. The objective of this study was to examine the effect of instructor feedback on long-term skill retention.

METHODS

A 6-month follow-up of a randomized trial. Participants were surgical novices (medical students). All participants (n = 99) initially practiced a laparoscopic salpingectomy on the LapSim virtual reality simulator to proficiency. The intervention group could request instructor feedback, whereas the control group could not. After 6 months, the participants (n = 65) practiced on the simulator until they reached proficiency again. The primary outcomes were the total time and the number of repetitions.

RESULTS

Initially, the intervention group used significantly fewer repetitions (29 vs 65, p < 0.0005) and less total training time (162 vs 342 min, p < 0.0005) than the control group to reach the proficiency level. At follow-up, both the groups used an equal number of repetitions (21 vs 20, p = 0.72) and time (83 vs 73 min, p = 0.37) to reach the same proficiency level.

CONCLUSIONS

Instructor feedback during proficiency-based laparoscopic simulator training does not affect the long-term retention of skills.

Laparoscopic skills and cognitive function are not affected in surgeons during a night shift

OBJECTIVE

To monitor surgeons' performance and cognition during night shifts.

DESIGN

Surgeons were monitored before call and on call (17-hour shift). Psychomotor performance was assessed by laparoscopic simulation and cognition by the d2 test of attention. The surgeons performed the laparoscopic simulation and the d2 test of attention at 8 a.m. before call and at 4 a.m. on call. Sleep was measured by wrist actigraphy and sleepiness by the Karolinska sleepiness scale.

SETTING

Department of Surgery at Herlev Hospital, Denmark.

PARTICIPANTS

Overall, 30 interns, residents, and attending surgeons were included and completed the study. One participant was subsequently excluded owing to myxedema.

RESULTS

The surgeons slept significantly less on call than before call. There was increasing sleepiness on call; however, no significant differences were found in the precall laparoscopic simulation values compared with on-call values. The d2 test of attention showed significantly improved values on call compared with before call.

CONCLUSION

Sleep deprivation during a 17-hour night shift did not impair surgeons' psychomotor or cognitive performance.

Evaluation of procedural learning transfer from a virtual environment to a real situation: a case study on tank maintenance training

Virtual reality opens new opportunities for operator training in complex tasks. It lowers costs and has fewer constraints than traditional training. The ultimate goal of virtual training is to transfer knowledge gained in a virtual environment to an actual real-world setting. This study tested whether a maintenance procedure could be learnt equally well by virtual-environment and conventional training. Forty-two adults were divided into three equally sized groups: virtual training (GVT® [generic virtual training]), conventional training (using a real tank suspension and preparation station) and control (no training). Participants then performed the procedure individually in the real environment. Both training types (conventional and virtual) produced similar levels of performance when the procedure was carried out in real conditions. Performance level for the two trained groups was better in terms of success and time taken to complete the task, time spent consulting job instructions and number of times the instructor provided guidance.

Simulator training and non-technical factors improve laparoscopic performance among OBGYN trainees

OBJECTIVE

To investigate how simulator training and non-technical factors affect laparoscopic performance among residents in obstetrics and gynecology.

DESIGN

In this prospective study, trainees were randomized into three groups. The first group was allocated to proficiency-based training in the LapSimGyn(®) virtual reality simulator. The second group received additional structured mentorship during subsequent laparoscopies. The third group served as control group. At baseline an operation was performed and visuospatial ability, flow and self-efficacy were assessed. All groups subsequently performed three tubal occlusions. Self-efficacy and flow were assessed before and/or after each operation.

SETTING

Simulator training was conducted at the Center for Advanced Medical Simulation and Training, Karolinska University Hospital. Sterilizations were performed at each trainee's home clinic.

POPULATION

Twenty-eight trainees/residents from 21 hospitals in Sweden were included.

METHODS/MAIN OUTCOME MEASURES

Visuospatial ability was tested by the Mental Rotation Test-A. Flow and self-efficacy were assessed by validated scales and questionnaires. Laparoscopic performance was measured as the duration of surgery. Visuospatial ability, self-efficacy and flow were correlated to the laparoscopic performance using Spearman's correlations. Differences between groups were analyzed by the Mann-Whitney U-test.

RESULTS

No differences across groups were detected at baseline. Self-efficacy scores before and flow scores after the third operation were significantly higher in the trained groups. Duration of surgery was significantly shorter in the trained groups. Flow and self-efficacy correlate positively with laparoscopic performance.

CONCLUSIONS

Simulator training and non-technical factors appear to improve the laparoscopic performance among trainees/residents in obstetrics and gynecology.

A structured four-step curriculum in basic laparoscopy: development and validation

OBJECTIVE

The objective of this study was to develop a four-step curriculum in basic laparoscopy consisting of validated modules integrating a cognitive component, a practical component and a procedural component.

DESIGN

A four-step curriculum was developed. The methodology was different for each step. Step 1: A 1-day course in basic laparoscopy developed on the background of a regional needs analysis. Step 2: A multiple-choice test, developed and validated through interviews with experts in laparoscopy and subsequently through a Delphi audit involving regional chief physicians. Step 3: A procedural training task (a salpingectomy) on a validated virtual reality simulator. Step 4: An operation on a patient (a salpingectomy) with following formative assessment based on a validated assessment scale.

SETTING

University hospital, Copenhagen, Denmark.

POPULATION

Fifty-two first-year residents in obstetrics and gynecology from 2009 to 2011.

METHOD

Observational cohort study.

MAIN OUTCOME MEASURE

Completion rate.

RESULTS

All participants completed step 1 and improved post-course test scores compared with pre-course test scores, p = 0.001. Step 2 was completed by 75% (37/52); all improved test scores after 6 months, p = 0.001. Step 3 was completed by 75%. Participants used 238 min (range 75-599) and 38 repetitions (range 8-99) to reach proficiency level on a virtual reality simulator. Step 4 was completed by 55%. There was no correlation between test scores and simulator training time. Protected training time was correlated with increasing completion rate.

CONCLUSION

A four-step curriculum in basic laparoscopy is applicable in residency training. Protected training time correlated with increasing completion rate.

Gynecologic endoscopy skills training and assessment: review

Training in and assessment of endoscopic skills is currently undergoing a period of evolution. Several recognized factors driving this evolution include working pattern, training opportunities, cost, and patient safety. In addition, the need to continuously monitor competence is punctuated by the rapid technologic changes and rising consumer expectation. These challenges present an opportunity to positively enhance the learning and performance of surgical practice.

Face and construct validation of a virtual peg transfer simulator

BACKGROUND

The Fundamentals of Laparoscopic Surgery (FLS) trainer box is now established as a standard for evaluating minimally invasive surgical skills. A particularly simple task in this trainer box is the peg transfer task which is aimed at testing the surgeon's bimanual dexterity, hand-eye coordination, speed, and precision. The Virtual Basic Laparoscopic Skill Trainer (VBLaST) is a virtual version of the FLS tasks which allows automatic scoring and real-time, subjective quantification of performance without the need of a human proctor. In this article we report validation studies of the VBLaST peg transfer (VBLaST-PT) simulator.

METHODS

Thirty-five subjects with medical background were divided into two groups: experts (PGY 4-5, fellows, and practicing surgeons) and novices (PGY 1-3). The subjects were asked to perform the peg transfer task on both the FLS trainer box and the VBLaST-PT simulator; their performance was evaluated based on established metrics of error and time. A new length of trajectory (LOT) metric has also been introduced for offline analysis. A questionnaire was used to rate the realism of the virtual system on a 5-point Likert scale.

RESULTS

Preliminary face validation of the VBLaST-PT with 34 subjects rated on a 5-point Likert scale questionnaire revealed high scores for all aspects of simulation, with 3.53 being the lowest mean score across all questions. A two-tailed Mann-Whitney test performed on the total scores showed significant (p = 0.001) difference between the groups. A similar test performed on the task time (p = 0.002) and the LOT (p = 0.004) separately showed statistically significant differences between the experts and the novices (p < 0.05). The experts appear to be traversing shorter overall trajectories in less time than the novices.

CONCLUSION

VBLaST-PT showed both face and construct validity and has promise as a substitute for the FLS for training peg transfer skills.

Objective structured assessment of technical competence in transthoracic echocardiography: a validity study in a standardised setting

BACKGROUND

Competence in transthoracic echocardiography (TTE) is unrelated to traditional measures of TTE competence, such as duration of training and number of examinations performed. This study aims to explore aspects of validity of an instrument for structured assessment of echocardiographic technical skills.

METHODS

The study included 45 physicians with three different clinical levels of echocardiography competence who all scanned the same healthy male following national guidelines. An expert in echocardiography (OG) evaluated all the recorded, de-identified TTE images blindly using the developed instrument for assessment of TTE technical skills. The instrument consisted of both a global rating scale and a procedure specific checklist. Two scores were calculated for each examination: A global rating score and a total checklist score. OG rated ten examinations twice for intra-rater reliability, and another expert rated the same ten examinations for inter-rater reliability. A small pilot study was then performed with focus on content validity. This pilot study included nine physicians who scanned three patients with different pathologies as well as different technical difficulties.

RESULTS

Validity of the TTE technical skills assessment instrument was supported by a significant correlation found between level of expertise and both the global score (Spearman 0.76, p<0.0001) and the checklist score (Spearman 0.74, p<0.001). Both scores were able to distinguish between the three levels of competence that were represented in the physician group. Reliability was supported by acceptable inter- and intra-rater values. The pilot study showed a tendency to improved scores with increasing expertise levels, suggesting that the instrument could also be used when pathologies were present.

CONCLUSIONS

We designed and developed a structured assessment instrument of echocardiographic technical skills that showed evidence of validity in terms of high correlations between test scores on a normal person and the level of physician competence, as well as acceptable inter- and intra-rater reliability scores. Further studies should, however, be performed to determine the adequate number of assessments needed to ensure high content validity and reliability in a clinical setting.

Construct validity of the LapSim virtual reality laparoscopic simulator within a urology residency program

OBJECTIVE

: We assessed the construct validity of the LapSim laparoscopic surgical simulator in a urology residency training program.

METHODS

: In total, 15 residents participated in the study between July 2007 and July 2008. The subjects were tested six times at one-month intervals on three skill tasks (lifting and grasping, cutting and clip application) using the LapSim laparoscopic simulator. The testing sessions were divided into the first three sessions (seminar 1), and the subsequent three sessions (seminar 2). We evaluated the following parameters: total time, path length, angular path length, tissue damage, maximum damage and stretch damage. The subjects were divided into junior (PGY 1,2) and senior resident groups (PGY 3,4,5). The Wilcoxon Signed-Rank test for paired samples was used to compare the performances of the juniors and seniors during seminar 1 to their performance in seminar 2 to determine whether there was improvement over time. The Wilcoxon Rank-Sum test for independent samples was used to compare the performance of the juniors to that of the seniors for seminar 1, seminar 2 and the combination of both seminars to determine whether the more experienced senior residents performed better than the less experienced juniors.

RESULTS

: No significant performance improvement between testing sessions could be demonstrated. Similarly, there was no significant difference in performance between junior and senior residents.

CONCLUSIONS

: Construct validity could not be demonstrated for the total time, path length, angular path length and tissue handling parameters of the LapSim laparoscopic surgical simulator when examined within the context of a urology residency program.

EVA: laparoscopic instrument tracking based on Endoscopic Video Analysis for psychomotor skills assessment

INTRODUCTION

The EVA (Endoscopic Video Analysis) tracking system is a new system for extracting motions of laparoscopic instruments based on nonobtrusive video tracking. The feasibility of using EVA in laparoscopic settings has been tested in a box trainer setup.

METHODS

EVA makes use of an algorithm that employs information of the laparoscopic instrument's shaft edges in the image, the instrument's insertion point, and the camera's optical center to track the three-dimensional position of the instrument tip. A validation study of EVA comprised a comparison of the measurements achieved with EVA and the TrEndo tracking system. To this end, 42 participants (16 novices, 22 residents, and 4 experts) were asked to perform a peg transfer task in a box trainer. Ten motion-based metrics were used to assess their performance.

RESULTS

Construct validation of the EVA has been obtained for seven motion-based metrics. Concurrent validation revealed that there is a strong correlation between the results obtained by EVA and the TrEndo for metrics, such as path length (ρ = 0.97), average speed (ρ = 0.94), or economy of volume (ρ = 0.85), proving the viability of EVA.

CONCLUSIONS

EVA has been successfully validated in a box trainer setup, showing the potential of endoscopic video analysis to assess laparoscopic psychomotor skills. The results encourage further implementation of video tracking in training setups and image-guided surgery.

Instructor feedback versus no instructor feedback on performance in a laparoscopic virtual reality simulator: a randomized educational trial

BACKGROUND

Several studies have found a positive effect on the learning curve as well as the improvement of basic psychomotor skills in the operating room after virtual reality training. Despite this, the majority of surgical and gynecological departments encounter hurdles when implementing this form of training. This is mainly due to lack of knowledge concerning the time and human resources needed to train novice surgeons to an adequate level. The purpose of this trial is to investigate the impact of instructor feedback regarding time, repetitions and self-perception when training complex operational tasks on a virtual reality simulator.

METHODS/DESIGN

The study population consists of medical students on their 4th to 6th year without prior laparoscopic experience. The study is conducted in a skills laboratory at a centralized university hospital. Based on a sample size estimation 98 participants will be randomized to an intervention group or a control group. Both groups have to achieve a predefined proficiency level when conducting a laparoscopic salpingectomy using a surgical virtual reality simulator. The intervention group receives standardized instructor feedback of 10 to 12 min a maximum of three times. The control group receives no instructor feedback. Both groups receive the automated feedback generated by the virtual reality simulator. The study follows the CONSORT Statement for randomized trials. Main outcome measures are time and repetitions to reach the predefined proficiency level on the simulator. We include focus on potential sex differences, computer gaming experience and self-perception.

DISCUSSION

The findings will contribute to a better understanding of optimal training methods in surgical education.

TRIAL REGISTRATION

NCT01497782.