Construct and face validity of a virtual reality–based camera navigation curriculum

Training and assessment using the LapSim laparoscopic simulator: a scoping review of validity evidence

INTRODUCTION

The LapSim (Surgical Science, Sweden) laparoscopic simulator is a high-fidelity virtual reality simulator for use in endoscopic surgical training. This review critiques the current validity evidence for the LapSim laparoscopic simulator, specifically with respect to its potential use as a tool and method of training and assessment in surgery.

METHODS

A scoping review of the MEDLINE (PubMed), EMBASE, Cochrane and Web of Science databases was conducted in accordance with PRISMA guidelines (2020)-scoping review extension. Articles were included if they presented validity evidence for the use of the LapSim in operative skill training or assessment, in accordance with Messick's validity framework. European Association of Endoscopic Surgeons (EAES) guidelines (2005) were used to provide recommendations for the use of the LapSim in operative performance training and assessments.

RESULTS

Forty-nine articles were included. An EAES level 2 recommendation was provided with regard to the internal consistency reliability of automated performance metrics in assessing performance. An EAES recommendation of 2 was awarded with respect to the ability of the LapSim to discriminate based on case volume and overall laparoscopic experience (relationships with other variables). Performance assessment metrics on the LapSim correlate with improved performance in the operating room (EAES level of recommendation 1-consequential validity).

CONCLUSION

The LapSim has accumulated substantial evidence supporting the validity of its use in surgical training and assessment. Future studies should explore the relationship between the achievement of performance benchmarks on the LapSim and subsequent patient outcomes, and interrogate the benefits of implementing virtual reality simulation training and assessment curricula in post-graduate surgical training programmes.

Comparison of a virtual reality compression-only Cardiopulmonary Resuscitation (CPR) course to the traditional course with content validation of the VR course - A randomized control pilot study

Introduction

Technology has been a major contributor to recent changes in education, where simulation plays a huge role by providing a unique safe environment, especially with the recent incorporation of immersive virtual reality (VR) training. Cardiopulmonary Resuscitation (CPR) is said to double, even triple survival from cardiac arrest, and hence it is crucial to ensure optimal acquisition and retention of these skills. In this study, we aim to compare a VR CPR teaching program to current teaching methods with content validation of the VR course.

Methods

A randomized single-blinded simulation-based pilot study where 26 participants underwent baseline assessment of their CPR skills using a validated checklist and Laerdal QCPR®. Participants were randomly allocated and underwent their respective courses. This was followed by a final assessment and a questionnaire for content validation, knowledge and confidence. The data was analysed using STATA 16.2 to determine the standardized mean difference using paired and unpaired t-test.

Results

Subjective assessment using the checklist showed statistically significant improvement in the overall scores of both groups (traditional group mean improved from 6.92 to 9.61 p-value 0.0005, VR group from 6.61 to 8.53 p-value 0.0016). However, no statistically significant difference was noted between the final scores in both the subjective and objective assessments. As for the questionnaire, knowledge and confidence seemed to improve equally. Finally, the content validation showed statistically significant improvement in ease of use (mean score 3 to 4.23 p-value of 0.0144), while for content, positivity of experience, usefulness and appropriateness participants showed similar satisfaction before and after use.

Conclusion

This pilot study suggests that VR teaching could deliver CPR skills in an attractive manner, with no inferiority in acquisition of these skills compared to traditional methods. To corroborate these findings, we suggest a follow-up study with a larger sample size after adding ventilation and Automated External Defibrillator (AED) skills to the VR course with re-examination after 3–6 months to test retention of the skills.

Distinguishing Intermediate and Novice Surgeons by Eye Movements

Surgical skill-level assessment is key to collecting the required feedback and adapting the educational programs accordingly. Currently, these assessments for the minimal invasive surgery programs are primarily based on subjective methods, and there is no consensus on skill level classifications. One of the most detailed of these classifications categorize skill levels as beginner, novice, intermediate, sub-expert, and expert. To properly integrate skill assessment into minimal invasive surgical education programs and provide skill-based training alternatives, it is necessary to classify the skill levels in as detailed a way as possible and identify the differences between all skill levels in an objective manner. Yet, despite the existence of very encouraging results in the literature, most of the studies have been conducted to better understand the differences between novice and expert surgical skill levels leaving out the other crucial skill levels between them. Additionally, there are very limited studies by considering the eye-movement behaviors of surgical residents. To this end, the present study attempted to distinguish novice- and intermediate-level surgical residents based on their eye movements. The eye-movement data was recorded from 23 volunteer surgical residents while they were performing four computer-based simulated surgical tasks under different hand conditions. The data was analyzed using logistic regression to estimate the skill levels of both groups. The best results of the estimation revealing a 91.3% recognition rate of predicting novice and intermediate surgical residents on one scenario were selected from four under the dominant hand condition. These results show that the eye-movements can be potentially used to identify surgeons with intermediate and novice skills. However, the results also indicate that the order in which the scenarios are provided, and the design of the scenario, the tasks, and their appropriateness with the skill levels of the participants are all critical factors to be considered in improving the estimation ratio, and hence require thorough assessment for future research.

Saving robots improves laparoscopic performance: transfer of skills from a serious game to a virtual reality simulator

BACKGROUND

Residents find it hard to commit to structural laparoscopic skills training. Serious gaming has been proposed as a solution on the premise that it is effective and more motivating than traditional simulation. We establish construct validity for the laparoscopic serious game Underground by comparing laparoscopic simulator performance for a control group and an Underground training group.

METHODS

A four-session laparoscopic basic skills course is part of the medical master students surgical internship at the Radboud University Medical Centre. Four cohorts, representing 107 participants, were assigned to either the Underground group or the control group. The control group trained on the FLS video trainer and the LapSim virtual reality simulator for four sessions. The Underground group played Underground for three sessions followed by a transfer session on the FLS video trainer and the LapSim. To assess the effect of engaging in serious gameplay on performance on two validated laparoscopic simulators, initial performance on the FLS video trainer and the LapSim was compared between the control group (first session) and the Underground group (fourth session).

RESULTS

We chose task duration as a proxy for laparoscopic performance. The Underground group outperformed the control group on all three LapSim tasks: Camera navigation F(1) = 12.71, p < .01; Instrument navigation F(1) = 8.04, p < .01; and Coordination F(1) = 6.36, p = .01. There was no significant effect of playing Underground for performance on the FLS video trainer Peg Transfer task, F(1) = 0.28, p = .60.

CONCLUSIONS

We demonstrated skills transfer between a serious game and validated laparoscopic simulator technology. Serious gaming may become a valuable, cost-effective addition to the skillslab, if transfer to the operating room can be established. Additionally, we discuss sources of transferable skills to help explain our and previous findings.

Gaze gesture based human robot interaction for laparoscopic surgery

While minimally invasive surgery offers great benefits in terms of reduced patient trauma, bleeding, as well as faster recovery time, it still presents surgeons with major ergonomic challenges. Laparoscopic surgery requires the surgeon to bimanually control surgical instruments during the operation. A dedicated assistant is thus required to manoeuvre the camera, which is often difficult to synchronise with the surgeon's movements. This article introduces a robotic system in which a rigid endoscope held by a robotic arm is controlled via the surgeon's eye movement, thus forgoing the need for a camera assistant. Gaze gestures detected via a series of eye movements are used to convey the surgeon's intention to initiate gaze contingent camera control. Hidden Markov Models (HMMs) are used for real-time gaze gesture recognition, allowing the robotic camera to pan, tilt, and zoom, whilst immune to aberrant or unintentional eye movements. A novel online calibration method for the gaze tracker is proposed, which overcomes calibration drift and simplifies its clinical application. This robotic system has been validated by comprehensive user trials and a detailed analysis performed on usability metrics to assess the performance of the system. The results demonstrate that the surgeons can perform their tasks quicker and more efficiently when compared to the use of a camera assistant or foot switches.

Face and content validity of the virtual reality simulator 'ScanTrainer®'

Background

Ultrasonography is a first-line imaging in the investigation of women’s irregular bleeding and other gynaecological pathologies, e.g. ovarian cysts and early pregnancy problems. However, teaching ultrasound, especially transvaginal scanning, remains a challenge for health professionals. New technology such as simulation may potentially facilitate and expedite the process of learning ultrasound. Simulation may prove to be realistic, very close to real patient scanning experience for the sonographer and objectively able to assist the development of basic skills such as image manipulation, hand-eye coordination and examination technique.

Objective

The aim of this study was to determine the face and content validity of a virtual reality simulator (ScanTrainer®, MedaPhor plc, Cardiff, Wales, UK) as reflective of real transvaginal ultrasound (TVUS) scanning.

Method

A questionnaire with 14 simulator-related statements was distributed to a number of participants with differing levels of sonography experience in order to determine the level of agreement between the use of the simulator in training and real practice.

Results

There were 36 participants: novices (n = 25) and experts (n = 11) who rated the simulator. Median scores of face validity statements between experts and non-experts using a 10-point visual analogue scale (VAS) ratings ranged between 7.5 and 9.0 (p > 0.05) indicated a high level of agreement. Experts’ median scores of content validity statements ranged from 8.4 to 9.0.

Conclusions

The findings confirm that the simulator has the feel and look of real-time scanning with high face validity. Similarly, its tutorial structures and learning steps confirm the content validity.

Electronic supplementary material

The online version of this article (10.1186/s10397-017-1020-6) contains supplementary material, which is available to authorized users.

Role of Simulation in Perioperative Echocardiography Training

Echocardiography plays a major role in the diagnosis and management of hemodynamic compromise during the perioperative period. Both transthoracic and transesophageal echocardiography have been shown to improve outcomes after cardiac and noncardiac surgery. Teaching basic echocardiographic skills to perioperative physicians remains a challenging task. Thus far, simulation-based medical education has been proven useful in teaching specific procedural skills and management of infrequent catastrophic events. Simulation-based echocardiography education has the potential to facilitate clinical training in echocardiography. Several small studies have shown the benefits of echocardiographic simulation on developing psychomotor and cognitive echocardiography skills. Future research should focus on the impact of simulation on actual clinical echocardiographic performance in the operating room and ultimately, patient outcomes.

Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery

Background

Effective visualization of the operative field is vital to surgical safety and education. However, additional metrics for visualization are needed to complement other common measures of surgeon proficiency, such as time or errors. Unlike other surgical modalities, robot-assisted minimally invasive surgery (RAMIS) enables data-driven feedback to trainees through measurement of camera adjustments. The purpose of this study was to validate and quantify the importance of novel camera metrics during RAMIS.

Methods

New (n = 18), intermediate (n = 8), and experienced (n = 13) surgeons completed 25 virtual reality simulation exercises on the da Vinci Surgical System. Three camera metrics were computed for all exercises and compared to conventional efficiency measures.

Results

Both camera metrics and efficiency metrics showed construct validity (p < 0.05) across most exercises (camera movement frequency 23/25, camera movement duration 22/25, camera movement interval 19/25, overall score 24/25, completion time 25/25). Camera metrics differentiated new and experienced surgeons across all tasks as well as efficiency metrics. Finally, camera metrics significantly (p < 0.05) correlated with completion time (camera movement frequency 21/25, camera movement duration 21/25, camera movement interval 20/25) and overall score (camera movement frequency 20/25, camera movement duration 19/25, camera movement interval 20/25) for most exercises.

Conclusions

We demonstrate construct validity of novel camera metrics and correlation between camera metrics and efficiency metrics across many simulation exercises. We believe camera metrics could be used to improve RAMIS proficiency-based curricula.

Fundamental arthroscopic skill differentiation with virtual reality simulation

PURPOSE

The purpose of this study was to investigate the use and validity of virtual reality modules as part of the educational approach to mastering arthroscopy in a safe environment by assessing the ability to distinguish between experience levels. Additionally, the study aimed to evaluate whether experts have greater ambidexterity than do novices.

METHODS

Three virtual reality modules (Swemac/Augmented Reality Systems, Linkoping, Sweden) were created to test fundamental arthroscopic skills. Thirty participants-10 experts consisting of faculty, 10 intermediate participants consisting of orthopaedic residents, and 10 novices consisting of medical students-performed each exercise. Steady and Telescope was designed to train centering and image stability. Steady and Probe was designed to train basic triangulation. Track and Moving Target was designed to train coordinated motions of arthroscope and probe. Metrics reflecting speed, accuracy, and efficiency of motion were used to measure construct validity.

RESULTS

Steady and Probe and Track a Moving Target both exhibited construct validity, with better performance by experts and intermediate participants than by novices (P < .05), whereas Steady and Telescope did not show validity. There was an overall trend toward better ambidexterity as a function of greater surgical experience, with experts consistently more proficient than novices throughout all 3 modules.

CONCLUSIONS

This study represents a new way to assess basic arthroscopy skills using virtual reality modules developed through task deconstruction. Participants with the most arthroscopic experience performed better and were more consistent than novices on all 3 virtual reality modules. Greater arthroscopic experience correlates with more symmetry of ambidextrous performance. However, further adjustment of the modules may better simulate fundamental arthroscopic skills and discriminate between experience levels.

CLINICAL RELEVANCE

Arthroscopy training is a critical element of orthopaedic surgery resident training. Developing techniques to safely and effectively train these skills is critical for patient safety and resident education.

Perceptions, training experiences, and preferences of surgical residents toward laparoscopic simulation training: a resident survey

INTRODUCTION

Simulation training for surgical residents can shorten learning curves, improve technical skills, and expedite competency. Several studies have shown that skills learned in the simulated environment are transferable to the operating room. Residency programs are trying to incorporate simulation into the resident training curriculum to supplement the hands-on experience gained in the operating room. Despite the availability and proven utility of surgical simulators and simulation laboratories, they are still widely underutilized by surgical trainees. Studies have shown that voluntary use leads to minimal participation in a training curriculum. Although there are several simulation tools, there is no clear evidence of the superiority of one tool over the other in skill acquisition. The purpose of this study was to explore resident perceptions, training experiences, and preferences regarding laparoscopic simulation training. Our goal was to profile resident participation in surgical skills simulation, recognize potential barriers to voluntary simulator use, and identify simulation tools and tasks preferred by residents. Furthermore, this study may help to inform whether mandatory/protected training time, as part of the residents' curriculum is essential to enhance participation in the simulation laboratory.

METHODS

A cross-sectional study on general surgery residents (postgraduate years 1-5) at Yale University School of Medicine and the University of Toronto via an online questionnaire was conducted. Overall, 67 residents completed the survey. The institutional review board approved the methods of the study.

RESULTS

Overall, 95.5% of the participants believed that simulation training improved their laparoscopic skills. Most respondents (92.5%) perceived that skills learned during simulation training were transferrable to the operating room. Overall, 56.7% of participants agreed that proficiency in a simulation curriculum should be mandatory before operating room experience. The simulation laboratory was most commonly used during work hours; lack of free time during work hours was most commonly cited as a reason for underutilization. Factors influencing use of the simulation laboratory in order of importance were the need for skill development, an interest in minimally invasive surgery, mandatory/protected time in a simulation environment as part of the residency program curriculum, a recommendation by an attending surgeon, and proximity of the simulation center. The most preferred simulation tool was the live animal model followed by cadaveric tissue. Virtual reality simulators were among the least-preferred (25%) simulation tools. Most residents (91.0%) felt that mandatory/protected time in a simulation environment should be introduced into resident training protocols.

CONCLUSIONS

Mandatory and protected time in a simulation environment as part of the resident training curriculum may improve participation in simulation training. A comprehensive curriculum, which includes the use of live animals, cadaveric tissue, and virtual reality simulators, may enhance the laparoscopic training experience and interest level of surgical trainees.

A Multicenter Prospective Cohort Study on Camera Navigation Training for Key User Groups in Minimally Invasive Surgery

Background. Untrained laparoscopic camera assistants in minimally invasive surgery (MIS) may cause suboptimal view of the operating field, thereby increasing risk for errors. Camera navigation is often performed by the least experienced member of the operating team, such as inexperienced surgical residents, operating room nurses, and medical students. The operating room nurses and medical students are currently not included as key user groups in structured laparoscopic training programs. A new virtual reality laparoscopic camera navigation (LCN) module was specifically developed for these key user groups. Methods. This multicenter prospective cohort study assesses face validity and construct validity of the LCN module on the Simendo virtual reality simulator. Face validity was assessed through a questionnaire on resemblance to reality and perceived usability of the instrument among experts and trainees. Construct validity was assessed by comparing scores of groups with different levels of experience on outcome parameters of speed and movement proficiency. Results. The results obtained show uniform and positive evaluation of the LCN module among expert users and trainees, signifying face validity. Experts and intermediate experience groups performed significantly better in task time and camera stability during three repetitions, compared to the less experienced user groups ( P < .007). Comparison of learning curves showed significant improvement of proficiency in time and camera stability for all groups during three repetitions ( P < .007). Conclusion. The results of this study show face validity and construct validity of the LCN module. The module is suitable for use in training curricula for operating room nurses and novice surgical trainees, aimed at improving team performance in minimally invasive surgery.