Virtual Reality vs. Dry-Lab Models: Comparing automated performance metrics and cognitive workload during robotic simulation training

Training in robotic-assisted surgery: a systematic review of training modalities and objective and subjective assessment methods

INTRODUCTION

The variety of robotic surgery systems, training modalities, and assessment tools within robotic surgery training is extensive. This systematic review aimed to comprehensively overview different training modalities and assessment methods for teaching and assessing surgical skills in robotic surgery, with a specific focus on comparing objective and subjective assessment methods.

METHODS

A systematic review was conducted following the PRISMA guidelines. The electronic databases Pubmed, EMBASE, and Cochrane were searched from inception until February 1, 2022. Included studies consisted of robotic-assisted surgery training (e.g., box training, virtual reality training, cadaver training and animal tissue training) with an assessment method (objective or subjective), such as assessment forms, virtual reality scores, peer-to-peer feedback or time recording.

RESULTS

The search identified 1591 studies. After abstract screening and full-texts examination, 209 studies were identified that focused on robotic surgery training and included an assessment tool. The majority of the studies utilized the da Vinci Surgical System, with dry lab training being the most common approach, followed by the da Vinci Surgical Skills Simulator. The most frequently used assessment methods included simulator scoring system (e.g., dVSS score), and assessment forms (e.g., GEARS and OSATS).

CONCLUSION

This systematic review provides an overview of training modalities and assessment methods in robotic-assisted surgery. Dry lab training on the da Vinci Surgical System and training on the da Vinci Skills Simulator are the predominant approaches. However, focused training on tissue handling, manipulation, and force interaction is lacking, despite the absence of haptic feedback. Future research should focus on developing universal objective assessment and feedback methods to address these limitations as the field continues to evolve.

Evaluation status of current and emerging minimally invasive robotic surgical platforms

BACKGROUND

The rapid adoption of robotics within minimally invasive surgical specialties has also seen an explosion of new technology including multi- and single port, natural orifice transluminal endoscopic surgery (NOTES), endoluminal and "on-demand" platforms. This review aims to evaluate the validation status of current and emerging MIS robotic platforms, using the IDEAL Framework.

METHODS

A scoping review exploring robotic minimally invasive surgical devices, technology and systems in use or being developed was performed, including general surgery, gynaecology, urology and cardiothoracics. Systems operating purely outside the abdomen or thorax and endoluminal or natural orifice platforms were excluded. PubMed, Google Scholar, journal reports and information from the public domain were collected. Each company was approached via email for a virtual interview to discover more about the systems and to quality check data. The IDEAL Framework is an internationally accepted tool to evaluate novel surgical technology, consisting of four stages: idea, development/exploration, assessment, and surveillance. An IDEAL stage, synonymous with validation status in this review, was assigned by reviewing the published literature.

RESULTS

21 companies with 23 different robotic platforms were identified for data collection, 13 with national and/or international regulatory approval. Of the 17 multiport systems, 1 is fully evaluated at stage 4, 2 are stage 3, 6 stage 2b, 2 at stage 2a, 2 stage 1, and 4 at the pre-IDEAL stage 0. Of the 6 single-port systems none have been fully evaluated with 1 at stage 3, 3 at stage 1 and 2 at stage 0.

CONCLUSIONS

The majority of existing robotic platforms are currently at the preclinical to developmental and exploratory stage of evaluation. Using the IDEAL framework will ensure that emerging robotic platforms are fully evaluated with long-term data, to inform the surgical workforce and ensure patient safety.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Virtual reality vs. physical models in surgical skills training. An update of the evidence

PURPOSE OF REVIEW

Simulation is a key component of surgical training, enabling trainees to develop their skills in a safe environment. With simulators broadly grouped into physical models and virtual-reality (VR) simulators, it is important to evaluate the comparative effectiveness of the simulator types in terms of validity as well as cost. The review aims to compare the benefits and drawbacks of novel VR and physical simulators within the broader themes of endourology, laparoscopic and robotic operations, and other urological procedures.

RECENT FINDINGS

Key benefits of bench models include their comparatively lower cost, easy access and provision of haptic feedback, whereas VR simulators are generally self-sufficient, reusable and enable skills of haemostasis to be practised. The advent of perfused 3D printed simulators across a range of urological procedures may replace cadavers as the traditional gold-standard simulation modality.

SUMMARY

Although possessing differing strengths and downsides, VR and physical simulators when used together can have an additive effect due to skill transferability across the platforms. Further comparative studies are required to directly quantify the differences between physical models and VR simulators in terms of performance metrics and cost-effectiveness. There is lack of validated VR simulators for open and reconstructive procedures.

Does Extended Reality Simulation Improve Surgical/Procedural Learning and Patient Outcomes When Compared With Standard Training Methods?: A Systematic Review

INTRODUCTION

The use of extended reality (XR) technologies, including virtual, augmented, and mixed reality, has increased within surgical and procedural training programs. Few studies have assessed experiential learning- and patient-based outcomes using XR compared with standard training methods.

METHODS

As a working group for the Society for Simulation in Healthcare, we used Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines and a PICO strategy to perform a systematic review of 4238 articles to assess the effectiveness of XR technologies compared with standard training methods. Outcomes were grouped into knowledge, time-to-completion, technical proficiency, reactions, and patient outcomes. Because of study heterogeneity, a meta-analysis was not feasible.

RESULTS

Thirty-two studies met eligibility criteria: 18 randomized controlled trials, 7 comparative studies, and 7 systematic reviews. Outcomes of most studies included Kirkpatrick levels of evidence I-III (reactions, knowledge, and behavior), while few reported level IV outcomes (patient). The overall risk of bias was low. With few exceptions, included studies showed XR technology to be more effective than standard training methods in improving objective skills and performance, shortening procedure time, and receiving more positive learner ratings. However, XR use did not show significant differences in gained knowledge.

CONCLUSIONS

Surgical or procedural XR training may improve technical skill development among trainees and is generally favored over standard training methods. However, there should be an additional focus on how skill development translates to clinically relevant outcomes. We recommend longitudinal studies to examine retention and transfer of training to clinical settings, methods to improve timely, adaptive feedback for deliberate practice, and cost analyses.

Differences Between Attending and Trainee Surgeon Performance Using Objective Performance Indicators During Robot-Assisted Lobectomy

OBJECTIVE

Existing approaches for assessing surgical performance are subjective and prone to bias. In contrast, utilizing digital kinematic and system data from the surgical robot allows the calculation of objective performance indicators (OPIs) that may differentiate technical skill and competency. This study compared OPIs of trainees and attending surgeons to assess differences during robotic lobectomy (RL).

METHODS

There were 50 cardiothoracic surgery residents and 7 attending surgeons who performed RL on a left upper lobectomy of an ex vivo perfused model. A novel recorder simultaneously captured video and data from the system and instruments. The lobectomy was annotated into discrete tasks, and OPIs were analyzed for both hands during 6 tasks: exposure of the superior pulmonary vein, upper division of the pulmonary artery and bronchus, and the stapling of these structures.

RESULTS

There were significant differences between attendings and trainees in all tasks. Among 20 OPIs during exposure tasks, significant differences were observed for the left hand in 31 of 60 (52%) of OPIs and for the right hand in 42 of 60 (70%). During stapling tasks, significant differences were observed for the stapling hand in 28 of 60 (47%) of OPIs and for the nonstapling hand in 14 of 60 (25%).

CONCLUSIONS

Use of a novel data and video recorder to generate OPIs for both hands revealed significant differences in the operative gestures performed by trainees compared to attendings during RL. This method of assessing performance has potential for establishing objective competency benchmarks and use for tracking progress.

Association of Suturing Technical Skill Assessment Scores Between Virtual Reality Simulation and Live Surgery

Introduction: Robotic surgical performance, in particular suturing, has been linked to postoperative clinical outcomes. Before attempting live surgery, virtual reality (VR) simulators afford opportunities for training surgeons to learn fundamental technical skills. Herein, we evaluate the association of suturing technical skill assessments between VR simulation and live surgery, and functional clinical outcomes. Materials and Methods: Twenty surgeons completed a VR suturing exercise on the Mimic™ Flex VR simulator and the anterior vesicourethral anastomosis during robot-assisted radical prostatectomy (RARP). Three independent and blinded graders provided technical skill scores using a validated assessment tool. Correlations between VR and live scores were assessed by Spearman's correlation coefficients (ρ). In addition, 117 historic RARP cases from participating surgeons were extracted, and the association between VR technical skill scores and urinary continence recovery was assessed by a multilevel mixed-effects model. Results: A total of 20 (6 training and 14 expert) surgeons participated. Statistically significant correlations for scores provided between VR simulation and live surgery were found for overall and needle driving scores (ρ = 0.555, p = 0.011; ρ = 0.570, p = 0.009, respectively). A subanalysis performed on training surgeons found significant correlations for overall scores between VR simulation and live surgery (ρ = 0.828, p = 0.042). Expert cases with high VR needle driving scores had significantly greater continence recovery rates at 24 months after RARP (98.5% vs 84.9%, p = 0.028). Conclusions: Our study found significant correlations in technical scores between VR and live surgery, especially among training surgeons. In addition, we found that VR needle driving scores were associated with continence recovery after RARP. Our data support the association of skill assessments between VR simulation and live surgery and potential implications for clinical outcomes.

Remote surgical education using synthetic models combined with an augmented reality headset

Objective

The objective was to investigate the use of an augmented reality headset to remotely train clinicians on medical devices using anatomic models.

Design

Disease-specific phantoms were developed to train physicians in mpMRI-guided fusion prostate biopsy, brachytherapy, and rectal spacer insertion. Training was remotely demonstrated using 1-way virtual video conferencing format. Participants responded to an educational content survey. A heads-up display with software and augmented reality was used for remote 2-way training with the proctor and student using on their own phantoms.

Setting

The virtual video meeting took place during a prostate cancer conference in 2020, while the augmented reality training occurred in 2021. The proctor and student wore a heads-up display containing a projector and webcam where the ultrasound image was displayed onto a see-through optic along with the physician's hands. The heads-up display allowed the proctor to teach by line-of-sight while the student watched and repeated the steps.

Participants

Faculty with expertise with the medical devices used in these procedures provided training to urologists unfamiliar with these techniques.

Results

Participants responded that the 1-way training on the phantoms was realistic and mimicked human tissue. A total of 70.9% requested more training or training on the phantoms. The remote training platform was successfully beta tested at the 2 locations in transperineal prostate biopsy and rectal spacer insertion.

Conclusion

Remote training using augmented reality eliminates the need for travel. For training programs and workshops, this technology may mitigate the risk of infectious exposures, reduce training cost, and increase proctor availability, allowing training from their own institution or clinic.

This investigation qualifies for the Accreditation Council for Graduate Medical Education competency in medical knowledge.

•

Disease and medical device specific simulation phantoms improve surgeons' skills.

•

Hands-on training using phantoms can be accomplished with both instructor and student at different locations.

•

A dedicated headset containing display optics and a webcam allows “line-of sight” instruction.

•

Broadcasting of proctor to student and student to proctor content is optimized with specialized telecasting software which organizes the video feeds vertically.