How to Bring Surgery to the Next Level: Interpretable Skills Assessment in Robotic-Assisted Surgery

Kinematic metrics and surgeon experience in robotic cholecystectomies: a pilot study on breaking down technical performance

BACKGROUND

Recent studies have correlated surgical skill measured by video-based assessment with improved clinical outcomes. Certain automated measures of operative performance in robotic surgery can be gathered beyond video review called objective performance indicators (OPIs). We explore the relationship between OPIs, surgeon experience, and postoperative recovery, hypothesizing that more efficient dissection will be associated with experience.

METHODS

Fifty-six robotic cholecystectomies between February 2022 and March 2023 were recorded at a large tertiary referral center. Surgeon experience and clinical outcomes data from the EMR were obtained for all 56 cases with 10 completing the QOL survey. Two steps of robotic cholecystectomies were reviewed: dissection of Calot's triangle (DCT) and dissection of the gallbladder from the liver (DGL). Postoperative recovery was measured using the SF-36 well-being survey. Univariate analysis was conducted using Pearson's coefficient.

RESULTS

Increased operative experience was associated with more efficient camera and instrument movements. DCT had 7 and DGL had 31 of 41 OPIs that correlated with experience. With respect to DGL, more experienced surgeons had reduced step duration and instrument path length and increased camera and instrument speeds.

CONCLUSIONS

Several OPIs correlate with surgical experience and may form the basis of more instructive feedback for trainees and less experienced surgeons in improving intraoperative technique.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Cumulative Sum Chart as Complement to Objective Assessment of Graduating Surgical Resident Competency: An Exploratory Study

BACKGROUND

Rater-based assessment and objective assessment play an important role in evaluating residents' clinical competencies. We hypothesize that a cumulative sum (CUSUM) chart of operative time is a complement to the assessment of chief general surgery residents' competencies with ACGME Milestones, aiding residency programs' determination of graduating residents' practice readiness.

STUDY DESIGN

We extracted ACGME Milestone evaluations of performance of operations and procedures (POP) and 3 objective metrics (operative time, case type, and case complexity) from 3 procedures (cholecystectomy, colectomy, and inguinal hernia) performed by 3 cohorts of residents (N = 15) during their PGY4-5. CUSUM charts were computed for each resident on each procedure type. A learning plateau was defined as at least 4 cases consistently locating around the centerline (target performance) at the end of a CUSUM chart with minimal deviations (range 0 to 1).

RESULTS

All residents reached the ACGME graduation targets for the overall POP by the end of chief year. A total of 2,446 cases were included (cholecystectomy N = 1234, colectomy N = 507, and inguinal hernia N = 705), and 3 CUSUM chart patterns emerged: skewed distribution, bimodal distribution, and peaks and valleys distribution. Analysis of CUSUM charts revealed surgery residents' development processes in the operating room towards a learning plateau vary, and only 46.7% residents reach a learning plateau in all 3 procedures upon graduation.

CONCLUSIONS

CUSUM charts of operative time complement the ACGME Milestones evaluations. The use of both may enable residency programs to holistically determine graduating residents' practice readiness and provide recommendations for their upcoming career/practice transition.

Comparison of Global Evaluative Assessment of Robotic Surgery with objective performance indicators for the assessment of skill during robotic-assisted thoracic surgery

BACKGROUND

The Global Evaluative Assessment of Robotic Skills is a popular but ultimately subjective assessment tool in robotic-assisted surgery. An alternative approach is to record system or console events or calculate instrument kinematics to derive objective performance indicators. The aim of this study was to compare these 2 approaches and correlate the Global Evaluative Assessment of Robotic Skills with different types of objective performance indicators during robotic-assisted lobectomy.

METHODS

Video, system event, and kinematic data were recorded from the robotic surgical system during left upper lobectomy on a standardized perfused and pulsatile ex vivo porcine heart-lung model. Videos were segmented into steps, and the superior vein dissection was graded independently by 2 blinded expert surgeons with Global Evaluative Assessment of Robotic Skills. Objective performance indicators representing categories for energy use, event data, movement, smoothness, time, and wrist articulation were calculated for the same task and compared to Global Evaluative Assessment of Robotic Skills scores.

RESULTS

Video and data from 51 cases were analyzed (44 fellows, 7 attendings). Global Evaluative Assessment of Robotic Skills scores were significantly higher for attendings (P < .05), but there was a significant difference in raters' scores of 31.4% (defined as >20% difference in total score). The interclass correlation was 0.44 for 1 rater and 0.61 for 2 raters. Objective performance indicators correlated with Global Evaluative Assessment of Robotic Skills to varying degrees. The most highly correlated Global Evaluative Assessment of Robotic Skills domain was efficiency. Instrument movement and smoothness were highly correlated among objective performance indicator categories. Of individual objective performance indicators, right-hand median jerk, an objective performance indicator of change of acceleration, had the highest correlation coefficient (0.55).

CONCLUSION

There was a relatively poor overall correlation between the Global Evaluative Assessment of Robotic Skills and objective performance indicators. However, both appear strongly correlated for certain metrics such as efficiency and smoothness. Objective performance indicators may be a potentially more quantitative and granular approach to assessing skill, given that they can be calculated mathematically and automatically without subjective interpretation.

A multifactorial evaluation of objective performance indicators and video analysis in the context of case complexity and clinical outcomes in robotic-assisted cholecystectomy

BACKGROUND

The increased digitization in robotic surgical procedures today enables surgeons to quantify their movements through data captured directly from the robotic system. These calculations, called objective performance indicators (OPIs), offer unprecedented detail into surgical performance. In this study, we link case- and surgical step-specific OPIs to case complexity, surgical experience and console utilization, and post-operative clinical complications across 87 robotic cholecystectomy (RC) cases.

METHODS

Videos of RCs performed by a principal surgeon with and without fellows were segmented into eight surgical steps and linked to patients' clinical data. Data for OPI calculations were extracted from an Intuitive Data Recorder and the da Vinci ® robotic system. RC cases were each assigned a Nassar and Parkland Grading score and categorized as standard or complex. OPIs were compared across complexity groups, console attributions, and post-surgical complication severities to determine objective relationships across variables.

RESULTS

Across cases, differences in camera control and head positioning metrics of the principal surgeon were observed when comparing standard and complex cases. Further, OPI differences across the principal surgeon and the fellow(s) were observed in standard cases and include differences in arm swapping, camera control, and clutching behaviors. Monopolar coagulation energy usage differences were also observed. Select surgical step duration differences were observed across complexities and console attributions, and additional surgical task analyses determine the adhesion removal and liver bed hemostasis steps to be the most impactful steps for case complexity and post-surgical complications, respectively.

CONCLUSION

This is the first study to establish the association between OPIs, case complexities, and clinical complications in RC. We identified OPI differences in intra-operative behaviors and post-surgical complications dependent on surgeon expertise and case complexity, opening the door for more standardized assessments of teaching cases, surgical behaviors and case complexities.

Kinematic data profile and clinical outcomes in robotic inguinal hernia repairs: a pilot study

BACKGROUND

Surgical training requires clinical knowledge and technical skills to operate safely and optimize clinical outcomes. Technical skills are hard to measure. The Intuitive Data Recorder (IDR), (Sunnyvale, CA) allows for the measurement of technical skills using objective performance indicators (OPIs) from kinematic event data. Our goal was to determine whether OPIs improve with surgeon experience and whether they are correlated with clinical outcomes for robotic inguinal hernia repair (RIHR).

METHODS

The IDR was used to record RIHRs from six surgeons. Data were obtained from 98 inguinal hernia repairs from February 2022 to February 2023. Patients were called on postoperative days 5-10 and asked to take the Carolina Comfort Scale (CCS) survey to evaluate acute clinical outcomes. A Pearson test was run to determine correlations between OPIs from the IDR with a surgeon's yearly RIHR experience and with CCS scores. Linear regression was then run for correlated OPIs.

RESULTS

Multiple OPIs were correlated with surgeon experience. Specifically, for the task of peritoneal flap exploration, we found that 23 OPIs were significantly correlated with surgeons' 1-year RIHR case number. Total angular motion distance of the left arm instrument had a correlation of - 0.238 (95% CI - 0.417, - 0.042) for RIHR yearly case number. Total angular motion distance of right arm instrument was also negatively correlated with RIHR in 1 year with a correlation of - 0.242 (95% CI - 0.420, - 0.046). For clinical outcomes, wrist articulation of the surgeon's console positively correlated with acute sensation scores from the CCS with a correlation of 0.453 (95% CI 0.013, 0.746).

CONCLUSIONS

This study defines multiple OPIs that correlate with surgeon experience and with outcomes. Using this knowledge, surgical simulation platforms can be designed to teach patterns to surgical trainees that are associated with increased surgical experience and with improved postoperative outcomes.

SDA-CLIP: surgical visual domain adaptation using video and text labels

Background

Surgical action recognition is an essential technology in context-aware-based autonomous surgery, whereas the accuracy is limited by clinical dataset scale. Leveraging surgical videos from virtual reality (VR) simulations to research algorithms for the clinical domain application, also known as domain adaptation, can effectively reduce the cost of data acquisition and annotation, and protect patient privacy.

Methods

We introduced a surgical domain adaptation method based on the contrastive language-image pretraining model (SDA-CLIP) to recognize cross-domain surgical action. Specifically, we utilized the Vision Transformer (ViT) and Transformer to extract video and text embeddings, respectively. Text embedding was developed as a bridge between VR and clinical domains. Inter- and intra-modality loss functions were employed to enhance the consistency of embeddings of the same class. Further, we evaluated our method on the MICCAI 2020 EndoVis Challenge SurgVisDom dataset.

Results

Our SDA-CLIP achieved a weighted F1-score of 65.9% (+18.9%) on the hard domain adaptation task (trained only with VR data) and 84.4% (+4.4%) on the soft domain adaptation task (trained with VR and clinical-like data), which outperformed the first place team of the challenge by a significant margin.

Conclusions

The proposed SDA-CLIP model can effectively extract video scene information and textual semantic information, which greatly improves the performance of cross-domain surgical action recognition. The code is available at https://github.com/Lycus99/SDA-CLIP.

Differences Between Attending and Trainee Surgeon Performance Using Objective Performance Indicators During Robot-Assisted Lobectomy

OBJECTIVE

Existing approaches for assessing surgical performance are subjective and prone to bias. In contrast, utilizing digital kinematic and system data from the surgical robot allows the calculation of objective performance indicators (OPIs) that may differentiate technical skill and competency. This study compared OPIs of trainees and attending surgeons to assess differences during robotic lobectomy (RL).

METHODS

There were 50 cardiothoracic surgery residents and 7 attending surgeons who performed RL on a left upper lobectomy of an ex vivo perfused model. A novel recorder simultaneously captured video and data from the system and instruments. The lobectomy was annotated into discrete tasks, and OPIs were analyzed for both hands during 6 tasks: exposure of the superior pulmonary vein, upper division of the pulmonary artery and bronchus, and the stapling of these structures.

RESULTS

There were significant differences between attendings and trainees in all tasks. Among 20 OPIs during exposure tasks, significant differences were observed for the left hand in 31 of 60 (52%) of OPIs and for the right hand in 42 of 60 (70%). During stapling tasks, significant differences were observed for the stapling hand in 28 of 60 (47%) of OPIs and for the nonstapling hand in 14 of 60 (25%).

CONCLUSIONS

Use of a novel data and video recorder to generate OPIs for both hands revealed significant differences in the operative gestures performed by trainees compared to attendings during RL. This method of assessing performance has potential for establishing objective competency benchmarks and use for tracking progress.

Parallel, component training in robotic total mesorectal excision

There has been widespread adoption of robotic total mesorectal excision (TME) for rectal cancer in recent years. There is now increasing interest in training robotic novice surgeons in robotic TME surgery using the principles of component-based learning. The aims of our study were to assess the feasibility of delivering a structured, parallel, component-based, training curriculum to surgical trainees and fellows. A prospective pilot study was undertaken between January 2021 and May 2021. A dedicated robotic training pathway was designed with two trainees trained in parallel per each robotic case based on prior experience, training grade and skill set. Component parts of each operation were allocated by the robotic trainer prior to the start of each case. Robotic proficiency was assessed using the Global Evaluative Assessment of Robotic Skills (GEARS) and the EARCS Global Assessment Score (GAS). Three trainees participated in this pilot study, performing a combined number of 52 TME resections. Key components of all 52 TME operations were performed by the trainees. GEARS scores improved throughout the study, with a mean overall baseline score of 17.3 (95% CI 15.1-1.4) compared to an overall final assessment mean score of 23.8 (95% CI 21.6-25.9), p = 0.003. The GAS component improved incrementally for all trainees at each candidate assessment (p < 0.001). Employing a parallel, component-based approach to training in robotic TME surgery is safe and feasible and can be used to train multiple trainees of differing grades simultaneously, whilst maintaining high-quality clinical outcomes.

Objective performance indicators of cardiothoracic residents are associated with vascular injury during robotic-assisted lobectomy on porcine models

Surgical training relies on subjective feedback on resident technical performance by attending surgeons. A novel data recorder connected to a robotic-assisted surgical platform captures synchronized kinematic and video data during an operation to calculate quantitative, objective performance indicators (OPIs). The aim of this study was to determine if OPIs during initial task of a resident's robotic-assisted lobectomy (RL) correlated with bleeding during the procedure. Forty-six residents from the 2019 Thoracic Surgery Directors Association Resident Boot Camp completed RL on an ex vivo perfused porcine model while continuous video and kinematic data were recorded. For this pilot study, RL was segmented into 12 tasks and OPIs were calculated for the initial major task. Cases were reviewed for major bleeding events and OPIs of bleeding cases were compared to those who did not. Data from 42 residents were complete and included in the analysis. 10/42 residents (23.8%) encountered bleeding: 10/40 residents who started with superior pulmonary vein exposure and 0/2 residents who started with pulmonary artery exposure. Twenty OPIs for both hands were assessed during the initial task. Six OPIs related to instrument usage or smoothness of motion were significant for bleeding. Differences were statistically significant for both hands (p < 0.05). OPIs showing bimanual asymmetry indicated lower proficiency. This study demonstrates that kinematic and video analytics can establish a correlation between objective performance metrics and bleeding events in an ex vivo perfused lobectomy. Further study could assist in the development of focused exercises and simulation on objective domains to help improve overall performance and reducing complications during RL.

Outcome prediction in bariatric surgery through video-based assessment

INTRODUCTION

The relationship between intraoperative surgical performance scores and patient outcomes has not been demonstrated at a single-case level. The GEARS score is a Likert-based scale that quantifies robotic surgical proficiency in 5 domains. Given that even highly skilled surgeons can have variability in their skill among their cases, we hypothesized that at a patient level, higher surgical skill as determined by the GEARS score will predict individual patient outcomes.

METHODS

Patients undergoing robotic sleeve gastrectomy between July 2018 and January 2021 at a single-health care system were captured in a prospective database. Bivariate Pearson's correlation was used to compare continuous variables, one-way ANOVA for categorical variables compared with a continuous variable, and chi-square for two categorical variables. Significant variables in the univariable screen were included in a multivariable linear regression model. Two-tailed p-value < 0.05 was considered significant.

RESULTS

Of 162 patients included, 9 patients (5.5%) experienced a serious morbidity within 30 days. The average excess weight loss (EWL) was 72 ± 12% at 6 months and 74 ± 15% at 12 months. GEARS score was not significantly correlated with EWL at 6 months (p = 0.349), 12 months (p = 0.468), or serious morbidity (p = 0.848) on unadjusted analysis. After adjusting, total GEARS score was not correlated with serious morbidity (p = 0.914); however, GEARS score did predict EWL at 6 (p < 0.001) and 12 months (p < 0.001). All GEARS subcomponent scores, bimanual dexterity, depth perception, efficiency, force sensitivity, and robotic control were predictive of EWL at 6 months (p < 0.001) and 12 months (p < 0.001) on multivariable analysis.

CONCLUSION

For patients undergoing sleeve gastrectomy, surgical skill as assessed by the GEARS score was correlated with EWL, suggesting that better performance of a sleeve gastrectomy can result in improved postoperative weight loss.

Identification of Main Influencers of Surgical Efficiency and Variability Using Task-Level Objective Metrics: A Five-Year Robotic Sleeve Gastrectomy Case Series

Objective

Surgical efficiency and variability are critical contributors to optimal outcomes, patient experience, care team experience, and total cost to treat per disease episode. Opportunities remain to develop scalable, objective methods to quantify surgical behaviors that maximize efficiency and reduce variability. Such objective measures can then be used to provide surgeons with timely and user-specific feedbacks to monitor performances and facilitate training and learning. In this study, we used objective task-level analysis to identify dominant contributors toward surgical efficiency and variability across the procedural steps of robotic-assisted sleeve gastrectomy (RSG) over a five-year period for a single surgeon. These results enable actionable insights that can both complement those from population level analyses and be tailored to an individual surgeon's practice and experience.

Methods

Intraoperative video recordings of 77 RSG procedures performed by a single surgeon from 2015 to 2019 were reviewed and segmented into surgical tasks. Surgeon-initiated events when controlling the robotic-assisted surgical system were used to compute objective metrics. A series of multi-staged regression analysis were used to determine: if any specific tasks or patient body mass index (BMI) statistically impacted procedure duration; which objective metrics impacted critical task efficiency; and which task(s) statistically contributed to procedure variability.

Results

Stomach dissection was found to be the most significant contributor to procedure duration (β = 0.344, p< 0.001; R = 0.81, p< 0.001) followed by surgical inactivity and stomach stapling. Patient BMI was not found to be statistically significantly correlated with procedure duration (R = −0.01, p = 0.90). Energy activation rate, a robotic system event-based metric, was identified as a dominant feature in predicting stomach dissection duration and differentiating earlier and later case groups. Reduction of procedure variability was observed between earlier (2015-2016) and later (2017-2019) groups (IQR = 14.20 min vs. 6.79 min). Stomach dissection was found to contribute most to procedure variability (β = 0.74, p < 0.001).

Conclusions

A surgical task-based objective analysis was used to identify major contributors to surgical efficiency and variability. We believe this data-driven method will enable clinical teams to quantify surgeon-specific performance and identify actionable opportunities focused on the dominant surgical tasks impacting overall procedure efficiency and consistency.

Machine learning for technical skill assessment in surgery: a systematic review

Accurate and objective performance assessment is essential for both trainees and certified surgeons. However, existing methods can be time consuming, labor intensive, and subject to bias. Machine learning (ML) has the potential to provide rapid, automated, and reproducible feedback without the need for expert reviewers. We aimed to systematically review the literature and determine the ML techniques used for technical surgical skill assessment and identify challenges and barriers in the field. A systematic literature search, in accordance with the PRISMA statement, was performed to identify studies detailing the use of ML for technical skill assessment in surgery. Of the 1896 studies that were retrieved, 66 studies were included. The most common ML methods used were Hidden Markov Models (HMM, 14/66), Support Vector Machines (SVM, 17/66), and Artificial Neural Networks (ANN, 17/66). 40/66 studies used kinematic data, 19/66 used video or image data, and 7/66 used both. Studies assessed the performance of benchtop tasks (48/66), simulator tasks (10/66), and real-life surgery (8/66). Accuracy rates of over 80% were achieved, although tasks and participants varied between studies. Barriers to progress in the field included a focus on basic tasks, lack of standardization between studies, and lack of datasets. ML has the potential to produce accurate and objective surgical skill assessment through the use of methods including HMM, SVM, and ANN. Future ML-based assessment tools should move beyond the assessment of basic tasks and towards real-life surgery and provide interpretable feedback with clinical value for the surgeon.

PROSPERO: CRD42020226071

A Probabilistic Approach To Surgical Tasks and Skill Metrics

Identifying and quantifying the activities that compose surgery is essential for effective interventions, computer-aided analyses and the advancement of surgical data science. For example, recent studies have shown that objective metrics (referred to as objective performance indicators, OPIs) computed during key surgical tasks correlate with surgeon skill and clinical outcomes. Unambiguous identification of these surgical tasks can be particularly challenging for both human annotators and algorithms. Each surgical procedure has multiple approaches, each surgeon has their own level of skill, and the initiation and termination of surgical tasks can be subject to interpretation. As such, human annotators and machine learning models face the same basic problem, accurately identifying the boundaries of surgical tasks despite variable and unstructured information. For use in surgeon feedback, OPIs should also be robust to the variability and diversity in this data. To mitigate this difficulty, we propose a probabilistic approach to surgical task identification and calculation of OPIs. Rather than relying on tasks that are identified by hard temporal boundaries, we demonstrate an approach that relies on distributions of start and stop times, for a probabilistic interpretation of when the task was performed. We first use hypothetical data to outline how this approach is superior to other conventional approaches. Then we present similar analyses on surgical data. We find that when surgical tasks are identified by their individual probabilities, the resulting OPIs are less sensitive to noise in the identification of the start and stop times. These results suggest that this probabilistic approach holds promise for the future of surgical data science.

Review of automated performance metrics to assess surgical technical skills in robot-assisted laparoscopy

INTRODUCTION

Robot-assisted laparoscopy is a safe surgical approach with several studies suggesting correlations between complication rates and the surgeon's technical skills. Surgical skills are usually assessed by questionnaires completed by an expert observer. With the advent of surgical robots, automated surgical performance metrics (APMs)-objective measures related to instrument movements-can be computed. The aim of this systematic review was thus to assess APMs use in robot-assisted laparoscopic procedures. The primary outcome was the assessment of surgical skills by APMs and the secondary outcomes were the association between APM and surgeon parameters and the prediction of clinical outcomes.

METHODS

A systematic review following the PRISMA guidelines was conducted. PubMed and Scopus electronic databases were screened with the query "robot-assisted surgery OR robotic surgery AND performance metrics" between January 2010 and January 2021. The quality of the studies was assessed by the medical education research study quality instrument. The study settings, metrics, and applications were analysed.

RESULTS

The initial search yielded 341 citations of which 16 studies were finally included. The study settings were either simulated virtual reality (VR) (4 studies) or real clinical environment (12 studies). Data to compute APMs were kinematics (motion tracking), and system and specific events data (actions from the robot console). APMs were used to differentiate expertise levels, and thus validate VR modules, predict outcomes, and integrate datasets for automatic recognition models. APMs were correlated with clinical outcomes for some studies.

CONCLUSIONS

APMs constitute an objective approach for assessing technical skills. Evidence of associations between APMs and clinical outcomes remain to be confirmed by further studies, particularly, for non-urological procedures. Concurrent validation is also required.