Development and Validation of an Objective Scoring Tool for Robot-Assisted Partial Nephrectomy: Scoring for Partial Nephrectomy

Current Standards for Training in Robot-assisted Surgery and Endourology: A Systematic Review

BACKGROUND AND OBJECTIVE

Different training programs have been developed to improve trainee outcomes in urology. However, evidence on the optimal training methodology is sparse. Our aim was to provide a comprehensive description of the training programs available for urological robotic surgery and endourology, assess their validity, and highlight the fundamental elements of future training pathways.

METHODS

We systematically reviewed the literature using PubMed/Medline, Embase, and Web of Science databases. The validity of each training model was assessed. The methodological quality of studies on metrics and curricula was graded using the MERSQI scale. The level of evidence (LoE) and level of recommendation for surgical curricula were awarded using the educational Oxford Centre for Evidence-Based Medicine classification.

KEY FINDINGS AND LIMITATIONS

A total of 75 studies were identified. Many simulators have been developed to aid trainees in mastering skills required for both robotic and endourology procedures, but only four demonstrated predictive validity. For assessment of trainee proficiency, we identified 18 in robotics training and six in endourology training; however, the majority are Likert-type scales. Although proficiency-based progression (PBP) curricula demonstrated superior outcomes to traditional training in preclinical settings, only four of six (67%) in robotics and three of nine (33%) in endourology are PBP-based. Among these, the Fundamentals of Robotic Surgery and the SIMULATE curricula have the highest LoE (level 1b). The lack of a quantitative synthesis is the main limitation of our study.

CONCLUSIONS AND CLINICAL IMPLICATIONS

Training curricula that integrate simulators and PBP methodology have been introduced to standardize trainee outcomes in robotics and endourology. However, evidence regarding their educational impact remains restricted to preclinical studies. Efforts should be made to expand these training programs to different surgical procedures and assess their clinical impact.

Surgeon Skill and Perioperative Outcomes in Robot-Assisted Partial Nephrectomy

Importance

Technical skill in complex surgical procedures may affect clinical outcomes, and there is growing interest in understanding the clinical implications of surgeon proficiency levels.

Objectives

To determine whether surgeon scores representing technical skills of robot-assisted kidney surgery are associated with patient outcomes.

Design, Setting, and Participants

This quality improvement study included 10 urological surgeons participating in a surgical collaborative in Michigan from July 2021 to September 2022. Each surgeon submitted up to 7 videos of themselves performing robot-assisted partial nephrectomy. Videos were segmented into 6 key steps, yielding 127 video clips for analysis. Each video clip was deidentified and distributed to at least 3 of the 24 blinded peer surgeons from the collaborative who also perform robot-assisted partial nephrectomy. Reviewers rated technical skill and provided written feedback. Statistical analysis was performed from May 2023 to January 2024.

Main Outcomes and Measures

Reviewers scored each video clip using a validated instrument to assess technical skill for partial nephrectomy on a scale of 1 to 5 (higher scores indicating greater skill). For all submitting surgeons, outcomes from a clinical registry were assessed for length of stay (LOS) greater than 3 days, estimated blood loss (EBL) greater than 500 mL, warm ischemia time (WIT) greater than 30 minutes, positive surgical margin (PSM), 30-day emergency department (ED) visits, and 30-day readmission.

Results

Among the 27 unique surgeons who participated in this study as reviewers and/or individuals performing the procedures, 3 (11%) were female, and the median age was 47 (IQR, 39-52) years. Risk-adjusted outcomes were associated with scores representing surgeon skills. The overall performance score ranged from 3.5 to 4.7 points with a mean (SD) of 4.1 (0.4) points. Greater skill was correlated with significantly lower rates of LOS greater than 3 days (-6.8% [95% CI, -8.3% to -5.2%]), EBL greater than 500 mL (-2.6% [95% CI, -3.0% to -2.1%]), PSM (-8.2% [95% CI, -9.2% to -7.2%]), ED visits (-3.9% [95% CI, -5.0% to -2.8%]), and readmissions (-5.7% [95% CI, -6.9% to -4.6%]) (P < .001 for all). Higher overall score was also associated with higher partial nephrectomy volume (β coefficient, 11.4 [95% CI, 10.0-12.7]; P < .001).

Conclusions and Relevance

In this quality improvement study on video-based evaluation of robot-assisted partial nephrectomy, higher technical skill was associated with lower rates of adverse clinical outcomes. These findings suggest that video-based evaluation plays a role in assessing surgical skill and can be used in quality improvement initiatives to improve patient care.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

The application of objective clinical human reliability analysis (OCHRA) in the assessment of basic robotic surgical skills

BACKGROUND

Using a validated, objective, and standardised assessment tool to assess progression and competency is essential for basic robotic surgical training programmes. Objective clinical human reliability analysis (OCHRA) is an error-based assessment tool that provides in-depth analysis of individual technical errors. We conducted a feasibility study to assess the concurrent validity and reliability of OCHRA when applied to basic, generic robotic technical skills assessment.

METHODS

Selected basic robotic surgical skill tasks, in virtual reality (VR) and dry lab equivalent, were performed by novice robotic surgeons during an intensive 5-day robotic surgical skills course on da Vinci® X and Xi surgical systems. For each task, we described a hierarchical task analysis. Our developed robotic surgical-specific OCHRA methodology was applied to error events in recorded videos with a standardised definition. Statistical analysis to assess concurrent validity with existing tools and inter-rater reliability were performed.

RESULTS

OCHRA methodology was applied to 272 basic robotic surgical skills tasks performed by 20 novice robotic surgeons. Performance scores improved from the start of the course to the end using all three assessment tools; Global Evaluative Assessment of Robotic Skills (GEARS) [VR: t(19) = - 9.33, p < 0.001] [dry lab: t(19) = - 10.17, p < 0.001], OCHRA [VR: t(19) = 6.33, p < 0.001] [dry lab: t(19) = 10.69, p < 0.001] and automated VR [VR: t(19) = - 8.26, p < 0.001]. Correlation analysis, for OCHRA compared to GEARS and automated VR scores, shows a significant and strong inverse correlation in every VR and dry lab task; OCHRA vs GEARS [VR: mean r = - 0.78, p < 0.001] [dry lab: mean r = - 0.82, p < 0.001] and OCHRA vs automated VR [VR: mean r = - 0.77, p < 0.001]. There is very strong and significant inter-rater reliability between two independent reviewers (r = 0.926, p < 0.001).

CONCLUSION

OCHRA methodology provides a detailed error analysis tool in basic robotic surgical skills with high reliability and concurrent validity with existing tools. OCHRA requires further evaluation in more advanced robotic surgical procedures.