Mapping the robotic hysterectomy learning curve and re-establishing surgical training metrics

Novel workflow analysis of robot-assisted hysterectomy through objective performance indicators: a pilot study

Introduction

The curriculum for a da Vinci surgeon in gynecology requires special training before a surgeon performs their first independent case, but standardized, objective assessments of a trainee's workflow or skills learned during clinical cases are lacking. This pilot study presents a methodology to evaluate intraoperative surgeon behavior in hysterectomy cases through standardized surgical step segmentation paired with objective performance indicators (OPIs) calculated directly from robotic data streams. This method can provide individual case analysis in a truly objective capacity.

Materials and methods

Surgical data from six robot-assisted total laparoscopic hysterectomies (rTLH) performed by two experienced surgeons was collected prospectively using an Intuitive Data Recorder. Each rTLH video was annotated and segmented into specific, functional surgical steps based on the recorded video. Once annotated, OPIs were compared through workflow analysis and across surgeons during two critical surgical steps: colpotomy and vaginal cuff closure.

Results

Through visualization of the individual steps over time, we observe workflow consistencies and variabilities across individual surgeons of a similar experience level at the same hospital, creating unique surgeon behavior signatures across each surgical case. OPI differences across surgeons were observed for both the colpotomy and vaginal cuff closure steps, specifically reflecting camera movement, energy usage and clutching behaviors. Comparing colpotomy and vaginal cuff closure time needed for the step and the events of energy use were significantly different (p < 0.001). For the comparison between the two surgeons only the event count for camera movement during colpotomy showed significant differences (p = 0.03).

Conclusion

This pilot study presents a novel methodology to analyze and compare individual rTLH procedures with truly objective measurements. Through collection of robotic data streams and standardized segmentation, OPI measurements for specific rTLH surgery steps can be reliably calculated and compared to those of other surgeons. This provides opportunity for critical standardization to the gynecology field, which can be integrated into individualized training plans in the future. However, more studies are needed to establish context surrounding these metrics in gynecology.

Skill assessment based on clutch use in cross-platform robot-assisted surgery

BACKGROUND

Many studies have investigated the transfer of skills between laparoscopic and robot-assisted surgery (RAS). These studies have considered time, error, and clinical outcomes in the assessment of skill transfer. However, little is known about the specific operations of the surgeon. Clutch control use is an important skill in RAS. Therefore, the present study aimed to propose a novel objective algorithm based on computer vision that can automatically evaluate a surgeon's clutch use. Additionally, the study aimed to evaluate the correlation between clutch metrics and surgical skill on different surgical robot platforms.

METHODS

The robotic surgery training center of Wuhan University trained 30 laparoscopic surgeons as the study group between 2023 and 2024. Laparoscopic surgeons were trained by combining robotic simulator exercises and RAS animal experiments. During the training, video and hand movement data were collected. Hand movements identified by a skin-color model were combined with labeling information to classify clutch use. The metrics were validated on different robotic platforms (dv-Trainer, EDGE MP1000, Toumai™ MT1000, and DaVinci Xi system) and among surgeons with different surgical skill levels.

RESULTS

On the robotic simulator, clutch accuracy in the expert group was significantly higher than in the study group for all tasks. No significant differences were observed in the number of clutches between the expert and study groups. In the RAS experiment, the number of clutches decreased significantly for both study and expert groups. The accuracy was maintained at a high level in the expert group but decreased rapidly in the study group.

CONCLUSIONS

We proposed a new objective assessment of surgical skills, clutch use metrics, in cross-platform RAS. Additionally, we verified that the metrics significantly correlated with the surgical skill levels of the surgeons.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

A systematic review of general surgery robotic training curriculums

Background

As of the most recent surveys of resident programs in 2018, only slightly more than half of programs have formal robotic training curriculums implemented. Fewer programs have further assessed their own curriculum and its benefit.

Method

We conducted a PubMed/MEDLINE literature search for robotic surgery curriculums and those that had assessment of their programs.

Results

A total of 11 studies were reviewed. When reviewed in chronological order, there has been a progression towards more robotic specific objective data analysis as opposed to subjective surveying. There is a wide variation in curriculums, but simulation use is pervasive.

Conclusions

Our review makes evident two important concepts-there is great variety in training curriculums and there is great benefit in implementation. The importance is in establishment of what makes resident training effective and supports the adaptable and successful surgeon. This may come from an adaptable curriculum but a structured test-out assessment.

Surgical Science - Simbionix Robotic Hysterectomy Simulator: Validating a New Tool

STUDY OBJECTIVE

To gather validity evidence for and determine acceptability of Surgical Science - Simbionix Hysterectomy Modules for the DaVinci Xi console simulation system and evaluate performance benchmarks between novice and experienced or expert surgeons.

DESIGN

Prospective education study (Messick validity framework) SETTING: Multi-center, academic medical institutions PARTICIPANTS: Residents, fellows, and faculty in Obstetrics and Gynecology were invited to participate at 3 institutions. Participants were categorized by experience level: less than 10 hysterectomies (novice), 10 to 50 hysterectomies (experienced), and greater than 50 hysterectomies (expert). A total of 10 novice, 10 experienced, and 14 expert surgeons were included.

INTERVENTIONS

Participants completed 4 simulator modules (ureter identification, bladder flap development, colpotomy, complete hysterectomy) and a qualitative survey. Simulator recordings were reviewed in duplicate by educators in minimally invasive gynecologic surgery using the Modified Global Evaluative Assessment of Robotic Skills (GEARS) rating scale.

MEASUREMENTS AND MAIN RESULTS

Most participants felt the simulator realistically simulated robotic hysterectomy (64.7%) and that feedback provided by the simulator was as or more helpful than feedback from previous simulators (88.2%) but less helpful than feedback provided in the OR (73.5%). Participants felt this simulator would be helpful for teaching junior residents. Simulator-generated metrics correlated with GEARS performance for bladder flap and ureter identification modules in multiple domains including total movements and total time for completion. GEARS performance for the bladder flap module correlated with experience level (novice vs experienced/expert) in domains of interest and total score but did not consistently correlate for the other procedural modules. Performance benchmarks were evaluated for the bladder flap module for each GEARS domain and total score.

CONCLUSION

The modules were well received by participants of all experience levels. Individual simulation modules appear to better discriminate between novice and experienced/expert users than overall simulator performance. Based on these data and participant feedback, use of individual modules in early residency education may be helpful for providing feedback and may ultimately serve as one component of determining readiness to perform robotic hysterectomy.