Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

Essential components and validation of multi-specialty robotic surgical training curricula: a systematic review

INTRODUCTION

The rapid adoption of robotic surgical systems has overtook the development of standardized training and competency assessment for surgeons, resulting in an unmet educational need in this field. This systematic review aims to identify the essential components and evaluate the validity of current robotic training curricula across all surgical specialties.

METHODS

A systematic search of MEDLINE, EMBASE, Emcare, and CINAHL databases was conducted to identify the studies reporting on multi-specialty or specialty-specific surgical robotic training curricula, between January 2000 and January 2024. We extracted the data according to Kirkpatrick's curriculum evaluation model and Messick's concept of validity. The quality of studies was assessed using the Medical Education Research Study Quality Instrument (MERSQI).

RESULTS

From the 3687 studies retrieved, 66 articles were included. The majority of studies were single-center ( n = 52, 78.8%) and observational ( n = 58, 87.9%) in nature. The most commonly reported curriculum components include didactic teaching ( n = 48, 72.7%), dry laboratory skills ( n = 46, 69.7%), and virtual reality (VR) simulation ( n = 44, 66.7%). Curriculum assessment methods varied, including direct observation ( n = 44, 66.7%), video assessment ( n = 26, 39.4%), and self-assessment (6.1%). Objective outcome measures were used in 44 studies (66.7%). None of the studies were fully evaluated according to Kirkpatrick's model, and five studies (7.6%) were fully evaluated according to Messick's framework. The studies were generally found to have moderate methodological quality with a median MERSQI of 11.

CONCLUSIONS

Essential components in robotic training curricula identified were didactic teaching, dry laboratory skills, and VR simulation. However, variability in assessment methods used and notable gaps in curricula validation remain evident. This highlights the need for standardized evidence-based development, evaluation, and reporting of robotic curricula to ensure the effective and safe adoption of robotic surgical systems.

Scoping review and proposed curriculum for robotic hepatopancreatobiliary surgery training

BACKGROUND

HPB surgery is being increasingly performed robotically worldwide. However, there is no consensus on what constitutes adequate training or an established curriculum. We evaluate the existing literature on formal education in robotic hepatopancreaticobiliary (HPB) surgery and propose a curriculum using Kern's six-step curriculum development model.

METHODS

A systematic search was performed across major databases and the methodology of the Joanna Briggs Institute was followed. The PRISMA-ScR was conformed in reporting. Evidence pertaining to cholecystectomy alone was excluded and studies that described formal training pathways were included.

RESULTS

Fifteen curricula were included with predilection towards the pancreas (n = 7, liver: n = 5, combination: n = 3). Almost all studies proposed initial robot system training through online modules, observership and console simulation exercises. Following this, six curricula described procedure-specific anastomosis training. Almost all studies described mentorship and proctorship. The assessment for implementation commonly described includes objective structured assessment of technical skill (OSATS) and cumulative sum technique (CUSUM) for operation time, conversion-to-open rate and postoperative complications.

DISCUSSION

This study has summarised the formal curricula for learning robotic HPB surgery. The majority share similar implementation tools. A comprehensive curriculum based on validated educational principles has been proposed which incorporates these elements.

Video-based robotic surgical action recognition and skills assessment on porcine models using deep learning

OBJECTIVES

This study aimed to develop an automated skills assessment tool for surgical trainees using deep learning.

BACKGROUND

Optimal surgical performance in robot-assisted surgery (RAS) is essential for ensuring good surgical outcomes. This requires effective training of new surgeons, which currently relies on supervision and skill assessment by experienced surgeons. Artificial Intelligence (AI) presents an opportunity to augment existing human-based assessments.

METHODS

We used a network architecture consisting of a convolutional neural network combined with a long short-term memory (LSTM) layer to create two networks for the extraction and analysis of spatial and temporal features from video recordings of surgical procedures, facilitating action recognition and skill assessment.

RESULTS

21 participants (16 novices and 5 experienced) performed 16 different intra-abdominal robot-assisted surgical procedures on porcine models. The action recognition network achieved an accuracy of 96.0% in identifying surgical actions. A GradCAM filter was used to enhance the model interpretability. The skill assessment network had an accuracy of 81.3% in classifying novices and experiences. Procedure plots were created to visualize the skill assessment.

CONCLUSION

Our study demonstrated that AI can be used to automate surgical action recognition and skill assessment. The use of a porcine model enables effective data collection at different levels of surgical performance, which is normally not available in the clinical setting. Future studies need to test how well AI developed within a porcine setting can be used to detect errors and provide feedback and actionable skills assessment in the clinical setting.

Validation of objective performance metrics via an intelligent medical network in gynecological oncology robotic surgery

BACKGROUND

Automated performance metrics (APMs) are potentially useful to accurately assess and improve surgeon skills and patient outcomes, while their clinical use is currently limited. We report on the use of the Medicaroid Intelligent Network System (MINS™), a network support system platform used together with the "hinotori" surgical robot system (hinotori™) for the collection of data logs from surgeries, and discuss its potential to improve surgical outcomes.

METHODS

This study prospectively evaluated the efficacy of MINS™ for collecting data logs in gynecologic oncologic robotic surgery between December 2022 and February 2024 in nine patients. MINS™ regularly communicated with the hinotori™ via a secure network to collect and send system logs to a cloud server, quantifying various performance data for the evaluation of surgical outcomes.

RESULTS

Clinical data on hinotori™ movement were successfully extracted. The number of operation arm (OA) changes was significantly higher in Patient No. 7, who underwent pelvic lymph node dissection. OA3 monopolar was used more frequently than OA1 bipolar for coagulation (mean: 6.7% vs 2.5%, P ˂0.001). The error count and percentage of inactive time associated with OA collisions decreased dramatically after Patient No. 6, following the version upgrade in July 2024.

CONCLUSION

MINS™ utilizes technology to connect the hinotori™ to various systems via the Internet, allowing objective evaluations of surgical procedures from data logs. MINS™ is a clinically applicable APM system that objectively analyzes a surgeon's individuality and has the potential to improve surgical techniques and promote standardization.

A pan-European survey of robotic training for gastrointestinal surgery: European Robotic Surgery Consensus (ERSC) initiative

BACKGROUND

There has been a recent rapid growth in the adoption of robotic systems across Europe. This study aimed to capture the current state of robotic training in gastrointestinal (GI) surgery and to identify potential challenges and barriers to training within Europe.

METHODS

A pan-European survey was designed to account for the opinion of the following GI surgery groups: (i) experts/independent practitioners; (ii) trainees with robotic access; (iii) trainees without robotic access; (iv) robotic industry representatives. The survey explored various aspects, including stakeholder opinions on bedside assisting, console operations, challenges faced and performance assessment. It was distributed through multiple European surgical societies and industry, in addition to social media and snowball sampling, between December 2023 and March 2024.

RESULTS

A total of 1360 participants responded, with valid/complete responses from 1045 participants across 38 European countries. Six hundred and ninety-five (68.0%) experts and trainees were not aware of a dedicated robotic training curriculum for trainees, with 13/23 (56.5%) industry representatives not incorporating training for trainees in their programme. Among trainees with access to robotic systems, 94/195 (48.2%) had not performed any robotic cases, citing challenges including a lack of certified robotic trainers and training lists. Both experts and trainees agreed that trainees should start bedside assisting and operating on the console earlier than they currently do. Assessment tools of trainee performance were not being used by 139/479 (29.0%) participants.

CONCLUSION

This pan-European survey highlights the need for a standardised robotic curriculum to address the gap in visceral training, assessment and certification. A greater emphasis may be required on implementing robotic training earlier through simulation training, dual console learning, bedside assisting, key clinical performance indicators, and assessment tools. The findings will guide the development of a pan-European consensus on the essential components of a comprehensive training programme for GI robotic surgery.

Detailed analysis of learning phases and outcomes in robotic and endoscopic thyroidectomy

BACKGROUND

Thyroid surgery has undergone significant transformation with the introduction of minimally invasive techniques, particularly robotic and endoscopic thyroidectomy. These advancements offer improved precision and faster recovery but also present unique challenges. This study aims to compare the learning curves, operational efficiencies, and patient outcomes of robotic versus endoscopic thyroidectomy.

METHODS

A retrospective cohort study was conducted, analyzing 258 robotic (da Vinci) and 214 endoscopic thyroidectomy cases. Key metrics such as operation duration, drainage volume, lymph node dissection outcomes, and hypoparathyroidism incidence were assessed to understand surgical learning curves and efficiency.

RESULTS

Robotic thyroidectomy showed a longer learning curve with initially longer operation times and higher drainage volumes but superior lymph node dissection outcomes. Both techniques were safe, with no permanent hypoparathyroidism or recurrent laryngeal nerve damage reported. The study delineated four distinct stages in the robotic and endoscopic surgery learning curve, each marked by specific improvements in proficiency. Endoscopic thyroidectomy displayed a shorter learning curve, leading to quicker operational efficiency gains.

CONCLUSION

Robotic and endoscopic thyroidectomies are viable minimally invasive approaches, each with its learning curves and efficiency metrics. Despite initial challenges and a longer learning period for robotic surgery, its benefits in complex dissections may justify specialized training. Structured training programs tailored to each technique are crucial for improving outcomes and efficiency. Future research should focus on optimizing training protocols and increasing accessibility to these technologies, enhancing patient care in thyroid surgery.

Initial surgical performance in robot-assisted radical prostatectomy is associated with clinical outcomes and learning curves

BACKGROUND

The association between surgical performance ratings and clinical outcomes in robotic surgery is poorly understood. Additionally, no studies have reported on the relationship between the surgeon's initial case-skill evaluation and the learning curve in robot-assisted surgery. We evaluated whether an objective surgical technique evaluation score for initial robot-assisted radical prostatectomy (RARP) was associated with clinical outcomes and surgeons' learning curves.

METHODS

Six surgeons who were trained in and started to perform RARP at our institution were included. Anonymized, unedited videos of each surgeon's 10th RARP case were evaluated by three reviewers, using modified Objective Structured Assessment of Technical Skill (OSATS) scores. We then divided the surgeons into two groups on the basis of these OSATS scores. We retrospectively compared the clinical outcomes and learning curves of the console time of the two groups for consecutive RARPs, performed from March 2018 to July 2023.

RESULTS

We analyzed 258 RARPs (43 cases/surgeon), including 129 cases performed by high-OSATS score surgeons (18.2-19.3 points) and 129 cases performed by low-OSATS score surgeons (11.9-16.0 points). Overall, the high-OSATS score group had significantly shorter operation and console times than the low-OSATS score group did (both P < 0.01) and their patients' rate of continence recovery by 3 months post-RARP was significantly higher (P = 0.03). However, complications, blood loss, and positive margins did not differ between the groups (P = 0.08, P = 0.51, and P = 0.90, respectively). The high-OSATS score group had a significantly shorter console time than the low-OSATS score group did after the 11-20 cases.

CONCLUSIONS

The OSATS score in early RARP cases can predict subsequent surgical outcomes and surgeons' learning curves.

Robotic performance metrics model fellow proficiency in living donor nephrectomy

We investigated the use of robotic objective performance metrics (OPM) to predict number of cases to proficiency and independence among abdominal transplant fellows performing robot-assisted donor nephrectomy (RDN). 101 RDNs were performed by 5 transplant fellows from September 2020 to October 2023. OPM included fellow percent active control time (%ACT) and handoff counts (HC). Proficiency was defined as ACT ≥ 80% and HC ≤ 2, and independence as ACT ≥ 99% and HC ≤ 1. Case number was significantly associated with increasing fellow %ACT, with proficiency estimated at 14 cases and independence at 32 cases (R2 = 0.56, p < 0.001). Similarly, case number was significantly associated with decreasing HC, with proficiency at 18 cases and independence at 33 cases (R2 = 0.29, p < 0.001). Case number was not associated with total active console time (p = 0.91). Patient demographics, operative characteristics, and outcomes were not associated with OPM, except for donor estimated blood loss (EBL), which positively correlated with HC. Abdominal transplant fellows demonstrated proficiency at 14-18 cases and independence at 32-33 cases. Total active console time remained unchanged, suggesting that increasing fellow autonomy does not impede operative efficiency. These findings may serve as a benchmark for training abdominal transplant surgery fellows independently and safely in RDN.

European Robotic Surgery Consensus (ERSC): Protocol for the development of a consensus in robotic training for gastrointestinal surgery trainees

BACKGROUND

The rapid adoption of robotic surgical systems across Europe has led to a critical gap in training and credentialing for gastrointestinal (GI) surgeons. Currently, there is no existing standardised curriculum to guide robotic training, assessment and certification for GI trainees. This manuscript describes the protocol to achieve a pan-European consensus on the essential components of a comprehensive training programme for GI robotic surgery through a five-stage process.

METHODS AND ANALYSIS

In Stage 1, a Steering Committee, consisting of international experts, trainees and educationalists, has been established to lead and coordinate the consensus development process. In Stage 2, a systematic review of existing multi-specialty robotic training curricula will be performed to inform the formulation of key position statements. In Stage 3, a comprehensive survey will be disseminated across Europe to capture the current state of robotic training and identify potential challenges and opportunities for improvement. In Stage 4, an international panel of GI surgeons, trainees, and robotic theatre staff will participate in a three-round Delphi process, seeking ≥ 70% agreement on crucial aspects of the training curriculum. Industry and patient representatives will be involved as external advisors throughout this process. In Stage 5, the robotic training curriculum for GI trainees will be finalised in a dedicated consensus meeting, culminating in the production of an Explanation and Elaboration (E&E) document.

REGISTRATION DETAILS

The study protocol has been registered on the Open Science Framework (https://osf.io/br87d/).

The deconstructed procedural description in robotic colorectal surgery

Increasing robotic surgical utilisation in colorectal surgery internationally has strengthened the need for standardised training. Deconstructed procedural descriptions identify components of an operation that can be integrated into proficiency-based progression training. This approach allows both access to skill level appropriate training opportunities and objective and comparable assessment. Robotic colorectal surgery has graded difficulty of operative procedures lending itself ideally to component training. Developing deconstructed procedural descriptions may assist in the structure and progression components in robotic colorectal surgical training. There is no currently published guide to procedural descriptions in robotic colorectal surgical or assessment of their training utility. This scoping review was conducted in June 2022 following the PRISMA-ScR guidelines to identify which robotic colorectal surgical procedures have available component-based procedural descriptions. Secondary aims were identifying the method of development of these descriptions and how they have been adapted in a training context. 20 published procedural descriptions were identified covering 8 robotic colorectal surgical procedures with anterior resection the most frequently described procedure. Five publications included descriptions of how the procedural description has been utilised for education and training. From these publications terminology relating to using deconstructed procedural descriptions in robotic colorectal surgical training is proposed. Development of deconstructed robotic colorectal procedural descriptions (DPDs) in an international context may assist in the development of a global curriculum of component operating competencies supported by objective metrics. This will allow for standardisation of robotic colorectal surgical training and supports a proficiency-based training approach.

Evaluation of different approaches to define expert benchmark scores for new robotic training simulators based on the Medtronic HUGO™ RAS surgical robot experience

New robot-assisted surgery platforms being developed will be required to have proficiency-based simulation training available. Scoring methodologies and performance feedback for trainees are currently not consistent across all robotic simulator platforms. Also, there are virtually no prior publications on how VR simulation passing benchmarks have been established. This paper compares methods evaluated to determine the proficiency-based scoring thresholds (a.k.a. benchmarks) for the new Medtronic Hugo™ RAS robotic simulator. Nine experienced robotic surgeons from multiple disciplines performed the 49 skills exercises 5 times each. The data were analyzed in 3 different ways: (1) include all data collected, (2) exclude first sessions, (3) exclude outliers. Eliminating the first session discounts becoming familiar with the exercise. Discounting outliers allows removal of potentially erroneous data that may be due to technical issues, unexpected distractions, etc. Outliers were identified using a common statistical technique involving the interquartile range of the data. Using each method above, mean and standard deviations were calculated, and the benchmark was set at a value of 1 standard deviation above the mean. In comparison to including all the data, when outliers are excluded, fewer data points are removed than just excluding first sessions, and the metric benchmarks are made more difficult by an average of 11%. When first sessions are excluded, the metric benchmarks are made easier by an average of about 2%. In comparison with benchmarks calculated using all data points, excluding outliers resulted in the biggest change making the benchmarks more challenging. We determined that this method provided the best representation of the data. These benchmarks should be validated with future clinical training studies.