Assessing system operation skills in robotic surgery trainees

Surgical training simulation modalities in minimally invasive surgery: How to achieve evidence-based curricula by translational research

BACKGROUND

Surgery has evolved from a hands-on discipline where skills were acquired via the "learning by doing" principle to a surgical science with attention to patient safety, health care effectiveness and evidence-based research. A variety of simulation modalities have been developed to meet the need for effective resident training. So far, research regarding surgical training for minimally invasive surgery has been extensive but also heterogenous in grade of evidence.

METHODS

A literature search was conducted to summarize current knowledge about simulation training and to guide research towards evidence-based curricula with translational effects. This was conducted using a variety of terms in PubMed for English articles up to October 2024. Results are presented in a structured narrative review.

RESULTS

For virtual reality simulators, there is sound evidence for effective training outcomes. The required instruments for the development of minimally invasive surgery curricula combining different simulation modalities to create a clinical benefit are known and published.

CONCLUSION

Surgeons are the main creators for minimally invasive surgery training curricula and often follow a hands-on oriented approach that leaves out equally important aspects of assessment, evaluation, and feedback. Further high-quality research that includes available evidence in this field promises to improve patient safety in surgical disciplines.

Evaluating the benefit of contact-force feedback in robotic surgery using the Saroa surgical system: A preclinical study

INTRODUCTION

Robotic surgery without contact-force feedback could be less safe, as forces exerted by the robot system may exceed tissue tolerance. This study aimed to evaluate the benefit of contact-force feedback.

METHODS

Nine junior and 11 senior surgeons performed two tasks using Saroa, a robotic surgical system with a force feedback function. In Task A, the participants estimated the order of stiffness of substances when feedback was on and off. In Task B, the effect of feedback on compression with a designated force (3 N) was assessed.

RESULTS

In Task A, the proportion of participants who correctly estimated the order of stiffness of the substances was similar when feedback was on and off. However, the median maximum force applied to the substances was significantly smaller when feedback was on than when it was off (5.0 vs. 6.9 N, p = .011), which was more obvious among the junior surgeons (5.0 vs. 7.7 N, p = .015) than among the senior surgeons (4.7 vs. 5.9 N, p = .288). In Task B, deviations from the designated force (3 N) for three substances were smaller when feedback was on (0, -0.1, and 0.7, respectively) than when it was off (-0.3, -0.5, and 1.3, respectively). Regarding the dispersion of the force to the substances, the interquartile range tended to be smaller with feedback; this trend was more obvious in the junior surgeons.

CONCLUSION

With contact-force feedback, tissue stiffness could be estimated with a small force, particularly by the junior surgeons; specified force could be accurately applied to the tissue.

Impact of artificial intelligence on the training of general surgeons of the future: a scoping review of the advances and challenges

PURPOSE

To explore artificial intelligence's impact on surgical education, highlighting its advantages and challenges.

METHODS

A comprehensive search across databases such as PubMed, Scopus, Scientific Electronic Library Online (SciELO), Embase, Web of Science, and Google Scholar was conducted to compile relevant studies.

RESULTS

Artificial intelligence offers several advantages in surgical training. It enables highly realistic simulation environments for the safe practice of complex procedures. Artificial intelligence provides personalized real-time feedback, improving trainees' skills. It efficiently processes clinical data, enhancing diagnostics and surgical planning. Artificial intelligence-assisted surgeries promise precision and minimally invasive procedures. Challenges include data security, resistance to artificial intelligence adoption, and ethical considerations.

CONCLUSIONS

Stricter policies and regulatory compliance are needed for data privacy. Addressing surgeons' and educators' reluctance to embrace artificial intelligence is crucial. Integrating artificial intelligence into curricula and providing ongoing training are vital. Ethical, bioethical, and legal aspects surrounding artificial intelligence demand attention. Establishing clear ethical guidelines, ensuring transparency, and implementing supervision and accountability are essential. As artificial intelligence evolves in surgical training, research and development remain crucial. Future studies should explore artificial intelligence-driven personalized training and monitor ethical and legal regulations. In summary, artificial intelligence is shaping the future of general surgeons, offering advanced simulations, personalized feedback, and improved patient care. However, addressing data security, adoption resistance, and ethical concerns is vital. Adapting curricula and providing continuous training are essential to maximize artificial intelligence's potential, promoting ethical and safe surgery.

Measures of performance and proficiency in robotic assisted surgery: a systematic review

Robotic assisted surgery (RAS) has seen a global rise in adoption. Despite this, there is not a standardised training curricula nor a standardised measure of performance. We performed a systematic review across the surgical specialties in RAS and evaluated tools used to assess surgeons' technical performance. Using the PRISMA 2020 guidelines, Pubmed, Embase and the Cochrane Library were searched systematically for full texts published on or after January 2020-January 2022. Observational studies and RCTs were included; review articles and systematic reviews were excluded. The papers' quality and bias score were assessed using the Newcastle Ottawa Score for the observational studies and Cochrane Risk Tool for the RCTs. The initial search yielded 1189 papers of which 72 fit the eligibility criteria. 27 unique performance metrics were identified. Global assessments were the most common tool of assessment (n = 13); the most used was GEARS (Global Evaluative Assessment of Robotic Skills). 11 metrics (42%) were objective tools of performance. Automated performance metrics (APMs) were the most widely used objective metrics whilst the remaining (n = 15, 58%) were subjective. The results demonstrate variation in tools used to assess technical performance in RAS. A large proportion of the metrics are subjective measures which increases the risk of bias amongst users. A standardised objective metric which measures all domains of technical performance from global to cognitive is required. The metric should be applicable to all RAS procedures and easily implementable. Automated performance metrics (APMs) have demonstrated promise in their wide use of accurate measures.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Automated performance metrics, learning curve and robotic colorectal surgery

BACKGROUND

The aim of this study was to evaluate the usefulness of Automated Performance Metrics (APMs) in assessing the learning curve.

METHODS

A retrospective review of 85 consecutive patients who underwent total robotic colorectal surgery at a single institution between August 2020 and October 2022 was performed. Patient demographics, operation type, and APMs were collected and analysed. Cumulative summation technique (CUSUM) was used to construct learning curves of surgeon console time (SCT), use of the fourth arm, clutch activation, instrument off screen (number and duration), and cut electrocautery activation.

RESULTS

Two phases with 50 and 35 cases were identified from the CUSUM graph for SCT. The SCT was significantly different between the two phases (176 and 251 min, p < 0.002). After adjustment for SCT, the APMs were not significantly different between the two phases.

CONCLUSIONS

Most APMs do not offer additional learning curve information when compared with SCT analysis alone.

An Automated Skill Assessment Framework Based on Visual Motion Signals and a Deep Neural Network in Robot-Assisted Minimally Invasive Surgery

Surgical skill assessment can quantify the quality of the surgical operation via the motion state of the surgical instrument tip (SIT), which is considered one of the effective primary means by which to improve the accuracy of surgical operation. Traditional methods have displayed promising results in skill assessment. However, this success is predicated on the SIT sensors, making these approaches impractical when employing the minimally invasive surgical robot with such a tiny end size. To address the assessment issue regarding the operation quality of robot-assisted minimally invasive surgery (RAMIS), this paper proposes a new automatic framework for assessing surgical skills based on visual motion tracking and deep learning. The new method innovatively combines vision and kinematics. The kernel correlation filter (KCF) is introduced in order to obtain the key motion signals of the SIT and classify them by using the residual neural network (ResNet), realizing automated skill assessment in RAMIS. To verify its effectiveness and accuracy, the proposed method is applied to the public minimally invasive surgical robot dataset, the JIGSAWS. The results show that the method based on visual motion tracking technology and a deep neural network model can effectively and accurately assess the skill of robot-assisted surgery in near real-time. In a fairly short computational processing time of 3 to 5 s, the average accuracy of the assessment method is 92.04% and 84.80% in distinguishing two and three skill levels. This study makes an important contribution to the safe and high-quality development of RAMIS.

Exploring the roles of artificial intelligence in surgical education: A scoping review

BACKGROUND

Technology-enhanced teaching and learning, including Artificial Intelligence (AI) applications, has started to evolve in surgical education. Hence, the purpose of this scoping review is to explore the current and future roles of AI in surgical education.

METHODS

Nine bibliographic databases were searched from January 2010 to January 2021. Full-text articles were included if they focused on AI in surgical education.

RESULTS

Out of 14,008 unique sources of evidence, 93 were included. Out of 93, 84 were conducted in the simulation setting, and 89 targeted technical skills. Fifty-six studies focused on skills assessment/classification, and 36 used multiple AI techniques. Also, increasing sample size, having balanced data, and using AI to provide feedback were major future directions mentioned by authors.

CONCLUSIONS

AI can help optimize the education of trainees and our results can help educators and researchers identify areas that need further investigation.

Objective assessment of robotic surgical skills: review of literature and future directions

BACKGROUND

Evaluation of robotic surgical skill has become increasingly important as robotic approaches to common surgeries become more widely utilized. However, evaluation of these currently lacks standardization. In this paper, we aimed to review the literature on robotic surgical skill evaluation.

METHODS

A review of literature on robotic surgical skill evaluation was performed and representative literature presented over the past ten years.

RESULTS

The study of reliability and validity in robotic surgical evaluation shows two main assessment categories: manual and automatic. Manual assessments have been shown to be valid but typically are time consuming and costly. Automatic evaluation and simulation are similarly valid and simpler to implement. Initial reports on evaluation of skill using artificial intelligence platforms show validity. Few data on evaluation methods of surgical skill connect directly to patient outcomes.

CONCLUSION

As evaluation in surgery begins to incorporate robotic skills, a simultaneous shift from manual to automatic evaluation may occur given the ease of implementation of these technologies. Robotic platforms offer the unique benefit of providing more objective data streams including kinematic data which allows for precise instrument tracking in the operative field. Such data streams will likely incrementally be implemented in performance evaluations. Similarly, with advances in artificial intelligence, machine evaluation of human technical skill will likely form the next wave of surgical evaluation.

Artificial Intelligence and Surgical Education: A Systematic Scoping Review of Interventions

OBJECTIVE

To synthesize peer-reviewed evidence related to the use of artificial intelligence (AI) in surgical education DESIGN: We conducted and reported a scoping review according to the standards outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis with extension for Scoping Reviews guideline and the fourth edition of the Joanna Briggs Institute Reviewer's Manual. We systematically searched eight interdisciplinary databases including MEDLINE-Ovid, ERIC, EMBASE, CINAHL, Web of Science: Core Collection, Compendex, Scopus, and IEEE Xplore. Databases were searched from inception until the date of search on April 13, 2021.

SETTING/PARTICIPANTS

We only examined original, peer-reviewed interventional studies that self-described as AI interventions, focused on medical education, and were relevant to surgical trainees (defined as medical or dental students, postgraduate residents, or surgical fellows) within the title and abstract (see Table 2). Animal, cadaveric, and in vivo studies were not eligible for inclusion.

RESULTS

After systematically searching eight databases and 4255 citations, our scoping review identified 49 studies relevant to artificial intelligence in surgical education. We found diverse interventions related to the evaluation of surgical competency, personalization of surgical education, and improvement of surgical education materials across surgical specialties. Many studies used existing surgical education materials, such as the Objective Structured Assessment of Technical Skills framework or the JHU-ISI Gesture and Skill Assessment Working Set database. Though most studies did not provide outcomes related to the implementation in medical schools (such as cost-effective analyses or trainee feedback), there are numerous promising interventions. In particular, many studies noted high accuracy in the objective characterization of surgical skill sets. These interventions could be further used to identify at-risk surgical trainees or evaluate teaching methods.

CONCLUSIONS

There are promising applications for AI in surgical education, particularly for the assessment of surgical competencies, though further evidence is needed regarding implementation and applicability.

Utilizing deep neural networks and electroencephalogram for objective evaluation of surgeon's distraction during robot-assisted surgery

OBJECTIVE

To develop an algorithm for objective evaluation of distraction of surgeons during robot-assisted surgery (RAS).

MATERIALS AND METHODS

Electroencephalogram (EEG) of 22 medical students was recorded while performing five key tasks on the robotic surgical simulator: Instrument Control, Ball Placement, Spatial Control II, Fourth Arm Tissue Retraction, and Hands-on Surgical Training Tasks. All students completed the Surgery Task Load Index (SURG-TLX), which includes one domain for subjective assessment of distraction (scale: 1-20). Scores were divided into low (score 1-6, subjective label: 1), intermediate (score 7-12, subjective label: 2), and high distraction (score 13-20, subjective label: 3). These cut-off values were arbitrarily considered based on a verbal assessment of participants and experienced surgeons. A Deep Convolutional Neural Network (CNN) algorithm was trained utilizing EEG recordings from the medical students and used to classify their distraction levels. The accuracy of our method was determined by comparing the subjective distraction scores on SURG-TLX and the results from the proposed classification algorithm. Also, Pearson correlation was utilized to assess the relationship between performance scores (generated by the simulator) and distraction (Subjective assessment scores).

RESULTS

The proposed end-to-end model classified distraction into low, intermediate, and high with 94%, 89%, and 95% accuracy, respectively. We found a significant negative correlation (r = -0.21; p = 0.003) between performance and SURG-TLX distraction scores.

CONCLUSIONS

Herein we report, to our knowledge, the first objective method to assess and quantify distraction while performing robotic surgical tasks on the robotic simulator, which may improve patient safety. Validation in the clinical setting is required.

Differentiating Operator Skill during Routine Fetal Ultrasound Scanning using Probe Motion Tracking

In this paper, we consider differentiating operator skill during fetal ultrasound scanning using probe motion tracking. We present a novel convolutional neural network-based deep learning framework to model ultrasound probe motion in order to classify operator skill levels, that is invariant to operators' personal scanning styles. In this study, probe motion data during routine second-trimester fetal ultrasound scanning was acquired by operators of known experience levels (2 newly-qualified operators and 10 expert operators). The results demonstrate that the proposed model can successfully learn underlying probe motion features that distinguish operator skill levels during routine fetal ultrasound with 95% accuracy.

Current use of telehealth in urology: a review

PURPOSE

Applications of telehealth have been growing in popularity. However, there is little information on how telehealth is being used in Urology. In this review, we examine current applications of telehealth in urological practices as well as barriers to implementation.

METHODS

A review was conducted of original research within the past 10 years describing telehealth applications in urology. Articles on telehealth as applied to other specialties were reviewed for discussion on real or perceived barriers to implementation.

RESULTS

Twenty-four articles met the inclusion criteria. The most common application of telehealth was using a video visit to assess or follow-up with patients. The second most commonly described applications of telehealth were telementorship, or the use of telehealth technology to help train providers, and telemedicine used in diagnostics. Studies consistently stated the effectiveness of the telehealth applications and the high level of patient and provider satisfaction.

CONCLUSIONS

Telehealth is sparingly used in urology. Barriers to implementation include technological literacy, reimbursement uncertainties, and resistance to change in workflow. When used, telehealth technologies are shown to be safe, effective, and satisfactory for patients and providers. Further investigation is necessary to determine the efficacy of telehealth applications.

Using Machine Learning to Assess Physician Competence: A Systematic Review

PURPOSE

To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different competence domains in several medical specialties.

METHOD

In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students', residents', fellows', or attending physicians' competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted.

RESULTS

Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24; 34.8%) and radiology (15; 21.7%) were the most studied specialties; natural language processing (24; 34.8%), support vector machine (15; 21.7%), and hidden Markov models (14; 20.3%) were the ML techniques most often applied; and patient care (63; 91.3%) and medical knowledge (45; 65.2%) were the most assessed competence domains.

CONCLUSIONS

A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.

Deep learning with convolutional neural network for objective skill evaluation in robot-assisted surgery

PURPOSE

With the advent of robot-assisted surgery, the role of data-driven approaches to integrate statistics and machine learning is growing rapidly with prominent interests in objective surgical skill assessment. However, most existing work requires translating robot motion kinematics into intermediate features or gesture segments that are expensive to extract, lack efficiency, and require significant domain-specific knowledge.

METHODS

We propose an analytical deep learning framework for skill assessment in surgical training. A deep convolutional neural network is implemented to map multivariate time series data of the motion kinematics to individual skill levels.

RESULTS

We perform experiments on the public minimally invasive surgical robotic dataset, JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our proposed learning model achieved competitive accuracies of 92.5%, 95.4%, and 91.3%, in the standard training tasks: Suturing, Needle-passing, and Knot-tying, respectively. Without the need of engineered features or carefully tuned gesture segmentation, our model can successfully decode skill information from raw motion profiles via end-to-end learning. Meanwhile, the proposed model is able to reliably interpret skills within a 1-3 second window, without needing an observation of entire training trial.

CONCLUSION

This study highlights the potential of deep architectures for efficient online skill assessment in modern surgical training.

Implementing assessments of robot-assisted technical skill in urological education: a systematic review and synthesis of the validity evidence

OBJECTIVES

To systematically review and synthesise the validity evidence supporting intraoperative and simulation-based assessments of technical skill in urological robot-assisted surgery (RAS), and make evidence-based recommendations for the implementation of these assessments in urological training.

MATERIALS AND METHODS

A literature search of the Medline, PsycINFO and Embase databases was performed. Articles using technical skill and simulation-based assessments in RAS were abstracted. Only studies involving urology trainees or faculty were included in the final analysis.

RESULTS

Multiple tools for the assessment of technical robotic skill have been published, with mixed sources of validity evidence to support their use. These evaluations have been used in both the ex vivo and in vivo settings. Performance evaluations range from global rating scales to psychometrics, and assessments are carried out through automation, expert analysts, and crowdsourcing.

CONCLUSION

There have been rapid expansions in approaches to RAS technical skills assessment, both in simulated and clinical settings. Alternative approaches to assessment in RAS, such as crowdsourcing and psychometrics, remain under investigation. Evidence to support the use of these metrics in high-stakes decisions is likely insufficient at present.

Energy-Based Metrics for Arthroscopic Skills Assessment

Minimally invasive skills assessment methods are essential in developing efficient surgical simulators and implementing consistent skills evaluation. Although numerous methods have been investigated in the literature, there is still a need to further improve the accuracy of surgical skills assessment. Energy expenditure can be an indication of motor skills proficiency. The goals of this study are to develop objective metrics based on energy expenditure, normalize these metrics, and investigate classifying trainees using these metrics. To this end, different forms of energy consisting of mechanical energy and work were considered and their values were divided by the related value of an ideal performance to develop normalized metrics. These metrics were used as inputs for various machine learning algorithms including support vector machines (SVM) and neural networks (NNs) for classification. The accuracy of the combination of the normalized energy-based metrics with these classifiers was evaluated through a leave-one-subject-out cross-validation. The proposed method was validated using 26 subjects at two experience levels (novices and experts) in three arthroscopic tasks. The results showed that there are statistically significant differences between novices and experts for almost all of the normalized energy-based metrics. The accuracy of classification using SVM and NN methods was between 70% and 95% for the various tasks. The results show that the normalized energy-based metrics and their combination with SVM and NN classifiers are capable of providing accurate classification of trainees. The assessment method proposed in this study can enhance surgical training by providing appropriate feedback to trainees about their level of expertise and can be used in the evaluation of proficiency.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Task-Level vs. Segment-Level Quantitative Metrics for Surgical Skill Assessment

OBJECTIVE

Task-level metrics of time and motion efficiency are valid measures of surgical technical skill. Metrics may be computed for segments (maneuvers and gestures) within a task after hierarchical task decomposition. Our objective was to compare task-level and segment (maneuver and gesture)-level metrics for surgical technical skill assessment.

DESIGN

Our analyses include predictive modeling using data from a prospective cohort study. We used a hierarchical semantic vocabulary to segment a simple surgical task of passing a needle across an incision and tying a surgeon's knot into maneuvers and gestures. We computed time, path length, and movements for the task, maneuvers, and gestures using tool motion data. We fit logistic regression models to predict experience-based skill using the quantitative metrics. We compared the area under a receiver operating characteristic curve (AUC) for task-level, maneuver-level, and gesture-level models.

SETTING

Robotic surgical skills training laboratory.

PARTICIPANTS

In total, 4 faculty surgeons with experience in robotic surgery and 14 trainee surgeons with no or minimal experience in robotic surgery.

RESULTS

Experts performed the task in shorter time (49.74s; 95% CI = 43.27-56.21 vs. 81.97; 95% CI = 69.71-94.22), with shorter path length (1.63m; 95% CI = 1.49-1.76 vs. 2.23; 95% CI = 1.91-2.56), and with fewer movements (429.25; 95% CI = 383.80-474.70 vs. 728.69; 95% CI = 631.84-825.54) than novices. Experts differed from novices on metrics for individual maneuvers and gestures. The AUCs were 0.79; 95% CI = 0.62-0.97 for task-level models, 0.78; 95% CI = 0.6-0.96 for maneuver-level models, and 0.7; 95% CI = 0.44-0.97 for gesture-level models. There was no statistically significant difference in AUC between task-level and maneuver-level (p = 0.7) or gesture-level models (p = 0.17).

CONCLUSIONS

Maneuver-level and gesture-level metrics are discriminative of surgical skill and can be used to provide targeted feedback to surgical trainees.

Analysis of the Structure of Surgical Activity for a Suturing and Knot-Tying Task

Background

Surgical tasks are performed in a sequence of steps, and technical skill evaluation includes assessing task flow efficiency. Our objective was to describe differences in task flow for expert and novice surgeons for a basic surgical task.

Methods

We used a hierarchical semantic vocabulary to decompose and annotate maneuvers and gestures for 135 instances of a surgeon’s knot performed by 18 surgeons. We compared counts of maneuvers and gestures, and analyzed task flow by skill level.

Results

Experts used fewer gestures to perform the task (26.29; 95% CI = 25.21 to 27.38 for experts vs. 31.30; 95% CI = 29.05 to 33.55 for novices) and made fewer errors in gestures than novices (1.00; 95% CI = 0.61 to 1.39 vs. 2.84; 95% CI = 2.3 to 3.37). Transitions among maneuvers, and among gestures within each maneuver for expert trials were more predictable than novice trials.

Conclusions

Activity segments and state flow transitions within a basic surgical task differ by surgical skill level, and can be used to provide targeted feedback to surgical trainees.

Conventional Laparoscopic vs Robotic Training: Which is Better for Naive Users? A Randomized Prospective Crossover Study

OBJECTIVE

Robotic training (RT) using the da Vinci skills simulator and conventional training (CT) using a laparoscopic "training box" are both used to augment operative skills in minimally invasive surgery. The current study tests the hypothesis that skill acquisition is more rapid using RT than using CT among naive learners.

DESIGN AND PARTICIPANTS

A total of 40 subjects without laparoscopic or robotic surgical experience were enrolled and randomized to begin with either RT or CT. Then, 2 specific RT tasks were reproduced for CT and repeated 5 times each with RT and CT. Time and quality indicators were measured quantitatively. A crossover technique was used to control for in-study experience bias.

RESULTS

The tasks "pick and place jacks" (PP) and "thread the rings" (TR) were achieved faster with RT than with CT despite crossover (p < 0.0001). An RT-favoring difference was observed in speed for both tasks when changing modality. Percentage improvement with increasing trials was similar for RT and CT: RT completion time averaged 39 seconds and 211 seconds (PP and TR, respectively), compared with 65 seconds and 362 seconds when using CT (p < 0.0001); final improvement averaged 26% and 46% for RT (PP and TR, respectively) vs 31% and 47% for CT (p was 0.76 for PP and 0.20 for TR). Within the PP task, RT times averaged 41 seconds without previous CT experience vs 35 seconds with previous CT experience (p = 0.20); CT times averaged 61 seconds without and 69 seconds with previous RT experience (p = 0.48). Comparable times for the TR task were 212 seconds vs 216 seconds (p = 0.66) and 388 seconds vs 334 seconds (p = 0.17). Both instrument collisions and excessive force occurred more commonly for RT than for CT within the TR task (p < 0.0001).

CONCLUSIONS

Speeds were faster overall with RT than with CT, but the percentage of speed improvement with trials was similar, suggesting similar learning curves, with minimal transfer effect appreciated.

A study of crowdsourced segment-level surgical skill assessment using pairwise rankings

PURPOSE

Currently available methods for surgical skills assessment are either subjective or only provide global evaluations for the overall task. Such global evaluations do not inform trainees about where in the task they need to perform better. In this study, we investigated the reliability and validity of a framework to generate objective skill assessments for segments within a task, and compared assessments from our framework using crowdsourced segment ratings from surgically untrained individuals and expert surgeons against manually assigned global rating scores.

METHODS

Our framework includes (1) a binary classifier trained to generate preferences for pairs of task segments (i.e., given a pair of segments, specification of which one was performed better), (2) computing segment-level percentile scores based on the preferences, and (3) predicting task-level scores using the segment-level scores. We conducted a crowdsourcing user study to obtain manual preferences for segments within a suturing and knot-tying task from a crowd of surgically untrained individuals and a group of experts. We analyzed the inter-rater reliability of preferences obtained from the crowd and experts, and investigated the validity of task-level scores obtained using our framework. In addition, we compared accuracy of the crowd and expert preference classifiers, as well as the segment- and task-level scores obtained from the classifiers.

RESULTS

We observed moderate inter-rater reliability within the crowd (Fleiss' kappa, κ = 0.41) and experts (κ = 0.55). For both the crowd and experts, the accuracy of an automated classifier trained using all the task segments was above par as compared to the inter-rater agreement [crowd classifier 85 % (SE 2 %), expert classifier 89 % (SE 3 %)]. We predicted the overall global rating scores (GRS) for the task with a root-mean-squared error that was lower than one standard deviation of the ground-truth GRS. We observed a high correlation between segment-level scores (ρ ≥ 0.86) obtained using the crowd and expert preference classifiers. The task-level scores obtained using the crowd and expert preference classifier were also highly correlated with each other (ρ ≥ 0.84), and statistically equivalent within a margin of two points (for a score ranging from 6 to 30). Our analyses, however, did not demonstrate statistical significance in equivalence of accuracy between the crowd and expert classifiers within a 10 % margin.

CONCLUSIONS

Our framework implemented using crowdsourced pairwise comparisons leads to valid objective surgical skill assessment for segments within a task, and for the task overall. Crowdsourcing yields reliable pairwise comparisons of skill for segments within a task with high efficiency. Our framework may be deployed within surgical training programs for objective, automated, and standardized evaluation of technical skills.

A review of training research and virtual reality simulators for the da Vinci surgical system

PHENOMENON: Virtual reality simulators are the subject of several recent studies of skills training for robot-assisted surgery. Yet no consensus exists regarding what a core skill set comprises or how to measure skill performance. Defining a core skill set and relevant metrics would help surgical educators evaluate different simulators.

APPROACH

This review draws from published research to propose a core technical skill set for using the da Vinci surgeon console. Publications on three commercial simulators were used to evaluate the simulators' content addressing these skills and associated metrics.

FINDINGS

An analysis of published research suggests that a core technical skill set for operating the surgeon console includes bimanual wristed manipulation, camera control, master clutching to manage hand position, use of third instrument arm, activating energy sources, appropriate depth perception, and awareness of forces applied by instruments. Validity studies of three commercial virtual reality simulators for robot-assisted surgery suggest that all three have comparable content and metrics. However, none have comprehensive content and metrics for all core skills. INSIGHTS: Virtual reality simulation remains a promising tool to support skill training for robot-assisted surgery, yet existing commercial simulator content is inadequate for performing and assessing a comprehensive basic skill set. The results of this evaluation help identify opportunities and challenges that exist for future developments in virtual reality simulation for robot-assisted surgery. Specifically, the inclusion of educational experts in the development cycle alongside clinical and technological experts is recommended.

Training to maintain surgical skills during periods of robotic surgery inactivity

BACKGROUND

The study was performed to establish a level of practice needed for newly-trained residents to maintain robotic surgical skills during periods of robotic inactivity.

METHODS

Ten surgical residents were trained to a standardized level of robotic surgery proficiency with inanimate models. At the end of two, four and six weeks, the residents practiced with the models for a total of one hour. Each resident performed a timed tissue closure task immediately after reaching the proficiency standards and twice in succession at eight weeks. Time to completion was compared between the three trials with a repeated measures ANOVA and a post-hoc test.

RESULTS

Average time to complete the tissue closure task decreased by more than 25% over the period between reaching the proficiency standards and the trials at eight weeks, with the difference significant (P < 0.004).

CONCLUSIONS

Biweekly practice for one hour was sufficient to maintain robotic surgical skills.

Objective assessment in residency-based training for transoral robotic surgery

OBJECTIVES/HYPOTHESIS

To develop a robotic surgery training regimen integrating objective skill assessment for otolaryngology and head and neck surgery trainees consisting of training modules of increasing complexity leading up to procedure-specific training. In particular, we investigated applications of such a training approach for surgical extirpation of oropharyngeal tumors via a transoral approach using the da Vinci robotic system.

STUDY DESIGN

Prospective blinded data collection and objective evaluation (Objective Structured Assessment of Technical Skills [OSATS]) of three distinct phases using the da Vinci robotic surgical system in an academic university medical engineering/computer science laboratory setting.

METHODS

Between September 2010 and July 2011, eight otolaryngology-head and neck surgery residents and four staff experts from an academic hospital participated in three distinct phases of robotic surgery training involving 1) robotic platform operational skills, 2) set up of the patient side system, and 3) a complete ex vivo surgical extirpation of an oropharyngeal tumor located in the base of tongue. Trainees performed multiple (four) approximately equally spaced training sessions in each stage of the training. In addition to trainees, baseline performance data were obtained for the experts. Each surgical stage was documented with motion and event data captured from the application programming interfaces of the da Vinci system, as well as separate video cameras as appropriate. All data were assessed using automated skill measures of task efficiency and correlated with structured assessment (OSATS and similar Likert scale) from three experts to assess expert and trainee differences and compute automated and expert assessed learning curves.

RESULTS

Our data show that such training results in an improved didactic robotic knowledge base and improved clinical efficiency with respect to the set up and console manipulation. Experts (e.g., average OSATS, 25; standard deviation [SD], 3.1; module 1, suturing) and trainees (average OSATS, 15.9; SD, 3.9; week 1) are well separated at the beginning of the training, and the separation reduces significantly (expert average OSATS, 27.6; SD, 2.7; trainee average OSATS, 24.2; SD, 6.8; module 3) at the conclusion of the training. Learning curves in each of the three stages show diminishing differences between the experts and trainees, which is also consistent with expert assessment. Subjective assessment by experts verified the clinical utility of the module 3 surgical environment, and a survey of trainees consistently rated the curriculum as very useful in progression to human operating room assistance.

CONCLUSIONS

Structured curricular robotic surgery training with objective assessment promises to reduce the overhead for mentors, allow detailed assessment of human-machine interface skills, and create customized training models for individualized training. This preliminary study verifies the utility of such training in improving human-machine operations skills (module 1), and operating room and surgical skills (modules 2 and 3). In contrast to current coarse measures of total operating time and subjective assessment of error for short mass training sessions, these methods may allow individual tasks to be removed from the trainee regimen when skill levels are within the standard deviation of the experts for these tasks, which can greatly enhance overall efficiency of the training regimen and allow time for additional and more complex training to be incorporated in the same time frame.