Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills

Multi-perspective analysis of daVinci surgical virtual reality training: a prospective randomized controlled study

This study explored the impact of virtual reality (VR) training on improving the acquisition of basic robotic surgical skills by analyzing the effects of training in the daVinci Surgical System (dVSS) simulator from multiple perspectives. 27 subjects were randomly divided into a VR-training group and a control group, with the VR-training group (n = 12) receiving training on the dVSS simulator (XI) to achieve expert-specific proficiency status, and the control group (n = 15) receiving no training. All the subjects subsequently wore electroencephalography (EEG) equipment to perform 6 tasks and repeated 3 times on the dVSS simulator (SI). The Global Evaluative Assessment of Robotic Skills (GEARS) scores, dVSS scores, the National Aeronautics and Space Administration Task Load Index (NASA TLX) scores, and EEG data of all the subjects were collected to conduct a comprehensive and multi-perspective analysis for dVSS training. Learning curve analysis revealed that all trainees improved their basic robotic surgical skills and reached a steady state after training. Compared with the control group, the VR-training group received higher the GEARS scores (24.91 ± 3.36 vs. 19.68 ± 3.07; p < 0.01) and dVSS scores and lower the NASA TLX scores (40.04 ± 10.55 vs. 48.2 ± 9.88; p < 0.01). In the EEG analysis, the VR-training group had higher scores in the Beta band and the Low-gamma band in the brain regions and had greater energy activation than did the control group. This randomized controlled trial combined subjective and objective evaluations to comprehensively analyze subjects' technical and nontechnical skills. It demonstrated that training on the dVSS simulator significantly improved trainees' basic robotic surgical skills and that they could achieve better basic robotic surgical skills at lower workloads. Trial registration: The study was retrospectively registered at the Chinese Clinical Trial Center. The trial registration number (TRN) was ChiCTR2400088465, and the registration day was August 20, 2024.

Objective skill assessment for cataract surgery from surgical microscope video

OBJECTIVE

Video offers an accessible method for automated surgical skill evaluation; however, many platforms still rely on traditional six-degree-of-freedom (6-DOF) tracking systems, which can be costly, cumbersome, and challenging to apply clinically. This study aims to demonstrate that trainee skill in cataract surgery can be assessed effectively using only object detection from monocular surgical microscope video.

METHODS

One ophthalmologist and four residents performed cataract surgery on a simulated eye five times each, generating 25 recordings. Recordings included both the surgical microscope video and 6-DOF instrument tracking data. Videos were graded by two expert ophthalmologists using the ICO-OSCAR:SICS rubric. We computed motion-based metrics using both object detection from video and 6-DOF tracking. We first examined correlations between each metric and expert scores for each rubric criteria. Then, using these findings, we trained an ordinal regression model to predict scores from each tracking modality and compared correlation strengths with expert scores.

RESULTS

Metrics from object detection generally showed stronger correlations with expert scores than 6-DOF tracking. For score prediction, 6-DOF tracking showed no significant advantage, while scores predicted from object detection achieved significantly stronger correlations with expert scores for four scoring criteria.

CONCLUSION

Our results indicate that skill assessment from monocular surgical microscope video can match, and in some cases exceed, the correlation strengths of 6-DOF tracking assessments. This finding supports the feasibility of using object detection for skill assessment without additional hardware.

Crowdsourced Assessment of Aesthetic Outcomes of Dorsal Preservation Rhinoplasty

BACKGROUND

The inherent subjectivity in aesthetic outcomes presents a unique challenge in assessing rhinoplasty. Crowdsourcing has provided a new metric for objective analysis. The authors designed a retrospective study to compare the aesthetic outcomes of dorsal preservation rhinoplasty vs structural rhinoplasty.

OBJECTIVES

We aimed to objectively quantify the relative aesthetic advantages of performing a dorsal preservation technique. Additionally, we aimed to demonstrate the efficacy of crowdsourcing as an efficient and reliable method for evaluating any plastic surgery aesthetic outcome.

METHODS

Patients' preoperative and postoperative photographs were divided into 2 cohorts. Photographs were evaluated by crowdworkers on a secure online rating platform based on the overall nasal appearance, dorsal profile, dorsal aesthetic line symmetry, and dorsal contour, and rated their confidence about whether a patient had undergone surgery. A delta was calculated by comparing preoperative to postoperative states to represent an absolute value of improvement after surgery. Each cohort was compared with non-paired t tests.

RESULTS

The structural rhinoplasty cohort included 34 patients. The dorsal preservation cohort included 30 patients. Both cohorts demonstrated improved aesthetic outcomes (dorsal preservation 0.300, 95% CI 0.047; structural 0.377, 95% CI 0.055). When raters were asked to predict whether a patient had surgery, the correlation coefficient of the structural cohort (0.74) suggested that a crowdworker was better able to identify whether a patient had surgery in those patients. The correlation coefficient in the dorsal preservation cohort (-0.0554) suggested that the raters were unable to identify which patients had surgery.

CONCLUSIONS

We found significant improvements in overall aesthetic outcomes with both techniques, although a more natural "unoperated" outcome was achieved when performing a dorsal preservation technique. We also provide evidence of the efficacy of crowdsourcing as an efficient and reliable method for evaluating aesthetic outcomes.

LEVEL OF EVIDENCE: 4 (THERAPEUTIC)

Objective performance indicators versus GEARS: an opportunity for more accurate assessment of surgical skill

PURPOSE

Surgical skill evaluation that relies on subjective scoring of surgical videos can be time-consuming and inconsistent across raters. We demonstrate differentiated opportunities for objective evaluation to improve surgeon training and performance.

METHODS

Subjective evaluation was performed using the Global evaluative assessment of robotic skills (GEARS) from both expert and crowd raters; whereas, objective evaluation used objective performance indicators (OPIs) derived from da Vinci surgical systems. Classifiers were trained for each evaluation method to distinguish between surgical expertise levels. This study includes one clinical task from a case series of robotic-assisted sleeve gastrectomy procedures performed by a single surgeon, and two training tasks performed by novice and expert surgeons, i.e., surgeons with no experience in robotic-assisted surgery (RAS) and those with more than 500 RAS procedures.

RESULTS

When comparing expert and novice skill levels, OPI-based classifier showed significantly higher accuracy than GEARS-based classifier on the more complex dissection task (OPI 0.93 ± 0.08 vs. GEARS 0.67 ± 0.18; 95% CI, 0.16-0.37; p = 0.02), but no significant difference was shown on the simpler suturing task. For the single-surgeon case series, both classifiers performed well when differentiating between early and late group cases with smaller group sizes and larger intervals between groups (OPI 0.9 ± 0.08; GEARS 0.87 ± 0.12; 95% CI, 0.02-0.04; p = 0.67). When increasing the group size to include more cases, thereby having smaller intervals between groups, OPIs demonstrated significantly higher accuracy (OPI 0.97 ± 0.06; GEARS 0.76 ± 0.07; 95% CI, 0.12-0.28; p = 0.004) in differentiating between the early/late cases.

CONCLUSIONS

Objective methods for skill evaluation in RAS outperform subjective methods when (1) differentiating expertise in a technically challenging training task, and (2) identifying more granular differences along early versus late phases of a surgeon learning curve within a clinical task. Objective methods offer an opportunity for more accessible and scalable skill evaluation in RAS.

Evaluation of objective tools and artificial intelligence in robotic surgery technical skills assessment: a systematic review

BACKGROUND

There is a need to standardize training in robotic surgery, including objective assessment for accreditation. This systematic review aimed to identify objective tools for technical skills assessment, providing evaluation statuses to guide research and inform implementation into training curricula.

METHODS

A systematic literature search was conducted in accordance with the PRISMA guidelines. Ovid Embase/Medline, PubMed and Web of Science were searched. Inclusion criterion: robotic surgery technical skills tools. Exclusion criteria: non-technical, laparoscopy or open skills only. Manual tools and automated performance metrics (APMs) were analysed using Messick's concept of validity and the Oxford Centre of Evidence-Based Medicine (OCEBM) Levels of Evidence and Recommendation (LoR). A bespoke tool analysed artificial intelligence (AI) studies. The Modified Downs-Black checklist was used to assess risk of bias.

RESULTS

Two hundred and forty-seven studies were analysed, identifying: 8 global rating scales, 26 procedure-/task-specific tools, 3 main error-based methods, 10 simulators, 28 studies analysing APMs and 53 AI studies. Global Evaluative Assessment of Robotic Skills and the da Vinci Skills Simulator were the most evaluated tools at LoR 1 (OCEBM). Three procedure-specific tools, 3 error-based methods and 1 non-simulator APMs reached LoR 2. AI models estimated outcomes (skill or clinical), demonstrating superior accuracy rates in the laboratory with 60 per cent of methods reporting accuracies over 90 per cent, compared to real surgery ranging from 67 to 100 per cent.

CONCLUSIONS

Manual and automated assessment tools for robotic surgery are not well validated and require further evaluation before use in accreditation processes.PROSPERO: registration ID CRD42022304901.

Laypersons versus experienced surgeons in assessing simulated robot-assisted radical prostatectomy

BACKGROUND

Feedback is important for surgical trainees but it can be biased and time-consuming. We examined crowd-sourced assessment as an alternative to experienced surgeons' assessment of robot-assisted radical prostatectomy (RARP).

METHODS

We used video recordings (n = 45) of three RARP modules on the RobotiX, Simbionix simulator from a previous study in a blinded comparative assessment study. A group of crowd workers (CWs) and two experienced RARP surgeons (ESs) evaluated all videos with the modified Global Evaluative Assessment of Robotic Surgery (mGEARS).

RESULTS

One hundred forty-nine CWs performed 1490 video ratings. Internal consistency reliability was high (0.94). Inter-rater reliability and test-retest reliability were low for CWs (0.29 and 0.39) and moderate for ESs (0.61 and 0.68). In an Analysis of Variance (ANOVA) test, CWs could not discriminate between the skill level of the surgeons (p = 0.03-0.89), whereas ES could (p = 0.034).

CONCLUSION

We found very low agreement between the assessments of CWs and ESs when they assessed robot-assisted radical prostatectomies. As opposed to ESs, CWs could not discriminate between surgical experience using the mGEARS ratings or when asked if they wanted the surgeons to perform their robotic surgery.

Current practises and the future of robotic surgical training

INTRODUCTION

This study reviews the current state of robotic surgery training for surgeons, including the various curricula, training methods, and tools available, as well as the challenges and limitations of these.

METHODS

The authors carried out a literature search across PubMed, MEDLINE, and Google Scholar using keywords related to 'robotic surgery', 'computer-assisted surgery', 'simulation', 'virtual reality', 'surgical training', and 'surgical education'. Full text analysis was performed on 112 articles.

TRAINING PROGRAMMES

The training program for robotic surgery should focus on proficiency, deliberation, and distribution principles. The curricula can be broadly split up into pre-console and console-side training. Pre-Console and Console-Side Training: Simulation training is an important aspect of robotic surgery training to improve technical skill acquisition and reduce mental workload, which helps prepare trainees for live procedures.

OPERATIVE PERFORMANCE ASSESSMENT

The study also discusses the various validated assessment tools used for operative performance assessments.

FUTURE ADVANCES

Finally, the authors propose potential future directions for robotic surgery training, including the use of emerging technologies such as AI and machine learning for real-time feedback, remote mentoring, and augmented reality platforms like Proximie to reduce costs and overcome geographic limitations.

CONCLUSION

Standardisation in trainee performance assessment is needed. Each of the robotic curricula and platforms has strengths and weaknesses. The ERUS Robotic Curriculum represents an evidence-based example of how to implement training from novice to expert. Remote mentoring and augmented reality platforms can overcome the challenges of high equipment costs and limited access to experts. Emerging technologies offer promising advancements for real-time feedback and immersive training environments, improving patient outcomes.

Effects of automated skill assessment on robotic surgery training

BACKGROUND

Several automated skill-assessment approaches have been proposed for robotic surgery, but their utility is not well understood. This article investigates the effects of one machine-learning-based skill-assessment approach on psychomotor skill development in robotic surgery training.

METHODS

N = 29 trainees (medical students and residents) with no robotic surgery experience performed five trials of inanimate peg transfer with an Intuitive Surgical da Vinci Standard robot. Half of the participants received no post-trial feedback. The other half received automatically calculated scores from five Global Evaluative Assessment of Robotic Skill domains post-trial.

RESULTS

There were no significant differences between the groups regarding overall improvement or skill improvement rate. However, participants who received post-trial feedback rated their overall performance improvement significantly lower than participants who did not receive feedback.

CONCLUSIONS

These findings indicate that automated skill evaluation systems might improve trainee self-awareness but not accelerate early stage psychomotor skill development in robotic surgery training.

Performance and Capability Assessment in Surgical Subtask Automation

Robot-Assisted Minimally Invasive Surgery (RAMIS) has reshaped the standard clinical practice during the past two decades. Many believe that the next big step in the advancement of RAMIS will be partial autonomy, which may reduce the fatigue and the cognitive load on the surgeon by performing the monotonous, time-consuming subtasks of the surgical procedure autonomously. Although serious research efforts are paid to this area worldwide, standard evaluation methods, metrics, or benchmarking techniques are still not formed. This article aims to fill the void in the research domain of surgical subtask automation by proposing standard methodologies for performance evaluation. For that purpose, a novel characterization model is presented for surgical automation. The current metrics for performance evaluation and comparison are overviewed and analyzed, and a workflow model is presented that can help researchers to identify and apply their choice of metrics. Existing systems and setups that serve or could serve as benchmarks are also introduced and the need for standard benchmarks in the field is articulated. Finally, the matter of Human-Machine Interface (HMI) quality, robustness, and the related legal and ethical issues are presented.

Adaptive Surgical Robotic Training Using Real-Time Stylistic Behavior Feedback Through Haptic Cues

Surgical skill directly affects surgical procedure outcomes; thus, effective training is needed to ensure satisfactory results. Many objective assessment metrics have been developed that provide the trainee with descriptive feedback about their performance however, often lack feedback on how to improve performance. The most effective training method is one that is intuitive, easy to understand, personalized to the user,and provided in a timely manner. We propose a framework to enable user-adaptive training using near real-time detection of performance, based on intuitive styles of surgical movements, and design a haptic feedback framework to assist with correcting styles of movement. We evaluate the ability of three types of force feedback (spring, damping, and spring plus damping feedback), computed based on prior user positions, to improve different stylistic behaviors of the user during kinematically constrained reaching movement tasks. The results indicate that five out of six styles studied here were improved using at least one of the three types of force feedback. Task performance metrics were compared in the presence of the three types of feedback. Task time was statistically significantly lower when applying spring feedback, compared to the other two types of feedback. Path straightness and targeting error were statistically significantly improved when using spring-damping feedback compared to the other two types of feedback. This study presents a groundwork for adaptive training in robotic surgery based on near real-time human-centric models of surgical behavior.

Implementation of a standardized robotic assistant surgical training curriculum

Since 2000, robotic-assisted surgery has rapidly expanded into almost every surgical sub-specialty. Despite the popularity of robotic surgery across the United States, a national consensus for standardized training and education of robotic surgeons or surgical teams remains absent. In this quality improvement initiative, a novel, stepwise iterative Robotic Assistant Surgical Training (RAST) curriculum was developed to broaden and standardize robotic bedside assistant training. Thirteen voluntary participants, capable of fulfilling the bedside assistant role, were evaluated to determine if RAST enhanced the learner's self-perceived level of confidence and comfort in their role as bedside assistant. A pre- and post-RAST training survey and a between-stages repeated-measures survey were conducted. All learner participants reported statistically significant increases in confidence and comfort after RAST training, (p =  < 0.001), and between each stage, F (2, 24 = 60.47, p < .001; [Formula: see text] = 0.834). Participant feedback regarding curriculum improvement was obtained, suggesting the desire for more training and practice, in smaller groups of 2-3 participants. One hundred percent of participants felt RAST was beneficial and that it should be implemented as standardized training during onboarding for all robotic bedside assistants. Thus, a standardized, stepwise iterative robotic bedside assistant curriculum increases learner preparedness, comfort, and confidence, safely away from the patient bedside.

Recent advances in benign gynecological laparoscopic surgery

Minimally invasive surgery continues to transform the field of gynecological surgery and is now the standard of care for the surgical treatment of many diseases in gynecology. Owing to minimally invasive surgery's clear advantages, new advances in technology are being employed rapidly and enabling even the most complicated procedures to be performed less invasively. We examine recent literature on minimally invasive surgical innovations, advances, and common practices in benign gynecology that, from our point of view, made an impact on the way laparoscopic surgery is performed and managed in the last decade.

Educational value of surgical telementoring

Educating surgeons is a time‐consuming process. In addition to theoretical knowledge, the practical tasks of surgical procedures must be mastered. Translation of such knowledge from mentor to mentee may be efficiently done by surgical telementoring (ST). This is a review on surgical telementoring. Recent technological advances have made this tool in surgical education more available and applicable but future applications of ST have to be wisely guided by high‐quality trials.

Crowdsourced Assessment of Surgical Skill Proficiency in Cataract Surgery

OBJECTIVE

To test whether crowdsourced lay raters can accurately assess cataract surgical skills.

DESIGN

Two-armed study: independent cross-sectional and longitudinal cohorts.

SETTING

Washington University Department of Ophthalmology.

PARTICIPANTS AND METHODS

Sixteen cataract surgeons with varying experience levels submitted cataract surgery videos to be graded by 5 experts and 300+ crowdworkers masked to surgeon experience. Cross-sectional study: 50 videos from surgeons ranging from first-year resident to attending physician, pooled by years of training. Longitudinal study: 28 videos obtained at regular intervals as residents progressed through 180 cases. Surgical skill was graded using the modified Objective Structured Assessment of Technical Skill (mOSATS). Main outcome measures were overall technical performance, reliability indices, and correlation between expert and crowd mean scores.

RESULTS

Experts demonstrated high interrater reliability and accurately predicted training level, establishing construct validity for the modified OSATS. Crowd scores were correlated with (r = 0.865, p < 0.0001) but consistently higher than expert scores for first, second, and third-year residents (p < 0.0001, paired t-test). Longer surgery duration negatively correlated with training level (r = -0.855, p < 0.0001) and expert score (r = -0.927, p < 0.0001). The longitudinal dataset reproduced cross-sectional study findings for crowd and expert comparisons. A regression equation transforming crowd score plus video length into expert score was derived from the cross-sectional dataset (r² = 0.92) and demonstrated excellent predictive modeling when applied to the independent longitudinal dataset (r² = 0.80). A group of student raters who had edited the cataract videos also graded them, producing scores that more closely approximated experts than the crowd.

CONCLUSIONS

Crowdsourced rankings correlated with expert scores, but were not equivalent; crowd scores overestimated technical competency, especially for novice surgeons. A novel approach of adjusting crowd scores with surgery duration generated a more accurate predictive model for surgical skill. More studies are needed before crowdsourcing can be reliably used for assessing surgical proficiency.

The Use of Crowdsourcing Technology to Evaluate Preoperative Severity in Patients With Unilateral Cleft Lip in a Multiethnic Population

ABSTRACT

Crowd sourcing has been used in multiple disciplines to quickly generate large amounts of diverse data. The objective of this study was to use crowdsourcing to grade preoperative severity of unilateral cleft lip phenotype in a multiethnic cohort with the hypothesis that crowdsourcing could efficiently achieve similar rankings compared to expert surgeons. Deidentified preoperative photos were collected for patients with primary, unilateral cleft lip with or without cleft palate (CL ± P). A platform was developed with C-SATS for pairwise comparisons utilizing Elo rankings by crowdsource workers through Amazon Mechanical Turk. Images were independently ranked by 2 senior surgeons for comparison. Seventy-six patients with varying severity of unilateral (CL ± P) phenotype were chosen from Operation Smile missions in Bolivia, Madagascar, Vietnam, and Morocco. Patients were an average of 1.2 years' old, ranging from 3 months to 3.3 years. Each image was compared with 10 others, for a total of 380 unique pairwise comparisons. A total of 4627 total raters participated with a median of 12 raters per pair. Data collection was completed in <20 hours. The crowdsourcing ranking and expert surgeon rankings were highly correlated with Pearson correlation coefficient of R = 0.77 (P = 0.0001). Crowdsourcing provides a rapid and convenient method of obtaining preoperative severity ratings, comparable to expert surgeon assessment, across multiple ethnicities. The method serves as a potential solution to the current lack of rating systems for preoperative severity and overcomes the difficulty of acquiring large-scale assessment from expert surgeons.

Rethinking Assessment Design: Evidence-Informed Strategies to Boost Educational Impact in the Anatomical Sciences

University assessment is in the midst of transformation. Assessments are no longer designed solely to determine that students can remember and regurgitate lecture content, nor in order to rank students to aid with some future selection process. Instead, assessments are expected to drive, support, and enhance learning and to contribute to student self-assessment and development of skills and attributes for a lifetime of learning. While traditional purposes of certifying achievement and determining readiness to progress remain important, these new expectations for assessment can create tensions in assessment design, selection, and deployment. With the recognition of these tensions, three contemporary approaches to assessment in medical education are described. These approaches include careful consideration of the educational impact of assessment-before, during (test or recall enhanced learning) and after assessments; development of student (and staff) assessment literacy; and planning of cohesive systems of assessment (with a range of assessment tools) designed to assess the various competencies demanded of future graduates. These approaches purposefully straddle the cross purposes of assessment in modern health professions education. The implications of these models are explored within the context of medical education and then linked with contemporary work in the anatomical sciences in order to highlight current synergies and potential future innovations when using evidence-informed strategies to boost the educational impact of assessments.

Training Novice Raters to Assess Nontechnical Skills of Operating Room Teams

OBJECTIVE

To our knowledge, no curricula have been described for training novice, nonclinician raters of nontechnical skills in the operating room (OR). We aimed to report the reliability of Oxford Non-Technical Skills (NOTECHS) ratings provided by novice raters who underwent a scalable curriculum for learning to assess nontechnical skills of OR teams.

DESIGN

In-person training course to apply the NOTECHS framework to assessing OR teams' nontechnical skill performance, led by 2 facilitators and involving 5 partial-day sessions of didactic presentations, video simulation, and live OR observation with postassessment debriefing. NOTECHS ratings were submitted after each of 11 video scenarios and 8 live operations for the total NOTECHS team rating (including surgical/anesthesiology/nursing subteams) and for each NOTECHS skill category-situation awareness, problem solving and decision making, teamwork and cooperation, leadership and management. Inter-rater reliability was determined by calculating the intraclass correlation coefficient (ICC, range 0-1).

SETTING

Training for outcome measurement during a quality improvement initiative focused on surgical safety in 3 public hospitals in Singapore. Two trainings were conducted in May 2019 and January 2020.

PARTICIPANTS

Ten novice raters who were existing hospital staff and had overall minimal OR experience and no prior experience with nontechnical skill assessment.

RESULTS

ICC for the total NOTECHS team rating was 0.89 (95% confidence interval [CI], 0.87-0.91). ICCs for each NOTECHS category were as follows: situation awareness, 0.83 (95% CI, 0.78-0.88); problem solving and decision-making, 0.76 (95% CI, 0.70-0.83); teamwork and cooperation, 0.84 (95% CI, 0.79-0.88); leadership and management, 0.81 (95% CI, 0.75-0.86).

CONCLUSIONS

This training curriculum for nontechnical skill assessments of OR teams was associated with high inter-rater reliability from novice raters with minimal collective OR experience. Using scalable training materials to produce reliable measurements of OR team performance, this nontechnical skills assessment curriculum may contribute to future QI projects aimed at improving surgical safety.

Crowdsourced Assessment of Aesthetic Outcomes after Breast Reconstruction

BACKGROUND

Evaluating the aesthetic success of breast reconstruction can be difficult. Patients, surgeons, and the general population may differ in what constitutes a successful outcome. Recently, crowdsourcing has emerged to accumulate and analyze data on a massive scale. The authors propose that crowdsourcing can be a useful tool to reliably rate aesthetic outcomes of breast reconstruction.

METHODS

One hundred one deidentified photographs of patients at various stages of breast reconstruction were gathered. Assessment tools included a five-point Likert scale and the transverse rectus abdominis myocutaneous (TRAM) visual assessment scale. Anonymous crowd workers and a group of expert reconstructive surgeons rated an identical set of photographs on the Likert scale. Crowd workers also rated the set of photographs on the TRAM scale.

RESULTS

The authors obtained 901 anonymous, layperson evaluations on both Likert and TRAM scales. Crowdsourced assessment data collection took 28.6 hours. Expert assessment took 15 months. Expert and crowdsourced scores were equivalent on the Likert scale (overall interrater reliability, κ = 0.99; 95 percent CI, 0.98 to 0.99). Intrarater reliability among each subcomponent was highly reproducible for the crowd (r = 0.98; 95 percent CI, 0.97 to 0.99) and experts (r = 0.82; 95 percent CI, 0.77 to 0.87). Breast contour and positioning were most predictive of overall aesthetic result. Skin patch and scar were least predictive of overall aesthetic appearance.

CONCLUSIONS

Aesthetic outcomes rated by crowds were reliable and correlated closely with those by expert surgeons. Crowdsourcing can be a rapid, reliable, and valid way to assess aesthetic outcomes in the breast reconstruction patient.

Current methods for assessing technical skill in cataract surgery

Surgery is a major source of errors in patient care. Preventing complications from surgical errors in the operating room is estimated to lead to reduction of up to 41 846 readmissions and save $620.3 million per year. It is now established that poor technical skill is associated with an increased risk of severe adverse events postoperatively and traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. This review discusses the current methods available for evaluating technical skills in cataract surgery and the recent technological advancements that have enabled capture and analysis of large amounts of complex surgical data for more automated objective skills assessment.

Robotic Surgery: The Impact of Simulation and Other Innovative Platforms on Performance and Training

OBJECTIVE

To review the current status of robotic training and the impact of various training platforms on the performance of robotic surgical trainees.

DATA SOURCES

Literature review of Google Scholar and PubMed. The search terms included a combination of the following: "robotic training," "simulation," "robotic curriculum," "obgyn residency robotic training," "virtual reality robotic training," "DaVinci training," "surgical simulation," "gyn surgical training." The sources considered for inclusion included peer-reviewed articles, literature reviews, textbook chapters, and statements from various institutions involved in resident training.

METHODS OF STUDY SELECTION

A literature search of Google Scholar and PubMed using terms related to robotic surgery and robotics training, as mentioned in the "Data Sources" section.

RESULTS

Multiple novel platforms that use machine learning and real-time video feedback to teach and evaluate robotic surgical skills have been developed over recent years. Various training curricula, virtual reality simulators, and other robotic training tools have been shown to enhance robotic surgical education and improve surgical skills. The integration of didactic learning, simulation, and intraoperative teaching into more comprehensive training curricula shows positive effects on robotic skills proficiency. Few robotic surgery training curricula have been validated through peer-reviewed study, and there is more work to be completed in this area. In addition, there is a lack of information about how the skills obtained through robotics curricula and simulation translate into operating room performance and patient outcomes.

CONCLUSION

Data collected to date show promising advances in the training of robotic surgeons. A diverse array of curricula for training robotic surgeons continue to emerge, and existing teaching modalities are evolving to keep up with the rapidly growing demand for proficient robotic surgeons. Futures areas of growth include establishing competency benchmarks for existing training tools, validating existing curricula, and determining how to translate the acquired skills in simulation into performance in the operating room and patient outcomes. Many surgical training platforms are beginning to expand beyond discrete robotic skills training to procedure-specific and team training. There is still a wealth of research to be done to understand how to create an effective training experience for gynecologic surgical trainees and robotics teams.

Applying the Delphi process for development of a hepatopancreaticobiliary robotic surgery training curriculum

BACKGROUND

Robotic hepatopancreaticobiliary (HPB) procedures are performed worldwide and establishing processes for safe adoption of this technology is essential for patient benefit. We report results of the Delphi process to define and optimize robotic training procedures for HPB surgeons.

METHODS

In 2019, a robotic HPB surgery panel with an interest in surgical training from the Americas and Europe was created and met. An e-consensus-finding exercise using the Delphi process was applied and consensus was defined as 80% agreement on each question. Iterations of anonymous voting continued over three rounds.

RESULTS

Members agreed on several points: there was need for a standardized robotic training curriculum for HPB surgery that considers experience of surgeons and based on a robotic hepatectomy includes a common approach for "basic robotic skills" training (e-learning module, including hardware description, patient selection, port placement, docking, troubleshooting, fundamentals of robotic surgery, team training and efficiency, and emergencies) and an "advanced technical skills curriculum" (e-learning, including patient selection information, cognitive skills, and recommended operative equipment lists). A modular approach to index procedures should be used with video demonstrations, port placement for index procedure, troubleshooting, and emergency scenario management information. Inexperienced surgeons should undergo training in basic robotic skills and console proficiency, transitioning to full procedure training of e-learning (video demonstration, simulation training, case observation, and final evaluation). Experienced surgeons should undergo basic training when using a new system (e-learning, dry lab, and operating room (OR) team training, virtual reality modules, and wet lab; case observations were unnecessary for basic training) and should complete the advanced index procedural robotic curriculum with assessment by wet lab, case observation, and OR team training.

CONCLUSIONS

Optimization and standardization of training and education of HPB surgeons in robotic procedures was agreed upon. Results are being incorporated into future curriculum for education in robotic surgery.

Updates and Controversies of Robotic-Assisted Surgery in Gynecologic Surgery

minimally invasive surgery (MIS) is the standard approach to performance of several gynecologic procedures, including hysterectomy, gynecologic cancer staging procedures, myomectomy, pelvic organ prolapse repair, and select adnexal procedures. Robotic-assisted surgery, a computer-based MIS approach, has been adopted widely in the United States and several other countries. Robotics may offer technological and ergonomic benefits that overcome limitations associated with conventional laparoscopy; however, it is not clear that reported claims of superiority translate into improved gynecologic patient outcomes compared with other MIS approaches. This review critically appraises the evolving role, benefits, limitations, and controversies of robotic-assisted surgery utilization in benign and oncologic gynecology settings.

Transoral Robotic Surgical Proficiency Via Real-Time Tactile Collision Awareness System

OBJECTIVES

In 2009, the Food and Drug Administration approved the use of the surgical robotic system for removal of benign and malignant conditions of the upper aerodigestive tract. This novel application of robotic-assisted surgery, termed transoral robotic surgery (TORS), places robotic instruments and camera system through the mouth to reach recessed areas of the pharynx and larynx. Over the successive decade, there was a rapid adoption of TORS with a surgical growth rate that continues to increase. Despite the rapid clinical acceptance, the field of TORS has not yet seen substantive changes or advances in the technical shortcomings, the lack of which has restricted objective TORS-specific surgical skills assessment as well as subsequent skills improvement efforts. One of the primary technical challenges of TORS is operating in a confined space, where the robotic system is maneuvered within the restrictive boundaries of the mouth and throat. Due to these confined boundaries of the pharynx, instruments can frequently collide with anatomic structures such as teeth and bone, producing anatomic collisions. Therefore, we hypothesized that anatomic collisions negatively impact TORS surgical performance. Secondarily, we hypothesized that avoidance of unwanted anatomic collisions could improve TORS surgical proficiency.

METHODS

Design and fidelity testing for a custom TORS training platform with an integrated anatomic collision-sensing system providing real-time tactile feedback is described. Following successful platform assembly and testing, validation study using the platform was carried through prospective surgical training with trial randomization. Twenty otolaryngology-head and neck surgery residents, each trainee performing three discrete mock surgical trials (n = 60), performed the initial system validation. Ten of the 20 residents were randomized to perform the surgical trials utilizing the real-time feedback system. The remaining 10 residents were randomized to perform the surgical trials without the feedback system, although the system still could record collision data. Surgical proficiency was measured by Global Evaluative Assessment of Robotic Skills (GEARS) score, time to completion, and tumor resection scores (categorical scale ranging 0-3, describing the adequacy of resection).

RESULTS

Major anatomic collisions (greater than 5N of force) negatively affected GEARS robotic skills. A mixed model analysis demonstrated that for every additional occurrence of a major collision, GEARS robotic skills assessment score would decrease by 0.29 points (P = .04). Real-time collision awareness created significantly fewer major (> 5 N) anatomic collisions with the tactile feedback system active (n = 30, mean collisions = 2.9 ± 4.2) as compared with trials without tactile feedback (n = 30, mean collisions = 12.53 ± 23.23) (P < .001). The second assessment measure of time to completion was unaffected by the presence of collisions or by the use of tactile feedback system. The third proficiency assessment was measured with tumor resection grading. Tumor resection scores was significantly (P = .02) improved with collision awareness system activated than trials without collision awareness.

CONCLUSION

In order to test our primary hypothesis, a novel TORS training platform was successfully developed that provides collision force measurements including frequency, severity, and duration of anatomic collisions. Additionally, the platform was modulated to provide real-time tactile feedback of the occurrence of out-of-field collisions. Utilizing this custom platform, our hypothesis that anatomic collisions during TORS diminishes surgical performance was supported. Additionally, our secondary hypothesis that subsequent reduction of anatomic collisions improves TORS proficiency was supported by the surgical trial. Dedicated investigation to characterize the effect size and clinical impact is required in order to translate this finding into training curriculums and into clinical utilization.

LEVEL OF EVIDENCE

II (Randomized trial) Laryngoscope, 130:S1-S17, 2020.

Crowdsourcing: A novel tool to elicit the student voice in the curriculum design process for an undergraduate diagnostic radiography degree programme

INTRODUCTION

Stakeholder participation in healthcare curriculum design is an important aspect of higher education with stakeholders including students, staff members, clinical partners, healthcare organisations, patients and members of the public. Significantly, student co-creation, of the curriculum, has become increasingly important. Yet there is limited research which addresses how to engage this group in design processes.

METHODS

This paper represents the first phase of a three stage action research spiral whereby the authors evaluated the use of a novel tool for curriculum design processes, anonymised crowdsourcing. This initial phase was open to all students enrolled on an undergraduate diagnostic radiography programme in the UK. To confirm the reliability of the crowdsource design an established eight point crowdsourcing verification tool was applied.

RESULTS

Twenty-three unique ideas were generated by participants, 40 comments made and 173 votes cast. Inductive analysis of the comments generated five themes. These included: the role of technology enhanced learning; simulation activities; patient focused curriculum; mental wealth (resilience) authentic assessment approaches. An evaluation of those who had and had not engaged highlighted areas of improvement for the administration of the second and third iterations which will include a wider pool of participants.

CONCLUSION

This study from a single programme offers lessons for others wishing to adopt and develop this approach elsewhere.

IMPLICATIONS FOR PRACTICE

Several ideas elicited by the crowdsource have been considered by the curriculum design team and will be implemented in the 2020 curriculum thus demonstrating the impact on local education practice of this research approach.

Multimodal Physiological Signals for Workload Prediction in Robot-assisted Surgery

Monitoring surgeon workload during robot-assisted surgery can guide allocation of task demands, adapt system interfaces, and assess the robotic system's usability. Current practices for measuring cognitive load primarily rely on questionnaires that are subjective and disrupt surgical workflow. To address this limitation, a computational framework is demonstrated to predict user workload during telerobotic surgery. This framework leverages wireless sensors to monitor surgeons’ cognitive load and predict their cognitive states. Continuous data across multiple physiological modalities (e.g., heart rate variability, electrodermal, and electroencephalogram activity) were simultaneously recorded for twelve surgeons performing surgical skills tasks on the validated da Vinci Skills Simulator. These surgical tasks varied in difficulty levels, e.g., requiring varying visual processing demand and degree of fine motor control. Collected multimodal physiological signals were fused using independent component analysis, and the predicted results were compared to the ground-truth workload level. Results compared performance of different classifiers, sensor fusion schemes, and physiological modality (i.e., prediction with single vs. multiple modalities). It was found that our multisensor approach outperformed individual signals and can correctly predict cognitive workload levels 83.2% of the time during basic and complex surgical skills tasks.

Crowdsourcing in health and medical research: a systematic review

Background

Crowdsourcing is used increasingly in health and medical research. Crowdsourcing is the process of aggregating crowd wisdom to solve a problem. The purpose of this systematic review is to summarize quantitative evidence on crowdsourcing to improve health.

Methods

We followed Cochrane systematic review guidance and systematically searched seven databases up to September 4th 2019. Studies were included if they reported on crowdsourcing and related to health or medicine. Studies were excluded if recruitment was the only use of crowdsourcing. We determined the level of evidence associated with review findings using the GRADE approach.

Results

We screened 3508 citations, accessed 362 articles, and included 188 studies. Ninety-six studies examined effectiveness, 127 examined feasibility, and 37 examined cost. The most common purposes were to evaluate surgical skills (17 studies), to create sexual health messages (seven studies), and to provide layperson cardio-pulmonary resuscitation (CPR) out-of-hospital (six studies). Seventeen observational studies used crowdsourcing to evaluate surgical skills, finding that crowdsourcing evaluation was as effective as expert evaluation (low quality). Four studies used a challenge contest to solicit human immunodeficiency virus (HIV) testing promotion materials and increase HIV testing rates (moderate quality), and two of the four studies found this approach saved money. Three studies suggested that an interactive technology system increased rates of layperson initiated CPR out-of-hospital (moderate quality). However, studies analyzing crowdsourcing to evaluate surgical skills and layperson-initiated CPR were only from high-income countries. Five studies examined crowdsourcing to inform artificial intelligence projects, most often related to annotation of medical data. Crowdsourcing was evaluated using different outcomes, limiting the extent to which studies could be pooled.

Conclusions

Crowdsourcing has been used to improve health in many settings. Although crowdsourcing is effective at improving behavioral outcomes, more research is needed to understand effects on clinical outcomes and costs. More research is needed on crowdsourcing as a tool to develop artificial intelligence systems in medicine.

Trial registration

PROSPERO: CRD42017052835. December 27, 2016.

Development of a Transoral Robotic Surgery Training Platform

Transoral robotic surgery (TORS) presents unique challenges due to difficulty manipulating surgical instruments within the tight confines of the oral cavity. Collisions between the end effectors and anatomical structures can be visualized through the endoscope; however, instrument shaft collisions are outside of the field-of-view. Acquiring the requisite skill set to minimize these collisions is challenging due to the lack of an appropriate training platform. In this paper, we present a TORS training platform with an integrated collision sensing system and real-time haptic feedback. Preliminary testing involved the recruitment of 10 Otolaryngology residents assigned to `feedback' (N=5) and `no feedback' (N=5) groups. Each trainee performed three mock surgical procedures involving the resection of a tumor from the base of the tongue. Superior surgical performance was observed in the feedback group suggesting that haptic feedback will enhance the acquisition of surgical skills.

Can a Structured, Video-Based Cadaver Curriculum Demonstrating Proficiency Enhance Resident Operative Autonomy?

INTRODUCTION

Autonomy is of foremost concern in the current era of surgical residency, and it is especially important to trainees when considering their surgical education. Factors impacting trainee independence include the restriction of clinical work hours and the development of advanced minimally invasive techniques such as robotics, which requires separate technical education outside of conventional surgical education. Moreover, when residents are left to learn fundamental exposures via their clinical experience alone, they run the risk of not being exposed to some fundamental skills based on case volume and type. The Department of Surgery at Emory University developed a cadaver-based simulation curriculum to standardize exposure to fundamental operative skills and enhance proficiency outside the operating room, with the larger aim of improving resident autonomy.

METHODS

Residents were assigned to small groups led by a chief resident with an even distribution of postgraduate year (PGY) levels. Each group participated in core surgical exposures and fundamental maneuvers on a cadaver over a 6-hour session. Residents were tested on skills according to their PGY level about 1 month after the course. Testing included recitation of the skill in an oral boards format, highlighting major steps, followed by performance of the skill. All steps were video-recorded with no resident identifiers. These were reviewed by 2 independent, blinded faculty examiners who assigned proficiency grades to each resident video.

RESULTS

Three hundred and thirty-three individual procedure evaluations were done over the 5-year period. Senior residents (PGY3-5) had 86% pass rate while junior residents (PGY1-2) had 70% pass rate. Overall, 21% of residents failed to achieve competence in their assigned skills. Junior residents were less likely to achieve competence compared to senior residents. Faculty graders had improved congruence in grading as the course progressed through the 5 years. The most recent 2 years had >80% congruence in faculty grading compared to less than 50% congruence in the first 2 years. 81% of attendings agreed this course positively influenced the granting of autonomy in the operating room.

CONCLUSIONS

A cadaveric skills course focused on fundamental maneuvers with objective confirmation of achieving competency is a viable adjunct to clinical operative experience. Video-recorded evaluation, of these fundamental skills improved both resident and attending confidence in trainee operative skill.

Crowdsourced Assessment of Inanimate Biotissue Drills: A Valid and Cost-Effective Way to Evaluate Surgical Trainees

OBJECTIVE

Providing feedback to surgical trainees is a critical component for assessment of technical skills, yet remains costly and time consuming. We hypothesize that statistical selection can identify a homogenous group of nonexpert crowdworkers capable of accurately grading inanimate surgical video.

DESIGN

Applicants auditioned by grading 9 training videos using the Objective Structured Assessment of Technical Skills (OSATS) tool and an error-based checklist. The summed OSATS, summed errors, and OSATS summary score were tested for outliers using Cronbach's Alpha and single measure intraclass correlation. Accepted crowdworkers then submitted grades for videos in 3 different compositions: full video 1× speed, full video 2× speed, and critical section segmented video. Graders were blinded to this study and a similar statistical analysis was performed.

SETTING

The study was conducted at the University of Pittsburgh Medical Center (Pittsburgh, PA), a tertiary care academic teaching hospital.

PARTICIPANTS

Thirty-six premedical students participated as crowdworker applicants and 2 surgery experts were compared as the gold-standard.

RESULTS

The selected hire intraclass correlation was 0.717 for Total Errors and 0.794 for Total OSATS for the first hire group and 0.800 for Total OSATS and 0.654 for Total Errors for the second hire group. There was very good correlation between full videos at 1× and 2× speed with an interitem statistic of 0.817 for errors and 0.86 for OSATS. Only moderate correlation was found with critical section segments. In 1 year 275hours of inanimate video was graded costing $22.27/video or $1.03/minute.

CONCLUSIONS

Statistical selection can be used to identify a homogenous cohort of crowdworkers used for grading trainees' inanimate drills. Crowdworkers can distinguish OSATS metrics and errors in full videos at 2× speed but were less consistent with segmented videos. The program is a comparatively cost-effective way to provide feedback to surgical trainees.

Crowdsourcing in health professions education: What radiography educators can learn from other disciplines

OBJECTIVES

Crowdsourcing works through an institution outsourcing a function normally performed by an employee or group of individuals. Within a crowdsource users, known as the crowd, form a community who voluntarily undertake a task which involves the pooling of knowledge resources. A literature review was undertaken to identify how the tool is being used in health professions education, and potential for use in radiography education.

KEY FINDINGS

17 papers were returned. Literature identified was assessed against an established crowdsourcing definition. Reviewing these yielded four themes for discussion: student selection procedures, lesson planning, teaching materials and assessment.

CONCLUSION

Crowdsourcing is associated with innovative activities through collective solution seeking via a large network of users. It is increasingly being adopted in healthcare training and maybe transferable to educational activities within the field of radiography education.

Automatic and near real-time stylistic behavior assessment in robotic surgery

PURPOSE

Automatic skill evaluation is of great importance in surgical robotic training. Extensive research has been done to evaluate surgical skill, and a variety of quantitative metrics have been proposed. However, these methods primarily use expert selected features which may not capture latent information in movement data. In addition, these features are calculated over the entire task time and are provided to the user after the completion of the task. Thus, these quantitative metrics do not provide users with information on how to modify their movements to improve performance in real time. This study focuses on automatic stylistic behavior recognition that has the potential to be implemented in near real time.

METHODS

We propose a sparse coding framework for automatic stylistic behavior recognition in short time intervals using only position data from the hands, wrist, elbow, and shoulder. A codebook is built for each stylistic adjective using the positive and negative labels provided for each trial through crowd sourcing. Sparse code coefficients are obtained for short time intervals (0.25 s) in a trial using this codebook. A support vector machine classifier is trained and validated through tenfold cross-validation using the sparse codes from the training set.

RESULTS

The results indicate that the proposed dictionary learning method is able to assess stylistic behavior performance in near real time using user joint position data with improved accuracy compared to using PCA features or raw data.

CONCLUSION

The possibility to automatically evaluate a trainee's style of movement in short time intervals could provide the user with online customized feedback and thus improve performance during surgical tasks.

Crowdsourced versus expert evaluations of the vesico-urethral anastomosis in the robotic radical prostatectomy: is one superior at discriminating differences in automated performance metrics?

Crowdsourcing from the general population is an efficient, inexpensive method of surgical performance evaluation. In this study, we compared the discriminatory ability of experts and crowdsourced evaluators (the Crowd) to detect differences in robotic automated performance metrics (APMs). APMs (instrument motion tracking and events data directly from the robot system) of anterior vesico-urethral anastomoses (VUAs) of robotic radical prostatectomies were captured by the dVLogger (Intuitive Surgical). Crowdsourced evaluators and four expert surgeons evaluated video footage using the Global Evaluative Assessment of Robotic Skills (GEARS) (individual domains and total score). Cases were then stratified into performance groups (high versus low quality) for each evaluator based on GEARS. APMs from each group were compared using the Mann-Whitney U test. 25 VUAs performed by 11 surgeons were evaluated. The Crowd displayed moderate correlation with averaged expert scores for all GEARS domains (r > 0.58, p < 0.01). Bland-Altman analysis showed a narrower total GEARS score distribution by the Crowd compared to experts. APMs compared amongst performance groups for each evaluator showed that through GEARS scoring, the most common differentiated metric by evaluators was the velocity of the dominant instrument arm. The Crowd outperformed two out of four expert evaluators by discriminating differences in three APMs using total GEARS scores. The Crowd assigns a narrower range of GEARS scores compared to experts but maintains overall agreement with experts. The discriminatory ability of the Crowd at discerning differences in robotic movements (via APMs) through GEARS scoring is quite refined, rivaling that of expert evaluators.

Crowdsourcing Public Perceptions of Plastic Surgeons: Is There a Gender Bias?

BACKGROUND

Implicit gender bias may result in lower wages for women, fewer leadership positions, and lower perceived competence. Understanding public and patient gender preferences for plastic surgeons may enable opportunities to address public perceptions. This investigation evaluates public preferences for a plastic surgeon's gender or demeanor.

METHODS

Members of the Amazon Mechanical Turk crowdsourcing platform read 1 of the 8 randomly assigned scenarios describing a hypothetical situation requiring a plastic surgeon to operate on their mother. The scenarios differed only by surgeon gender, surgeon demeanor ("agentic," traditionally more masculine versus "communal," traditionally more feminine), or type of surgery. Using a Likert scale, respondents indicated their agreement with 7 statements on surgeon competence, skills, leadership qualities, likeability, respect, trustworthiness, and, ultimately, preference as a surgeon. Independent t tests were used to compare scores. Lower scores indicated a more negative response.

RESULTS

Overall, 341 responses were received: 55.7% were male and 45.5% white. There were no significant differences in any of the 7 characteristics assessed when examining by surgeon gender, only. However, female surgeons with a communal demeanor were perceived as less competent (4.32 versus 4.51, P = 0.018) and less skilled (4.36 versus 4.56, P = 0.019) than agentic female surgeons. Male respondents rated female surgeons lower than male surgeons in terms of competence (P = 0.018), skills (P = 0.034), likeability (P = 0.042), and preferred choice as a surgeon (P = 0.033).

CONCLUSIONS

Women plastic surgeons' demeanor and respondent gender affected perception of certain characteristics. Women plastic surgeons may consider ways to engage with the public to address possible gender role stereotypes.

Nonspecialist Raters Can Provide Reliable Assessments of Procedural Skills

BACKGROUND

Competency-based learning has become a crucial component in medical education. Despite the advantages of competency-based learning, there are still challenges that need to be addressed. Currently, the common perception is that specialist assessment is needed for evaluating procedural skills which is difficult owing to the limited availability of faculty time. The aim of this study was to explore the validity of assessments of video recorded procedures performed by nonspecialist raters.

METHODS

This study was a blinded observational trial. Twenty-three novices (senior medical students) and 9 experienced doctors were video recorded while each performing 2 flexible cystoscopies on patients. The recordings were anonymized and placed in random order and then rated by 2 experienced cystoscopists (specialist raters) and 2 medical students (nonspecialist raters). Flexible cystoscopy was chosen as it is a simple procedural skill that is crucial to master in a resident urology program.

RESULTS

The internal consistency of assessments was high, Cronbach's α = 0.93 and 0.95 for nonspecialist and specialist raters, respectively (p < 0.001 for both correlations). The interrater reliability was significant (p < 0.001) with a Pearson's correlation of 0.77 for the nonspecialists and 0.75 for the specialists. The test-retest reliability showed the biggest difference between the 2 groups, 0.59 and 0.38 for the nonspecialist raters and the specialist raters, respectively (p < 0.001).

CONCLUSION

Our study suggests that nonspecialist raters can provide reliable and valid assessments of video recorded cystoscopies. This could make mastery learning and competency-based education more feasible.

Crowdsourcing Evaluation of Ureteroscopic Videos Using the Post-Ureteroscopic Lesion Scale to Assess Ureteral Injury

INTRODUCTION AND OBJECTIVES

We hypothesized that crowdsourcing assessments could be applied to the Postureteroscopic Lesion Scale (PULS) for ureteral injury.

METHODS

At a single institution, we prospectively digitally recorded 14 ureters at the terminal portion of standard ureteroscopic procedures. Each recording was reviewed by 10 global experts to determine a mean PULS score. Following training, the Crowd-Sourced Assessment of Technical Skills, C-SATS^® (C-SATS, Inc., Seattle, WA) platform was used to obtain crowd-based reviews. The mean crowd PULS scores was determined using the linear mixed-effects (LME) model. The intraclass correlation coefficient (ICC) was calculated to measure the agreement among experts. Spearman's rank correlation (rho) was used to quantify the strength of the relationship between the crowd LME mean and the experts.

RESULTS

Ten expert's reviews and 2100 layman reviews were obtained in 21 days and 49 hours, respectively. The ICC for the 10 experts was 0.68 (95% confidence interval 0.49, 0.86). When the expert mean PULS was <1, the crowd scored those recordings at 1 or greater. The highest scored recording by the experts was a 3.2, which the crowd scored at 2.25. The correlation between the crowd LME means and expert means across all videos was 0.70 (p = 0.0056) indicative of moderately strong agreement.

CONCLUSION

In this initial application of crowd-sourced evaluation of ureteral injury, there was a moderately strong correlation between crowd and expert ratings. Refinement of the training, through exposure to the nuances of ureteral injuries, in particular for PULS <1 or ≥3, may lead to better crowd/expert correlation. Compared to expert review, crowd data can be collected with much greater efficiency.

Crowdsourcing in Surgical Skills Acquisition: A Developing Technology in Surgical Education

BACKGROUND

The application of crowdsourcing to surgical education is a recent phenomenon and adds to increasing demands on surgical residency training. The efficacy, range, and scope of this technology for surgical education remains incompletely defined.

OBJECTIVE

A systematic review was performed using the PubMed database of English-language literature on crowdsourced evaluation of surgical technical tasks up to April 2017.

METHODS

Articles were reviewed, abstracted, and analyzed, and were assessed for quality using the Medical Education Research Study Quality Instrument (MERSQI). Articles were evaluated with eligibility criteria for inclusion. Study information, performance task, subjects, evaluative standards, crowdworker compensation, time to response, and correlation between crowd and expert or standard evaluations were abstracted and analyzed.

RESULTS

Of 63 unique publications initially identified, 13 with MERSQI scores ranging from 10 to 13 (mean = 11.85) were included in the review. Overall, crowd and expert evaluations demonstrated good to excellent correlation across a wide range of tasks (Pearson's coefficient 0.59-0.95, Cronbach's alpha 0.32-0.92), with 1 exception being a study involving medical students. There was a wide range of reported interrater variability among experts. Nonexpert evaluation was consistently quicker than expert evaluation (ranging from 4.8 to 150.9 times faster), and was more cost effective.

CONCLUSIONS

Crowdsourced feedback appears to be comparable to expert feedback and is cost effective and efficient. Further work is needed to increase consistency in expert evaluations, to explore sources of discrepant assessments between surgeons and crowds, and to identify optimal populations and novel applications for this technology.

Training in Robotic Surgery-an Overview

PURPOSE OF REVIEW

There has been a rapid and widespread adoption of the robotic surgical system with a lag in the development of a comprehensive training and credentialing framework. A literature search on robotic surgical training techniques and benchmarks was conducted to provide an evidence-based road map for the development of a robotic surgical skills for the novice robotic surgeon.

RECENT FINDINGS

A structured training curriculum is suggested incorporating evidence-based training techniques and benchmarks for progress. This usually involves sequential progression from observation, case assisting, acquisition of basic robotic skills in the dry and wet lab setting along with achievement of individual and team-based non-technical skills, modular console training under supervision, and finally independent practice. Robotic surgical training must be based on demonstration of proficiency and safety in executing basic robotic skills and procedural tasks prior to independent practice.

C-SATS: Assessing Surgical Skills Among Urology Residency Applicants

BACKGROUND

We hypothesized that surgical skills assessment could aid in the selection process of medical student applicants to a surgical program. Recently, crowdsourcing has been shown to provide an accurate assessment of surgical skills at all levels of training. We compared expert and crowd assessment of surgical tasks performed by resident applicants during their interview day at the urology program at the University of California, Irvine.

MATERIALS AND METHODS

Twenty-five resident interviewees performed four tasks: open square knot tying, laparoscopic peg transfer, robotic suturing, and skill task 8 on the LAP Mentor™ (Simbionix Ltd., Lod, Israel). Faculty experts and crowd workers (Crowd-Sourced Assessment of Technical Skills [C-SATS], Seattle, WA) assessed recorded performances using the Objective Structured Assessment of Technical Skills (OSATS), Global Evaluative Assessment of Robotic Skills (GEARS), and the Global Operative Assessment of Laparoscopic Skills (GOALS) validated assessment tools.

RESULTS

Overall, 3938 crowd assessments were obtained for the four tasks in less than 3.5 hours, whereas the average time to receive 150 expert assessments was 22 days. Inter-rater agreement between expert and crowd assessment scores was 0.62 for open knot tying, 0.92 for laparoscopic peg transfer, and 0.86 for robotic suturing. Agreement between applicant rank on skill task 8 on the LAP Mentor assessment and crowd assessment was 0.32. The crowd match rank based solely on skills performance did not compare well with the final faculty match rank list (0.46); however, none of the bottom five crowd-rated applicants appeared in the top five expert-rated applicants and none of the top five crowd-rated applicants appeared in the bottom five expert-rated applicants.

CONCLUSIONS

Crowd-source assessment of resident applicant surgical skills has good inter-rater agreement with expert physician raters but not with a computer-based objective motion metrics software assessment. Overall applicant rank was affected to some degree by the crowd performance rating.

Milestone assessment of minimally invasive surgery in Pediatric Urology fellowship programs

INTRODUCTION

Minimally invasive surgery has become an important aspect of Pediatric Urology fellowship training. In 2014, the Accreditation Council for Graduate Medical Education published the Pediatric Urology Milestone Project as a metric of fellow proficiency in multiple facets of training, including laparoscopic/robotic procedures.

OBJECTIVE

The present study assessed trends in minimally invasive surgery training and utilization of the Milestones among recent Pediatric Urology fellows.

STUDY DESIGN

Using an electronic survey instrument, Pediatric Urology fellowship program directors and fellows who completed their clinical year in 2015 were surveyed. Participants were queried regarding familiarity with the Milestone Project, utilization of the Milestones, robotic/laparoscopic case volume and training experience, and perceived competency with robotic/laparoscopic surgery at the start and end of the fellowship clinical year according to Milestone criteria. Responses were accepted between August and November 2015.

RESULTS

Surveys were distributed via e-mail to 35 fellows and 30 program directors. Sixteen fellows (46%) and 14 (47%) program directors responded. All fellows reported some robotic experience prior to fellowship, and 69% performed >50 robotic/laparoscopic surgeries during residency. Fellow robotic/laparoscopic case volume varied: three had 1-10 cases (19%), four had 11-20 cases (25%), and nine had >20 cases (56%). Supplementary or robotic training modalities included simulation (9), animal models (6), surgical videos (7), and courses (2). Comparison of beginning and end of fellowship robotic/laparoscopic Milestone assessment (Summary Fig.) revealed scores of <3 in (10) 62% of fellow self-assessments and 10 (75%) of program director assessments. End of training Milestone scores >4 were seen in 12 (75%) of fellow self-assessment and eight (57%) of program director assessments.

DISCUSSION

An improvement in robotic/laparoscopic Milestone scores by both fellow self-assessment and program director assessment was observed during the course of training; however, 43% of program directors rated their fellow below the graduation target of a Milestone score of 4.

CONCLUSION

The best ways to teach minimally invasive surgery in fellowship training must be critically considered.