Generalized approach for modeling minimally invasive surgery as a stochastic process using a discrete Markov model

Surgical Artificial Intelligence in Urology: Educational Applications

Surgical education has seen immense change recently. Increased demand for iterative evaluation of trainees from medical school to independent practice has led to the generation of an overwhelming amount of data related to an individual's competency. Artificial intelligence has been proposed as a solution to automate and standardize the ability of stakeholders to assess the technical and nontechnical abilities of a surgical trainee. In both the simulation and clinical environments, evidence supports the use of machine learning algorithms to both evaluate trainee skill and provide real-time and automated feedback, enabling a shortened learning curve for many key procedural skills and ensuring patient safety.

Assessment of open surgery suturing skill: Simulator platform, force-based, and motion-based metrics

Objective

This paper focuses on simulator-based assessment of open surgery suturing skill. We introduce a new surgical simulator designed to collect synchronized force, motion, video and touch data during a radial suturing task adapted from the Fundamentals of Vascular Surgery (FVS) skill assessment. The synchronized data is analyzed to extract objective metrics for suturing skill assessment.

Methods

The simulator has a camera positioned underneath the suturing membrane, enabling visual tracking of the needle during suturing. Needle tracking data enables extraction of meaningful metrics related to both the process and the product of the suturing task. To better simulate surgical conditions, the height of the system and the depth of the membrane are both adjustable. Metrics for assessment of suturing skill based on force/torque, motion, and physical contact are presented. Experimental data are presented from a study comparing attending surgeons and surgery residents.

Results

Analysis shows force metrics (absolute maximum force/torque in z-direction), motion metrics (yaw, pitch, roll), physical contact metric, and image-enabled force metrics (orthogonal and tangential forces) are found to be statistically significant in differentiating suturing skill between attendings and residents.

Conclusion and significance

The results suggest that this simulator and accompanying metrics could serve as a useful tool for assessing and teaching open surgery suturing skill.

Analysis of executional and procedural errors in dry‐lab robotic surgery experiments

Background

Analysing kinematic and video data can help identify potentially erroneous motions that lead to sub‐optimal surgeon performance and safety‐critical events in robot‐assisted surgery.

Methods

We develop a rubric for identifying task and gesture‐specific executional and procedural errors and evaluate dry‐lab demonstrations of suturing and needle passing tasks from the JIGSAWS dataset. We characterise erroneous parts of demonstrations by labelling video data, and use distribution similarity analysis and trajectory averaging on kinematic data to identify parameters that distinguish erroneous gestures.

Results

Executional error frequency varies by task and gesture, and correlates with skill level. Some predominant error modes in each gesture are distinguishable by analysing error‐specific kinematic parameters. Procedural errors could lead to lower performance scores and increased demonstration times but also depend on surgical style.

Conclusions

This study provides insights into context‐dependent errors that can be used to design automated error detection mechanisms and improve training and skill assessment.

Fine-Grained Video Captioning via Graph-based Multi-Granularity Interaction Learning

Learning to generate continuous linguistic descriptions for multi-subject interactive videos in great details has particular applications in team sports auto-narrative. In contrast to traditional video caption, this task is more challenging as it requires simultaneous modeling of fine-grained individual actions, uncovering of spatio-temporal dependency structures of frequent group interactions, and then accurate mapping of these complex interaction details into long and detailed commentary. To explicitly address these challenges, we propose a novel framework Graph-based Learning for Multi-Granularity Interaction Representation (GLMGIR) for fine-grained team sports auto-narrative task. A multi-granular interaction modeling module is proposed to extract among-subjects' interactive actions in a progressive way for encoding both intra- and inter-team interactions. Based on the above multi-granular representations, a multi-granular attention module is developed to consider action/event descriptions of multiple spatio-temporal resolutions. Both modules are integrated seamlessly and work in a collaborative way to generate the final narrative. In the meantime, to facilitate reproducible research, we collect a new video dataset from YouTube.com called Sports Video Narrative dataset (SVN). It is a novel direction as it contains 6K team sports videos (i.e., NBA basketball games) with 10K ground-truth narratives(e.g., sentences). Furthermore, as previous metrics such as METEOR (i.e., used in coarse-grained video caption task) DO NOT cope with fine-grained sports narrative task well, we hence develop a novel evaluation metric named Fine-grained Captioning Evaluation (FCE), which measures how accurate the generated linguistic description reflects fine-grained action details as well as the overall spatio-temporal interactional structure. Extensive experiments on our SVN dataset have demonstrated the effectiveness of the proposed framework for fine-grained team sports video auto-narrative.

Continuous Operator Authentication for Teleoperated Systems Using Hidden Markov Models

In this article, we present a novel approach for continuous operator authentication in teleoperated robotic processes based on Hidden Markov Models (HMM). While HMMs were originally developed and widely used in speech recognition, they have shown great performance in human motion and activity modeling. We make an analogy between human language and teleoperated robotic processes (i.e., words are analogous to a teleoperator’s gestures, sentences are analogous to the entire teleoperated task or process) and implement HMMs to model the teleoperated task. To test the continuous authentication performance of the proposed method, we conducted two sets of analyses. We built a virtual reality (VR) experimental environment using a commodity VR headset (HTC Vive) and haptic feedback enabled controller (Sensable PHANToM Omni) to simulate a real teleoperated task. An experimental study with 10 subjects was then conducted. We also performed simulated continuous operator authentication by using the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). The performance of the model was evaluated based on the continuous (real-time) operator authentication accuracy as well as resistance to a simulated impersonation attack. The results suggest that the proposed method is able to achieve 70% (VR experiment) and 81% (JIGSAWS dataset) continuous classification accuracy with as short as a 1-second sample window. It is also capable of detecting an impersonation attack in real-time.

Application of Design Structure Matrix to Simulate Surgical Procedures and Predict Surgery Duration

Background

The complexities of surgery require an efficient and explicit method to evaluate and standardize surgical procedures. A reliable surgical evaluation tool will be able to serve various purposes such as development of surgery training programs and improvement of surgical skills.

Objectives

(a) To develop a modeling framework based on integration of dexterity analysis and design structure matrix (DSM), to be generally applicable to predict total duration of a surgical procedure, and (b) to validate the model by comparing its results with laparoscopic cholecystectomy surgery protocol.

Method

A modeling framework is developed through DSM, a tool used in engineering design, systems engineering and management, to hierarchically decompose and describe relationships among individual surgical activities. Individual decomposed activities are assumed to have uncertain parameters so that a rework probability is introduced. The simulation produces a distribution of the duration of the modeled procedure. A statistical approach is then taken to evaluate surgery duration through integrated numerical parameters. The modeling framework is applied for the first time to analyze a surgery; laparoscopic cholecystectomy, a common surgical procedure, is selected for the analysis.

Results

The present simulation model is validated by comparing its results of predicted surgery duration with the standard laparoscopic cholecystectomy protocols from the Atlas of Minimally Invasive Surgery with 2.5% error and that from the Atlas of Pediatric Laparoscopy and Thoracoscopy with 4% error.

Conclusion

The present model, developed based on dexterity analysis and DSM, demonstrates a validated capability of predicting laparoscopic cholecystectomy surgery duration. Future studies will explore its potential applications to other surgery procedures and in improving surgeons' performance and training novices.

Endowing Robots with Longer-term Autonomy by Recovering from External Disturbances in Manipulation Through Grounded Anomaly Classification and Recovery Policies

Robots are poised to interact with humans in unstructured environments. Despite increasingly robust control algorithms, failure modes arise whenever the underlying dynamics are poorly modeled, especially in unstructured environments. We contribute a set of recovery policies to deal with anomalies produced by external disturbances. The recoveries work when various different types of anomalies are triggered any number of times at any point in the task, including during already running recoveries. Our recovery critic stands atop of a tightly-integrated, graph-based online motion-generation and introspection system. Policies, skills, and introspection models are learned incrementally and contextually over time. Recoveries are studied via a collaborative kitting task where a wide range of anomalous conditions are experienced in the system. We also contribute an extensive analysis of the performance of the tightly integrated anomaly identification, classification, and recovery system under extreme anomalous conditions. We show how the integration of such a system achieves performances greater than the sum of its parts.

Visual Intelligence: Prediction of Unintentional Surgical-Tool-Induced Bleeding during Robotic and Laparoscopic Surgery

Unintentional vascular damage can result from a surgical instrument’s abrupt movements during minimally invasive surgery (laparoscopic or robotic). A novel real-time image processing algorithm based on local entropy is proposed that can detect abrupt movements of surgical instruments and predict bleeding occurrence. The uniform nature of the texture of surgical tools is utilized to segment the tools from the background. By comparing changes in entropy over time, the algorithm determines when the surgical instruments are moved abruptly. We tested the algorithm using 17 videos of minimally invasive surgery, 11 of which had tool-induced bleeding. Our preliminary testing shows that the algorithm is 88% accurate and 90% precise in predicting bleeding. The average advance warning time for the 11 videos is 0.662 s, with the standard deviation being 0.427 s. The proposed approach has the potential to eventually lead to a surgical early warning system or even proactively attenuate tool movement (for robotic surgery) to avoid dangerous surgical outcomes.

Surgical Performance Analysis and Classification Based on Video Annotation of Laparoscopic Tasks

Background and Objectives

Current approaches in surgical skills assessment employ virtual reality simulators, motion sensors, and task-specific checklists. Although accurate, these methods may be complex in the interpretation of the generated measures of performance. The aim of this study is to propose an alternative methodology for skills assessment and classification, based on video annotation of laparoscopic tasks.

Methods

Two groups of 32 trainees (students and residents) performed two laparoscopic tasks: peg transfer (PT) and knot tying (KT). Each task was annotated via a video analysis software based on a vocabulary of eight surgical gestures (surgemes) that denote the elementary gestures required to perform a task. The extracted metrics included duration/counts of each surgeme, penalty events, and counts of sequential surgemes (transitions). Our analysis focused on trainees' skill level comparison and classification using a nearest neighbor approach. The classification was assessed via accuracy, sensitivity, and specificity.

Results

For PT, almost all metrics showed significant performance difference between the two groups (p < 0.001). Residents were able to complete the task with fewer, shorter surgemes and fewer penalty events. Moreover, residents performed significantly fewer transitions (p < 0.05). For KT, residents performed two surgemes in significantly shorter time (p < 0.05). The metrics derived from the video annotations were also able to recognize the trainees' skill level with 0.71 - 0.86 accuracy, 0.80 - 1.00 sensitivity, and 0.60 - 0.80 specificity.

Conclusion

The proposed technique provides a tool for skills assessment and experience classification of surgical trainees, as well as an intuitive way for describing what and how surgemes are performed.

Motion analysis for better understanding of psychomotor skills in laparoscopy: objective assessment-based simulation training using animal organs

Background

Our aim was to characterize the motions of multiple laparoscopic surgical instruments among participants with different levels of surgical experience in a series of wet-lab training drills, in which participants need to perform a range of surgical procedures including grasping tissue, tissue traction and dissection, applying a Hem-o-lok clip, and suturing/knotting, and digitize the level of surgical competency.

Methods

Participants performed tissue dissection around the aorta, dividing encountered vessels after applying a Hem-o-lok (Task 1), and renal parenchymal closure (Task 2: suturing, Task 3: suturing and knot-tying), using swine cadaveric organs placed in a box trainer under a motion capture (Mocap) system. Motion-related metrics were compared according to participants’ level of surgical experience (experts: 50 ≤ laparoscopic surgeries, intermediates: 10–49, novices: 0–9), using the Kruskal–Wallis test, and significant metrics were subjected to principal component analysis (PCA).

Results

A total of 15 experts, 12 intermediates, and 18 novices participated in the training. In Task 1, a shorter path length and faster velocity/acceleration/jerk were observed using both scissors and a Hem-o-lok applier in the experts, and Hem-o-lok-related metrics markedly contributed to the 1st principal component on PCA analysis, followed by scissors-related metrics. Higher-level skills including a shorter path length and faster velocity were observed in both hands of the experts also in tasks 2 and 3. Sub-analysis showed that, in experts with 100 ≤  cases, scissors moved more frequently in the “close zone (0  ≤ to < 2.0 cm from aorta)” than those with 50–99 cases.

Conclusion

Our novel Mocap system recognized significant differences in several metrics in multiple instruments according to the level of surgical experience. “Applying a Hem-o-lok clip on a pedicle” strongly reflected the level of surgical experience, and zone-metrics may be a promising tool to assess surgical expertise. Our next challenge is to give completely objective feedback to trainees on-site in the wet-lab.

Electronic supplementary material

The online version of this article (10.1007/s00464-020-07940-7) contains supplementary material, which is available to authorized users.

Using Surgeon Hand Motions to Predict Surgical Maneuvers

OBJECTIVE

This study explores how common machine learning techniques can predict surgical maneuvers from a continuous video record of surgical benchtop simulations.

BACKGROUND

Automatic computer vision recognition of surgical maneuvers (suturing, tying, and transition) could expedite video review and objective assessment of surgeries.

METHOD

We recorded hand movements of 37 clinicians performing simple and running subcuticular suturing benchtop simulations, and applied three machine learning techniques (decision trees, random forests, and hidden Markov models) to classify surgical maneuvers every 2 s (60 frames) of video.

RESULTS

Random forest predictions of surgical video correctly classified 74% of all video segments into suturing, tying, and transition states for a randomly selected test set. Hidden Markov model adjustments improved the random forest predictions to 79% for simple interrupted suturing on a subset of randomly selected participants.

CONCLUSION

Random forest predictions aided by hidden Markov modeling provided the best prediction of surgical maneuvers. Training of models across all users improved prediction accuracy by 10% compared with a random selection of participants.

APPLICATION

Marker-less video hand tracking can predict surgical maneuvers from a continuous video record with similar accuracy as robot-assisted surgical platforms, and may enable more efficient video review of surgical procedures for training and coaching.

Using Machine Learning to Assess Physician Competence: A Systematic Review

PURPOSE

To identify the different machine learning (ML) techniques that have been applied to automate physician competence assessment and evaluate how these techniques can be used to assess different competence domains in several medical specialties.

METHOD

In May 2017, MEDLINE, EMBASE, PsycINFO, Web of Science, ACM Digital Library, IEEE Xplore Digital Library, PROSPERO, and Cochrane Database of Systematic Reviews were searched for articles published from inception to April 30, 2017. Studies were included if they applied at least one ML technique to assess medical students', residents', fellows', or attending physicians' competence. Information on sample size, participants, study setting and design, medical specialty, ML techniques, competence domains, outcomes, and methodological quality was extracted. MERSQI was used to evaluate quality, and a qualitative narrative synthesis of the medical specialties, ML techniques, and competence domains was conducted.

RESULTS

Of 4,953 initial articles, 69 met inclusion criteria. General surgery (24; 34.8%) and radiology (15; 21.7%) were the most studied specialties; natural language processing (24; 34.8%), support vector machine (15; 21.7%), and hidden Markov models (14; 20.3%) were the ML techniques most often applied; and patient care (63; 91.3%) and medical knowledge (45; 65.2%) were the most assessed competence domains.

CONCLUSIONS

A growing number of studies have attempted to apply ML techniques to physician competence assessment. Although many studies have investigated the feasibility of certain techniques, more validation research is needed. The use of ML techniques may have the potential to integrate and analyze pragmatic information that could be used in real-time assessments and interventions.

Using Brain Activation to Evaluate Arrangements Aiding Hand-Eye Coordination in Surgical Robot Systems

GOAL

To realize intuitive, minimally invasive surgery, surgical robots are often controlled using master-slave systems. However, the surgical robot's structure often differs from that of the human body, so the arrangement between the monitor and master must reflect this physical difference. In this study, we validate the feasibility of an embodiment evaluation method that determines the arrangement between the monitor and master. In our constructed cognitive model, the brain's intraparietal sulcus activates significantly when somatic and visual feedback match. Using this model, we validate a cognitively appropriate arrangement between the monitor and master.

METHODS

In experiments, we measure participants' brain activation using an imaging device as they control the virtual surgical simulator. Two experiments are carried out that vary the monitor and hand positions.

CONCLUSION

There are two common arrangements of the monitor and master at the brain activation's peak: One is placing the monitor behind the master, so the user feels that the system is an extension of his arms into the monitor; the other arranges the monitor in front of the master, so the user feels the correspondence between his own arm and the virtual arm in the monitor.

SIGNIFICANCE

From these results, we conclude that the arrangement between the monitor and master impacts embodiment, enabling the participant to feel apparent posture matches in master-slave surgical robot systems.

Real-Time Visual Tracking of Dynamic Surgical Suture Threads

In order to realize many of the potential benefits associated with robotically assisted minimally invasive surgery, the robot must be more than a remote controlled device. Currently, using a surgical robot can be challenging, fatiguing, and time consuming. Teaching the robot to actively assist surgical tasks, such as suturing, has the potential to vastly improve both patient outlook and the surgeon's efficiency. One obstacle to completing surgical sutures autonomously is the difficulty in tracking surgical suture threads. This paper presents novel stereo image processing algorithms for the detection, initialization, and tracking of a surgical suture thread. A Non Uniform Rational B-Spline (NURBS) curve is used to model a thin, deformable, and dynamic length thread. The NURBS model is initialized and grown from a single selected point located on the thread. The NURBS curve is optimized by minimizing the image matching energy between the projected stereo NURBS image and the segmented thread image. The algorithms are evaluated using suture threads, a calibrated test pattern, and a simulated thread image. Additionally, the accuracy of the algorithms presented are validated as they track a suture thread undergoing translation, deformation, and apparent length changes. All of the tracking is in real-time. Note to Practioners: Abstract-The problem of tracking a surgical suture thread was addressed in this work. Since the suture thread is highly deformable, any tracking algorithm must be robust to intersections, occlusions, knot tying, and length changes. The detection algorithm introduced in this paper is capable of distinguishing different threads when they intersect. The tracking algorithm presented here demonstrate that it is possible, using polynomial curves, to track a suture thread as it deforms, becomes occluded, changes length, and even ties a knot in real time. The detection algorithm can enhance directional thin features while the polynomial curve modeling can track any string like structure. Further integration of the polynomial curve with a feed-forward thread model could improve the stability and robustness of the thread tracking.

Vessel Ligation Fundamentals: A Comparison of Technical Evaluations by Crowdsourced Nonclinical Personnel and Surgical Faculty

BACKGROUND

Evaluation of fundamental surgical skills is invaluable to the training of medical students and junior residents. This study assessed the effectiveness of crowdsourcing nonmedical personnel to evaluate technical proficiency at simulated vessel ligation.

STUDY DESIGN

Fifteen videos were captured of participants performing vessel ligation using a low-fidelity model (5 attending surgeons and 5 medical students before and after training). These videos were evaluated by nonmedical personnel recruited through Amazon Mechanical Turk, as well as by 3 experienced surgical faculty. Evaluation criteria were based on Objective Structured Assessment of Technical Skills (scale: 5-25). Results were compared using Wilcoxon signed rank-sum and Cronbach's alpha (α).

RESULTS

Thirty-two crowd workers evaluated all 15 videos. Crowd workers scored attending surgeon videos significantly higher than pretraining medical student videos (20.5 vs 14.9, p < 0.001), demonstrating construct validity. Across all videos, crowd evaluations were more lenient than expert evaluations (19.1 vs 14.5, p < 0.001). However, average volunteer evaluations correlated more strongly with average expert evaluations (α = 0.95) than the strength of correlation between any 2 individual expert evaluators (α = 0.72-0.88). Combined reimbursement for all workers was $80.00.

CONCLUSION

After adjustments for score inflation, crowdsourced can evaluate surgical fundamentals with excellent validity. This resource is considerably less costly and potentially more reliable than individual expert evaluations.

Convergent validation and transfer of learning studies of a virtual reality-based pattern cutting simulator

INTRODUCTION

Research has clearly shown the benefits of surgical simulators to train laparoscopic motor skills required for positive patient outcomes. We have developed the Virtual Basic Laparoscopic Skill Trainer (VBLaST) that simulates tasks from the Fundamentals of Laparoscopic Surgery (FLS) curriculum. This study aims to show convergent validity of the VBLaST pattern cutting module via the CUSUM method to quantify learning curves along with motor skill transfer from simulation environments to ex vivo tissue samples.

METHODS

18 medical students at the University at Buffalo, with no prior laparoscopic surgical skills, were placed into the control, FLS training, or VBLaST training groups. Each training group performed pattern cutting trials for 12 consecutive days on their respective simulation trainers. Following a 2-week break period, the trained students performed three pattern cutting trials on each simulation platform to measure skill retention. All subjects then performed one pattern cutting task on ex vivo cadaveric peritoneal tissue. FLS and VBLaST pattern cutting scores, CUSUM scores, and transfer task completion times were reported.

RESULTS

Results indicate that the FLS and VBLaST trained groups have significantly higher task performance scores than the control group in both the VBLaST and FLS environments (p < 0.05). Learning curve results indicate that three out of seven FLS training subjects and four out of six VBLaST training subjects achieved the "senior" performance level. Furthermore, both the FLS and VBLaST trained groups had significantly lower transfer task completion times on ex vivo peritoneal tissue models (p < 0.05).

CONCLUSION

We characterized task performance scores for trained VBLaST and FLS subjects via CUSUM analysis of the learning curves and showed evidence that both groups have significant improvements in surgical motor skill. Furthermore, we showed that learned surgical skills in the FLS and VBLaST environments transfer not only to the different simulation environments, but also to ex vivo tissue models.

Video analysis in basic skills training: a way to expand the value and use of BlackBox training?

Background

Basic skills training in laparoscopic high-fidelity simulators (LHFS) improves laparoscopic skills. However, since LHFS are expensive, their availability is limited. The aim of this study was to assess whether automated video analysis of low-cost BlackBox laparoscopic training could provide an alternative to LHFS in basic skills training.

Methods

Medical students volunteered to participate during their surgical semester at the Karolinska University Hospital. After written informed consent, they performed two laparoscopic tasks (PEG-transfer and precision-cutting) on a BlackBox trainer. All tasks were videotaped and sent to MPLSC for automated video analysis, generating two parameters (Pl and Prtcl_tot) that assess the total motion activity. The students then carried out final tests on the MIST-VR simulator. This study was a European collaboration among two simulation centers, located in Sweden and Greece, within the framework of ACS-AEI.

Results

31 students (19 females and 12 males), mean age of 26.2 ± 0.8 years, participated in the study. However, since two of the students completed only one of the three MIST-VR tasks, they were excluded. The three MIST-VR scores showed significant positive correlations to both the Pl variable in the automated video analysis of the PEG-transfer (RSquare 0.48, P < 0.0001; 0.34, P = 0.0009; 0.45, P < 0.0001, respectively) as well as to the Prtcl_tot variable in that same exercise (RSquare 0.42, P = 0.0002; 0.29, P = 0.0024; 0.45, P < 0.0001). However, the correlations were exclusively shown in the group with less PC gaming experience as well as in the female group.

Conclusions

Automated video analysis provides accurate results in line with those of the validated MIST-VR. We believe that a more frequent use of automated video analysis could provide an extended value to cost-efficient laparoscopic BlackBox training. However, since there are gender-specific as well as PC gaming experience differences, this should be taken in account regarding the value of automated video analysis.

Surgical process modeling

Due to the rapidly evolving medical, technological, and technical possibilities, surgical procedures are becoming more and more complex. On the one hand, this offers an increasing number of advantages for patients, such as enhanced patient safety, minimal invasive interventions, and less medical malpractices. On the other hand, it also heightens pressure on surgeons and other clinical staff and has brought about a new policy in hospitals, which must rely on a great number of economic, social, psychological, qualitative, practical, and technological resources. As a result, medical disciplines, such as surgery, are slowly merging with technical disciplines. However, this synergy is not yet fully matured. The current information and communication technology in hospitals cannot manage the clinical and operational sequence adequately. The consequences are breaches in the surgical workflow, extensions in procedure times, and media disruptions. Furthermore, the data accrued in operating rooms (ORs) by surgeons and systems are not sufficiently implemented. A flood of information, “big data”, is available from information systems. That might be deployed in the context of Medicine 4.0 to facilitate the surgical treatment. However, it is unused due to infrastructure breaches or communication errors. Surgical process models (SPMs) alleviate these problems. They can be defined as simplified, formal, or semiformal representations of a network of surgery-related activities, reflecting a predefined subset of interest. They can employ different means of generation, languages, and data acquisition strategies. They can represent surgical interventions with high resolution, offering qualifiable and quantifiable information on the course of the intervention on the level of single, minute, surgical work-steps. The basic idea is to gather information concerning the surgical intervention and its activities, such as performance time, surgical instrument used, trajectories, movements, or intervention phases. These data can be gathered by means of workflow recordings. These recordings are abstracted to represent an individual surgical process as a model and are an essential requirement to enable Medicine 4.0 in the OR. Further abstraction can be generated by merging individual process models to form generic SPMs to increase the validity for a larger number of patients. Furthermore, these models can be applied in a wide variety of use-cases. In this regard, the term “modeling” can be used to support either one or more of the following tasks: “to describe”, “to understand”, “to explain”, to optimize”, “to learn”, “to teach”, or “to automate”. Possible use-cases are requirements analyses, evaluating surgical assist systems, generating surgeon-specific training-recommendation, creating workflow management systems for ORs, and comparing different surgical strategies. The presented chapter will give an introduction into this challenging topic, presenting different methods to generate SPMs from the workflow in the OR, as well as various use-cases, and state-of-the-art research in this field. Although many examples in the article are given according to SPMs that were computed based on observations, the same approaches can be easily applied to SPMs that were measured automatically and mined from big data.

Virtual reality-based assessment of basic laparoscopic skills using the Leap Motion controller

BACKGROUND

The majority of the current surgical simulators employ specialized sensory equipment for instrument tracking. The Leap Motion controller is a new device able to track linear objects with sub-millimeter accuracy. The aim of this study was to investigate the potential of a virtual reality (VR) simulator for assessment of basic laparoscopic skills, based on the low-cost Leap Motion controller.

METHODS

A simple interface was constructed to simulate the insertion point of the instruments into the abdominal cavity. The controller provided information about the position and orientation of the instruments. Custom tools were constructed to simulate the laparoscopic setup. Three basic VR tasks were developed: camera navigation (CN), instrument navigation (IN), and bimanual operation (BO). The experiments were carried out in two simulation centers: MPLSC (Athens, Greece) and CRESENT (Riyadh, Kingdom of Saudi Arabia). Two groups of surgeons (28 experts and 21 novices) participated in the study by performing the VR tasks. Skills assessment metrics included time, pathlength, and two task-specific errors. The face validity of the training scenarios was also investigated via a questionnaire completed by the participants.

RESULTS

Expert surgeons significantly outperformed novices in all assessment metrics for IN and BO (p < 0.05). For CN, a significant difference was found in one error metric (p < 0.05). The greatest difference between the performances of the two groups occurred for BO. Qualitative analysis of the instrument trajectory revealed that experts performed more delicate movements compared to novices. Subjects' ratings on the feedback questionnaire highlighted the training value of the system.

CONCLUSIONS

This study provides evidence regarding the potential use of the Leap Motion controller for assessment of basic laparoscopic skills. The proposed system allowed the evaluation of dexterity of the hand movements. Future work will involve comparison studies with validated simulators and development of advanced training scenarios on current Leap Motion controller.

Objective Assessment of Surgical Technical Skill and Competency in the Operating Room

Training skillful and competent surgeons is critical to ensure high quality of care and to minimize disparities in access to effective care. Traditional models to train surgeons are being challenged by rapid advances in technology, an intensified patient-safety culture, and a need for value-driven health systems. Simultaneously, technological developments are enabling capture and analysis of large amounts of complex surgical data. These developments are motivating a "surgical data science" approach to objective computer-aided technical skill evaluation (OCASE-T) for scalable, accurate assessment; individualized feedback; and automated coaching. We define the problem space for OCASE-T and summarize 45 publications representing recent research in this domain. We find that most studies on OCASE-T are simulation based; very few are in the operating room. The algorithms and validation methodologies used for OCASE-T are highly varied; there is no uniform consensus. Future research should emphasize competency assessment in the operating room, validation against patient outcomes, and effectiveness for surgical training.

Phase Segmentation Methods for an Automatic Surgical Workflow Analysis

In this paper, we present robust methods for automatically segmenting phases in a specified surgical workflow by using latent Dirichlet allocation (LDA) and hidden Markov model (HMM) approaches. More specifically, our goal is to output an appropriate phase label for each given time point of a surgical workflow in an operating room. The fundamental idea behind our work lies in constructing an HMM based on observed values obtained via an LDA topic model covering optical flow motion features of general working contexts, including medical staff, equipment, and materials. We have an awareness of such working contexts by using multiple synchronized cameras to capture the surgical workflow. Further, we validate the robustness of our methods by conducting experiments involving up to 12 phases of surgical workflows with the average length of each surgical workflow being 12.8 minutes. The maximum average accuracy achieved after applying leave-one-out cross-validation was 84.4%, which we found to be a very promising result.

Using Contact Forces and Robot Arm Accelerations to Automatically Rate Surgeon Skill at Peg Transfer

OBJECTIVE

Most trainees begin learning robotic minimally invasive surgery by performing inanimate practice tasks with clinical robots such as the Intuitive Surgical da Vinci. Expert surgeons are commonly asked to evaluate these performances using standardized five-point rating scales, but doing such ratings is time consuming, tedious, and somewhat subjective. This paper presents an automatic skill evaluation system that analyzes only the contact force with the task materials, the broad-bandwidth accelerations of the robotic instruments and camera, and the task completion time.

METHODS

We recruited N = 38 participants of varying skill in robotic surgery to perform three trials of peg transfer with a da Vinci Standard robot instrumented with our Smart Task Board. After calibration, three individuals rated these trials on five domains of the Global Evaluative Assessment of Robotic Skill (GEARS) structured assessment tool, providing ground-truth labels for regression and classification machine learning algorithms that predict GEARS scores based on the recorded force, acceleration, and time signals.

RESULTS

Both machine learning approaches produced scores on the reserved testing sets that were in good to excellent agreement with the human raters, even when the force information was not considered. Furthermore, regression predicted GEARS scores more accurately and efficiently than classification.

CONCLUSION

A surgeon's skill at robotic peg transfer can be reliably rated via regression using features gathered from force, acceleration, and time sensors external to the robot.

SIGNIFICANCE

We expect improved trainee learning as a result of providing these automatic skill ratings during inanimate task practice on a surgical robot.

Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery

Background

Effective visualization of the operative field is vital to surgical safety and education. However, additional metrics for visualization are needed to complement other common measures of surgeon proficiency, such as time or errors. Unlike other surgical modalities, robot-assisted minimally invasive surgery (RAMIS) enables data-driven feedback to trainees through measurement of camera adjustments. The purpose of this study was to validate and quantify the importance of novel camera metrics during RAMIS.

Methods

New (n = 18), intermediate (n = 8), and experienced (n = 13) surgeons completed 25 virtual reality simulation exercises on the da Vinci Surgical System. Three camera metrics were computed for all exercises and compared to conventional efficiency measures.

Results

Both camera metrics and efficiency metrics showed construct validity (p < 0.05) across most exercises (camera movement frequency 23/25, camera movement duration 22/25, camera movement interval 19/25, overall score 24/25, completion time 25/25). Camera metrics differentiated new and experienced surgeons across all tasks as well as efficiency metrics. Finally, camera metrics significantly (p < 0.05) correlated with completion time (camera movement frequency 21/25, camera movement duration 21/25, camera movement interval 20/25) and overall score (camera movement frequency 20/25, camera movement duration 19/25, camera movement interval 20/25) for most exercises.

Conclusions

We demonstrate construct validity of novel camera metrics and correlation between camera metrics and efficiency metrics across many simulation exercises. We believe camera metrics could be used to improve RAMIS proficiency-based curricula.

System events: readily accessible features for surgical phase detection

PURPOSE

Surgical phase recognition using sensor data is challenging due to high variation in patient anatomy and surgeon-specific operating styles. Segmenting surgical procedures into constituent phases is of significant utility for resident training, education, self-review, and context-aware operating room technologies. Phase annotation is a highly labor-intensive task and would benefit greatly from automated solutions.

METHODS

We propose a novel approach using system events-for example, activation of cautery tools-that are easily captured in most surgical procedures. Our method involves extracting event-based features over 90-s intervals and assigning a phase label to each interval. We explore three classification techniques: support vector machines, random forests, and temporal convolution neural networks. Each of these models independently predicts a label for each time interval. We also examine segmental inference using an approach based on the semi-Markov conditional random field, which jointly performs phase segmentation and classification. Our method is evaluated on a data set of 24 robot-assisted hysterectomy procedures.

RESULTS

Our framework is able to detect surgical phases with an accuracy of 74 % using event-based features over a set of five different phases-ligation, dissection, colpotomy, cuff closure, and background. Precision and recall values for the cuff closure (Precision: 83 %, Recall: 98 %) and dissection (Precision: 75 %, Recall: 88 %) classes were higher than other classes. The normalized Levenshtein distance between predicted and ground truth phase sequence was 25 %.

CONCLUSIONS

Our findings demonstrate that system events features are useful for automatically detecting surgical phase. Events contain phase information that cannot be obtained from motion data and that would require advanced computer vision algorithms to extract from a video. Many of these events are not specific to robotic surgery and can easily be recorded in non-robotic surgical modalities. In future work, we plan to combine information from system events, tool motion, and videos to automate phase detection in surgical procedures.

Analysis of the Structure of Surgical Activity for a Suturing and Knot-Tying Task

Background

Surgical tasks are performed in a sequence of steps, and technical skill evaluation includes assessing task flow efficiency. Our objective was to describe differences in task flow for expert and novice surgeons for a basic surgical task.

Methods

We used a hierarchical semantic vocabulary to decompose and annotate maneuvers and gestures for 135 instances of a surgeon’s knot performed by 18 surgeons. We compared counts of maneuvers and gestures, and analyzed task flow by skill level.

Results

Experts used fewer gestures to perform the task (26.29; 95% CI = 25.21 to 27.38 for experts vs. 31.30; 95% CI = 29.05 to 33.55 for novices) and made fewer errors in gestures than novices (1.00; 95% CI = 0.61 to 1.39 vs. 2.84; 95% CI = 2.3 to 3.37). Transitions among maneuvers, and among gestures within each maneuver for expert trials were more predictable than novice trials.

Conclusions

Activity segments and state flow transitions within a basic surgical task differ by surgical skill level, and can be used to provide targeted feedback to surgical trainees.

A Hands-Free Interface for Controlling Virtual Electric-Powered Wheelchairs

This paper focuses on how to provide mobility to people with motor impairments with the integration of robotics and wearable computing systems. The burden of learning to control powered mobility devices should not fall entirely on the people with disabilities. Instead, the system should be able to learn the user's movements. This requires learning the degrees of freedom of user movement, and mapping these degrees of freedom onto electric-powered wheelchair (EPW) controls. Such mapping cannot be static because in some cases users will eventually improve with practice. Our goal in this paper is to present a hands-free interface (HFI) that can be customized to the varying needs of EPW users with appropriate mapping between the users' degrees of freedom and EPW controls. EPW users with different impairment types must learn how to operate a wheelchair with their residual body motions. EPW interfaces are often customized to fit their needs. An HFI utilizes the signals generated by the user's voluntary shoulder and elbow movements and translates them into an EPW control scheme. We examine the correlation of kinematics that occur during moderately paced repetitive elbow and shoulder movements for a range of motion. The output of upper-limb movements (shoulder and elbows) was tested on six participants, and compared with an output of a precision position tracking (PPT) optical system for validation. We find strong correlations between the HFI signal counts and PPT optical system during different upper-limb movements (ranged from r = 0.86 to 0.94). We also tested the HFI performance in driving the EPW in a virtual reality environment on a spinal-cord-injured (SCI) patient. The results showed that the HFI was able to adapt and translate the residual mobility of the SCI patient into efficient control commands within a week's training. The results are encouraging for the development of more efficient HFIs, especially for wheelchair users.

Surgical robotics beyond enhanced dexterity instrumentation: a survey of machine learning techniques and their role in intelligent and autonomous surgical actions

PURPOSE

Advances in technology and computing play an increasingly important role in the evolution of modern surgical techniques and paradigms. This article reviews the current role of machine learning (ML) techniques in the context of surgery with a focus on surgical robotics (SR). Also, we provide a perspective on the future possibilities for enhancing the effectiveness of procedures by integrating ML in the operating room.

METHODS

The review is focused on ML techniques directly applied to surgery, surgical robotics, surgical training and assessment. The widespread use of ML methods in diagnosis and medical image computing is beyond the scope of the review. Searches were performed on PubMed and IEEE Explore using combinations of keywords: ML, surgery, robotics, surgical and medical robotics, skill learning, skill analysis and learning to perceive.

RESULTS

Studies making use of ML methods in the context of surgery are increasingly being reported. In particular, there is an increasing interest in using ML for developing tools to understand and model surgical skill and competence or to extract surgical workflow. Many researchers begin to integrate this understanding into the control of recent surgical robots and devices.

CONCLUSION

ML is an expanding field. It is popular as it allows efficient processing of vast amounts of data for interpreting and real-time decision making. Already widely used in imaging and diagnosis, it is believed that ML will also play an important role in surgery and interventional treatments. In particular, ML could become a game changer into the conception of cognitive surgical robots. Such robots endowed with cognitive skills would assist the surgical team also on a cognitive level, such as possibly lowering the mental load of the team. For example, ML could help extracting surgical skill, learned through demonstration by human experts, and could transfer this to robotic skills. Such intelligent surgical assistance would significantly surpass the state of the art in surgical robotics. Current devices possess no intelligence whatsoever and are merely advanced and expensive instruments.

Performance comparison of various feature detector-descriptors and temporal models for video-based assessment of laparoscopic skills

BACKGROUND

Despite the significant progress in hand gesture analysis for surgical skills assessment, video-based analysis has not received much attention. In this study we investigate the application of various feature detector-descriptors and temporal modeling techniques for laparoscopic skills assessment.

METHODS

Two different setups were designed: static and dynamic video-histogram analysis. Four well-known feature detection-extraction methods were investigated: SIFT, SURF, STAR-BRIEF and STIP-HOG. For the dynamic setup two temporal models were employed (LDS and GMMAR model). Each method was evaluated for its ability to classify experts and novices on peg transfer and knot tying.

RESULTS

STIP-HOG yielded the best performance (static: 74-79%; dynamic: 80-89%). Temporal models had equivalent performance. Important differences were found between the two groups with respect to the underlying dynamics of the video-histogram sequences.

CONCLUSIONS

Temporal modeling of feature histograms extracted from laparoscopic training videos provides information about the skill level and motion pattern of the operator. Copyright © 2015 John Wiley & Sons, Ltd.

The effect of a robot-assisted surgical system on the kinematics of user movements

Teleoperated robot-assisted surgery (RAS) offers many advantages over traditional minimally invasive surgery. However, RAS has not yet realized its full potential, and it is not clear how to optimally train surgeons to use these systems. We hypothesize that the dynamics of the master manipulator impact the ability of users to make desired movements with the robot. We compared freehand and teleoperated movements of novices and experienced surgeons. To isolate the effects of dynamics from procedural knowledge, we chose simple movements rather than surgical tasks. We found statistically significant effects of teleoperation and user expertise in several aspects of motion, including target acquisition error, movement speed, and movement smoothness. Such quantitative assessment of human motor performance in RAS can impact the design of surgical robots, their control, and surgeon training methods, and eventually, improve patient outcomes.

Towards structuring contextual information for workflow-driven surgical assistance functionalities

A workflow-driven cooperative working environment needs to be established in order to successfully unburden the surgeon and the OR staff from technical configuration and information-seeking tasks. An important prerequisite for autonomous situationaware adaptation of medical devices is a comprehensive representation of the operating context regarding the surgical process and situation.

We propose a hierarchical structuring of process-related and situation-related information entities and include assessment scores that intraoperative workflow information systems may provide via OR networks. The conducted experiments on the proposed assessment scores included sixty recorded brain tumour removal procedures and considered 344 distinguishable surgical situations.

A comprehensive modelling of surgical situations and process context will be a significant pre-requisite for reliable autonomous adaptation of medical devices and systems in digital operating rooms.

Characterizing workflow for pediatric asthma patients in emergency departments using electronic health records

OBJECTIVE

The purpose of this study was to describe a workflow analysis approach and apply it in emergency departments (EDs) using data extracted from the electronic health record (EHR) system.

MATERIALS AND METHODS

We used data that were obtained during 2013 from the ED of a children's hospital and its four satellite EDs. Workflow-related data were extracted for all patient visits with either a primary or secondary diagnosis on discharge of asthma (ICD-9 code=493). For each patient visit, eight different a priori time-stamped events were identified. Data were also collected on mode of arrival, patient demographics, triage score (i.e. acuity level), and primary/secondary diagnosis. Comparison groups were by acuity levels 2 and 3 with 2 being more acute than 3, arrival mode (ambulance versus walk-in), and site. Data were analyzed using a visualization method and Markov Chains.

RESULTS

To demonstrate the viability and benefit of the approach, patient care workflows were visually and quantitatively compared. The analysis of the EHR data allowed for exploration of workflow patterns and variation across groups. Results suggest that workflow was different for different arrival modes, settings and acuity levels.

DISCUSSION

EHRs can be used to explore workflow with statistical and visual analytics techniques novel to the health care setting. The results generated by the proposed approach could be utilized to help institutions identify workflow issues, plan for varied workflows and ultimately improve efficiency in caring for diverse patient groups.

CONCLUSION

EHR data and novel analytic techniques in health care can expand our understanding of workflow in both large and small ED units.

Crowd-Sourced Assessment of Technical Skill: A Valid Method for Discriminating Basic Robotic Surgery Skills

BACKGROUND

A surgeon's skill in the operating room has been shown to correlate with a patient's clinical outcome. The prompt accurate assessment of surgical skill remains a challenge, in part, because expert faculty reviewers are often unavailable. By harnessing the power of large readily available crowds through the Internet, rapid, accurate, and low-cost assessments may be achieved. We hypothesized that assessments provided by crowd workers highly correlate with expert surgeons' assessments.

MATERIALS AND METHODS

A group of 49 surgeons from two hospitals performed two dry-laboratory robotic surgical skill assessment tasks. The performance of these tasks was video recorded and posted online for evaluation using Amazon Mechanical Turk. The surgical tasks in each video were graded by (n=30) varying crowd workers and (n=3) experts using a modified global evaluative assessment of Robotic Skills (GEARS) grading tool, and the mean scores were compared using Cronbach's alpha statistic.

RESULTS

GEARS evaluations from the crowd were obtained for each video and task and compared with the GEARS ratings from the expert surgeons. The crowd-based performance scores agreed with the performance assessments by experts with a Cronbach's alpha of 0.84 and 0.92 for the two tasks, respectively.

CONCLUSION

The assessment of surgical skill by crowd workers resulted in a high degree of agreement with the scores provided by expert surgeons in the evaluation of basic robotic surgical dry-laboratory tasks. Crowd responses cost less and were much faster to acquire. This study provides evidence that crowds may provide an adjunctive method for rapidly providing feedback of skills to training and practicing surgeons.

A simple sensor calibration technique for estimating the 3D pose of endoscopic instruments

INTRODUCTION

The aim of this study was to describe a simple and easy-to-use calibration method that is able to estimate the pose (tip position and orientation) of a rigid endoscopic instrument with respect to an electromagnetic tracking device attached to the handle.

METHODS

A two-step calibration protocol was developed. First, the orientation of the instrument shaft is derived by performing a 360° rotation of the instrument around its shaft using a firmly positioned surgical trocar. Second, the 3D position of the instrument tip is obtained by allowing the tip to come in contact with a planar surface.

RESULTS

The results indicate submillimeter accuracy in the estimation of the tooltip position, and subdegree accuracy in the estimation of the shaft orientation, both with respect to a known reference frame. The assets of the proposed method are also highlighted by illustrating an indicative application in the field of augmented reality simulation.

CONCLUSIONS

The proposed method is simple, inexpensive, does not require employment of special calibration frames, and has potential applications not only in training systems but also in the operating room.

Robot-assisted surgery: an emerging platform for human neuroscience research

Classic studies in human sensorimotor control use simplified tasks to uncover fundamental control strategies employed by the nervous system. Such simple tasks are critical for isolating specific features of motor, sensory, or cognitive processes, and for inferring causality between these features and observed behavioral changes. However, it remains unclear how these theories translate to complex sensorimotor tasks or to natural behaviors. Part of the difficulty in performing such experiments has been the lack of appropriate tools for measuring complex motor skills in real-world contexts. Robot-assisted surgery (RAS) provides an opportunity to overcome these challenges by enabling unobtrusive measurements of user behavior. In addition, a continuum of tasks with varying complexity-from simple tasks such as those in classic studies to highly complex tasks such as a surgical procedure-can be studied using RAS platforms. Finally, RAS includes a diverse participant population of inexperienced users all the way to expert surgeons. In this perspective, we illustrate how the characteristics of RAS systems make them compelling platforms to extend many theories in human neuroscience, as well as, to develop new theories altogether.

Relative Hidden Markov Models for Video-Based Evaluation of Motion Skills in Surgical Training

A proper temporal model is essential to analysis tasks involving sequential data. In computer-assisted surgical training, which is the focus of this study, obtaining accurate temporal models is a key step towards automated skill-rating. Conventional learning approaches can have only limited success in this domain due to insufficient amount of data with accurate labels. We propose a novel formulation termed Relative Hidden Markov Model and develop algorithms for obtaining a solution under this formulation. The method requires only relative ranking between input pairs, which are readily available from training sessions in the target application, hence alleviating the requirement on data labeling. The proposed algorithm learns a model from the training data so that the attribute under consideration is linked to the likelihood of the input, hence supporting comparing new sequences. For evaluation, synthetic data are first used to assess the performance of the approach, and then we experiment with real videos from a widely-adopted surgical training platform. Experimental results suggest that the proposed approach provides a promising solution to video-based motion skill evaluation. To further illustrate the potential of generalizing the method to other applications of temporal analysis, we also report experiments on using our model on speech-based emotion recognition.

Multi-perspective workflow modeling for online surgical situation models

INTRODUCTION

Surgical workflow management is expected to enable situation-aware adaptation and intelligent systems behavior in an integrated operating room (OR). The overall aim is to unburden the surgeon and OR staff from both manual maintenance and information seeking tasks. A major step toward intelligent systems behavior is a stable classification of the surgical situation from multiple perspectives based on performed low-level tasks.

MATERIAL AND METHODS

The present work proposes a method for the classification of surgical situations based on multi-perspective workflow modeling. A model network that interconnects different types of surgical process models is described. Various aspects of a surgical situation description were considered: low-level tasks, high-level tasks, patient status, and the use of medical devices. A study with sixty neurosurgical interventions was conducted to evaluate the performance of our approach and its robustness against incomplete workflow recognition input.

RESULTS

A correct classification rate of over 90% was measured for high-level tasks and patient status. The device usage models for navigation and neurophysiology classified over 95% of the situations correctly, whereas the ultrasound usage was more difficult to predict. Overall, the classification rate decreased with an increasing level of input distortion.

DISCUSSION

Autonomous adaptation of medical devices and intelligent systems behavior do not currently depend solely on low-level tasks. Instead, they require a more general type of understanding of the surgical condition. The integration of various surgical process models in a network provided a comprehensive representation of the interventions and allowed for the generation of extensive situation descriptions.

CONCLUSION

Multi-perspective surgical workflow modeling and online situation models will be a significant pre-requisite for reliable and intelligent systems behavior. Hence, they will contribute to a cooperative OR environment.

Work domain constraints for modelling surgical performance

PURPOSE

Three main approaches can be identified for modelling surgical performance: a competency-based approach, a task-based approach, both largely explored in the literature, and a less known work domain-based approach. The work domain-based approach first describes the work domain properties that constrain the agent's actions and shape the performance. This paper presents a work domain-based approach for modelling performance during cervical spine surgery, based on the idea that anatomical structures delineate the surgical performance. This model was evaluated through an analysis of junior and senior surgeons' actions.

METHOD

Twenty-four cervical spine surgeries performed by two junior and two senior surgeons were recorded in real time by an expert surgeon. According to a work domain-based model describing an optimal progression through anatomical structures, the degree of adjustment of each surgical procedure to a statistical polynomial function was assessed.

RESULTS

Each surgical procedure showed a significant suitability with the model and regression coefficient values around 0.9. However, the surgeries performed by senior surgeons fitted this model significantly better than those performed by junior surgeons. Analysis of the relative frequencies of actions on anatomical structures showed that some specific anatomical structures discriminate senior from junior performances.

CONCLUSION

The work domain-based modelling approach can provide an overall statistical indicator of surgical performance, but in particular, it can highlight specific points of interest among anatomical structures that the surgeons dwelled on according to their level of expertise.

Idle time: an underdeveloped performance metric for assessing surgical skill

BACKGROUND

The aim of this study was to evaluate validity evidence using idle time as a performance measure in open surgical skills assessment.

METHODS

This pilot study tested psychomotor planning skills of surgical attendings (n = 6), residents (n = 4) and medical students (n = 5) during suturing tasks of varying difficulty. Performance data were collected with a motion tracking system. Participants' hand movements were analyzed for idle time, total operative time, and path length. We hypothesized that there will be shorter idle times for more experienced individuals and on the easier tasks.

RESULTS

A total of 365 idle periods were identified across all participants. Attendings had fewer idle periods during 3 specific procedure steps (P < .001). All participants had longer idle time on friable tissue (P < .005).

CONCLUSIONS

Using an experimental model, idle time was found to correlate with experience and motor planning when operating on increasingly difficult tissue types. Further work exploring idle time as a valid psychomotor measure is warranted.

Uncontrolled manifold analysis of arm joint angle variability during robotic teleoperation and freehand movement of surgeons and novices

Teleoperated robot-assisted surgery (RAS) is used to perform a wide variety of minimally invasive procedures. However, current understanding of the effect of robotic manipulation on the motor coordination of surgeons is limited. Recent studies in human motor control suggest that we optimize hand movement stability and task performance while minimizing control effort and improving robustness to unpredicted disturbances. To achieve this, the variability of joint angles and muscle activations is structured to reduce task-relevant variability and increase task-irrelevant variability. In this study, we determine whether teleoperation of a da Vinci Si surgical system in a nonclinical task of simple planar movements changes this structure of variability in experienced surgeons and novices. To answer this question, we employ the UnControlled manifold analysis that partitions users' joint angle variability into task-irrelevant and task-relevant manifolds. We show that experienced surgeons coordinate their joint angles to stabilize hand movements more than novices, and that the effect of teleoperation depends on experience--experts increase teleoperated stabilization relative to freehand whereas novices decrease it. We suggest that examining users' exploitation of the task-irrelevant manifold for stabilization of hand movements may be applied to: (1) evaluation and optimization of teleoperator design and control parameters, and (2) skill assessment and optimization of training in RAS.

Effects of robotic manipulators on movements of novices and surgeons

BACKGROUND

Robot-assisted surgery is widely adopted for many procedures but has not realized its full potential to date. Based on human motor control theories, the authors hypothesized that the dynamics of the master manipulators impose challenges on the motor system of the user and may impair performance and slow down learning. Although studies have shown that robotic outcomes are correlated with the case experience of the surgeon, the relative contribution of cognitive versus motor skill is unknown. This study quantified the effects of da Vinci Si master manipulator dynamics on movements of novice users and experienced surgeons and suggests possible implications for training and robot design.

METHODS

In the reported study, six experienced robotic surgeons and ten novice nonmedical users performed movements under two conditions: teleoperation of a da Vinci Si Surgical system and freehand. A linear mixed model was applied to nine kinematic metrics (including endpoint error, movement time, peak speed, initial jerk, and deviation from a straight line) to assess the effects of teleoperation and expertise. To assess learning effects, t tests between the first and last movements of each type were used.

RESULTS

All the users moved slower during teleoperation than during freehand movements (F(1,9343) = 345; p < 0.001). The experienced surgeons had smaller errors than the novices (F(1,14) = 36.8; p < 0.001). The straightness of movements depended on their direction (F(7,9343) = 117; p < 0.001). Learning effects were observed in all conditions. Novice users first learned the task and then the dynamics of the manipulator.

CONCLUSIONS

The findings showed differences between the novices and the experienced surgeons for extremely simple point-to-point movements. The study demonstrated that manipulator dynamics affect user movements, suggesting that these dynamics could be improved in future robot designs. The authors showed the partial adaptation of novice users to the dynamics. Future studies are needed to evaluate whether it will be beneficial to include early training sessions dedicated to learning the dynamics of the manipulator.