Eye-Tracking Metrics Predict Perceived Workload in Robotic Surgical Skills Training

Eye-Tracking in Physical Human-Robot Interaction: Mental Workload and Performance Prediction

BACKGROUND

In Physical Human-Robot Interaction (pHRI), the need to learn the robot's motor-control dynamics is associated with increased cognitive load. Eye-tracking metrics can help understand the dynamics of fluctuating mental workload over the course of learning.

OBJECTIVE

The aim of this study was to test eye-tracking measures' sensitivity and reliability to variations in task difficulty, as well as their performance-prediction capability, in physical human-robot collaboration tasks involving an industrial robot for object comanipulation.

METHODS

Participants (9M, 9F) learned to coperform a virtual pick-and-place task with a bimanual robot over multiple trials. Joint stiffness of the robot was manipulated to increase motor-coordination demands. The psychometric properties of eye-tracking measures and their ability to predict performance was investigated.

RESULTS

Stationary Gaze Entropy and pupil diameter were the most reliable and sensitive measures of workload associated with changes in task difficulty and learning. Increased task difficulty was more likely to result in a robot-monitoring strategy. Eye-tracking measures were able to predict the occurrence of success or failure in each trial with 70% sensitivity and 71% accuracy.

CONCLUSION

The sensitivity and reliability of eye-tracking measures was acceptable, although values were lower than those observed in cognitive domains. Measures of gaze behaviors indicative of visual monitoring strategies were most sensitive to task difficulty manipulations, and should be explored further for the pHRI domain where motor-control and internal-model formation will likely be strong contributors to workload.

APPLICATION

Future collaborative robots can adapt to human cognitive state and skill-level measured using eye-tracking measures of workload and visual attention.

On the impact of on-road partially-automated driving on drivers' cognitive workload and attention allocation

The use of partially-automated or SAE level-2 vehicles is expected to change the role of the human driver from operator to supervisor, which may have an effect on the driver's workload and visual attention. In this study, 30 Ontario drivers operated a vehicle in manual and partially-automated mode. Cognitive workload was measured by means of the Detection Response Task, and visual attention was measured by means of coding glances on and off the forward roadway. No difference in cognitive workload was found between driving modes. However, drivers spent less time glancing at the forward roadway, and more time glancing at the vehicle's touchscreen. These data add to our knowledge of how vehicle automation affects cognitive workload and attention allocation, and show potential safety risks associated with the adoption of partially-automated driving.

Training in robotic-assisted surgery: a systematic review of training modalities and objective and subjective assessment methods

INTRODUCTION

The variety of robotic surgery systems, training modalities, and assessment tools within robotic surgery training is extensive. This systematic review aimed to comprehensively overview different training modalities and assessment methods for teaching and assessing surgical skills in robotic surgery, with a specific focus on comparing objective and subjective assessment methods.

METHODS

A systematic review was conducted following the PRISMA guidelines. The electronic databases Pubmed, EMBASE, and Cochrane were searched from inception until February 1, 2022. Included studies consisted of robotic-assisted surgery training (e.g., box training, virtual reality training, cadaver training and animal tissue training) with an assessment method (objective or subjective), such as assessment forms, virtual reality scores, peer-to-peer feedback or time recording.

RESULTS

The search identified 1591 studies. After abstract screening and full-texts examination, 209 studies were identified that focused on robotic surgery training and included an assessment tool. The majority of the studies utilized the da Vinci Surgical System, with dry lab training being the most common approach, followed by the da Vinci Surgical Skills Simulator. The most frequently used assessment methods included simulator scoring system (e.g., dVSS score), and assessment forms (e.g., GEARS and OSATS).

CONCLUSION

This systematic review provides an overview of training modalities and assessment methods in robotic-assisted surgery. Dry lab training on the da Vinci Surgical System and training on the da Vinci Skills Simulator are the predominant approaches. However, focused training on tissue handling, manipulation, and force interaction is lacking, despite the absence of haptic feedback. Future research should focus on developing universal objective assessment and feedback methods to address these limitations as the field continues to evolve.

Comparison of virtual reality and computed tomography in the preoperative planning of complex tibial plateau fractures

INTRODUCTION

Preoperative planning is a critical step in the success of any complex surgery. The pur-pose of this study is to evaluate the advantage of VR glasses in surgical planning of complex tibial plateau fractures compared to CT planning.

MATERIALS AND METHODS

Five orthopedic surgeons performed preoperative planning for 30 fractures using either conventional CT slices or VR visualization with a VR headset. Planning was performed in a randomized order with a 3-month interval between planning sessions. A standardized questionnaire assessed planned operative time, planning time, fracture classification and understanding, and surgeons' subjective confidence in surgical planning.

RESULTS

The mean planned operative time of 156 (SD 47) minutes was significantly lower (p < 0.001) in the VR group than in the CT group (172 min; SD 44). The mean planning time in the VR group was 3.48 min (SD 2.4), 17% longer than in the CT group (2.98 min, SD 1.9; p = 0.027). Relevant parameters influencing planning time were surgeon experience (-0.61 min) and estimated complexity of fracture treatment (+ 0.65 min).

CONCLUSION

The use of virtual reality for surgical planning of complex tibial plateau fractures resulted in significantly shorter planned operative time, while planning time was longer compared to CT planning. After VR planning, more surgeons felt (very) well prepared for surgery.

A Review of Recent Advancements in Sensor-Integrated Medical Tools

A medical tool is a general instrument intended for use in the prevention, diagnosis, and treatment of diseases in humans or other animals. Nowadays, sensors are widely employed in medical tools to analyze or quantify disease-related parameters for the diagnosis and monitoring of patients' diseases. Recent explosive advancements in sensor technologies have extended the integration and application of sensors in medical tools by providing more versatile in vivo sensing capabilities. These unique sensing capabilities, especially for medical tools for surgery or medical treatment, are getting more attention owing to the rapid growth of minimally invasive surgery. In this review, recent advancements in sensor-integrated medical tools are presented, and their necessity, use, and examples are comprehensively introduced. Specifically, medical tools often utilized for medical surgery or treatment, for example, medical needles, catheters, robotic surgery, sutures, endoscopes, and tubes, are covered, and in-depth discussions about the working mechanism used for each sensor-integrated medical tool are provided.

Objective Evaluation of Gaze Location Patterns Using Eye Tracking During Cystoscopy and Artificial Intelligence-Assisted Lesion Detection

Background: The diagnostic accuracy of cystoscopy varies according to the knowledge and experience of the performing physician. In this study, we evaluated the difference in cystoscopic gaze location patterns between medical students and urologists and assessed the differences in their eye movements when simultaneously observing conventional cystoscopic images and images with lesions detected by artificial intelligence (AI). Methodology: Eye-tracking measurements were performed, and observation patterns of participants (24 medical students and 10 urologists) viewing images from routine cystoscopic videos were analyzed. The cystoscopic video was captured preoperatively in a case of initial-onset noninvasive bladder cancer with three low-lying papillary tumors in the posterior, anterior, and neck areas (urothelial carcinoma, high grade, and pTa). The viewpoint coordinates and stop times during observation were obtained using a noncontact type of gaze tracking and gaze measurement system for screen-based gaze tracking. In addition, observation patterns of medical students and urologists during parallel observation of conventional cystoscopic videos and AI-assisted lesion detection videos were compared. Results: Compared with medical students, urologists exhibited a significantly higher degree of stationary gaze entropy when viewing cystoscopic images (p < 0.05), suggesting that urologists with expertise in identifying lesions efficiently observed a broader range of bladder mucosal surfaces on the screen, presumably with the conscious intent of identifying pathologic changes. When the participants observed conventional and AI-assisted lesion detection images side by side, contrary to urologists, medical students showed a higher proportion of attention directed toward AI-detected lesion images. Conclusion: Eye-tracking measurements during cystoscopic image assessment revealed that experienced specialists efficiently observed a wide range of video screens during cystoscopy. In addition, this study revealed how lesion images detected by AI are viewed. Observation patterns of observers' gaze may have implications for assessing and improving proficiency and serving educational purposes. To the best of our knowledge, this is the first study to utilize eye tracking in cystoscopy. University of Tsukuba Hospital, clinical research reference number R02-122.

Cognitive effort detection for tele-robotic surgery via personalized pupil response modeling

PURPOSE

Gaze tracking and pupillometry are established proxies for cognitive load, giving insights into a user's mental effort. In tele-robotic surgery, knowing a user's cognitive load can inspire novel human-machine interaction designs, fostering contextual surgical assistance systems and personalized training programs. While pupillometry-based methods for estimating cognitive effort have been proposed, their application in surgery is limited by the pupil's sensitivity to brightness changes, which can mask pupil's response to cognitive load. Thus, methods considering pupil and brightness conditions are essential for detecting cognitive effort in unconstrained scenarios.

METHODS

To contend with this challenge, we introduce a personalized pupil response model integrating pupil and brightness-based features. Discrepancies between predicted and measured pupil diameter indicate dilations due to non-brightness-related sources, i.e., cognitive effort. Combined with gaze entropy, it can detect cognitive load using a random forest classifier. To test our model, we perform a user study with the da Vinci Research Kit, where 17 users perform pick-and-place tasks in addition to auditory tasks known to generate cognitive effort responses.

RESULTS

We compare our method to two baselines (BCPD and CPD), demonstrating favorable performance in varying brightness conditions. Our method achieves an average true positive rate of 0.78, outperforming the baselines (0.57 and 0.64).

CONCLUSION

We present a personalized brightness-aware model for cognitive effort detection able to operate under unconstrained brightness conditions, comparing favorably to competing approaches, contributing to the advancement of cognitive effort detection in tele-robotic surgery. Future work will consider alternative learning strategies, handling the difficult positive-unlabeled scenario in user studies, where only some positive and no negative events are reliably known.

An Adaptive Human-Robotic Interaction Architecture for Augmenting Surgery Performance Using Real-Time Workload Sensing-Demonstration of a Semi-autonomous Suction Tool

OBJECTIVE

This study developed and evaluated a mental workload-based adaptive automation (MWL-AA) that monitors surgeon cognitive load and assist during cognitively demanding tasks and assists surgeons in robotic-assisted surgery (RAS).

BACKGROUND

The introduction of RAS makes operators overwhelmed. The need for precise, continuous assessment of human mental workload (MWL) states is important to identify when the interventions should be delivered to moderate operators' MWL.

METHOD

The MWL-AA presented in this study was a semi-autonomous suction tool. The first experiment recruited ten participants to perform surgical tasks under different MWL levels. The physiological responses were captured and used to develop a real-time multi-sensing model for MWL detection. The second experiment evaluated the effectiveness of the MWL-AA, where nine brand-new surgical trainees performed the surgical task with and without the MWL-AA. Mixed effect models were used to compare task performance, objective- and subjective-measured MWL.

RESULTS

The proposed system predicted high MWL hemorrhage conditions with an accuracy of 77.9%. For the MWL-AA evaluation, the surgeons' gaze behaviors and brain activities suggested lower perceived MWL with MWL-AA than without. This was further supported by lower self-reported MWL and better task performance in the task condition with MWL-AA.

CONCLUSION

A MWL-AA systems can reduce surgeons' workload and improve performance in a high-stress hemorrhaging scenario. Findings highlight the potential of utilizing MWL-AA to enhance the collaboration between the autonomous system and surgeons. Developing a robust and personalized MWL-AA is the first step that can be used do develop additional use cases in future studies.

APPLICATION

The proposed framework can be expanded and applied to more complex environments to improve human-robot collaboration.

Cognitive ergonomics and robotic surgery

Cognitive ergonomics refer to mental resources and is associated with memory, sensory motor response, and perception. Cognitive workload (CWL) involves use of working memory (mental strain and effort) to complete a task. The three types of cognitive loads have been divided into intrinsic (dependent on complexity and expertise), extraneous (the presentation of tasks) and germane (the learning process) components. The effect of robotic surgery on CWL is complex because the postural, visualisation, and manipulation ergonomic benefits for the surgeon may be offset by the disadvantages associated with team separation and reduced situation awareness. Physical fatigue and workflow disruptions have a negative impact on CWL. Intraoperative CWL can be measured subjectively post hoc with the use of self-reported instruments or objectively with real-time physiological response metrics. Cognitive training can play a crucial role in the process of skill acquisition during the three stages of motor learning: from cognitive to integrative and then to autonomous. Mentorship, technical practice and watching videos are the most common traditional cognitive training methods in surgery. Cognitive training can also occur with computer-based cognitive simulation, mental rehearsal, and cognitive task analysis. Assessment of cognitive skills may offer a more effective way to differentiate robotic expertise level than automated performance (tool-based) metrics.

Quantifying Workload and Stress in Intensive Care Unit Nurses: Preliminary Evaluation Using Continuous Eye-Tracking

OBJECTIVE

(1) To assess mental workloads of intensive care unit (ICU) nurses in 12-hour working shifts (days and nights) using eye movement data; (2) to explore the impact of stress on the ocular metrics of nurses performing patient care in the ICU.

BACKGROUND

Prior studies have employed workload scoring systems or accelerometer data to assess ICU nurses' workload. This is the first naturalistic attempt to explore nurses' mental workload using eye movement data.

METHODS

Tobii Pro Glasses 2 eye-tracking and Empatica E4 devices were used to collect eye movement and physiological data from 15 nurses during 12-hour shifts (252 observation hours). We used mixed-effect models and an ordinal regression model with a random effect to analyze the changes in eye movement metrics during high stress episodes.

RESULTS

While the cadence and characteristics of nurse workload can vary between day shift and night shift, no significant difference in eye movement values was detected. However, eye movement metrics showed that the initial handoff period of nursing shifts has a higher mental workload compared with other times. Analysis of ocular metrics showed that stress is positively associated with an increase in number of eye fixations and gaze entropy, but negatively correlated with the duration of saccades and pupil diameter.

CONCLUSION

Eye-tracking technology can be used to assess the temporal variation of stress and associated changes with mental workload in the ICU environment. A real-time system could be developed for monitoring stress and workload for intervention development.

Development of performance and learning rate evaluation models in robot-assisted surgery using electroencephalography and eye-tracking

The existing performance evaluation methods in robot-assisted surgery (RAS) are mainly subjective, costly, and affected by shortcomings such as the inconsistency of results and dependency on the raters' opinions. The aim of this study was to develop models for an objective evaluation of performance and rate of learning RAS skills while practicing surgical simulator tasks. The electroencephalogram (EEG) and eye-tracking data were recorded from 26 subjects while performing Tubes, Suture Sponge, and Dots and Needles tasks. Performance scores were generated by the simulator program. The functional brain networks were extracted using EEG data and coherence analysis. Then these networks, along with community detection analysis, facilitated the extraction of average search information and average temporal flexibility features at 21 Brodmann areas (BA) and four band frequencies. Twelve eye-tracking features were extracted and used to develop linear random intercept models for performance evaluation and multivariate linear regression models for the evaluation of the learning rate. Results showed that subject-wise standardization of features improved the R2 of the models. Average pupil diameter and rate of saccade were associated with performance in the Tubes task (multivariate analysis; p-value = 0.01 and p-value = 0.04, respectively). Entropy of pupil diameter was associated with performance in Dots and Needles task (multivariate analysis; p-value = 0.01). Average temporal flexibility and search information in several BAs and band frequencies were associated with performance and rate of learning. The models may be used to objectify performance and learning rate evaluation in RAS once validated with a broader sample size and tasks.

Advanced workstations and collaborative robots: exploiting eye-tracking and cardiac activity indices to unveil senior workers' mental workload in assembly tasks

Introduction: As a result of Industry 5.0's technological advancements, collaborative robots (cobots) have emerged as pivotal enablers for refining manufacturing processes while re-focusing on humans. However, the successful integration of these cutting-edge tools hinges on a better understanding of human factors when interacting with such new technologies, eventually fostering workers' trust and acceptance and promoting low-fatigue work. This study thus delves into the intricate dynamics of human-cobot interactions by adopting a human-centric view. Methods: With this intent, we targeted senior workers, who often contend with diminishing work capabilities, and we explored the nexus between various human factors and task outcomes during a joint assembly operation with a cobot on an ergonomic workstation. Exploiting a dual-task manipulation to increase the task demand, we measured performance, subjective perceptions, eye-tracking indices and cardiac activity during the task. Firstly, we provided an overview of the senior workers' perceptions regarding their shared work with the cobot, by measuring technology acceptance, perceived wellbeing, work experience, and the estimated social impact of this technology in the industrial sector. Secondly, we asked whether the considered human factors varied significantly under dual-tasking, thus responding to a higher mental load while working alongside the cobot. Finally, we explored the predictive power of the collected measurements over the number of errors committed at the work task and the participants' perceived workload. Results: The present findings demonstrated how senior workers exhibited strong acceptance and positive experiences with our advanced workstation and the cobot, even under higher mental strain. Besides, their task performance suffered increased errors and duration during dual-tasking, while the eye behavior partially reflected the increased mental demand. Some interesting outcomes were also gained about the predictive power of some of the collected indices over the number of errors committed at the assembly task, even though the same did not apply to predicting perceived workload levels. Discussion: Overall, the paper discusses possible applications of these results in the 5.0 manufacturing sector, emphasizing the importance of adopting a holistic human-centered approach to understand the human-cobot complex better.

Utilizing Eye Tracking to Assess Medical Student Non-Technical Performance During Scenario-Based Simulation: Results of a Pilot Study

Background

Non-technical skills (NTS) are essential for safe surgical patient management. However, assessing NTS involves observer-based ratings, which can introduce bias. Eye tracking (ET) has been proposed as an effective method to capture NTS. The purpose of the current study was to determine if ET metrics are associated with NTS performance.

Methods

Participants wore a mobile ET system and participated in two patient care simulations, where they managed a deteriorating patient. The scenarios featured several challenges to leadership, which were evaluated using a 4-point Likert scale. NTS were evaluated by trained raters using the Non-Technical Skills for Surgeons (NOTSS) scale. ET metrics included percentage of fixations and visits on areas of interest.

Results

Ten medical students participated. Average visit duration on the patient was negatively correlated with participants' communication and leadership. Average visit duration on the patient's intravenous access was negatively correlated with participants' decision making and situation awareness.

Conclusions

Our preliminary data suggests that visual attention on the patient was negatively associated with NTS and may indicate poor comprehension of the patient's status due to heightened cognitive load. In future work, researchers and educators should consider using ET to objectively evaluate and provide feedback on their NTS.

Eye-tracking in surgery: a systematic review

BACKGROUND

Surgery is constantly evolving with the assistance of rapidly developing novel technology. Eye-tracking devices provide opportunities to monitor the acquisition of surgical skills, gain insight into performance, and enhance surgical practice. The aim of this review was to consolidate the available evidence for the use of eye-tracking in the surgical disciplines.

METHODS

A systematic literature review was conducted in accordance with PRISMA guidelines. A search of OVID Medline, EMBASE, Cochrane library, Scopus, and Science Direct was conducted January 2000 until December 2022. Studies involving eye-tracking in surgical training, assessment and technical innovation were included in the review. Non-surgical procedures, animal studies, and studies not involving surgical participants were excluded from the review.

RESULTS

The search returned a total of 12 054 articles, 80 of which were included in the final analysis and review. Seventeen studies involved eye-tracking in surgical training, 48 surgical assessment, and 20 were focussing on technical aspects of this technology. Twenty-six different eye-tracking devices were used in the included studies. Metrics such as the number of fixations, duration of fixations, dwell time, and cognitive workload were able to differentiate between novice and expert performance. Eight studies demonstrated the effectiveness of gaze-training for improving surgical skill.

CONCLUSION

The current literature shows a broad range of utility for a variety of eye-tracking devices in surgery. There remains a lack of standardization for metric parameters and gaze analysis techniques. Further research is required to validate its use to establish reliability and create uniform practices.

Machine Learning and Artificial Intelligence in Surgical Research

Machine learning, a subtype of artificial intelligence, is an emerging field of surgical research dedicated to predictive modeling. From its inception, machine learning has been of interest in medical and surgical research. Built on traditional research metrics for optimal success, avenues of research include diagnostics, prognosis, operative timing, and surgical education, in a variety of surgical subspecialties. Machine learning represents an exciting and developing future in the world of surgical research that will not only allow for more personalized and comprehensive medical care.

Eye Tracking and Motion Data Predict Endoscopic Sinus Surgery Skill

OBJECTIVE

Endoscopic surgery has a considerable learning curve due to dissociation of the visual-motor axes, coupled with decreased tactile feedback and mobility. In particular, endoscopic sinus surgery (ESS) lacks objective skill assessment metrics to provide specific feedback to trainees. This study aims to identify summary metrics from eye tracking, endoscope motion, and tool motion to objectively assess surgeons' ESS skill.

METHODS

In this cross-sectional study, expert and novice surgeons performed ESS tasks of inserting an endoscope and tool into a cadaveric nose, touching an anatomical landmark, and withdrawing the endoscope and tool out of the nose. Tool and endoscope motion were collected using an electromagnetic tracker, and eye gaze was tracked using an infrared camera. Three expert surgeons provided binary assessments of low/high skill. 20 summary statistics were calculated for eye, tool, and endoscope motion and used in logistic regression models to predict surgical skill.

RESULTS

14 metrics (10 eye gaze, 2 tool motion, and 2 endoscope motion) were significantly different between surgeons with low and high skill. Models to predict skill for 6/9 ESS tasks had an AUC >0.95. A combined model of all tasks (AUC 0.95, PPV 0.93, NPV 0.89) included metrics from eye tracking data and endoscope motion, indicating that these metrics are transferable across tasks.

CONCLUSIONS

Eye gaze, endoscope, and tool motion data can provide an objective and accurate measurement of ESS surgical performance. Incorporation of these algorithmic techniques intraoperatively could allow for automated skill assessment for trainees learning endoscopic surgery.

LEVEL OF EVIDENCE

N/A Laryngoscope, 133:500-505, 2023.

Eye tracking metrics of orthopedic surgeons with different competency levels who practice simulation-based hip arthroscopic procedures

Objective

This study aimed to investigate the feasibility of using eye tracking data to identify orthopedic trainees' technical proficiency in hip arthroscopic procedures during simulation-based training.

Design

A cross sectional study.

Setting

A simulation-based training session for hip arthroscopy was conducted. Eye tracking devices were used to record participants' eye movements while performing simulated operations. The NASA Task Load Index survey was then used to measure subjective opinions on the perceived workload of the training. Statistical analyses were performed to determine the significance of the eye metrics and survey data.

Participants

A total of 12 arthroscopic trainees, including resident doctors, junior specialist surgeons, and consultant surgeons from the Department of Orthopedics in five hospitals, participated in this study. They were divided into three subgroups based on their prior clinical experience.

Results

Significant differences, including those for dwell time, number of fixations, number of saccades, saccade duration, peak velocity of the saccade, and pupil entropy, were observed among the three subgroups. Additionally, there were clear trends in the perceived workload of the simulation-based training based on feedback from the participants.

Conclusion

Based on this preliminary study, a correlation was identified between the eye tracking metrics and participants' experience levels. Hence, it is feasible to apply eye tracking data as a supplementary objective assessment tool to benchmark the technical proficiency of surgical trainees in hip arthroscopy, and enhance simulation-based training.

The Measurement of Cognitive Workload in Surgery Using Pupil Metrics: A Systematic Review and Narrative Analysis

INTRODUCTION

Increased cognitive workload (CWL) is a well-established entity that can impair surgical performance and increase the likelihood of surgical error. The use of pupil and gaze tracking data is increasingly being used to measure CWL objectively in surgery. The aim of this review is to summarize and synthesize the existing evidence that surrounds this.

METHODS

A systematic review was undertaken in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. A search of OVID MEDLINE, IEEE Xplore, Web of Science, Google Scholar, APA PsychINFO, and EMBASE was conducted for articles published in English between 1990 and January 2021. In total, 6791 articles were screened and 32 full-text articles were selected based on the inclusion criteria. A narrative analysis was undertaken in view of the heterogeneity of studies.

RESULTS

Seventy-eight percent of selected studies were deemed high quality. The most frequent surgical environment and task studied was surgical simulation (75%) and performance of laparoscopic skills (56%) respectively. The results demonstrated that the current literature can be broadly categorized into pupil, blink, and gaze metrics used in the assessment of CWL. These can be further categorized according to their use in the context of CWL: (1) direct measurement of CWL (n = 16), (2) determination of expertise level (n = 14), and (3) predictors of performance (n = 2).

CONCLUSIONS

Eye-tracking data provide a wealth of information; however, there is marked study heterogeneity. Pupil diameter and gaze entropy demonstrate promise in CWL assessment. Future work will entail the use of artificial intelligence in the form of deep learning and the use of a multisensor platform to accurately measure CWL.

Reappraisal of telesurgery in the era of high-speed, high-bandwidth, secure communications: Evaluation of surgical performance in local and remote environments

Aim

Communication and video transmission delays negatively affect telerobotic surgery. Since latency varies by communication environment and robot, to realize remote surgery, both must perform well. This study aims to examine the feasibility of telerobotic surgery by validating the communication environment and local/remote robot operation, using secure commercial lines and newly developed robots.

Methods

Hirosaki University and Mutsu General Hospital, 150 km apart, were connected via a Medicaroid surgical robot. Ten surgeons performed a simple task remotely using information encoding and decoding. The required bandwidth, delay time, task completion time, number of errors, and image quality were evaluated. Next, 11 surgeons performed a complex task using gallbladder and intestinal models in local/remote environments; round trip time (RTT), packet loss, time to completion, operator fatigue, operability, and image were observed locally and remotely.

Results

Image quality was not so degraded as to affect remote robot operation. Median RTT was 4 msec (2-12), and added delay was 29 msec. There was no significant difference in accuracy or number of errors for cholecystectomy, intestinal suturing, completion time, surgeon fatigue, or image evaluation.

Conclusion

The fact that remote surgery succeeded equally to local surgery showed that this system has the necessary elemental technology for widespread social implementation.

Eye Tracking Use in Surgical Research: A Systematic Review

INTRODUCTION

Eye tracking (ET) is a popular tool to study what factors affect the visual behaviour of surgical team members. To our knowledge, there have been no reviews to date that evaluate the broad use of ET in surgical research. This review aims to identify and assess the quality of this evidence, to synthesize how ET can be used to inform surgical practice, and to provide recommendations to improve future ET surgical studies.

METHODS

In line with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines, a systematic literature review was conducted. An electronic search was performed in MEDLINE, Cochrane Central, Embase, and Web of Science databases up to September 2020. Included studies used ET to measure the visual behaviour of members of the surgical team during surgery or surgical tasks. The included studies were assessed by two independent reviewers.

RESULTS

A total of 7614 studies were identified, and 111 were included for data extraction. Eleven applications were identified; the four most common were skill assessment (41%), visual attention assessment (22%), workload measurement (17%), and skills training (10%). A summary was provided of the various ways ET could be used to inform surgical practice, and three areas were identified for the improvement of future ET studies in surgery.

CONCLUSIONS

This review provided a comprehensive summary of the various applications of ET in surgery and how ET could be used to inform surgical practice, including how to use ET to improve surgical education. The information provided in this review can also aid in the design and conduct of future ET surgical studies.

A Perspective Review on Integrating VR/AR with Haptics into STEM Education for Multi-Sensory Learning

As a result of several governments closing educational facilities in reaction to the COVID-19 pandemic in 2020, almost 80% of the world’s students were not in school for several weeks. Schools and universities are thus increasing their efforts to leverage educational resources and provide possibilities for remote learning. A variety of educational programs, platforms, and technologies are now accessible to support student learning; while these tools are important for society, they are primarily concerned with the dissemination of theoretical material. There is a lack of support for hands-on laboratory work and practical experience. This is particularly important for all disciplines related to science, technology, engineering, and mathematics (STEM), where labs and pedagogical assets must be continuously enhanced in order to provide effective study programs. In this study, we describe a unique perspective to achieving multi-sensory learning through the integration of virtual and augmented reality (VR/AR) with haptic wearables in STEM education. We address the implications of a novel viewpoint on established pedagogical notions. We want to encourage worldwide efforts to make fully immersive, open, and remote laboratory learning a reality.

When medical trainees encountering a performance difficulty: evidence from pupillary responses

BACKGROUND

Medical trainees are required to learn many procedures following instructions to improve their skills. This study aims to investigate the pupillary response of trainees when they encounter moment of performance difficulty (MPD) during skill learning. Detecting the moment of performance difficulty is essential for educators to assist trainees when they need it.

METHODS

Eye motions were recorded while trainees practiced the thoracostomy procedure in the simulation model. To make pupillary data comparable among trainees, we proposed the adjusted pupil size (APS) normalizing pupil dilation for each trainee in their entire procedure. APS variables including APS, maxAPS, minAPS, meanAPS, medianAPS, and max interval indices were compared between easy and difficult subtasks; the APSs were compared among the three different performance situations, the moment of normal performance (MNP), MPD, and moment of seeking help (MSH).

RESULTS

The mixed ANOVA revealed that the adjusted pupil size variables, such as the maxAPS, the minAPS, the meanAPS, and the medianAPS, had significant differences between performance situations. Compared to MPD and MNP, pupil size was reduced during MSH. Trainees displayed a smaller accumulative frequency of APS during difficult subtask when compared to easy subtasks.

CONCLUSIONS

Results from this project suggest that pupil responses can be a good behavioral indicator. This study is a part of our research aiming to create an artificial intelligent system for medical trainees with automatic detection of their performance difficulty and delivering instructional messages using augmented reality technology.

Artificial Intelligence and Surgical Education: A Systematic Scoping Review of Interventions

OBJECTIVE

To synthesize peer-reviewed evidence related to the use of artificial intelligence (AI) in surgical education DESIGN: We conducted and reported a scoping review according to the standards outlined in the Preferred Reporting Items for Systematic Reviews and Meta-Analysis with extension for Scoping Reviews guideline and the fourth edition of the Joanna Briggs Institute Reviewer's Manual. We systematically searched eight interdisciplinary databases including MEDLINE-Ovid, ERIC, EMBASE, CINAHL, Web of Science: Core Collection, Compendex, Scopus, and IEEE Xplore. Databases were searched from inception until the date of search on April 13, 2021.

SETTING/PARTICIPANTS

We only examined original, peer-reviewed interventional studies that self-described as AI interventions, focused on medical education, and were relevant to surgical trainees (defined as medical or dental students, postgraduate residents, or surgical fellows) within the title and abstract (see Table 2). Animal, cadaveric, and in vivo studies were not eligible for inclusion.

RESULTS

After systematically searching eight databases and 4255 citations, our scoping review identified 49 studies relevant to artificial intelligence in surgical education. We found diverse interventions related to the evaluation of surgical competency, personalization of surgical education, and improvement of surgical education materials across surgical specialties. Many studies used existing surgical education materials, such as the Objective Structured Assessment of Technical Skills framework or the JHU-ISI Gesture and Skill Assessment Working Set database. Though most studies did not provide outcomes related to the implementation in medical schools (such as cost-effective analyses or trainee feedback), there are numerous promising interventions. In particular, many studies noted high accuracy in the objective characterization of surgical skill sets. These interventions could be further used to identify at-risk surgical trainees or evaluate teaching methods.

CONCLUSIONS

There are promising applications for AI in surgical education, particularly for the assessment of surgical competencies, though further evidence is needed regarding implementation and applicability.

Objective Measures of Surgeon Non-Technical Skills in Surgery: A Scoping Review

OBJECTIVE

The purpose of this study was to identify, synthesize, and discuss objective behavioral or physiological metrics of surgeons' nontechnical skills (NTS) in the literature.

BACKGROUND

NTS, or interpersonal or cognitive skills, have been identified to contribute to safe and efficient surgical performance; however, current assessments are subjective, checklist-based tools. Intraoperative skill evaluation, such as technical skills, has been previously utilized as an objective measure to address such limitations.

METHODS

Five databases in engineering, behavioral science, and medicine were searched following PRISMA reporting guidelines. Eligibility criteria included studies with NTS objective measurements, surgeons, and took place within simulated or live operations.

RESULTS

Twenty-three articles were included in this review. Objective metrics included communication metrics and measures from physiological responses such as changes in brain activation and motion of the eye. Frequencies of content-coded communication in surgery were utilized in 16 studies and were associated with not only the communication construct but also cognitive constructs of situation awareness and decision making. This indicates the underlying importance of communication in evaluating the NTS constructs. To synthesize the scoped literature, a framework based on the one-way communication model was used to map the objective measures to NTS constructs.

CONCLUSION

Objective NTS measurement of surgeons is still preliminary, and future work on leveraging objective metrics in parallel with current assessment tools is needed.

APPLICATION

Findings from this work identify objective NTS metrics for measurement applications in a surgical environment.

Eye-Tracking Indicators of Workload in Surgery: A Systematic Review

BackgroundEye tracking is a powerful tool for unobtrusive and real time assessment of workload in clinical settings. Before the complex eye tracking derived surrogates can be proactively utilized to improve surgical safety, the indications, validity and reliability requires careful evaluation.MethodsWe conducted a systematic review of literature from 2010 to 2020 according to PRISMA guidelines. A search on PubMed, Cochrane, Scopus, Web of science, PsycInfo and Google scholar databases was conducted on July 2020. The following search query was used" ("eye tracking" OR "gaze tracking") AND (surgery OR surgical OR operative OR intraoperative) AND (workload OR stress)". Short papers, no peer reviewed or papers in which eye-tracking methodology was not used to investigate workload or stress factors in surgery, were omitted.ResultsA total of 17 (N = 17) studies were identified eligible to this review. Most of the studies (n = 15) measured workload in simulated setting. Task difficulty and expertise were the most studied factors. Studies consistently showed surgeon's eye movements such as pupil responses, gaze patterns, blinks were associated with the level of perceived workload. However, differences between measurements in operational room and simulated environments have been found.ConclusionPupil responses, blink rate and gaze indices are valid indicators of workload. However, the effect of distractions and non-technical factors on workload is underrepresented aspect in the literature even though recognized as underlying factors in successful surgery.

Exploring the roles of artificial intelligence in surgical education: A scoping review

BACKGROUND

Technology-enhanced teaching and learning, including Artificial Intelligence (AI) applications, has started to evolve in surgical education. Hence, the purpose of this scoping review is to explore the current and future roles of AI in surgical education.

METHODS

Nine bibliographic databases were searched from January 2010 to January 2021. Full-text articles were included if they focused on AI in surgical education.

RESULTS

Out of 14,008 unique sources of evidence, 93 were included. Out of 93, 84 were conducted in the simulation setting, and 89 targeted technical skills. Fifty-six studies focused on skills assessment/classification, and 36 used multiple AI techniques. Also, increasing sample size, having balanced data, and using AI to provide feedback were major future directions mentioned by authors.

CONCLUSIONS

AI can help optimize the education of trainees and our results can help educators and researchers identify areas that need further investigation.

A meta-analytic review of the reliability of the Index of Cognitive Activity concerning task-evoked cognitive workload and light influences

The Index of Cognitive Activity (ICA) was introduced as a promising pupillary workload measure for field investigations since, unlike pupil dilation, it is not affected by illumination. Recent studies have investigated the ICA for task-evoked cognitive workload with contradictory findings. However, few studies investigated the influence of illumination on the ICA. Therefore, to examine inconsistencies regarding the reliability for workload measurement and the effects of light, a meta-analysis was conducted based on a structured literature review. The meta-analysis considered k = 14 studies with a total sample size of N = 751 participants. Results showed significant effects for workload (r = 0.61) and light (r = 0.45) on the ICA. Since moderating effects were found for several between-study differences, it seems likely that different cognitive processes and settings affect the indicator and should be considered in empirical investigations. According to the findings, the ICA is a reliable indicator for task-evoked workload. However, light influences were found which indicates that evidence-based conclusions regarding the ICA's practical applicability require further research.

Evolving robotic surgery training and improving patient safety, with the integration of novel technologies

INTRODUCTION

Robot-assisted surgery is becoming increasingly adopted by multiple surgical specialties. There is evidence of inherent risks of utilising new technologies that are unfamiliar early in the learning curve. The development of standardised and validated training programmes is crucial to deliver safe introduction. In this review, we aim to evaluate the current evidence and opportunities to integrate novel technologies into modern digitalised robotic training curricula.

METHODS

A systematic literature review of the current evidence for novel technologies in surgical training was conducted online and relevant publications and information were identified. Evaluation was made on how these technologies could further enable digitalisation of training.

RESULTS

Overall, the quality of available studies was found to be low with current available evidence consisting largely of expert opinion, consensus statements and small qualitative studies. The review identified that there are several novel technologies already being utilised in robotic surgery training. There is also a trend towards standardised validated robotic training curricula. Currently, the majority of the validated curricula do not incorporate novel technologies and training is delivered with more traditional methods that includes centralisation of training services with wet laboratories that have access to cadavers and dedicated training robots.

CONCLUSIONS

Improvements to training standards and understanding performance data have good potential to significantly lower complications in patients. Digitalisation automates data collection and brings data together for analysis. Machine learning has potential to develop automated performance feedback for trainees. Digitalised training aims to build on the current gold standards and to further improve the 'continuum of training' by integrating PBP training, 3D-printed models, telementoring, telemetry and machine learning.

Visuomotor control dynamics of quiet standing under single and dual task conditions in younger and older adults

Visual input facilitates stable postural control; however, ageing alters visual gaze strategies and visual input processing times. Understanding the complex interaction between visual gaze behaviour and the effects of age may inform future interventions to improve postural control in older adults. The purpose of this study was to determine effects of age and dual task on gaze and postural sway dynamics, and the sway-gaze complexity coupling to explore the coupling between sensory input and motor output. Ten older and 10 younger adults performed single and dual task quiet standing while gaze behaviour and centre of mass motion were recorded. The complexity and stability of postural sway, saccade characteristics, visual input duration and complexity of gaze were calculated in addition to sway-gaze coupling quantified by cross-sample entropy. Dual tasking increased complexity and decreased stability of sway with increased gaze complexity and visual input duration, suggesting greater automaticity of sway with greater exploration of the visual field but with longer visual inputs to maintain postural stability in dual task conditions. In addition, older adults had lower complexity and stability of sway than younger adults indicating less automated and stable postural control. Older adults also demonstrated lower gaze complexity, longer visual input durations and greater sway-gaze coupling. These findings suggest older adults adopted a strategy to increase the capacity for visual information input, whilst exploring less of the visual field than younger adults.

Non-Technical Skill Assessment and Mental Load Evaluation in Robot-Assisted Minimally Invasive Surgery

BACKGROUND: Sensor technologies and data collection practices are changing and improving quality metrics across various domains. Surgical skill assessment in Robot-Assisted Minimally Invasive Surgery (RAMIS) is essential for training and quality assurance. The mental workload on the surgeon (such as time criticality, task complexity, distractions) and non-technical surgical skills (including situational awareness, decision making, stress resilience, communication, leadership) may directly influence the clinical outcome of the surgery. METHODS: A literature search in PubMed, Scopus and PsycNet databases was conducted for relevant scientific publications. The standard PRISMA method was followed to filter the search results, including non-technical skill assessment and mental/cognitive load and workload estimation in RAMIS. Publications related to traditional manual Minimally Invasive Surgery were excluded, and also the usability studies on the surgical tools were not assessed. RESULTS: 50 relevant publications were identified for non-technical skill assessment and mental load and workload estimation in the domain of RAMIS. The identified assessment techniques ranged from self-rating questionnaires and expert ratings to autonomous techniques, citing their most important benefits and disadvantages. CONCLUSIONS: Despite the systematic research, only a limited number of articles was found, indicating that non-technical skill and mental load assessment in RAMIS is not a well-studied area. Workload assessment and soft skill measurement do not constitute part of the regular clinical training and practice yet. Meanwhile, the importance of the research domain is clear based on the publicly available surgical error statistics. Questionnaires and expert-rating techniques are widely employed in traditional surgical skill assessment; nevertheless, recent technological development in sensors and Internet of Things-type devices show that skill assessment approaches in RAMIS can be much more profound employing automated solutions. Measurements and especially big data type analysis may introduce more objectivity and transparency to this critical domain as well. SIGNIFICANCE: Non-technical skill assessment and mental load evaluation in Robot-Assisted Minimally Invasive Surgery is not a well-studied area yet; while the importance of this domain from the clinical outcome's point of view is clearly indicated by the available surgical error statistics.

Modeling patient-related workload in the emergency department using electronic health record data

INTRODUCTION

Understanding and managing clinician workload is important for clinician (nurses, physicians and advanced practice providers) occupational health as well as patient safety. Efforts have been made to develop strategies for managing clinician workload by improving patient assignment. The goal of the current study is to use electronic health record (EHR) data to predict the amount of work that individual patients contribute to clinician workload (patient-related workload).

METHODS

One month of EHR data was retrieved from an emergency department (ED). A list of workload indicators and five potential workload proxies were extracted from the data. Linear regression and four machine learning classification algorithms were utilized to model the relationship between the indicators and the proxies.

RESULTS

Linear regression proved that the indicators explained a substantial amount of variance of the proxies (four out of five proxies were modeled with R² > 0.80). Classification algorithms also showed success in classifying a patient as having high or low task demand based on data from early in the ED visit (e.g. 80 % accurate binary classification with data from the first hour).

CONCLUSION

The main contribution of this study is demonstrating the potential of using EHR data to predict patient-related workload automatically in the ED. The predicted workload can potentially help in managing clinician workload by supporting decisions around the assignment of new patients to providers. Future work should focus on identifying the relationship between workload proxies and actual workload, as well as improving prediction performance of regression and multi-class classification.

Sensor-based indicators of performance changes between sessions during robotic surgery training

Training of surgeons is essential for safe and effective use of robotic surgery, yet current assessment tools for learning progression are limited. The objective of this study was to measure changes in trainees' cognitive and behavioral states as they progressed in a robotic surgeon training curriculum at a medical institution. Seven surgical trainees in urology who had no formal robotic training experience participated in the simulation curriculum. They performed 12 robotic skills exercises with varying levels of difficulty repetitively in separate sessions. EEG (electroencephalogram) activity and eye movements were measured throughout to calculate three metrics: engagement index (indicator of task engagement), pupil diameter (indicator of mental workload) and gaze entropy (indicator of randomness in gaze pattern). Performance scores (completion of task goals) and mental workload ratings (NASA-Task Load Index) were collected after each exercise. Changes in performance scores between training sessions were calculated. Analysis of variance, repeated measures correlation, and machine learning classification were used to diagnose how cognitive and behavioral states associate with performance increases or decreases between sessions. The changes in performance were correlated with changes in engagement index (rrm=-.25,p<.001) and gaze entropy (rrm=-.37,p<.001). Changes in cognitive and behavioral states were able to predict training outcomes with 72.5% accuracy. Findings suggest that cognitive and behavioral metrics correlate with changes in performance between sessions. These measures can complement current feedback tools used by medical educators and learners for skills assessment in robotic surgery training.

Identifying mental health status using deep neural network trained by visual metrics

Mental health is an integral part of the quality of life of cancer patients. It has been found that mental health issues, such as depression and anxiety, are more common in cancer patients. They may result in catastrophic consequences, including suicide. Therefore, monitoring mental health metrics (such as hope, anxiety, and mental well-being) is recommended. Currently, there is lack of objective method for mental health evaluation, and most of the available methods are limited to subjective face-to-face discussions between the patient and psychotherapist. In this study we introduced an objective method for mental health evaluation using a combination of convolutional neural network and long short-term memory (CNN-LSTM) algorithms learned and validated by visual metrics time-series. Data were recorded by the TobiiPro eyeglasses from 16 patients with cancer after major oncologic surgery and nine individuals without cancer while viewing18 artworks in an in-house art gallery. Pre-study and post-study questionnaires of Herth Hope Index (HHI; for evaluation of hope), anxiety State-Trait Anxiety Inventory for Adults (STAI; for evaluation of anxiety) and Warwick-Edinburgh Mental Wellbeing Scale (WEMWBS; for evaluation of mental well-being) were completed by participants. Clinical psychotherapy and statistical suggestions for cutoff scores were used to assign an individual's mental health metrics level during each session into low (class 0), intermediate (class 1), and high (class 2) levels. Our proposed model was used to objectify evaluation and categorize HHI, STAI, and WEMWBS status of individuals. Classification accuracy of the model was 93.81%, 94.76%, and 95.00% for HHI, STAI, and WEMWBS metrics, respectively. The proposed model can be integrated into applications for home-based mental health monitoring to be used by patients after oncologic surgery to identify patients at risk.