Neural insights on expert surgeons' mental workload during live robotic surgeries

Despite its adoption and benefits, robotic surgeries can impose additional mental workload on surgeons. Validated questionnaires mostly administered at the end of procedures may not accurately capture the dynamic nature of mental workload over an entire procedure. Hence, we sought to determine if electroencephalogram (EEG) based neural activities in different brain regions can measure variations in expert surgeons' mental workload intraoperatively. EEG data was collected from five different surgeons performing 13 robotic-assisted urological procedures. Data analysis focused on three surgery phases (before, critical, and after). After performing each phase, surgeons provided a rating of their perceived mental workload. A linear mixed effects model was applied to explore the impact of the study phases on the relative spectral band power of EEG signals. The relative theta band power in the frontal brain region was highest during the critical portions of the procedure (p < 0.05). As the subjective ratings increased, the relative frontal theta band power increased (p < 0.001) while the relative parietal alpha band power decreased across all phases. We show that EEG signals can distinguish intraoperative workload in robotic surgeries. This has several applications including predicting risk factors for increased case complexity and surgical education.

Evaluating robotic actions: spatiotemporal brain dynamics of performance assessment in robot-assisted laparoscopic training

Introduction

Enhancing medical robot training traditionally relies on explicit feedback from physicians to identify optimal and suboptimal robotic actions during surgery. Passive brain-computer interfaces (BCIs) offer an emerging alternative by enabling implicit brain-based performance evaluations. However, effectively decoding these evaluations of robot performance requires a comprehensive understanding of the spatiotemporal brain dynamics identifying optimal and suboptimal robot actions within realistic settings.

Methods

We conducted an electroencephalographic study with 16 participants who mentally assessed the quality of robotic actions while observing simulated robot-assisted laparoscopic surgery scenarios designed to approximate real-world conditions. We aimed to identify key spatiotemporal dynamics using the surface Laplacian technique and two complementary data-driven methods: a mass-univariate permutation-based clustering and multivariate pattern analysis (MVPA)-based temporal decoding. A second goal was to identify the optimal time interval of evoked brain signatures for single-trial classification.

Results

Our analyses revealed three distinct spatiotemporal brain dynamics differentiating the quality assessment of optimal vs. suboptimal robotic actions during video-based laparoscopic training observations. Specifically, an enhanced left fronto-temporal current source, consistent with P300, LPP, and P600 components, indicated heightened attentional allocation and sustained evaluation processes during suboptimal robot actions. Additionally, amplified current sinks in right frontal and mid-occipito-parietal regions suggested prediction-based processing and conflict detection, consistent with the oERN and interaction-based ERN/N400. Both mass-univariate clustering and MVPA provided convergent evidence supporting these neural distinctions.

Discussion

The identified neural signatures propose that suboptimal robotic actions elicit enhanced, sustained brain dynamics linked to continuous attention allocation, action monitoring, conflict detection, and ongoing evaluative processing. The findings highlight the importance of prioritizing late evaluative brain signatures in BCIs to classify robotic actions reliably. These insights have significant implications for advancing machine-learning-based training paradigms.

Using passive BCI for personalization of assistive wearable devices: a proof-of-concept study

Assistive wearable devices can significantly enhance the quality of life for individuals with movement impairments, aid the rehabilitation process, and augment movement abilities of healthy users. However, personalizing the assistance to individual preferences and needs remains a challenge. Brain-Computer Interface (BCI) offers a promising solution for this personalization problem. The overarching goal of this study is to investigate the feasibility of utilizing passive BCI technology to personalize the assistance provided by a knee exoskeleton. Participants performed seated knee flexion-extension tasks while wearing a one-degree-of-freedom knee exoskeleton with varying levels of applied force. Their brain activities were recorded throughout the movements using electroencephalography (EEG). EEG spectral bands from several brain regions were compared between the conditions with the lowest and highest exoskeleton forces to identify statistically significant changes. A Naive Bayes classifier was trained on these spectral features to distinguish between the two conditions. Statistical analysis revealed significant increases in δ and θ activity and decreases in α and β activity in the frontal, motor, and occipital cortices. These changes suggest heightened attention, concentration, and motor engagement when the task became more difficult. The trained Naive Bayes classifier achieved an average accuracy of approximately 72% in distinguishing between the two conditions. The outcomes of our study demonstrate the potential of passive BCI in personalizing assistance provided by wearable devices. Future research should further explore integrating passive BCI into assistive wearable devices to enhance user experience.

Using mobile EEG to study auditory work strain during simulated surgical procedures

Surgical personnel face various stressors in the workplace, including environmental sounds. Mobile electroencephalography (EEG) offers a promising approach for objectively measuring how individuals perceive sounds. Because surgical performance does not necessarily decrease with higher levels of distraction, EEG could help guide noise reduction strategies that are independent of performance measures. In this study, we utilized mobile EEG to explore how a realistic soundscape is perceived during simulated laparoscopic surgery. To examine the varying demands placed on personnel in different situations, we manipulated the cognitive demand during the surgical task, using a memory task. To assess responses to the soundscape, we calculated event-related potentials for distinct sound events and temporal response functions for the ongoing soundscape. Although participants reported varying degrees of demand under different conditions, no significant effects were observed on surgical task performance or EEG parameters. However, changes in surgical task performance and EEG parameters over time were noted, while subjective results remained consistent over time. These findings highlight the importance of using multiple measures to fully understand the complex relationship between sound processing and cognitive demand. Furthermore, in the context of combined EEG and audio recordings in real-life scenarios, a sparse representation of the soundscape has the advantage that it can be recorded in a data-protected way compared to more detailed representations. However, it is unclear whether information get lost with sparse representations. Our results indicate that sparse and detailed representations are equally effective in eliciting neural responses. Overall, this study marks a significant step towards objectively investigating sound processing in applied settings.

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human-robot collaboration

Objective. Brain-computer interface (BCI) technology is poised to play a prominent role in modern work environments, especially a collaborative environment where humans and machines work in close proximity, often with physical contact. In a physical human robot collaboration (pHRC), the robot performs complex motion sequences. Any unexpected robot behavior or faulty interaction might raise safety concerns. Error-related potentials, naturally generated by the brain when a human partner perceives an error, have been extensively employed in BCI as implicit human feedback to adapt robot behavior to facilitate a safe and intuitive interaction. However, the integration of BCI technology with error-related potential for robot control demands failure-free integration of highly uncertain electroencephalography (EEG) signals, particularly influenced by the physical and cognitive state of the user. As a higher workload on the user compromises their access to cognitive resources needed for error awareness, it is crucial to study how mental workload variations impact the error awareness as it might raise safety concerns in pHRC. In this study, we aim to study how cognitive workload affects the error awareness of a human user engaged in a pHRC.Approach. We designed a blasting task with an abrasive industrial robot and manipulated the mental workload with a secondary arithmetic task of varying difficulty. EEG data, perceived workload, task and physical performance were recorded from 24 participants moving the robot arm. The error condition was achieved by the unexpected stopping of the robot in 33% of trials.Main results. We observed a diminished amplitude for the prediction error negativity (PEN) and error positivity (Pe), indicating reduced error awareness with increasing mental workload. We further observed an increased frontal theta power and increasing trend in the central alpha and central beta power after the unexpected robot stopping compared to when the robot stopped correctly at the target. We also demonstrate that a popular convolution neural network model, EEGNet, could predict the amplitudes of PEN and Pe from the EEG data prior to the error.Significance. This prediction model could be instrumental in developing an online prediction model that could forewarn the system and operators of the diminished error awareness of the user, alluding to a potential safety breach in error-related potential-based BCI system for pHRC. Therefore, our work paves the way for embracing BCI technology in pHRC to optimally adapt the robot behavior for personalized user experience using real-time brain activity, enriching the quality of the interaction.

Humans and robots: Friends of the future? A bird's eye view of biomanufacturing industry 5.0

The evolution of industries have introduced versatile technologies, motivating limitless possibilities of tackling pivotal global predicaments in the arenas of medicine, environment, defence, and national security. In this direction, ardently emerges the new era of Industry 5.0 through the eyes of biomanufacturing, which integrates the most advanced systems 21st century has to offer by means of integrating artificial systems to mimic and nativize the natural milieu to substitute the deficits of nature, thence leading to a new meta world. Albeit, it questions the natural order of the living world, which necessitates certain paramount stipulations to be addressed for a successful expansion of biomanufacturing Industry 5.0. Can humans live in synergism with artificial beings? How can humans establish dominance of hierarchy with artificial counterparts? This perspective provides a bird's eye view on the plausible direction of a new meta world inquisitively. For this purpose, we propose the influence of internet of things (IoT) via new generation interfacial systems, such as, human-machine interface (HMI) and brain-computer interface (BCI) in the domain of tissue engineering and regenerative medicine, which can be extended to target modern warfare and smart healthcare.

The neurophysiology of intraoperative error: An EEG study of trainee surgeons during robotic-assisted surgery simulations

Surgeons operate in mentally and physically demanding workspaces where the impact of error is highly consequential. Accurately characterizing the neurophysiology of surgeons during intraoperative error will help guide more accurate performance assessment and precision training for surgeons and other teleoperators. To better understand the neurophysiology of intraoperative error, we build and deploy a system for intraoperative error detection and electroencephalography (EEG) signal synchronization during robot-assisted surgery (RAS). We then examine the association between EEG data and detected errors. Our results suggest that there are significant EEG changes during intraoperative error that are detectable irrespective of surgical experience level.

Electroencephalography can provide advance warning of technical errors during laparoscopic surgery

BACKGROUND

Intraoperative adverse events lead to patient injury and death, and are increasing. Early warning systems (EWSs) have been used to detect patient deterioration and save lives. However, few studies have used EWSs to monitor surgical performance and caution about imminent technical errors. Previous (non-surgical) research has investigated neural activity to predict future motor errors using electroencephalography (EEG). The present proof-of-concept cohort study investigates whether EEG could predict technical errors in surgery.

METHODS

In a large academic hospital, three surgical fellows performed 12 elective laparoscopic general surgeries. Audiovisual data of the operating room and the surgeon's neural activity were recorded. Technical errors and epochs of good surgical performance were coded into events. Neural activity was observed 40 s prior and 10 s after errors and good events to determine how far in advance errors were detected. A hierarchical regression model was used to account for possible clustering within surgeons. This prospective, proof-of-concept, cohort study was conducted from July to November 2021, with a pilot period from February to March 2020 used to optimize the technique of data capture and included participants who were blinded from study hypotheses.

RESULTS

Forty-five technical errors, mainly due to too little force or distance (n = 39), and 27 good surgical events were coded during grasping and dissection. Neural activity representing error monitoring (p = .008) and motor uncertainty (p = .034) was detected 17 s prior to errors, but not prior to good surgical performance.

CONCLUSIONS

These results show that distinct neural signatures are predictive of technical error in laparoscopic surgery. If replicated with low false-alarm rates, an EEG-based EWS of technical errors could be used to improve individualized surgical training by flagging imminent unsafe actions-before errors occur and cause patient harm.

Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces

Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.

Changes in brain activity of trainees during laparoscopic surgical virtual training assessed with electroencephalography

OBJECTIVE

Evaluate changes in brain activity of trainees during laparoscopic surgical training from electroencephalographic (EEG) signals in an ecological scenario with few restrictions for the user. Design Longitudinal study with two follow-up measurements in the first and last session of a 4-week training with LapSim laparoscopic surgery simulator. Variables analyzed include EEG neuronal activations in theta and alpha bands, tasks performance measures, and subjective measures such as perception of mental workload. Setting Medical School, Universidad de Antioquia, Medellin, Colombia. Participants First-year surgical residents (n = 16, age = 28.0 ± 2.6 years old, right-handed, 9 females) RESULTS: Significant improvements in tasks performance were found together with changes in neuronal activity over frontal and parietal cortex. These changes were also correlated with task performance through training sessions.

CONCLUSIONS

The use of neurophysiological measures such as electroencephalography combined with source separation techniques allows evaluating neural changes associated with motor training. The experiment proposed in this work establishes less controlled recording conditions leading to a more realistic analysis scenario to cognitive assessment in residents training.

Mental State Detection Using Riemannian Geometry on Electroencephalogram Brain Signals

The goal of this study was to implement a Riemannian geometry (RG)-based algorithm to detect high mental workload (MWL) and mental fatigue (MF) using task-induced electroencephalogram (EEG) signals. In order to elicit high MWL and MF, the participants performed a cognitively demanding task in the form of the letter n-back task. We analyzed the time-varying characteristics of the EEG band power (BP) features in the theta and alpha frequency band at different task conditions and cortical areas by employing a RG-based framework. MWL and MF were considered as too high, when the Riemannian distances of the task-run EEG reached or surpassed the threshold of the baseline EEG. The results of this study showed a BP increase in the theta and alpha frequency bands with increasing experiment duration, indicating elevated MWL and MF that impedes/hinders the task performance of the participants. High MWL and MF was detected in 8 out of 20 participants. The Riemannian distances also showed a steady increase toward the threshold with increasing experiment duration, with the most detections occurring toward the end of the experiment. To support our findings, subjective ratings (questionnaires concerning fatigue and workload levels) and behavioral measures (performance accuracies and response times) were also considered.

State of the Art of Non-Invasive Electrode Materials for Brain-Computer Interface

The brain-computer interface (BCI) has emerged in recent years and has attracted great attention. As an indispensable part of the BCI signal acquisition system, brain electrodes have a great influence on the quality of the signal, which determines the final effect. Due to the special usage scenario of brain electrodes, some specific properties are required for them. In this study, we review the development of three major types of EEG electrodes from the perspective of material selection and structural design, including dry electrodes, wet electrodes, and semi-dry electrodes. Additionally, we provide a reference for the current chaotic performance evaluation of EEG electrodes in some aspects such as electrochemical performance, stability, and so on. Moreover, the challenges and future expectations for EEG electrodes are analyzed.

Developments in the human machine interface technologies and their applications: a review

Human-machine interface (HMI) techniques use bioelectrical signals to gain real-time synchronised communication between the human body and machine functioning. HMI technology not only provides a real-time control access but also has the ability to control multiple functions at a single instance of time with modest human inputs and increased efficiency. The HMI technologies yield advanced control access on numerous applications such as health monitoring, medical diagnostics, development of prosthetic and assistive devices, automotive and aerospace industry, robotic controls and many more fields. In this paper, various physiological signals, their acquisition and processing techniques along with their respective applications in different HMI technologies have been discussed.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction

Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.

Assessment of changes in neural activity during acquisition of spatial knowledge using EEG signal classification

OBJECTIVE

This study explored the classification of electroencephalography (EEG) signals to assess changes in neural activity as individuals performed a training task in a virtual environment simulator. Commonly, task behavior and perception are used to assess a trainee's ability to perform a task, however, changes in cognition are not usually measured and could be important to provide a true indication of an individual's level of knowledge or skill.

APPROACH

In this study, 15 participants acquired spatial knowledge via 60 navigation trials (divided into ten blocks) in a novel virtual environment. Time performance, perceived certainty, and EEG signal data were collected.

MAIN RESULTS

A significant increase in alpha power and classification accuracy of EEG data from block 1 against blocks 2-10 was observed and stabilized after block 7, while time performance and perceived certainty measures improved and stabilized after block 5 and 6, respectively.

SIGNIFICANCE

Results suggest that changes in neural activity, which may reflect an increase in cognitive efficiency, could provide additional insight beyond time performance and perceived certainty.

SEREEGA: Simulating event-related EEG activity

BACKGROUND

Electroencephalography (EEG) is a popular method to monitor brain activity, but it is difficult to evaluate EEG-based analysis methods because no ground-truth brain activity is available for comparison. Therefore, in order to test and evaluate such methods, researchers often use simulated EEG data instead of actual EEG recordings. Simulated data can be used, among other things, to assess or compare signal processing and machine learning algorithms, to model EEG variabilities, and to design source reconstruction methods.

NEW METHOD

We present SEREEGA, Simulating Event-Related EEG Activity. SEREEGA is a free and open-source MATLAB-based toolbox dedicated to the generation of simulated epochs of EEG data. It is modular and extensible, at initial release supporting five different publicly available head models and capable of simulating multiple different types of signals mimicking brain activity. This paper presents the architecture and general workflow of this toolbox, as well as a simulated data set demonstrating some of its functions. The toolbox is available at https://github.com/lrkrol/SEREEGA.

RESULTS

The simulated data allows established analysis pipelines and classification methods to be applied and is capable of producing realistic results.

COMPARISON WITH EXISTING METHODS

Most simulated EEG is coded from scratch. The few open-source methods in existence focus on specific applications or signal types, such as connectivity. SEREEGA unifies the majority of past simulation methods reported in the literature into one toolbox.

CONCLUSION

SEREEGA is a general-purpose toolbox to simulate ground-truth EEG data.

Decoding human mental states by whole-head EEG+fNIRS during category fluency task performance

OBJECTIVE

Concurrent scalp electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), which we refer to as EEG+fNIRS, promises greater accuracy than the individual modalities while remaining nearly as convenient as EEG. We sought to quantify the hybrid system's ability to decode mental states and compare it with its unimodal components.

APPROACH

We recorded from healthy volunteers taking the category fluency test and applied machine learning techniques to the data.

MAIN RESULTS

EEG+fNIRS's decoding accuracy was greater than that of its subsystems, partly due to the new type of neurovascular features made available by hybrid data.

SIGNIFICANCE

Availability of an accurate and practical decoding method has potential implications for medical diagnosis, brain-computer interface design, and neuroergonomics.

In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations.