EEG error-related potentials encode magnitude of errors and individual perceptual thresholds

Counting on AR: EEG responses to incongruent information with real-world context

Augmented Reality (AR) technologies enhance the real world by integrating contextual digital information about physical entities. However, inconsistencies between physical reality and digital augmentations, which may arise from errors in the visualized information or the user's mental context, can considerably impact user experience. This work characterizes the brain dynamics associated with processing incongruent information within an AR environment. To study these effects, we designed an interactive paradigm featuring the manipulation of a Rubik's cube serving as a physical referent. Congruent and incongruent information regarding the cube's current status was presented via symbolic (digits) and non-symbolic (graphs) stimuli, thus examining the impact of different means of data representation. The analysis of electroencephalographic signals from 19 participants revealed the presence of centro-parietal N400 and P600 components following the processing of incongruent information, with significantly increased latencies for non-symbolic stimuli. Additionally, we explored the feasibility of exploiting incongruency effects for brain-computer interfaces. Hence, we implemented decoders using linear discriminant analysis, support vector machines, and EEGNet, achieving comparable performances with all methods. Therefore, this work contributes to the design of adaptive AR systems by demonstrating that above-chance detection of incongruent information based on physiological signals is feasible. The successful decoding of incongruency-induced modulations can inform systems about the current mental state of users without making it explicit, aiming for more coherent and contextually appropriate AR interactions.

Error-related potentials during multitasking involving sensorimotor control: an ERP and offline decoding study for brain-computer interface

Humans achieve efficient behaviors by perceiving and responding to errors. Error-related potentials (ErrPs) are electrophysiological responses that occur upon perceiving errors. Leveraging ErrPs to improve the accuracy of brain-computer interfaces (BCIs), utilizing the brain's natural error-detection processes to enhance system performance, has been proposed. However, the influence of external and contextual factors on the detectability of ErrPs remains poorly understood, especially in multitasking scenarios involving both BCI operations and sensorimotor control. Herein, we hypothesized that the difficulty in sensorimotor control would lead to the dispersion of neural resources in multitasking, resulting in a reduction in ErrP features. To examine this, we conducted an experiment in which participants were instructed to keep a ball within a designated area on a board, while simultaneously attempting to control a cursor on a display through motor imagery. The BCI provided error feedback with a random probability of 30%. Three scenarios-without a ball (single-task), lightweight ball (easy-task), and heavyweight ball (hard-task)-were used for the characterization of ErrPs based on the difficulty of sensorimotor control. In addition, to examine the impact of multitasking on ErrP-BCI performance, we analyzed single-trial classification accuracy offline. Contrary to our hypothesis, varying the difficulty of sensorimotor control did not result in significant changes in ErrP features. However, multitasking significantly affected ErrP classification accuracy. Post-hoc analyses revealed that the classifier trained on single-task ErrPs exhibited reduced accuracy under hard-task scenarios. To our knowledge, this study is the first to investigate how ErrPs are modulated in a multitasking environment involving both sensorimotor control and BCI operation in an offline framework. Although the ErrP features remained unchanged, the observed variation in accuracy suggests the need to design classifiers that account for task load even before implementing a real-time ErrP-based BCI.

Neural interfaces: Bridging the brain to the world beyond healthcare

Neural interfaces, emerging at the intersection of neurotechnology and urban planning, promise to transform how we interact with our surroundings and communicate. By recording and decoding neural signals, these interfaces facilitate direct connections between the brain and external devices, enabling seamless information exchange and shared experiences. Nevertheless, their development is challenged by complexities in materials science, electrochemistry, and algorithmic design. Electrophysiological crosstalk and the mismatch between electrode rigidity and tissue flexibility further complicate signal fidelity and biocompatibility. Recent closed-loop brain-computer interfaces, while promising for mood regulation and cognitive enhancement, are limited by decoding accuracy and the adaptability of user interfaces. This perspective outlines these challenges and discusses the progress in neural interfaces, contrasting non-invasive and invasive approaches, and explores the dynamics between stimulation and direct interfacing. Emphasis is placed on applications beyond healthcare, highlighting the need for implantable interfaces with high-resolution recording and stimulation capabilities.

Robustness and adaptability of sensorimotor skills in expert piano performance

Skillful sequential action requires the delicate balance of sensorimotor control, encompassing both robustness and adaptability. However, it remains unknown whether both motor and neural responses triggered by sensory perturbation undergo plastic adaptation as a consequence of extensive sensorimotor experience. We assessed the effects of transiently delayed tone production on the subsequent motor actions and event-related potentials (ERPs) during piano performance by comparing pianists and non-musicians. Following the perturbation, the inter-keystroke interval was abnormally prolonged in non-musicians but not in pianists. By contrast, the keystroke velocity following the perturbation was increased only in the pianists. A regression model demonstrated that the change in the inter-keystroke interval covaried with the ERPs, particularly at the frontal and parietal regions. The alteration in the keystroke velocity was associated with the P300 component of the temporal region. These findings suggest that different neural mechanisms underlie robust and adaptive sensorimotor skills across proficiency level.

Yes or no? A study of ErrPs in the "guess what I am thinking" paradigm with stimuli of different visual content

Error-related potentials (ErrPs) have attracted attention in part because of their practical potential for building brain-computer interface (BCI) paradigms. BCIs, facilitating direct communication between the brain and machines, hold great promise for brain-AI interaction. Therefore, a comprehensive understanding of ErrPs is crucial to ensure reliable BCI outcomes. In this study, we investigated ErrPs in the context of the "guess what I am thinking" paradigm. 23 healthy participants were instructed to imagine an object from a predetermined set, while an algorithm randomly selected another object that was either the same as or different from the imagined object. We recorded and analyzed the participants' EEG activity to capture their mental responses to the algorithm's "predictions". The study identified components distinguishing correct from incorrect responses. It discusses their nature and how they differ from ErrPs extensively studied in other BCI paradigms. We observed pronounced variations in the shape of ErrPs across different stimulus sets, underscoring the significant influence of visual stimulus appearance on ErrP peaks. These findings have implications for designing effective BCI systems, especially considering the less conventional BCI paradigm employed. They emphasize the necessity of accounting for stimulus factors in BCI development.

Customizing the human-avatar mapping based on EEG error related potentials

Objective.A key challenge of virtual reality (VR) applications is to maintain a reliable human-avatar mapping. Users may lose the sense of controlling (sense of agency), owning (sense of body ownership), or being located (sense of self-location) inside the virtual body when they perceive erroneous interaction, i.e. a break-in-embodiment (BiE). However, the way to detect such an inadequate event is currently limited to questionnaires or spontaneous reports from users. The ability to implicitly detect BiE in real-time enables us to adjust human-avatar mapping without interruption.Approach.We propose and empirically demonstrate a novel brain computer interface (BCI) approach that monitors the occurrence of BiE based on the users' brain oscillatory activity in real-time to adjust the human-avatar mapping in VR. We collected EEG activity of 37 participants while they performed reaching movements with their avatar with different magnitude of distortion.Main results.Our BCI approach seamlessly predicts occurrence of BiE in varying magnitude of erroneous interaction. The mapping has been customized by BCI-reinforcement learning (RL) closed-loop system to prevent BiE from occurring. Furthermore, a non-personalized BCI decoder generalizes to new users, enabling 'Plug-and-Play' ErrP-based non-invasive BCI. The proposed VR system allows customization of human-avatar mapping without personalized BCI decoders or spontaneous reports.Significance.We anticipate that our newly developed VR-BCI can be useful to maintain an engaging avatar-based interaction and a compelling immersive experience while detecting when users notice a problem and seamlessly correcting it.

Inferring individual evaluation criteria for reaching trajectories with obstacle avoidance from EEG signals

During reaching actions, the human central nerve system (CNS) generates the trajectories that optimize effort and time. When there is an obstacle in the path, we make sure that our arm passes the obstacle with a sufficient margin. This comfort margin varies between individuals. When passing a fragile object, risk-averse individuals may adopt a larger margin by following the longer path than risk-prone people do. However, it is not known whether this variation is associated with a personalized cost function used for the individual optimal control policies and how it is represented in our brain activity. This study investigates whether such individual variations in evaluation criteria during reaching results from differentiated weighting given to energy minimization versus comfort, and monitors brain error-related potentials (ErrPs) evoked when subjects observe a robot moving dangerously close to a fragile object. Seventeen healthy participants monitored a robot performing safe, daring and unsafe trajectories around a wine glass. Each participant displayed distinct evaluation criteria on the energy efficiency and comfort of robot trajectories. The ErrP-BCI outputs successfully inferred such individual variation. This study suggests that ErrPs could be used in conjunction with an optimal control approach to identify the personalized cost used by CNS. It further opens new avenues for the use of brain-evoked potential to train assistive robotic devices through the use of neuroprosthetic interfaces.