Single trial detection of error-related potentials in brain-machine interfaces: a survey and comparison of methods

Counting on AR: EEG responses to incongruent information with real-world context

Augmented Reality (AR) technologies enhance the real world by integrating contextual digital information about physical entities. However, inconsistencies between physical reality and digital augmentations, which may arise from errors in the visualized information or the user's mental context, can considerably impact user experience. This work characterizes the brain dynamics associated with processing incongruent information within an AR environment. To study these effects, we designed an interactive paradigm featuring the manipulation of a Rubik's cube serving as a physical referent. Congruent and incongruent information regarding the cube's current status was presented via symbolic (digits) and non-symbolic (graphs) stimuli, thus examining the impact of different means of data representation. The analysis of electroencephalographic signals from 19 participants revealed the presence of centro-parietal N400 and P600 components following the processing of incongruent information, with significantly increased latencies for non-symbolic stimuli. Additionally, we explored the feasibility of exploiting incongruency effects for brain-computer interfaces. Hence, we implemented decoders using linear discriminant analysis, support vector machines, and EEGNet, achieving comparable performances with all methods. Therefore, this work contributes to the design of adaptive AR systems by demonstrating that above-chance detection of incongruent information based on physiological signals is feasible. The successful decoding of incongruency-induced modulations can inform systems about the current mental state of users without making it explicit, aiming for more coherent and contextually appropriate AR interactions.

Error-related potentials during multitasking involving sensorimotor control: an ERP and offline decoding study for brain-computer interface

Humans achieve efficient behaviors by perceiving and responding to errors. Error-related potentials (ErrPs) are electrophysiological responses that occur upon perceiving errors. Leveraging ErrPs to improve the accuracy of brain-computer interfaces (BCIs), utilizing the brain's natural error-detection processes to enhance system performance, has been proposed. However, the influence of external and contextual factors on the detectability of ErrPs remains poorly understood, especially in multitasking scenarios involving both BCI operations and sensorimotor control. Herein, we hypothesized that the difficulty in sensorimotor control would lead to the dispersion of neural resources in multitasking, resulting in a reduction in ErrP features. To examine this, we conducted an experiment in which participants were instructed to keep a ball within a designated area on a board, while simultaneously attempting to control a cursor on a display through motor imagery. The BCI provided error feedback with a random probability of 30%. Three scenarios-without a ball (single-task), lightweight ball (easy-task), and heavyweight ball (hard-task)-were used for the characterization of ErrPs based on the difficulty of sensorimotor control. In addition, to examine the impact of multitasking on ErrP-BCI performance, we analyzed single-trial classification accuracy offline. Contrary to our hypothesis, varying the difficulty of sensorimotor control did not result in significant changes in ErrP features. However, multitasking significantly affected ErrP classification accuracy. Post-hoc analyses revealed that the classifier trained on single-task ErrPs exhibited reduced accuracy under hard-task scenarios. To our knowledge, this study is the first to investigate how ErrPs are modulated in a multitasking environment involving both sensorimotor control and BCI operation in an offline framework. Although the ErrP features remained unchanged, the observed variation in accuracy suggests the need to design classifiers that account for task load even before implementing a real-time ErrP-based BCI.

Improving non-invasive trajectory decoding via neural correlates of continuous erroneous feedback processing

Objective. Over the last decades, error-related potentials (ErrPs) have repeatedly proven especially useful as corrective mechanisms in invasive and non-invasive brain-computer interfaces (BCIs). However, research in this context exclusively investigated the distinction of discrete events intocorrectorerroneousto the present day. Due to this predominant formulation as a binary classification problem, classical ErrP-based BCIs fail to monitor tasks demanding quantitative information on error severity rather than mere qualitative decisions on error occurrence. As a result, fine-tuned and natural feedback control based on continuously perceived deviations from an intended target remains beyond the capabilities of previously used BCI setups.Approach.To address this issue for future BCI designs, we investigated the feasibility of regressing rather than classifying error-related activity non-invasively from the brain.Main results.Using pre-recorded data from ten able-bodied participants in three sessions each and a multi-output convolutional neural network, we demonstrated the above-chance regression of ongoing target-feedback discrepancies from brain signals in a pseudo-online fashion. In a second step, we used this inferred information about the target deviation to correct the initially displayed feedback accordingly, reporting significant improvements in correlations between corrected feedback and target trajectories across feedback conditions.Significance.Our results indicate that continuous information on target-feedback discrepancies can be successfully regressed from cortical activity, paving the way to increasingly naturalistic, fine-tuned correction mechanisms for future BCI applications.

Yes or no? A study of ErrPs in the "guess what I am thinking" paradigm with stimuli of different visual content

Error-related potentials (ErrPs) have attracted attention in part because of their practical potential for building brain-computer interface (BCI) paradigms. BCIs, facilitating direct communication between the brain and machines, hold great promise for brain-AI interaction. Therefore, a comprehensive understanding of ErrPs is crucial to ensure reliable BCI outcomes. In this study, we investigated ErrPs in the context of the "guess what I am thinking" paradigm. 23 healthy participants were instructed to imagine an object from a predetermined set, while an algorithm randomly selected another object that was either the same as or different from the imagined object. We recorded and analyzed the participants' EEG activity to capture their mental responses to the algorithm's "predictions". The study identified components distinguishing correct from incorrect responses. It discusses their nature and how they differ from ErrPs extensively studied in other BCI paradigms. We observed pronounced variations in the shape of ErrPs across different stimulus sets, underscoring the significant influence of visual stimulus appearance on ErrP peaks. These findings have implications for designing effective BCI systems, especially considering the less conventional BCI paradigm employed. They emphasize the necessity of accounting for stimulus factors in BCI development.

A generic error-related potential classifier based on simulated subjects

Error-related potentials (ErrPs) are brain signals known to be generated as a reaction to erroneous events. Several works have shown that not only self-made errors but also mistakes generated by external agents can elicit such event-related potentials. The possibility of reliably measuring ErrPs through non-invasive techniques has increased the interest in the brain-computer interface (BCI) community in using such signals to improve performance, for example, by performing error correction. Extensive calibration sessions are typically necessary to gather sufficient trials for training subject-specific ErrP classifiers. This procedure is not only time-consuming but also boresome for participants. In this paper, we explore the effectiveness of ErrPs in closed-loop systems, emphasizing their dependency on precise single-trial classification. To guarantee the presence of an ErrPs signal in the data we employ and to ensure that the parameters defining ErrPs are systematically varied, we utilize the open-source toolbox SEREEGA for data simulation. We generated training instances and evaluated the performance of the generic classifier on both simulated and real-world datasets, proposing a promising alternative to conventional calibration techniques. Results show that a generic support vector machine classifier reaches balanced accuracies of 72.9%, 62.7%, 71.0%, and 70.8% on each validation dataset. While performing similarly to a leave-one-subject-out approach for error class detection, the proposed classifier shows promising generalization across different datasets and subjects without further adaptation. Moreover, by utilizing SEREEGA, we can systematically adjust parameters to accommodate the variability in the ErrP, facilitating the systematic validation of closed-loop setups. Furthermore, our objective is to develop a universal ErrP classifier that captures the signal's variability, enabling it to determine the presence or absence of an ErrP in real EEG data.

From lab to life: assessing the impact of real-world interactions on the operation of rapid serial visual presentation-based brain-computer interfaces

Objective.Brain-computer interfaces (BCI) have been extensively researched in controlled lab settings where the P300 event-related potential (ERP), elicited in the rapid serial visual presentation (RSVP) paradigm, has shown promising potential. However, deploying BCIs outside of laboratory settings is challenging due to the presence of contaminating artifacts that often occur as a result of activities such as talking, head movements, and body movements. These artifacts can severely contaminate the measured EEG signals and consequently impede detection of the P300 ERP. Our goal is to assess the impact of these real-world noise factors on the performance of a RSVP-BCI, specifically focusing on single-trial P300 detection.Approach.In this study, we examine the impact of movement activity on the performance of a P300-based RSVP-BCI application designed to allow users to search images at high speed. Using machine learning, we assessed P300 detection performance using both EEG data captured in optimal recording conditions (e.g. where participants were instructed to refrain from moving) and a variety of conditions where the participant intentionally produced movements to contaminate the EEG recording.Main results.The results, presented as area under the receiver operating characteristic curve (ROC-AUC) scores, provide insight into the significant impact of noise on single-trial P300 detection. Notably, there is a reduction in classifier detection accuracy when intentionally contaminated RSVP trials are used for training and testing, when compared to using non-intentionally contaminated RSVP trials.Significance.Our findings underscore the necessity of addressing and mitigating noise in EEG recordings to facilitate the use of BCIs in real-world settings, thus extending the reach of EEG technology beyond the confines of the laboratory.

A deep neural network and transfer learning combined method for cross-task classification of error-related potentials

Background

Error-related potentials (ErrPs) are electrophysiological responses that naturally occur when humans perceive wrongdoing or encounter unexpected events. It offers a distinctive means of comprehending the error-processing mechanisms within the brain. A method for detecting ErrPs with high accuracy holds significant importance for various ErrPs-based applications, such as human-in-the-loop Brain-Computer Interface (BCI) systems. Nevertheless, current methods fail to fulfill the generalization requirements for detecting such ErrPs due to the high non-stationarity of EEG signals across different tasks and the limited availability of ErrPs datasets.

Methods

This study introduces a deep learning-based model that integrates convolutional layers and transformer encoders for the classification of ErrPs. Subsequently, a model training strategy, grounded in transfer learning, is proposed for the effective training of the model. The datasets utilized in this study are available for download from the publicly accessible databases.

Results

In cross-task classification, an average accuracy of about 78% was achieved, exceeding the baseline. Furthermore, in the leave-one-subject-out, within-session, and cross-session classification scenarios, the proposed model outperformed the existing techniques with an average accuracy of 71.81, 78.74, and 77.01%, respectively.

Conclusions

Our approach contributes to mitigating the challenge posed by limited datasets in the ErrPs field, achieving this by reducing the requirement for extensive training data for specific target tasks. This may serve as inspiration for future studies that concentrate on ErrPs and their applications.

Multimodal decoding of error processing in a virtual reality flight simulation

Technological advances in head-mounted displays (HMDs) facilitate the acquisition of physiological data of the user, such as gaze, pupil size, or heart rate. Still, interactions with such systems can be prone to errors, including unintended behavior or unexpected changes in the presented virtual environments. In this study, we investigated if multimodal physiological data can be used to decode error processing, which has been studied, to date, with brain signals only. We examined the feasibility of decoding errors solely with pupil size data and proposed a hybrid decoding approach combining electroencephalographic (EEG) and pupillometric signals. Moreover, we analyzed if hybrid approaches can improve existing EEG-based classification approaches and focused on setups that offer increased usability for practical applications, such as the presented game-like virtual reality flight simulation. Our results indicate that classifiers trained with pupil size data can decode errors above chance. Moreover, hybrid approaches yielded improved performance compared to EEG-based decoders in setups with a reduced number of channels, which is crucial for many out-of-the-lab scenarios. These findings contribute to the development of hybrid brain-computer interfaces, particularly in combination with wearable devices, which allow for easy acquisition of additional physiological data.

A toolbox for decoding BCI commands based on event-related potentials

Commands in brain-computer interface (BCI) applications often rely on the decoding of event-related potentials (ERP). For instance, the P300 potential is frequently used as a marker of attention to an oddball event. Error-related potentials and the N2pc signal are further examples of ERPs used for BCI control. One challenge in decoding brain activity from the electroencephalogram (EEG) is the selection of the most suitable channels and appropriate features for a particular classification approach. Here we introduce a toolbox that enables ERP-based decoding using the full set of channels, while automatically extracting informative components from relevant channels. The strength of our approach is that it handles sequences of stimuli that encode multiple items using binary classification, such as target vs. nontarget events typically used in ERP-based spellers. We demonstrate examples of application scenarios and evaluate the performance of four openly available datasets: a P300-based matrix speller, a P300-based rapid serial visual presentation (RSVP) speller, a binary BCI based on the N2pc, and a dataset capturing error potentials. We show that our approach achieves performances comparable to those in the original papers, with the advantage that only conventional preprocessing is required by the user, while channel weighting and decoding algorithms are internally performed. Thus, we provide a tool to reliably decode ERPs for BCI use with minimal programming requirements.

Error detection and correction in intracortical brain-machine interfaces controlling two finger groups

Objective.While brain-machine interfaces (BMIs) are promising technologies that could provide direct pathways for controlling the external world and thus regaining motor capabilities, their effectiveness is hampered by decoding errors. Previous research has demonstrated the detection and correction of BMI outcome errors, which occur at the end of trials. Here we focus on continuous detection and correction of BMI execution errors, which occur during real-time movements.Approach.Two adult male rhesus macaques were implanted with Utah arrays in the motor cortex. The monkeys performed single or two-finger group BMI tasks where a Kalman filter decoded binned spiking-band power into intended finger kinematics. Neural activity was analyzed to determine how it depends not only on the kinematics of the fingers, but also on the distance of each finger-group to its target. We developed a method to detect erroneous movements, i.e. consistent movements away from the target, from the same neural activity used by the Kalman filter. Detected errors were corrected by a simple stopping strategy, and the effect on performance was evaluated.Mainresults.First we show that including distance to target explains significantly more variance of the recorded neural activity. Then, for the first time, we demonstrate that neural activity in motor cortex can be used to detect execution errors during BMI controlled movements. Keeping false positive rate below5%, it was possible to achieve mean true positive rate of28.1%online. Despite requiring 200 ms to detect and react to suspected errors, we were able to achieve a significant improvement in task performance via reduced orbiting time of one finger group.Significance.Neural activity recorded in motor cortex for BMI control can be used to detect and correct BMI errors and thus to improve performance. Further improvements may be obtained by enhancing classification and correction strategies.