Optimizing the P300-based brain-computer interface: current status, limitations and future directions

Multi-information improves the performance of CCA-based SSVEP classification

The target recognition algorithm based on canonical correlation analysis (CCA) has been widely used in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces. To reduce visual fatigue and improve the information transfer rate (ITR), how to improve the accuracy of algorithms within a short time window has become one of the main problems at present. There were filter bank CCA (FBCCA), individual template CCA (ITCCA), and temporally local CCA (TCCA), which improve the CCA algorithm from different aspects.This paper proposed to consider individual, frequency, and time information at the same time, so as to extract features more effectively. A comparison of the various methods was performed using benchmark dataset. Classification accuracy and ITR were used for performance evaluation. In the different extensions of CCA, the method incorporating the above three kinds of information simultaneously achieved the best performance within a short time window. This study explores the effect of using a variety of information to improve the CCA algorithm.

An auditory brain-computer interface to detect changes in sound pressure level for automatic volume control

Volume control is necessary to adjust sound levels for a comfortable audio or video listening experience. This study aims to develop an automatic volume control system based on a brain-computer interface (BCI). We thus focused on a BCI using an auditory oddball paradigm, and conducted two types of experiments. In the first experiment, the participant was asked to pay attention to a target sound where the sound level was high (70 dB) compared with the other sounds (60 dB). The brain activity measured by electroencephalogram showed large positive activity (P300) for the target sound, and classification of the target and nontarget sounds achieved an accuracy of 0.90. The second experiment adopted a two-target paradigm where a low sound level (50 dB) was introduced as the second target sound. P300 was also observed in the second experiment, and a value of 0.76 was obtained for the binary classification of the target and nontarget sounds. Further, we found that better accuracy was observed in large sound levels compared to small ones. These results suggest the possibility of using BCI for automatic volume control; however, it is necessary to improve its accuracy for application in daily life.

Classification of Targets and Distractors in an Audiovisual Attention Task Based on Electroencephalography

Within the broader context of improving interactions between artificial intelligence and humans, the question has arisen regarding whether auditory and rhythmic support could increase attention for visual stimuli that do not stand out clearly from an information stream. To this end, we designed an experiment inspired by pip-and-pop but more appropriate for eliciting attention and P3a-event-related potentials (ERPs). In this study, the aim was to distinguish between targets and distractors based on the subject's electroencephalography (EEG) data. We achieved this objective by employing different machine learning (ML) methods for both individual-subject (IS) and cross-subject (CS) models. Finally, we investigated which EEG channels and time points were used by the model to make its predictions using saliency maps. We were able to successfully perform the aforementioned classification task for both the IS and CS scenarios, reaching classification accuracies up to 76%. In accordance with the literature, the model primarily used the parietal-occipital electrodes between 200 ms and 300 ms after the stimulus to make its prediction. The findings from this research contribute to the development of more effective P300-based brain-computer interfaces. Furthermore, they validate the EEG data collected in our experiment.

Post-stimulus encoding of decision confidence in EEG: toward a brain-computer interface for decision making

Objective.When making decisions, humans can evaluate how likely they are to be correct. If this subjective confidence could be reliably decoded from brain activity, it would be possible to build a brain-computer interface (BCI) that improves decision performance by automatically providing more information to the user if needed based on their confidence. But this possibility depends on whether confidence can be decoded right after stimulus presentation and before the response so that a corrective action can be taken in time. Although prior work has shown that decision confidence is represented in brain signals, it is unclear if the representation is stimulus-locked or response-locked, and whether stimulus-locked pre-response decoding is sufficiently accurate for enabling such a BCI.Approach.We investigate the neural correlates of confidence by collecting high-density electroencephalography (EEG) during a perceptual decision task with realistic stimuli. Importantly, we design our task to include a post-stimulus gap that prevents the confounding of stimulus-locked activity by response-locked activity and vice versa, and then compare with a task without this gap.Main results.We perform event-related potential and source-localization analyses. Our analyses suggest that the neural correlates of confidence are stimulus-locked, and that an absence of a post-stimulus gap could cause these correlates to incorrectly appear as response-locked. By preventing response-locked activity from confounding stimulus-locked activity, we then show that confidence can be reliably decoded from single-trial stimulus-locked pre-response EEG alone. We also identify a high-performance classification algorithm by comparing a battery of algorithms. Lastly, we design a simulated BCI framework to show that the EEG classification is accurate enough to build a BCI and that the decoded confidence could be used to improve decision making performance particularly when the task difficulty and cost of errors are high.Significance.Our results show feasibility of non-invasive EEG-based BCIs to improve human decision making.

Minimally Invasive Local-Skull Electrophysiological Modification With Piezoelectric Drill

The research on non-invasive BCI is nowadays hitting the bottleneck due to the humble quality of scalp EEG signals. Whereas invasive solutions that offer higher signal quality in contrast are suffocated in their spreading because of the potential surgical complication and health risks caused by electrode implantation. Therefore, it puts forward a necessity to explore a scheme that could both collect high-quality EEG signals and guarantee high-level operation safety.This study proposed a Minimally Invasive Local-skull Electrophysiological Modification method to improve scalp EEG signals qualities at specific brain regions. Six eight-month-old SD rats were used for in vivo verification experiment. A hole with a diameter of about 500 micrometers was drilled in the skull above the visual cortex of rats. Significant changes in rsEEG and SSVEP signals before and after modification were observed. After modification, the skull impedance of rats decreases by about 84 %, the average maximum bandwidth of rsEEG increase by 57 %, and the broadband SNR of SSVEP is increased by 5.13 dB. The time of piezoelectric drilling operation is strictly controlled under 30 seconds for each rat to prevent possible brain damage from overheating. Compared with traditional invasive procedures such as ECoG, Minimally Invasive Local-skull Electrophysiological Modification operation time is shorter and no electrode implantation is needed while it remarkably boosts the scalp EEG signal quality. This technical solution has the potential to replace the use of ECoG in certain application scenarios and further invigorate studies in the field of scalp EEG in the future.

A Review of the Role of Machine Learning Techniques towards Brain–Computer Interface Applications

This review article provides a deep insight into the Brain–Computer Interface (BCI) and the application of Machine Learning (ML) technology in BCIs. It investigates the various types of research undertaken in this realm and discusses the role played by ML in performing different BCI tasks. It also reviews the ML methods used for mental state detection, mental task categorization, emotion classification, electroencephalogram (EEG) signal classification, event-related potential (ERP) signal classification, motor imagery categorization, and limb movement classification. This work explores the various methods employed in BCI mechanisms for feature extraction, selection, and classification and provides a comparative study of reviewed methods. This paper assists the readers to gain information regarding the developments made in BCI and ML domains and future improvements needed for improving and designing better BCI applications.

The Impact of Vigorous Cycling Exercise on Visual Attention: A Study With the BR8 Wireless Dry EEG System

Many studies have reported that exercise can influence cognitive performance. But advancing our understanding of the interrelations between psychology and physiology in sports neuroscience requires the study of real-time brain dynamics during exercise in the field. Electroencephalography (EEG) is one of the most powerful brain imaging technologies. However, the limited portability and long preparation time of traditional wet-sensor systems largely limits their use to laboratory settings. Wireless dry-sensor systems are emerging with much greater potential for practical application in sports. Hence, in this paper, we use the BR8 wireless dry-sensor EEG system to measure P300 brain dynamics while cycling at various intensities. The preparation time was mostly less than 2 min as BR8 system’s dry sensors were able to attain the required skin-sensor interface impedance, enabling its operation without any skin preparation or application of conductive gel. Ten participants performed four sessions of a 3 min rapid serial visual presentation (RSVP) task while resting and while cycling. These four sessions were pre-CE (RSVP only), low-CE (RSVP in 40–50% of max heart rate), vigorous-CE (RSVP in 71–85% of max heart rate) and post-CE (RSVP only). The recorded brain signals demonstrate that the P300 amplitudes, observed at the Pz channel, for the target and non-target responses were significantly different in all four sessions. The results also show decreased reaction times to the visual attention task during vigorous exercise, enriching our understanding of the ways in which exercise can enhance cognitive performance. Even though only a single channel was evaluated in this study, the quality and reliability of the measurement using these dry sensor-based EEG systems is clearly demonstrated by our results. Further, the smooth implementation of the experiment with a dry system and the success of the data analysis demonstrate that wireless dry EEG devices can open avenues for real-time measurement of cognitive functions in athletes outside the laboratory.

EEG-Inception: A Novel Deep Convolutional Neural Network for Assistive ERP-Based Brain-Computer Interfaces

In recent years, deep-learning models gained attention for electroencephalography (EEG) classification tasks due to their excellent performance and ability to extract complex features from raw data. In particular, convolutional neural networks (CNN) showed adequate results in brain-computer interfaces (BCI) based on different control signals, including event-related potentials (ERP). In this study, we propose a novel CNN, called EEG-Inception, that improves the accuracy and calibration time of assistive ERP-based BCIs. To the best of our knowledge, EEG-Inception is the first model to integrate Inception modules for ERP detection, which combined efficiently with other structures in a light architecture, improved the performance of our approach. The model was validated in a population of 73 subjects, of which 31 present motor disabilities. Results show that EEG-Inception outperforms 5 previous approaches, yielding significant improvements for command decoding accuracy up to 16.0%, 10.7%, 7.2%, 5.7% and 5.1% in comparison to rLDA, xDAWN + Riemannian geometry, CNN-BLSTM, DeepConvNet and EEGNet, respectively. Moreover, EEG-Inception requires very few calibration trials to achieve state-of-the-art performances taking advantage of a novel training strategy that combines cross-subject transfer learning and fine-tuning to increase the feasibility of this approach for practical use in assistive applications.

Recommendations for Integrating a P300-Based Brain–Computer Interface in Virtual Reality Environments for Gaming: An Update

The integration of a P300-based brain–computer interface (BCI) into virtual reality (VR) environments is promising for the video games industry. However, it faces several limitations, mainly due to hardware constraints and limitations engendered by the stimulation needed by the BCI. The main restriction is still the low transfer rate that can be achieved by current BCI technology, preventing movement while using VR. The goal of this paper is to review current limitations and to provide application creators with design recommendations to overcome them, thus significantly reducing the development time and making the domain of BCI more accessible to developers. We review the design of video games from the perspective of BCI and VR with the objective of enhancing the user experience. An essential recommendation is to use the BCI only for non-complex and non-critical tasks in the game. Also, the BCI should be used to control actions that are naturally integrated into the virtual world. Finally, adventure and simulation games, especially if cooperative (multi-user), appear to be the best candidates for designing an effective VR game enriched by BCI technology.

Improvement of P300-Based Brain-Computer Interfaces for Home Appliances Control by Data Balancing Techniques

The oddball paradigm used in P300-based brain-computer interfaces (BCIs) intrinsically poses the issue of data imbalance between target stimuli and nontarget stimuli. Data imbalance can cause overfitting problems and, consequently, poor classification performance. The purpose of this study is to improve BCI performance by solving this data imbalance problem with sampling techniques. The sampling techniques were applied to BCI data in 15 subjects controlling a door lock, 15 subjects an electric light, and 14 subjects a Bluetooth speaker. We explored two categories of sampling techniques: oversampling and undersampling. Oversampling techniques, including random oversampling, synthetic minority oversampling technique (SMOTE), borderline-SMOTE, support vector machine (SVM) SMOTE, and adaptive synthetic sampling, were used to increase the number of samples for the class of target stimuli. Undersampling techniques, including random undersampling, neighborhood cleaning rule, Tomek's links, and weighted undersampling bagging, were used to reduce the class size of nontarget stimuli. The over- or undersampled data were classified by an SVM classifier. Overall, some oversampling techniques improved BCI performance while undersampling techniques often degraded performance. Particularly, using borderline-SMOTE yielded the highest accuracy (87.27%) and information transfer rate (8.82 bpm) across all three appliances. Moreover, borderline-SMOTE led to performance improvement, especially for poor performers. A further analysis showed that borderline-SMOTE improved SVM by generating more support vectors within the target class and enlarging margins. However, there was no difference in the accuracy between borderline-SMOTE and the method of applying the weighted regularization parameter of the SVM. Our results suggest that although oversampling improves performance of P300-based BCIs, it is not just the effect of the oversampling techniques, but rather the effect of solving the data imbalance problem.

A 36-Class Bimodal ERP Brain-Computer Interface Using Location-Congruent Auditory-Tactile Stimuli

To date, traditional visual-based event-related potential brain-computer interface (ERP-BCI) systems continue to dominate the mainstream BCI research. However, these conventional BCIs are unsuitable for the individuals who have partly or completely lost their vision. Considering the poor performance of gaze independent ERP-BCIs, it is necessary to study techniques to improve the performance of these BCI systems. In this paper, we developed a novel 36-class bimodal ERP-BCI system based on tactile and auditory stimuli, in which six-virtual-direction audio files produced via head related transfer functions (HRTF) were delivered through headphones and location-congruent electro-tactile stimuli were simultaneously delivered to the corresponding position using electrodes placed on the abdomen and waist. We selected the eight best channels, trained a Bayesian linear discriminant analysis (BLDA) classifier and acquired the optimal trial number for target selection in online process. The average online information transfer rate (ITR) of the bimodal ERP-BCI reached 11.66 bit/min, improvements of 35.11% and 36.69% compared to the auditory (8.63 bit/min) and tactile approaches (8.53 bit/min), respectively. The results demonstrate the performance of the bimodal system is superior to each unimodal system. These facts indicate that the proposed bimodal system has potential utility as a gaze-independent BCI in future real-world applications.

The Utility of Event-Related Potentials in Clinical Psychology

Event-related potentials (ERPs) are direct measures of brain activity that can be leveraged for clinically meaningful research. They can relate robustly both to continuous measures of individual difference and to categorical diagnoses in ways that clarify similarities and distinctions between apparently related disorders and traits. ERPs can be linked to genetic risk, can act as moderators of developmental trajectories and responses to stress, and can be leveraged to identify those at greater risk for psychopathology, especially when used in combination with other neural and self-report measures. ERPs can inform models of the development of, and risk for, psychopathology. Finally, ERPs can be used as targets for existing and novel interventions and prevention efforts. We provide concrete examples for each of these possibilities by focusing on programmatic research on the error-related negativity and anxiety, and thus show that ERPs are poised to make greater contributions toward the identification, prediction, treatment, and prevention of mental disorders.

An Asynchronous Hybrid Spelling Approach Based on EEG-EOG Signals for Chinese Character Input

In this paper, we presented a novel asynchronous speller for Chinese sinogram input by incorporating electroencephalography (EOG) into the conventional electroencephalography (EEG)-based spelling paradigm. An EOG-based brain switch was used to activate a classic row-column P300-based speller only when spelling was needed, enabling an asynchronous operation of the system. Then, the user could input sinograms by alternately performing P300 and double-blink tasks until he or she intended to stop spelling. With the incorporation of an EOG detector, the system achieved rapid sinogram input. In addition to asynchronous operation, the performance of the proposed speller was compared with that achieved by a P300-based method alone across 18 subjects. The proposed system showed a mean communication speed of approximately 2.39 sinograms per minute, an increase of 0.83 sinograms per minute compared with the P300-based method. The preliminary online performance indicated that the proposed paradigm is a very promising approach for practical Chinese sinogram input application. This system may also be expanded to users whose languages are written in logographic scripts to serve as an assistive communication tool.

Cross-correlation task-related component analysis (xTRCA) for enhancing evoked and induced responses of event-related potentials

We propose an analysis method that extracts trial-reproducible (i.e., recurring) event-related spatiotemporal EEG patterns by optimizing a spatial filter as well as trial timings of task-related components in the time domain simultaneously in a unified manner. Event-related responses are broadly categorized into evoked and induced responses, but those are analyzed commonly in the time and the time-frequency domain, respectively. To facilitate a comparison of evoked and induced responses, a unified method for analyzing both evoked and induced responses is desired. Here we propose a method of cross-correlation task-related component analysis (xTRCA) as an extension of our previous method. xTRCA constructs a linear spatial filter and then optimizes trial timings of single trials based on trial reproducibility as an objective function. The spatial filter enhances event-related responses, and the temporal optimization compensates trial-by-trial latencies that are inherent to ERPs. We first applied xTRCA to synthetic data of induced responses whose phases varied from trial to trial, and found that xTRCA could realign the induced responses by compensating the phase differences. We then demonstrated with mismatch negativity data that xTRCA enhanced the event-related-potential waveform observed at a single channel. Finally, a classification accuracy was improved when trial timings were optimized by xTRCA, suggesting a practical application of the method for a brain computer interface. We conclude that xTRCA provides a unified framework to analyze and enhance event-related evoked and induced responses in the time domain by objectively maximizing trial reproducibility.

Learning Discriminative Spatiospectral Features of ERPs for Accurate Brain-Computer Interfaces

Constructing accurate predictive models is at the heart of brain-computer interfaces (BCIs) because these models can ultimately translate brain activities into communication and control commands. The majority of the previous work in BCI use spatial, temporal, or spatiotemporal features of event-related potentials (ERPs). In this study, we examined the discriminatory effect of their spatiospectral features to capture the most relevant set of neural activities from electroencephalographic recordings that represent users' mental intent. In this regard, we model ERP waveforms using a sum of sinusoids with unknown amplitudes, frequencies, and phases. The effect of this signal modeling step is to represent high-dimensional ERP waveforms in a substantially lower dimensionality space, which includes their dominant power spectral contents. We found that the most discriminative frequencies for accurate decoding of visual attention modulated ERPs lie in a spectral range less than 6.4 Hz. This was empirically verified by treating dominant frequency contents of ERP waveforms as feature vectors in the state-of-the-art machine learning techniques used herein. The constructed predictive models achieved remarkable performance, which for some subjects was as high as 94% as measured by the area under curve. Using these spectral contents, we further studied the discriminatory effect of each channel and proposed an efficient strategy to choose subject-specific subsets of channels that generally led to classifiers with comparable performance.

An Online Interactive Paradigm for P300 Brain-Computer Interface Speller

For each brain-computer interface system, efficiency is a key issue that considers both accuracy and speed. The P300 spellers built upon oddball paradigm are usually less efficient due to the repetitive stimulation of multiple characters for reliable detection. In this paper, based on the online EEG signal, we propose an interactive paradigm for P300 speller to improve its efficiency, primarily focusing within the single characterP300 paradigm. Specifically, after each stimulation, we first evaluate the posterior probability of each character in the stimuli set to be the target. The lowprobability characters are then removed fromthe stimuli set in the subsequent round(s), as character flash continues until the probability of any character surpasses a predefined threshold. Then, the character is selected as the target and data collection for the trial terminates. By reducing stimulus sequence characters, the system efficiency can be substantially improved. The spelling accuracy is insignificantly affected as the characters being removed have low probability to be the target. The online experimental results from a total of eight subjects show that an average practical information transfer rate of 50.26 bits/min (i.e. 9.07 characters/min) has been achieved, with 91% average spelling accuracy rate.

Three-Dimensional Brain-Computer Interface Control Through Simultaneous Overt Spatial Attentional and Motor Imagery Tasks

OBJECTIVE

While noninvasive electroenceph-alography (EEG) based brain-computer interfacing (BCI) has been successfully demonstrated in two-dimensional (2-D) control tasks, little work has been published regarding its extension to practical three-dimensional (3-D) control.

METHODS

In this study, we developed a new BCI approach for 3-D control by combining a novel form of endogenous visuospatial attentional modulation, defined as overt spatial attention (OSA), and motor imagery (MI).

RESULTS

OSA modulation was shown to provide comparable control to conventional MI modulation in both 1-D and 2-D tasks. Furthermore, this paper provides evidence for the functional independence of traditional MI and OSA, as well as an investigation into the simultaneous use of both. Using this newly proposed BCI paradigm, 16 participants successfully completed a 3-D eight-target control task. Nine of these subjects further demonstrated robust 3-D control in a 12-target task, significantly outperforming the information transfer rate achieved in the 1-D and 2-D control tasks (29.7 ± 1.6 b/min).

CONCLUSION

These results strongly support the hypothesis that noninvasive EEG-based BCI can provide robust 3-D control through endogenous neural modulation in broader populations with limited training.

SIGNIFICANCE

Through the combination of the two strategies (MI and OSA), a substantial portion of the recruited subjects were capable of robustly controlling a virtual cursor in 3-D space. The proposed novel approach could broaden the dimensionality of BCI control and shorten the training time.

A comparison of feature extraction strategies using wavelet dictionaries and feature selection methods for single trial P300-based BCI

The P300 component of event-related potentials (ERPs) is widely used in the implementation of brain computer interfaces (BCI). In this context, one of the main issues to solve is the binary classification problem that entails differentiating between electroencephalographic (EEG) signals with and without P300. Given the particularly unfavorable signal-to-noise ratio (SNR) in the single-trial detection scenario, this is a challenging problem in the pattern recognition field. To the best of our knowledge, there are no previous experimental studies comparing feature extraction and selection methods for single trial P300-based BCIs using unified criteria and data. In order to improve the performance and robustness of single-trial classifiers, we analyzed and compared different alternatives for the feature generation and feature selection blocks. We evaluated different orthogonal decompositions based on the wavelet transform for feature extraction, as well as different filter, wrapper, and embedded alternatives for feature selection. Accuracies over 75% were obtained for most of the analyzed strategies with a relatively low computational cost, making them attractive for a practical BCI implementation using inexpensive hardware. Graphical Abstract Experiments performed for P300 detection.

A Feasibility Clinical Trial to Improve Social Attention in Autistic Spectrum Disorder (ASD) Using a Brain Computer Interface

Deficits in the interpretation of others' intentions from gaze-direction or other social attention cues are well-recognized in ASD. Here we investigated whether an EEG brain computer interface (BCI) can be used to train social cognition skills in ASD patients. We performed a single-arm feasibility clinical trial and enrolled 15 participants (mean age 22y 2m) with high-functioning ASD (mean full-scale IQ 103). Participants were submitted to a BCI training paradigm using a virtual reality interface over seven sessions spread over 4 months. The first four sessions occurred weekly, and the remainder monthly. In each session, the subject was asked to identify objects of interest based on the gaze direction of an avatar. Attentional responses were extracted from the EEG P300 component. A final follow-up assessment was performed 6-months after the last session. To analyze responses to joint attention cues participants were assessed pre and post intervention and in the follow-up, using an ecologic “Joint-attention task.” We used eye-tracking to identify the number of social attention items that a patient could accurately identify from an avatar's action cues (e.g., looking, pointing at). As secondary outcome measures we used the Autism Treatment Evaluation Checklist (ATEC) and the Vineland Adaptive Behavior Scale (VABS). Neuropsychological measures related to mood and depression were also assessed. In sum, we observed a decrease in total ATEC and rated autism symptoms (Sociability; Sensory/Cognitive Awareness; Health/Physical/Behavior); an evident improvement in Adapted Behavior Composite and in the DLS subarea from VABS; a decrease in Depression (from POMS) and in mood disturbance/depression (BDI). BCI online performance and tolerance were stable along the intervention. Average P300 amplitude and alpha power were also preserved across sessions. We have demonstrated the feasibility of BCI in this kind of intervention in ASD. Participants engage successfully and consistently in the task. Although the primary outcome (rate of automatic responses to joint attention cues) did not show changes, most secondary neuropsychological outcome measures showed improvement, yielding promise for a future efficacy trial.

(clinical-trial ID: NCT02445625—clinicaltrials.gov).

Toward an Attention-Based Diagnostic Tool for Patients With Locked-in Syndrome

Electroencephalography (EEG) has been proposed as a supplemental tool for reducing clinical misdiagnosis in severely brain-injured populations helping to distinguish conscious from unconscious patients. We studied the use of spectral entropy as a measure of focal attention in order to develop a motor-independent, portable, and objective diagnostic tool for patients with locked-in syndrome (LIS), answering the issues of accuracy and training requirement. Data from 20 healthy volunteers, 6 LIS patients, and 10 patients with a vegetative state/unresponsive wakefulness syndrome (VS/UWS) were included. Spectral entropy was computed during a gaze-independent 2-class (attention vs rest) paradigm, and compared with EEG rhythms (delta, theta, alpha, and beta) classification. Spectral entropy classification during the attention-rest paradigm showed 93% and 91% accuracy in healthy volunteers and LIS patients respectively. VS/UWS patients were at chance level. EEG rhythms classification reached a lower accuracy than spectral entropy. Resting-state EEG spectral entropy could not distinguish individual VS/UWS patients from LIS patients. The present study provides evidence that an EEG-based measure of attention could detect command-following in patients with severe motor disabilities. The entropy system could detect a response to command in all healthy subjects and LIS patients, while none of the VS/UWS patients showed a response to command using this system.

Enhancing performance of P300-Speller under mental workload by incorporating dual-task data during classifier training

As one of the most important brain-computer interface (BCI) paradigms, P300-Speller was shown to be significantly impaired once applied in practical situations due to effects of mental workload. This study aims to provide a new method of building training models to enhance performance of P300-Speller under mental workload. Three experiment conditions based on row-column P300-Speller paradigm were performed including speller-only, 3-back-speller and mental-arithmetic-speller. Data under dual-task conditions were introduced to speller-only data respectively to build new training models. Then performance of classifiers with different models was compared under the same testing condition. The results showed that when tasks of imported training data and testing data were the same, character recognition accuracies and round accuracies of P300-Speller with mixed-data training models significantly improved (FDR, p < 0.005). When they were different, performance significantly improved when tested on mental-arithmetic-speller (FDR, p < 0.05) while the improvement was modest when tested on n-back-speller (FDR, p < 0.1). The analysis of ERPs revealed that ERP difference between training data and testing data was significantly diminished when the dual-task data was introduced to training data (FDR, p < 0.05). The new method of training classifier on mixed data proved to be effective in enhancing performance of P300-Speller under mental workload, confirmed the feasibility to build a universal training model and overcome the effects of mental workload in its practical applications.

Robot-to-human feedback and automatic object grasping using an RGB-D camera–projector system

This paper presents a novel system for human–robot interaction in object-grasping applications. Consisting of an RGB-D camera, a projector and a robot manipulator, the proposed system provides intuitive information to the human by analyzing the scene, detecting graspable objects and directly projecting numbers or symbols in front of objects. Objects are detected using a visual attention model that incorporates color, shape and depth information. The positions and orientations of the projected numbers are based on the shapes, positions and orientations of the corresponding objects. Users select a grasping target by indicating the corresponding number. Projected arrows are then created on the fly to guide a robotic arm to grasp the selected object using visual servoing and deliver the object to the human user. Experimental results are presented to demonstrate how the system is used in robot grasping tasks.

Decoding Saccadic Directions Using Epidural ECoG in Non-Human Primates

A brain-computer interface (BCI) can be used to restore some communication as an alternative interface for patients suffering from locked-in syndrome. However, most BCI systems are based on SSVEP, P300, or motor imagery, and a diversity of BCI protocols would be needed for various types of patients. In this paper, we trained the choice saccade (CS) task in 2 non-human primate monkeys and recorded the brain signal using an epidural electrocorticogram (eECoG) to predict eye movement direction. We successfully predicted the direction of the upcoming eye movement using a support vector machine (SVM) with the brain signals after the directional cue onset and before the saccade execution. The mean accuracies were 80% for 2 directions and 43% for 4 directions. We also quantified the spatial-spectro-temporal contribution ratio using SVM recursive feature elimination (RFE). The channels over the frontal eye field (FEF), supplementary eye field (SEF), and superior parietal lobule (SPL) area were dominantly used for classification. The α-band in the spectral domain and the time bins just after the directional cue onset and just before the saccadic execution were mainly useful for prediction. A saccade based BCI paradigm can be projected in the 2D space, and will hopefully provide an intuitive and convenient communication platform for users.

A novel Brain Computer Interface for classification of social joint attention in autism and comparison of 3 experimental setups: A feasibility study

BACKGROUND

We present a novel virtual-reality P300-based Brain Computer Interface (BCI) paradigm using social cues to direct the focus of attention. We combined interactive immersive virtual-reality (VR) technology with the properties of P300 signals in a training tool which can be used in social attention disorders such as autism spectrum disorder (ASD).

NEW METHOD

We tested the novel social attention training paradigm (P300-based BCI paradigm for rehabilitation of joint-attention skills) in 13 healthy participants, in 3 EEG systems. The more suitable setup was tested online with 4 ASD subjects. Statistical accuracy was assessed based on the detection of P300, using spatial filtering and a Naïve-Bayes classifier.

RESULTS

We compared: 1 - g.Mobilab+ (active dry-electrodes, wireless transmission); 2 - g.Nautilus (active electrodes, wireless transmission); 3 - V-Amp with actiCAP Xpress dry-electrodes. Significant statistical classification was achieved in all systems. g.Nautilus proved to be the best performing system in terms of accuracy in the detection of P300, preparation time, speed and reported comfort. Proof of concept tests in ASD participants proved that this setup is feasible for training joint attention skills in ASD.

COMPARISON WITH EXISTING METHODS

This work provides a unique combination of 'easy-to-use' BCI systems with new technologies such as VR to train joint-attention skills in autism.

CONCLUSIONS

Our P300 BCI paradigm is feasible for future Phase I/II clinical trials to train joint-attention skills, with successful classification within few trials, online in ASD participants. The g.Nautilus system is the best performing one to use with the developed BCI setup.

A Multifunctional Brain-Computer Interface Intended for Home Use: An Evaluation with Healthy Participants and Potential End Users with Dry and Gel-Based Electrodes

Current brain-computer interface (BCIs) software is often tailored to the needs of scientists and technicians and therefore complex to allow for versatile use. To facilitate home use of BCIs a multifunctional P300 BCI with a graphical user interface intended for non-expert set-up and control was designed and implemented. The system includes applications for spelling, web access, entertainment, artistic expression and environmental control. In addition to new software, it also includes new hardware for the recording of electroencephalogram (EEG) signals. The EEG system consists of a small and wireless amplifier attached to a cap that can be equipped with gel-based or dry contact electrodes. The system was systematically evaluated with a healthy sample, and targeted end users of BCI technology, i.e., people with a varying degree of motor impairment tested the BCI in a series of individual case studies. Usability was assessed in terms of effectiveness, efficiency and satisfaction. Feedback of users was gathered with structured questionnaires. Two groups of healthy participants completed an experimental protocol with the gel-based and the dry contact electrodes (N = 10 each). The results demonstrated that all healthy participants gained control over the system and achieved satisfactory to high accuracies with both gel-based and dry electrodes (average error rates of 6 and 13%). Average satisfaction ratings were high, but certain aspects of the system such as the wearing comfort of the dry electrodes and design of the cap, and speed (in both groups) were criticized by some participants. Six potential end users tested the system during supervised sessions. The achieved accuracies varied greatly from no control to high control with accuracies comparable to that of healthy volunteers. Satisfaction ratings of the two end-users that gained control of the system were lower as compared to healthy participants. The advantages and disadvantages of the BCI and its applications are discussed and suggestions are presented for improvements to pave the way for user friendly BCIs intended to be used as assistive technology by persons with severe paralysis.

Brain-Machine Interfaces: From Basic Science to Neuroprostheses and Neurorehabilitation

Brain-machine interfaces (BMIs) combine methods, approaches, and concepts derived from neurophysiology, computer science, and engineering in an effort to establish real-time bidirectional links between living brains and artificial actuators. Although theoretical propositions and some proof of concept experiments on directly linking the brains with machines date back to the early 1960s, BMI research only took off in earnest at the end of the 1990s, when this approach became intimately linked to new neurophysiological methods for sampling large-scale brain activity. The classic goals of BMIs are 1) to unveil and utilize principles of operation and plastic properties of the distributed and dynamic circuits of the brain and 2) to create new therapies to restore mobility and sensations to severely disabled patients. Over the past decade, a wide range of BMI applications have emerged, which considerably expanded these original goals. BMI studies have shown neural control over the movements of robotic and virtual actuators that enact both upper and lower limb functions. Furthermore, BMIs have also incorporated ways to deliver sensory feedback, generated from external actuators, back to the brain. BMI research has been at the forefront of many neurophysiological discoveries, including the demonstration that, through continuous use, artificial tools can be assimilated by the primate brain's body schema. Work on BMIs has also led to the introduction of novel neurorehabilitation strategies. As a result of these efforts, long-term continuous BMI use has been recently implicated with the induction of partial neurological recovery in spinal cord injury patients.

Oculomotor activity parameters of the operator in the P300 brain–computer interface with variating stimulus situations

We tested the hypotheses about the correlation of visual environment properties in the BCI P300 with oculomotor activity and operator efficiency. We varied level of stimulus intensification and the frame surrounding the stimulus elements. So we had four situation: 1) low contrast, without frame; 2) low contrast, with frame; 3) high contrast, without frame; 4) high contrast, with frame. 12 subjects participated. Our study showed that visual environment which provides lowest level of operator’s errors and so the highest efficiency of the BCI P300 workflow combined with lowest fixation dispersion and highest fixation duration. However, various subjects demonstrated the highest level of the efficiency at the different visual environments. We did not define the best type of the visual environment for the most efficient BCI P300 workflow. This results demonstrate the opportunity to use the eyetracking for optimization visual environment of the BCI P300 for most efficient and comfort operator’s workflow. The study was funded by RFH, grant 15-36-01386 “Consistent pattern of organization oculomotor activity in an environ- ment of brain-computer interface”.

Optimizing the Face Paradigm of BCI System by Modified Mismatch Negative Paradigm

Many recent studies have focused on improving the performance of event-related potential (ERP) based brain computer interfaces (BCIs). The use of a face pattern has been shown to obtain high classification accuracies and information transfer rates (ITRs) by evoking discriminative ERPs (N200 and N400) in addition to P300 potentials. Recently, it has been proved that the performance of traditional P300-based BCIs could be improved through a modification of the mismatch pattern. In this paper, a mismatch inverted face pattern (MIF-pattern) was presented to improve the performance of the inverted face pattern (IF-pattern), one of the state of the art patterns used in visual-based BCI systems. Ten subjects attended in this experiment. The result showed that the mismatch inverted face pattern could evoke significantly larger vertex positive potentials (p < 0.05) and N400s (p < 0.05) compared to the inverted face pattern. The classification accuracy (mean accuracy is 99.58%) and ITRs (mean bit rate is 27.88 bit/min) of the mismatch inverted face pattern was significantly higher than that of the inverted face pattern (p < 0.05).

Using the detectability index to predict P300 speller performance

OBJECTIVE

The P300 speller is a popular brain-computer interface (BCI) system that has been investigated as a potential communication alternative for individuals with severe neuromuscular limitations. To achieve acceptable accuracy levels for communication, the system requires repeated data measurements in a given signal condition to enhance the signal-to-noise ratio of elicited brain responses. These elicited brain responses, which are used as control signals, are embedded in noisy electroencephalography (EEG) data. The discriminability between target and non-target EEG responses defines a user's performance with the system. A previous P300 speller model has been proposed to estimate system accuracy given a certain amount of data collection. However, the approach was limited to a static stopping algorithm, i.e. averaging over a fixed number of measurements, and the row-column paradigm. A generalized method that is also applicable to dynamic stopping (DS) algorithms and other stimulus paradigms is desirable.

APPROACH

We developed a new probabilistic model-based approach to predicting BCI performance, where performance functions can be derived analytically or via Monte Carlo methods. Within this framework, we introduce a new model for the P300 speller with the Bayesian DS algorithm, by simplifying a multi-hypothesis to a binary hypothesis problem using the likelihood ratio test. Under a normality assumption, the performance functions for the Bayesian algorithm can be parameterized with the detectability index, a measure which quantifies the discriminability between target and non-target EEG responses.

MAIN RESULTS

Simulations with synthetic and empirical data provided initial verification of the proposed method of estimating performance with Bayesian DS using the detectability index. Analysis of results from previous online studies validated the proposed method.

SIGNIFICANCE

The proposed method could serve as a useful tool to initially assess BCI performance without extensive online testing, in order to estimate the amount of data required to achieve a desired accuracy level.

Exploring Combinations of Different Color and Facial Expression Stimuli for Gaze-Independent BCIs

Background: Some studies have proven that a conventional visual brain computer interface (BCI) based on overt attention cannot be used effectively when eye movement control is not possible. To solve this problem, a novel visual-based BCI system based on covert attention and feature attention has been proposed and was called the gaze-independent BCI. Color and shape difference between stimuli and backgrounds have generally been used in examples of gaze-independent BCIs. Recently, a new paradigm based on facial expression changes has been presented, and obtained high performance. However, some facial expressions were so similar that users couldn't tell them apart, especially when they were presented at the same position in a rapid serial visual presentation (RSVP) paradigm. Consequently, the performance of the BCI is reduced.

New Method: In this paper, we combined facial expressions and colors to optimize the stimuli presentation in the gaze-independent BCI. This optimized paradigm was called the colored dummy face pattern. It is suggested that different colors and facial expressions could help users to locate the target and evoke larger event-related potentials (ERPs). In order to evaluate the performance of this new paradigm, two other paradigms were presented, called the gray dummy face pattern and the colored ball pattern.

Comparison with Existing Method(s): The key point that determined the value of the colored dummy faces stimuli in BCI systems was whether the dummy face stimuli could obtain higher performance than gray faces or colored balls stimuli. Ten healthy participants (seven male, aged 21–26 years, mean 24.5 ± 1.25) participated in our experiment. Online and offline results of four different paradigms were obtained and comparatively analyzed.

Results: The results showed that the colored dummy face pattern could evoke higher P300 and N400 ERP amplitudes, compared with the gray dummy face pattern and the colored ball pattern. Online results showed that the colored dummy face pattern had a significant advantage in terms of classification accuracy (p < 0.05) and information transfer rate (p < 0.05) compared to the other two patterns.

Conclusions: The stimuli used in the colored dummy face paradigm combined color and facial expressions. This had a significant advantage in terms of the evoked P300 and N400 amplitudes and resulted in high classification accuracies and information transfer rates. It was compared with colored ball and gray dummy face stimuli.

Endogenous Sensory Discrimination and Selection by a Fast Brain Switch for a High Transfer Rate Brain-Computer Interface

In this study, we present a novel multi-class brain-computer interface (BCI) for communication and control. In this system, the information processing is shared by the algorithm (computer) and the user (human). Specifically, an electro-tactile cycle was presented to the user, providing the choice (class) by delivering timely sensory input. The user discriminated these choices by his/her endogenous sensory ability and selected the desired choice with an intuitive motor task. This selection was detected by a fast brain switch based on real-time detection of movement-related cortical potentials from scalp EEG. We demonstrated the feasibility of such a system with a four-class BCI, yielding a true positive rate of  ∼ 80% and  ∼ 70%, and an information transfer rate of  ∼ 7 bits/min and  ∼ 5 bits/min, for the movement and imagination selection command, respectively. Furthermore, when the system was extended to eight classes, the throughput of the system was improved, demonstrating the capability of accommodating a large number of classes. Combining the endogenous sensory discrimination with the fast brain switch, the proposed system could be an effective, multi-class, gaze-independent BCI system for communication and control applications.

Faster P300 Classifier Training Using Spatiotemporal Beamforming

The linearly-constrained minimum-variance (LCMV) beamformer is traditionally used as a spatial filter for source localization, but here we consider its spatiotemporal extension for P300 classification. We compare two variants and show that the spatiotemporal LCMV beamformer is at par with state-of-the-art P300 classifiers, but several orders of magnitude faster in training the classifier.

An Auditory-Tactile Visual Saccade-Independent P300 Brain–Computer Interface

Most P300 event-related potential (ERP)-based brain–computer interface (BCI) studies focus on gaze shift-dependent BCIs, which cannot be used by people who have lost voluntary eye movement. However, the performance of visual saccade-independent P300 BCIs is generally poor. To improve saccade-independent BCI performance, we propose a bimodal P300 BCI approach that simultaneously employs auditory and tactile stimuli. The proposed P300 BCI is a vision-independent system because no visual interaction is required of the user. Specifically, we designed a direction-congruent bimodal paradigm by randomly and simultaneously presenting auditory and tactile stimuli from the same direction. Furthermore, the channels and number of trials were tailored to each user to improve online performance. With 12 participants, the average online information transfer rate (ITR) of the bimodal approach improved by 45.43% and 51.05% over that attained, respectively, with the auditory and tactile approaches individually. Importantly, the average online ITR of the bimodal approach, including the break time between selections, reached 10.77 bits/min. These findings suggest that the proposed bimodal system holds promise as a practical visual saccade-independent P300 BCI.

P300-based brain-computer interface (BCI) event-related potentials (ERPs): People with amyotrophic lateral sclerosis (ALS) vs. age-matched controls

OBJECTIVE

Brain-computer interfaces (BCIs) aimed at restoring communication to people with severe neuromuscular disabilities often use event-related potentials (ERPs) in scalp-recorded EEG activity. Up to the present, most research and development in this area has been done in the laboratory with young healthy control subjects. In order to facilitate the development of BCI most useful to people with disabilities, the present study set out to: (1) determine whether people with amyotrophic lateral sclerosis (ALS) and healthy, age-matched volunteers (HVs) differ in the speed and accuracy of their ERP-based BCI use; (2) compare the ERP characteristics of these two groups; and (3) identify ERP-related factors that might enable improvement in BCI performance for people with disabilities.

METHODS

Sixteen EEG channels were recorded while people with ALS or healthy age-matched volunteers (HVs) used a P300-based BCI. The subjects with ALS had little or no remaining useful motor control (mean ALS Functional Rating Scale-Revised 9.4 (±9.5SD) (range 0-25)). Each subject attended to a target item as the items in a 6×6 visual matrix flashed. The BCI used a stepwise linear discriminant function (SWLDA) to determine the item the user wished to select (i.e., the target item). Offline analyses assessed the latencies, amplitudes, and locations of ERPs to the target and non-target items for people with ALS and age-matched control subjects.

RESULTS

BCI accuracy and communication rate did not differ significantly between ALS users and HVs. Although ERP morphology was similar for the two groups, their target ERPs differed significantly in the location and amplitude of the late positivity (P300), the amplitude of the early negativity (N200), and the latency of the late negativity (LN).

CONCLUSIONS

The differences in target ERP components between people with ALS and age-matched HVs are consistent with the growing recognition that ALS may affect cortical function. The development of BCIs for use by this population may begin with studies in HVs but also needs to include studies in people with ALS. Their differences in ERP components may affect the selection of electrode montages, and might also affect the selection of presentation parameters (e.g., matrix design, stimulation rate).

SIGNIFICANCE

P300-based BCI performance in people severely disabled by ALS is similar to that of age-matched control subjects. At the same time, their ERP components differ to some degree from those of controls. Attention to these differences could contribute to the development of BCIs useful to those with ALS and possibly to others with severe neuromuscular disabilities.

Comparison of eye tracking, electrooculography and an auditory brain-computer interface for binary communication: a case study with a participant in the locked-in state

BACKGROUND

In this study, we evaluated electrooculography (EOG), an eye tracker and an auditory brain-computer interface (BCI) as access methods to augmentative and alternative communication (AAC). The participant of the study has been in the locked-in state (LIS) for 6 years due to amyotrophic lateral sclerosis. He was able to communicate with slow residual eye movements, but had no means of partner independent communication. We discuss the usability of all tested access methods and the prospects of using BCIs as an assistive technology.

METHODS

Within four days, we tested whether EOG, eye tracking and a BCI would allow the participant in LIS to make simple selections. We optimized the parameters in an iterative procedure for all systems.

RESULTS

The participant was able to gain control over all three systems. Nonetheless, due to the level of proficiency previously achieved with his low-tech AAC method, he did not consider using any of the tested systems as an additional communication channel. However, he would consider using the BCI once control over his eye muscles would no longer be possible. He rated the ease of use of the BCI as the highest among the tested systems, because no precise eye movements were required; but also as the most tiring, due to the high level of attention needed to operate the BCI.

CONCLUSIONS

In this case study, the partner based communication was possible due to the good care provided and the proficiency achieved by the interlocutors. To ease the transition from a low-tech AAC method to a BCI once control over all muscles is lost, it must be simple to operate. For persons, who rely on AAC and are affected by a progressive neuromuscular disease, we argue that a complementary approach, combining BCIs and standard assistive technology, can prove valuable to achieve partner independent communication and ease the transition to a purely BCI based approach. Finally, we provide further evidence for the importance of a user-centered approach in the design of new assistive devices.

EEG resolutions in detecting and decoding finger movements from spectral analysis

Mu/beta rhythms are well-studied brain activities that originate from sensorimotor cortices. These rhythms reveal spectral changes in alpha and beta bands induced by movements of different body parts, e.g., hands and limbs, in electroencephalography (EEG) signals. However, less can be revealed in them about movements of different fine body parts that activate adjacent brain regions, such as individual fingers from one hand. Several studies have reported spatial and temporal couplings of rhythmic activities at different frequency bands, suggesting the existence of well-defined spectral structures across multiple frequency bands. In the present study, spectral principal component analysis (PCA) was applied on EEG data, obtained from a finger movement task, to identify cross-frequency spectral structures. Features from identified spectral structures were examined in their spatial patterns, cross-condition pattern changes, detection capability of finger movements from resting, and decoding performance of individual finger movements in comparison to classic mu/beta rhythms. These new features reveal some similar, but more different spatial and spectral patterns as compared with classic mu/beta rhythms. Decoding results further indicate that these new features (91%) can detect finger movements much better than classic mu/beta rhythms (75.6%). More importantly, these new features reveal discriminative information about movements of different fingers (fine body-part movements), which is not available in classic mu/beta rhythms. The capability in decoding fingers (and hand gestures in the future) from EEG will contribute significantly to the development of non-invasive BCI and neuroprosthesis with intuitive and flexible controls.

Re-establishing the disrupted sensorimotor loop in deafferented and deefferented people: The case of spinal cord injuries

Acting efficiently in the world depends on the activity of motor and somatosensory systems, the integration of which is necessary for the proper functioning of the sensorimotor loop (SL). Profound alterations of SL functioning follow spinal cord injury (SCI), a condition that brings about a disconnection of the body from the brain. Such disconnection creates a substantial deprivation of somatosensorial inputs and motor outputs. Consequent somatic deficits and motor paralysis affect the body below the lesion level. A complete restoration of normal functions of the SL cannot be expected until basic neuroscience has found a way to re-establish the interrupted neural connectivity. Meanwhile, studies should focus on the development of technical solutions for dealing with the disruption of the sensorimotor loop. This review discusses the structural and functional adaptive reorganization of the brain after SCI, and the maladaptive mechanisms that impact on the processing of body related information, which alter motor imagery strategies and EEG signals. Studies that show how residual functions (e.g. face tactile sensitivity) may help people to restore a normal body image are also reviewed. Finally, data on how brain and residual body signals may be used to improve brain computer interface systems is discussed in relation to the issue of how such systems may help SCI people to re-enter the world and interact with objects and other individuals.

Rapid P300 brain-computer interface communication with a head-mounted display

Visual ERP (P300) based brain-computer interfaces (BCIs) allow for fast and reliable spelling and are intended as a muscle-independent communication channel for people with severe paralysis. However, they require the presentation of visual stimuli in the field of view of the user. A head-mounted display could allow convenient presentation of visual stimuli in situations, where mounting a conventional monitor might be difficult or not feasible (e.g., at a patient's bedside). To explore if similar accuracies can be achieved with a virtual reality (VR) headset compared to a conventional flat screen monitor, we conducted an experiment with 18 healthy participants. We also evaluated it with a person in the locked-in state (LIS) to verify that usage of the headset is possible for a severely paralyzed person. Healthy participants performed online spelling with three different display methods. In one condition a 5 × 5 letter matrix was presented on a conventional 22 inch TFT monitor. Two configurations of the VR headset were tested. In the first (glasses A), the same 5 × 5 matrix filled the field of view of the user. In the second (glasses B), single letters of the matrix filled the field of view of the user. The participant in the LIS tested the VR headset on three different occasions (glasses A condition only). For healthy participants, average online spelling accuracies were 94% (15.5 bits/min) using three flash sequences for spelling with the monitor and glasses A and 96% (16.2 bits/min) with glasses B. In one session, the participant in the LIS reached an online spelling accuracy of 100% (10 bits/min) using the glasses A condition. We also demonstrated that spelling with one flash sequence is possible with the VR headset for healthy users (mean: 32.1 bits/min, maximum reached by one user: 71.89 bits/min at 100% accuracy). We conclude that the VR headset allows for rapid P300 BCI communication in healthy users and may be a suitable display option for severely paralyzed persons.