Which Physiological Components are More Suitable for Visual ERP Based Brain–Computer Interface? A Preliminary MEG/EEG Study

Task-irrelevant features can be ignored in feature-based encoding

The present study aimed to explore whether individuals could selectively remember task-relevant features while ignoring task-irrelevant features for given items. Participants were initially asked to remember the task-relevant feature of one (low load), two (medium load), or four (high load) items, while ignoring their task-irrelevant features. Participants were required to make responses to the target in the subsequent search task, while being presented with distractors that contained either task-irrelevant or task-relevant features. No features matched with the studied items in the neutral trials. The items' color was manipulated as a task-relevant feature in Experiment 1, while their shape was designated as a task-irrelevant feature. Conversely, the items' shape was manipulated as a task-relevant feature in Experiment 2, and their color was designated as task-irrelevant. The event-related potentials evoked by the visual search task were also examined. The results showed that, in both experiments, 1) The response time showed no differences between task-irrelevant trials and neutral trials among different load conditions, suggesting that the task-irrelevant distractors may not slow down the target searching. 2) The magnitude of the target-elicited N2pc was similar between the neutral and the task-irrelevant trials among different load conditions, indicating that the task-irrelevant distractor received no attention and had no effect on the target processing. The results indicated that the task-irrelevant features were suppressed or completely disregarded.

Enhanced Spatial Division Multiple Access BCI Performance via Incorporating MEG With EEG

Spatial division multiple access (SDMA) is a way of encoding BCI systems based on spatial distribution of brain signal characteristics. However, SDMA-BCI based on EEG had poor system performance limited by spatial resolution. MEG-EEG fusion modality analysis can help solve this problem. According retina-cortical relationship, this study used stimulus out of the central visual field and tiny fixation points to construct a 16-command SDMA coded MEG-EEG fusion modality BCI system. We achieved this by synchronously acquiring MEG and EEG signals from 10 subjects. We compared the spatiotemporal features between MEG and EEG by analyzing signals in the occipital region. We fused MEG and EEG modalities without any signal processing and used the multi-class discriminative canonical pattern matching (Multi-DCPM) algorithm to evaluate and compare the system performance of EEG, MEG, and MEG-EEG fusion modalities. The result showed that MEG and EEG had obvious differences in spatial distribution characteristics. MEG improves offline classification accuracy of the 16 fixation points by 27.81% over EEG at 4s data length. Specially, the MEG-EEG fusion modality achieves an impressive average offline accuracy of 91.71%, which was a significant improvement over MEG (p<0.01, ANOVA). The MEG-EEG fusion modality achieved average information transfer rate (ITR) of 60.74 bits/min with a data length of 1 s, which was a 14% improvement over MEG. The MEG-EEG fusion modality significantly enhanced the spatial features and performance of SDMA-encoded BCIs. These results highlight the potential and feasibility of MEG-EEG fusion modality BCI, and provide theoretical insights and practical value for promoting the further development and application of SDMA in BCI.

Brain activity underlying visual search in depth when viewing volumetric multiplanar images

The study investigated the cortical activity associated with 3D and 2D image perception on a volumetric multiplanar display by analyzing event-related potentials (ERPs) and power spectral density (PSD). In this study, we used a volumetric multiplanar display to present visual targets, and the brain signals were recorded via an EEG amplifier and analyzed using the EEGLAB toolbox on MATLAB. The study found no significant differences in amplitude between the 3D and 2D conditions across five occipital and parietal electrodes. However, there was a significant difference in latency of the P3 component on the Pz electrode. The analysis of PSD showed no significant differences between the two conditions, although there was a slightly higher alpha and beta activity observed in the 2D visualization. The study concluded that 3D image representation on a volumetric multiplanar display has no more sensory or cognitive load on the human brain than 2D representation, and that depth perception on a multiplanar display requires less brain activity.

On the Influence of Aging on Classification Performance in the Visual EEG Oddball Paradigm Using Statistical and Temporal Features

The utilization of a non-invasive electroencephalogram (EEG) as an input sensor is a common approach in the field of the brain-computer interfaces (BCI). However, the collected EEG data pose many challenges, one of which may be the age-related variability of event-related potentials (ERPs), which are often used as primary EEG BCI signal features. To assess the potential effects of aging, a sample of 27 young and 43 older healthy individuals participated in a visual oddball study, in which they passively viewed frequent stimuli among randomly occurring rare stimuli while being recorded with a 32-channel EEG set. Two types of EEG datasets were created to train the classifiers, one consisting of amplitude and spectral features in time and another with extracted time-independent statistical ERP features. Among the nine classifiers tested, linear classifiers performed best. Furthermore, we show that classification performance differs between dataset types. When temporal features were used, maximum individuals' performance scores were higher, had lower variance, and were less affected overall by within-class differences such as age. Finally, we found that the effect of aging on classification performance depends on the classifier and its internal feature ranking. Accordingly, performance will differ if the model favors features with large within-class differences. With this in mind, care must be taken in feature extraction and selection to find the correct features and consequently avoid potential age-related performance degradation in practice.

Merging Brain-Computer Interface P300 speller datasets: Perspectives and pitfalls

Background

In the last decades, the P300 Speller paradigm was replicated in many experiments, and collected data were released to the public domain to allow research groups, particularly those in the field of machine learning, to test and improve their algorithms for higher performances of brain-computer interface (BCI) systems. Training data is needed to learn the identification of brain activity. The more training data are available, the better the algorithms will perform. The availability of larger datasets is highly desirable, eventually obtained by merging datasets from different repositories. The main obstacle to such merging is that all public datasets are released in various file formats because no standard way is established to share these data. Additionally, all datasets necessitate reading documents or scientific papers to retrieve relevant information, which prevents automating the processing. In this study, we thus adopted a unique file format to demonstrate the importance of having a standard and to propose which information should be stored and why.

Methods

We described our process to convert a dozen of P300 Speller datasets and reported the main encountered problems while converting them into the same file format. All the datasets are characterized by the same 6 × 6 matrix of alphanumeric symbols (characters and numbers or symbols) and by the same subset of acquired signals (8 EEG sensors at the same recording sites).

Results and discussion

Nearly a million stimuli were converted, relative to about 7000 spelled characters and belonging to 127 subjects. The converted stimuli represent the most extensively available platform for training and testing new algorithms on the specific paradigm - the P300 Speller. The platform could potentially allow exploring transfer learning procedures to reduce or eliminate the time needed for training a classifier to improve the performance and accuracy of such BCI systems.

Spatial and Time Domain Feature of ERP Speller System Extracted via Convolutional Neural Network

Feature of event-related potential (ERP) has not been completely understood and illiteracy problem remains unsolved. To this end, P300 peak has been used as the feature of ERP in most brain–computer interface applications, but subjects who do not show such peak are common. Recent development of convolutional neural network provides a way to analyze spatial and temporal features of ERP. Here, we train the convolutional neural network with 2 convolutional layers whose feature maps represented spatial and temporal features of event-related potential. We have found that nonilliterate subjects' ERP show high correlation between occipital lobe and parietal lobe, whereas illiterate subjects only show correlation between neural activities from frontal lobe and central lobe. The nonilliterates showed peaks in P300, P500, and P700, whereas illiterates mostly showed peaks in around P700. P700 was strong in both subjects. We found that P700 peak may be the key feature of ERP as it appears in both illiterate and nonilliterate subjects.

Predictors for drug effects with brain disease: Shed new light from EEG parameters to brain connectomics

Though researchers spent a lot of effort to develop treatments for neuropsychiatric disorders, the poor translation of drug efficacy data from animals to human hampered the success of these therapeutic approaches in human. Pharmaceutical industry is challenged by low clinical success rates for new drug registration. To maximize the success in drug development, biomarkers are required to act as surrogate end points and predictors of drug effects. The pathology of brain disease could be in part due to synaptic dysfunction. Electroencephalogram (EEG), generating from the result of the postsynaptic potential discharge between cells, could be a potential measure to bridge the gaps between animal and human data. Here we discuss recent progress on using relevant EEG characteristics and brain connectomics as biomarkers to monitor drug effects and measure cognitive changes on animal models and human in real-time. It is expected that the novel approach, i.e. EEG connectomics, will offer a deeper understanding on the drug efficacy at a microcirculatory level, which will be useful to support the development of new treatments for neuropsychiatric disorders.

Spatiotemporal Beamforming: A Transparent and Unified Decoding Approach to Synchronous Visual Brain-Computer Interfacing

Brain-Computer Interfaces (BCIs) decode brain activity with the aim to establish a direct communication channel with an external device. Albeit they have been hailed to (re-)establish communication in persons suffering from severe motor- and/or communication disabilities, only recently BCI applications have been challenging other assistive technologies. Owing to their considerably increased performance and the advent of affordable technological solutions, BCI technology is expected to trigger a paradigm shift not only in assistive technology but also in the way we will interface with technology. However, the flipside of the quest for accuracy and speed is most evident in EEG-based visual BCI where it has led to a gamut of increasingly complex classifiers, tailored to the needs of specific stimulation paradigms and use contexts. In this contribution, we argue that spatiotemporal beamforming can serve several synchronous visual BCI paradigms. We demonstrate this for three popular visual paradigms even without attempting to optimizing their electrode sets. For each selectable target, a spatiotemporal beamformer is applied to assess whether the corresponding signal-of-interest is present in the preprocessed multichannel EEG signals. The target with the highest beamformer output is then selected by the decoder (maximum selection). In addition to this simple selection rule, we also investigated whether interactions between beamformer outputs could be employed to increase accuracy by combining the outputs for all targets into a feature vector and applying three common classification algorithms. The results show that the accuracy of spatiotemporal beamforming with maximum selection is at par with that of the classification algorithms and interactions between beamformer outputs do not further improve that accuracy.

A Comparative Study on the Detection of Covert Attention in Event-Related EEG and MEG Signals to Control a BCI

In brain-computer interface (BCI) applications the detection of neural processing as revealed by event-related potentials (ERPs) is a frequently used approach to regain communication for people unable to interact through any peripheral muscle control. However, the commonly used electroencephalography (EEG) provides signals of low signal-to-noise ratio, making the systems slow and inaccurate. As an alternative noninvasive recording technique, the magnetoencephalography (MEG) could provide more advantageous electrophysiological signals due to a higher number of sensors and the magnetic fields not being influenced by volume conduction. We investigated whether MEG provides higher accuracy in detecting event-related fields (ERFs) compared to detecting ERPs in simultaneously recorded EEG, both evoked by a covert attention task, and whether a combination of the modalities is advantageous. In our approach, a detection algorithm based on spatial filtering is used to identify ERP/ERF components in a data-driven manner. We found that MEG achieves higher decoding accuracy (DA) compared to EEG and that the combination of both further improves the performance significantly. However, MEG data showed poor performance in cross-subject classification, indicating that the algorithm's ability for transfer learning across subjects is better in EEG. Here we show that BCI control by covert attention is feasible with EEG and MEG using a data-driven spatial filter approach with a clear advantage of the MEG regarding DA but with a better transfer learning in EEG.

The Role of the Interplay between Stimulus Type and Timing in Explaining BCI-Illiteracy for Visual P300-Based Brain-Computer Interfaces

Visual P300-based Brain-Computer Interface (BCI) spellers enable communication or interaction with the environment by flashing elements in a matrix and exploiting consequent changes in end-user's brain activity. Despite research efforts, performance variability and BCI-illiteracy still are critical issues for real world applications. Moreover, there is a quite unaddressed kind of BCI-illiteracy, which becomes apparent when the same end-user operates BCI-spellers intended for different applications: our aim is to understand why some well performers can become BCI-illiterate depending on speller type. We manipulated stimulus type (factor STIM: either characters or icons), color (factor COLOR: white, green) and timing (factor SPEED: fast, slow). Each BCI session consisted of training (without feedback) and performance phase (with feedback), both in copy-spelling. For fast flashing spellers, we observed a performance worsening for white icon-speller. Our findings are consistent with existing results reported on end-users using identical white×fast spellers, indicating independence of worsening trend from users' group. The use of slow stimulation timing shed a new light on the perceptual and cognitive phenomena related to the use of a BCI-speller during both the training and the performance phase. We found a significant STIM main effect for the N1 component on P z and PO7 during the training phase and on PO8 during the performance phase, whereas in both phases neither the STIM×COLOR interaction nor the COLOR main effect was statistically significant. After collapsing data for factor COLOR, it emerged a statistically significant modulation of N1 amplitude depending to the phase of BCI session: N1 was more negative for icons than for characters both on P z and PO7 (training), whereas the opposite modulation was observed for PO8 (performance). Results indicate that both feedback and expertise with respect to the stimulus type can modulate the N1 component and that icons require more perceptual analysis. Therefore, fast flashing is likely to be more detrimental for end-users' performance in case of icon-spellers. In conclusion, the interplay between stimulus type and timing seems relevant for a satisfactory and efficient end-user's BCI-experience.

Fusion of P300 and eye-tracker data for spelling using BCI2000

OBJECTIVE

Various augmentative and alternative communication (AAC) devices have been developed in order to aid communication for individuals with communication disorders. Recently, there has been interest in combining EEG data and eye-gaze data with the goal of developing a hybrid (or 'fused') BCI (hBCI) AAC system. This work explores the effectiveness of a speller that fuses data from an eye-tracker and the P300 speller in order to create a hybrid P300 speller.

APPROACH

This hybrid speller collects both eye-tracking and EEG data in parallel, and the user spells characters on the screen in the same way that they would if they were only using the P300 speller. Online and offline experiments were performed. The online experiments measured the performance of the speller for sixteen non-disabled participants, while the offline simulations were used to assess the robustness of the hybrid system.

MAIN RESULTS

Online results showed that for fifteen non-disabled participants, using eye-gaze in a Bayesian framework with EEG data from the P300 speller improved accuracy ([Formula: see text], [Formula: see text], [Formula: see text] for estimated, medium and high variance configurations) and reduced the average number of flashes required to spell a character compared to the standard P300 speller that relies solely on EEG data ([Formula: see text], [Formula: see text], [Formula: see text] for estimated, medium and high variance configurations). Offline simulations indicate that the system provides more robust performance than a standalone eye gaze system.

SIGNIFICANCE

The results of this work on non-disabled participants shows the potential efficacy of hybrid P300 and eye-tracker speller. Further validation on the amyotrophic lateral sceloris population is needed to assess the benefit of this hybrid system.

Testing a novel method for improving wayfinding by means of a P3b Virtual Reality Visual Paradigm in normal aging

BACKGROUND

We propose a virtual reality (VR) model, reproducing a house environment, where color modification of target places, obtainable by home automation in a real ambient, was tested by means of a P3b paradigm. The target place (bathroom door) was designed to be recognized during a virtual wayfinding in a realistic reproduction of a house environment. Different color and luminous conditions, easily obtained in the real ambient from a remote home automation control, were applied to the target and standard places, all the doors being illuminated in white (W), and only target doors colored with a green (G) or red (R) spotlight. Three different Virtual Environments (VE) were depicted, as the bathroom was designed in the aisle (A), living room (L) and bedroom (B). EEG was recorded from 57 scalp electrodes in 10 healthy subjects in the 60-80 year age range (O-old group) and 12 normal cases in the 20-30 year age range (Y-young group).

RESULTS

In Young group, all the target stimuli determined a significant increase in P3b amplitude on the parietal, occipital and central electrodes compared to frequent stimuli condition, whatever was the color of the target door, while in elderly group the P3b obtained by the green and red colors was significantly different from the frequent stimulus, on the parietal, occipital, and central derivations, while the White stimulus did not evoke a significantly larger P3b with respect to frequent stimulus.

DISCUSSION

The modulation of P3b amplitude, obtained by color and luminance change of target place, suggests that cortical resources, able to compensate the age-related progressive loss of cognitive performance, need to be facilitated even in normal elderly. The event-related responses obtained by virtual reality may be a reliable method to test the environmental feasibility to age-related cognitive changes.

Neurofeedback Therapy for Enhancing Visual Attention: State-of-the-Art and Challenges

We have witnessed a rapid development of brain-computer interfaces (BCIs) linking the brain to external devices. BCIs can be utilized to treat neurological conditions and even to augment brain functions. BCIs offer a promising treatment for mental disorders, including disorders of attention. Here we review the current state of the art and challenges of attention-based BCIs, with a focus on visual attention. Attention-based BCIs utilize electroencephalograms (EEGs) or other recording techniques to generate neurofeedback, which patients use to improve their attention, a complex cognitive function. Although progress has been made in the studies of neural mechanisms of attention, extraction of attention-related neural signals needed for BCI operations is a difficult problem. To attain good BCI performance, it is important to select the features of neural activity that represent attentional signals. BCI decoding of attention-related activity may be hindered by the presence of different neural signals. Therefore, BCI accuracy can be improved by signal processing algorithms that dissociate signals of interest from irrelevant activities. Notwithstanding recent progress, optimal processing of attentional neural signals remains a fundamental challenge for the development of efficient therapies for disorders of attention.

BCI in practice

Brain-computer interfaces are systems that use signals recorded from the brain to enable communication and control applications for individuals who have impaired function. This technology has developed to the point that it is now being used by individuals who can actually benefit from it. However, there are several outstanding issues that prevent widespread use. These include the ease of obtaining high-quality recordings by home users, the speed, and accuracy of current devices and adapting applications to the needs of the user. In this chapter, we discuss some of these unsolved issues.

Exploring Combinations of Different Color and Facial Expression Stimuli for Gaze-Independent BCIs

Background: Some studies have proven that a conventional visual brain computer interface (BCI) based on overt attention cannot be used effectively when eye movement control is not possible. To solve this problem, a novel visual-based BCI system based on covert attention and feature attention has been proposed and was called the gaze-independent BCI. Color and shape difference between stimuli and backgrounds have generally been used in examples of gaze-independent BCIs. Recently, a new paradigm based on facial expression changes has been presented, and obtained high performance. However, some facial expressions were so similar that users couldn't tell them apart, especially when they were presented at the same position in a rapid serial visual presentation (RSVP) paradigm. Consequently, the performance of the BCI is reduced.

New Method: In this paper, we combined facial expressions and colors to optimize the stimuli presentation in the gaze-independent BCI. This optimized paradigm was called the colored dummy face pattern. It is suggested that different colors and facial expressions could help users to locate the target and evoke larger event-related potentials (ERPs). In order to evaluate the performance of this new paradigm, two other paradigms were presented, called the gray dummy face pattern and the colored ball pattern.

Comparison with Existing Method(s): The key point that determined the value of the colored dummy faces stimuli in BCI systems was whether the dummy face stimuli could obtain higher performance than gray faces or colored balls stimuli. Ten healthy participants (seven male, aged 21–26 years, mean 24.5 ± 1.25) participated in our experiment. Online and offline results of four different paradigms were obtained and comparatively analyzed.

Results: The results showed that the colored dummy face pattern could evoke higher P300 and N400 ERP amplitudes, compared with the gray dummy face pattern and the colored ball pattern. Online results showed that the colored dummy face pattern had a significant advantage in terms of classification accuracy (p < 0.05) and information transfer rate (p < 0.05) compared to the other two patterns.

Conclusions: The stimuli used in the colored dummy face paradigm combined color and facial expressions. This had a significant advantage in terms of the evoked P300 and N400 amplitudes and resulted in high classification accuracies and information transfer rates. It was compared with colored ball and gray dummy face stimuli.

Concentration on performance with P300-based BCI systems: a matter of interface features

People who suffer from severe motor disabilities have difficulties to communicate with others or to interact with their environment using natural, i.e., muscular channels. These limitations can be overcome to some extent by using brain-computer interfaces (BCIs), because such systems allow users to communicate on the basis of their brain activity only. Among the several types of BCIs for spelling purposes, those that rely on the P300 event related potential-P300-based spellers-are chosen preferentially due to their high reliability. However, they demand from the user to sustain his/her attention to the desired character over a relatively long period of time. Therefore, the user's capacity to concentrate can affect his/her performance with a P300-based speller. The aim of this study was to test this hypothesis using three different interfaces: one based on the classic P300 speller paradigm, another also based on that speller but including a word predictor, and a third one that was based on the T9 interface developed for mobile phones. User performance was assessed by measuring the time to complete a spelling task and the accuracy of character selection. The d2 test was applied to assess attention and concentration. Sample (N = 14) was divided into two groups basing on of concentration scores. As a result, performance was better with the predictor-enriched interfaces: less time was needed to solve the task and participants made fewer errors (p < .05). There were also significant effects of concentration (p < .05) on performance with the standard P300 speller. In conclusion, the performance of those users with lower concentration level can be improved by providing BCIs with more interactive interfaces. These findings provide substantial evidence in order to highlight the impact of psychological features on BCI performance and should be taken into account for future assistive technology systems.

A Fuzzy Integral Ensemble Method in Visual P300 Brain-Computer Interface

We evaluate the possibility of application of combination of classifiers using fuzzy measures and integrals to Brain-Computer Interface (BCI) based on electroencephalography. In particular, we present an ensemble method that can be applied to a variety of systems and evaluate it in the context of a visual P300-based BCI. Offline analysis of data relative to 5 subjects lets us argue that the proposed classification strategy is suitable for BCI. Indeed, the achieved performance is significantly greater than the average of the base classifiers and, broadly speaking, similar to that of the best one. Thus the proposed methodology allows realizing systems that can be used by different subjects without the need for a preliminary configuration phase in which the best classifier for each user has to be identified. Moreover, the ensemble is often capable of detecting uncertain situations and turning them from misclassifications into abstentions, thereby improving the level of safety in BCI for environmental or device control.

P300-based brain-computer interface (BCI) event-related potentials (ERPs): People with amyotrophic lateral sclerosis (ALS) vs. age-matched controls

OBJECTIVE

Brain-computer interfaces (BCIs) aimed at restoring communication to people with severe neuromuscular disabilities often use event-related potentials (ERPs) in scalp-recorded EEG activity. Up to the present, most research and development in this area has been done in the laboratory with young healthy control subjects. In order to facilitate the development of BCI most useful to people with disabilities, the present study set out to: (1) determine whether people with amyotrophic lateral sclerosis (ALS) and healthy, age-matched volunteers (HVs) differ in the speed and accuracy of their ERP-based BCI use; (2) compare the ERP characteristics of these two groups; and (3) identify ERP-related factors that might enable improvement in BCI performance for people with disabilities.

METHODS

Sixteen EEG channels were recorded while people with ALS or healthy age-matched volunteers (HVs) used a P300-based BCI. The subjects with ALS had little or no remaining useful motor control (mean ALS Functional Rating Scale-Revised 9.4 (±9.5SD) (range 0-25)). Each subject attended to a target item as the items in a 6×6 visual matrix flashed. The BCI used a stepwise linear discriminant function (SWLDA) to determine the item the user wished to select (i.e., the target item). Offline analyses assessed the latencies, amplitudes, and locations of ERPs to the target and non-target items for people with ALS and age-matched control subjects.

RESULTS

BCI accuracy and communication rate did not differ significantly between ALS users and HVs. Although ERP morphology was similar for the two groups, their target ERPs differed significantly in the location and amplitude of the late positivity (P300), the amplitude of the early negativity (N200), and the latency of the late negativity (LN).

CONCLUSIONS

The differences in target ERP components between people with ALS and age-matched HVs are consistent with the growing recognition that ALS may affect cortical function. The development of BCIs for use by this population may begin with studies in HVs but also needs to include studies in people with ALS. Their differences in ERP components may affect the selection of electrode montages, and might also affect the selection of presentation parameters (e.g., matrix design, stimulation rate).

SIGNIFICANCE

P300-based BCI performance in people severely disabled by ALS is similar to that of age-matched control subjects. At the same time, their ERP components differ to some degree from those of controls. Attention to these differences could contribute to the development of BCIs useful to those with ALS and possibly to others with severe neuromuscular disabilities.

Simultaneous detection of P300 and steady-state visually evoked potentials for hybrid brain-computer interface

Objective

We study the feasibility of a hybrid Brain-Computer Interface (BCI) combining simultaneous visual oddball and Steady-State Visually Evoked Potential (SSVEP) paradigms, where both types of stimuli are superimposed on a computer screen. Potentially, such a combination could result in a system being able to operate faster than a purely P300-based BCI and encode more targets than a purely SSVEP-based BCI.

Approach

We analyse the interactions between the brain responses of the two paradigms, and assess the possibility to detect simultaneously the brain activity evoked by both paradigms, in a series of 3 experiments where EEG data are analysed offline.

Main Results

Despite differences in the shape of the P300 response between pure oddball and hybrid condition, we observe that the classification accuracy of this P300 response is not affected by the SSVEP stimulation. We do not observe either any effect of the oddball stimulation on the power of the SSVEP response in the frequency of stimulation. Finally results from the last experiment show the possibility of detecting both types of brain responses simultaneously and suggest not only the feasibility of such hybrid BCI but also a gain over pure oddball- and pure SSVEP-based BCIs in terms of communication rate.

Coherent Activity in Bilateral Parieto-Occipital Cortices during P300-BCI Operation

The visual P300 brain–computer interface (BCI), a popular system for electroencephalography (EEG)-based BCI, uses the P300 event-related potential to select an icon arranged in a flicker matrix. In earlier studies, we used green/blue (GB) luminance and chromatic changes in the P300-BCI system and reported that this luminance and chromatic flicker matrix was associated with better performance and greater subject comfort compared with the conventional white/gray (WG) luminance flicker matrix. To highlight areas involved in improved P300-BCI performance, we used simultaneous EEG–fMRI recordings and showed enhanced activities in bilateral and right lateralized parieto-occipital areas. Here, to capture coherent activities of the areas during P300-BCI, we collected whole-head 306-channel magnetoencephalography data. When comparing functional connectivity between the right and left parieto-occipital channels, significantly greater functional connectivity in the alpha band was observed under the GB flicker matrix condition than under the WG flicker matrix condition. Current sources were estimated with a narrow-band adaptive spatial filter, and mean imaginary coherence was computed in the alpha band. Significantly greater coherence was observed in the right posterior parietal cortex under the GB than under the WG condition. Re-analysis of previous EEG-based P300-BCI data showed significant correlations between the power of the coherence of the bilateral parieto-occipital cortices and their performance accuracy. These results suggest that coherent activity in the bilateral parieto-occipital cortices plays a significant role in effectively driving the P300-BCI.

Utilizing a language model to improve online dynamic data collection in P300 spellers

P300 spellers provide a means of communication for individuals with severe physical limitations, especially those with locked-in syndrome, such as amyotrophic lateral sclerosis. However, P300 speller use is still limited by relatively low communication rates due to the multiple data measurements that are required to improve the signal-to-noise ratio of event-related potentials for increased accuracy. Therefore, the amount of data collection has competing effects on accuracy and spelling speed. Adaptively varying the amount of data collection prior to character selection has been shown to improve spelling accuracy and speed. The goal of this study was to optimize a previously developed dynamic stopping algorithm that uses a Bayesian approach to control data collection by incorporating a priori knowledge via a language model. Participants ( n = 17) completed online spelling tasks using the dynamic stopping algorithm, with and without a language model. The addition of the language model resulted in improved participant performance from a mean theoretical bit rate of 46.12 bits/min at 88.89% accuracy to 54.42 bits/min ( ) at 90.36% accuracy.

Noninvasive brain-computer interfaces for augmentative and alternative communication

Brain-computer interfaces (BCIs) promise to provide a novel access channel for assistive technologies, including augmentative and alternative communication (AAC) systems, to people with severe speech and physical impairments (SSPI). Research on the subject has been accelerating significantly in the last decade and the research community took great strides toward making BCI-AAC a practical reality to individuals with SSPI. Nevertheless, the end goal has still not been reached and there is much work to be done to produce real-world-worthy systems that can be comfortably, conveniently, and reliably used by individuals with SSPI with help from their families and care givers who will need to maintain, setup, and debug the systems at home. This paper reviews reports in the BCI field that aim at AAC as the application domain with a consideration on both technical and clinical aspects.

Gamma band activity associated with BCI performance: simultaneous MEG/EEG study

While brain computer interface (BCI) can be employed with patients and healthy subjects, there are problems that must be resolved before BCI can be useful to the public. In the most popular motor imagery (MI) BCI system, a significant number of target users (called “BCI-Illiterates”) cannot modulate their neuronal signals sufficiently to use the BCI system. This causes performance variability among subjects and even among sessions within a subject. The mechanism of such BCI-Illiteracy and possible solutions still remain to be determined. Gamma oscillation is known to be involved in various fundamental brain functions, and may play a role in MI. In this study, we investigated the association of gamma activity with MI performance among subjects. Ten simultaneous MEG/EEG experiments were conducted; MI performance for each was estimated by EEG data, and the gamma activity associated with BCI performance was investigated with MEG data. Our results showed that gamma activity had a high positive correlation with MI performance in the prefrontal area. This trend was also found across sessions within one subject. In conclusion, gamma rhythms generated in the prefrontal area appear to play a critical role in BCI performance.

The component structure of event-related potentials in the p300 speller paradigm

We investigated the componential structure of event-related potentials elicited while participants use the P300 BCI. Six healthy participants "typed" all characters in a 6 × 6 matrix twice in a random sequence. A principal component analysis indicated that in addition to the P300, target flashes elicited an earlier frontal positivity, possibly a Novelty P3. The amplitudes of both P300 and the Novelty P3 varied with the matrix row in which the target character was located. However, the P300 elicited by row flashes was largest for targets in the lower part of the matrix, whereas the Novelty P3 elicited by column flashes was largest in the top part. Classification accuracy using stepwise linear discriminant analysis mirrored the pattern in the Novelty P3 (an accuracy difference of 0.1 between rows 1 and 6). When separate classifiers were generated to rely solely on the P300 or solely on the Novelty P3, the latter function led to higher accuracy (a mean accuracy difference of about 0.2 between classifiers). A possible explanation is that some nontarget flashes elicit a P300, leading to lower selection accuracy of the respective classifier. In an additional set of data from six different participants we replicated the ERP structure of the initial analyses and characterized the spatial distributions more closely by using a dense electrode array. Overall, our findings provide new insights in the componential structure of ERPs elicited in the P300 speller paradigm and have important implications for optimizing the speller's selection accuracy.

A P300-based brain-computer interface with stimuli on moving objects: four-session single-trial and triple-trial tests with a game-like task design

Brain-computer interfaces (BCIs) are tools for controlling computers and other devices without using muscular activity, employing user-controlled variations in signals recorded from the user's brain. One of the most efficient noninvasive BCIs is based on the P300 wave of the brain's response to stimuli and is therefore referred to as the P300 BCI. Many modifications of this BCI have been proposed to further improve the BCI's characteristics or to better adapt the BCI to various applications. However, in the original P300 BCI and in all of its modifications, the spatial positions of stimuli were fixed relative to each other, which can impose constraints on designing applications controlled by this BCI. We designed and tested a P300 BCI with stimuli presented on objects that were freely moving on a screen at a speed of 5.4°/s. Healthy participants practiced a game-like task with this BCI in either single-trial or triple-trial mode within four sessions. At each step, the participants were required to select one of nine moving objects. The mean online accuracy of BCI-based selection was 81% in the triple-trial mode and 65% in the single-trial mode. A relatively high P300 amplitude was observed in response to targets in most participants. Self-rated interest in the task was high and stable over the four sessions (the medians in the 1st/4th sessions were 79/84% and 76/71% in the groups practicing in the single-trial and triple-trial modes, respectively). We conclude that the movement of stimulus positions relative to each other may not prevent the efficient use of the P300 BCI by people controlling their gaze, e.g., in robotic devices and in video games.

A comparison of two spelling Brain-Computer Interfaces based on visual P3 and SSVEP in Locked-In Syndrome

OBJECTIVES

We study the applicability of a visual P3-based and a Steady State Visually Evoked Potentials (SSVEP)-based Brain-Computer Interfaces (BCIs) for mental text spelling on a cohort of patients with incomplete Locked-In Syndrome (LIS).

METHODS

Seven patients performed repeated sessions with each BCI. We assessed BCI performance, mental workload and overall satisfaction for both systems. We also investigated the effect of the quality of life and level of motor impairment on the performance.

RESULTS

All seven patients were able to achieve an accuracy of 70% or more with the SSVEP-based BCI, compared to 3 patients with the P3-based BCI, showing a better performance with the SSVEP BCI than with the P3 BCI in the studied cohort. Moreover, the better performance of the SSVEP-based BCI was accompanied by a lower mental workload and a higher overall satisfaction. No relationship was found between BCI performance and level of motor impairment or quality of life.

CONCLUSION

Our results show a better usability of the SSVEP-based BCI than the P3-based one for the sessions performed by the tested population of locked-in patients with respect to all the criteria considered. The study shows the advantage of developing alternative BCIs with respect to the traditional matrix-based P3 speller using different designs and signal modalities such as SSVEPs to build a faster, more accurate, less mentally demanding and more satisfying BCI by testing both types of BCIs on a convenience sample of LIS patients.

Asynchronous gaze-independent event-related potential-based brain-computer interface

OBJECTIVE

In this study a gaze independent event related potential (ERP)-based brain computer interface (BCI) for communication purpose was combined with an asynchronous classifier endowed with dynamical stopping feature. The aim was to evaluate if and how the performance of such asynchronous system could be negatively affected in terms of communication efficiency and robustness to false positives during the intentional no-control state.

MATERIAL AND METHODS

The proposed system was validated with the participation of 9 healthy subjects. A comparison was performed between asynchronous and synchronous classification technique outputs while users were controlling the same gaze independent BCI interface. The performance of both classification techniques were assessed both off-line and on-line by means of the efficiency metric introduced by Bianchi et al. (2007). This latter metric allows to set a different misclassification cost for wrong classifications and abstentions. Robustness was evaluated as the rate of false positives occurring during voluntary no-control states.

RESULTS

The asynchronous classifier did not exhibited significantly higher accuracy or lower error rate with respect to the synchronous classifier (accuracy: 74.66% versus 87.96%, error rate: 7.11% versus 12.04% respectively). However, the on-line and off-line analysis revealed that the communication efficiency was significantly improved (p<.05) with the asynchronous classification modality as compared with the synchronous. Furthermore, the asynchronous classifier proved to be robust to false positives during intentional no-control state which occur during the ongoing visual stimulation (less than 1 false positive every 6min).

CONCLUSION

As such, the proposed ERP-BCI system which combines an asynchronous classifier with a gaze independent interface is a promising solution to be further explored in order to increase the general usability of ERP-based BCI systems designed for severely disabled people with an impairment of the voluntary control of eye movements. In fact, the asynchronous classifier can improve communication efficiency automatically adapting the number of stimulus repetitions to the current user's state and suspending the control if he/she does not intend to select an item.

User-centered design in brain-computer interfaces-a case study

OBJECTIVE

The array of available brain-computer interface (BCI) paradigms has continued to grow, and so has the corresponding set of machine learning methods which are at the core of BCI systems. The latter have evolved to provide more robust data analysis solutions, and as a consequence the proportion of healthy BCI users who can use a BCI successfully is growing. With this development the chances have increased that the needs and abilities of specific patients, the end-users, can be covered by an existing BCI approach. However, most end-users who have experienced the use of a BCI system at all have encountered a single paradigm only. This paradigm is typically the one that is being tested in the study that the end-user happens to be enrolled in, along with other end-users. Though this corresponds to the preferred study arrangement for basic research, it does not ensure that the end-user experiences a working BCI. In this study, a different approach was taken; that of a user-centered design. It is the prevailing process in traditional assistive technology. Given an individual user with a particular clinical profile, several available BCI approaches are tested and - if necessary - adapted to him/her until a suitable BCI system is found.

METHODS

Described is the case of a 48-year-old woman who suffered from an ischemic brain stem stroke, leading to a severe motor- and communication deficit. She was enrolled in studies with two different BCI systems before a suitable system was found. The first was an auditory event-related potential (ERP) paradigm and the second a visual ERP paradigm, both of which are established in literature.

RESULTS

The auditory paradigm did not work successfully, despite favorable preconditions. The visual paradigm worked flawlessly, as found over several sessions. This discrepancy in performance can possibly be explained by the user's clinical deficit in several key neuropsychological indicators, such as attention and working memory. While the auditory paradigm relies on both categories, the visual paradigm could be used with lower cognitive workload. Besides attention and working memory, several other neurophysiological and -psychological indicators - and the role they play in the BCIs at hand - are discussed.

CONCLUSION

The user's performance on the first BCI paradigm would typically have excluded her from further ERP-based BCI studies. However, this study clearly shows that, with the numerous paradigms now at our disposal, the pursuit for a functioning BCI system should not be stopped after an initial failed attempt.

Bayesian approach to dynamically controlling data collection in P300 spellers

P300 spellers provide a noninvasive method of communication for people who may not be able to use other communication aids due to severe neuromuscular disabilities. However, P300 spellers rely on event-related potentials (ERPs) which often have low signal-to-noise ratios (SNRs). In order to improve detection of the ERPs, P300 spellers typically collect multiple measurements of the electroencephalography (EEG) response for each character. The amount of collected data can affect both the accuracy and the communication rate of the speller system. The goal of the present study was to develop an algorithm that would automatically determine the necessary amount of data to collect during operation. Dynamic data collection was controlled by a threshold on the probabilities that each possible character was the target character, and these probabilities were continually updated with each additional measurement. This Bayesian technique differs from other dynamic data collection techniques by relying on a participant-independent, probability-based metric as the stopping criterion. The accuracy and communication rate for dynamic and static data collection in P300 spellers were compared for 26 users. Dynamic data collection resulted in a significant increase in accuracy and communication rate.

A multi-signature brain-computer interface: use of transient and steady-state responses

OBJECTIVE

The aim of this paper was to increase the information transfer in brain-computer interfaces (BCI). Therefore, a multi-signature BCI was developed and investigated. Stimuli were designed to simultaneously evoke transient somatosensory event-related potentials (ERPs) and steady-state somatosensory potentials (SSSEPs) and the ERPs and SSSEPs in isolation.

APPROACH

Twelve subjects participated in two sessions. In the first session, the single and combined stimulation conditions were compared on these somatosensory responses and on the classification performance. In the second session the on-line performance with the combined stimulation was evaluated while subjects received feedback. Furthermore, in both sessions, the performance based on ERP and SSSEP features was compared.

MAIN RESULTS

No difference was found in the ERPs and SSSEPs between stimulation conditions. The combination of ERP and SSSEP features did not perform better than with ERP features only. In both sessions, the classification performances based on ERP and combined features were higher than the classification based on SSSEP features.

SIGNIFICANCE

Although the multi-signature BCI did not increase performance, it also did not negatively impact it. Therefore, such stimuli could be used and the best performing feature set could then be chosen individually.

Classifier-based latency estimation: a novel way to estimate and predict BCI accuracy

OBJECTIVE

Brain-computer interfaces (BCIs) that detect event-related potentials (ERPs) rely on classification schemes that are vulnerable to latency jitter, a phenomenon known to occur with ERPs such as the P300 response. The objective of this work was to investigate the role that latency jitter plays in BCI classification.

APPROACH

We developed a novel method, classifier-based latency estimation (CBLE), based on a generalization of Woody filtering. The technique works by presenting the time-shifted data to the classifier, and using the time shift that corresponds to the maximal classifier score.

MAIN RESULTS

The variance of CBLE estimates correlates significantly (p < 10(-42)) with BCI accuracy in the Farwell-Donchin BCI paradigm. Additionally, CBLE predicts same-day accuracy, even from small datasets or datasets that have already been used for classifier training, better than the accuracy on the small dataset (p < 0.05). The technique should be relatively classifier-independent, and the results were confirmed on two linear classifiers.

SIGNIFICANCE

The results suggest that latency jitter may be an important cause of poor BCI performance, and methods that correct for latency jitter may improve that performance. CBLE can also be used to decrease the amount of data needed for accuracy estimation, allowing research on effects with shorter timescales.

Navigation of a telepresence robot via covert visuospatial attention and real-time fMRI

Brain-computer interfaces (BCIs) allow people with severe neurological impairment and without ability to control their muscles to regain some control over their environment. The BCI user performs a mental task to regulate brain activity, which is measured and translated into commands controlling some external device. We here show that healthy participants are capable of navigating a robot by covertly shifting their visuospatial attention. Covert Visuospatial Attention (COVISA) constitutes a very intuitive brain function for spatial navigation and does not depend on presented stimuli or on eye movements. Our robot is equipped with motors and a camera that sends visual feedback to the user who can navigate it from a remote location. We used an ultrahigh field MRI scanner (7 Tesla) to obtain fMRI signals that were decoded in real time using a support vector machine. Four healthy subjects with virtually no training succeeded in navigating the robot to at least three of four target locations. Our results thus show that with COVISA BCI, realtime robot navigation can be achieved. Since the magnitude of the fMRI signal has been shown to correlate well with the magnitude of spectral power changes in the gamma frequency band in signals measured by intracranial electrodes, the COVISA concept may in future translate to intracranial application in severely paralyzed people.

Regularized logistic regression and multiobjective variable selection for classifying MEG data

This paper addresses the question of maximizing classifier accuracy for classifying task-related mental activity from Magnetoencelophalography (MEG) data. We propose the use of different sources of information and introduce an automatic channel selection procedure. To determine an informative set of channels, our approach combines a variety of machine learning algorithms: feature subset selection methods, classifiers based on regularized logistic regression, information fusion, and multiobjective optimization based on probabilistic modeling of the search space. The experimental results show that our proposal is able to improve classification accuracy compared to approaches whose classifiers use only one type of MEG information or for which the set of channels is fixed a priori.

Eye-gaze independent EEG-based brain-computer interfaces for communication

The present review systematically examines the literature reporting gaze independent interaction modalities in non-invasive brain-computer interfaces (BCIs) for communication. BCIs measure signals related to specific brain activity and translate them into device control signals. This technology can be used to provide users with severe motor disability (e.g. late stage amyotrophic lateral sclerosis (ALS); acquired brain injury) with an assistive device that does not rely on muscular contraction. Most of the studies on BCIs explored mental tasks and paradigms using visual modality. Considering that in ALS patients the oculomotor control can deteriorate and also other potential users could have impaired visual function, tactile and auditory modalities have been investigated over the past years to seek alternative BCI systems which are independent from vision. In addition, various attentional mechanisms, such as covert attention and feature-directed attention, have been investigated to develop gaze independent visual-based BCI paradigms. Three areas of research were considered in the present review: (i) auditory BCIs, (ii) tactile BCIs and (iii) independent visual BCIs. Out of a total of 130 search results, 34 articles were selected on the basis of pre-defined exclusion criteria. Thirteen articles dealt with independent visual BCIs, 15 reported on auditory BCIs and the last six on tactile BCIs, respectively. From the review of the available literature, it can be concluded that a crucial point is represented by the trade-off between BCI systems/paradigms with high accuracy and speed, but highly demanding in terms of attention and memory load, and systems requiring lower cognitive effort but with a limited amount of communicable information. These issues should be considered as priorities to be explored in future studies to meet users' requirements in a real-life scenario.

Brain-computer interfacing in disorders of consciousness

BACKGROUND

Recent neuroimaging research has strikingly demonstrated the existence of covert awareness in some patients with disorders of consciousness (DoC). These findings have highlighted the potential for the development of simple brain-computer interfaces (BCI) as a diagnosis in behaviourally unresponsive patients.

OBJECTIVES

This study here reviews current EEG-based BCIs that hold potential for assessing and eventually assisting patients with DoC. It highlights key areas for further development that might eventually make their application feasible in this challenging patient group.

METHODS

The major types of BCIs proposed in the literature are considered, namely those based on the P3 potential, sensorimotor rhythms, steady state oscillations and slow cortical potentials. In each case, a brief overview of the relevant literature is provided and then their relative merits for BCI applications in DoC are considered.

RESULTS

A range of BCI designs have been proposed and tested for enabling communication in fully conscious, paralysed patients. Although many of these have potential applicability for patients with DoC, they share some key challenges that need to be overcome, including limitations of stimulation modality, feedback, user training and consistency.

CONCLUSION

Future work will need to address the technical and practical challenges facing reliable implementation at the patient's bedside.

Evaluation of the performances of different P300 based brain-computer interfaces by means of the efficiency metric

The aim of this paper is to show how to use the Efficiency, a brain-computer interface (BCI) performance indicator, to evaluate the performances of a wide range of BCI systems. Unlike the most used metrics in the BCI research field, the Efficiency takes into account the penalties and the strategies to recover errors and this makes it a reliable instrument to describe the behavior of real BCIs. The Efficiency is compared with the accuracy and the information transfer rate, both in the Wolpaw and Nykopp definitions. The comparison covers four widely used classifiers and different stimulation sequences. Results show that the Efficiency is able to predict if the communication will not be possible, because the time spent to correct mistakes is longer than the time needed to generate a correct selection, and therefore it provides a much more realistic evaluation of a system. It can also be easily adapted to evaluate different applications, so it reveals a more general and versatile indicator for BCI systems.

A Novel 9-Class Auditory ERP Paradigm Driving a Predictive Text Entry System

Brain–computer interfaces (BCIs) based on event related potentials (ERPs) strive for offering communication pathways which are independent of muscle activity. While most visual ERP-based BCI paradigms require good control of the user's gaze direction, auditory BCI paradigms overcome this restriction. The present work proposes a novel approach using auditory evoked potentials for the example of a multiclass text spelling application. To control the ERP speller, BCI users focus their attention to two-dimensional auditory stimuli that vary in both, pitch (high/medium/low) and direction (left/middle/right) and that are presented via headphones. The resulting nine different control signals are exploited to drive a predictive text entry system. It enables the user to spell a letter by a single nine-class decision plus two additional decisions to confirm a spelled word. This paradigm – called PASS2D – was investigated in an online study with 12 healthy participants. Users spelled with more than 0.8 characters per minute on average (3.4 bits/min) which makes PASS2D a competitive method. It could enrich the toolbox of existing ERP paradigms for BCI end users like people with amyotrophic lateral sclerosis disease in a late stage.

Impact of spatial filters during sensor selection in a visual P300 brain-computer interface

A challenge in designing a Brain-Computer Interface (BCI) is the choice of the channels, e.g. the most relevant sensors. Although a setup with many sensors can be more efficient for the detection of Event-Related Potential (ERP) like the P300, it is relevant to consider only a low number of sensors for a commercial or clinical BCI application. Indeed, a reduced number of sensors can naturally increase the user comfort by reducing the time required for the installation of the EEG (electroencephalogram) cap and can decrease the price of the device. In this study, the influence of spatial filtering during the process of sensor selection is addressed. Two of them maximize the Signal to Signal-plus-Noise Ratio (SSNR) for the different sensor subsets while the third one maximizes the differences between the averaged P300 waveform and the non P300 waveform. We show that the locations of the most relevant sensors subsets for the detection of the P300 are highly dependent on the use of spatial filtering. Applied on data from 20 healthy subjects, this study proves that subsets obtained where sensors are suppressed in relation to their individual SSNR are less efficient than when sensors are suppressed in relation to their contribution once the different selected sensors are combined for enhancing the signal. In other words, it highlights the difference between estimating the P300 projection on the scalp and evaluating the more efficient sensor subsets for a P300-BCI. Finally, this study explores the issue of channel commonality across subjects. The results support the conclusion that spatial filters during the sensor selection procedure allow selecting better sensors for a visual P300 Brain-Computer Interface.

Brain-computer interfacing using modulations of alpha activity induced by covert shifts of attention

Background

Visual brain-computer interfaces (BCIs) often yield high performance only when targets are fixated with the eyes. Furthermore, many paradigms use intense visual stimulation, which can be irritating especially in long BCI sessions. However, BCIs can more directly directly tap the neural processes underlying visual attention. Covert shifts of visual attention induce changes in oscillatory alpha activity in posterior cortex, even in the absence of visual stimulation. The aim was to investigate whether different pairs of directions of attention shifts can be reliably differentiated based on the electroencephalogram. To this end, healthy participants (N = 8) had to strictly fixate a central dot and covertly shift visual attention to one out of six cued directions.

Results

Covert attention shifts induced a prolonged alpha synchronization over posterior electrode sites (PO and O electrodes). Spectral changes had specific topographies so that different pairs of directions could be differentiated. There was substantial variation across participants with respect to the direction pairs that could be reliably classified. Mean accuracy for the best-classifiable pair amounted to 74.6%. Furthermore, an alpha power index obtained during a relaxation measurement showed to be predictive of peak BCI performance (r = .66).

Conclusions

Results confirm posterior alpha power modulations as a viable input modality for gaze-independent EEG-based BCIs. The pair of directions yielding optimal performance varies across participants. Consequently, participants with low control for standard directions such as left-right might resort to other pairs of directions including top and bottom. Additionally, a simple alpha index was shown to predict prospective BCI performance.