Multiband task related components enhance rapid cognition decoding for both small and similar objects

The cortically-coupled target recognition system based on rapid serial visual presentation (RSVP) has a wide range of applications in brain computer interface (BCI) fields such as medical and military. However, in the complex natural environment backgrounds, the identification of event-related potentials (ERP) of both small and similar objects that are quickly presented is a research challenge. Therefore, we designed corresponding experimental paradigms and proposed a multi-band task related components matching (MTRCM) method to improve the rapid cognitive decoding of both small and similar objects. We compared the areas under the receiver operating characteristic curve (AUC) between MTRCM and other 9 methods under different numbers of training sample using RSVP-ERP data from 50 subjects. The results showed that MTRCM maintained an overall superiority and achieved the highest average AUC (0.6562 ± 0.0091). We also optimized the frequency band and the time parameters of the method. The verification on public data sets further showed the necessity of designing MTRCM method. The MTRCM method provides a new approach for neural decoding of both small and similar RSVP objects, which is conducive to promote the further development of RSVP-BCI.

A spatial-temporal linear feature learning algorithm for P300-based brain-computer interfaces

Speller brain-computer interface (BCI) systems can help neuromuscular disorders patients write their thoughts by using the electroencephalogram (EEG) signals by just focusing on the speller tasks. For practical speller-based BCI systems, the P300 event-related brain potential is measured by using the EEG signal. In this paper, we design a robust machine-learning algorithm for P300 target detection. The novel spatial-temporal linear feature learning (STLFL) algorithm is proposed to extract high-level P300 features. The STLFL method is a modified linear discriminant analysis technique focusing on the spatial-temporal aspects of information extraction. A new P300 detection structure is then proposed based on the combination of the novel STLFL feature extraction and discriminative restricted Boltzmann machine (DRBM) for the classification approach (STLFL + DRBM). The effectiveness of the proposed technique is evaluated using two state-of-the-art P300 BCI datasets. Across the two available databases, we show that in terms of average target recognition accuracy and standard deviation values, the proposed STLFL + DRBM method outperforms traditional methods by 33.5, 78.5, 93.5, and 98.5% for 1, 5, 10, and 15 repetitions, respectively, in BCI competition III datasets II and by 71.3, 100, 100, and 100% for 1, 5, 10, and 15 repetitions, respectively, in BCI competition II datasets II and by 67.5 ± 4, 84.2 ± 2.5, 93.5 ± 1, 96.3 ± 1, and 98.4 ± 0.5% for rapid serial visual presentation (RSVP) based dataset in repetitions 1-5. The method has some advantages over the existing variants including its efficiency, robustness with a small number of training samples, and a high ability to create discriminative features between classes.

Stopping Criterion Design for Recursive Bayesian Classification: Analysis and Decision Geometry

Systems that are based on recursive Bayesian updates for classification limit the cost of evidence collection through certain stopping/termination criteria and accordingly enforce decision making. Conventionally, two termination criteria based on pre-defined thresholds over (i) the maximum of the state posterior distribution; and (ii) the state posterior uncertainty are commonly used. In this paper, we propose a geometric interpretation over the state posterior progression and accordingly we provide a point-by-point analysis over the disadvantages of using such conventional termination criteria. For example, through the proposed geometric interpretation we show that confidence thresholds defined over maximum of the state posteriors suffer from stiffness that results in unnecessary evidence collection whereas uncertainty based thresholding methods are fragile to number of categories and terminate prematurely if some state candidates are already discovered to be unfavorable. Moreover, both types of termination methods neglect the evolution of posterior updates. We then propose a new stopping/termination criterion with a geometrical insight to overcome the limitations of these conventional methods and provide a comparison in terms of decision accuracy and speed. We validate our claims using simulations and using real experimental data obtained through a brain computer interfaced typing system.

Detection of Stroke-Induced Visual Neglect and Target Response Prediction Using Augmented Reality and Electroencephalography

We aim to build a system incorporating electroencephalography (EEG) and augmented reality (AR) that is capable of identifying the presence of visual spatial neglect (SN) and mapping the estimated neglected visual field. An EEG-based brain-computer interface (BCI) was used to identify those spatiospectral features that best detect participants with SN among stroke survivors using their EEG responses to ipsilesional and contralesional visual stimuli. Frontal-central delta and alpha, frontal-parietal theta, Fp1 beta, and left frontal gamma were found to be important features for neglect detection. Additionally, temporal analysis of the responses shows that the proposed model is accurate in detecting potentially neglected targets. These targets were predicted using common spatial patterns as the feature extraction algorithm and regularized discriminant analysis combined with kernel density estimation for classification. With our preliminary results, our system shows promise for reliably detecting the presence of SN and predicting visual target responses in stroke patients with SN.

Target-Related Alpha Attenuation in a Brain-Computer Interface Rapid Serial Visual Presentation Calibration

This study evaluated the feasibility of using occipitoparietal alpha activity to drive target/non-target classification in a brain-computer interface (BCI) for communication. EEG data were collected from 12 participants who completed BCI Rapid Serial Visual Presentation (RSVP) calibrations at two different presentation rates: 1 and 4 Hz. Attention-related changes in posterior alpha activity were compared to two event-related potentials (ERPs): N200 and P300. Machine learning approaches evaluated target/non-target classification accuracy using alpha activity. Results indicated significant alpha attenuation following target letters at both 1 and 4 Hz presentation rates, though this effect was significantly reduced in the 4 Hz condition. Target-related alpha attenuation was not correlated with coincident N200 or P300 target effects. Classification using posterior alpha activity was above chance and benefitted from individualized tuning procedures. These findings suggest that target-related posterior alpha attenuation is detectable in a BCI RSVP calibration and that this signal could be leveraged in machine learning algorithms used for RSVP or comparable attention-based BCI paradigms.

Feedback Related Potentials for EEG-Based Typing Systems

Error related potentials (ErrP), which are elicited in the EEG in response to a perceived error, have been used for error correction and adaption in the event related potential (ERP)-based brain computer interfaces designed for typing. In these typing interfaces, ERP evidence is collected in response to a sequence of stimuli presented usually in the visual form and the intended user stimulus is probabilistically inferred (stimulus with highest probability) and presented to the user as the decision. If the inferred stimulus is incorrect, ErrP is expected to be elicited in the EEG. Early approaches to use ErrP in the design of typing interfaces attempt to make hard decisions on the perceived error such that the perceived error is corrected and either the sequence of stimuli are repeated to obtain further ERP evidence, or without further repetition the stimulus with the second highest probability is presented to the user as the decision of the system. Moreover, none of the existing approaches use a language model to increase the performance of typing. In this work, unlike the existing approaches, we study the potential benefits of fusing feedback related potentials (FRP), a form of ErrP, with ERP and context information (language model, LM) in a Bayesian fashion to detect the user intent. We present experimental results based on data from 12 healthy participants using RSVP Keyboard™ to complete a copy-phrase-task. Three paradigms are compared: [P1] uses only ERP/LM Bayesian fusion; [P2] each RSVP sequence is appended with the top candidate in the alphabet according to posterior after ERP evidence fusion; corresponding FRP is then incorporated; and [P3] the top candidate is shown as a prospect to generate FRP evidence only if its posterior exceeds a threshold. Analyses indicate that ERP/LM/FRP evidence fusion during decision making yields significant speed-accuracy benefits for the user.

Visual P300 Mind-Speller Brain-Computer Interfaces: A Walk Through the Recent Developments With Special Focus on Classification Algorithms

Brain-computer interfaces are sophisticated signal processing systems, which directly operate on neuronal signals to identify specific human intents. These systems can be applied to overcome certain disabilities or to enhance the natural capabilities of human beings. The visual P300 mind-speller is a prominent one among them, which has opened up tremendous possibilities in movement and communication applications. Today, there exist many state-of-the-art visual P300 mind-speller implementations in the literature as a result of numerous researches in this domain over the past 2 decades. Each of these systems can be evaluated in terms of performance metrics like classification accuracy, information transfer rate, and processing time. Various classification techniques associated with these systems, which include but are not limited to discriminant analysis, support vector machine, neural network, distance-based and ensemble of classifiers, have major roles in determining the overall system performances. The significance of a proper review on the recent developments in visual P300 mind-spellers with proper emphasis on their classification algorithms is the key insight for this work. This article is organized with a brief introduction to P300, concepts of visual P300 mind-spellers, the survey of literature with special focus on classification algorithms, followed by the discussion of various challenges and future directions.

An Event-Driven AR-Process Model for EEG-Based BCIs With Rapid Trial Sequences

Electroencephalography (EEG) is an effective non-invasive measurement method to infer user intent in brain-computer interface (BCI) systems for control and communication, however, these systems often lack sufficient accuracy and speed due to low separability of class-conditional EEG feature distributions. Many factors impact system performance, including inadequate training datasets and models' ignorance of the temporal dependency of brain responses to serial stimuli. Here, we propose a signal model for event-related responses in the EEG evoked with a rapid sequence of stimuli in BCI applications. The model describes the EEG as a superposition of impulse responses time-locked to stimuli corrupted with an autoregressive noise process. The performance of the signal model is assessed in the context of RSVP keyboard, a language-model-assisted EEG-based BCI for typing. EEG data obtained for model calibration from 10 healthy participants are used to fit and compare two models: the proposed sequence-based EEG model and the trial-based feature-class-conditional distribution model that ignores temporal dependencies, which has been used in the previous work. The simulation studies indicate that the earlier model that ignores temporal dependencies may be causing drastic reductions in achievable information transfer rate (ITR). Furthermore, the proposed model, with better regularization, may achieve improved accuracy with fewer calibration data samples, potentially helping to reduce calibration time. Specifically, results show an average 8.6% increase in (cross-validated) calibration AUC for a single channel of EEG, and 54% increase in the ITR in a typing task.

New and emerging access technologies for adults with complex communication needs and severe motor impairments: State of the science

Individuals with complex communication needs often use alternative access technologies to control their augmentative and alternative communication (AAC) devices, their computers, and mobile technologies. While a range of access devices is available, many challenges continue to exist, particularly for those with severe motor-control limitations. For some, access options may not be readily available or access itself may be inaccurate and frustrating. For others, access may be available but only under optimal conditions and support. There is an urgent need to develop new options for individuals with severe motor impairments and to leverage existing technology to improve efficiency, increase accuracy, and decrease fatigue of access. This paper describes person-centred research and development activities related to new and emerging access technologies, with a particular focus on adults with acquired neurological conditions.

A Parametric EEG Signal Model for BCIs with Rapid-Trial Sequences

Electroencephalogram (EEG) signals have been shown very effective for inferring user intents in brain-computer interface (BCI) applications. However, existing EEG-based BCIs, in many cases, lack sufficient performance due to utilizing classifiers that operate on EEG signals induced by individual trials. While many factors influence the classification performance, an important aspect that is often ignored is the temporal dependency of these trial-EEG signals, in some cases impacted by interference of brain responses to consecutive target and non-target trials. In this study, the EEG signals are analyzed in a parametric sequence-based fashion, which considers all trials that induce brain responses in a rapid-sequence fashion, including a mixture of consecutive target and non-target trials. EEG signals are described as a linear combination of time-shifted cortical source activities plus measurement noise. Using a superposition of time invariant with an auto-regressive (AR) process, EEG signals are treated as a linear combination of a stationary Gaussian process and time-locked impulse responses to the stimulus (input events) onsets. The model performance is assessed in the framework of a rapid serial visualization presentation (RSVP) based typing task for three healthy subjects across two sessions. Signal modeling in this fashion yields promising performance outcomes considering a single EEG channel to estimate the user intent.

INCORPORATING TEMPORAL DEPENDENCY ON ERP BASED BCI

In brain computer interface (BCI) systems based on event related potentials (ERPs), a windowed electroencephalography (EEG) signal is taken into consideration for the assumed duration of the ERP potential. In BCI applications inter stimuli interval is shorter than the ERP duration. This causes temporal dependencies over observation potentials thus disallows taking the data into consideration independently. However, conventionally the data is assumed to be independent for decreasing complexity. In this paper we propose a graphical model which covers the temporal dependency into consideration by labeling each time sample. We also propose a formulation to exploit the time series structure of the EEG.

A novel P300 BCI speller based on the Triple RSVP paradigm

A brain–computer interface (BCI) is an advanced human–machine interaction technology. The BCI speller is a typical application that detects the stimulated source-induced EEG signal to identify the expected characters of the subjects. The current mainstream matrix-based BCI speller involves two problems that remain unsolved, namely, gaze-dependent and space-dependent problems. Some scholars have designed gaze-independent and space-independent spelling systems. However, this system still cannot achieve a satisfactory information transfer rate (ITR). In this paper, we propose a novel triple RSVP speller with gaze-independent and space-independent characteristics and higher ITR. The triple RSVP speller uses rapid serial visual presentation (RSVP) paradigm, each time presents three different characters, and each character is presented three times to increase the ITR. The results of the experiments show the triple RSVP speller online average accuracy of 0.790 and average online ITR of 20.259 bit/min, where the system spelled at a speed of 10 s per character, and the stimulus presentation interface is a 90 × 195 pixel rectangle. Thus, the triple RSVP speller can be integrated into mobile smart devices (such as smartphones, smart watches, and others).

An Active RBSE Framework to Generate Optimal Stimulus Sequences in a BCI for Spelling

A class of brain computer interfaces (BCIs) employs noninvasive recordings of electroencephalography (EEG) signals to enable users with severe speech and motor impairments to interact with their environment and social network. For example, EEG based BCIs for typing popularly utilize event related potentials (ERPs) for inference. Presentation paradigm design in current ERP-based letter by letter typing BCIs typically query the user with an arbitrary subset characters. However, the typing accuracy and also typing speed can potentially be enhanced with more informed subset selection and flash assignment. In this manuscript, we introduce the active recursive Bayesian state estimation (active-RBSE) framework for inference and sequence optimization. Prior to presentation in each iteration, rather than showing a subset of randomly selected characters, the developed framework optimally selects a subset based on a query function. Selected queries are made adaptively specialized for users during each intent detection. Through a simulation-based study, we assess the effect of active-RBSE on the performance of a language-model assisted typing BCI in terms of typing speed and accuracy. To provide a baseline for comparison, we also utilize standard presentation paradigms namely, row and column matrix presentation paradigm and also random rapid serial visual presentation paradigms. The results show that utilization of active-RBSE can enhance the online performance of the system, both in terms of typing accuracy and speed. Moreover, we conduct real time experiments with human participants to study the human-in-the-loop effect on the performance of the proposed active-RBSE framework and consistent with the simulation results, the results of these experiments show improvement both in typing speed and accuracy.

Context-aware recursive bayesian graph traversal in BCIs

Noninvasive brain computer interfaces (BCI), and more specifically Electroencephalography (EEG) based systems for intent detection need to compensate for the low signal to noise ratio of EEG signals. In many applications, the temporal dependency information from consecutive decisions and contextual data can be used to provide a prior probability for the upcoming decision. In this study we proposed two probabilistic graphical models (PGMs), using context information and previously observed EEG evidences to estimate a probability distribution over the decision space in graph based decision-making mechanism. In this approach, user moves a pointer to the desired vertex in the graph in which each vertex represents an action. To select a vertex, a "Select" command, or a proposed probabilistic Selection criterion (PSC) can be used to automatically detect the user intended vertex. Performance of different PGMs and Selection criteria combinations are compared over a keyboard based on a graph layout. Based on the simulation results, probabilistic Selection criterion along with the probabilistic graphical model provides the highest performance boost for individuals with pour calibration performance and achieving the same performance for individuals with high calibration performance.

Spatio-Temporal EEG Models for Brain Interfaces

Multichannel electroencephalography (EEG) is widely used in non-invasive brain computer interfaces (BCIs) for user intent inference. EEG can be assumed to be a Gaussian process with unknown mean and autocovariance, and the estimation of parameters is required for BCI inference. However, the relatively high dimensionality of the EEG feature vectors with respect to the number of labeled observations lead to rank deficient covariance matrix estimates. In this manuscript, to overcome ill-conditioned covariance estimation, we propose a structure for the covariance matrices of the multichannel EEG signals. Specifically, we assume that these covariances can be modeled as a Kronecker product of temporal and spatial covariances. Our results over the experimental data collected from the users of a letter-by-letter typing BCI show that with less number of parameter estimations, the system can achieve higher classification accuracies compared to a method that uses full unstructured covariance estimation. Moreover, in order to illustrate that the proposed Kronecker product structure could enable shortening the BCI calibration data collection sessions, using Cramer-Rao bound analysis on simulated data, we demonstrate that a model with structured covariance matrices will achieve the same estimation error as a model with no covariance structure using fewer labeled EEG observations.

Integrating language models into classifiers for BCI communication: a review

OBJECTIVE

The present review systematically examines the integration of language models to improve classifier performance in brain-computer interface (BCI) communication systems.

APPROACH

The domain of natural language has been studied extensively in linguistics and has been used in the natural language processing field in applications including information extraction, machine translation, and speech recognition. While these methods have been used for years in traditional augmentative and assistive communication devices, information about the output domain has largely been ignored in BCI communication systems. Over the last few years, BCI communication systems have started to leverage this information through the inclusion of language models.

MAIN RESULTS

Although this movement began only recently, studies have already shown the potential of language integration in BCI communication and it has become a growing field in BCI research. BCI communication systems using language models in their classifiers have progressed down several parallel paths, including: word completion; signal classification; integration of process models; dynamic stopping; unsupervised learning; error correction; and evaluation.

SIGNIFICANCE

Each of these methods have shown significant progress, but have largely been addressed separately. Combining these methods could use the full potential of language model, yielding further performance improvements. This integration should be a priority as the field works to create a BCI system that meets the needs of the amyotrophic lateral sclerosis population.