Learning across multi-stimulus enhances target recognition methods in SSVEP-based BCIs

Multi-Stimulus Least-Squares Transformation With Online Adaptation Scheme to Reduce Calibration Effort for SSVEP-Based BCIs

Steady-state visual evoked potential (SSVEP), one of the most popular electroencephalography (EEG)-based brain-computer interface (BCI) paradigms, can achieve high performance using calibration-based recognition algorithms. As calibration-based recognition algorithms are time-consuming to collect calibration data, the least-squares transformation (LST) has been used to reduce the calibration effort for SSVEP-based BCI. However, the transformation matrices constructed by current LST methods are not precise enough, resulting in large differences between the transformed data and the real data of the target subject. This ultimately leads to the constructed spatial filters and reference templates not being effective enough. To address these issues, this paper proposes multi-stimulus LST with online adaptation scheme (ms-LST-OA).

METHODS

The proposed ms-LST-OA consists of two parts. Firstly, to improve the precision of the transformation matrices, we propose the multi-stimulus LST (ms-LST) using cross-stimulus learning scheme as the cross-subject data transformation method. The ms-LST uses the data from neighboring stimuli to construct a higher precision transformation matrix for each stimulus to reduce the differences between transformed data and real data. Secondly, to further optimize the constructed spatial filters and reference templates, we use an online adaptation scheme to learn more features of the EEG signals of the target subject through an iterative process trial-by-trial.

RESULTS

ms-LST-OA performance was measured for three datasets (Benchmark Dataset, BETA Dataset, and UCSD Dataset). Using few calibration data, the ITR of ms-LST-OA achieved 210.01±10.10 bits/min, 172.31±7.26 bits/min, and 139.04±14.90 bits/min for all three datasets, respectively.

CONCLUSION

Using ms-LST-OA can reduce calibration effort for SSVEP-based BCIs.

MetaBCI: An open-source platform for brain-computer interfaces

BACKGROUND

Recently, brain-computer interfaces (BCIs) have attracted worldwide attention for their great potential in clinical and real-life applications. To implement a complete BCI system, one must set up several links to translate the brain intent into computer commands. However, there is not an open-source software platform that can cover all links of the BCI chain.

METHOD

This study developed a one-stop open-source BCI software, namely MetaBCI, to facilitate the construction of a BCI system. MetaBCI is written in Python, and has the functions of stimulus presentation (Brainstim), data loading and processing (Brainda), and online information flow (Brainflow). This paper introduces the detailed information of MetaBCI and presents four typical application cases.

RESULTS

The results showed that MetaBCI was an extensible and feature-rich software platform for BCI research and application, which could effectively encode, decode, and feedback brain activities.

CONCLUSIONS

MetaBCI can greatly lower the BCI's technical threshold for BCI beginners and can save time and cost to build up a practical BCI system. The source code is available at https://github.com/TBC-TJU/MetaBCI, expecting new contributions from the BCI community.

A data expansion technique based on training and testing sample to boost the detection of SSVEPs for brain-computer interfaces

Objective.Currently, steady-state visual evoked potentials (SSVEPs)-based brain-computer interfaces (BCIs) have achieved the highest interaction accuracy and speed among all BCI paradigms. However, its decoding efficacy depends deeply on the number of training samples, and the system performance would have a dramatic drop when the training dataset decreased to a small size. To date, no study has been reported to incorporate the unsupervised learning information from testing trails into the construction of supervised classification model, which is a potential way to mitigate the overfitting effect of limited samples.Approach.This study proposed a novel method for SSVEPs detection, i.e. cyclic shift trials (CSTs), which could combine unsupervised learning information from test trials and supervised learning information from train trials. Furthermore, since SSVEPs are time-locked and phase-locked to the onset of specific flashes, CST could also expand training samples on the basis of its regularity and periodicity. In order to verify the effectiveness of CST, we designed an online SSVEP-BCI system, and tested this system combined CST with two common classification algorithms, i.e. extended canonical correlation analysis and ensemble task-related component analysis.Main results.CST could significantly enhance the signal to noise ratios of SSVEPs and improve the performance of systems especially for the condition of few training samples and short stimulus time. The online information transfer rate could reach up to 236.19 bits min-1using 36 s calibration time of only one training sample for each category.Significance.The proposed CST method can take full advantages of supervised learning information from training samples and unsupervised learning information of testing samples. Furthermore, it is a data expansion technique, which can enhance the SSVEP characteristics and reduce dependence on sample size. Above all, CST is a promising method to improve the performance of SSVEP-based BCI without any additional experimental burden.

Almost free of calibration for SSVEP-based brain-computer interfaces

Objective. Steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) is a promising technology that can achieve high information transfer rate (ITR) with supervised algorithms such as ensemble task-related component analysis (eTRCA) and task-discriminant component analysis (TDCA). However, training individual models requires a tedious and time-consuming calibration process, which hinders the real-life use of SSVEP-BCIs. A recent data augmentation method, called source aliasing matrix estimation (SAME), can generate new EEG samples from a few calibration trials. But SAME does not exploit the information across stimuli as well as only reduces the number of calibration trials per command, so it still has some limitations.Approach. This study proposes an extended version of SAME, called multi-stimulus SAME (msSAME), which exploits the similarity of the aliasing matrix across frequencies to enhance the performance of SSVEP-BCI with insufficient calibration trials. We also propose a semi-supervised approach based on msSAME that can further reduce the number of SSVEP frequencies needed for calibration. We evaluate our method on two public datasets, Benchmark and BETA, and an online experiment.Main results. The results show that msSAME outperforms SAME for both eTRCA and TDCA on the public datasets. Moreover, the semi-supervised msSAME-based method achieves comparable performance to the fully calibrated methods and outperforms the conventional free-calibrated methods. Remarkably, our method only needs 24 s to calibrate 40 targets in the online experiment and achieves an average ITR of 213.8 bits min-1with a peak of 242.6 bits min-1.Significance. This study significantly reduces the calibration effort for individual SSVEP-BCIs, which is beneficial for developing practical plug-and-play SSVEP-BCIs.

Phase-Locked Time-Shift Data Augmentation Method for SSVEP Brain-Computer Interfaces

Steady-state visual evoked potential (SSVEP) based brain-computer interfaces (BCIs) have achieved an information transfer rate (ITR) of over 300 bits/min, but abundant training data is required. The performance of SSVEP algorithms deteriorates greatly under limited data, and the existing time-shift data augmentation method fails to improve it because the phase-locked requirement between training samples is violated. To address this issue, this study proposes a novel augmentation method, namely phase-locked time-shift (PLTS), for SSVEP-BCI. The similarity between epochs at different time moments was evaluated, and a unique time-shift step was calculated for each class to augment additional data epochs in each trial. The results showed that the PLTS significantly improved the classification performance of SSVEP algorithms on the BETA SSVEP datasets. Moreover, under the condition of one calibration block, by slightly prolonging the calibration duration (from 48 s to 51.5 s), the ITR increased from 40.88±4.54 bits/min to 122.61±7.05 bits/min with the PLTS. This study provides a new perspective on augmenting data epochs for training-based SSVEP-BCI, promotes the classification accuracy and ITR under limited training data, and thus facilitates the real-life applications of SSVEP-based brain spellers.

Cross Domain Correlation Maximization for Enhancing the Target Recognition of SSVEP-Based Brain-Computer Interfaces

The target recognition performance of steady-state visual evoked potential (SSVEP)-based brain-computer interfaces can be significantly improved with a training-based approach. However, the training procedure is time consuming and often causes fatigue. Consequently, the number of training data should be limited, which may reduce the classification performance. Thus, how to improve classification accuracy without increasing the training time is crucial to SSVEP-based BCI system. This study proposes a transfer-related component analysis (TransRCA) method for addressing the above issue. In this method, the SSVEP-related components are extracted from a small number of training data of the current individual and combined with those extracted from a large number of existing training data of other individuals. The TransRCA method maximizes not only the inter-trial covariances between the source and target subjects, but also the correlation between the reference signals and SSVEP signals from the source and target subjects. The proposed method was validated on the SSVEP public Benchmark and BETA datasets, and the classification accuracy and information transmission rate of the ensemble version of the proposed TransRCA method were compared with those of the state-of-the-art eCCA, eTRCA, ttCCA, LSTeTRCA, and eIISMC methods on both datasets. The comparison results indicate that the proposed method provides a superior performance compared with these state-of-the-art methods, and thus has high potential for the development of a SSVEP-based brain-computer interface system with high classification performance that only uses a small number of training data.

Cross-Subject Transfer Method Based on Domain Generalization for Facilitating Calibration of SSVEP-Based BCIs

In steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs), various spatial filtering methods based on individual calibration data have been proposed to alleviate the interference of spontaneous activities in SSVEP signals for enhancing the SSVEP detection performance. However, the time-consuming calibration session would increase the visual fatigue of subjects and reduce the usability of the BCI system. The key idea of this study is to propose a cross-subject transfer method based on domain generalization, which transfers the domain-invariant spatial filters and templates learned from source subjects to the target subject with no access to the EEG data from the target subject. The transferred spatial filters and templates are obtained by maximizing the intra- and inter-subject correlations using the SSVEP data corresponding to the target and its neighboring stimuli. For SSVEP detection of the target subject, four types of correlation coefficients are calculated to construct the feature vector. Experimental results estimated with three SSVEP datasets show that the proposed cross-subject transfer method improves the SSVEP detection performance compared to state-of-art methods. The satisfactory results demonstrate that the proposed method provides an effective transfer learning strategy requiring no tedious data collection process for new users, holding the potential of promoting practical applications of SSVEP-based BCI.

Improving AR-SSVEP Recognition Accuracy Under High Ambient Brightness Through Iterative Learning

Augmented reality-based brain-computer interface (AR-BCI) system is one of the important ways to promote BCI technology outside of the laboratory due to its portability and mobility, but its performance in real-world scenarios has not been fully studied. In the current study, we first investigated the effect of ambient brightness on AR-BCI performance. 5 different light intensities were set as experimental conditions to simulate typical brightness in real scenes, while the same steady-state visual evoked potentials (SSVEP) stimulus was displayed in the AR glass. The data analysis results showed that SSVEP can be evoked under all 5 light intensities, but the response intensity became weaker when the brightness increased. The recognition accuracies of AR-SSVEP were negatively correlated to light intensity, the highest accuracies were 89.35% with FBCCA and 83.33% with CCA under 0 lux light intensity, while they decreased to 62.53% and 49.24% under 1200 lux. To solve the accuracy loss problem in high ambient brightness, we further designed a SSVEP recognition algorithm with iterative learning capability, named ensemble online adaptive CCA (eOACCA). The main strategy is to provide initial filters for high-intensity data by iteratively learning low-light-intensity AR-SSVEP data. The experimental results showed that the eOACCA algorithm had significant advantages under higher light intensities ( 600 lux). Compared with FBCCA, the accuracy of eOACCA under 1200 lux was increased by 13.91%. In conclusion, the current study contributed to the in-depth understanding of the performance variations of AR-BCI under different lighting conditions, and was helpful in promoting the AR-BCI application in complex lighting environments.

An Adaptive Task-Related Component Analysis Method for SSVEP Recognition

Steady-State Visual Evoked Potential (SSVEP) recognition methods use a subject's calibration data to differentiate between brain responses, hence, providing the SSVEP-based brain-computer interfaces (BCIs) with high performance. However, they require sufficient calibration EEG trials to achieve that. This study develops a new method to learn from limited calibration EEG trials, and it proposes and evaluates a novel adaptive data-driven spatial filtering approach for enhancing SSVEP detection. The spatial filter learned from each stimulus utilizes temporal information from the corresponding EEG trials. To introduce the temporal information into the overall procedure, a multitask learning approach, based on the Bayesian framework, is adopted. The performance of the proposed method was evaluated into two publicly available benchmark datasets, and the results demonstrated that our method outperformed competing methods by a significant margin.

An efficient CNN-LSTM Network with spectral normalization and label smoothing technologies for SSVEP frequency recognition

OBJECTIVE

Steady-state visual evoked potentials(SSVEPs) based braincomputer interface(BCI) has received great interests owing to the high information transfer rate(ITR) and available large number of targets. However, the performance of frequency recognition methods heavily depends on the amount of the calibration data for intra-subject classification. Some research adopted the deep learning(DL) algorithm to conduct the inter-subject classification, which could reduce the calculation procedure, but the performance still has large room to improve compared with the intra-subject classification.

APPROACH

To address these issues, we proposed an efficient SSVEP DL NETwork (termed SSVEPNET) based on 1D convolution and long short-term memory (LSTM) module. To enhance the performance of SSVEPNT, we adopted the spectral normalization and label smoothing technologies during implementing the network architecture. We evaluated the SSVEPNET and compared it with other methods for the intra- and inter-subject classification under different conditions, i.e., two datasets, two time-window lengths (1 s and 0.5 s), three sizes of training data.

MAIN RESULTS

Under all the experimental settings, the proposed SSVEPNET achieved the highest average accuracy for the intra- and inter-subject classification on the two SSVEP datasets, when compared with other traditional and DL baseline methods. Signif icance. The extensive experimental results demonstrate that the proposed DL model holds promise to enhance frequency recognition performance in SSVEP-based BCIs. Besides, the mixed network structures with CNN and LSTM, and the spectral normalization and label smoothing could be useful optimization strategies to design efficient models for EEG data.

An improved cross-subject spatial filter transfer method for SSVEP-based BCI

Steady-state visual evoked potential (SSVEP) training feature recognition algorithms utilize user training data to reduce the interference of spontaneous electroencephalogram (EEG) activities on SSVEP response for improved recognition accuracy. The data collection process can be tedious, increasing the mental fatigue of users and also seriously affecting the practicality of SSVEP-based brain-computer interface (BCI) systems. As an alternative, a cross-subject spatial filter transfer (CSSFT) method to transfer an existing user data model with good SSVEP response to new user test data has been proposed. The CSSFT method uses superposition averages of data for multiple blocks of data as transfer data. However, the amplitude and pattern of brain signals are often significantly different across trials. The goal of this study was to improve superposition averaging for the CSSFT method and propose an Ensemble scheme based on ensemble learning, and an Expansion scheme based on matrix expansion. The feature recognition performance was compared for CSSFT and the proposed improved CSSFT method using two public datasets. The results demonstrated that the improved CSSFT method can significantly improve the recognition accuracy and information transmission rate of existing methods. This strategy avoids a tedious data collection process, and promotes the potential practical application of BCI systems.

FB-EEGNet: A fusion neural network across multi-stimulus for SSVEP target detection

BACKGROUND

Steady-state visual evoked potential (SSVEP) is a prevalent paradigm of brain-computer interface (BCI). Recently, deep neural networks (DNNs) have been employed for SSVEP target recognition. However, current DNN models can not fully extract information from SSVEP harmonic components, and ignore the influence of non-target stimuli.

NEW METHOD

To employ information of multiple sub-bands and non-target stimulus data, we propose a DNN model for SSVEP target detection, i.e., FB-EEGNet, which fuses features of multiple neural networks. Additionally, we design a multi-label for each sample and optimize the parameters of FB-EEGNet across multi-stimulus to incorporate the information from non-target stimuli.

RESULTS

Under the subject-specific condition, FB-EEGNet achieves the average classification accuracies (information transfer rate (ITR)) of 76.75 % (50.70 bits/min) and 89.14 % (70.45 bits/min) in a time widow of 0.7 s under the public 12-target dataset and our experimental 9-target dataset, respectively. Under the cross-subject condition, FB-EEGNet achieved mean accuracies (ITRs) of 81.72 % (67.99 bits/min) and 92.15 % (76.12 bits/min) on the public and experimental datasets in a time window of 1 s, respectively.

COMPARISON WITH EXISTING METHODS

FB-EEGNet shows superior performance than CCNN, EEGNet, CCA and FBCCA both for subject-dependent and subject-independent SSVEP target recognition.

CONCLUSION

FB-EEGNet can effectively extract information from multiple sub-bands and cross-stimulus targets, providing a promising way for extracting deep features in SSVEP using neural networks.

Multisymbol Time Division Coding for High-Frequency Steady-State Visual Evoked Potential-Based Brain-Computer Interface

The optimization of coding stimulus is a crucial factor in the study of steady-state visual evoked potential (SSVEP)-based brain-computer interface(BCI).This study proposed an encoding approach named Multi-Symbol Time Division Coding (MSTDC). This approach is based on a protocol of maximizing the distance between neural responses, which aims to encode stimulation systems implementing any number of targets with finite stimulations of different frequencies and phases. Firstly, this study designed an SSVEP-based BCI system containing forty targets with this approach. The stimulation encoding of this system was achieved with four temporal-divided stimuli that adopt the same frequency of 30 Hz and different phases. During the online experiments of twelve subjects, this system achieved an average accuracy of 96.77 ±2.47 % and an average information transfer rate (ITR) of 119.05 ± 6.11 bits/min. This study also devised an SSVEP-based BCI system containing 72 targets and proposed a Template Splicing task-related component analysis (TRCA) algorithm that utilized the dataset of the previous system containing forty targets as the training dataset. The subjects acquired an average accuracy of 86.23 ± 7.75% and an average ITR of 95.68 ± 14.19 bits/min. It can be inferred that MSTDC can encode multiple targets with limited frequencies and phases of stimuli. Meanwhile, this protocol can be effortlessly expanded into other systems and sufficiently reduce the cost of collecting training data. This study provides a feasible technique for obtaining a comfortable SSVEP-based BCI with multiple targets while maintaining high information transfer rate.

Facilitating Applications of SSVEP-Based BCIs by Within-Subject Information Transfer

The steady-state visual evoked potential based brain–computer interface (SSVEP–BCI) can provide high-speed alternative and augmentative communication in real-world applications. For individuals using a long-term BCI, within-subject (i.e., cross-day and cross-electrode) transfer learning could improve the BCI performance and reduce the calibration burden. To validate the within-subject transfer learning scheme, this study designs a 40-target SSVEP–BCI. Sixteen subjects are recruited, each of whom has performed experiments on three different days and has undergone the experiments of the SSVEP–BCIs based on the dry and wet electrodes. Several transfer directions, including the cross-day directions in parallel with the cross-electrode directions, are analyzed, and it is found that the transfer learning-based approach can maintain stable performance by zero training. Compared with the fully calibrated approaches, the transfer learning-based approach can achieve significantly better or comparable performance in different transfer directions. This result verifies that the transfer learning-based scheme is well suited for implementing a high-speed zero-training SSVEP–BCI, especially the dry electrode-based SSVEP–BCI system. A validation experiment of the cross-day wet-to-dry transfer, involving nine subjects, has shown that the average accuracy is 85.97 ± 5.60% for the wet-to-dry transfer and 77.69 ± 6.42% for the fully calibrated method with dry electrodes. By leveraging the electroencephalography data acquired on different days by different electrodes via transfer learning, this study lays the foundation for facilitating the long-term usage of the SSVEP–BCI and advancing the frontier of the dry electrode-based SSVEP–BCI in real-world applications.

eldBETA: A Large Eldercare-oriented Benchmark Database of SSVEP-BCI for the Aging Population

Global population aging poses an unprecedented challenge and calls for a rising effort in eldercare and healthcare. Steady-state visual evoked potential based brain-computer interface (SSVEP-BCI) boasts its high transfer rate and shows great promise in real-world applications to support aging. Public database is critically important for designing the SSVEP-BCI systems. However, the SSVEP-BCI database tailored for the elder is scarce in existing studies. Therefore, in this study, we present a large eldercare-oriented BEnchmark database of SSVEP-BCI for The Aging population (eldBETA). The eldBETA database consisted of the 64-channel electroencephalogram (EEG) from 100 elder participants, each of whom performed seven blocks of 9-target SSVEP-BCI task. The quality and characteristics of the eldBETA database were validated by a series of analyses followed by a classification analysis of thirteen frequency recognition methods. We expect that the eldBETA database would provide a substrate for the design and optimization of the BCI systems intended for the elders. The eldBETA database is open-access for research and can be downloaded from the website 10.6084/m9.figshare.18032669.

Measurement(s)	Steady-state visual evoked potential (SSVEP)
Technology Type(s)	Electroencephalography (EEG)
Factor Type(s)	Elder population
Sample Characteristic - Organism	Homo sapiens
Sample Characteristic - Environment	Electromagnetic shielding room

Task-Related Component Analysis Combining Paired Character Decoding for Miniature Asymmetric Visual Evoked Potentials

Brain-computer interface (BCI) technology based on event-related potentials (ERP) of electroencephalography (EEG) is widely used in daily life and medical treatment. However, the research of identifying the miniature and more informative asymmetric visual evoked potentials (aVEPs), which belongs to ERP, needs further exploration. Herein, a task-related component analysis combining paired character decoding (TRCA-PCD) method, which can enhance reproducibility of aVEPs in multiple trials and strengthen the features of different samples, was designed to realize fast decoding of aVEPs. The BCI performance and the influence of repetition times between the TRCA-PCD method, the discriminative canonical pattern matching (DCPM) method and traditional task-related component analysis (TRCA) method were compared using a 32-class aVEPs dataset recorded from 32 subjects. The highest average recognition accuracy and information transfer rate (ITR) of TRCA-PCD after parameter selection were 70.37 ± 2.49% (DCPM: 64.91 ± 2.81%, TRCA: 44.01 ± 3.25%) with the peak value of 97.92% and 28.90 ± 3.83 bits/min (DCPM: 21.29 ± 3.35 bits/min, TRCA: 11.54 ± 2.81 bits/min) with the peak value of 94.55 bits/min respectively. Statistical analysis indicated that the highest average recognition rate could be obtained when the repetition time was six, and the highest ITR could be obtained when the repetition time was one. Overall, the results verified the effectiveness and superiority of TRCA-PCD in recognition of aVEPs and provided a reference for parameter selection. Therefore, the TRCA-PCD method can promote the further application of aVEPs in the BCI speller field.

Cross-subject spatial filter transfer method for SSVEP-EEG feature recognition

Objective.Steady-state visual evoked potential (SSVEP) is an important control method of the brain-computer interface (BCI) system. The development of an efficient SSVEP feature decoding algorithm is the core issue in SSVEP-BCI. It has been proposed to use user training data to reduce the spontaneous electroencephalogram activity interference on SSVEP response, thereby improving the feature recognition accuracy of the SSVEP signal. Nevertheless, the tedious data collection process increases the mental fatigue of the user and severely affects the applicability of the BCI system.Approach.A cross-subject spatial filter transfer (CSSFT) method that transfer the existing user model with good SSVEP response to the new user test data without collecting any training data from the new user is proposed.Main results.Experimental results demonstrate that the transfer model increases the distinction of the feature discriminant coefficient between the gaze following target and the non-gaze following target and accurately identifies the wrong target in the fundamental algorithm model. The public datasets show that the CSSFT method significantly increases the recognition performance of canonical correlation analysis (CCA) and filter bank CCA. Additionally, when the data used to calculate the transfer model contains one data block only, the CSSFT method retains its effective feature recognition capabilities.Significance.The proposed method requires no tedious data calibration process for new users, provides an effective technical solution for the transfer of the cross-subject model, and has potential application value for promoting the application of the BCI system.

A Zero-Padding Frequency Domain Convolutional Neural Network for SSVEP Classification

The brain-computer interface (BCI) of steady-state visual evoked potential (SSVEP) is one of the fundamental ways of human-computer communication. The main challenge is that there may be a nonlinear relationship between different SSVEP in other states. For improving the performance of SSVEP BCI, a novel CNN algorithm model is proposed in this study. Based on the discrete Fourier transform to calculate the signal's power spectral density (PSD), we perform zero-padding in the signal's time domain to improve its performance on the PSD and make it more refined. In this way, the frequency point interval in the PSD of the SSVEP is consistent with the minimum gap between the stimulation frequency. Combining the nonlinear transformation capabilities of CNN in deep learning, a zero-padding frequency domain convolutional neural network (ZPFDCNN) model is proposed. Extensive experiments based on the SSVEP dataset validate the effectiveness of our method. The study verifies that the proposed ZPFDCNN method can improve the effectiveness of the SSVEP-based high-speed BCI ITR. It has massive potential in the application of BCI.

Feedback Related Potentials for EEG-Based Typing Systems

Error related potentials (ErrP), which are elicited in the EEG in response to a perceived error, have been used for error correction and adaption in the event related potential (ERP)-based brain computer interfaces designed for typing. In these typing interfaces, ERP evidence is collected in response to a sequence of stimuli presented usually in the visual form and the intended user stimulus is probabilistically inferred (stimulus with highest probability) and presented to the user as the decision. If the inferred stimulus is incorrect, ErrP is expected to be elicited in the EEG. Early approaches to use ErrP in the design of typing interfaces attempt to make hard decisions on the perceived error such that the perceived error is corrected and either the sequence of stimuli are repeated to obtain further ERP evidence, or without further repetition the stimulus with the second highest probability is presented to the user as the decision of the system. Moreover, none of the existing approaches use a language model to increase the performance of typing. In this work, unlike the existing approaches, we study the potential benefits of fusing feedback related potentials (FRP), a form of ErrP, with ERP and context information (language model, LM) in a Bayesian fashion to detect the user intent. We present experimental results based on data from 12 healthy participants using RSVP Keyboard™ to complete a copy-phrase-task. Three paradigms are compared: [P1] uses only ERP/LM Bayesian fusion; [P2] each RSVP sequence is appended with the top candidate in the alphabet according to posterior after ERP evidence fusion; corresponding FRP is then incorporated; and [P3] the top candidate is shown as a prospect to generate FRP evidence only if its posterior exceeds a threshold. Analyses indicate that ERP/LM/FRP evidence fusion during decision making yields significant speed-accuracy benefits for the user.

SSVEP-EEG Feature Enhancement Method Using an Image Sharpening Filter

Steady-state visual evoked potential (SSVEP) is widely used in brain computer interface (BCI), medical detection, and neuroscience, so there is significant interest in enhancing SSVEP features via signal processing for better performance. In this study, an image processing method was combined with brain signal analysis and a sharpening filter was used to extract image details and features for the enhancement of SSVEP features. The results demonstrated that sharpening filter could eliminate the SSVEP signal trend term and suppress its low-frequency component. Meanwhile, sharpening filter effectively enhanced the signal-to-noise ratios (SNRs) of the single-channel and multi-channel fused signals. Image sharpening filter also significantly improved the recognition accuracy of canonical correlation analysis (CCA), filter bank canonical correlation analysis (FBCCA), and task-related component analysis (TRCA). The tools developed here effectively enhanced the SSVEP signal features, suggesting that image processing methods can be considered for improved brain signal analysis.

Online Adaptation Boosts SSVEP-Based BCI Performance

OBJECTIVE

A user-friendly steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) prefers no calibration for its target recognition algorithm, however, the existing calibration-free schemes perform still far behind their calibration-based counterparts. To tackle this issue, learning online from the subject's unlabeled data is investigated as a potential approach to boost the performance of the calibration-free SSVEP-based BCIs.

METHODS

An online adaptation scheme is developed to tune the spatial filters using the online unlabeled data from previous trials, and then developing the online adaptive canonical correlation analysis (OACCA) method.

RESULTS

A simulation study on two public SSVEP datasets (Dataset I and II) with a total of 105 subjects demonstrated that the proposed online adaptation scheme can boost the CCA's averaged information transfer rate (ITR) from 94.60 to 158.87 bits/min in Dataset I and from 85.80 to 123.91 bits/min in Dataset II. Furthermore, in our online experiment it boosted the CCA's ITR from 55.81 bits/min to 95.73 bits/min. More importantly, this online adaptation scheme can be easily combined with any spatial filtering-based algorithms to achieve online learning.

CONCLUSION

By online adaptation, the proposed OACCA performed much better than the calibration-free CCA, and comparable to the calibration-based algorithms.

SIGNIFICANCE

This work provides a general way for the SSVEP-based BCIs to learn online from unlabeled data and thus avoid calibration.

Filter bank approach for enhancement of supervised Canonical Correlation Analysis methods for SSVEP-based BCI spellers

Canonical correlation analysis (CCA) is one of the most used algorithms in the steady-state visual evoked potentials (SSVEP)-based brain-computer interface (BCI) systems due to its simplicity, efficiency, and robustness. Researchers have proposed modifications to CCA to improve its speed, allowing high-speed spelling and thus a more natural communication. In this work, we combine two approaches, the filter-bank (FB) approach to extract more information from the harmonics, and a range of different supervised methods which optimize the reference signals to improve the SSVEP detection. The proposed models are tested on the publicly available benchmark dataset for SSVEP-based BCIs and the results show improved performance compared to the state-of-the-art methods and, in particular, the proposed FBMwayCCA approach achieves the best results with an information transfer rate (ITR) of 134.8±8.4 bits/minute. This study indeed suggests the feasibility of combining the fundamental and harmonic SSVEP components with supervised methods in target identification to develop high-speed BCI spellers.

Common Spatial Pattern Reformulated for Regularizations in Brain-Computer Interfaces

Common spatial pattern (CSP) is one of the most successful feature extraction algorithms for brain-computer interfaces (BCIs). It aims to find spatial filters that maximize the projected variance ratio between the covariance matrices of the multichannel electroencephalography (EEG) signals corresponding to two mental tasks, which can be formulated as a generalized eigenvalue problem (GEP). However, it is challenging in principle to impose additional regularization onto the CSP to obtain structural solutions (e.g., sparse CSP) due to the intrinsic nonconvexity and invariance property of GEPs. This article reformulates the CSP as a constrained minimization problem and establishes the equivalence of the reformulated and the original CSPs. An efficient algorithm is proposed to solve this optimization problem by alternately performing singular value decomposition (SVD) and least squares. Under this new formulation, various regularization techniques for linear regression can then be easily implemented to regularize the CSPs for different learning paradigms, such as the sparse CSP, the transfer CSP, and the multisubject CSP. Evaluations on three BCI competition datasets show that the regularized CSP algorithms outperform other baselines, especially for the high-dimensional small training set. The extensive results validate the efficiency and effectiveness of the proposed CSP formulation in different learning contexts.

Information Bottleneck as Optimisation Method for SSVEP-Based BCI

In this study, the information bottleneck method is proposed as an optimisation method for steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI). The information bottleneck is an information-theoretic optimisation method for solving problems with a trade-off between preserving meaningful information and compression. Its main practical application in machine learning is in representation learning or feature extraction. In this study, we use the information bottleneck to find optimal classification rule for a BCI. This is a novel application for the information bottleneck. This approach is particularly suitable for BCIs since the information bottleneck optimises the amount of information transferred by the BCI. Steady-state visual evoked potential-based BCIs often classify targets using very simple rules like choosing the class corresponding to the largest feature value. We call this classifier the arg max classifier. It is unlikely that this approach is optimal, and in this study, we propose a classification method specifically designed to optimise the performance measure of BCIs. This approach gives an advantage over standard machine learning methods, which aim to optimise different measures. The performance of the proposed algorithm is tested on two publicly available datasets in offline experiments. We use the standard power spectral density analysis (PSDA) and canonical correlation analysis (CCA) feature extraction methods on one dataset and show that the current approach outperforms most of the related studies on this dataset. On the second dataset, we use the task-related component analysis (TRCA) method and demonstrate that the proposed method outperforms the standard argmax classification rule in terms of information transfer rate when using a small number of classes. To our knowledge, this is the first time the information bottleneck is used in the context of SSVEP-based BCIs. The approach is unique in the sense that optimisation is done over the space of classification functions. It potentially improves the performance of BCIs and makes it easier to calibrate the system for different subjects.

Improving the Performance of Individually Calibrated SSVEP-BCI by Task- Discriminant Component Analysis

A brain-computer interface (BCI) provides a direct communication channel between a brain and an external device. Steady-state visual evoked potential based BCI (SSVEP-BCI) has received increasing attention due to its high information transfer rate, which is accomplished by individual calibration for frequency recognition. Task-related component analysis (TRCA) is a recent and state-of-the-art method for individually calibrated SSVEP-BCIs. However, in TRCA, the spatial filter learned from each stimulus may be redundant and temporal information is not fully utilized. To address this issue, this paper proposes a novel method, i.e., task-discriminant component analysis (TDCA), to further improve the performance of individually calibrated SSVEP-BCI. The performance of TDCA was evaluated by two publicly available benchmark datasets, and the results demonstrated that TDCA outperformed ensemble TRCA and other competing methods by a significant margin. An offline and online experiment testing 12 subjects further validated the effectiveness of TDCA. The present study provides a new perspective for designing decoding methods in individually calibrated SSVEP-BCI and presents insight for its implementation in high-speed brain speller applications.

A Deep Neural Network for SSVEP-based Brain-Computer Interfaces

OBJECTIVE

Target identification in brain-computer interface (BCI) spellers refers to the electroencephalogram (EEG) classification for predicting the target character that the subject intends to spell. When the visual stimulus of each character is tagged with a distinct frequency, the EEG records steady-state visually evoked potentials (SSVEP) whose spectrum is dominated by the harmonics of the target frequency. In this setting, we address the target identification and propose a novel deep neural network (DNN) architecture.

METHOD

The proposed DNN processes the multi-channel SSVEP with convolutions across the sub-bands of harmonics, channels, time, and classifies at the fully connected layer. We test with two publicly available large scale (the benchmark and BETA) datasets consisting of in total 105 subjects with 40 characters. Our first stage training learns a global model by exploiting the statistical commonalities among all subjects, and the second stage fine tunes to each subject separately by exploiting the individualities.

RESULTS

Our DNN achieves impressive information transfer rates (ITRs) on both datasets, 265.23 bits/min and 196.59 bits/min, respectively, with only 0.4 seconds of stimulation. The code is available for reproducibility at https://github.com/osmanberke/Deep-SSVEP-BCI.

CONCLUSION

The presented DNN strongly outperforms the state-of-the-art techniques as our accuracy and ITR rates are the highest ever reported performance results on these datasets.

SIGNIFICANCE

Due to its unprecedentedly high speller ITRs and flawless applicability to general SSVEP systems, our technique has great potential in various biomedical engineering settings of BCIs such as communication, rehabilitation and control.

Align and pool for EEG headset domain adaptation (ALPHA) to facilitate dry electrode based SSVEP-BCI

OBJECTIVE

The steady-state visual evoked potential based brain-computer interface (SSVEP-BCI) implemented in dry electrodes is a promising paradigm for alternative and augmentative communication in real-world applications. To improve its performance and reduce the calibration effort for dry-electrode systems, we utilize cross-device transfer learning by exploiting auxiliary individual wet-electrode electroencephalogram (EEG).

METHODS

We proposed a novel transfer learning framework named ALign and Pool for EEG Headset domain Adaptation (ALPHA), which aligns the spatial pattern and the covariance for domain adaptation. To evaluate its efficacy, 75 subjects performed an experiment of 2 sessions involving a 12-target SSVEP-BCI task.

RESULTS

ALPHA significantly outperformed a baseline approach (canonical correlation analysis, CCA) and two competing transfer learning approaches (transfer template CCA, ttCCA and least square transformation, LST) in two transferring directions. When transferring from wet to dry EEG headsets, ALPHA significantly outperformed the fully calibrated approach of task-related component analysis (TRCA).

CONCLUSION

ALPHA advances the frontier of recalibration-free cross-device transfer learning for SSVEP-BCIs and boosts the performance of dry electrode based systems.

SIGNIFICANCE

ALPHA has methodological and practical implications and pushes the boundary of dry electrode based SSVEP-BCI toward real-world applications.

Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface

A brain-computer interface (BCI) can connect humans and machines directly and has achieved successful applications in the past few decades. Many new BCI paradigms and algorithms have been developed in recent years. Therefore, it is necessary to review new progress in BCIs. This paper summarizes progress for EEG-based BCIs from the perspective of encoding paradigms and decoding algorithms, which are two key elements of BCI systems. Encoding paradigms are grouped by their underlying neural meachanisms, namely sensory- and motor-related, vision-related, cognition-related and hybrid paradigms. Decoding algorithms are reviewed in four categories, namely decomposition algorithms, Riemannian geometry, deep learning and transfer learning. This review will provide a comprehensive overview of both modern primary paradigms and algorithms, making it helpful for those who are developing BCI systems.

Signal processing algorithms for SSVEP-based brain computer interface: State-of-the-art and recent developments

Over past two decades, steady-state evoked potentials (SSVEP)-based brain computer interface (BCI) systems have been extensively developed. As we all know, signal processing algorithms play an important role in this BCI. However, there is no comprehensive review of the latest development of signal processing algorithms for SSVEP-based BCI. By analyzing the papers published in authoritative journals in nearly five years, signal processing algorithms of preprocessing, feature extraction and classification modules are discussed in detail. In addition, other aspects existed in this BCI are mentioned. The following key problems are solved. (1) In recent years, which signal processing algorithms are frequently used in each module? (2) Which signal processing algorithms attract more attention in recent years? (3) Which modules are the key to signal processing in BCI field? This information is very important for choosing the appropriate algorithms, and can also be considered as a reference for further research. Simultaneously, we hope that this work can provide relevant BCI researchers with valuable information about the latest trends of signal processing algorithms for SSVEP-based BCI systems.

Similarity-constrained task-related component analysis for enhancing SSVEP detection

Objective. Task-related component analysis (TRCA) is a representative subject-specific training algorithm in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces. Task-related components (TRCs), extracted by the TRCA-based spatial filtering from electroencephalogram (EEG) signals through maximizing the reproducibility across trials, may contain some task-related inherent noise that is still trial-reproducible.Approach. To address this problem, this study proposed a similarity-constrained TRCA (scTRCA) algorithm to remove the task-related noise and extract TRCs maximally correlated with SSVEPs for enhancing SSVEP detection. Similarity constraints, which were created by introducing covariance matrices between EEG training data and an artificial SSVEP template, were added to the objective function of TRCA. Therefore, a better spatial filter was obtained by maximizing not only the reproducibility across trials but also the similarity between TRCs and SSVEPs. The proposed scTRCA was compared with TRCA, multi-stimulus TRCA, and sine-cosine reference signal based on two public datasets.Main results. The performance of TRCA in target identification of SSVEPs is improved by introducing similarity constraints. The proposed scTRCA significantly outperformed the other three methods, and the improvement was more significant especially with insufficient training data.Significance. The proposed scTRCA algorithm is promising for enhancing SSVEP detection considering its better performance and robustness against insufficient calibration.

Enhancing detection of steady-state visual evoked potentials using channel ensemble method

OBJECTIVE

This study proposed and evaluated a channel ensemble approach to enhance detection of steady-state visual evoked potentials (SSVEPs).

APPROACH

Collected multi-channel electroencephalogram (EEG) signals were classified into multiple groups of new analysis signals based on correlation analysis, and each group of analysis signals contained signals from a different number of electrode channels. These groups of analysis signals were used as the input of a training-free feature extraction model, and the obtained feature coefficients were converted into feature probability values using the softmax function. The ensemble value of multiple sets of feature probability values was determined and used as the final discrimination coefficient.

MAIN RESULTS

Compared with canonical correlation analysis (CCA), likelihood ratio test (LRT), and multivariate synchronization index (MSI) analysis methods using a standard approach, the recognition accuracies of the methods using a channel ensemble approach were improved by 5.05%, 3.87%, and 3.42%, and the information transfer rates (ITRs) were improved by 6.00%, 4.61%, and 3.71%, respectively. The channel ensemble method also obtained better recognition results than the standard algorithm on the public dataset. This study validated the efficiency of the proposed method to enhance the detection of SSVEPs, demonstrating its potential use in practical brain-computer interface (BCI) systems.

SIGNIFICANCE

A SSVEP-based BCI system using a channel ensemble method could achieve high ITR, indicating great potential of this design for various applications with improved control and interaction.

An Open Dataset for Wearable SSVEP-Based Brain-Computer Interfaces

Brain-computer interfaces (BCIs) provide humans a new communication channel by encoding and decoding brain activities. Steady-state visual evoked potential (SSVEP)-based BCI stands out among many BCI paradigms because of its non-invasiveness, little user training, and high information transfer rate (ITR). However, the use of conductive gel and bulky hardware in the traditional Electroencephalogram (EEG) method hinder the application of SSVEP-based BCIs. Besides, continuous visual stimulation in long time use will lead to visual fatigue and pose a new challenge to the practical application. This study provides an open dataset, which is collected based on a wearable SSVEP-based BCI system, and comprehensively compares the SSVEP data obtained by wet and dry electrodes. The dataset consists of 8-channel EEG data from 102 healthy subjects performing a 12-target SSVEP-based BCI task. For each subject, 10 consecutive blocks were recorded using wet and dry electrodes, respectively. The dataset can be used to investigate the performance of wet and dry electrodes in SSVEP-based BCIs. Besides, the dataset provides sufficient data for developing new target identification algorithms to improve the performance of wearable SSVEP-based BCIs.

Inter- and Intra-Subject Transfer Reduces Calibration Effort for High-Speed SSVEP-Based BCIs

OBJECTIVE

Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) that can deliver a high information transfer rate (ITR) usually require subject's calibration data to learn the class- and subject-specific model parameters (e.g. the spatial filters and SSVEP templates). Normally, the amount of the calibration data for learning is proportional to the number of classes (or visual stimuli), which could be huge and consequently lead to a time-consuming calibration. This study presents a transfer learning scheme to substantially reduce the calibration effort.

METHODS

Inspired by the parameter-based and instance-based transfer learning techniques, we propose a subject transfer based canonical correlation analysis (stCCA) method which utilizes the knowledge within subject and between subjects, thus requiring few calibration data from a new subject.

RESULTS

The evaluation study on two SSVEP datasets (from Tsinghua and UCSD) shows that the stCCA method performs well with only a small amount of calibration data, providing an ITR at 198.18±59.12 (bits/min) with 9 calibration trials in the Tsinghua dataset and 111.04±57.24 (bits/min) with 3 trials in the UCSD dataset. Such performances are comparable to those from using the multi-stimulus CCA (msCCA) and the ensemble task-related component analysis (eTRCA) methods with the minimally required calibration data (i.e., at least 40 trials in the Tsinghua dataset and at least 12 trials in the UCSD dataset), respectively.

CONCLUSION

Inter- and intra-subject transfer helps the recognition method achieve high ITR with extremely little calibration effort.

SIGNIFICANCE

The proposed approach saves much calibration effort without sacrificing the ITR, which would be significant for practical SSVEP-based BCIs.

Optimizing SSVEP-Based BCI System towards Practical High-Speed Spelling

The brain–computer interface (BCI) spellers based on steady-state visual evoked potentials (SSVEPs) have recently been widely investigated for their high information transfer rates (ITRs). This paper aims to improve the practicability of the SSVEP-BCIs for high-speed spelling. The system acquired the electroencephalogram (EEG) data from a self-developed dedicated EEG device and the stimulation was arranged as a keyboard. The task-related component analysis (TRCA) spatial filter was modified (mTRCA) for target classification and showed significantly higher performance compared with the original TRCA in the offline analysis. In the online system, the dynamic stopping (DS) strategy based on Bayesian posterior probability was utilized to realize alterable stimulating time. In addition, the temporal filtering process and the programs were optimized to facilitate the online DS operation. Notably, the online ITR reached 330.4 ± 45.4 bits/min on average, which is significantly higher than that of fixed stopping (FS) strategy, and the peak value of 420.2 bits/min is the highest online spelling ITR with a SSVEP-BCI up to now. The proposed system with portable EEG acquisition, friendly interaction, and alterable time of command output provides more flexibility for SSVEP-based BCIs and is promising for practical high-speed spelling.

BETA: A Large Benchmark Database Toward SSVEP-BCI Application

The brain-computer interface (BCI) provides an alternative means to communicate and it has sparked growing interest in the past two decades. Specifically, for Steady-State Visual Evoked Potential (SSVEP) based BCI, marked improvement has been made in the frequency recognition method and data sharing. However, the number of pubic databases is still limited in this field. Therefore, we present a BEnchmark database Towards BCI Application (BETA) in the study. The BETA database is composed of 64-channel Electroencephalogram (EEG) data of 70 subjects performing a 40-target cued-spelling task. The design and the acquisition of the BETA are in pursuit of meeting the demand from real-world applications and it can be used as a test-bed for these scenarios. We validate the database by a series of analyses and conduct the classification analysis of eleven frequency recognition methods on BETA. We recommend using the metric of wide-band signal-to-noise ratio (SNR) and BCI quotient to characterize the SSVEP at the single-trial and population levels, respectively. The BETA database can be downloaded from the following link http://bci.med.tsinghua.edu.cn/download.html.

Spatial Filtering in SSVEP-Based BCIs: Unified Framework and New Improvements

OBJECTIVE

In the steady-state visual evoked potential (SSVEP)-based brain computer interfaces (BCIs), spatial filtering, which combines the multi-channel electroencephalography (EEG) signals in order to reduce the non-SSVEP-related component and thus enhance the signal-to-noise ratio (SNR), plays an important role in target recognition. Recently, various spatial filtering algorithms have been developed employing different prior knowledge and characteristics of SSVEPs, however how these algorithms interconnect and differ is not yet fully explored, leading to difficulties in further understanding, utilizing and improving them.

METHODS

We propose a unified framework under which the spatial filtering algorithms can be formulated as generalized eigenvalue problems (GEPs) with four different elements: data, temporal filter, orthogonal projection and spatial filter. Based on the framework, we design new spatial filtering algorithms for improvements through the choice of different elements.

RESULTS

The similarities, differences and relationships among nineteen mainstream spatial filtering algorithms are revealed under the proposed framework. Particularly, it is found that they originate from the canonical correlation analysis (CCA), principal component analysis (PCA), and multi-set CCA, respectively. Furthermore, three new spatial filtering algorithms are developed with enhanced performance validated on two public SSVEP datasets with 45 subjects.

CONCLUSION

The proposed framework provides insights into the underlying relationships among different spatial filtering algorithms and helps the design of new spatial filtering algorithms.

SIGNIFICANCE

This is a systematic study to explore, compare and improve the existing spatial filtering algorithms, which would be significant for further understanding and future development of high performance SSVEP-based BCIs.

Simultaneous Decoding of Eccentricity and Direction Information for a Single-Flicker SSVEP BCI

The feasibility of a steady-state visual evoked potential (SSVEP) brain–computer interface (BCI) with a single-flicker stimulus for multiple-target decoding has been demonstrated in a number of recent studies. The single-flicker BCIs have mainly employed the direction information for encoding the targets, i.e., different targets are placed at different spatial directions relative to the flicker stimulus. The present study explored whether visual eccentricity information can also be used to encode targets for the purpose of increasing the number of targets in the single-flicker BCIs. A total number of 16 targets were encoded, placed at eight spatial directions, and two eccentricities (2.5° and 5°) relative to a 12 Hz flicker stimulus. Whereas distinct SSVEP topographies were elicited when participants gazed at targets of different directions, targets of different eccentricities were mainly represented by different signal-to-noise ratios (SNRs). Using a canonical correlation analysis-based classification algorithm, simultaneous decoding of both direction and eccentricity information was achieved, with an offline 16-class accuracy of 66.8 ± 16.4% averaged over 12 participants and a best individual accuracy of 90.0%. Our results demonstrate a single-flicker BCI with a substantially increased target number towards practical applications.