Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain-computer interface

Toward the enhancement of affective brain-computer interfaces using dependence within EEG series

In recent years, electroencephalogram (EEG)-based affective brain-computer interfaces (aBCI) has made remarkable advances.Objective. However, a subtle but crucial problem caused by the sliding window method has long been overlooked, which is the serious quantity mismatch between stimuli and short-term EEG frames. This may be an important factor limiting the performance of aBCIs.Approach. We refer to this mismatch as the quantity-independence imbalance (Q/I imbalance) and we propose the weak independence hypothesis to explain the mismatch. To validate this hypothesis and explore the effects of the Q/I imbalance on short-term EEG frames, we design four experiments from four perspectives, which are visualization, cross-validation, randomness test, and redundancy test.Main results. Inspired by validation experiments, we propose an inference correction (IC) method to enhancing the emotional predictions by leveraging the majority of the classifier's outputs. The proposed IC method is evaluated on two datasets involving 60 subjects using both intra-subject and inter-subject validation protocols. Our IC achieves a significant improvement of 14.97% in classification accuracy.Significance. This study promotes the understanding of the time-dependent nature of EEG signals in aBCI.

Enhancing detection of SSVEP-based BCIs via a novel temporally local canonical correlation analysis

BACKGROUND

In recent years, spatial filter-based frequency recognition methods have become popular in steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems. However, these methods are ineffective in suppressing local noise, and they rely on the length of the data. In practical applications, enhancing recognition performance with short data windows is a significant challenge for the BCI systems.

NEW METHOD

With extracting temporal information and eliminating local noise, a temporally local canonical correlation analysis based on training data-driven (TI-tdCCA) method is proposed to enhance the recognition performance of SSVEPs. Based on a novel framework, the filters are derived by incorporating the Laplacian matrix through the use of TI-CCA between the concatenated training data and individual templates. The target frequency is subsequently determined by applying the appropriate spatial filters and Laplacian matrix.

RESULTS

The experimental results on two datasets, consisting of 40 classes and recording from 35 and 70 subjects respectively, demonstrate that the proposed method consistently outperforms the eight competing methods in the majority of cases. The proposed method is simultaneously evaluated by an extended version that incorporates artificial reference signals. The extended method demonstrates a significant improvement over the proposed method. Specifically, with a time window of 0.7 s, the average recognition accuracy of the subjects increases by 10.71 % on the Benchmark dataset and by 6.98 % on the BETA dataset, respectively.

COMPARISON WITH EXISTING METHODS

Our extended method outperforms the state-of-the-art methods by at least 3 %, and it effectively suppresses local noise and maintains excellent scalability.

CONCLUSIONS FOR RESEARCH ARTICLES

The proposed method can effectively combine spatial and temporal filters to improve the recognition performance of SSVEPs.

Stimulus-Informed Generalized Canonical Correlation Analysis for Group Analysis of Neural Responses to Natural Stimuli

Various new brain-computer interface technologies or neuroscience applications require decoding stimulus-following neural responses to natural stimuli such as speech and video from, e.g., electroencephalography (EEG) signals. In this context, generalized canonical correlation analysis (GCCA) is often used as a group analysis technique, which allows the extraction of correlated signal components from the neural activity of multiple subjects attending to the same stimulus. GCCA can be used to improve the signal-to-noise ratio of the stimulus-following neural responses relative to all other irrelevant (non-)neural activity, or to quantify the correlated neural activity across multiple subjects in a group-wise coherence metric. However, the traditional GCCA technique is stimulus-unaware: no information about the stimulus is used to estimate the correlated components from the neural data of several subjects. Therefore, the GCCA technique might fail to extract relevant correlated signal components in practical situations where the amount of information is limited, for example, because of a limited amount of training data or group size. This motivates a new stimulus-informed GCCA (SI-GCCA) framework that allows taking the stimulus into account to extract the correlated components. We show that SI-GCCA outperforms GCCA in various practical settings, for both auditory and visual stimuli. Moreover, we showcase how SI-GCCA can be used to steer the estimation of the components towards the stimulus. As such, SI-GCCA substantially improves upon GCCA for various purposes, ranging from preprocessing to quantifying attention.

A Bibliometric Review of Brain-Computer Interfaces in Motor Imagery and Steady-State Visually Evoked Potentials for Applications in Rehabilitation and Robotics

In this paper, a bibliometric review is conducted on brain-computer interfaces (BCI) in non-invasive paradigms like motor imagery (MI) and steady-state visually evoked potentials (SSVEP) for applications in rehabilitation and robotics. An exploratory and descriptive approach is used in the analysis. Computational tools such as the biblioshiny application for R-Bibliometrix and VOSViewer are employed to generate data on years, sources, authors, affiliation, country, documents, co-author, co-citation, and co-occurrence. This article allows for the identification of different bibliometric indicators such as the research process, evolution, visibility, volume, influence, impact, and production in the field of brain-computer interfaces for MI and SSVEP paradigms in rehabilitation and robotics applications from 2000 to August 2024.

Novel Sinusoidal Signal Assisted Multivariate Variational Mode Decomposition Combined With Task-Related Component Analysis for Enhancing SSVEP-Based BCI Performance

Brain-computer interfaces (BCIs) based on steady-state visually evoked potential (SSVEP) have a broad application prospect owing to their multiple command output and high performance. Each harmonic component of SSVEP individually contains unique features, which can be utilized to enhance the recognition performance of SSVEP-based BCIs. However, the existing subband analysis methods for SSVEP, including those based on filter banks and existing mode decomposition methods, have limitations in extracting and utilizing independent harmonic components. This study proposes a sinusoidal signal assisted multivariate variational mode decomposition (SA-MVMD) algorithm that allows the constraint of the center frequencies and narrowband filtering structures of the intrinsic mode functions (IMFs) based on the prior frequency knowledge of the signal. It preserves the target information of the signal during decomposition while avoiding mode mixing and incorrect decomposition, thereby enabling the effective extraction of each independent harmonic component of SSVEP. Building on this, a SA-MVMD based task-related component analysis (SA-MVMD-TRCA) method is further proposed to fully utilize the features within the overall SSVEP as well as its independent harmonics, thereby enhancing the recognition performance. Testing on the public SSVEP Benchmark dataset demonstrates that the proposed method significantly outperforms the filter bank-based control methods. This study confirms the effectiveness of SA-MVMD and the potential of this approach, which analyzes and utilizes each independent harmonic of SSVEP, providing new strategies and perspectives for performance enhancement in SSVEP-based BCIs.

Development of a humanoid robot control system based on AR-BCI and SLAM navigation

Brain-computer interface (BCI)-based robot combines BCI and robotics technology to realize the brain's intention to control the robot, which not only opens up a new way for the daily care of the disabled individuals, but also provides a new way of communication for normal people. However, the existing systems still have shortcomings in many aspects such as friendliness of human-computer interaction, and interaction efficient. This study developed a humanoid robot control system by integrating an augmented reality (AR)-based BCI with a simultaneous localization and mapping (SLAM)-based scheme for autonomous indoor navigation. An 8-target steady-state visual evoked potential (SSVEP)-based BCI was implemented to enable direct control of the humanoid robot by the user. A Microsoft HoloLens was utilized to display visual stimuli for eliciting SSVEPs. Filter bank canonical correlation analysis (FBCCA), a training-free method, was used to detect SSVEPs in this study. By leveraging SLAM technology, the proposed system alleviates the need for frequent control commands transmission from the user, thereby effectively reducing their workload. Online results from 12 healthy subjects showed this developed BCI system was able to select a command out of eight potential targets with an average accuracy of 94.79%. The autonomous navigation subsystem enabled the humanoid robot to autonomously navigate to a destination chosen utilizing the proposed BCI. Furthermore, all participants successfully completed the experimental task using the developed system without any prior training. These findings illustrate the feasibility of the developed system and its potential to contribute novel insights into humanoid robots control strategies.

Using the Cocktail Party Effect to Add the Coding Dimension of Auditory Event Related Potential Brain-Computer Interface

OBJECTIVE

The auditory event-related potential based brain-computer interface (aERP-BCI) is a classical paradigm of brain-computer communication. To improve the coding efficiency of aERP-BCI, this study proposes a method using two parallel voice channels to add the coding dimension based on the cocktail party effect.

METHODS

The novel paradigm used male and female voices to establish two parallel oddball sound stimulus sequences. In comparison, the baseline paradigm only presented male or female stimulus sequences. Both the double voice condition (DVC) and the single voice condition (SVC) paradigms carried out offline experiments and the DVC also carried out online experiment. Subsequently, the EEG signal and BCI operation results were compared and analyzed.

CONCLUSION

The cocktail party effect caused a significant difference in the EEG responses of non-target stimulus between the focused vocal channel and the ignored vocal channel under the DVC paradigm, and the focused and ignored channels achieved a recognition accuracy of 97.2%. The target recognition rate of DVC was 82.3%, with no significant difference compared with 85% of SVC while the information transfer rate (ITR) of DVC reaching 15.3 bits/min was significantly higher than that of SVC.

SIGNIFICANCE

The cocktail party effect improves the coding efficiency by adding parallel channels without reducing the target/non-target stimulus recognition in the focused vocal channel. This provides a novel direction for the performance improvement of aERP-BCI.

Wasserstein generative adversarial network with gradient penalty and convolutional neural network based motor imagery EEG classification

Objective.Due to the difficulty in acquiring motor imagery electroencephalography (MI-EEG) data and ensuring its quality, insufficient training data often leads to overfitting and inadequate generalization capabilities of deep learning-based classification networks. Therefore, we propose a novel data augmentation method and deep learning classification model to enhance the decoding performance of MI-EEG further.Approach.The raw EEG signals were transformed into the time-frequency maps as the input to the model by continuous wavelet transform. An improved Wasserstein generative adversarial network with gradient penalty data augmentation method was proposed, effectively expanding the dataset used for model training. Additionally, a concise and efficient deep learning model was designed to improve decoding performance further.Main results.It has been demonstrated through validation by multiple data evaluation methods that the proposed generative network can generate more realistic data. Experimental results on the BCI Competition IV 2a and 2b datasets and the actual collected dataset show that classification accuracies are 83.4%, 89.1% and 73.3%, and Kappa values are 0.779, 0.782 and 0.644, respectively. The results indicate that the proposed model outperforms state-of-the-art methods.Significance.Experimental results demonstrate that this method effectively enhances MI-EEG data, mitigates overfitting in classification networks, improves MI classification accuracy, and holds positive implications for MI tasks.

Multi-Task Heterogeneous Ensemble Learning-Based Cross-Subject EEG Classification Under Stroke Patients

Robot-assisted motor training is applied for neurorehabilitation in stroke patients, using motor imagery (MI) as a representative paradigm of brain-computer interfaces to offer real-life assistance to individuals facing movement challenges. However, the effectiveness of training with MI may vary depending on the location of the stroke lesion, which should be considered. This paper introduces a multi-task electroencephalogram-based heterogeneous ensemble learning (MEEG-HEL) specifically designed for cross-subject training. In the proposed framework, common spatial patterns were used for feature extraction, and the features according to stroke lesions are shared and selected through sequential forward floating selection. The heterogeneous ensembles were used as classifiers. Nine patients with chronic ischemic stroke participated, engaging in MI and motor execution (ME) paradigms involving finger tapping. The classification criteria for the multi-task were established in two ways, taking into account the characteristics of stroke patients. In the cross-subject session, the first involved a direction recognition task for two-handed classification, achieving a performance of 0.7419 (±0.0811) in MI and 0.7061 (±0.1270) in ME. The second task focused on motor assessment for lesion location, resulting in a performance of 0.7457 (±0.1317) in MI and 0.6791 (±0.1253) in ME. Comparing the specific-subject session, except for ME on the motor assessment task, performance on both tasks was significantly higher than the cross-subject session. Furthermore, classification performance was similar to or statistically higher in cross-subject sessions compared to baseline models. The proposed MEEG-HEL holds promise in improving the practicality of neurorehabilitation in clinical settings and facilitating the detection of lesions.

BCI-controlled wheelchairs: end-users' perceptions, needs, and expectations, an interview-based study

PURPOSE

Brain-computer interface (BCI)-controlled wheelchairs have the potential to improve the independence of people with mobility impairments. The low uptake of BCI devices has been linked to a lack of knowledge among researchers of the needs of end-users that should influence BCI development.

MATERIALS AND METHODS

This study used semi-structured interviews to learn about the perceptions, needs, and expectations of spinal cord injury (SCI) patients with regards to a BCI-controlled wheelchair. Topics discussed in the interview include: paradigms, shared control, safety, robustness, channel selection, hardware, and experimental design. The interviews were recorded and then transcribed. Analysis was carried out using coding based on grounded theory principles.

RESULTS

The majority of participants had a positive view of BCI-controlled wheelchair technology and were willing to use the technology. Core issues were raised regarding safety, cost and aesthetics. Interview discussions were linked to state-of-the-art BCI technology. The results challenge the current reliance of researchers on the motor-imagery paradigm by suggesting end-users expect highly intuitive paradigms. There also needs to be a stronger focus on obstacle avoidance and safety features in BCI wheelchairs. Finally, the development of control approaches that can be personalized for individual users may be instrumental for widespread adoption of these devices.

CONCLUSIONS

This study, based on interviews with SCI patients, indicates that BCI-controlled wheelchairs are a promising assistive technology that would be well received by end-users. Recommendations for a more person-centered design of BCI controlled wheelchairs are made and clear avenues for future research are identified.

A comparative study of stereo-dependent SSVEP targets and their impact on VR-BCI performance

Steady-state visual evoked potential brain-computer interfaces (SSVEP-BCI) have attracted significant attention due to their ease of deployment and high performance in terms of information transfer rate (ITR) and accuracy, making them a promising candidate for integration with consumer electronics devices. However, as SSVEP characteristics are directly associated with visual stimulus attributes, the influence of stereoscopic vision on SSVEP as a critical visual attribute has yet to be fully explored. Meanwhile, the promising combination of virtual reality (VR) devices and BCI applications is hampered by the significant disparity between VR environments and traditional 2D displays. This is not only due to the fact that screen-based SSVEP generally operates under static, stable conditions with simple and unvaried visual stimuli but also because conventional luminance-modulated stimuli can quickly induce visual fatigue. This study attempts to address these research gaps by designing SSVEP paradigms with stereo-related attributes and conducting a comparative analysis with the traditional 2D planar paradigm under the same VR environment. This study proposed two new paradigms: the 3D paradigm and the 3D-Blink paradigm. The 3D paradigm induces SSVEP by modulating the luminance of spherical targets, while the 3D-Blink paradigm employs modulation of the spheres' opacity instead. The results of offline 4-object selection experiments showed that the accuracy of 3D and 2D paradigm was 85.67 and 86.17% with canonical correlation analysis (CCA) and 86.17 and 91.73% with filter bank canonical correlation analysis (FBCCA), which is consistent with the reduction in the signal-to-noise ratio (SNR) of SSVEP harmonics for the 3D paradigm observed in the frequency-domain analysis. The 3D-Blink paradigm achieved 75.00% of detection accuracy and 27.02 bits/min of ITR with 0.8 seconds of stimulus time and task-related component analysis (TRCA) algorithm, demonstrating its effectiveness. These findings demonstrate that the 3D and 3D-Blink paradigms supported by VR can achieve improved user comfort and satisfactory performance, while further algorithmic optimization and feature analysis are required for the stereo-related paradigms. In conclusion, this study contributes to a deeper understanding of the impact of binocular stereoscopic vision mechanisms on SSVEP paradigms and promotes the application of SSVEP-BCI in diverse VR environments.

A Robotic Teleoperation System Enhanced by Augmented Reality for Natural Human-Robot Interaction

Telekinesis, as commonly portrayed in science fiction literature and cinema, is a super power wherein users control and manipulate objects absent in physical interaction. In real world, enhancing human-robot interaction needs the synthesis of human intuitive processes with robotic arm. This paper introduces a robotic teleoperation system achieving the essence of telekinetic operations, combining the profound capabilities of augmented reality (AR) with the robotic arm operations. Utilizing AR, the proposed methodology offers operators with a visual feedback, facilitating a level of control surpassing the capacities of natural interfaces. By using AR-driven visual recognition, this system achieves operations in a virtual environment, subsequently actualized in the real world through the robotic arm. Through multiple experiments, we found that the system has a small margin of error in telekinesis operations, meeting the needs of remote operation. Furthermore, our system can operate on objects in the real world. These experiments underscore the capability of the remote control system to assist humans in accomplishing a wider range of tasks through the integration of AR and robotic arms, providing a natural human-robot interaction approach.

A novel brain-controlled prosthetic hand method integrating AR-SSVEP augmentation, asynchronous control, and machine vision assistance

Background and objective

The brain-computer interface (BCI) system based on steady-state visual evoked potentials (SSVEP) is expected to help disabled patients achieve alternative prosthetic hand assistance. However, the existing study still has some shortcomings in interaction aspects such as stimulus paradigm and control logic. The purpose of this study is to innovate the visual stimulus paradigm and asynchronous decoding/control strategy by integrating augmented reality technology, and propose an asynchronous pattern recognition algorithm, thereby improving the interaction logic and practical application capabilities of the prosthetic hand with the BCI system.

Methods

An asynchronous visual stimulus paradigm based on an augmented reality (AR) interface was proposed in this paper, in which there were 8 control modes, including Grasp, Put down, Pinch, Point, Fist, Palm push, Hold pen, and Initial. According to the attentional orienting characteristics of the paradigm, a novel asynchronous pattern recognition algorithm that combines center extended canonical correlation analysis and support vector machine (Center-ECCA-SVM) was proposed. Then, this study proposed an intelligent BCI system switch based on a deep learning object detection algorithm (YOLOv4) to improve the level of user interaction. Finally, two experiments were designed to test the performance of the brain-controlled prosthetic hand system and its practical performance in real scenarios.

Results

Under the AR paradigm of this study, compared with the liquid crystal display (LCD) paradigm, the average SSVEP spectrum amplitude of multiple subjects increased by 17.41%, and the signal-noise ratio (SNR) increased by 3.52%. The average stimulus pattern recognition accuracy was 96.71 ± 3.91%, which was 2.62% higher than the LCD paradigm. Under the data analysis time of 2s, the Center-ECCA-SVM classifier obtained 94.66 ± 3.87% and 97.40 ± 2.78% asynchronous pattern recognition accuracy under the Normal metric and the Tolerant metric, respectively. And the YOLOv4-tiny model achieves a speed of 25.29fps and a 96.4% confidence in the prosthetic hand in real-time detection. Finally, the brain-controlled prosthetic hand helped the subjects to complete 4 kinds of daily life tasks in the real scene, and the time-consuming were all within an acceptable range, which verified the effectiveness and practicability of the system.

Conclusion

This research is based on improving the user interaction level of the prosthetic hand with the BCI system, and has made improvements in the SSVEP paradigm, asynchronous pattern recognition, interaction, and control logic. Furthermore, it also provides support for BCI areas for alternative prosthetic control, and movement disorder rehabilitation programs.

Optimizing Visual Stimulation Paradigms for User-Friendly SSVEP-Based BCIs

In steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems, traditional flickering stimulation patterns face challenges in achieving a trade-off in both BCI performance and visual comfort across various frequency bands. To investigate the optimal stimulation paradigms with high performance and high comfort for each frequency band, this study systematically compared the characteristics of SSVEP and user experience of different stimulation paradigms with a wide stimulation frequency range of 1-60 Hz. The findings suggest that, for a better balance between system performance and user experience, ON and OFF grid stimuli with a Weber contrast of 50% can be utilized as alternatives to traditional flickering stimulation paradigms in the frequency band of 1-25 Hz. In the 25-35 Hz range, uniform flicker stimuli with the same 50% contrast are more suitable. In the higher frequency band, traditional uniform flicker stimuli with a high 300% contrast are preferred. These results are significant for developing high performance and user-friendly SSVEP-based BCI systems.

A Novel Data Augmentation Approach Using Mask Encoding for Deep Learning-Based Asynchronous SSVEP-BCI

Deep learning (DL)-based methods have been successfully employed as asynchronous classification algorithms in the steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) system. However, these methods often suffer from the limited amount of electroencephalography (EEG) data, leading to overfitting. This study proposes an effective data augmentation approach called EEG mask encoding (EEG-ME) to mitigate overfitting. EEG-ME forces models to learn more robust features by masking partial EEG data, leading to enhanced generalization capabilities of models. Three different network architectures, including an architecture integrating convolutional neural networks (CNN) with Transformer (CNN-Former), time domain-based CNN (tCNN), and a lightweight architecture (EEGNet) are utilized to validate the effectiveness of EEG-ME on publicly available benchmark and BETA datasets. The results demonstrate that EEG-ME significantly enhances the average classification accuracy of various DL-based methods with different data lengths of time windows on two public datasets. Specifically, CNN-Former, tCNN, and EEGNet achieve respective improvements of 3.18%, 1.42%, and 3.06% on the benchmark dataset as well as 11.09%, 3.12%, and 2.81% on the BETA dataset, with the 1-second time window as an example. The enhanced performance of SSVEP classification with EEG-ME promotes the implementation of the asynchronous SSVEP-BCI system, leading to improved robustness and flexibility in human-machine interaction.

Temporal hyper-connectivity and frontal hypo-connectivity within gamma band in schizophrenia: A resting state EEG study

OBJECTIVE

The brain network serves as the physiological foundation for information processing of the brain. Many studies have reported abnormalities of gamma oscillations in Schizophrenia. The aim of this study was to investigate the gamma-band connectivity in Schizophrenia patients.

METHODS

We recorded the resting state electroencephalogram (EEG) for 15 schizophrenia patients with refractory auditory hallucinations and 14 healthy controls, with eyes open and closed. The brain network was constructed based on weighted phase lag index for gamma band. Whole scalp metrics (clustering coefficient, global efficiency and local efficiency) and local region metrics (degree and betweenness centrality) in the frontal and temporal lobes were computed. Correlation analyses between network metrics and symptom scales were examined to find associations with symptom severity.

RESULTS

Schizophrenia patients had larger global efficiency and local efficiency (p < 0.05) with eyes closed, probably representing greater brain activity and information exchange. For degree and betweenness centrality, schizophrenia patients showed an increase (p < 0.05) in the temporal lobe but a decrease (p < 0.05) in the frontal lobe with eyes closed and open, potentially account for the patients' symptoms such as hallucinations and thought disorders. Local efficiency and frontal lobe degree were positively and negatively correlated with the scales, respectively (both p < 0.05).

CONCLUSIONS

Altered connectivity of the resting state brain network has been revealed and may be associated with the core symptoms of schizophrenia. Our study provides promising evidence for the investigation of the pathological basis of Schizophrenia and could aid in objective diagnosis.

Classification of EEG Signals Based on Sparrow Search Algorithm-Deep Belief Network for Brain-Computer Interface

In brain-computer interface (BCI) systems, challenges are presented by the recognition of motor imagery (MI) brain signals. Established recognition approaches have achieved favorable performance from patterns like SSVEP, AEP, and P300, whereas the classification methods for MI need to be improved. Hence, seeking a classification method that exhibits high accuracy and robustness for application in MI-BCI systems is essential. In this study, the Sparrow search algorithm (SSA)-optimized Deep Belief Network (DBN), called SSA-DBN, is designed to recognize the EEG features extracted by the Empirical Mode Decomposition (EMD). The performance of the DBN is enhanced by the optimized hyper-parameters obtained through the SSA. Our method's efficacy was tested on three datasets: two public and one private. Results indicate a relatively high accuracy rate, outperforming three baseline methods. Specifically, on the private dataset, our approach achieved an accuracy of 87.83%, marking a significant 10.38% improvement over the standard DBN algorithm. For the BCI IV 2a dataset, we recorded an accuracy of 86.14%, surpassing the DBN algorithm by 9.33%. In the SMR-BCI dataset, our method attained a classification accuracy of 87.21%, which is 5.57% higher than that of the conventional DBN algorithm. This study demonstrates enhanced classification capabilities in MI-BCI, potentially contributing to advancements in the field of BCI.

A Novel Hybrid Brain-Computer Interface Combining the Illusion-Induced VEP and SSVEP

Traditional single-modality brain-computer interface (BCI) systems are limited by their reliance on a single characteristic of brain signals. To address this issue, incorporating multiple features from EEG signals can provide robust information to enhance BCI performance. In this study, we designed and implemented a novel hybrid paradigm that combined illusion-induced visual evoked potential (IVEP) and steady-state visual evoked potential (SSVEP) with the aim of leveraging their features simultaneously to improve system efficiency. The proposed paradigm was validated through two experimental studies, which encompassed feature analysis of IVEP with a static paradigm, and performance evaluation of hybrid paradigm in comparison with the conventional SSVEP paradigm. The characteristic analysis yielded significant differences in response waveforms among different motion illusions. The performance evaluation of the hybrid BCI demonstrates the advantage of integrating illusory stimuli into the SSVEP paradigm. This integration effectively enhanced the spatio-temporal features of EEG signals, resulting in higher classification accuracy and information transfer rate (ITR) within a short time window when compared to traditional SSVEP-BCI in four-command task. Furthermore, the questionnaire results of subjective estimation revealed that proposed hybrid BCI offers less eye fatigue, and potentially higher levels of concentration, physical condition, and mental condition for users. This work first introduced the IVEP signals in hybrid BCI system that could enhance performance efficiently, which is promising to fulfill the requirements for efficiency in practical BCI control systems.

A High-Rate Hybrid BCI System Based on High-Frequency SSVEP and sEMG

Recently, various biosignals have been combined with electroencephalography (EEG) to build hybrid brain-computer interface (BCI) systems to improve system performance. Since steady-state visual evoked potential (SSVEP) and surface electromyography (sEMG) are easy-to-use, non-invasive techniques, and have high signal-to-noise ratio (SNR), hybrid BCI systems combining SSVEP and sEMG have received much attention in the BCI literature. However, most existing studies regarding hybrid BCIs based on SSVEP and sEMG adopt low-frequency visual stimuli to induce SSVEPs. The comfort of these systems needs further improvement to meet the practical application requirements. The present study realized a novel hybrid BCI combining high-frequency SSVEP and sEMG signals for spelling applications. EEG and sEMG were obtained simultaneously from the scalp and skin surface of subjects, respectively. These two types of signals were analyzed independently and then combined to determine the target stimulus. Our online results demonstrated that the developed hybrid BCI yielded a mean accuracy of 88.07 ± 1.43% and ITR of 159.12 ± 4.31 bits/min. These results exhibited the feasibility and effectiveness of fusing high-frequency SSVEP and sEMG towards improving the total BCI system performance.

Phase-Locked Time-Shift Data Augmentation Method for SSVEP Brain-Computer Interfaces

Steady-state visual evoked potential (SSVEP) based brain-computer interfaces (BCIs) have achieved an information transfer rate (ITR) of over 300 bits/min, but abundant training data is required. The performance of SSVEP algorithms deteriorates greatly under limited data, and the existing time-shift data augmentation method fails to improve it because the phase-locked requirement between training samples is violated. To address this issue, this study proposes a novel augmentation method, namely phase-locked time-shift (PLTS), for SSVEP-BCI. The similarity between epochs at different time moments was evaluated, and a unique time-shift step was calculated for each class to augment additional data epochs in each trial. The results showed that the PLTS significantly improved the classification performance of SSVEP algorithms on the BETA SSVEP datasets. Moreover, under the condition of one calibration block, by slightly prolonging the calibration duration (from 48 s to 51.5 s), the ITR increased from 40.88±4.54 bits/min to 122.61±7.05 bits/min with the PLTS. This study provides a new perspective on augmenting data epochs for training-based SSVEP-BCI, promotes the classification accuracy and ITR under limited training data, and thus facilitates the real-life applications of SSVEP-based brain spellers.

Investigating EEG-based cross-session and cross-task vigilance estimation in BCI systems

Objective. The state of vigilance is crucial for effective performance in brain-computer interface (BCI) tasks, and therefore, it is essential to investigate vigilance levels in BCI tasks. Despite this, most studies have focused on vigilance levels in driving tasks rather than on BCI tasks, and the electroencephalogram (EEG) patterns of vigilance states in different BCI tasks remain unclear. This study aimed to identify similarities and differences in EEG patterns and performances of vigilance estimation in different BCI tasks and sessions.Approach.To achieve this, we built a steady-state visual evoked potential-based BCI system and a rapid serial visual presentation-based BCI system and recruited 18 participants to carry out four BCI experimental sessions over four days.Main results. Our findings demonstrate that specific neural patterns for high and low vigilance levels are relatively stable across sessions. Differential entropy features significantly differ between different vigilance levels in all frequency bands and between BCI tasks in the delta and theta frequency bands, with the theta frequency band features playing a critical role in vigilance estimation. Additionally, prefrontal, temporal, and occipital regions are more relevant to the vigilance state in BCI tasks. Our results suggest that cross-session vigilance estimation is more accurate than cross-task estimation.Significance.Our study clarifies the underlying mechanisms of vigilance state in two BCI tasks and provides a foundation for further research in vigilance estimation in BCI applications.

Shared Three-Dimensional Robotic Arm Control Based on Asynchronous BCI and Computer Vision

OBJECTIVE

A brain-computer interface (BCI) can be used to translate neuronal activity into commands to control external devices. However, using noninvasive BCI to control a robotic arm for movements in three-dimensional (3D) environments and accomplish complicated daily tasks, such as grasping and drinking, remains a challenge.

APPROACH

In this study, a shared robotic arm control system based on hybrid asynchronous BCI and computer vision was presented. The BCI model, which combines steady-state visual evoked potentials (SSVEPs) and blink-related electrooculography (EOG) signals, allows users to freely choose from fifteen commands in an asynchronous mode corresponding to robot actions in a 3D workspace and reach targets with a wide movement range, while computer vision can identify objects and assist a robotic arm in completing more precise tasks, such as grasping a target automatically.

RESULTS

Ten subjects participated in the experiments and achieved an average accuracy of more than 92% and a high trajectory efficiency for robot movement. All subjects were able to perform the reach-grasp-drink tasks successfully using the proposed shared control method, with fewer error commands and shorter completion time than with direct BCI control.

SIGNIFICANCE

Our results demonstrated the feasibility and efficiency of generating practical multidimensional control of an intuitive robotic arm by merging hybrid asynchronous BCI and computer vision-based recognition.

BCI Control of a Robotic Arm Based on SSVEP With Moving Stimuli for Reach and Grasp Tasks

Brain-computer interface (BCI) provides a novel technology for patients and healthy human subjects to control a robotic arm. Currently, BCI control of a robotic arm to complete the reaching and grasping tasks in an unstructured environment is still challenging because the current BCI technology does not meet the requirement of manipulating a multi-degree robotic arm accurately and robustly. BCI based on steady-state visual evoked potential (SSVEP) could output a high information transfer rate; however, the conventional SSVEP paradigm failed to control a robotic arm to move continuously and accurately because the users have to switch their gaze between the flickering stimuli and the target frequently. This study proposed a novel SSVEP paradigm in which the flickering stimuli were attached to the robotic arm's gripper and moved with it. First, an offline experiment was designed to investigate the effects of moving flickering stimuli on the SSVEP's responses and decoding accuracy. After that, contrast experiments were conducted, and twelve subjects were recruited to participate in a robotic arm control experiment using both the paradigm one (P1, with moving flickering stimuli) and the paradigm two (P2, conventional fixed flickering stimuli) using a block randomization design to balance their sequences. Double blinks were used to trigger the grasping action asynchronously whenever the subjects were confident that the position of the robotic arm's gripper was accurate enough. Experimental results showed that the paradigm P1 with moving flickering stimuli provided a much better control performance than the conventional paradigm P2 in completing a reaching and grasping task in an unstructured environment. Subjects' subjective feedback scored by a NASA-TLX mental workload scale also corroborated the BCI control performance. The results of this study suggest that the proposed control interface based on SSVEP BCI provides a better solution for robotic arm control to complete the accurate reaching and grasping tasks.

SSVEP detection assessment by combining visual stimuli paradigms and no-training detection methods

Introduction

Brain-Computer Interfaces (BCI) based on Steady-State Visually Evoked Potentials (SSVEP) have great potential for use in communication applications because of their relatively simple assembly and in some cases the possibility of bypassing the time-consuming training stage. However, among multiple factors, the efficient performance of this technology is highly dependent on the stimulation paradigm applied in combination with the SSVEP detection algorithm employed. This paper proposes the performance assessment of the classification of target events with respect to non-target events by applying four types of visual paradigms, rectangular modulated On-Off (OOR), sinusoidal modulated On-Off (OOS), rectangular modulated Checkerboard (CBR), and sinusoidal modulated Checkerboard (CBS), with three types of SSVEP detection methods, Canonical Correlation Analysis (CCA), Filter-Bank CCA (FBCCA), and Minimum Energy Combination (MEC).

Methods

We set up an experimental protocol in which the four types of visual stimuli were presented randomly to twenty-seven participants and after acquiring their electroencephalographic responses to five stimulation frequencies (8.57, 10.909, 15, 20, and 24 Hz), the three detection methods were applied to the collected data.

Results

The results are conclusive, obtaining the best performance with the combination of either OOR or OOS visual stimulus and the FBCCA as a detection method, however, this finding contrasts with the opinion of almost half of the participants in terms of visual comfort, where the 51.9% of the subjects felt more comfortable and focused with CBR or CBS stimulation.

Discussion

Finally, the EEG recordings correspond to the SSVEP response of 27 subjects to four visual paradigms when selecting five items on a screen, which is useful in BCI navigation applications. The dataset is available to anyone interested in studying and evaluating signal processing and machine-learning algorithms for SSVEP-BCI systems.

CVT-Based Asynchronous BCI for Brain-Controlled Robot Navigation

Brain-computer interface (BCI) is a typical direction of integration of human intelligence and robot intelligence. Shared control is an essential form of combining human and robot agents in a common task, but still faces a lack of freedom for the human agent. This paper proposes a Centroidal Voronoi Tessellation (CVT)-based road segmentation approach for brain-controlled robot navigation by means of asynchronous BCI. An electromyogram-based asynchronous mechanism is introduced into the BCI system for self-paced control. A novel CVT-based road segmentation method is provided to generate optional navigation goals in the road area for arbitrary goal selection. An event-related potential of the BCI is designed for target selection to communicate with the robot. The robot has an autonomous navigation function to reach the human selected goals. A comparison experiment in the single-step control pattern is executed to verify the effectiveness of the CVT-based asynchronous (CVT-A) BCI system. Eight subjects participated in the experiment, and they were instructed to control the robot to navigate toward a destination with obstacle avoidance tasks. The results show that the CVT-A BCI system can shorten the task duration, decrease the command times, and optimize navigation path, compared with the single-step pattern. Moreover, this shared control mechanism of the CVT-A BCI system contributes to the promotion of human and robot agent integration control in unstructured environments.

A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment

At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system.

Activation of the brain during motor imagination task with auditory stimulation

Introduction

Auditory stimulation is one of the most important influence factors in the cognitive process. It is an important guiding role in cognitive motor process. However, previous studies on auditory stimuli mainly focused on the cognitive effects of auditory stimuli on the cortex, while the role of auditory stimuli in motor imagery tasks is still unclear.

Methods

In order to explore the role of auditory stimuli in motor imagery tasks, we studied the EEG power spectrum distribution characteristics, frontal parietal mismatch negative (MMN) wave characteristics, and the Inter trial phase locking consistency (ITPC) characteristics of the prefrontal cognitive cortex and parietal motor cortex. In this study, 18 subjects were hired to complete the motor imagery tasks, induced by auditory stimuli of task related verbs and task independent nouns.

Results

EEG power spectrum analysis showed that the activity of the contralateral motor cortex was significantly increased under the stimulation of verbs, and the amplitude of mismatch negative wave was also significantly increased. ITPC is mainly concentrated in μ, α, and γ bands in the process of motor imagery task guided by the auditory stimulus of verbs, while it is mainly concentrated in the β band under the nouns stimulation. This difference may be due to the impact of auditory cognitive process on motor imagery.

Discussion

We speculate that there may be a more complex mechanism for the effect of auditory stimulation on the inter test phase lock consistency. When the stimulus sound has the corresponding meaning to the motor action, the parietal motor cortex may be more affected by the cognitive prefrontal cortex, thus changing its normal response mode. This mode change is due to the joint action of motor imagination, cognitive and auditory stimuli. This study provides new insight into the neural mechanism of motor imagery task guided by auditory stimuli, and provides more information on the activity characteristics of the brain network in motor imagery task by cognitive auditory stimulation.

A hybrid P300-SSVEP brain-computer interface speller with a frequency enhanced row and column paradigm

Objective

This study proposes a new hybrid brain-computer interface (BCI) system to improve spelling accuracy and speed by stimulating P300 and steady-state visually evoked potential (SSVEP) in electroencephalography (EEG) signals.

Methods

A frequency enhanced row and column (FERC) paradigm is proposed to incorporate the frequency coding into the row and column (RC) paradigm so that the P300 and SSVEP signals can be evoked simultaneously. A flicker (white-black) with a specific frequency from 6.0 to 11.5 Hz with an interval of 0.5 Hz is assigned to one row or column of a 6 × 6 layout, and the row/column flashes are carried out in a pseudorandom sequence. A wavelet and support vector machine (SVM) combination is adopted for P300 detection, an ensemble task-related component analysis (TRCA) method is used for SSVEP detection, and the two detection possibilities are fused using a weight control approach.

Results

The implemented BCI speller achieved an accuracy of 94.29% and an information transfer rate (ITR) of 28.64 bit/min averaged across 10 subjects during the online tests. An accuracy of 96.86% is obtained during the offline calibration tests, higher than that of only using P300 (75.29%) or SSVEP (89.13%). The SVM in P300 outperformed the previous linear discrimination classifier and its variants (61.90–72.22%), and the ensemble TRCA in SSVEP outperformed the canonical correlation analysis method (73.33%).

Conclusion

The proposed hybrid FERC stimulus paradigm can improve the performance of the speller compared with the classical single stimulus paradigm. The implemented speller can achieve comparable accuracy and ITR to its state-of-the-art counterparts with advanced detection algorithms.

Enhancing Detection of Control State for High-Speed Asynchronous SSVEP-BCIs Using Frequency-Specific Framework

This study proposed a novel frequency-specific (FS) algorithm framework for enhancing control state detection using short data length toward high-performance asynchronous steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCI). The FS framework sequentially incorporated task-related component analysis (TRCA)-based SSVEP identification and a classifier bank containing multiple FS control state detection classifiers. For an input EEG epoch, the FS framework first identified its potential SSVEP frequency using the TRCA-based method and then recognized its control state using one of the classifiers trained on the features specifically related to the identified frequency. A frequency-unified (FU) framework that conducted control state detection using a unified classifier trained on features related to all candidate frequencies was proposed to compare with the FS framework. Offline evaluation using data lengths within 1 s found that the FS framework achieved excellent performance and significantly outperformed the FU framework. 14-target FS and FU asynchronous systems were separately constructed by incorporating a simple dynamic stopping strategy and validated using a cue-guided selection task in an online experiment. Using averaged data length of 591.63±5.65 ms, the online FS system significantly outperformed the FU system and achieved an information transfer rate, true positive rate, false positive rate, and balanced accuracy of 124.95±12.35 bits/min, 93.16±4.4%, 5.21±5.85%, and 92.89±4.02%, respectively. The FS system was also of higher reliability by accepting more correctly identified SSVEP trials and rejecting more wrongly identified ones. These results suggest that the FS framework has great potential to enhance the control state detection for high-speed asynchronous SSVEP-BCIs.

Enhancing Detection of Multi-Frequency-Modulated SSVEP Using Phase Difference Constrained Canonical Correlation Analysis

OBJECTIVE

Multi-frequency-modulated visual stimulation scheme has been shown effective for the steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) recently, especially in increasing the visual target number with less stimulus frequencies and mitigating the visual fatigue. However, the existing calibration-free recognition algorithms based on the traditional canonical correlation analysis (CCA) cannot provide the merited performance.

APPROACH

To improve the recognition performance, this study proposes a phase difference constrained CCA (pdCCA), which assumes that the multi-frequency-modulated SSVEPs share a common spatial filter over different frequencies and have a specified phase difference. Specifically, during the CCA computation, the phase differences of the spatially filtered SSVEPs are constrained using the temporal concatenation of the sine-cosine reference signals with the pre-defined initial phases.

MAIN RESULTS

We evaluate the performance of the proposed pdCCA-based method on three representative multi-frequency-modulated visual stimulation paradigms (i.e., based on the multi-frequency sequential coding, the dual-frequency, and the amplitude modulation). The evaluation results on four SSVEP datasets (Dataset Ia, Ib, II, and III) show that the pdCCA-based method can significantly outperform the current CCA method in terms of recognition accuracy. It improves the accuracy by 22.09% in Dataset Ia, 20.86% in Dataset Ib, 8.61% in Dataset II, and 25.85% in Dataset III.

SIGNIFICANCE

The pdCCA-based method, which actively controls the phase difference of the multi-frequency-modulated SSVEPs after spatial filtering, is a new calibration-free method for multi-frequency-modulated SSVEP-based BCIs.

Improving Performance of Motor Imagery-Based Brain-Computer Interface in Poorly Performing Subjects Using a Hybrid-Imagery Method Utilizing Combined Motor and Somatosensory Activity

The phenomena of brain-computer interface-inefficiency in transfer rates and reliability can hinder development and use of brain-computer interface technology. This study aimed to enhance the classification performance of motor imagery-based brain-computer interface (three-class: left hand, right hand, and right foot) of poor performers using a hybrid-imagery approach that combined motor and somatosensory activity. Twenty healthy subjects participated in these experiments involving the following three paradigms: (1) Control-condition: motor imagery only, (2) Hybrid-condition I: combined motor and somatosensory stimuli (same stimulus: rough ball), and (3) Hybrid-condition II: combined motor and somatosensory stimuli (different stimulus: hard and rough, soft and smooth, and hard and rough ball). The three paradigms for all participants, achieved an average accuracy of 63.60± 21.62%, 71.25± 19.53%, and 84.09± 12.79% using the filter bank common spatial pattern algorithm (5-fold cross-validation), respectively. In the poor performance group, the Hybrid-condition II paradigm achieved an accuracy of 81.82%, showing a significant increase of 38.86% and 21.04% in accuracy compared to the control-condition (42.96%) and Hybrid-condition I (60.78%), respectively. Conversely, the good performance group showed a pattern of increasing accuracy, with no significant difference between the three paradigms. The Hybrid-condition II paradigm provided high concentration and discrimination to poor performers in the motor imagery-based brain-computer interface and generated the enhanced event-related desynchronization pattern in three modalities corresponding to different types of somatosensory stimuli in motor and somatosensory regions compared to the Control-condition and Hybrid-condition I. The hybrid-imagery approach can help improve motor imagery-based brain-computer interface performance, especially for poorly performing users, thus contributing to the practical use and uptake of brain-computer interface.

Hybrid mental tasks based human computer interface via integration of pronunciation and motor imagery

Objective.Among the existing active brain-computer interfaces (BCI), the motor imagination (MI) is widely used. To operate the MI BCI effectively, subjects need to carry out trainings on corresponding imagining tasks. Here, we studied how to reduce the discomfort and fatigue of active BCI imaginary tasks and the inability to concentrate on them while improving the accuracy.Approach.This paper proposes a hybrid BCI composed of MI and pronunciation imagination (PI). The electroencephalogram signals of ten subjects are recognized by the adaptive Riemannian distance classification and the improved frequency selective filter-bank Common Spatial Pattern recognition.Main results.The results show that under the new paradigm with the combination of MI and PI, the recognition accuracy is higher than the MI alone. The highest recognition rate of the proposed hybrid system can reach more than 90%. Furthermore, through the subjects' scoring results of the operation difficulty, it is concluded that the designed hybrid paradigm is more operable than the traditional BCI paradigm.Significance.The separable tasks in the active BCI are limited and the accuracy needs to be improved. The new hybrid paradigm proposed by us improves the accuracy and operability of the active BCI system, providing a new possibility for the research direction of the active BCI.

An extended binary subband canonical correlation analysis detection algorithm oriented to the radial contraction-expansion motion steady-state visual evoked paradigm

The radial contraction-expansion motion paradigm is a novel steady-state visual evoked experimental paradigm, and the electroencephalography (EEG) evoked potential is different from the traditional luminance modulation paradigm. The signal energy is concentrated chiefly in the fundamental frequency, while the higher harmonic power is lower. Therefore, the conventional steady-state visual evoked potential recognition algorithms optimizing multiple harmonic response components, such as the extended canonical correlation analysis (eCCA) and task-related component analysis (TRCA) algorithm, have poor recognition performance under the radial contraction-expansion motion paradigm. This paper proposes an extended binary subband canonical correlation analysis (eBSCCA) algorithm for the radial contraction-expansion motion paradigm. For the radial contraction-expansion motion paradigm, binary subband filtering was used to optimize the weighting coefficients of different frequency response signals, thereby improving the recognition performance of EEG signals. The results of offline experiments involving 13 subjects showed that the eBSCCA algorithm exhibits a better performance than the eCCA and TRCA algorithms under the stimulation of the radial contraction-expansion motion paradigm. In the online experiment, the average recognition accuracy of 13 subjects was 88.68% ± 6.33%, and the average information transmission rate (ITR) was 158.77 ± 43.67 bits/min, which proved that the algorithm had good recognition effect signals evoked by the radial contraction-expansion motion paradigm.

Using Non-linear Dynamics of EEG Signals to Classify Primary Hand Movement Intent Under Opposite Hand Movement

Decoding human hand movement from electroencephalograms (EEG) signals is essential for developing an active human augmentation system. Although existing studies have contributed much to decoding single-hand movement direction from EEG signals, decoding primary hand movement direction under the opposite hand movement condition remains open. In this paper, we investigated the neural signatures of the primary hand movement direction from EEG signals under the opposite hand movement and developed a novel decoding method based on non-linear dynamics parameters of movement-related cortical potentials (MRCPs). Experimental results showed significant differences in MRCPs between hand movement directions under an opposite hand movement. Furthermore, the proposed method performed well with an average binary decoding accuracy of 89.48 ± 5.92% under the condition of the opposite hand movement. This study may lay a foundation for the future development of EEG-based human augmentation systems for upper limbs impaired patients and healthy people and open a new avenue to decode other hand movement parameters (e.g., velocity and position) from EEG signals.