Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

Co-localized optode-electrode design for multimodal functional near infrared spectroscopy and electroencephalography

Significance

Neuroscience of the everyday world requires continuous mobile brain imaging in real time and in ecologically valid environments, which aids in directly translating research for human benefit. Combined functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) studies have increased in demand, as the combined systems can provide great insights into cortical hemodynamics, neuronal activity, and neurovascular coupling. However, fNIRS-EEG studies remain limited in modularity and portability due to restrictions in combined cap designs, especially for high-density (HD) fNIRS measurements.

Aim

We have built and tested custom fNIRS sources that attach to electrodes without decreasing the overall modularity and portability of the probe.

Approach

To demonstrate the design's utility, we screened for any potential interference and performed a HD-fNIRS-EEG measurement with co-located opto-electrode positions during a modified Stroop task.

Results

No observable interference was present from the fNIRS source optodes in the EEG spectral analysis. The performance, fNIRS, and EEG results of the Stroop task supported the trends from previous research. We observed increased activation with both fNIRS and EEG within the regions of interest.

Conclusion

Overall, these results suggest that the co-localization method is a promising approach to multimodal imaging.

Response coupling with an auxiliary neural signal for enhancing brain signal detection

Brain-computer interfaces (BCIs) offer an implicit, non-linguistic communication channel between users and machines. Despite their potential, BCIs are far from becoming a mainstream communication modality like text and speech. While non-invasive BCIs, such as Electroencephalography, are favored for their ease of use, their broader adoption is limited by challenges related to signal noise, artifacts, and variability across users. In this paper, we propose a novel method called response coupling, aimed at enhancing brain signal detection and reliability by pairing a brain signal with an artificially induced auxiliary signal and leveraging their interaction. Specifically, we use error-related potentials (ErrPs) as the primary signal and steady-state visual evoked potentials (SSVEPs) as the auxiliary signal. SSVEPs, known for their phase-locked responses to rhythmic stimuli, are selected because rhythmic neural activity plays a critical role in sensory and cognitive processes, with evidence suggesting that reinforcing these oscillations can improve neural performance. By exploring the interaction between these two signals, we demonstrate that response coupling significantly improves the detection accuracy of ErrPs, especially in the parietal and occipital regions. This method introduces a new paradigm for enhancing BCI performance, where the interaction between a primary and an auxiliary signal is harnessed to enhance the detection performance. Additionally, the phase-locking properties of SSVEPs allow for unsupervised rejection of suboptimal data, further increasing BCI reliability.

Electroencephalogram features reflect effort corresponding to graded finger extension: implications for hemiparetic stroke

Brain-computer interfaces (BCIs) offer disabled individuals the means to interact with devices by decoding the electroencephalogram (EEG). However, decoding intent in fine motor tasks can be challenging, especially in stroke survivors with cortical lesions. Here, we attempt to decode graded finger extension from the EEG in stroke patients with left-hand paresis and healthy controls. Participants extended their fingers to one of four levels: low, medium, high, or 'no-go' (none), while hand, muscle (electromyography: EMG), and brain (EEG) activity were monitored. Event-related desynchronization (ERD) was measured as the change in 8-30 Hz EEG power during movement. Classifiers were trained on EEG features, EMG power, or both (EEG+EMG) to decode finger extension, and accuracy assessed via four-fold cross-validation for each hand of each participant. Mean accuracy exceeded chance (25%) for controls (n = 11) at 62% for EMG, 60% for EEG, and 71% for EEG+EMG on the left hand; and 67%, 60%, and 74%, respectively, on the right hand. Accuracies were similar on the unimpaired right hand for the stroke group (n = 3): 61%, 68%, and 78%, respectively. But on the paretic left hand, EMG only discriminated no-go from movement above chance (41%); in contrast, EEG gave 65% accuracy (68% for EEG+EMG), comparable to the non-paretic hand. The median ERD was significant (p < 0.01) over the cortical hand area in both groups and increased with each level of finger extension. But while the ERD favored the hemisphere contralateral to the active hand as expected, it was ipsilateral for the left hand of stroke due to the lesion in the right hemisphere, which may explain its discriminative ability. Hence, the ERD captures effort in finger extension regardless of success or failure at the task; and harnessing residual EMG improves the correlation. This marker could be leveraged in rehabilitative protocols that focus on fine motor control.

Real-Time Attention Regulation and Cognitive Monitoring Using a Wearable EEG-Based BCI

OBJECTIVE

Attention regulation is an essential ability in daily life that affects learning and work efficiency and is closely related to mental health. The effectiveness of brain-computer interface (BCI) systems in attention regulation has been proven, but most of these systems rely on bulky and expensive equipment and are still in the experimental stage. This study proposes a wearable BCI system for real-time attention regulation and cognitive monitoring.

METHODS

The BCI system integrates a wearable single-channel electroencephalogram (EEG) headband with wireless data streaming for real-time analysis. Twenty healthy subjects participated in the long-term attention regulation experiment and were evenly divided into an experimental group and a control group based on the presence of real-time neurofeedback. The neurofeedback is represented by output value of attention, which calculated from single-channel EEG data. Before and after the regulation sessions, baseline assessments were conducted for each subject, incorporating multi-channel EEG data analysis and cognitive behavioral evaluations, to verify the effectiveness of system for attention regulation.

RESULTS

The online experimental results indicate that the average attention level in the experimental group increased from 0.625 to 0.812, while no significant improvement was observed in the control group. Further comparative analysis revealed the reasons for the enhancement of attention regulation ability in terms of both brain network patterns and cognitive performance.

SIGNIFICANCE

The proposed wearable BCI system is effective at improving attention regulation ability and could be a promising tool for assisting people with attention disorders.

A hybrid CNN model for classification of motor tasks obtained from hybrid BCI system

The Hybrid-Brain Computer Interface (BCI) has shown improved performance, especially in classifying multi-class data. Two non-invasive BCI modules are combined to achieve an improved classification which are Electroencephalogram (EEG) and functional Near Infra-red Spectroscopy (fNIRS). Classifying contralateral and ipsilateral motor movements is found challenging among the other mental activity signals. The current work focuses on the performance of deep learning methods like - Convolutional Neural Networks (CNN) and Bidirectional Long-Short term memory (Bi-LSTM) in classifying a four-class motor execution of Right Hand, Left Hand, Right Arm and Left Arm taken from the CORE dataset. The model performance was evaluated using metrics such as Accuracy, F1 - score, Precision, Recall, AUC and ROC curve. The CNN and Hybrid CNN models have resulted in 98.3% and 99% accuracy respectively.

Brain and muscle derived features to discriminate simple hand motor tasks for a rehabilitative BCI: comparative study on healthy and post-stroke individuals

Objective.Brain-Computer Interfaces targeting post-stroke recovery of the upper limb employ mainly electroencephalography to decode movement-related brain activation. Recently hybrid systems including muscular activity were introduced. We compared the motor task discrimination abilities of three different features, namely event-related desynchronization/synchronization (ERD/ERS) and movement-related cortical potential (MRCP) as brain-derived features and cortico-muscular coherence (CMC) as a hybrid brain-muscle derived feature, elicited in 13 healthy subjects and 13 stroke patients during the execution/attempt of two simple hand motor tasks (finger extension and grasping) commonly employed in upper limb rehabilitation protocols.Approach. We employed a three-way statistical design to investigate whether their ability to discriminate the two movements follows a specific temporal evolution along the movement execution and is eventually different among the three features and between the two groups. We also investigated the differences in performance at the single-subject level.Main results. The ERD/ERS and the CMC-based classification showed similar temporal evolutions of the performance with a significant increase in accuracy during the execution phase while MRCP-based accuracy peaked at movement onset. Such temporal dynamics were similar but slower in stroke patients when the movements were attempted with the affected hand (AH). Moreover, CMC outperformed the two brain features in healthy subjects and stroke patients when performing the task with their unaffected hand, whereas a higher variability across subjects was observed in patients performing the tasks with their AH. Interestingly, brain features performed better in this latter condition with respect to healthy subjects.Significance.Our results provide hints to improve the design of Brain-Computer Interfaces for post-stroke rehabilitation, emphasizing the need for personalized approaches tailored to patients' characteristics and to the intended rehabilitative target.

Hybrid Functional Near-Infrared Spectroscopy System and Electromyography for Prosthetic Knee Control

The increasing number of individuals with limb loss worldwide highlights the need for advancements in prosthetic knee technology. To improve control and quality of life, integrating brain-computer communication with motor imagery offers a promising solution. This study introduces a hybrid system that combines electromyography (EMG) and functional near-infrared spectroscopy (fNIRS) to address these limitations and enhance the control of knee movements for individuals with above-knee amputations. The study involved an experiment with nine healthy male participants, consisting of two sessions: real execution and imagined execution using motor imagery. The OpenBCI Cyton board collected EMG signals corresponding to the desired movements, while fNIRS monitored brain activity in the prefrontal and motor cortices. The analysis of the simultaneous measurement of the muscular and hemodynamic responses demonstrated that combining these data sources significantly improved the classification accuracy compared to using each dataset alone. The results showed that integrating both the EMG and fNIRS data consistently achieved a higher classification accuracy. More specifically, the Support Vector Machine performed the best during the motor imagery tasks, with an average accuracy of 49.61%, while the Linear Discriminant Analysis excelled in the real execution tasks, achieving an average accuracy of 89.67%. This research validates the feasibility of using a hybrid approach with EMG and fNIRS to enable prosthetic knee control through motor imagery, representing a significant advancement potential in prosthetic technology.

A Brain Network Analysis Model for Motion Sickness in Electric Vehicles Based on EEG and fNIRS Signal Fusion

Motion sickness is a common issue in electric vehicles, significantly impacting passenger comfort. This study aims to develop a functional brain network analysis model by integrating electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals to evaluate motion sickness symptoms. During real-world testing with the Feifan F7 series of new energy-electric vehicles from SAIC Motor Corp, data were collected from 32 participants. The EEG signals were divided into four frequency bands: delta-range, theta-range, alpha-range, and beta-range, and brain oxygenation variation was calculated from the fNIRS signals. Functional connectivity between brain regions was measured to construct functional brain network models for motion sickness analysis. A motion sickness detection model was developed using a graph convolutional network (GCN) to integrate EEG and fNIRS data. Our results show significant differences in brain functional connectivity between participants in motion and non-motion sickness states. The model that combined fNIRS data with high-frequency EEG signals achieved the best performance, improving the F1 score by 11.4% compared to using EEG data alone and by 8.2% compared to using fNIRS data alone. These results highlight the effectiveness of integrating EEG and fNIRS signals using GCN for motion sickness detection. They demonstrate the model's superiority over single-modality approaches, showcasing its potential for real-world applications in electric vehicles.

Recent progress on smart lower prosthetic limbs: a comprehensive review on using EEG and fNIRS devices in rehabilitation

The global rise in lower limb amputation cases necessitates advancements in prosthetic limb technology to enhance the quality of life for affected patients. This review paper explores recent advancements in the integration of EEG and fNIRS modalities for smart lower prosthetic limbs for rehabilitation applications. The paper synthesizes current research progress, focusing on the synergy between brain-computer interfaces and neuroimaging technologies to enhance the functionality and user experience of lower limb prosthetics. The review discusses the potential of EEG and fNIRS in decoding neural signals, enabling more intuitive and responsive control of prosthetic devices. Additionally, the paper highlights the challenges, innovations, and prospects associated with the incorporation of these neurotechnologies in the field of rehabilitation. The insights provided in this review contribute to a deeper understanding of the evolving landscape of smart lower prosthetic limbs and pave the way for more effective and user-friendly solutions in the realm of neurorehabilitation.

An Adaptive Hybrid Brain-Computer Interface for Hand Function Rehabilitation of Stroke Patients

Motor imagery (MI) based brain computer interface (BCI) has been extensively studied to improve motor recovery for stroke patients by inducing neuroplasticity. However, due to the lower spatial resolution and signal-to-noise ratio (SNR) of electroencephalograph (EEG), MI based BCI system that involves decoding hand movements within the same limb remains lower classification accuracy and poorer practicality. To overcome the limitations, an adaptive hybrid BCI system combining MI and steady-state visually evoked potential (SSVEP) is developed to improve decoding accuracy while enhancing neural engagement. On the one hand, the SSVEP evoked by visual stimuli based on action-state flickering coding approach significantly improves the recognition accuracy compared to the pure MI based BCI. On the other hand, to reduce the impact of SSVEP on MI due to the dual-task interference effect, the event-related desynchronization (ERD) based neural engagement is monitored and employed for feedback in real-time to ensure the effective execution of MI tasks. Eight healthy subjects and six post-stroke patients were recruited to verify the effectiveness of the system. The results showed that the four-class gesture recognition accuracies of healthy individuals and patients could be improved to 94.37 ± 4.77 % and 79.38 ± 6.26 %, respectively. Moreover, the designed hybrid BCI could maintain the same degree of neural engagement as observed when subjects solely performed MI tasks. These phenomena demonstrated the interactivity and clinical utility of the developed system for the rehabilitation of hand function in stroke patients.

Enhancing Word-Level Imagined Speech BCI Through Heterogeneous Transfer Learning

In this study, we proposed a novel heterogeneous transfer learning approach named Focused Speech Feature Transfer Learning (FSFTL), aimed at enhancing the performance of electroencephalogram (EEG)-based word-level Imagined Speech (IS) Brain-Computer Interface (BCI). In IS BCI, the classification accuracy for imagining specific words is relatively low due to the inherent complexity in high-level feature variations. However, the binary classification accuracy for IS/rest is significantly higher. FSFTL leverages the refined feature focusing capability of the binary IS/Rest classification task to effectively locate relevant features for the word-level task. The feature extractor in the IS/Rest model demonstrates robust decoding ability for low-level IS features in EEG signals. We applied this high-performance yet low-resolution feature extractor to a public dataset for five-word IS task. The classifier was retrained to handle an increased number of classification categories, and the feature extractor was further fine-tuned to accommodate higher-level classification tasks. Before the experiment, we aligned the data from the two datasets to maintain the versatility of the feature extractor. Our proposed FSFTL approach was compared with existing EEG models, showing a significant improvement. The FSFTL approach outperformed the backbone strategy with a 6% increase in mean accuracy across all fifteen subjects. This study highlights the commonality of features in EEG data of IS and their transferability across various datasets and tasks, which is beneficial for improving the decoding ability of word-level IS BCI.

Millán J, Sulzer J, Lewis-Peacock JA. Evaluating the Feasibility of Visual Imagery for an EEG-Based Brain-Computer Interface

Visual imagery, or the mental simulation of visual information from memory, could serve as an effective control paradigm for a brain-computer interface (BCI) due to its ability to directly convey the user's intention with many natural ways of envisioning an intended action. However, multiple initial investigations into using visual imagery as a BCI control strategies have been unable to fully evaluate the capabilities of true spontaneous visual mental imagery. One major limitation in these prior works is that the target image is typically displayed immediately preceding the imagery period. This paradigm does not capture spontaneous mental imagery as would be necessary in an actual BCI application but something more akin to short-term retention in visual working memory. Results from the present study show that short-term visual imagery following the presentation of a specific target image provides a stronger, more easily classifiable neural signature in EEG than spontaneous visual imagery from long-term memory following an auditory cue for the image. We also show that short-term visual imagery and visual perception share commonalities in the most predictive electrodes and spectral features. However, visual imagery received greater influence from frontal electrodes whereas perception was mostly confined to occipital electrodes. This suggests that visual perception is primarily driven by sensory information whereas visual imagery has greater contributions from areas associated with memory and attention. This work provides the first direct comparison of short-term and long-term visual imagery tasks and provides greater insight into the feasibility of using visual imagery as a BCI control strategy.

Low frequency oscillations reflect neurovascular coupling and disappear after cerebral death

Spectrum power analysis in the low frequency oscillations (LFO) region of functional near infrared spectroscopy (fNIRS) is a promising method to deliver information about brain activation and therefore might be used for prognostication in patients with disorders of consciousness in the neurocritical care unit alongside with established methods. In this study, we measure the cortical hemodynamic response measured by fNIRS in the LFO region following auditory and somatosensory stimulation in healthy subjects. The significant hemodynamic reaction in the contralateral hemisphere correlation with the physiologic electric response suggests neurovascular coupling. In addition, we investigate power spectrum changes in steady state measurements of cerebral death patients and healthy subjects in the LFO region, the frequency of the heartbeat and respiration. The spectral power within the LFO region was lower in the patients with cerebral death compared to the healthy subjects, whereas there were no differences in spectral power for physiological activities such as heartbeat and respiration rate. This finding indicates the cerebral origin of our low frequency measurements. Therefore, LFO measurements are a potential method to detect brain activation in patients with disorders of consciousness and cerebral death. However, further studies in patients are needed to investigate its potential clinical use.

A novel brain-controlled prosthetic hand method integrating AR-SSVEP augmentation, asynchronous control, and machine vision assistance

Background and objective

The brain-computer interface (BCI) system based on steady-state visual evoked potentials (SSVEP) is expected to help disabled patients achieve alternative prosthetic hand assistance. However, the existing study still has some shortcomings in interaction aspects such as stimulus paradigm and control logic. The purpose of this study is to innovate the visual stimulus paradigm and asynchronous decoding/control strategy by integrating augmented reality technology, and propose an asynchronous pattern recognition algorithm, thereby improving the interaction logic and practical application capabilities of the prosthetic hand with the BCI system.

Methods

An asynchronous visual stimulus paradigm based on an augmented reality (AR) interface was proposed in this paper, in which there were 8 control modes, including Grasp, Put down, Pinch, Point, Fist, Palm push, Hold pen, and Initial. According to the attentional orienting characteristics of the paradigm, a novel asynchronous pattern recognition algorithm that combines center extended canonical correlation analysis and support vector machine (Center-ECCA-SVM) was proposed. Then, this study proposed an intelligent BCI system switch based on a deep learning object detection algorithm (YOLOv4) to improve the level of user interaction. Finally, two experiments were designed to test the performance of the brain-controlled prosthetic hand system and its practical performance in real scenarios.

Results

Under the AR paradigm of this study, compared with the liquid crystal display (LCD) paradigm, the average SSVEP spectrum amplitude of multiple subjects increased by 17.41%, and the signal-noise ratio (SNR) increased by 3.52%. The average stimulus pattern recognition accuracy was 96.71 ± 3.91%, which was 2.62% higher than the LCD paradigm. Under the data analysis time of 2s, the Center-ECCA-SVM classifier obtained 94.66 ± 3.87% and 97.40 ± 2.78% asynchronous pattern recognition accuracy under the Normal metric and the Tolerant metric, respectively. And the YOLOv4-tiny model achieves a speed of 25.29fps and a 96.4% confidence in the prosthetic hand in real-time detection. Finally, the brain-controlled prosthetic hand helped the subjects to complete 4 kinds of daily life tasks in the real scene, and the time-consuming were all within an acceptable range, which verified the effectiveness and practicability of the system.

Conclusion

This research is based on improving the user interaction level of the prosthetic hand with the BCI system, and has made improvements in the SSVEP paradigm, asynchronous pattern recognition, interaction, and control logic. Furthermore, it also provides support for BCI areas for alternative prosthetic control, and movement disorder rehabilitation programs.

A scoping review of gaze and eye tracking-based control methods for assistive robotic arms

Background: Assistive Robotic Arms are designed to assist physically disabled people with daily activities. Existing joysticks and head controls are not applicable for severely disabled people such as people with Locked-in Syndrome. Therefore, eye tracking control is part of ongoing research. The related literature spans many disciplines, creating a heterogeneous field that makes it difficult to gain an overview. Objectives: This work focuses on ARAs that are controlled by gaze and eye movements. By answering the research questions, this paper provides details on the design of the systems, a comparison of input modalities, methods for measuring the performance of these controls, and an outlook on research areas that gained interest in recent years. Methods: This review was conducted as outlined in the PRISMA 2020 Statement. After identifying a wide range of approaches in use the authors decided to use the PRISMA-ScR extension for a scoping review to present the results. The identification process was carried out by screening three databases. After the screening process, a snowball search was conducted. Results: 39 articles and 6 reviews were included in this article. Characteristics related to the system and study design were extracted and presented divided into three groups based on the use of eye tracking. Conclusion: This paper aims to provide an overview for researchers new to the field by offering insight into eye tracking based robot controllers. We have identified open questions that need to be answered in order to provide people with severe motor function loss with systems that are highly useable and accessible.

Unleashing the potential of fNIRS with machine learning: classification of fine anatomical movements to empower future brain-computer interface

In this study, we explore the potential of using functional near-infrared spectroscopy (fNIRS) signals in conjunction with modern machine-learning techniques to classify specific anatomical movements to increase the number of control commands for a possible fNIRS-based brain-computer interface (BCI) applications. The study focuses on novel individual finger-tapping, a well-known task in fNIRS and fMRI studies, but limited to left/right or few fingers. Twenty-four right-handed participants performed the individual finger-tapping task. Data were recorded by using sixteen sources and detectors placed over the motor cortex according to the 10-10 international system. The event's average oxygenated Δ HbO and deoxygenated Δ HbR hemoglobin data were utilized as features to assess the performance of diverse machine learning (ML) models in a challenging multi-class classification setting. These methods include LDA, QDA, MNLR, XGBoost, and RF. A new DL-based model named "Hemo-Net" has been proposed which consists of multiple parallel convolution layers with different filters to extract the features. This paper aims to explore the efficacy of using fNRIS along with ML/DL methods in a multi-class classification task. Complex models like RF, XGBoost, and Hemo-Net produce relatively higher test set accuracy when compared to LDA, MNLR, and QDA. Hemo-Net has depicted a superior performance achieving the highest test set accuracy of 76%, however, in this work, we do not aim at improving the accuracies of models rather we are interested in exploring if fNIRS has the neural signatures to help modern ML/DL methods in multi-class classification which can lead to applications like brain-computer interfaces. Multi-class classification of fine anatomical movements, such as individual finger movements, is difficult to classify with fNIRS data. Traditional ML models like MNLR and LDA show inferior performance compared to the ensemble-based methods of RF and XGBoost. DL-based method Hemo-Net outperforms all methods evaluated in this study and demonstrates a promising future for fNIRS-based BCI applications.

Multimodal pre-screening can predict BCI performance variability: A novel subject-specific experimental scheme

BACKGROUND

Brain-computer interface (BCI) systems currently lack the required robustness for long-term daily use due to inter- and intra-subject performance variability. In this study, we propose a novel personalized scheme for a multimodal BCI system, primarily using functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), to identify, predict, and compensate for factors affecting competence-related and interfering factors associated with performance.

METHOD

11 (out of 13 recruited) participants, including five participants with motor deficits, completed four sessions on average. During the training sessions, the subjects performed a short pre-screening phase, followed by three variations of a novel visou-mental (VM) protocol. Features extracted from the pre-screening phase were used to construct predictive platforms using stepwise multivariate linear regression (MLR) models. In the test sessions, we employed a task-correction phase where our predictive models were used to predict the ideal task variation to maximize performance, followed by an interference-correction phase. We then investigated the associations between predicted and actual performances and evaluated the outcome of correction strategies.

RESULT

The predictive models resulted in respective adjusted R-squared values of 0.942, 0.724, and 0.939 for the first, second, and third variation of the task, respectively. The statistical analyses showed significant associations between the performances predicted by predictive models and the actual performances for the first two task variations, with rhos of 0.7289 (p-value = 0.011) and 0.6970 (p-value = 0.017), respectively. For 81.82 % of the subjects, the task/workload correction stage correctly determined which task variation provided the highest accuracy, with an average performance gain of 5.18 % when applying the correction strategies.

CONCLUSION

Our proposed method can lead to an integrated multimodal predictive framework to compensate for BCI performance variability, particularly, for people with severe motor deficits.

The clinical effects of brain-computer interface with robot on upper-limb function for post-stroke rehabilitation: a meta-analysis and systematic review

PURPOSE

Many recent clinical studies have suggested that the combination of brain-computer interfaces (BCIs) can induce neurological recovery and improvement in motor function. In this review, we performed a systematic review and meta-analysis to evaluate the clinical effects of BCI-robot systems.

METHODS

The articles published from January 2010 to December 2020 have been searched by using the databases (EMBASE, PubMed, CINAHL, EBSCO, Web of Science and manual search). The single-group studies were qualitatively described, and only the controlled-trial studies were included for the meta-analysis. The mean difference (MD) of Fugl-Meyer Assessment (FMA) scores were pooled and the random-effects model method was used to perform the meta-analysis. The PRISMA criteria were followed in current review.

RESULTS

A total of 897 records were identified, eight single-group studies and 11 controlled-trial studies were included in our review. The systematic analysis indicated that the BCI-robot systems had a significant improvement on motor function recovery. The meta-analysis showed there were no statistic differences between BCI-robot groups and robot groups, neither in the immediate effects nor long-term effects (p > 0.05).

CONCLUSION

The use of BCI-robot systems has significant improvement on the motor function recovery of hemiparetic upper-limb, and there is a sustaining effect. The meta-analysis showed no statistical difference between the experimental group (BCI-robot) and the control group (robot). However, there are a few shortcomings in the experimental design of existing studies, more clinical trials need to be conducted, and the experimental design needs to be more rigorous.Implications for RehabilitationIn this review, we evaluated the clinical effects of brain-computer interface with robot on upper-limb function for post-stroke rehabilitation. After we screened the databases, 19 articles were included in this review. These articles all clinical trial research, they all used non-invasive brain-computer interfaces and upper-limb robot.We conducted the systematic review with nine articles, the result indicated that the BCI-robot system had a significant improvement on motor function recovery. Eleven articles were included for the meta-analysis, the result showed there were no statistic differences between BCI-robot groups and robot groups, neither in the immediate effects nor long-term effects.We thought the result of meta-analysis which showed no statistic difference was probably caused by the heterogenicity of clinical trial designs of these articles.We thought the BCI-robot systems are promising strategies for post-stroke rehabilitation. And we gave several suggestions for further research: (1) The experimental design should be more rigorous, and describe the experimental designs in detail, especially the control group intervention, to make the experiment replicability. (2) New evaluation criteria need to be established, more objective assessment such as biomechanical assessment, fMRI should be utilised as the primary outcome. (3) More clinical studies with larger sample size, novel external devices, and BCI systems need to be conducted to investigate the differences between BCI-robot system and other interventions. (4) Further research could shift the focus to the patients who are in subacute stage, to explore if the early BCI training can make a positive impact on cerebral cortical recovery.

A Novel Hybrid Brain-Computer Interface Combining the Illusion-Induced VEP and SSVEP

Traditional single-modality brain-computer interface (BCI) systems are limited by their reliance on a single characteristic of brain signals. To address this issue, incorporating multiple features from EEG signals can provide robust information to enhance BCI performance. In this study, we designed and implemented a novel hybrid paradigm that combined illusion-induced visual evoked potential (IVEP) and steady-state visual evoked potential (SSVEP) with the aim of leveraging their features simultaneously to improve system efficiency. The proposed paradigm was validated through two experimental studies, which encompassed feature analysis of IVEP with a static paradigm, and performance evaluation of hybrid paradigm in comparison with the conventional SSVEP paradigm. The characteristic analysis yielded significant differences in response waveforms among different motion illusions. The performance evaluation of the hybrid BCI demonstrates the advantage of integrating illusory stimuli into the SSVEP paradigm. This integration effectively enhanced the spatio-temporal features of EEG signals, resulting in higher classification accuracy and information transfer rate (ITR) within a short time window when compared to traditional SSVEP-BCI in four-command task. Furthermore, the questionnaire results of subjective estimation revealed that proposed hybrid BCI offers less eye fatigue, and potentially higher levels of concentration, physical condition, and mental condition for users. This work first introduced the IVEP signals in hybrid BCI system that could enhance performance efficiently, which is promising to fulfill the requirements for efficiency in practical BCI control systems.

fNIRS-EEG BCIs for Motor Rehabilitation: A Review

Motor impairment has a profound impact on a significant number of individuals, leading to a substantial demand for rehabilitation services. Through brain-computer interfaces (BCIs), people with severe motor disabilities could have improved communication with others and control appropriately designed robotic prosthetics, so as to (at least partially) restore their motor abilities. BCI plays a pivotal role in promoting smoother communication and interactions between individuals with motor impairments and others. Moreover, they enable the direct control of assistive devices through brain signals. In particular, their most significant potential lies in the realm of motor rehabilitation, where BCIs can offer real-time feedback to assist users in their training and continuously monitor the brain's state throughout the entire rehabilitation process. Hybridization of different brain-sensing modalities, especially functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), has shown great potential in the creation of BCIs for rehabilitating the motor-impaired populations. EEG, as a well-established methodology, can be combined with fNIRS to compensate for the inherent disadvantages and achieve higher temporal and spatial resolution. This paper reviews the recent works in hybrid fNIRS-EEG BCIs for motor rehabilitation, emphasizing the methodologies that utilized motor imagery. An overview of the BCI system and its key components was introduced, followed by an introduction to various devices, strengths and weaknesses of different signal processing techniques, and applications in neuroscience and clinical contexts. The review concludes by discussing the possible challenges and opportunities for future development.

A High-Rate Hybrid BCI System Based on High-Frequency SSVEP and sEMG

Recently, various biosignals have been combined with electroencephalography (EEG) to build hybrid brain-computer interface (BCI) systems to improve system performance. Since steady-state visual evoked potential (SSVEP) and surface electromyography (sEMG) are easy-to-use, non-invasive techniques, and have high signal-to-noise ratio (SNR), hybrid BCI systems combining SSVEP and sEMG have received much attention in the BCI literature. However, most existing studies regarding hybrid BCIs based on SSVEP and sEMG adopt low-frequency visual stimuli to induce SSVEPs. The comfort of these systems needs further improvement to meet the practical application requirements. The present study realized a novel hybrid BCI combining high-frequency SSVEP and sEMG signals for spelling applications. EEG and sEMG were obtained simultaneously from the scalp and skin surface of subjects, respectively. These two types of signals were analyzed independently and then combined to determine the target stimulus. Our online results demonstrated that the developed hybrid BCI yielded a mean accuracy of 88.07 ± 1.43% and ITR of 159.12 ± 4.31 bits/min. These results exhibited the feasibility and effectiveness of fusing high-frequency SSVEP and sEMG towards improving the total BCI system performance.

A global bibliometric and visualized analysis of gait analysis and artificial intelligence research from 1992 to 2022

Gait is an important basic function of human beings and an integral part of life. Many mental and physical abnormalities can cause noticeable differences in a person's gait. Abnormal gait can lead to serious consequences such as falls, limited mobility and reduced life satisfaction. Gait analysis, which includes joint kinematics, kinetics, and dynamic Electromyography (EMG) data, is now recognized as a clinically useful tool that can provide both quantifiable and qualitative information on performance to aid in treatment planning and evaluate its outcome. With the assistance of new artificial intelligence (AI) technology, the traditional medical environment has undergone great changes. AI has the potential to reshape medicine, making gait analysis more accurate, efficient and accessible. In this study, we analyzed basic information about gait analysis and AI articles that met inclusion criteria in the WoS Core Collection database from 1992-2022, and the VosViewer software was used for web visualization and keyword analysis. Through bibliometric and visual analysis, this article systematically introduces the research status of gait analysis and AI. We introduce the application of artificial intelligence in clinical gait analysis, which affects the identification and management of gait abnormalities found in various diseases. Machine learning (ML) and artificial neural networks (ANNs) are the most often utilized AI methods in gait analysis. By comparing the predictive capability of different AI algorithms in published studies, we evaluate their potential for gait analysis in different situations. Furthermore, the current challenges and future directions of gait analysis and AI research are discussed, which will also provide valuable reference information for investors in this field.

Spectrum-Enhanced TRCA (SE-TRCA): A novel approach for direction detection in SSVEP-based BCI

The Steady State Visual Evoked Potential (SSVEP) is a widely used component in BCIs due to its high noise resistance and low equipment requirements. Recently, a novel SSVEP-based paradigm has been introduced for direction detection, in which, unlike the common SSVEP paradigms that use several frequency stimuli, only one flickering stimulus is used and it makes direction detection very challenging. So far, only the CCA method has been used for direction detection using SSVEP component analysis. Since Canonical Correlation Analysis (CCA) has some limitations, a Task-Related Component Analysis (TRCA) based method has been introduced for feature extraction to improve the direction detection performance. Although these methods have been proven efficient, they do not utilize the latent frequency information in the EEG signal. Therefore, the performance of direction detection using SSVEP component analysis is still suboptimal. For further improvement, the TRCA-based algorithm is enhanced by incorporating frequency information and introducing Spectrum-Enhanced TRCA (SE-TRCA). SE-TRCA method can utilize frequency information in conjunction with spatial information by concatenating the EEG signal and its shifted version. Accordingly, the obtained spatio-spectral filters perform as a Finite Impulse Response (FIR) filter. To evaluate the proposed SE-TRCA method, two different sorts of datasets (1) a hybrid BCI dataset (including SSVEP component for direction detection) and (2) a pure benchmark SSVEP dataset (including SSVEP component for frequency detection) have been used. Our experiments showed that the accuracy of direction detection using the proposed SE-TRCA and TRCA approaches compared to CCA-based approach have been increased by 23.35% and 28.24%, respectively. Furthermore, the accuracy of character recognition obtained from integrating P300 and SSVEP components in CCA, TRCA, and SETRCA approaches are 54.01%, 56.02%, and 58.56%, on the hybrid dataset, respectively. The evaluation of the SE-TRCA method on the benchmark SSVEP dataset demonstrates that the SE-TRCA method outperforms both CCA and TRCA, particularly regarding frequency detection accuracy. In this specific dataset, the SE-TRCA method achieved an impressive frequency detection accuracy of 98.19% for a 3-s signal, surpassing the accuracies of TRCA and CCA, which were 97.91% and 90.47%, respectively. These results demonstrated that the TRCA-based approach is more efficient than the CCA approach to extracting spatial filters. Moreover, SE-TRCA, extracting both Spectral and spatial information from the EEG signal, can capture more discriminative features from the SSVEP component and increase the accuracy of classification. The results of this study emphasize the effectiveness of the proposed SE-TRCA approach across different SSVEP paradigms and tasks. These findings provide strong evidence for the method's ability to generalize well in SSVEP analysis.

Coupling Effects of Cross-Corticomuscular Association during Object Manipulation Tasks on Different Haptic Sensations

The effects of corticomuscular connectivity during object manipulation tasks with different haptic sensations have not been quantitatively investigated. Connectivity analyses enable the study of cortical effects and muscle responses during movements, revealing communication pathways between the brain and muscles. This study aims to examine the corticomuscular connectivity of three Electroencephalography (EEG) channels and five muscles during object manipulation tasks involving contact surfaces of Sandpaper, Suede, and Silk. The analyses included 12 healthy subjects performing tasks with their right hand. Power-Based Connectivity (PBC) and Mutual Information (MI) measures were utilized to evaluate significant differences in connectivity between contact surfaces, EEG channels, muscles, and frequency bands. The research yielded the following findings: Suede contact surface exhibited higher connectivity; Mu and Gamma frequency bands exerted greater influence; significant connectivity was observed between the three EEG channels (C 3 ,C z ,C 4 ) and the Anterior Deltoid (AD) and Brachioradialis (B) muscles; and connectivity was primarily involved during active movement in the AD muscle compared to the resting state. These findings suggest potential implementation in motor rehabilitation for more complex movements using novel alternative training systems with high effectiveness.

The characteristics of brain network in patient with post-stroke depression under cognitive task condition

Objectives

Post-stroke depression (PSD) may be associated with the altered brain network property. This study aimed at exploring the brain network characteristics of PSD under the classic cognitive task, i.e., the oddball task, in order to promote our understanding of the pathogenesis and the diagnosis of PSD.

Methods

Nineteen stroke survivors with PSD and 18 stroke survivors with no PSD (non-PSD) were recruited. The functional near-infrared spectroscopy (fNIRS) covering the dorsolateral prefrontal cortex was recorded during the oddball task state and the resting state. The brain network characteristics were extracted using the graph theory and compared between the PSD and the non-PSD subjects. In addition, the classification performance between the PSD and non-PSD subjects was evaluated using features in the resting and the task state, respectively.

Results

Compared with the resting state, more brain network characteristics in the task state showed significant differences between the PSD and non-PSD groups, resulting in better classification performance. In the task state, the assortativity, clustering coefficient, characteristic path length, and local efficiency of the PSD subjects was larger compared with the non-PSD subjects while the global efficiency of the PSD subjects was smaller than that of the non-PSD subjects.

Conclusion

The altered brain network properties associated with PSD in the cognitive task state were more distinct compared with the resting state, and the ability of the brain network to resist attack and transmit information was reduced in PSD patients in the task state.

Significance

This study demonstrated the feasibility and superiority of investigating brain network properties in the task state for the exploration of the pathogenesis and new diagnosis methods for PSD.

Multi-Class Classification of Upper Limb Movements With Filter Bank Task-Related Component Analysis

The classification of limb movements can provide with control commands in non-invasive brain-computer interface. Previous studies on the classification of limb movements have focused on the classification of left/right limbs; however, the classification of different types of upper limb movements has often been ignored despite that it provides more active-evoked control commands in the brain-computer interface. Nevertheless, few machine learning method can be used as the state-of-the-art method in the multi-class classification of limb movements. This work focuses on the multi-class classification of upper limb movements and proposes the multi-class filter bank task-related component analysis (mFBTRCA) method, which consists of three steps: spatial filtering, similarity measuring and filter bank selection. The spatial filter, namely the task-related component analysis, is first used to remove noise from EEG signals. The canonical correlation measures the similarity of the spatial-filtered signals and is used for feature extraction. The correlation features are extracted from multiple low-frequency filter banks. The minimum-redundancy maximum-relevance selects the essential features from all the correlation features, and finally, the support vector machine is used to classify the selected features. The proposed method compared against previously used models is evaluated using two datasets. mFBTRCA achieved a classification accuracy of 0.4193 ± 0.0780 (7 classes) and 0.4032 ± 0.0714 (5 classes), respectively, which improves on the best accuracies achieved using the compared methods (0.3590 ± 0.0645 and 0.3159 ± 0.0736, respectively). The proposed method is expected to provide more control commands in the applications of non-invasive brain-computer interfaces.

Driving drowsiness detection using spectral signatures of EEG-based neurophysiology

Introduction: Drowsy driving is a significant factor causing dire road crashes and casualties around the world. Detecting it earlier and more effectively can significantly reduce the lethal aftereffects and increase road safety. As physiological conditions originate from the human brain, so neurophysiological signatures in drowsy and alert states may be investigated for this purpose. In this preface, A passive brain-computer interface (pBCI) scheme using multichannel electroencephalography (EEG) brain signals is developed for spatially localized and accurate detection of human drowsiness during driving tasks.

Methods: This pBCI modality acquired electrophysiological patterns of 12 healthy subjects from the prefrontal (PFC), frontal (FC), and occipital cortices (OC) of the brain. Neurological states are recorded using six EEG channels spread over the right and left hemispheres in the PFC, FC, and OC of the sleep-deprived subjects during simulated driving tasks. In post-hoc analysis, spectral signatures of the δ, θ, α, and β rhythms are extracted in terms of spectral band powers and their ratios with a temporal correlation over the complete span of the experiment. Minimum redundancy maximum relevance, Chi-square, and ReliefF feature selection methods are used and aggregated with a Z-score based approach for global feature ranking. The extracted drowsiness attributes are classified using decision trees, discriminant analysis, logistic regression, naïve Bayes, support vector machines, k-nearest neighbors, and ensemble classifiers. The binary classification results are reported with confusion matrix-based performance assessment metrics.

Results: In inter-classifier comparison, the optimized ensemble model achieved the best results of drowsiness classification with 85.6% accuracy and precision, 89.7% recall, 87.6% F1-score, 80% specificity, 70.3% Matthews correlation coefficient, 70.2% Cohen’s kappa score, and 91% area under the receiver operating characteristic curve with 76-ms execution time. In inter-channel comparison, the best results were obtained at the F8 electrode position in the right FC of the brain. The significance of all the results was validated with a p-value of less than 0.05 using statistical hypothesis testing methods.

Conclusions: The proposed scheme has achieved better results for driving drowsiness detection with the accomplishment of multiple objectives. The predictor importance approach has reduced the feature extraction cost and computational complexity is minimized with the use of conventional machine learning classifiers resulting in low-cost hardware and software requirements. The channel selection approach has spatially localized the most promising brain region for drowsiness detection with only a single EEG channel (F8) which reduces the physical intrusiveness in normal driving operation. This pBCI scheme has a good potential for practical applications requiring earlier, more accurate, and less disruptive drowsiness detection using the spectral information of EEG biosignals.

Functional Near-Infrared Spectroscopy as a Personalized Digital Healthcare Tool for Brain Monitoring

The sustained growth of digital healthcare in the field of neurology relies on portable and cost-effective brain monitoring tools that can accurately monitor brain function in real time. Functional near-infrared spectroscopy (fNIRS) is one such tool that has become popular among researchers and clinicians as a practical alternative to functional magnetic resonance imaging, and as a complementary tool to modalities such as electroencephalography. This review covers the contribution of fNIRS to the personalized goals of digital healthcare in neurology by identifying two major trends that drive current fNIRS research. The first major trend is multimodal monitoring using fNIRS, which allows clinicians to access more data that will help them to understand the interconnection between the cerebral hemodynamics and other physiological phenomena in patients. This allows clinicians to make an overall assessment of physical health to obtain a more-detailed and individualized diagnosis. The second major trend is that fNIRS research is being conducted with naturalistic experimental paradigms that involve multisensory stimulation in familiar settings. Cerebral monitoring of multisensory stimulation during dynamic activities or within virtual reality helps to understand the complex brain activities that occur in everyday life. Finally, the scope of future fNIRS studies is discussed to facilitate more-accurate assessments of brain activation and the wider clinical acceptance of fNIRS as a medical device for digital healthcare.

Exploring high-density corticomuscular networks after stroke to enable a hybrid Brain-Computer Interface for hand motor rehabilitation

BACKGROUND

Brain-Computer Interfaces (BCI) promote upper limb recovery in stroke patients reinforcing motor related brain activity (from electroencephalogaphy, EEG). Hybrid BCIs which include peripheral signals (electromyography, EMG) as control features could be employed to monitor post-stroke motor abnormalities. To ground the use of corticomuscular coherence (CMC) as a hybrid feature for a rehabilitative BCI, we analyzed high-density CMC networks (derived from multiple EEG and EMG channels) and their relation with upper limb motor deficit by comparing data from stroke patients with healthy participants during simple hand tasks.

METHODS

EEG (61 sensors) and EMG (8 muscles per arm) were simultaneously recorded from 12 stroke (EXP) and 12 healthy participants (CTRL) during simple hand movements performed with right/left (CTRL) and unaffected/affected hand (EXP, UH/AH). CMC networks were estimated for each movement and their properties were analyzed by means of indices derived ad-hoc from graph theory and compared among groups.

RESULTS

Between-group analysis showed that CMC weight of the whole brain network was significantly reduced in patients during AH movements. The network density was increased especially for those connections entailing bilateral non-target muscles. Such reduced muscle-specificity observed in patients was confirmed by muscle degree index (connections per muscle) which indicated a connections' distribution among non-target and contralateral muscles and revealed a higher involvement of proximal muscles in patients. CMC network properties correlated with upper-limb motor impairment as assessed by Fugl-Meyer Assessment and Manual Muscle Test in patients.

CONCLUSIONS

High-density CMC networks can capture motor abnormalities in stroke patients during simple hand movements. Correlations with upper limb motor impairment support their use in a BCI-based rehabilitative approach.

Identifying Thematics in a Brain-Computer Interface Research

This umbrella review is motivated to understand the shift in research themes on brain-computer interfacing (BCI) and it determined that a shift away from themes that focus on medical advancement and system development to applications that included education, marketing, gaming, safety, and security has occurred. The background of this review examined aspects of BCI categorisation, neuroimaging methods, brain control signal classification, applications, and ethics. The specific area of BCI software and hardware development was not examined. A search using One Search was undertaken and 92 BCI reviews were selected for inclusion. Publication demographics indicate the average number of authors on review papers considered was 4.2 ± 1.8. The results also indicate a rapid increase in the number of BCI reviews from 2003, with only three reviews before that period, two in 1972, and one in 1996. While BCI authors were predominantly Euro-American in early reviews, this shifted to a more global authorship, which China dominated by 2020-2022. The review revealed six disciplines associated with BCI systems: life sciences and biomedicine (n = 42), neurosciences and neurology (n = 35), and rehabilitation (n = 20); (2) the second domain centred on the theme of functionality: computer science (n = 20), engineering (n = 28) and technology (n = 38). There was a thematic shift from understanding brain function and modes of interfacing BCI systems to more applied research novel areas of research-identified surround artificial intelligence, including machine learning, pre-processing, and deep learning. As BCI systems become more invasive in the lives of "normal" individuals, it is expected that there will be a refocus and thematic shift towards increased research into ethical issues and the need for legal oversight in BCI application.

Investigating User Proficiency of Motor Imagery for EEG-Based BCI System to Control Simulated Wheelchair

The research on the electroencephalography (EEG)-based brain-computer interface (BCI) is widely utilized for wheelchair control. The ability of the user is one factor of BCI efficiency. Therefore, we focused on BCI tasks and protocols to yield high efficiency from the robust EEG features of individual users. This study proposes a task-based brain activity to gain the power of the alpha band, which included eyes closed for alpha response at the occipital area, attention to an upward arrow for alpha response at the frontal area, and an imagined left/right motor for alpha event-related desynchronization at the left/right motor cortex. An EPOC X neuroheadset was used to acquire the EEG signals. We also proposed user proficiency in motor imagery sessions with limb movement paradigms by recommending motor imagination tasks. Using the proposed system, we verified the feature extraction algorithms and command translation. Twelve volunteers participated in the experiment, and the conventional paradigm of motor imagery was used to compare the efficiencies. With utilized user proficiency in motor imagery, an average accuracy of 83.7% across the left and right commands was achieved. The recommended MI paradigm via user proficiency achieved an approximately 4% higher accuracy than the conventional MI paradigm. Moreover, the real-time control results of a simulated wheelchair revealed a high efficiency based on the time condition. The time results for the same task as the joystick-based control were still approximately three times longer. We suggest that user proficiency be used to recommend an individual MI paradigm for beginners. Furthermore, the proposed BCI system can be used for electric wheelchair control by people with severe disabilities.

fNIRS-based brain functional response to robot-assisted training for upper-limb in stroke patients with hemiplegia

Background

Robot-assisted therapy (RAT) has received considerable attention in stroke motor rehabilitation. Characteristics of brain functional response associated with RAT would provide a theoretical basis for choosing the appropriate protocol for a patient. However, the cortical response induced by RAT remains to be fully elucidated due to the lack of dynamic brain functional assessment tools.

Objective

To guide the implementation of clinical therapy, this study focused on the brain functional responses induced by RAT in patients with different degrees of motor impairment.

Methods

A total of 32 stroke patients were classified into a low score group (severe impairment, n = 16) and a high score group (moderate impairment, n = 16) according to the motor function of the upper limb and then underwent RAT training in assistive mode with simultaneous cerebral haemodynamic measurement by functional near-infrared spectroscopy (fNIRS). Functional connectivity (FC) and the hemisphere autonomy index (HAI) were calculated based on the wavelet phase coherence among fNIRS signals covering bilateral prefrontal, motor and occipital areas.

Results

Specific cortical network response related to RAT was observed in patients with unilateral moderate-to-severe motor deficits in the subacute stage. Compared with patients with moderate dysfunction, patients with severe impairment showed a wide range of significant FC responses in the bilateral hemispheres induced by RAT with the assistive mode, especially task-related involvement of ipsilesional supplementary motor areas.

Conclusion

Under assisted mode, RAT-related extensive cortical response in patients with severe dysfunction might contribute to brain functional organization during motor performance, which is considered the basic neural substrate of motor-related processes. In contrast, the limited cortical response related to RAT in patients with moderate dysfunction may indicate that the training intensity needs to be adjusted in time according to the brain functional state. fNIRS-based assessment of brain functional response assumes great importance for the customization of an appropriate protocol training in the clinical practice.

A Review of Brain Activity and EEG-Based Brain-Computer Interfaces for Rehabilitation Application

Patients with severe CNS injuries struggle primarily with their sensorimotor function and communication with the outside world. There is an urgent need for advanced neural rehabilitation and intelligent interaction technology to provide help for patients with nerve injuries. Recent studies have established the brain-computer interface (BCI) in order to provide patients with appropriate interaction methods or more intelligent rehabilitation training. This paper reviews the most recent research on brain-computer-interface-based non-invasive rehabilitation systems. Various endogenous and exogenous methods, advantages, limitations, and challenges are discussed and proposed. In addition, the paper discusses the communication between the various brain-computer interface modes used between severely paralyzed and locked patients and the surrounding environment, particularly the brain-computer interaction system utilizing exogenous (induced) EEG signals (such as P300 and SSVEP). This discussion reveals with an examination of the interface for collecting EEG signals, EEG components, and signal postprocessing. Furthermore, the paper describes the development of natural interaction strategies, with a focus on signal acquisition, data processing, pattern recognition algorithms, and control techniques.

EEG hybrid brain-computer interfaces: A scoping review applying an existing hybrid-BCI taxonomy and considerations for pediatric applications

Most hybrid brain-computer interfaces (hBCI) aim at improving the performance of single-input BCI. Many combinations are possible to configure an hBCI, such as using multiple brain input signals, different stimuli or more than one input system. Multiple studies have been done since 2010 where such interfaces have been tested and analyzed. Results and conclusions are promising but little has been discussed as to what is the best approach for the pediatric population, should they use hBCI as an assistive technology. Children might face greater challenges when using BCI and might benefit from less complex interfaces. Hence, in this scoping review we included 42 papers that developed hBCI systems for the purpose of control of assistive devices or communication software, and we analyzed them through the lenses of potential use in clinical settings and for children. We extracted taxonomic categories proposed in previous studies to describe the types of interfaces that have been developed. We also proposed interface characteristics that could be observed in different hBCI, such as type of target, number of targets and number of steps before selection. Then, we discussed how each of the extracted characteristics could influence the overall complexity of the system and what might be the best options for applications for children. Effectiveness and efficiency were also collected and included in the analysis. We concluded that the least complex hBCI interfaces might involve having a brain inputs and an external input, with a sequential role of operation, and visual stimuli. Those interfaces might also use a minimal number of targets of the strobic type, with one or two steps before the final selection. We hope this review can be used as a guideline for future hBCI developments and as an incentive to the design of interfaces that can also serve children who have motor impairments.

EEG-EMG coupling as a hybrid method for steering detection in car driving settings

Understanding mental processes in complex human behavior is a key issue in driving, representing a milestone for developing user-centered assistive driving devices. Here, we propose a hybrid method based on electroencephalographic (EEG) and electromyographic (EMG) signatures to distinguish left and right steering in driving scenarios. Twenty-four participants took part in the experiment consisting of recordings of 128-channel EEG and EMG activity from deltoids and forearm extensors in non-ecological and ecological steering tasks. Specifically, we identified the EEG mu rhythm modulation correlates with motor preparation of self-paced steering actions in the non-ecological task, while the concurrent EMG activity of the left (right) deltoids correlates with right (left) steering. Consequently, we exploited the mu rhythm de-synchronization resulting from the non-ecological task to detect the steering side using cross-correlation analysis with the ecological EMG signals. Results returned significant cross-correlation values showing the coupling between the non-ecological EEG feature and the muscular activity collected in ecological driving conditions. Moreover, such cross-correlation patterns discriminate the steering side earlier relative to the single EMG signal. This hybrid system overcomes the limitation of the EEG signals collected in ecological settings such as low reliability, accuracy, and adaptability, thus adding to the EMG the characteristic predictive power of the cerebral data. These results prove how it is possible to complement different physiological signals to control the level of assistance needed by the driver.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-021-09776-w.

EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

High-Frequency Vibrating Stimuli Using the Low-Cost Coin-Type Motors for SSSEP-Based BCI

Steady-state somatosensory-evoked potential- (SSSEP-) based brain-computer interfaces (BCIs) have been applied for assisting people with physical disabilities since it does not require gaze fixation or long-time training. Despite the advancement of various noninvasive electroencephalogram- (EEG-) based BCI paradigms, researches on SSSEP with the various frequency range and related classification algorithms are relatively unsettled. In this study, we investigated the feasibility of classifying the SSSEP within high-frequency vibration stimuli induced by a versatile coin-type eccentric rotating mass (ERM) motor. Seven healthy subjects performed selective attention (SA) tasks with vibration stimuli attached to the left and right index fingers. Three EEG feature extraction methods, followed by a support vector machine (SVM) classifier, have been tested: common spatial pattern (CSP), filter-bank CSP (FBCSP), and mutual information-based best individual feature (MIBIF) selection after the FBCSP. Consequently, the FBCSP showed the highest performance at $71.5\pm 2.5$ % for classifying the left and right-hand SA tasks than the other two methods (i.e., CSP and FBCSP-MIBIF). Based on our findings and approach, the high-frequency vibration stimuli using low-cost coin motors with the FBCSP-based feature selection can be potentially applied to developing practical SSSEP-based BCI systems.

Bimodal EEG-fNIRS in Neuroergonomics. Current Evidence and Prospects for Future Research

Neuroergonomics focuses on the brain signatures and associated mental states underlying behavior to design human-machine interfaces enhancing performance in the cognitive and physical domains. Brain imaging techniques such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) have been considered key methods for achieving this goal. Recent research stresses the value of combining EEG and fNIRS in improving these interface systems' mental state decoding abilities, but little is known about whether these improvements generalize over different paradigms and methodologies, nor about the potentialities for using these systems in the real world. We review 33 studies comparing mental state decoding accuracy between bimodal EEG-fNIRS and unimodal EEG and fNIRS in several subdomains of neuroergonomics. In light of these studies, we also consider the challenges of exploiting wearable versions of these systems in real-world contexts. Overall the studies reviewed suggest that bimodal EEG-fNIRS outperforms unimodal EEG or fNIRS despite major differences in their conceptual and methodological aspects. Much work however remains to be done to reach practical applications of bimodal EEG-fNIRS in naturalistic conditions. We consider these points to identify aspects of bimodal EEG-fNIRS research in which progress is expected or desired.

Temel Duygusal Durumların Hemodinamik Karşılıklarının Taşınabilir bir İşlevsel Yakın Kızılaltı Spektroskopi Sistemi ile Tanımlanması

Objective: The aim of this study was to test the feasibility of a functional near infrared spectroscopy (fNIRS) system for quantification of the similarities and differences in the spatial localization of cerebral hemodynamic activation induced by visual presentation of neutral, negative and positive valence emotional stimuli. Method: 13 healthy subjects viewed neutral, pleasant and unpleasant pictures from the International Affective Picture System (IAPS) database in a block design experiment while the prefrontal cortical hemodynamic changes induced by emotional stimuli were continuously recorded with a 20 channel fNIRS system that covered the forehead region. Results: Negative valence pictures induced higher hemodynamic activity in right lateralized regions involving dorsolateral and oribtofrontal cortex when compared to neutral and positive valence stimuli (pFDR

MCFHNet: Multi-Channel Fusion Hybrid Network for Efficient EEG-fNIRS Multi-modal Motor Imagery Decoding

Motor Imagery-based Brain Computer Interface (MI-BCI) is a typical active BCI with a main focus on motor intention identification. Hybrid motor imagery (MI) decoding methods that based on multi-modal fusion of Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), especially deep learning-based methods, become popular in recent MI-BCI studies. However, the fusion strategy and network design in deep learning-based methods are complex. To solve this problem, we proposed the multi-channel fusion method (MCF) to simplify current fusion methods, and we designed a multi-channel fusion hybrid network (MCFHNet) based on MCF. MCFHNet combines depthwise convolutional layers, channel attention mechanism, and Bidirectional Long Short Term Memory (Bi-LSTM) layers, which enables strong capability of feature extraction in spatial and temporal domain. The comparison between MCFHNet and representative deep learning-based methods was performed on an open EEG-fNIRS dataset. We found the proposed method can yield superior performance (mean accuracy of 99.641 % in 5-fold cross validation of an intra-subject experiment). This work provides a new option for multi-modal MI decoding, which can be applied in the rehabilitation field based on hybrid BCI systems.

Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli – an exploding and a burning box – interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern – a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by –1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.

A Unified Local–Global Feature Extraction Network for Human Gait Recognition Using Smartphone Sensors

Smartphone-based gait recognition has been considered a unique and promising technique for biometric-based identification. It is integrated with multiple sensors to collect inertial data while a person walks. However, captured data may be affected by several covariate factors due to variations of gait sequences such as holding loads, wearing types, shoe types, etc. Recent gait recognition approaches either work on global or local features, causing failure to handle these covariate-based features. To address these issues, a novel weighted multi-scale CNN (WMsCNN) architecture is designed to extract local to global features for boosting recognition accuracy. Specifically, a weight update sub-network (Ws) is proposed to increase or reduce the weights of features concerning their contribution to the final classification task. Thus, the sensitivity of these features toward the covariate factors decreases using the weight updated technique. Later, these features are fed to a fusion module used to produce global features for the overall classification. Extensive experiments have been conducted on four different benchmark datasets, and the demonstrated results of the proposed model are superior to other state-of-the-art deep learning approaches.

Unmanned Aerial Vehicle for Laser Based Biomedical Sensor Development and Examination of Device Trajectory

Controller design and signal processing for the control of air-vehicles have gained extreme importance while interacting with humans to form a brain–computer interface. This is because fewer commands need to be mapped into multiple controls. For our anticipated biomedical sensor for breath analysis, it is mandatory to provide medication to the patients on an urgent basis. To address this increasingly tense situation in terms of emergencies, we plan to design an unmanned vehicle that can aid spontaneously to monitor the person’s health, and help the physician spontaneously during the rescue mission. Simultaneously, that must be done in such a computationally efficient algorithm that the minimum amount of energy resources are consumed. For this purpose, we resort to an unmanned logistic air-vehicle which flies from the medical centre to the affected person. After obtaining restricted permission from the regional administration, numerous challenges are identified for this design. The device is able to lift a weight of 2 kg successfully which is required for most emergency medications, while choosing the smallest distance to the destination with the GPS. By recording the movement of the vehicle in numerous directions, the results deviate to a maximum of 2% from theoretical investigations. In this way, our biomedical sensor provides critical information to the physician, who is able to provide medication to the patient urgently. On account of reasonable supply of medicines to the destination in terms of weight and time, this experimentation has been rendered satisfactory by the relevant physicians in the vicinity.

Dual Passive Reactive Brain-Computer Interface: A Novel Approach to Human-Machine Symbiosis

The present study proposes a novel concept of neuroadaptive technology, namely a dual passive-reactive Brain-Computer Interface (BCI), that enables bi-directional interaction between humans and machines. We have implemented such a system in a realistic flight simulator using the NextMind classification algorithms and framework to decode pilots' intention (reactive BCI) and to infer their level of attention (passive BCI). Twelve pilots used the reactive BCI to perform checklists along with an anti-collision radar monitoring task that was supervised by the passive BCI. The latter simulated an automatic avoidance maneuver when it detected that pilots missed an incoming collision. The reactive BCI reached 100% classification accuracy with a mean reaction time of 1.6 s when exclusively performing the checklist task. Accuracy was up to 98.5% with a mean reaction time of 2.5 s when pilots also had to fly the aircraft and monitor the anti-collision radar. The passive BCI achieved a F1-score of 0.94. This first demonstration shows the potential of a dual BCI to improve human-machine teaming which could be applied to a variety of applications.

Involvement of the Rostromedial Prefrontal Cortex in Human-Robot Interaction: fNIRS Evidence From a Robot-Assisted Motor Task

Assistive exoskeleton robots are being widely applied in neurorehabilitation to improve upper-limb motor and somatosensory functions. During robot-assisted exercises, the central nervous system appears to highly attend to external information-processing (IP) to efficiently interact with robotic assistance. However, the neural mechanisms underlying this process remain unclear. The rostromedial prefrontal cortex (rmPFC) may be the core of the executive resource allocation that generates biases in the allocation of processing resources toward an external IP according to current behavioral demands. Here, we used functional near-infrared spectroscopy to investigate the cortical activation associated with executive resource allocation during a robot-assisted motor task. During data acquisition, participants performed a right-arm motor task using elbow flexion-extension movements in three different loading conditions: robotic assistive loading (ROB), resistive loading (RES), and non-loading (NON). Participants were asked to strive for kinematic consistency in their movements. A one-way repeated measures analysis of variance and general linear model-based methods were employed to examine task-related activity. We demonstrated that hemodynamic responses in the ventral and dorsal rmPFC were higher during ROB than during NON. Moreover, greater hemodynamic responses in the ventral rmPFC were observed during ROB than during RES. Increased activation in ventral and dorsal rmPFC subregions may be involved in the executive resource allocation that prioritizes external IP during human-robot interactions. In conclusion, these findings provide novel insights regarding the involvement of executive control during a robot-assisted motor task.

Novel fNIRS study on homogeneous symmetric feature-based transfer learning for brain-computer interface

The brain-computer interface (BCI) provides an alternate means of communication between the brain and external devices by recognizing the brain activities and translating them into external commands. The functional Near-Infrared Spectroscopy (fNIRS) is becoming popular as a non-invasive modality for brain activity detection. The recent trends show that deep learning has significantly enhanced the performance of the BCI systems. But the inherent bottleneck for deep learning (in the domain of BCI) is the requirement of the vast amount of training data, lengthy recalibrating time, and expensive computational resources for training deep networks. Building a high-quality, large-scale annotated dataset for deep learning-based BCI systems is exceptionally tedious, complex, and expensive. This study investigates the novel application of transfer learning for fNIRS-based BCI to solve three objective functions (concerns), i.e., the problem of insufficient training data, reduced training time, and increased accuracy. We applied symmetric homogeneous feature-based transfer learning on convolutional neural network (CNN) designed explicitly for fNIRS data collected from twenty-six (26) participants performing the n-back task. The results suggested that the proposed method achieves the maximum saturated accuracy sooner and outperformed the traditional CNN model on averaged accuracy by 25.58% in the exact duration of training time, reducing the training time, recalibrating time, and computational resources.

Novel Hybrid Brain-Computer Interface for Virtual Reality Applications Using Steady-State Visual-Evoked Potential-Based Brain-Computer Interface and Electrooculogram-Based Eye Tracking for Increased Information Transfer Rate

Brain-computer interfaces (BCIs) based on electroencephalogram (EEG) have recently attracted increasing attention in virtual reality (VR) applications as a promising tool for controlling virtual objects or generating commands in a "hands-free" manner. Video-oculography (VOG) has been frequently used as a tool to improve BCI performance by identifying the gaze location on the screen, however, current VOG devices are generally too expensive to be embedded in practical low-cost VR head-mounted display (HMD) systems. In this study, we proposed a novel calibration-free hybrid BCI system combining steady-state visual-evoked potential (SSVEP)-based BCI and electrooculogram (EOG)-based eye tracking to increase the information transfer rate (ITR) of a nine-target SSVEP-based BCI in VR environment. Experiments were repeated on three different frequency configurations of pattern-reversal checkerboard stimuli arranged in a 3 × 3 matrix. When a user was staring at one of the nine visual stimuli, the column containing the target stimulus was first identified based on the user's horizontal eye movement direction (left, middle, or right) classified using horizontal EOG recorded from a pair of electrodes that can be readily incorporated with any existing VR-HMD systems. Note that the EOG can be recorded using the same amplifier for recording SSVEP, unlike the VOG system. Then, the target visual stimulus was identified among the three visual stimuli vertically arranged in the selected column using the extension of multivariate synchronization index (EMSI) algorithm, one of the widely used SSVEP detection algorithms. In our experiments with 20 participants wearing a commercial VR-HMD system, it was shown that both the accuracy and ITR of the proposed hybrid BCI were significantly increased compared to those of the traditional SSVEP-based BCI in VR environment.

Steady-State Visual Evoked Potential-Based Brain–Computer Interface Using a Novel Visual Stimulus with Quick Response (QR) Code Pattern

Steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) systems suffer from low SSVEP response intensity and visual fatigue, resulting in lower accuracy when operating the system for continuous commands, such as an electric wheelchair control. This study proposes two SSVEP improvements to create a practical BCI for communication and control in disabled people. The first is flicker pattern modification for increasing SSVEP response through mixing (1) fundamental and first harmonic frequencies, and (2) two fundamental frequencies for an additional number of commands. The second method utilizes a quick response (QR) code for visual stimulus patterns to increase the SSVEP response and reduce visual fatigue. Eight different stimulus patterns from three flickering frequencies (7, 13, and 17 Hz) were presented to twelve participants for the test and score levels of visual fatigue. Two popular SSVEP methods, i.e., power spectral density (PSD) with Welch periodogram and canonical correlation analysis (CCA) with overlapping sliding window, are used to detect SSVEP intensity and response, compared to the checkerboard pattern. The results suggest that the QR code patterns can yield higher accuracy than checkerboard patterns for both PSD and CCA methods. Moreover, a QR code pattern with low frequency can reduce visual fatigue; however, visual fatigue can be easily affected by high flickering frequency. The findings can be used in the future to implement a real-time, SSVEP-based BCI for verifying user and system performance in actual environments.