Hybrid EEG-fNIRS Asynchronous Brain-Computer Interface for Multiple Motor Tasks

Co-localized optode-electrode design for multimodal functional near infrared spectroscopy and electroencephalography

Significance

Neuroscience of the everyday world requires continuous mobile brain imaging in real time and in ecologically valid environments, which aids in directly translating research for human benefit. Combined functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) studies have increased in demand, as the combined systems can provide great insights into cortical hemodynamics, neuronal activity, and neurovascular coupling. However, fNIRS-EEG studies remain limited in modularity and portability due to restrictions in combined cap designs, especially for high-density (HD) fNIRS measurements.

Aim

We have built and tested custom fNIRS sources that attach to electrodes without decreasing the overall modularity and portability of the probe.

Approach

To demonstrate the design's utility, we screened for any potential interference and performed a HD-fNIRS-EEG measurement with co-located opto-electrode positions during a modified Stroop task.

Results

No observable interference was present from the fNIRS source optodes in the EEG spectral analysis. The performance, fNIRS, and EEG results of the Stroop task supported the trends from previous research. We observed increased activation with both fNIRS and EEG within the regions of interest.

Conclusion

Overall, these results suggest that the co-localization method is a promising approach to multimodal imaging.

From light to insight: Functional near-infrared spectroscopy for unravelling cognitive impairment during task performance

Cognitive impairment refers to the impairment of higher brain functions such as perception, thinking or memory that affects the individual's ability to perform daily or social activities. Studies have found that changes in neuronal activity during tasks in patients with cognitive impairment are closely related to changes in cerebral cortical hemodynamics. Functional near-infrared spectroscopy is an indirect method to measure neural activity based on changes in blood oxygen concentration in the cerebral cortex. Due to its strong anti-motion interference, high compatibility, and almost no restriction on participants and environment, it has shown great potential in the research field of cognitive impairment. Recognizing these benefits, this comprehensive review systematically elucidates the rationale, historical development, advantages and disadvantages of functional near-infrared spectroscopy, and also discusses the applications of combining functional near-infrared spectroscopy with other detection techniques. Additionally, this review summarized how functional near-infrared spectroscopy can be applied to cognitive impairment caused by different diseases, ultimately aiding the study of neural mechanisms of cognitive activities, which is crucial for the diagnosis, differentiation and treatment of cognitive impairment.

A hybrid CNN model for classification of motor tasks obtained from hybrid BCI system

The Hybrid-Brain Computer Interface (BCI) has shown improved performance, especially in classifying multi-class data. Two non-invasive BCI modules are combined to achieve an improved classification which are Electroencephalogram (EEG) and functional Near Infra-red Spectroscopy (fNIRS). Classifying contralateral and ipsilateral motor movements is found challenging among the other mental activity signals. The current work focuses on the performance of deep learning methods like - Convolutional Neural Networks (CNN) and Bidirectional Long-Short term memory (Bi-LSTM) in classifying a four-class motor execution of Right Hand, Left Hand, Right Arm and Left Arm taken from the CORE dataset. The model performance was evaluated using metrics such as Accuracy, F1 - score, Precision, Recall, AUC and ROC curve. The CNN and Hybrid CNN models have resulted in 98.3% and 99% accuracy respectively.

Temporal attention fusion network with custom loss function for EEG-fNIRS classification

Objective.Methods that can detect brain activities accurately are crucial owing to the increasing prevalence of neurological disorders. In this context, a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) offers a powerful approach to understanding normal and pathological brain functions, thereby overcoming the limitations of each modality, such as susceptibility to artifacts of EEG and limited temporal resolution of fNIRS. However, challenges such as class imbalance and inter-class variability within multisubject data hinder their full potential.Approach.To address this issue, we propose a novel temporal attention fusion network (TAFN) with a custom loss function. The TAFN model incorporates attention mechanisms to its long short-term memory and temporal convolutional layers to accurately capture spatial and temporal dependencies in the EEG-fNIRS data. The custom loss function combines class weights and asymmetric loss terms to ensure the precise classification of cognitive and motor intentions, along with addressing class imbalance issues.Main results.Rigorous testing demonstrated the exceptional cross-subject accuracy of the TAFN, exceeding 99% for cognitive tasks and 97% for motor imagery (MI) tasks. Additionally, the ability of the model to detect subtle differences in epilepsy was analyzed using scalp topography in MI tasks.Significance.This study presents a technique that outperforms traditional methods for detecting high-precision brain activity with subtle differences in the associated patterns. This makes it a promising tool for applications such as epilepsy and seizure detection, in which discerning subtle pattern differences is of paramount importance.

Identification of drug use degree by integrating multi-modal features with dual-input deep learning method

Most of studies on drug use degree are based on subjective judgments without objective quantitative assessment, in this paper, a dual-input bimodal fusion algorithm is proposed to study drug use degree by using electroencephalogram (EEG) and near-infrared spectroscopy (NIRS). Firstly, this paper uses the optimized dual-input multi-modal TiCBnet for extracting the deep encoding features of the bimodal signal, then fuses and screens the features using different methods, and finally fused deep encoding features are classified. The classification accuracy of bimodal is found to be higher than that of single modal, and the classification accuracy is up to 89.9%.

Investigating the relationship between CSAT scores and prefrontal fNIRS signals during cognitive tasks using a quantum annealing algorithm

Academic achievement is a critical measure of intellectual ability, prompting extensive research into cognitive tasks as potential predictors. Neuroimaging technologies, such as functional near-infrared spectroscopy (fNIRS), offer insights into brain hemodynamics, allowing understanding of the link between cognitive performance and academic achievement. Herein, we explored the association between cognitive tasks and academic achievement by analyzing prefrontal fNIRS signals. A novel quantum annealer (QA) feature selection algorithm was applied to fNIRS data to identify cognitive tasks correlated with CSAT scores. Twelve features (signal mean, median, variance, peak, number of peaks, sum of peaks, range, minimum, kurtosis, skewness, standard deviation, and root mean square) were extracted from fNIRS signals at two time windows (10- and 60-s) to compare results from various feature variable conditions. The feature selection results from the QA-based and XGBoost regressor algorithms were compared to validate the former's performance. In a two-step validation process using multiple linear regression models, model fitness (adjusted R2) and model prediction error (RMSE) values were calculated. The quantum annealer demonstrated comparable performance to classical machine learning models, and specific cognitive tasks, including verbal fluency, recognition, and the Corsi block tapping task, were correlated with academic achievement. Group analyses revealed stronger associations between Tower of London and N-back tasks with higher CSAT scores. Quantum annealing algorithms have significant potential in feature selection using fNIRS data, and represents a novel research approach. Future studies should explore predictors of academic achievement and cognitive ability.

Simultaneous EEG-fNIRS Data Classification Through Selective Channel Representation and Spectrogram Imaging

The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) can facilitate the advancement of brain-computer interfaces (BCIs). However, existing research in this domain has grappled with the challenge of the efficient selection of features, resulting in the underutilization of the temporal richness of EEG and the spatial specificity of fNIRS data.To effectively address this challenge, this study proposed a deep learning architecture called the multimodal DenseNet fusion (MDNF) model that was trained on two-dimensional (2D) EEG data images, leveraging advanced feature extraction techniques. The model transformed EEG data into 2D images using a short-time Fourier transform, applied transfer learning to extract discriminative features, and consequently integrated them with fNIRS-derived spectral entropy features. This approach aimed to bridge existing gaps in EEG-fNIRS-based BCI research by enhancing classification accuracy and versatility across various cognitive and motor imagery tasks.Experimental results on two public datasets demonstrated the superiority of our model over existing state-of-the-art methods.Thus, the high accuracy and precise feature utilization of the MDNF model demonstrates the potential in clinical applications for neurodiagnostics and rehabilitation, thereby paving the method for patient-specific therapeutic strategies.

A Within-Subject Multimodal NIRS-EEG Classifier for Infant Data

Functional Near Infrared Spectroscopy (fNIRS) and Electroencephalography (EEG) are commonly employed neuroimaging methods in developmental neuroscience. Since they offer complementary strengths and their simultaneous recording is relatively easy, combining them is highly desirable. However, to date, very few infant studies have been conducted with NIRS-EEG, partly because analyzing and interpreting multimodal data is challenging. In this work, we propose a framework to carry out a multivariate pattern analysis that uses an NIRS-EEG feature matrix, obtained by selecting EEG trials presented within larger NIRS blocks, and combining the corresponding features. Importantly, this classifier is intended to be sensitive enough to apply to individual-level, and not group-level data. We tested the classifier on NIRS-EEG data acquired from five newborn infants who were listening to human speech and monkey vocalizations. We evaluated how accurately the model classified stimuli when applied to EEG data alone, NIRS data alone, or combined NIRS-EEG data. For three out of five infants, the classifier achieved high and statistically significant accuracy when using features from the NIRS data alone, but even higher accuracy when using combined EEG and NIRS data, particularly from both hemoglobin components. For the other two infants, accuracies were lower overall, but for one of them the highest accuracy was still achieved when using combined EEG and NIRS data with both hemoglobin components. We discuss how classification based on joint NIRS-EEG data could be modified to fit the needs of different experimental paradigms and needs.

Correlation-Filter-Based Channel and Feature Selection Framework for Hybrid EEG-fNIRS BCI Applications

The proposed study is based on a feature and channel selection strategy that uses correlation filters for brain-computer interface (BCI) applications using electroencephalography (EEG)-functional near-infrared spectroscopy (fNIRS) brain imaging modalities. The proposed approach fuses the complementary information of the two modalities to train the classifier. The channels most closely correlated with brain activity are extracted using a correlation-based connectivity matrix for fNIRS and EEG separately. Furthermore, the training vector is formed through the identification and fusion of the statistical features of both modalities (i.e., slope, skewness, maximum, skewness, mean, and kurtosis). The constructed fused feature vector is passed through various filters (including ReliefF, minimum redundancy maximum relevance, chi-square test, analysis of variance, and Kruskal-Wallis filters) to remove redundant information before training. Traditional classifiers such as neural networks, support-vector machines, linear discriminant analysis, and ensembles were used for the purpose of training and testing. A publicly available dataset with motor imagery information was used for validation of the proposed approach. Our findings indicate that the proposed correlation-filter-based channel and feature selection framework significantly enhances the classification accuracy of hybrid EEG-fNIRS. The ReliefF-based filter outperformed other filters with the ensemble classifier with a high accuracy of 94.77 ± 4.26%. The statistical analysis also validated the significance (p < 0.01) of the results. A comparison of the proposed framework with the prior findings was also presented. Our results show that the proposed approach can be used in future EEG-fNIRS-based hybrid BCI applications.

Exploring Aesthetic Perception in Impaired Aging: A Multimodal Brain-Computer Interface Study

In the field of neuroscience, brain-computer interfaces (BCIs) are used to connect the human brain with external devices, providing insights into the neural mechanisms underlying cognitive processes, including aesthetic perception. Non-invasive BCIs, such as EEG and fNIRS, are critical for studying central nervous system activity and understanding how individuals with cognitive deficits process and respond to aesthetic stimuli. This study assessed twenty participants who were divided into control and impaired aging (AI) groups based on MMSE scores. EEG and fNIRS were used to measure their neurophysiological responses to aesthetic stimuli that varied in pleasantness and dynamism. Significant differences were identified between the groups in P300 amplitude and late positive potential (LPP), with controls showing greater reactivity. AI subjects showed an increase in oxyhemoglobin in response to pleasurable stimuli, suggesting hemodynamic compensation. This study highlights the effectiveness of multimodal BCIs in identifying the neural basis of aesthetic appreciation and impaired aging. Despite its limitations, such as sample size and the subjective nature of aesthetic appreciation, this research lays the groundwork for cognitive rehabilitation tailored to aesthetic perception, improving the comprehension of cognitive disorders through integrated BCI methodologies.

Mental workload assessment by monitoring brain, heart, and eye with six biomedical modalities during six cognitive tasks

Introduction

The efficiency and safety of complex high precision human-machine systems such as in aerospace and robotic surgery are closely related to the cognitive readiness, ability to manage workload, and situational awareness of their operators. Accurate assessment of mental workload could help in preventing operator error and allow for pertinent intervention by predicting performance declines that can arise from either work overload or under stimulation. Neuroergonomic approaches based on measures of human body and brain activity collectively can provide sensitive and reliable assessment of human mental workload in complex training and work environments.

Methods

In this study, we developed a new six-cognitive-domain task protocol, coupling it with six biomedical monitoring modalities to concurrently capture performance and cognitive workload correlates across a longitudinal multi-day investigation. Utilizing two distinct modalities for each aspect of cardiac activity (ECG and PPG), ocular activity (EOG and eye-tracking), and brain activity (EEG and fNIRS), 23 participants engaged in four sessions over 4 weeks, performing tasks associated with working memory, vigilance, risk assessment, shifting attention, situation awareness, and inhibitory control.

Results

The results revealed varying levels of sensitivity to workload within each modality. While certain measures exhibited consistency across tasks, neuroimaging modalities, in particular, unveiled meaningful differences between task conditions and cognitive domains.

Discussion

This is the first comprehensive comparison of these six brain-body measures across multiple days and cognitive domains. The findings underscore the potential of wearable brain and body sensing methods for evaluating mental workload. Such comprehensive neuroergonomic assessment can inform development of next generation neuroadaptive interfaces and training approaches for more efficient human-machine interaction and operator skill acquisition.

Transferring a deep learning model from healthy subjects to stroke patients in a motor imagery brain-computer interface

Objective.Motor imagery (MI) brain-computer interfaces (BCIs) based on electroencephalogram (EEG) have been developed primarily for stroke rehabilitation, however, due to limited stroke data, current deep learning methods for cross-subject classification rely on healthy data. This study aims to assess the feasibility of applying MI-BCI models pre-trained using data from healthy individuals to detect MI in stroke patients.Approach.We introduce a new transfer learning approach where features from two-class MI data of healthy individuals are used to detect MI in stroke patients. We compare the results of the proposed method with those obtained from analyses within stroke data. Experiments were conducted using Deep ConvNet and state-of-the-art subject-specific machine learning MI classifiers, evaluated on OpenBMI two-class MI-EEG data from healthy subjects and two-class MI versus rest data from stroke patients.Main results.Results of our study indicate that through domain adaptation of a model pre-trained using healthy subjects' data, an average MI detection accuracy of 71.15% (±12.46%) can be achieved across 71 stroke patients. We demonstrate that the accuracy of the pre-trained model increased by 18.15% after transfer learning (p<0.001). Additionally, the proposed transfer learning method outperforms the subject-specific results achieved by Deep ConvNet and FBCSP, with significant enhancements of 7.64% (p<0.001) and 5.55% (p<0.001) in performance, respectively. Notably, the healthy-to-stroke transfer learning approach achieved similar performance to stroke-to-stroke transfer learning, with no significant difference (p>0.05). Explainable AI analyses using transfer models determined channel relevance patterns that indicate contributions from the bilateral motor, frontal, and parietal regions of the cortex towards MI detection in stroke patients.Significance.Transfer learning from healthy to stroke can enhance the clinical use of BCI algorithms by overcoming the challenge of insufficient clinical data for optimal training.

Beta bursts question the ruling power for brain-computer interfaces

Objective: Current efforts to build reliable brain-computer interfaces (BCI) span multiple axes from hardware, to software, to more sophisticated experimental protocols, and personalized approaches. However, despite these abundant efforts, there is still room for significant improvement. We argue that a rather overlooked direction lies in linking BCI protocols with recent advances in fundamental neuroscience.Approach: In light of these advances, and particularly the characterization of the burst-like nature of beta frequency band activity and the diversity of beta bursts, we revisit the role of beta activity in 'left vs. right hand' motor imagery (MI) tasks. Current decoding approaches for such tasks take advantage of the fact that MI generates time-locked changes in induced power in the sensorimotor cortex and rely on band-passed power changes in single or multiple channels. Although little is known about the dynamics of beta burst activity during MI, we hypothesized that beta bursts should be modulated in a way analogous to their activity during performance of real upper limb movements.Main results and Significance: We show that classification features based on patterns of beta burst modulations yield decoding results that are equivalent to or better than typically used beta power across multiple open electroencephalography datasets, thus providing insights into the specificity of these bio-markers.

fNIRS-EEG BCIs for Motor Rehabilitation: A Review

Motor impairment has a profound impact on a significant number of individuals, leading to a substantial demand for rehabilitation services. Through brain-computer interfaces (BCIs), people with severe motor disabilities could have improved communication with others and control appropriately designed robotic prosthetics, so as to (at least partially) restore their motor abilities. BCI plays a pivotal role in promoting smoother communication and interactions between individuals with motor impairments and others. Moreover, they enable the direct control of assistive devices through brain signals. In particular, their most significant potential lies in the realm of motor rehabilitation, where BCIs can offer real-time feedback to assist users in their training and continuously monitor the brain's state throughout the entire rehabilitation process. Hybridization of different brain-sensing modalities, especially functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG), has shown great potential in the creation of BCIs for rehabilitating the motor-impaired populations. EEG, as a well-established methodology, can be combined with fNIRS to compensate for the inherent disadvantages and achieve higher temporal and spatial resolution. This paper reviews the recent works in hybrid fNIRS-EEG BCIs for motor rehabilitation, emphasizing the methodologies that utilized motor imagery. An overview of the BCI system and its key components was introduced, followed by an introduction to various devices, strengths and weaknesses of different signal processing techniques, and applications in neuroscience and clinical contexts. The review concludes by discussing the possible challenges and opportunities for future development.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

OptEF-BCI: An Optimization-Based Hybrid EEG and fNIRS-Brain Computer Interface

Multimodal data fusion (electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS)) has been developed as an important neuroimaging research field in order to circumvent the inherent limitations of individual modalities by combining complementary information from other modalities. This study employed an optimization-based feature selection algorithm to systematically investigate the complementary nature of multimodal fused features. After preprocessing the acquired data of both modalities (i.e., EEG and fNIRS), the temporal statistical features were computed separately with a 10 s interval for each modality. The computed features were fused to create a training vector. A wrapper-based binary enhanced whale optimization algorithm (E-WOA) was used to select the optimal/efficient fused feature subset using the support-vector-machine-based cost function. An online dataset of 29 healthy individuals was used to evaluate the performance of the proposed methodology. The findings suggest that the proposed approach enhances the classification performance by evaluating the degree of complementarity between characteristics and selecting the most efficient fused subset. The binary E-WOA feature selection approach showed a high classification rate (94.22 ± 5.39%). The classification performance exhibited a 3.85% increase compared with the conventional whale optimization algorithm. The proposed hybrid classification framework outperformed both the individual modalities and traditional feature selection classification (p < 0.01). These findings indicate the potential efficacy of the proposed framework for several neuroclinical applications.

Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

Objective. A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. Multimodal MI-BCI that uses multiple neural signals contains rich common and complementary information and is promising for enhancing the decoding accuracy of MI-BCI. However, the heterogeneity of different modalities makes the multimodal decoding task difficult. How to effectively utilize multimodal information remains to be further studied.Approach. In this study, a multimodal MI decoding neural network was proposed. Spatial feature alignment losses were designed to enhance the feature representations extracted from the heterogeneous data and guide the fusion of features from different modalities. An attention-based modality fusion module was built to align and fuse the features in the temporal dimension. To evaluate the proposed decoding method, a five-class MI electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) dataset were constructed.Main results and significance. The comparison experimental results showed that the proposed decoding method achieved higher decoding accuracy than the compared methods on both the self-collected dataset and a public dataset. The ablation results verified the effectiveness of each part of the proposed method. Feature distribution visualization results showed that the proposed losses enhance the feature representation of EEG and fNIRS modalities. The proposed method based on EEG and fNIRS modalities has significant potential for improving decoding performance of MI tasks.

Explainable artificial intelligence model to predict brain states from fNIRS signals

Objective: Most Deep Learning (DL) methods for the classification of functional Near-Infrared Spectroscopy (fNIRS) signals do so without explaining which features contribute to the classification of a task or imagery. An explainable artificial intelligence (xAI) system that can decompose the Deep Learning mode's output onto the input variables for fNIRS signals is described here. Approach: We propose an xAI-fNIRS system that consists of a classification module and an explanation module. The classification module consists of two separately trained sliding window-based classifiers, namely, (i) 1-D Convolutional Neural Network (CNN); and (ii) Long Short-Term Memory (LSTM). The explanation module uses SHAP (SHapley Additive exPlanations) to explain the CNN model's output in terms of the model's input. Main results: We observed that the classification module was able to classify two types of datasets: (a) Motor task (MT), acquired from three subjects; and (b) Motor imagery (MI), acquired from 29 subjects, with an accuracy of over 96% for both CNN and LSTM models. The explanation module was able to identify the channels contributing the most to the classification of MI or MT and therefore identify the channel locations and whether they correspond to oxy- or deoxy-hemoglobin levels in those locations. Significance: The xAI-fNIRS system can distinguish between the brain states related to overt and covert motor imagery from fNIRS signals with high classification accuracy and is able to explain the signal features that discriminate between the brain states of interest.

Early-stage fusion of EEG and fNIRS improves classification of motor imagery

Introduction

Many research papers have reported successful implementation of hybrid brain-computer interfaces by complementarily combining EEG and fNIRS, to improve classification performance. However, modality or feature fusion of EEG and fNIRS was usually designed for specific user cases, which were generally customized and hard to be generalized. How to effectively utilize information from the two modalities was still unclear.

Methods

In this paper, we conducted a study to investigate the stage of bi-modal fusion based on EEG and fNIRS. A Y-shaped neural network was proposed and evaluated on an open dataset, which fuses the bimodal information in different stages.

Results

The results suggests that the early-stage fusion of EEG and fNIRS have significantly higher performance compared to middle-stage and late-stage fusion network configuration (N = 57, P < 0.05). With the proposed framework, the average accuracy of 29 participants reaches 76.21% in the left-or-right hand motor imagery task in leave-one-out cross-validation, using bi-modal data as network inputs respectively, which is in the same level as the state-of-the-art hybrid BCI methods based on EEG and fNIRS data.

EEG hybrid brain-computer interfaces: A scoping review applying an existing hybrid-BCI taxonomy and considerations for pediatric applications

Most hybrid brain-computer interfaces (hBCI) aim at improving the performance of single-input BCI. Many combinations are possible to configure an hBCI, such as using multiple brain input signals, different stimuli or more than one input system. Multiple studies have been done since 2010 where such interfaces have been tested and analyzed. Results and conclusions are promising but little has been discussed as to what is the best approach for the pediatric population, should they use hBCI as an assistive technology. Children might face greater challenges when using BCI and might benefit from less complex interfaces. Hence, in this scoping review we included 42 papers that developed hBCI systems for the purpose of control of assistive devices or communication software, and we analyzed them through the lenses of potential use in clinical settings and for children. We extracted taxonomic categories proposed in previous studies to describe the types of interfaces that have been developed. We also proposed interface characteristics that could be observed in different hBCI, such as type of target, number of targets and number of steps before selection. Then, we discussed how each of the extracted characteristics could influence the overall complexity of the system and what might be the best options for applications for children. Effectiveness and efficiency were also collected and included in the analysis. We concluded that the least complex hBCI interfaces might involve having a brain inputs and an external input, with a sequential role of operation, and visual stimuli. Those interfaces might also use a minimal number of targets of the strobic type, with one or two steps before the final selection. We hope this review can be used as a guideline for future hBCI developments and as an incentive to the design of interfaces that can also serve children who have motor impairments.

A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI

Decoding neural responses from multimodal information sources, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), has the transformative potential to advance hybrid brain-computer interfaces (hBCIs). However, existing modest performance improvement of hBCIs might be attributed to the lack of computational frameworks that exploit complementary synergistic properties in multimodal features. This study proposes a multimodal data fusion framework to represent and decode synergistic multimodal motor imagery (MI) neural responses. We hypothesize that exploiting EEG nonlinear dynamics adds a new informative dimension to the commonly combined EEG-fNIRS features and will ultimately increase the synergy between EEG and fNIRS features toward an enhanced hBCI. The EEG nonlinear dynamics were quantified by extracting graph-based recurrence quantification analysis (RQA) features to complement the commonly used spectral features for an enhanced multimodal configuration when combined with fNIRS. The high-dimensional multimodal features were further given to a feature selection algorithm relying on the least absolute shrinkage and selection operator (LASSO) for fused feature selection. Linear support vector machine (SVM) was then used to evaluate the framework. The mean hybrid classification performance improved by up to 15% and 4% compared to the unimodal EEG and fNIRS, respectively. The proposed graph-based framework substantially increased the contribution of EEG features for hBCI classification from 28.16% up to 52.9% when introduced the nonlinear dynamics and improved the performance by approximately 2%. These findings suggest that graph-based nonlinear dynamics can increase the synergy between EEG and fNIRS features for an enhanced MI response representation that is not dominated by a single modality.

Bimodal EEG-fNIRS in Neuroergonomics. Current Evidence and Prospects for Future Research

Neuroergonomics focuses on the brain signatures and associated mental states underlying behavior to design human-machine interfaces enhancing performance in the cognitive and physical domains. Brain imaging techniques such as functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) have been considered key methods for achieving this goal. Recent research stresses the value of combining EEG and fNIRS in improving these interface systems' mental state decoding abilities, but little is known about whether these improvements generalize over different paradigms and methodologies, nor about the potentialities for using these systems in the real world. We review 33 studies comparing mental state decoding accuracy between bimodal EEG-fNIRS and unimodal EEG and fNIRS in several subdomains of neuroergonomics. In light of these studies, we also consider the challenges of exploiting wearable versions of these systems in real-world contexts. Overall the studies reviewed suggest that bimodal EEG-fNIRS outperforms unimodal EEG or fNIRS despite major differences in their conceptual and methodological aspects. Much work however remains to be done to reach practical applications of bimodal EEG-fNIRS in naturalistic conditions. We consider these points to identify aspects of bimodal EEG-fNIRS research in which progress is expected or desired.

Concurrent fNIRS and EEG for Brain Function Investigation: A Systematic, Methodology-Focused Review

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) stand as state-of-the-art techniques for non-invasive functional neuroimaging. On a unimodal basis, EEG has poor spatial resolution while presenting high temporal resolution. In contrast, fNIRS offers better spatial resolution, though it is constrained by its poor temporal resolution. One important merit shared by the EEG and fNIRS is that both modalities have favorable portability and could be integrated into a compatible experimental setup, providing a compelling ground for the development of a multimodal fNIRS-EEG integration analysis approach. Despite a growing number of studies using concurrent fNIRS-EEG designs reported in recent years, the methodological reference of past studies remains unclear. To fill this knowledge gap, this review critically summarizes the status of analysis methods currently used in concurrent fNIRS-EEG studies, providing an up-to-date overview and guideline for future projects to conduct concurrent fNIRS-EEG studies. A literature search was conducted using PubMed and Web of Science through 31 August 2021. After screening and qualification assessment, 92 studies involving concurrent fNIRS-EEG data recordings and analyses were included in the final methodological review. Specifically, three methodological categories of concurrent fNIRS-EEG data analyses, including EEG-informed fNIRS analyses, fNIRS-informed EEG analyses, and parallel fNIRS-EEG analyses, were identified and explained with detailed description. Finally, we highlighted current challenges and potential directions in concurrent fNIRS-EEG data analyses in future research.

Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have potentially complementary characteristics that reflect the electrical and hemodynamic characteristics of neural responses, so EEG-fNIRS-based hybrid brain-computer interface (BCI) is the research hotspots in recent years. However, current studies lack a comprehensive systematic approach to properly fuse EEG and fNIRS data and exploit their complementary potential, which is critical for improving BCI performance. To address this issue, this study proposes a novel multimodal fusion framework based on multi-level progressive learning with multi-domain features. The framework consists of a multi-domain feature extraction process for EEG and fNIRS, a feature selection process based on atomic search optimization, and a multi-domain feature fusion process based on multi-level progressive machine learning. The proposed method was validated on EEG-fNIRS-based motor imagery (MI) and mental arithmetic (MA) tasks involving 29 subjects, and the experimental results show that multi-domain features provide better classification performance than single-domain features, and multi-modality provides better classification performance than single-modality. Furthermore, the experimental results and comparison with other methods demonstrated the effectiveness and superiority of the proposed method in EEG and fNIRS information fusion, it can achieve an average classification accuracy of 96.74% in the MI task and 98.42% in the MA task. Our proposed method may provide a general framework for future fusion processing of multimodal brain signals based on EEG-fNIRS.

Toward an Adapted Neurofeedback for Post-stroke Motor Rehabilitation: State of the Art and Perspectives

Stroke is a severe health issue, and motor recovery after stroke remains an important challenge in the rehabilitation field. Neurofeedback (NFB), as part of a brain–computer interface, is a technique for modulating brain activity using on-line feedback that has proved to be useful in motor rehabilitation for the chronic stroke population in addition to traditional therapies. Nevertheless, its use and applications in the field still leave unresolved questions. The brain pathophysiological mechanisms after stroke remain partly unknown, and the possibilities for intervention on these mechanisms to promote cerebral plasticity are limited in clinical practice. In NFB motor rehabilitation, the aim is to adapt the therapy to the patient’s clinical context using brain imaging, considering the time after stroke, the localization of brain lesions, and their clinical impact, while taking into account currently used biomarkers and technical limitations. These modern techniques also allow a better understanding of the physiopathology and neuroplasticity of the brain after stroke. We conducted a narrative literature review of studies using NFB for post-stroke motor rehabilitation. The main goal was to decompose all the elements that can be modified in NFB therapies, which can lead to their adaptation according to the patient’s context and according to the current technological limits. Adaptation and individualization of care could derive from this analysis to better meet the patients’ needs. We focused on and highlighted the various clinical and technological components considering the most recent experiments. The second goal was to propose general recommendations and enhance the limits and perspectives to improve our general knowledge in the field and allow clinical applications. We highlighted the multidisciplinary approach of this work by combining engineering abilities and medical experience. Engineering development is essential for the available technological tools and aims to increase neuroscience knowledge in the NFB topic. This technological development was born out of the real clinical need to provide complementary therapeutic solutions to a public health problem, considering the actual clinical context of the post-stroke patient and the practical limits resulting from it.

fNIRS-Based Upper Limb Motion Intention Recognition Using an Artificial Neural Network for Transhumeral Amputees

Prosthetic arms are designed to assist amputated individuals in the performance of the activities of daily life. Brain machine interfaces are currently employed to enhance the accuracy as well as number of control commands for upper limb prostheses. However, the motion prediction for prosthetic arms and the rehabilitation of amputees suffering from transhumeral amputations is limited. In this paper, functional near-infrared spectroscopy (fNIRS)-based approach for the recognition of human intention for six upper limb motions is proposed. The data were extracted from the study of fifteen healthy subjects and three transhumeral amputees for elbow extension, elbow flexion, wrist pronation, wrist supination, hand open, and hand close. The fNIRS signals were acquired from the motor cortex region of the brain by the commercial NIRSport device. The acquired data samples were filtered using finite impulse response (FIR) filter. Furthermore, signal mean, signal peak and minimum values were computed as feature set. An artificial neural network (ANN) was applied to these data samples. The results show the likelihood of classifying the six arm actions with an accuracy of 78%. The attained results have not yet been reported in any identical study. These achieved fNIRS results for intention detection are promising and suggest that they can be applied for the real-time control of the transhumeral prosthesis.

Recent Advances and Current Trends in Brain-Computer Interface (BCI) Research and Their Applications

Brain-Computer Interface (BCI) provides direct communication between the brain and an external device. BCI systems have become a trendy field of research in recent years. These systems can be used in a variety of applications to help both disabled and healthy people. Concerning significant BCI progress, we may assume that these systems are not very far from real-world applications. This review has taken into account current trends in BCI research. In this survey, one hundred most cited articles from the WOS database were selected over the last four years. This survey is divided into several sectors. These sectors are Medicine, Communication and Control, Entertainment, and Other BCI applications. The application area, recording method, signal acquisition types, and countries of origin have been identified in each article. This survey provides an overview of the BCI articles published from 2016 to 2020 and their current trends and advances in different application areas.

A bimodal deep learning architecture for EEG-fNIRS decoding of overt and imagined speech

OBJECTIVE

Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding.

METHODS

We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding.

RESULTS

Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance: 25%)) indicated a significant improvement on EEG used independently for imagined speech (p=0.020) while tending towards significance for overt speech (p=0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of ~12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways.

CONCLUSION

The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding.

SIGNIFICANCE

This novel architecture can be used to enhance speech decoding from bimodal neural signals.

EEG Electrode Selection for a Two-Class Motor Imagery Task in a BCI Using fNIRS Prior Data

This study investigated the possibility of using functional near infrared spectroscopy (fNIRS) during right- and left-hand motor imagery tasks to select an optimum set of electroencephalography (EEG) electrodes for a brain computer interface. fNIRS has better spatial resolution allowing areas of brain activity to more readily be identified. The ReliefF algorithm was used to identify the most reliable fNIRS channels. Then, EEG electrodes adjacent to those channels were selected for classification. This study used three different classifiers of linear and quadratic discriminant analyses, and support vector machine to examine the proposed method.Clinical Relevance- Reducing the number of sensors in a BCI makes the system more usable for patients with severe disabilities.

Brain–Computer Interfacing Using Functional Near-Infrared Spectroscopy (fNIRS)

Functional Near-Infrared Spectroscopy (fNIRS) is a wearable optical spectroscopy system originally developed for continuous and non-invasive monitoring of brain function by measuring blood oxygen concentration. Recent advancements in brain–computer interfacing allow us to control the neuron function of the brain by combining it with fNIRS to regulate cognitive function. In this review manuscript, we provide information regarding current advancement in fNIRS and how it provides advantages in developing brain–computer interfacing to enable neuron function. We also briefly discuss about how we can use this technology for further applications.

Research on blood oxygen activity in cerebral cortical motor function areas with adjustment intention during gait

BACKGROUND

The study of the neural mechanism of human gait control can provide a theoretical basis for the treatment of walking disorders or the improvement of rehabilitation strategies, and further promote the functional rehabilitation of patients with movement disorders. However, the performance and changes of cerebral cortex activity corresponding to gait adjustment intentions are still not clear.

OBJECTIVE

The purpose of this study was to detect the blood oxygen activation characterization of the cerebral cortex motor function area when people have the intention to adjust gait during walking.

METHODS

Thirty young volunteers (21 ± 1 years old) performed normal walking, speed increase, speed reduction, step increase, and step reduction, during which oxygenated hemoglobin (HbO), deoxygenated hemoglobin (HbR), and total oxyhemoglobin (HbT) information in the prefrontal cortex (PFC), premotor cortex (PMC), supplementary motor area (SMA) was continuous monitored using near-infrared brain functional imaging.

RESULTS

(1) With the intention to adjust gait, the HbO concentration in the SMA increased significantly, while the HbT concentration in the medial-PFC decreased significantly. (2) In the HbO concentration, step reduction is more activated than the step increase in the left-PMC (p= 0.0130); step adjustment is more activated than speed adjustment in the right-PMC (p= 0.0067). In the HbR concentration, the speed reduction is more activated than the speed increase in the left-PFC (p= 0.0103).

CONCLUSIONS

When the intention of gait adjustment occurs, the increase of HbO concentration in the SMA indicates the initial stage of gait adjustment will increase the cognitive-locomotor demand of the brain. The left brain area meets the additional nerve needs of speed adjustment. The preliminary findings of this study can lay an important theoretical foundation for the realization of gait control based on fNIRS-BCI technology.

Wearable, Integrated EEG-fNIRS Technologies: A Review

There has been considerable interest in applying electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) simultaneously for multimodal assessment of brain function. EEG–fNIRS can provide a comprehensive picture of brain electrical and hemodynamic function and has been applied across various fields of brain science. The development of wearable, mechanically and electrically integrated EEG–fNIRS technology is a critical next step in the evolution of this field. A suitable system design could significantly increase the data/image quality, the wearability, patient/subject comfort, and capability for long-term monitoring. Here, we present a concise, yet comprehensive, review of the progress that has been made toward achieving a wearable, integrated EEG–fNIRS system. Significant marks of progress include the development of both discrete component-based and microchip-based EEG–fNIRS technologies; modular systems; miniaturized, lightweight form factors; wireless capabilities; and shared analogue-to-digital converter (ADC) architecture between fNIRS and EEG data acquisitions. In describing the attributes, advantages, and disadvantages of current technologies, this review aims to provide a roadmap toward the next generation of wearable, integrated EEG–fNIRS systems.

Crossing time windows optimization based on mutual information for hybrid BCI

Hybrid EEG-fNIRS brain-computer interface (HBCI) is widely employed to enhance BCI performance. EEG and fNIRS signals are combined to increase the dimensionality of the information. Time windows are used to select EEG and fNIRS singles synchronously. However, it ignores that specific modal signals have their own characteristics, when the task is stimulated, the information between the modalities will mismatch at the moment, which has a significant impact on the classification performance. Here we propose a novel crossing time windows optimization for mental arithmetic (MA) based BCI. The EEG and fNIRS signals were segmented separately by sliding time windows. Then crossing time windows (CTW) were combined with each one segment from EEG and fNIRS selected independently. Furthermore, EEG and fNIRS features were extracted using Filter Bank Common Spatial Pattern (FBCSP) and statistical methods from each sample. Mutual information was calculated for FBCSP and statistical features to characterize the discrimination of crossing time windows, and the optimal window would be selected based on the largest mutual information. Finally, a sparse structured framework of Fisher Lasso feature selection (FLFS) was designed to select the joint features, and conventional Linear Discriminant Analysis (LDA) was employed to perform classification. We used proposed method for a MA dataset. The classification accuracy of the proposed method is 92.52 ± 5.38% and higher than other methods, which shows the rationality and superiority of the proposed method.

Application of bi-modal signal in the classification and recognition of drug addiction degree based on machine learning

Most studies on drug addiction degree are made based on statistical scales, addicts' account, and subjective judgement of rehabilitation doctors. No objective, quantified evaluation has been made. This paper uses devises the synchronous bimodal signal collection and experimentation paradigm with electroencephalogram (EEG) and forehead high-density near-infrared spectroscopy (NIRS) device. The drug addicts are classified into mild, moderate and severe groups with reference to the suggestions of researchers and medical experts. Data of 45 drug addicts (mild: 15; moderate: 15; and severe: 15) is collected, and then used to design an addiction degree testing algorithm based on decision fusion. The algorithm is used to classify mild, moderate and severe addiction. This paper pioneers to use two types of Convolutional Neural Network (CNN) to abstract the EEG and NIR data of drug addicts, and introduces batch normalization to CNN, thus accelerating training process, reducing parameter sensitivity, and enhancing system robustness. The characteristics output by two CNNs are transformed into dimensions. Two new characteristics are assigned with a weight of 50% each. The data is used for decision fusion. In the networks, 27 subjects are used as training sets, 9 as validation sets, and 9 as testing sets. The 3-class accuracy remains to be 63.15%, preliminarily justifying this method as an effective approach to measure drug addiction degree. And the method is ready to use, objective, and offers results in real time.

Bimodal Data Fusion of Simultaneous Measurements of EEG and fNIRS during Lower Limb Movements

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have temporal and spatial characteristics that may complement each other and, therefore, pose an intriguing approach for brain-computer interaction (BCI). In this work, the relationship between the hemodynamic response and brain oscillation activity was investigated using the concurrent recording of fNIRS and EEG during ankle joint movements. Twenty subjects participated in this experiment. The EEG was recorded using 20 electrodes and hemodynamic responses were recorded using 32 optodes positioned over the motor cortex areas. The event-related desynchronization (ERD) feature was extracted from the EEG signal in the alpha band (8-11) Hz, and the concentration change of the oxy-hemoglobin (oxyHb) was evaluated from the hemodynamics response. During the motor execution of the ankle joint movements, a decrease in the alpha (8-11) Hz amplitude (desynchronization) was found to be correlated with an increase of the oxyHb (r = -0.64061, p < 0.00001) observed on the Cz electrode and the average of the fNIRS channels (ch28, ch25, ch32, ch35) close to the foot area representation. Then, the correlated channels in both modalities were used for ankle joint movement classification. The result demonstrates that the integrated modality based on the correlated channels provides a substantial enhancement in ankle joint classification accuracy of 93.01 ± 5.60% (p < 0.01) compared with single modality. These results highlight the potential of the bimodal fNIR-EEG approach for the development of future BCI for lower limb rehabilitation.

Data Processing in Functional Near-Infrared Spectroscopy (fNIRS) Motor Control Research

FNIRS pre-processing and processing methodologies are very important-how a researcher chooses to process their data can change the outcome of an experiment. The purpose of this review is to provide a guide on fNIRS pre-processing and processing techniques pertinent to the field of human motor control research. One hundred and twenty-three articles were selected from the motor control field and were examined on the basis of their fNIRS pre-processing and processing methodologies. Information was gathered about the most frequently used techniques in the field, which included frequency cutoff filters, wavelet filters, smoothing filters, and the general linear model (GLM). We discuss the methodologies of and considerations for these frequently used techniques, as well as those for some alternative techniques. Additionally, general considerations for processing are discussed.

Multimodal fusion of EEG-fNIRS: a mutual information-based hybrid classification framework

Multimodal data fusion is one of the current primary neuroimaging research directions to overcome the fundamental limitations of individual modalities by exploiting complementary information from different modalities. Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are especially compelling modalities due to their potentially complementary features reflecting the electro-hemodynamic characteristics of neural responses. However, the current multimodal studies lack a comprehensive systematic approach to properly merge the complementary features from their multimodal data. Identifying a systematic approach to properly fuse EEG-fNIRS data and exploit their complementary potential is crucial in improving performance. This paper proposes a framework for classifying fused EEG-fNIRS data at the feature level, relying on a mutual information-based feature selection approach with respect to the complementarity between features. The goal is to optimize the complementarity, redundancy and relevance between multimodal features with respect to the class labels as belonging to a pathological condition or healthy control. Nine amyotrophic lateral sclerosis (ALS) patients and nine controls underwent multimodal data recording during a visuo-mental task. Multiple spectral and temporal features were extracted and fed to a feature selection algorithm followed by a classifier, which selected the optimized subset of features through a cross-validation process. The results demonstrated considerably improved hybrid classification performance compared to the individual modalities and compared to conventional classification without feature selection, suggesting a potential efficacy of our proposed framework for wider neuro-clinical applications.

Progress in Brain Computer Interface: Challenges and Opportunities

Brain computer interfaces (BCI) provide a direct communication link between the brain and a computer or other external devices. They offer an extended degree of freedom either by strengthening or by substituting human peripheral working capacity and have potential applications in various fields such as rehabilitation, affective computing, robotics, gaming, and neuroscience. Significant research efforts on a global scale have delivered common platforms for technology standardization and help tackle highly complex and non-linear brain dynamics and related feature extraction and classification challenges. Time-variant psycho-neurophysiological fluctuations and their impact on brain signals impose another challenge for BCI researchers to transform the technology from laboratory experiments to plug-and-play daily life. This review summarizes state-of-the-art progress in the BCI field over the last decades and highlights critical challenges.

Optical brain imaging and its application to neurofeedback

Besides passive recording of brain electric or magnetic activity, also non-ionizing electromagnetic or optical radiation can be used for real-time brain imaging. Here, changes in the radiation's absorption or scattering allow for continuous in vivo assessment of regional neurometabolic and neurovascular activity. Besides magnetic resonance imaging (MRI), over the last years, also functional near-infrared spectroscopy (fNIRS) was successfully established in real-time metabolic brain imaging. In contrast to MRI, fNIRS is portable and can be applied at bedside or in everyday life environments, e.g., to restore communication and movement. Here we provide a comprehensive overview of the history and state-of-the-art of real-time optical brain imaging with a special emphasis on its clinical use towards neurofeedback and brain-computer interface (BCI) applications. Besides pointing to the most critical challenges in clinical use, also novel approaches that combine real-time optical neuroimaging with other recording modalities (e.g. electro- or magnetoencephalography) are described, and their use in the context of neuroergonomics, neuroenhancement or neuroadaptive systems discussed.

Analysis of Human Gait Using Hybrid EEG-fNIRS-Based BCI System: A Review

Human gait is a complex activity that requires high coordination between the central nervous system, the limb, and the musculoskeletal system. More research is needed to understand the latter coordination's complexity in designing better and more effective rehabilitation strategies for gait disorders. Electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) are among the most used technologies for monitoring brain activities due to portability, non-invasiveness, and relatively low cost compared to others. Fusing EEG and fNIRS is a well-known and established methodology proven to enhance brain-computer interface (BCI) performance in terms of classification accuracy, number of control commands, and response time. Although there has been significant research exploring hybrid BCI (hBCI) involving both EEG and fNIRS for different types of tasks and human activities, human gait remains still underinvestigated. In this article, we aim to shed light on the recent development in the analysis of human gait using a hybrid EEG-fNIRS-based BCI system. The current review has followed guidelines of preferred reporting items for systematic reviews and meta-Analyses (PRISMA) during the data collection and selection phase. In this review, we put a particular focus on the commonly used signal processing and machine learning algorithms, as well as survey the potential applications of gait analysis. We distill some of the critical findings of this survey as follows. First, hardware specifications and experimental paradigms should be carefully considered because of their direct impact on the quality of gait assessment. Second, since both modalities, EEG and fNIRS, are sensitive to motion artifacts, instrumental, and physiological noises, there is a quest for more robust and sophisticated signal processing algorithms. Third, hybrid temporal and spatial features, obtained by virtue of fusing EEG and fNIRS and associated with cortical activation, can help better identify the correlation between brain activation and gait. In conclusion, hBCI (EEG + fNIRS) system is not yet much explored for the lower limb due to its complexity compared to the higher limb. Existing BCI systems for gait monitoring tend to only focus on one modality. We foresee a vast potential in adopting hBCI in gait analysis. Imminent technical breakthroughs are expected using hybrid EEG-fNIRS-based BCI for gait to control assistive devices and Monitor neuro-plasticity in neuro-rehabilitation. However, although those hybrid systems perform well in a controlled experimental environment when it comes to adopting them as a certified medical device in real-life clinical applications, there is still a long way to go.

Hybrid EEG-fNIRS BCI Fusion Using Multi-Resolution Singular Value Decomposition (MSVD)

Brain-computer interface (BCI) multi-modal fusion has the potential to generate multiple commands in a highly reliable manner by alleviating the drawbacks associated with single modality. In the present work, a hybrid EEG-fNIRS BCI system—achieved through a fusion of concurrently recorded electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals—is used to overcome the limitations of uni-modality and to achieve higher tasks classification. Although the hybrid approach enhances the performance of the system, the improvements are still modest due to the lack of availability of computational approaches to fuse the two modalities. To overcome this, a novel approach is proposed using Multi-resolution singular value decomposition (MSVD) to achieve system- and feature-based fusion. The two approaches based up different features set are compared using the KNN and Tree classifiers. The results obtained through multiple datasets show that the proposed approach can effectively fuse both modalities with improvement in the classification accuracy.

A Computationally Efficient Method for Hybrid EEG-fNIRS BCI Based on the Pearson Correlation

A hybrid brain computer interface (BCI) system considered here is a combination of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS). EEG-fNIRS signals are simultaneously recorded to achieve high motor imagery task classification. This integration helps to achieve better system performance, but at the cost of an increase in system complexity and computational time. In hybrid BCI studies, channel selection is recognized as the key element that directly affects the system's performance. In this paper, we propose a novel channel selection approach using the Pearson product-moment correlation coefficient, where only highly correlated channels are selected from each hemisphere. Then, four different statistical features are extracted, and their different combinations are used for the classification through KNN and Tree classifiers. As far as we know, there is no report available that explored the Pearson product-moment correlation coefficient for hybrid EEG-fNIRS BCI channel selection. The results demonstrate that our hybrid system significantly reduces computational burden while achieving a classification accuracy with high reliability comparable to the existing literature.

Enhanced Performance of a Brain Switch by Simultaneous Use of EEG and NIRS Data for Asynchronous Brain-Computer Interface

Previous studies have shown the superior performance of hybrid electroencephalography (EEG)/ near-infrared spectroscopy (NIRS) brain-computer interfaces (BCIs). However, it has been veiled whether the use of a hybrid EEG/NIRS modality can provide better performance for a brain switch that can detect the onset of the intention to turn on a BCI. In this study, we developed such a hybrid EEG/NIRS brain switch and compared its performance with single modality EEG- and NIRS-based brain switch respectively, in terms of true positive rate (TPR), false positive rate (FPR), onset detection time (ODT), and information transfer rate (ITR). In an offline analysis, the performance of a hybrid EEG/NIRS brain switch was significantly improved over that of EEG- and NIRS-based brain switches in general, and in particular a significantly lower FPR was observed for the hybrid EEG/NIRS brain switch. A pseudo-online analysis was additionally performed to confirm the feasibility of implementing an online BCI system with our hybrid EEG/NIRS brain switch. The overall trend of pseudo-online analysis results generally coincided with that of the offline analysis results. No significant difference in all performance measures was also found between offline and pseudo online analysis schemes when the amount of training data was same, with one exception for the ITRs of an EEG brain switch. These offline and pseudo-online results demonstrate that a hybrid EEG/NIRS brain switch can be used to provide a better onset detection performance than that of a single neuroimaging modality.

Discrimination of Two-Class Motor Imagery in a fNIRS Based Brain Computer Interface

The purpose of this study was to discriminate between left- and right-hand motor imagery tasks. We recorded the brain signals from two participants using a fNIRS system and compared different feature extraction (mean, peak, minimum, skewness and kurtosis) and classification techniques (linear (LDA) and quadratic discriminant analysis (QDA), support vector machine (SVM), logistic regression, K-nearest-neighbor (KNN), and neural networks with Levenberg-Marquardt (LMA), Bayesian Regularization (BRANN) and Scaled Conjugate Gradient (SCGA) training algorithms). The results showed poor classification accuracies (<; 58%) when skewness and kurtosis were used. When mean, peak, and minimum were used as features, QDA, SVM and KNN produced higher classification accuracies relative to LDA and logistic regression. Overall, BRANN led to the highest accuracies (>98%) when mean, peak and minimum were used as features.

On fractality of functional near-infrared spectroscopy signals: analysis and applications

Significance: The human brain is a highly complex system with nonlinear, dynamic behavior. A majority of brain imaging studies employing functional near-infrared spectroscopy (fNIRS), however, have considered only the spatial domain and have ignored the temporal properties of fNIRS recordings. Methods capable of revealing nonlinearities in fNIRS recordings can provide new insights about how the brain functions. Aim: The temporal characteristics of fNIRS signals are explored by comprehensively investigating their fractal properties. Approach: Fractality of fNIRS signals is analyzed using scaled windowed variance (SWV), as well as using visibility graph (VG), a method which converts a given time series into a graph. Additionally, the fractality of fNIRS signals obtained under resting-state and task-based conditions is compared, and the application of fractality in differentiating brain states is demonstrated for the first time via various classification approaches. Results: Results from SWV analysis show the existence of high fractality in fNIRS recordings. It is shown that differences in the temporal characteristics of fNIRS signals related to task-based and resting-state conditions can be revealed via the VGs constructed for each case. Conclusions: fNIRS recordings, regardless of the experimental conditions, exhibit high fractality. Furthermore, VG-based metrics can be employed to differentiate rest and task-execution brain states.