Deep learning networks based decision fusion model of EEG and fNIRS for classification of cognitive tasks

Sg-snn: a self-organizing spiking neural network based on temporal information

Neurodynamic observations indicate that the cerebral cortex evolved by self-organizing into functional networks, These networks, or distributed clusters of regions, display various degrees of attention maps based on input. Traditionally, the study of network self-organization relies predominantly on static data, overlooking temporal information in dynamic neuromorphic data. This paper proposes Temporal Self-Organizing (TSO) method for neuromorphic data processing using a spiking neural network. The TSO method incorporates information from multiple time steps into the selection strategy of the Best Matching Unit (BMU) neurons. It enables the coupled BMUs to radiate the weight across the same layer of neurons, ultimately forming a hierarchical self-organizing topographic map of concern. Additionally, we simulate real neuronal dynamics, introduce a glial cell-mediated Glial-LIF (Leaky Integrate-and-fire) model, and adjust multiple levels of BMUs to optimize the attention topological map.Experiments demonstrate that the proposed Self-organizing Glial Spiking Neural Network (SG-SNN) can generate attention topographies for dynamic event data from coarse to fine. A heuristic method based on cognitive science effectively guides the network's distribution of excitatory regions. Furthermore, the SG-SNN shows improved accuracy on three standard neuromorphic datasets: DVS128-Gesture, CIFAR10-DVS, and N-Caltech 101, with accuracy improvements of 0.3%, 2.4%, and 0.54% respectively. Notably, the recognition accuracy on the DVS128-Gesture dataset reaches 99.3%, achieving state-of-the-art (SOTA) performance.

Neurophysiological data augmentation for EEG-fNIRS multimodal features based on a denoising diffusion probabilistic model

BACKGROUND AND OBJECTIVE

The hybrid brain computer interfaces (BCI) combining electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) have attracted extensive attention for overcoming the decoding limitations of the single-modality BCI. With the deepening application of deep learning approaches in BCI systems, its significant performance improvement has become apparent. However, the scarcity of brain signal data limits the performance of deep learning models.

METHODS

In this study, we proposed an EEG-fNIRS data augmentation framework based on the combination of denoising diffusion probabilistic model (DDPM) and adding Gaussian noise (EFDA-CDG), for enhancing the performance of hybrid BCI systems. Firstly, we unified the temporal and spatial dimensions of EEG and fNIRS by manually extracting features and spatial mapping interpolation to create EEG-fNIRS joint distribution samples. Then, the DDPM generative model was combined with the traditional method of adding Gaussian noise to provide richer training data for the classifier. Finally, we constructed a classification module that applies EEG feature attention and fNIRS terrain attention to improve classification accuracy.

RESULTS

In order to evaluate the effectiveness of EFDA-CDG framework, experiments were conducted and fully validated on three publicly available databases and one self-collected database. In the context of a participant-dependent training approach, our method achieves accuracy rates of 82.02% for motor imagery, 91.93% for mental arithmetic, and 90.54% for n-back tasks on public databases. Additionally, our method boasts an accuracy rate of 97.82% for drug addiction discrimination task on the self-collected database.

CONCLUSIONS

EFDA-CDG framework successfully facilitates data augmentation, thereby enhancing the performance of EEG-fNIRS hybrid BCI systems.

Data collection, enhancement, and classification of functional near-infrared spectroscopy motor execution and imagery

Recognition and execution of motor imagery play a key role in brain-computer interface (BCI) and are prerequisites for converting thoughts into executable instructions. However, to date, data acquired through commonly used electroencephalography (EEG) methods are very sensitive to motion interference, which will affect the accuracy of the data classification. The emerging functional near-infrared spectroscopy (fNIRS) technique, while overcoming the drawbacks of EEG's susceptibility to interference and difficulty in detecting motor signals, has less publicly available data. In this paper, we designed a motor execution and imagery experiment based on a wearable fNIRS device to acquire brain signals and proposed a modified Kolmogorov-Arnold network (named SE-KAN) for recognizing fNIRS signals corresponding to the task. Due to the small number of subjects in this experiment, the Wasserstein generative adversarial network was used to enhance the data processing. For the fNIRS data recognition task, the SE-KAN method achieved 96.36 ± 2.43% single-subject accuracy and 84.72 ± 3.27% cross-subject accuracy. It is believed that the dataset and method of this paper will help the development of BCI.

Enhancing learning experiences: EEG-based passive BCI system adapts learning speed to cognitive load in real-time, with motivation as catalyst

Computer-based learning has gained popularity in recent years, providing learners greater flexibility and freedom. However, these learning environments do not consider the learner's mental state in real-time, resulting in less optimized learning experiences. This research aimed to explore the effect on the learning experience of a novel EEG-based Brain-Computer Interface (BCI) that adjusts the speed of information presentation in real-time during a learning task according to the learner's cognitive load. We also explored how motivation moderated these effects. In accordance with three experimental groups (non-adaptive, adaptive, and adaptive with motivation), participants performed a calibration task (n-back), followed by a memory-based learning task concerning astrological constellations. Learning gains were assessed based on performance on the learning task. Self-perceived mental workload, cognitive absorption and satisfaction were assessed using a post-test questionnaire. Between-group analyses using Mann-Whitney tests suggested that combining BCI and motivational factors led to more significant learning gains and an improved learning experience. No significant difference existed between the BCI without motivational factor and regular non-adaptive interface for overall learning gains, self-perceived mental workload, and cognitive absorption. However, participants who undertook the experiment with an imposed learning pace reported higher overall satisfaction with their learning experience and a higher level of temporal stress. Our findings suggest BCI's potential applicability and feasibility in improving memorization-based learning experiences. Further work should seek to optimize the BCI adaptive index and explore generalizability to other learning contexts.

A working memory model based on recurrent neural networks using reinforcement learning

Numerous electrophysiological experiments have reported that the prefrontal cortex (PFC) is involved in the process of working memory. PFC neurons continue firing to maintain stimulus information in the delay period without external stimuli in working memory tasks. Further findings indicate that while the activity of single neurons exhibits strong temporal and spatial dynamics (heterogeneity), the activity of population neurons can encode spatiotemporal information of stimuli stably and reliably. From the perspective of neural networks, the computational mechanism underlying this phenomenon is not well demonstrated. The main purpose of this paper is to adopt a new strategy to explore the neural computation mechanism of working memory. We used reinforcement learning to train a recurrent neural network model to learn a spatial working memory task. The model is composed of a decision network and a baseline network. The decision network is responsible for updating strategies to make action choices, while the baseline network evaluates action choices to predict rewards. Simulated results demonstrate that the model can perform the spatial working memory task. The activity of the recurrent units has characteristics such as temporal dynamics and preferred direction selectivity, but their population activity encodes the stimulus information stably during the delay period in a low-dimensional subspace. These activity characteristics displayed by the model units are similar to those of PFC neurons observed in the same experiments. Meanwhile, as the network model continued learning the task, the temporal stability and spatial separability of the stimulus information encoded by the activity of model units in the low-dimensional subspace gradually strengthened, and the accuracy of the network's action choices also increased. In summary, this network model provides a new simulation method for spatial working memory tasks and a new perspective for understanding the characteristics of neuron activity in the PFC.

Classification algorithm for motor imagery fusing CNN and attentional mechanisms based on functional near-infrared spectroscopy brain image

With the continuing development of brain-computer interface technology, the analysis and interpretation of brain signals are becoming increasingly important. In the field of brain-computer interfaces, motor imagery (MI) is an important paradigm for generating specific brain signals through thought alone, rather than actual movement, for computer decoding. Functional near-infrared spectroscopy (fNIRS) imaging technology has been increasingly used in brain-computer interfaces due to its advantages of non-invasiveness, low resource requirements, low cost, and high spatial resolution. Scientists have done a lot of work in channel selection, feature selection, and then applying traditional machine learning methods for classification, but the results achieved so far are still insufficient to meet the conditions for realizing fNIRS brain-computer interfaces. To achieve a higher level of classification of fNIRS signals, we propose a method that fuses CNN and attention mechanisms to analyze the near-infrared signals of motor imagery and mental arithmetic data, which is fed into a neural network by deriving signals of changes in oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) concentrations through the modified Beer-Lambert law, and then applied to the fNIRS dataset of 29 healthy subjects to validate the proposed method. In the fNIRS-based BCI, the average classification accuracy of the MI signal from HbR and HbO reaches 85.92% and 86.21%, respectively, and the average classification accuracy of the MA signal reaches 89.66% and 88.79%, respectively. The advantage of our approach is that it is lightweight and improves the classification accuracy of current BCI fNIRS signals.

Enhancing Classification Accuracy with Integrated Contextual Gate Network: Deep Learning Approach for Functional Near-Infrared Spectroscopy Brain-Computer Interface Application

Brain-computer interface (BCI) systems include signal acquisition, preprocessing, feature extraction, classification, and an application phase. In fNIRS-BCI systems, deep learning (DL) algorithms play a crucial role in enhancing accuracy. Unlike traditional machine learning (ML) classifiers, DL algorithms eliminate the need for manual feature extraction. DL neural networks automatically extract hidden patterns/features within a dataset to classify the data. In this study, a hand-gripping (closing and opening) two-class motor activity dataset from twenty healthy participants is acquired, and an integrated contextual gate network (ICGN) algorithm (proposed) is applied to that dataset to enhance the classification accuracy. The proposed algorithm extracts the features from the filtered data and generates the patterns based on the information from the previous cells within the network. Accordingly, classification is performed based on the similar generated patterns within the dataset. The accuracy of the proposed algorithm is compared with the long short-term memory (LSTM) and bidirectional long short-term memory (Bi-LSTM). The proposed ICGN algorithm yielded a classification accuracy of 91.23 ± 1.60%, which is significantly (p < 0.025) higher than the 84.89 ± 3.91 and 88.82 ± 1.96 achieved by LSTM and Bi-LSTM, respectively. An open access, three-class (right- and left-hand finger tapping and dominant foot tapping) dataset of 30 subjects is used to validate the proposed algorithm. The results show that ICGN can be efficiently used for the classification of two- and three-class problems in fNIRS-based BCI applications.