Cross-task Cognitive Workload Recognition Based on EEG and Domain Adaptation

Cross-subject mental workload recognition using bi-classifier domain adversarial learning

To deploy Electroencephalogram (EEG) based Mental Workload Recognition (MWR) systems in the real world, it is crucial to develop general models that can be applied across subjects. Previous studies have utilized domain adaptation to mitigate inter-subject discrepancies in EEG data distributions. However, they have focused on reducing global domain discrepancy, while neglecting local workload-categorical domain divergence. This degrades the workload-discriminating ability of subject-invariant features. To deal with this problem, we propose a novel joint category-wise and domain-wise alignment Domain Adaptation (cdaDA) algorithm, using bi-classifier learning and domain discriminative adversarial learning. The bi-classifier learning approach is adopted to address the similarities and differences between categories, helping to align EEG data within the same mental workload categories. Additionally, the domain discriminative adversarial learning technique is adopted to consider global domain information by minimizing global domain discrepancy. By integrating both local category information and global domain information, the cdaDA model performs a coarse-to-fine alignment and achieves promising cross-subject MWR results.

Task-Independent Cognitive Workload Discrimination Based on EEG with Stacked Graph Attention Convolutional Networks

In the field of neuroeconomics, the assessment of cognitive workload is a crucial issue with significant implications for real-world applications. Previous research has made progress in task-based germane cognitive load classification, but decentralized studies focusing on task-independent assessment have often produced less than optimal results. In this study, we present a stacked graph attention convolutional networks (SGATCNs) model to tackle the challenges related to task-independent cognitive workload assessment using EEG spatial information. The model employs the differential entropy (DE) and power spectral density (PSD) features of each EEG channel across four frequency bands (delta, theta, alpha, and beta) as node information. For the construction of the network structure, phase-locked values (PLVs), phase-lag indices (PLIs), Pearson correlation coefficients (PCCs), and mutual information (MI) are utilized and evaluated to generate a functional brain network. Specifically, the model aggregates spatial information on the dynamic map by stacking the graph attention layers and utilizes the convolution module to extract the frequency domain information from between the networks under each frequency band. We conducted a cognitive workload experiment with 15 subjects and selected three representative psychological experimental task paradigms (N-back, mental arithmetic, and Sternberg) to induce different levels of cognitive workload (low, medium, and high). Our framework achieved an average accuracy of 65.11% in recognizing the task-independent cognitive workload across the three scenarios.

Cognitive load recognition in simulated flight missions: an EEG study

Cognitive load recognition (CLR) utilizing EEG signals has experienced significant advancement in recent years. However, current load-eliciting paradigms often rely on simplistic cognitive tasks such as arithmetic calculations, failing to adequately replicate real-world scenarios and lacking applicability. This study explores simulated flight missions over time to better reflect operational environments and investigate temporal dynamics of multiple load states. Thirty-six participants were recruited to perform simulated flight tasks with varying cognitive load levels of low, medium, and high. Throughout the experiments, we collected EEG load data from three sessions, pre- and post-task resting-state EEG data, subjective ratings, and objective performance metrics. Then, we employed several deep convolutional neural network (CNN) models, utilizing raw EEG data as model input, to assess cognitive load levels with six classification designs. Key findings from the study include (1) a notable distinction between resting-state and post-fatigue EEG data; (2) superior performance of shallow CNN models compared to more complex ones; and (3) temporal dynamics decline in CLR as the missions progressed. This paper establishes a potential foundation for assessing cognitive states during intricate simulated tasks across different individuals.

AdaptEEG: A Deep Subdomain Adaptation Network With Class Confusion Loss for Cross-Subject Mental Workload Classification

EEG signals exhibit non-stationary characteristics, particularly across different subjects, which presents significant challenges in the precise classification of mental workload levels when applying a trained model to new subjects. Domain adaptation techniques have shown effectiveness in enhancing the accuracy of cross-subject classification. However, current state-of-the-art methods for cross-subject mental workload classification primarily focus on global domain adaptation, which may lack fine-grained information and result in ambiguous classification boundaries. We proposed a novel approach called deep subdomain adaptation network with class confusion loss (DSAN-CCL) to enhance the performance of cross-subject mental workload classification. DSAN-CCL utilizes the local maximum mean discrepancy to align the feature distributions between the source domain and the target domain for each mental workload category. Moreover, the class confusion matrix was constructed by the product of the weighted class probabilities (class probabilities predicted by the label classifier) and the transpose of the class probabilities. The loss for maximizing diagonal elements and minimizing non-diagonal elements of the class confusion matrix was added to increase the credibility of pseudo-labels, thus improving the transfer performance. The proposed DSAN-CCL method was validated on two datasets, and the results indicate a significant improvement of 3∼10 percentage points compared to state-of-the-art domain adaptation methods. In addition, our proposed method is not dependent on a specific feature extractor. It can be replaced by any other feature extractor to fit new applications. This makes our approach universal to cross-domain classification problems.

Using Semi-Supervised Domain Adaptation to Enhance EEG-Based Cross-Task Mental Workload Classification Performance

Mental workload (MWL) assessment is critical for accident prevention and operator safety. However, achieving cross-task generalization of MWL classification models is a significant challenge for real-world applications. Classifiers trained on labeled samples from one task often experience a notable performance drop when directly applied to samples from other tasks, limiting its use cases. To address this issue, we propose a semi-supervised cross-task domain adaptation (SCDA) method using power spectral density (PSD) features for MWL recognition across tasks (MATB-II and n-back). Our results demonstrated that the SCDA method achieved the best cross-task classification performance on our data and COG-BCI public dataset, with accuracies of 90.98% ± 9.36% and 96.61% ± 4.35%, respectively. Furthermore, in the cross-task classification of cross-subject scenarios, SCDA showed the highest average accuracy (75.39% ± 9.56% on our data, 90.98% ± 9.36% on the COG-BCI public dataset). The findings indicate that the semi-supervised transfer learning approach using PSD features is feasible and effective for cross-task MWL assessment.

Avoidance of specific calibration sessions in motor intention recognition for exoskeleton-supported rehabilitation through transfer learning on EEG data

Exoskeleton-based support for patients requires the learning of individual machine-learning models to recognize movement intentions of patients based on the electroencephalogram (EEG). A major issue in EEG-based movement intention recognition is the long calibration time required to train a model. In this paper, we propose a transfer learning approach that eliminates the need for a calibration session. This approach is validated on healthy subjects in this study. We will use the proposed approach in our future rehabilitation application, where the movement intention of the affected arm of a patient can be inferred from the EEG data recorded during bilateral arm movements enabled by the exoskeleton mirroring arm movements from the unaffected to the affected arm. For the initial evaluation, we compared two trained models for predicting unilateral and bilateral movement intentions without applying a classifier transfer. For the main evaluation, we predicted unilateral movement intentions without a calibration session by transferring the classifier trained on data from bilateral movement intentions. Our results showed that the classification performance for the transfer case was comparable to that in the non-transfer case, even with only 4 or 8 EEG channels. Our results contribute to robotic rehabilitation by eliminating the need for a calibration session, since EEG data for training is recorded during the rehabilitation session, and only a small number of EEG channels are required for model training.

A Dual-Adversarial Model for Cross-Time and Cross-Subject Cognitive Workload Decoding

Electroencephalogram (EEG) signals are widely utilized in the field of cognitive workload decoding (CWD). However, when the recognition scenario is shifted from subject-dependent to subject-independent or spans a long period, the accuracy of CWD deteriorates significantly. Current solutions are either dependent on extensive training datasets or fail to maintain clear distinctions between categories, additionally lacking a robust feature extraction mechanism. In this paper, we tackle these issues by proposing a Bi-Classifier Joint Domain Adaptation (BCJDA) model for EEG-based cross-time and cross-subject CWD. Specifically, the model consists of a feature extractor, a domain discriminator, and a Bi-Classifier, containing two sets of adversarial processes for domain-wise alignment and class-wise alignment. In the adversarial domain adaptation, the feature extractor is forced to learn the common domain features deliberately. The Bi-Classifier also fosters the feature extractor to retain the category discrepancies of the unlabeled domain, so that its classification boundary is consistent with the labeled domain. Furthermore, different adversarial distance functions of the Bi-Classifier are adopted and evaluated in this model. We conduct classification experiments on a publicly available BCI competition dataset for recognizing low, medium, and high cognitive workload levels. The experimental results demonstrate that our proposed BCJDA model based on cross-gradient difference maximization achieves the best performance.

A fused multi-subfrequency bands and CBAM SSVEP-BCI classification method based on convolutional neural network

For the brain-computer interface (BCI) system based on steady-state visual evoked potential (SSVEP), it is difficult to obtain satisfactory classification performance for short-time window SSVEP signals by traditional methods. In this paper, a fused multi-subfrequency bands and convolutional block attention module (CBAM) classification method based on convolutional neural network (CBAM-CNN) is proposed for discerning SSVEP-BCI tasks. This method extracts multi-subfrequency bands SSVEP signals as the initial input of the network model, and then carries out feature fusion on all feature inputs. In addition, CBAM is embedded in both parts of the initial input and feature fusion for adaptive feature refinement. To verify the effectiveness of the proposed method, this study uses the datasets of Inner Mongolia University of Technology (IMUT) and Tsinghua University (THU) to evaluate the performance of the proposed method. The experimental results show that the highest accuracy of CBAM-CNN reaches 0.9813 percentage point (pp). Within 0.1-2 s time window, the accuracy of CBAM-CNN is 0.0201-0.5388 (pp) higher than that of CNN, CCA-CWT-SVM, CCA-SVM, CCA-GNB, FBCCA, and CCA. Especially in the short-time window range of 0.1-1 s, the performance advantage of CBAM-CNN is more significant. The maximum information transmission rate (ITR) of CBAM-CNN is 503.87 bit/min, which is 227.53 bit/min-503.41 bit/min higher than the above six EEG decoding methods. The study further results show that CBAM-CNN has potential application value in SSVEP decoding.

Granular estimation of user cognitive workload using multi-modal physiological sensors

Mental workload (MWL) is a crucial area of study due to its significant influence on task performance and potential for significant operator error. However, measuring MWL presents challenges, as it is a multi-dimensional construct. Previous research on MWL models has focused on differentiating between two to three levels. Nonetheless, tasks can vary widely in their complexity, and little is known about how subtle variations in task difficulty influence workload indicators. To address this, we conducted an experiment inducing MWL in up to 5 levels, hypothesizing that our multi-modal metrics would be able to distinguish between each MWL stage. We measured the induced workload using task performance, subjective assessment, and physiological metrics. Our simulated task was designed to induce diverse MWL degrees, including five different math and three different verbal tiers. Our findings indicate that all investigated metrics successfully differentiated between various MWL levels induced by different tiers of math problems. Notably, performance metrics emerged as the most effective assessment, being the only metric capable of distinguishing all the levels. Some limitations were observed in the granularity of subjective and physiological metrics. Specifically, the subjective overall mental workload couldn't distinguish lower levels of workload, while all physiological metrics could detect a shift from lower to higher levels, but did not distinguish between workload tiers at the higher or lower ends of the scale (e.g., between the easy and the easy-medium tiers). Despite these limitations, each pair of levels was effectively differentiated by one or more metrics. This suggests a promising avenue for future research, exploring the integration or combination of multiple metrics. The findings suggest that subtle differences in workload levels may be distinguishable using combinations of subjective and physiological metrics.

Multi-Action Knee Contact Force Prediction by Domain Adaptation

Most recent musculoskeletal dynamics estimation methods are designed for predefined actions, such as gait, and don't generalize to various tasks. In this work, we address the problem of estimating internal biomechanical forces during more than one actions by introducing unsupervised domain adaptation into a deep learning model. More specifically, we developed a Bidirectional Long Short-Term Memory network for knee contact force prediction, enhanced with correlation alignment layers, in order to minimize the domain shift between kinematic data from different actions. Furthermore, we used the novel Neural State Machine (NSM) as a simulation platform to test and visualize our model predictions in a wide range of trajectories adapted to different 3D scene geometries in real-time. We conducted multiple experiments, including comparison with previous models, model alignment across action classes and real-to-synthetic data alignment. The results showed that the proposed deep learning architecture with domain adaptation performs better than the benchmark in terms of NRMSE and t-test. Overall, our method is capable of predicting knee contact forces for more than one action classes using a single architecture and thereby opens the path for estimating internal forces for intermediate actions, while the knowledge of the hidden state of motion may be used to support personalized rehabilitation. Moreover, our model can be easily integrated into any human motion simulation environment, which shows its potential in enabling biomechanical analysis in an automated and computationally efficient way.

Cross-Subject Transfer Learning for Boosting Recognition Performance in SSVEP-Based BCIs

Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) have been substantially studied in recent years due to their fast communication rate and high signal-to-noise ratio. The transfer learning is typically utilized to improve the performance of SSVEP-based BCIs with auxiliary data from the source domain. This study proposed an inter-subject transfer learning method for enhancing SSVEP recognition performance through transferred templates and transferred spatial filters. In our method, the spatial filter was trained via multiple covariance maximization to extract SSVEP-related information. The relationships between the training trial, the individual template, and the artificially constructed reference are involved in the training process. The spatial filters are applied to the above templates to form two new transferred templates, and the transferred spatial filters are obtained accordingly via the least-square regression. The contribution scores of different source subjects can be calculated based on the distance between the source subject and the target subject. Finally, a four-dimensional feature vector is constructed for SSVEP detection. To demonstrate the effectiveness of the proposed method, a publicly available dataset and a self-collected dataset were employed for performance evaluation. The extensive experimental results validated the feasibility of the proposed method for improving SSVEP detection.

A Spiking Neural Network With Adaptive Graph Convolution and LSTM for EEG-Based Brain-Computer Interfaces

Electroencephalography (EEG) signals classification is essential for the brain-computer interface (BCI). Recently, energy-efficient spiking neural networks (SNNs) have shown great potential in EEG analysis due to their ability to capture the complex dynamic properties of biological neurons while also processing stimulus information through precisely timed spike trains. However, most existing methods do not effectively mine the specific spatial topology of EEG channels and temporal dependencies of the encoded EEG spikes. Moreover, most are designed for specific BCI tasks and lack some generality. Hence, this study presents a novel SNN model with the customized spike-based adaptive graph convolution and long short-term memory (LSTM), termed SGLNet, for EEG-based BCIs. Specifically, we first adopt a learnable spike encoder to convert the raw EEG signals into spike trains. Then, we tailor the concepts of the multi-head adaptive graph convolution to SNN so that it can make good use of the intrinsic spatial topology information among distinct EEG channels. Finally, we design the spike-based LSTM units to further capture the temporal dependencies of the spikes. We evaluate our proposed model on two publicly available datasets from two representative fields of BCI, notably emotion recognition, and motor imagery decoding. The empirical evaluations demonstrate that SGLNet consistently outperforms existing state-of-the-art EEG classification algorithms. This work provides a new perspective for exploring high-performance SNNs for future BCIs with rich spatiotemporal dynamics.

A Multi-Source Transfer Joint Matching Method for Inter-Subject Motor Imagery Decoding

Individual differences among different subjects pose a great challenge to motor imagery (MI) decoding. Multi-source transfer learning (MSTL) is one of the most promising ways to reduce individual differences, which can utilize rich information and align the data distribution among different subjects. However, most MSTL methods in MI-BCI combine all data in the source subjects into a single mixed domain, which will ignore the effect of important samples and the large differences in multiple source subjects. To address these issues, we introduce transfer joint matching and improve it to multi-source transfer joint matching (MSTJM) and weighted MSTJM (wMSTJM). Different from previous MSTL methods in MI, our methods align the data distribution for each pair of subjects, and then integrate the results by decision fusion. Besides that, we design an inter-subject MI decoding framework to verify the effectiveness of these two MSTL algorithms. It mainly consists of three modules: covariance matrix centroid alignment in the Riemannian space, source selection in the Euclidean space after tangent space mapping to reduce negative transfer and computation overhead, and further distribution alignment by MSTJM or wMSTJM. The superiority of this framework is verified on two common public MI datasets from BCI competition IV. The average classification accuracy of the MSTJM and wMSTJ methods outperformed other state-of-the-art methods by at least 4.24% and 2.62% respectively. It's promising to advance the practical applications of MI-BCI.

Predictions of task using neural modeling

Introduction

A well-designed brain-computer interface (BCI) can make accurate and reliable predictions of a user's state through the passive assessment of their brain activity; in turn, BCI can inform an adaptive system (such as artificial intelligence, or AI) to intelligently and optimally aid the user to maximize the human-machine team (HMT) performance. Various groupings of spectro-temporal neural features have shown to predict the same underlying cognitive state (e.g., workload) but vary in their accuracy to generalize across contexts, experimental manipulations, and beyond a single session. In our work we address an outstanding challenge in neuroergonomic research: we quantify if (how) identified neural features and a chosen modeling approach will generalize to various manipulations defined by the same underlying psychological construct, (multi)task cognitive workload.

Methods

To do this, we train and test 20 different support vector machine (SVM) models, each given a subset of neural features as recommended from previous research or matching the capabilities of commercial devices. We compute each model's accuracy to predict which (monitoring, communications, tracking) and how many (one, two, or three) task(s) were completed simultaneously. Additionally, we investigate machine learning model accuracy to predict task(s) within- vs. between-sessions, all at the individual-level.

Results

Our results indicate gamma activity across all recording locations consistently outperformed all other subsets from the full model. Our work demonstrates that modelers must consider multiple types of manipulations which may each influence a common underlying psychological construct.

Discussion

We offer a novel and practical modeling solution for system designers to predict task through brain activity and suggest next steps in expanding our framework to further contribute to research and development in the neuroergonomics community. Further, we quantified the cost in model accuracy should one choose to deploy our BCI approach using a mobile EEG-systems with fewer electrodes-a practical recommendation from our work.