Pattern Recognition of Cognitive Load Using EEG and ECG Signals

Study on incentive mechanism of reward and punishment on work efficiency of PCB welder based on recurrence quantification analysis and electroencephalogram signals

Traditional methods often struggle to objectively quantify the impact of salary incentives on employees' productivity, leaving enterprise incentive strategies without a solid scientific foundation. To address this issue, this study innovatively combines recurrence quantification analysis (RQA) with electroencephalogram (EEG) signals, proposing a dynamic incentive evaluation model based on the analysis of brain chaos characteristics. By comparing the EEG signals of workers with and without reward and punishment incentives (control group vs. experimental group), key features such as deterministic (DET) and average diagonal line length (DLL) are extracted to reveal how incentives regulate work efficiency. The experiment shows that RQA diagrams of workers' EEG under reward and punishment incentives exhibit significantly enhanced chaotic characteristics, with DET and DLL values decreasing by 13.3% and 10.4%, respectively. The accuracy of the twin support vector machine (TWSVM) reaches 98.71%, which is 0.79% and 14.37% higher than existing EEG-based incentive evaluation methods, such as the phase-locking value combined with convolutional neural network (accuracy: 97.92%) and spectral power features (accuracy: 84.34%). This study not only confirms the feasibility of EEG in incentive evaluation but also addresses the insufficient sensitivity of traditional cognitive load monitoring by integrating RQA features and a dynamic classification framework, providing a quantifiable neuroscientific basis for optimizing enterprise incentive mechanisms.

A Pilot Study on Video Game Training Effects on Visual Working Memory: Behavioral and Neural Insights

Background/Objectives: Recent research suggests that video games may serve as cognitive training tools to enhance visual working memory (VWM) capacity. However, the effectiveness of game-based cognitive training remains debated, and the underlying neural mechanisms, as well as the relationship between training efficacy and game design factors, are unclear. This study aimed to evaluate the impact of video game training on VWM capacity and explore its neural correlates. Methods: Two groups underwent 56 daily 20 min training sessions with two distinct video games targeting different cognitive skills: a reaction-time training game and a VWM-specific training game. Behavioral assessments included accuracy, hit response times, correct rejection response times, and Cowan's K values. Neural correlates were measured through Negative Slow Wave (NSW) activity using EEG. Decision tree classification analyses were applied to NSW data across sessions and set sizes to identify patterns linked to VWM capacity. Results: Preliminary results are that both groups showed improvements in behavioral measures (accuracy, response times, and Cowan's K values). NSW analyses revealed a main effect of set size in both groups, and classification results indicated that NSW patterns differed between groups, across sessions, and set sizes, supporting the relationship between NSW and VWM capacity. Conclusions: These findings contribute to understanding NSW as a neurophysiological correlate of VWM capacity, demonstrating its plasticity through video game training. Simple video games could effectively enhance behavioral and neural aspects of VWM, encouraging their potential as accessible cognitive training tools.

A cross-attention swin transformer network for EEG-based subject-independent cognitive load assessment

EEG signals play a crucial role in assessing cognitive load, which is a key element in ensuring the secure operation of human-computer interaction systems. However, the variability of EEG signals across different subjects poses a challenge in applying the pre-trained cognitive load assessment model to new subjects. Moreover, previous domain adaptation research has primarily focused on developing complex network architectures to learn more domain-invariant features, overlooking the noise introduced by pseudo-labels and the challenges posed by domain migration problems. Therefore, this study proposes a novel cross-attention swin-transformer network for cross-subject cognitive load assessment, which achieves inter-domain feature alignment through parameter sharing in cross attention mechanism without using pseudo-labels, and utilizes maximum mean discrepancy (MMD) to measure the difference between the feature distributions of the source and target domains, further promoting feature alignment between domains. This method aims to leverage the advantages of cross-attention mechanism and MMD to better mitigate individual differences among subjects in cross-subject cognitive workload assessment. To validate the classification performance of the proposed network, two datasets of image recognition task and N-back task were employed for testing. Results show that, the proposed model outperformed advanced methods with cross-subject classification results of 88.13% and 81.27% on the on local and public datasets. The ablation experiment results reveal that using either the cross-attention mechanism or the MMD strategy alone improves cross-subject classification performance by 2.11% and 2.95% on the local dataset, respectively. Furthermore, the results of the EEG features distribution differences between all subjects before and after network training showed a significant reduction in feature distribution differences between subjects, further confirming the network's effectiveness in minimizing inter-subject differences.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11571-024-10160-7.

Accurate Mental Stress Detection Using Sequential Backward Selection and Adaptive Synthetic Methods

The daily experience of mental stress profoundly influences our health and work performance while concurrently triggering alterations in brain electrical activity. Electroencephalogram (EEG) is a widely adopted method for assessing cognitive and affective states. This study delves into the EEG correlates of stress and the potential use of resting EEG in evaluating stress levels. Over 13 weeks, our longitudinal study focuses on the real-life experiences of college students, collecting data from each of the 18 participants across multiple days in classroom settings. To tackle the complexity arising from the multitude of EEG features and the imbalance in data samples across stress levels, we use the sequential backward selection (SBS) method for feature selection and the adaptive synthetic (ADASYN) sampling algorithm for imbalanced data. Our findings unveil that delta and theta features account for approximately 50% of the selected features through the SBS process. In leave-one-out (LOO) cross-validation, the combination of band power and pair-wise coherence (COH) achieves a maximum balanced accuracy of 94.8% in stress-level detection for the above daily stress dataset. Notably, using ADASYN and borderline synthesized minority over-sampling technique (borderline-SMOTE) methods enhances model accuracy compared to the traditional SMOTE approach. These results provide valuable insights into using EEG signals for assessing stress levels in real-life scenarios, shedding light on potential strategies for managing stress more effectively.

Multimodal Physiological Analysis of Impact of Emotion on Cognitive Control in VR

Cognitive control is often perplexing to elucidate and can be easily influenced by emotions. Understanding the individual cognitive control level is crucial for enhancing VR interaction and designing adaptive and self-correcting VR/AR applications. Emotions can reallocate processing resources and influence cognitive control performance. However, current research has primarily emphasized the impact of emotional valence on cognitive control tasks, neglecting emotional arousal. In this study, we comprehensively investigate the influence of emotions on cognitive control based on the arousal-valence model. A total of 26 participants are recruited, inducing emotions through VR videos with high ecological validity and then performing related cognitive control tasks. Leveraging physiological data including EEG, HRV, and EDA, we employ classification techniques such as SVM, KNN, and deep learning to categorize cognitive control levels. The experiment results demonstrate that high-arousal emotions significantly enhance users' cognitive control abilities. Utilizing complementary information among multi-modal physiological signal features, we achieve an accuracy of 84.52% in distinguishing between high and low cognitive control. Additionally, time-frequency analysis results confirm the existence of neural patterns related to cognitive control, contributing to a better understanding of the neural mechanisms underlying cognitive control in VR. Our research indicates that physiological signals measured from both the central and autonomic nervous systems can be employed for cognitive control classification, paving the way for novel approaches to improve VR/AR interactions.

Machine learning algorithms for detection of visuomotor neural control differences in individuals with PASC and ME

The COVID-19 pandemic has affected millions worldwide, giving rise to long-term symptoms known as post-acute sequelae of SARS-CoV-2 (PASC) infection, colloquially referred to as long COVID. With an increasing number of people experiencing these symptoms, early intervention is crucial. In this study, we introduce a novel method to detect the likelihood of PASC or Myalgic Encephalomyelitis (ME) using a wearable four-channel headband that collects Electroencephalogram (EEG) data. The raw EEG signals are processed using Continuous Wavelet Transform (CWT) to form a spectrogram-like matrix, which serves as input for various machine learning and deep learning models. We employ models such as CONVLSTM (Convolutional Long Short-Term Memory), CNN-LSTM, and Bi-LSTM (Bidirectional Long short-term memory). Additionally, we test the dataset on traditional machine learning models for comparative analysis. Our results show that the best-performing model, CNN-LSTM, achieved an accuracy of 83%. In addition to the original spectrogram data, we generated synthetic spectrograms using Wasserstein Generative Adversarial Networks (WGANs) to augment our dataset. These synthetic spectrograms contributed to the training phase, addressing challenges such as limited data volume and patient privacy. Impressively, the model trained on synthetic data achieved an average accuracy of 93%, significantly outperforming the original model. These results demonstrate the feasibility and effectiveness of our proposed method in detecting the effects of PASC and ME, paving the way for early identification and management of the condition. The proposed approach holds significant potential for various practical applications, particularly in the clinical domain. It can be utilized for evaluating the current condition of individuals with PASC or ME, and monitoring the recovery process of those with PASC, or the efficacy of any interventions in the PASC and ME populations. By implementing this technique, healthcare professionals can facilitate more effective management of chronic PASC or ME effects, ensuring timely intervention and improving the quality of life for those experiencing these conditions.

Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines

BACKGROUND AND OBJECTIVE

Mental workload (MWL) is a relevant construct involved in all cognitively demanding activities, and its assessment is an important goal in many research fields. This paper aims at evaluating the reproducibility and sensitivity of MWL assessment from EEG signals considering the effects of different electrode configurations and pre-processing pipelines (PPPs).

METHODS

Thirteen young healthy adults were enrolled and were asked to perform 45 min of Simon's task to elicit a cognitive demand. EEG data were collected using a 32-channel system with different electrode configurations (fronto-parietal; Fz and Pz; Cz) and analyzed using different PPPs, from the simplest bandpass filtering to the combination of filtering, Artifact Subspace Reconstruction (ASR) and Independent Component Analysis (ICA). The reproducibility of MWL indexes estimation and the sensitivity of their changes were assessed using Intraclass Correlation Coefficient and statistical analysis.

RESULTS

MWL assessed with different PPPs showed reliability ranging from good to very good in most of the electrode configurations (average consistency > 0.87 and average absolute agreement > 0.92). Larger fronto-parietal electrode configurations, albeit being more affected by the choice of PPPs, provide better sensitivity in the detection of MWL changes if compared to a single-electrode configuration (18 vs. 10 statistically significant differences detected, respectively).

CONCLUSIONS

The most complex PPPs have been proven to ensure good reliability (>0.90) and sensitivity in all experimental conditions. In conclusion, we propose to use at least a two-electrode configuration (Fz and Pz) and complex PPPs including at least the ICA algorithm (even better including ASR) to mitigate artifacts and obtain reliable and sensitive MWL assessment during cognitive tasks.

Feature Selection for Continuous within- and Cross-User EEG-Based Emotion Recognition

The monitoring of emotional state is important in the prevention and management of mental health problems and is increasingly being used to support affective computing. As such, researchers are exploring various modalities from which emotion can be inferred, such as through facial images or via electroencephalography (EEG) signals. Current research commonly investigates the performance of machine-learning-based emotion recognition systems by exposing users to stimuli that are assumed to elicit a single unchanging emotional response. Moreover, in order to demonstrate better results, many models are tested in evaluation frameworks that do not reflect realistic real-world implementations. Consequently, in this paper, we explore the design of EEG-based emotion recognition systems using longer, variable stimuli using the publicly available AMIGOS dataset. Feature engineering and selection results are evaluated across four different cross-validation frameworks, including versions of leave-one-movie-out (testing with a known user, but a previously unseen movie), leave-one-person-out (testing with a known movie, but a previously unseen person), and leave-one-person-and-movie-out (testing on both a new user and new movie). Results of feature selection lead to a 13% absolute improvement over comparable previously reported studies, and demonstrate the importance of evaluation framework on the design and performance of EEG-based emotion recognition systems.

Modeling Cognitive Load as a Self-Supervised Brain Rate with Electroencephalography and Deep Learning

The principal reason for measuring mental workload is to quantify the cognitive cost of performing tasks to predict human performance. Unfortunately, a method for assessing mental workload that has general applicability does not exist yet. This is due to the abundance of intuitions and several operational definitions from various fields that disagree about the sources or workload, its attributes, the mechanisms to aggregate these into a general model and their impact on human performance. This research built upon these issues and presents a novel method for mental workload modelling from EEG data employing deep learning. This method is self-supervised, employing a continuous brain rate, an index of cognitive activation, and does not require human declarative knowledge. The aim is to induce models automatically from data, supporting replicability, generalisability and applicability across fields and contexts. This specific method is a convolutional recurrent neural network trainable with spatially preserving spectral topographic head-maps from EEG data, aimed at fitting a novel brain rate variable. Findings demonstrate the capacity of the convolutional layers to learn meaningful high-level representations from EEG data since within-subject models had, on average, a test Mean Absolute Percentage Error of around 11%. The addition of a Long-Short Term Memory layer for handling sequences of high-level representations was not significant, although it did improve their accuracy. These findings point to the existence of quasi-stable blocks of automatically learnt high-level representations of cognitive activation because they can be induced through convolution and seem not to be dependent on each other over time, intuitively matching the non-stationary nature of brain responses. Additionally, across-subject models, induced with data from an increasing number of participants, thus trained with data containing more variability, obtained a similar accuracy to the within-subject models. This highlights the potential generalisability of the induced high-level representations across people, suggesting the existence of subject-independent cognitive activation patterns. This research contributes to the body of knowledge by providing scholars with a novel computational method for mental workload modelling that aims to be generally applicable and does not rely on ad hoc human crafted models.

Single feature spatio-temporal architecture for EEG Based cognitive load assessment

The study of electroencephalography (EEG) data for cognitive load analysis plays an important role in identification of stress-inducing tasks. This can be useful in applications such as optimal work allocation, increasing efficiency in the workplace and ensuring safety in difficult work environments. In order for such systems to be realistically deployable, easy acquisition and processing of the data on a wearable device is imperative. Current techniques primarily perform offline processing to analyse a multi-channel EEG to make a post facto assessment. This work focusses on building a new deep learning architecture that performs a single feature based spatio-temporal analysis of EEG data. This is achieved by creating a brain topographic map based on a single feature followed by spatio-temporal analysis using the developed network architecture. Data from two cognitive load experiments on the Physionet EEGMAT dataset were used to validate the performance. The network achieves an accuracy of 98.3% which is better than similar state-of-the-art approaches. Moreover, the proposed approach facilitates analysis of the spatial propagation of a signal, which is not possible through conventional EEG signal representations.

Assessment of mental fatigue and stress on electronic sport players with data fusion

Stress and mental fatigue are in existence constantly in daily life, and decrease our productivity while performing our daily routines. The purpose of this study was to analyze the states of stress and mental fatigue using data fusion while e-sport activity. In the study, ten volunteers performed e-sport duty which required both physical and mental effort and skills for 2 min. Volunteers' electroencephalogram (EEG), galvanic skin response (GSR), heart rate variability (HRV), and eye tracking data were obtained before and during game and then were analyzed. In addition, the effects of e-sports were evaluated with visual analogue scale and d2 attention tests. The d2 tests are performed after the game, and the game has a positive effect on attention and concentration. EEG from the frontal region indicates that the game is partly caused by stress and mental fatigue. HRV analysis showed that the sympathetic and vagal activities created by e-sports on people are different. By evaluating HRV and GSR together, it was seen that the emotional processes of the participants were stressed in some and excited in others. Data fusion can serve a variety of purposes such as determining the effect of e-sports activity on the person and the appropriate game type.

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.

Emotion Recognition Based on Skin Potential Signals with a Portable Wireless Device

Emotion recognition is of great importance for artificial intelligence, robots, and medicine etc. Although many techniques have been developed for emotion recognition, with certain successes, they rely heavily on complicated and expensive equipment. Skin potential (SP) has been recognized to be correlated with human emotions for a long time, but has been largely ignored due to the lack of systematic research. In this paper, we propose a single SP-signal-based method for emotion recognition. Firstly, we developed a portable wireless device to measure the SP signal between the middle finger and left wrist. Then, a video induction experiment was designed to stimulate four kinds of typical emotion (happiness, sadness, anger, fear) in 26 subjects. Based on the device and video induction, we obtained a dataset consisting of 397 emotion samples. We extracted 29 features from each of the emotion samples and used eight well-established algorithms to classify the four emotions based on these features. Experimental results show that the gradient-boosting decision tree (GBDT), logistic regression (LR) and random forest (RF) algorithms achieved the highest accuracy of 75%. The obtained accuracy is similar to, or even better than, that of other methods using multiple physiological signals. Our research demonstrates the feasibility of the SP signal’s integration into existing physiological signals for emotion recognition.