Emotion Recognition Based on Fusion of Local Cortical Activations and Dynamic Functional Networks Connectivity: An EEG Study

The effects of spatial leakage correction on the reliability of EEG-based functional connectivity networks

Electroencephalography (EEG) functional connectivity (FC) estimates are confounded by the volume conduction problem. This effect can be greatly reduced by applying FC measures insensitive to instantaneous, zero-lag dependencies (corrected measures). However, numerous studies showed that FC measures sensitive to volume conduction (uncorrected measures) exhibit higher reliability and higher subject-level identifiability. We tested how source reconstruction contributed to the reliability difference of EEG FC measures on a large (n = 201) resting-state data set testing eight FC measures (including corrected and uncorrected measures). We showed that the high reliability of uncorrected FC measures in resting state partly stems from source reconstruction: idiosyncratic noise patterns define a baseline resting-state functional network that explains a significant portion of the reliability of uncorrected FC measures. This effect remained valid for template head model-based, as well as individual head model-based source reconstruction. Based on our findings we made suggestions how to best use spatial leakage corrected and uncorrected FC measures depending on the main goals of the study.

Decoding emotion with phase-amplitude fusion features of EEG functional connectivity network

Decoding emotional neural representations from the electroencephalographic (EEG)-based functional connectivity network (FCN) is of great scientific importance for uncovering emotional cognition mechanisms and developing harmonious human-computer interactions. However, existing methods mainly rely on phase-based FCN measures (e.g., phase locking value [PLV]) to capture dynamic interactions between brain oscillations in emotional states, which fail to reflect the energy fluctuation of cortical oscillations over time. In this study, we initially examined the efficacy of amplitude-based functional networks (e.g., amplitude envelope correlation [AEC]) in representing emotional states. Subsequently, we proposed an efficient phase-amplitude fusion framework (PAF) to fuse PLV and AEC and used common spatial pattern (CSP) to extract fused spatial topological features from PAF for multi-class emotion recognition. We conducted extensive experiments on the DEAP and MAHNOB-HCI datasets. The results showed that: (1) AEC-derived discriminative spatial network topological features possess the ability to characterize emotional states, and the differential network patterns of AEC reflect dynamic interactions in brain regions associated with emotional cognition. (2) The proposed fusion features outperformed other state-of-the-art methods in terms of classification accuracy for both datasets. Moreover, the spatial filter learned from PAF is separable and interpretable, enabling a description of affective activation patterns from both phase and amplitude perspectives.

Improved emotion differentiation under reduced acoustic variability of speech in autism

BACKGROUND

Socio-emotional impairments are among the diagnostic criteria for autism spectrum disorder (ASD), but the actual knowledge has substantiated both altered and intact emotional prosodies recognition. Here, a Bayesian framework of perception is considered suggesting that the oversampling of sensory evidence would impair perception within highly variable environments. However, reliable hierarchical structures for spectral and temporal cues would foster emotion discrimination by autistics.

METHODS

Event-related spectral perturbations (ERSP) extracted from electroencephalographic (EEG) data indexed the perception of anger, disgust, fear, happiness, neutral, and sadness prosodies while listening to speech uttered by (a) human or (b) synthesized voices characterized by reduced volatility and variability of acoustic environments. The assessment of mechanisms for perception was extended to the visual domain by analyzing the behavioral accuracy within a non-social task in which dynamics of precision weighting between bottom-up evidence and top-down inferences were emphasized. Eighty children (mean 9.7 years old; standard deviation 1.8) volunteered including 40 autistics. The symptomatology was assessed at the time of the study via the Autism Diagnostic Observation Schedule, Second Edition, and parents' responses on the Autism Spectrum Rating Scales. A mixed within-between analysis of variance was conducted to assess the effects of group (autism versus typical development), voice, emotions, and interaction between factors. A Bayesian analysis was implemented to quantify the evidence in favor of the null hypothesis in case of non-significance. Post hoc comparisons were corrected for multiple testing.

RESULTS

Autistic children presented impaired emotion differentiation while listening to speech uttered by human voices, which was improved when the acoustic volatility and variability of voices were reduced. Divergent neural patterns were observed from neurotypicals to autistics, emphasizing different mechanisms for perception. Accordingly, behavioral measurements on the visual task were consistent with the over-precision ascribed to the environmental variability (sensory processing) that weakened performance. Unlike autistic children, neurotypicals could differentiate emotions induced by all voices.

CONCLUSIONS

This study outlines behavioral and neurophysiological mechanisms that underpin responses to sensory variability. Neurobiological insights into the processing of emotional prosodies emphasized the potential of acoustically modified emotional prosodies to improve emotion differentiation by autistics.

TRIAL REGISTRATION

BioMed Central ISRCTN Registry, ISRCTN18117434. Registered on September 20, 2020.

Mental Stress Classification Based on Selected Electroencephalography Channels Using Correlation Coefficient of Hjorth Parameters

Electroencephalography (EEG) signals offer invaluable insights into diverse activities of the human brain, including the intricate physiological and psychological responses associated with mental stress. A major challenge, however, is accurately identifying mental stress while mitigating the limitations associated with a large number of EEG channels. Such limitations encompass computational complexity, potential overfitting, and the prolonged setup time for electrode placement, all of which can hinder practical applications. To address these challenges, this study presents the novel CCHP method, aimed at identifying and ranking commonly optimal EEG channels based on their sensitivity to the mental stress state. This method's uniqueness lies in its ability not only to find common channels, but also to prioritize them according to their responsiveness to stress, ensuring consistency across subjects and making it potentially transformative for real-world applications. From our rigorous examinations, eight channels emerged as universally optimal in detecting stress variances across participants. Leveraging features from the time, frequency, and time-frequency domains of these channels, and employing machine learning algorithms, notably RLDA, SVM, and KNN, our approach achieved a remarkable accuracy of 81.56% with the SVM algorithm outperforming existing methodologies. The implications of this research are profound, offering a stepping stone toward the development of real-time stress detection devices, and consequently, enabling clinicians to make more informed therapeutic decisions based on comprehensive brain activity monitoring.

Machine learning for cognitive behavioral analysis: datasets, methods, paradigms, and research directions

Human behaviour reflects cognitive abilities. Human cognition is fundamentally linked to the different experiences or characteristics of consciousness/emotions, such as joy, grief, anger, etc., which assists in effective communication with others. Detection and differentiation between thoughts, feelings, and behaviours are paramount in learning to control our emotions and respond more effectively in stressful circumstances. The ability to perceive, analyse, process, interpret, remember, and retrieve information while making judgments to respond correctly is referred to as Cognitive Behavior. After making a significant mark in emotion analysis, deception detection is one of the key areas to connect human behaviour, mainly in the forensic domain. Detection of lies, deception, malicious intent, abnormal behaviour, emotions, stress, etc., have significant roles in advanced stages of behavioral science. Artificial Intelligence and Machine learning (AI/ML) has helped a great deal in pattern recognition, data extraction and analysis, and interpretations. The goal of using AI and ML in behavioral sciences is to infer human behaviour, mainly for mental health or forensic investigations. The presented work provides an extensive review of the research on cognitive behaviour analysis. A parametric study is presented based on different physical characteristics, emotional behaviours, data collection sensing mechanisms, unimodal and multimodal datasets, modelling AI/ML methods, challenges, and future research directions.

Classification of Mental Stress Levels using EEG Connectivity and Convolutional Neural Networks

Classifying mental stress is important as it helps in identifying the type and severity of stress, which can inform the most appropriate treatment or intervention. In this study, we propose utilizing electroencephalography (EEG) signals with convolutional neural networks (CNNs) to classify four mental states: rest, control-alert, stress and stress mitigation. The mental stress state was induced using Stroop color word test (SCWT) with time constrains and was then mitigated using 16 Hz Binaural beat stimulation (BBs). We quantified the four mental states using the reaction time (RT) to stimuli, accuracy of target detection, subjective score, and functional connectivity images of EEG estimated by Phase Locking Value (PLV). Our results show that, the SCWT reduced the accuracy of target detection by 70% with (F= 24.56, p = .00001), and the BBs improved the accuracy by 28% (F= 4.54, p = .00470). The functional connectivity network showed different patterns between the frontal/occipital and parietal regions, under the four mental states. The proposed CNNs with PLV images differentiated between the four mental states with highest classification performance at beta frequency band with 80.95% accuracy, 80.36% sensitivity, 94.75% specificity, 83.63% precision and 81.96% F-score. The overall results suggest that 16 Hz BBs can be used as an effective method to mitigate stress and the proposed CNNs with EEG-PLV images as a promising method for classifying different mental states.

Detection of Astronaut's Stress Levels During 240-Day Confinement using EEG Signals and Machine Learning. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023;2023:1-6. [PMID: 38082733 DOI: 10.1109/embc40787.2023.10340035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Early detection of mental stress is particularly important in prolonged space missions. In this study, we propose utilizing electroencephalography (EEG) with multiple machine learning models to detect elevated stress levels during a 240-day confinement. We quantified the levels of stress using alpha amylase levels, reaction time (RT) to stimuli, accuracy of target detection, and functional connectivity of EEG estimated by Phase Locking Value (PLV). Our results show that, alpha amylase level increased every 60-days (with 0.76 correlation) In-mission resulting in four elevated levels of stress. The RT and accuracy of target detection did not show any significant difference with time In-mission. The functional connectivity network showed different patterns between the frontal/occipital with other regions, and parietal to central region. The machine learning classifiers differentiate between four levels of stress with classification accuracy of 91.8%, 91.4%, 90.2%, 87.8, and 81% using linear discriminate analysis (LDA), Support Vector Machine (SVM), k-nearest neighbor (KNN), Naïve bayes (NB) and decision trees (DT). Our results suggest that EEG and machine learning can be used to detect elevated levels of mental stress in isolation and confined environments.

Stress management using fNIRS and binaural beats stimulation

In this study, we investigate the effectiveness of binaural beats stimulation (BBs) in enhancing cognitive vigilance and mitigating mental stress level at the workplace. We developed an experimental protocol under four cognitive conditions: high vigilance (HV), vigilance enhancement (VE), mental stress (MS) and stress mitigation (SM). The VE and SM conditions were achieved by listening to 16 Hz of BBs. We assessed the four cognitive conditions using salivary alpha-amylase, behavioral responses, and Functional Near-Infrared Spectroscopy (fNIRS). We quantified the vigilance and stress levels using the reaction time (RT) to stimuli, accuracy of detection, and the functional connectivity metrics of the fNIRS estimated by Phase Locking Values (PLV). We propose using the orthogonal minimum spanning tree (OMST) to determine the true connectivity network patterns of the PLV. Our results show that listening to 16-Hz BBs has significantly reduced the level of alpha amylase by 44%, reduced the RT to stimuli by 20% and increased the accuracy of target detection by 25%, (p < 0.001). The analysis of the connectivity network across the four different cognitive conditions revealed several statistically significant trends. Specifically, a significant increase in connectivity between the right and left dorsolateral prefrontal cortex (DLPFC) areas and left orbitofrontal cortex was found during the vigilance enhancement condition compared to the high vigilance. Likewise, similar patterns were found between the right and left DLPFC, orbitofrontal cortex, right ventrolateral prefrontal cortex (VLPFC) and right frontopolar PFC (prefrontal cortex) area during stress mitigation compared to mental stress. Furthermore, the connectivity network under stress condition alone showed significant connectivity increase between the VLPFC and DLPFC compared to other areas. The laterality index demonstrated left frontal laterality under high vigilance and VE conditions, and right DLPFC and left frontopolar PFC while under mental stress. Overall, our results showed that BBs can be used for vigilance enhancement and stress mitigation.

Evaluation of Brain Functional Connectivity from Electroencephalographic Signals Under Different Emotional States

The identification of the emotional states corresponding to the four quadrants of the valence/arousal space has been widely analyzed in the scientific literature by means of multiple techniques. Nevertheless, most of these methods were based on the assessment of each brain region separately, without considering the possible interactions among different areas. In order to study these interconnections, this study computes for the first time the functional connectivity metric called cross-sample entropy for the analysis of the brain synchronization in four groups of emotions from electroencephalographic signals. Outcomes reported a strong synchronization in the interconnections among central, parietal and occipital areas, while the interactions between left frontal and temporal structures with the rest of brain regions presented the lowest coordination. These differences were statistically significant for the four groups of emotions. All emotions were simultaneously classified with a 95.43% of accuracy, overcoming the results reported in previous studies. Moreover, the differences between high and low levels of valence and arousal, taking into account the state of the counterpart dimension, also provided notable findings about the degree of synchronization in the brain within different emotional conditions and the possible implications of these outcomes from a psychophysiological point of view.

Semi-supervised EEG Emotion Recognition Model Based on Enhanced Graph Fusion and GCN

OBJECTIVE

To take full advantage of both labeled data and unlabeled ones, the Graph Convolutional Network (GCN) was introduced in electroencephalography (EEG) based emotion recognition to achieve feature propagation. However, a single feature cannot represent the emotional state entirely and precisely due to the instability of the EEG signal and the complexity of the emotional state. In addition, the noise existing in the graph may affect the performance greatly. To solve these problems, it was necessary to introduce feature/similarity fusion and noise reduction strategies.

APPROACH

A semi-supervised EEG emotion recognition model combining graph fusion, network enhancement, and feature fusion was proposed. Firstly, different features were extracted from EEG and then compacted by Principal Component Analysis (PCA), respectively. Secondly, a Sample-by-sample Similarity Matrix (SSM) was constructed based on each feature, and Similarity Network Fusion (SNF) was adopted to fuse the graphs corresponding to different SSMs to take advantage of their complementarity. Then, Network Enhancement (NE) was performed on the fused graph to reduce the noise in it. Finally, GCN was performed on the concatenated features and the enhanced fused graph to achieve feature propagation.

MAIN RESULTS

Experimental results demonstrated that: i) When 5.30% of SEED and 7.20% of SEED-IV samples were chosen as the labeled samples, respectively, the minimum classification accuracy improvement achieved by the proposed scheme over state-of-the-art schemes were 1.52% on SEED and 13.14% on SEED-IV, respectively. ii) When 8.00% of SEED and 9.60% of SEED-IV samples were chosen as the labeled samples, respectively, the minimum training time reduction achieved by the proposed scheme over state-of-the-art schemes were 46.75s and 22.55s, respectively. iii) Graph fusion, network enhancement, and feature fusion all contributed to the performance enhancement. iv) The key hyperparameters that affect the performance were relatively few and easy to set to obtain outstanding performance.

SIGNIFICANCE

This paper demonstrated that the combination of graph fusion, network enhancement, and feature fusion help to enhance GCN-based EEG emotion recognition.

Automated emotion recognition: Current trends and future perspectives

BACKGROUND

Human emotions greatly affect the actions of a person. The automated emotion recognition has applications in multiple domains such as health care, e-learning, surveillance, etc. The development of computer-aided diagnosis (CAD) tools has led to the automated recognition of human emotions.

OBJECTIVE

This review paper provides an insight into various methods employed using electroencephalogram (EEG), facial, and speech signals coupled with multi-modal emotion recognition techniques. In this work, we have reviewed most of the state-of-the-art papers published on this topic.

METHOD

This study was carried out by considering the various emotion recognition (ER) models proposed between 2016 and 2021. The papers were analysed based on methods employed, classifier used and performance obtained.

RESULTS

There is a significant rise in the application of deep learning techniques for ER. They have been widely applied for EEG, speech, facial expression, and multimodal features to develop an accurate ER model.

CONCLUSION

Our study reveals that most of the proposed machine and deep learning-based systems have yielded good performances for automated ER in a controlled environment. However, there is a need to obtain high performance for ER even in an uncontrolled environment.

Investigating EEG-based functional connectivity patterns for multimodal emotion recognition

Objective. Previous studies on emotion recognition from electroencephalography (EEG) mainly rely on single-channel-based feature extraction methods, which ignore the functional connectivity between brain regions. Hence, in this paper, we propose a novel emotion-relevant critical subnetwork selection algorithm and investigate three EEG functional connectivity network features: strength, clustering coefficient, and eigenvector centrality. Approach. After constructing the brain networks by the correlations between pairs of EEG signals, we calculated critical subnetworks through the average of brain network matrices with the same emotion label to eliminate the weak associations. Then, three network features were conveyed to a multimodal emotion recognition model using deep canonical correlation analysis along with eye movement features. The discrimination ability of the EEG connectivity features in emotion recognition is evaluated on three public datasets: SEED, SEED-V, and DEAP. Main results. The experimental results reveal that the strength feature outperforms the state-of-the-art features based on single-channel analysis. The classification accuracies of multimodal emotion recognition are 95.08 ± 6.42 % on the SEED dataset, 84.51 ± 5.11 % on the SEED-V dataset, and 85.34 ± 2.90 % and 86.61 ± 3.76 % for arousal and valence on the DEAP dataset, respectively, which all achieved the best performance. In addition, the brain networks constructed with 18 channels achieve comparable performance with that of the 62-channel network and enable easier setups in real scenarios. Significance. The EEG functional connectivity networks combined with emotion-relevant critical subnetworks selection algorithm we proposed is a successful exploration to excavate the information between channels.

EEG Evidence of Altered Functional Connectivity and Microstate in Children Orphaned by HIV/AIDS

Children orphaned by HIV/AIDS ("AIDS orphans") suffer numerous early-life adverse events which have a long-lasting effect on brain function. Although previous studies found altered electroencephalography (EEG) oscillation during resting state in children orphaned by HIV/AIDS, data are limited regarding the alterations in connectivity and microstate. The current study aimed to investigate the functional connectivity (FC) and microstate in children orphaned by HIV/AIDS with resting-state EEG data. Data were recorded from 63 children orphaned by HIV/AIDS and 65 non-orphan controls during a close-eyes resting state. The differences in phase-locking value (PLV) of global average FC and temporal dynamics of microstate were compared between groups. For functional connectivity, children orphaned by HIV/AIDS showed decreased connectivity in alpha, beta, theta, and delta band compared with non-orphan controls. For microstate, EEG results demonstrated that children orphaned by HIV/AIDS show increased duration and coverage of microstate C, decreased occurrence and coverage of microstate B, and decreased occurrence of microstate D than non-orphan controls. These findings suggest that the microstate and functional connectivity has altered in children orphaned by HIV/AIDS compared with non-orphan controls and provide additional evidence that early life stress (ELS) would alter the structure and function of the brain and increase the risk of psychiatric disorders.

Enhancing EEG-Based Mental Stress State Recognition Using an Improved Hybrid Feature Selection Algorithm

In real-life applications, electroencephalogram (EEG) signals for mental stress recognition require a conventional wearable device. This, in turn, requires an efficient number of EEG channels and an optimal feature set. This study aims to identify an optimal feature subset that can discriminate mental stress states while enhancing the overall classification performance. We extracted multi-domain features within the time domain, frequency domain, time-frequency domain, and network connectivity features to form a prominent feature vector space for stress. We then proposed a hybrid feature selection (FS) method using minimum redundancy maximum relevance with particle swarm optimization and support vector machines (mRMR-PSO-SVM) to select the optimal feature subset. The performance of the proposed method is evaluated and verified using four datasets, namely EDMSS, DEAP, SEED, and EDPMSC. To further consolidate, the effectiveness of the proposed method is compared with that of the state-of-the-art metaheuristic methods. The proposed model significantly reduced the features vector space by an average of 70% compared with the state-of-the-art methods while significantly increasing overall detection performance.

Asymmetry of Regional Phase Synchrony Cortical Networks Under Cognitive Alertness and Vigilance Decrement States

This study investigates intra-regional connectivity and regional hemispheric asymmetry under two vigilance states: alertness and vigilance decrement. The vigilance states were induced on nine healthy subjects while performing 30 min in-congruent Stroop color-word task (I-SCWT). We measured brain activity using Electroencephalography (EEG) signals with 64-channels. We quantified the regional network connectivity using the phase-locking value (PLV) with graph theory analysis (GTA) and Support Vector Machines (SVM). Results showed that the vigilance decrement state was associated with impaired information processing within the frontal and central regions in delta and theta frequency bands. Meanwhile, the hemispheric asymmetry results showed that the laterality shifted to the right-temporal in delta, right-central, parietal, and left frontal in theta, right-frontal and left-central, temporal and parietal in alpha, and right-parietal and left temporal in beta frequency bands. These findings represent the first demonstration of intra-regional connectivity and hemispheric asymmetry changes as a function of cognitive vigilance states. The overall results showed that vigilance decrement is region and frequency band-specific. Our SVM model achieved the highest classification accuracy of 99.73% in differentiating between the two vigilance states based on the frontal and central connectivity networks measures.

EEG Mental Stress Assessment Using Hybrid Multi-Domain Feature Sets of Functional Connectivity Network and Time-Frequency Features

Exposure to mental stress for long period leads to serious accidents and health problems. To avoid negative consequences on health and safety, it is very important to detect mental stress at its early stages, i.e., when it is still limited to acute or episodic stress. In this study, we developed an experimental protocol to induce two different levels of stress by utilizing a mental arithmetic task with time pressure and negative feedback as the stressors. We assessed the levels of stress on 22 healthy subjects using frontal electroencephalogram (EEG) signals, salivary alpha-amylase level (AAL), and multiple machine learning (ML) classifiers. The EEG signals were analyzed using a fusion of functional connectivity networks estimated by the Phase Locking Value (PLV) and temporal and spectral domain features. A total of 210 different features were extracted from all domains. Only the optimum multi-domain features were used for classification. We then quantified stress levels using statistical analysis and seven ML classifiers. Our result showed that the AAL level was significantly increased (p < 0.01) under stress condition in all subjects. Likewise, the functional connectivity network demonstrated a significant decrease under stress, p < 0.05. Moreover, we achieved the highest stress classification accuracy of 93.2% using the Support Vector Machine (SVM) classifier. Other classifiers produced relatively similar results.

A Review on Mental Stress Assessment Methods Using EEG Signals

Mental stress is one of the serious factors that lead to many health problems. Scientists and physicians have developed various tools to assess the level of mental stress in its early stages. Several neuroimaging tools have been proposed in the literature to assess mental stress in the workplace. Electroencephalogram (EEG) signal is one important candidate because it contains rich information about mental states and condition. In this paper, we review the existing EEG signal analysis methods on the assessment of mental stress. The review highlights the critical differences between the research findings and argues that variations of the data analysis methods contribute to several contradictory results. The variations in results could be due to various factors including lack of standardized protocol, the brain region of interest, stressor type, experiment duration, proper EEG processing, feature extraction mechanism, and type of classifier. Therefore, the significant part related to mental stress recognition is choosing the most appropriate features. In particular, a complex and diverse range of EEG features, including time-varying, functional, and dynamic brain connections, requires integration of various methods to understand their associations with mental stress. Accordingly, the review suggests fusing the cortical activations with the connectivity network measures and deep learning approaches to improve the accuracy of mental stress level assessment.