[Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals]

Emotion recognition refers to the process of determining and identifying an individual's current emotional state by analyzing various signals such as voice, facial expressions, and physiological indicators etc. Using electroencephalogram (EEG) signals and virtual reality (VR) technology for emotion recognition research helps to better understand human emotional changes, enabling applications in areas such as psychological therapy, education, and training to enhance people's quality of life. However, there is a lack of comprehensive review literature summarizing the combined researches of EEG signals and VR environments for emotion recognition. Therefore, this paper summarizes and synthesizes relevant research from the past five years. Firstly, it introduces the relevant theories of VR and EEG signal emotion recognition. Secondly, it focuses on the analysis of emotion induction, feature extraction, and classification methods in emotion recognition using EEG signals within VR environments. The article concludes by summarizing the research's application directions and providing an outlook on future development trends, aiming to serve as a reference for researchers in related fields.

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This article introduces a novel emotion recognition framework, primarily targeting the analysis of users' psychological reactions and stimuli. It is important to note that the stimuli eliciting emotional responses are as critical as the responses themselves. Hence, our approach synergizes stimulus data with physical and physiological signals, pioneering a multimodal method for emotional cognition. Our proposed framework unites stimulus source data with physiological signals, aiming to enhance the accuracy and robustness of emotion recognition through data integration. We initiated an emotional cognition experiment to gather EEG and ET data alongside recording emotional responses. Building on this, we developed the Emotion-Multimodal Fusion Neural Network (E-MFNN), optimized for multimodal data fusion to process both stimulus and physiological data. We conducted extensive comparisons between our framework's outcomes and those from existing models, also assessing various algorithmic approaches within our framework. This comparison underscores our framework's efficacy in multimodal emotion recognition. The source code is publicly available at https://figshare.com/s/8833d837871c78542b29.

Duration of face mask exposure matters: evidence from Swiss and Brazilian kindergartners' ability to recognise emotions

Wearing facial masks became a common practice worldwide during the COVID-19 pandemic. This study investigated (1) whether facial masks that cover adult faces affect 4- to 6-year-old children's recognition of emotions in those faces and (2) whether the duration of children's exposure to masks is associated with emotion recognition. We tested children from Switzerland (N = 38) and Brazil (N = 41). Brazil represented longer mask exposure due to a stricter mandate during COVID-19. Children had to choose a face displaying a specific emotion (happy, angry, or sad) when the face wore either no cover, a facial mask, or sunglasses. The longer hours of mask exposure were associated with better emotion recognition. Controlling for the hours of exposure, children were less likely to recognise emotions in partially hideen faces. Moreover, Brazilian children were more accurate in recognising happy faces than Swiss children. Overall, facial masks may negatively impact children's emotion recognition. However, prolonged exposure appears to buffer the lack of facial cues from the nose and mouth. In conclusion, restricting facial cues due to masks may impair kindergarten children's emotion recognition in the short run. However, it may facilitate their broader reading of facial emotional cues in the long run.

Supraphysiological testosterone levels from anabolic steroid use and reduced sensitivity to negative facial expressions in men

RATIONALE

Anabolic-androgenic steroids (AAS) are used to improve physical performance and appearance, but have been associated with deficits in social cognitive functioning. Approximately 30% of people who use AAS develop a dependence, increasing the risk for undesired effects.

OBJECTIVES

To assess the relationship between AAS use (current/previous), AAS dependence, and the ability to recognize emotional facial expressions, and investigate the potential mediating role of hormone levels.

METHODS

In total 156 male weightlifters, including those with current (n = 45) or previous (n = 34) AAS use and never-using controls (n = 77), completed a facial Emotion Recognition Task (ERT). Participants were presented with faces expressing one out of six emotions (sadness, happiness, fear, anger, disgust, and surprise) and were instructed to indicate which of the six emotions each face displayed. ERT accuracy and response time were recorded and evaluated for association with AAS use status, AAS dependence, and serum reproductive hormone levels. Mediation models were used to evaluate the mediating role of androgens in the relationship between AAS use and ERT performance.

RESULTS

Compared to never-using controls, men currently using AAS exhibited lower recognition accuracy for facial emotional expressions, particularly anger (Cohen's d = -0.57, pFDR = 0.03) and disgust (d = -0.51, pFDR = 0.05). Those with AAS dependence (n = 47) demonstrated worse recognition of fear relative to men without dependence (d = 0.58, p = 0.03). Recognition of disgust was negatively correlated with serum free testosterone index (FTI); however, FTI did not significantly mediate the association between AAS use and recognition of disgust.

CONCLUSIONS

Our findings demonstrate impaired facial emotion recognition among men currently using AAS compared to controls. While further studies are needed to investigate potential mechanisms, our analysis did not support a simple mediation effect of serum FTI.

EEG-based emotion recognition using a temporal-difference minimizing neural network

Electroencephalogram (EEG) emotion recognition plays an important role in human-computer interaction. An increasing number of algorithms for emotion recognition have been proposed recently. However, it is still challenging to make efficient use of emotional activity knowledge. In this paper, based on prior knowledge that emotion varies slowly across time, we propose a temporal-difference minimizing neural network (TDMNN) for EEG emotion recognition. We use maximum mean discrepancy (MMD) technology to evaluate the difference in EEG features across time and minimize the difference by a multibranch convolutional recurrent network. State-of-the-art performances are achieved using the proposed method on the SEED, SEED-IV, DEAP and DREAMER datasets, demonstrating the effectiveness of including prior knowledge in EEG emotion recognition.

Decoding emotion with phase-amplitude fusion features of EEG functional connectivity network

Decoding emotional neural representations from the electroencephalographic (EEG)-based functional connectivity network (FCN) is of great scientific importance for uncovering emotional cognition mechanisms and developing harmonious human-computer interactions. However, existing methods mainly rely on phase-based FCN measures (e.g., phase locking value [PLV]) to capture dynamic interactions between brain oscillations in emotional states, which fail to reflect the energy fluctuation of cortical oscillations over time. In this study, we initially examined the efficacy of amplitude-based functional networks (e.g., amplitude envelope correlation [AEC]) in representing emotional states. Subsequently, we proposed an efficient phase-amplitude fusion framework (PAF) to fuse PLV and AEC and used common spatial pattern (CSP) to extract fused spatial topological features from PAF for multi-class emotion recognition. We conducted extensive experiments on the DEAP and MAHNOB-HCI datasets. The results showed that: (1) AEC-derived discriminative spatial network topological features possess the ability to characterize emotional states, and the differential network patterns of AEC reflect dynamic interactions in brain regions associated with emotional cognition. (2) The proposed fusion features outperformed other state-of-the-art methods in terms of classification accuracy for both datasets. Moreover, the spatial filter learned from PAF is separable and interpretable, enabling a description of affective activation patterns from both phase and amplitude perspectives.

Assessing multimodal emotion recognition in multiple sclerosis with a clinically accessible measure

BACKGROUND

Multiple sclerosis (MS) negatively impacts cognition and has been associated with deficits in social cognition, including emotion recognition. There is a lack of research examining emotion recognition from multiple modalities in MS. The present study aimed to employ a clinically available measure to assess multimodal emotion recognition abilities among individuals with MS.

METHOD

Thirty-one people with MS and 21 control participants completed the Advanced Clinical Solutions Social Perceptions Subtest (ACS-SP), BICAMS, and measures of premorbid functioning, mood, and fatigue. ANCOVAs examined group differences in all outcomes while controlling for education. Correlational analyses examined potential correlates of emotion recognition in both groups.

RESULTS

The MS group performed significantly worse on the ACS-SP than the control group, F(1, 49) = 5.32, p = .025. Significant relationships between emotion recognition and cognitive functions were found only in the MS group, namely for information processing speed (r = 0.59, p < .001), verbal learning (r = 0.52, p = .003) and memory (r = 0.65, p < 0.001), and visuospatial learning (r = 0.62, p < 0.001) and memory (r = 0.52, p = .003). Emotion recognition did not correlate with premorbid functioning, mood, or fatigue in either group.

CONCLUSIONS

This study was the first to employ the ACS-SP to assess emotion recognition in MS. The results suggest that emotion recognition is impacted in MS and is related to other cognitive processes, such as information processing speed. The results provide information for clinicians amidst calls to include social cognition measures in standard MS assessments.

A novel transformer autoencoder for multi-modal emotion recognition with incomplete data

Multi-modal signals have become essential data for emotion recognition since they can represent emotions more comprehensively. However, in real-world environments, it is often impossible to acquire complete data on multi-modal signals, and the problem of missing modalities causes severe performance degradation in emotion recognition. Therefore, this paper represents the first attempt to use a transformer-based architecture, aiming to fill the modality-incomplete data from partially observed data for multi-modal emotion recognition (MER). Concretely, this paper proposes a novel unified model called transformer autoencoder (TAE), comprising a modality-specific hybrid transformer encoder, an inter-modality transformer encoder, and a convolutional decoder. The modality-specific hybrid transformer encoder bridges a convolutional encoder and a transformer encoder, allowing the encoder to learn local and global context information within each particular modality. The inter-modality transformer encoder builds and aligns global cross-modal correlations and models long-range contextual information with different modalities. The convolutional decoder decodes the encoding features to produce more precise recognition. Besides, a regularization term is introduced into the convolutional decoder to force the decoder to fully leverage the complete and incomplete data for emotional recognition of missing data. 96.33%, 95.64%, and 92.69% accuracies are attained on the available data of the DEAP and SEED-IV datasets, and 93.25%, 92.23%, and 81.76% accuracies are obtained on the missing data. Particularly, the model acquires a 5.61% advantage with 70% missing data, demonstrating that the model outperforms some state-of-the-art approaches in incomplete multi-modal learning.

Suicide-related cognitions and emotional bias performance in a community sample

BACKGROUND

Suicide is theorized to be connected to social interactions and feelings of belongingness. Those with suicide-related cognitions (SRCs) demonstrate attentional bias toward negative or suicide-related words, which can lead to increased feelings of rejection or alienation. As social interactions employ both verbal and nonverbal cues, there exists a gap in understanding how perception of emotional expressions can contribute to the development or exacerbation of suicidal ideation.

METHODS

The current sample (N = 114, 60.5 % female, 74.6 % white) completed the Suicide Cognitions Scale-Revised (SCS-R) and Patient Health Questionnaire (PHQ-9) to assess SRCs and depression severity. The Emotional Bias Task (EBT) was used to assess emotional response latency.

RESULTS

Multiple regression analyses on EBT results showed that endorsement of SRCs and depression severity were not associated with any particular emotional response bias. However, presence of SRCs showed an association with longer latencies to identify ambiguous emotional expressions, even when controlling for depressive symptoms and age LIMITATIONS: Measures were self-completed online. Relative homogeneity of the sample and cross-sectional design limits interpretation of the results.

CONCLUSIONS

Those with more severe SRCs take longer to recognize positive, nonverbal cues. Irregular processing of positive emotional stimuli combined with bias toward negative verbal cues could worsen feelings of rejection or alienation in social interactions, therefore increasing risk of developing SI. This suggests that interventions focusing on allocation of attentional resources to process positive social cues may be beneficial for those with SRCs to reduce severity and risk of suicide.

Relationship between neuroimaging and emotion recognition in mild cognitive impairment patients

OBJECTIVE

Dementia is a major public health problem with high needs for early detection, efficient treatment, and prognosis evaluation. Social cognition impairment could be an early dementia indicator and can be assessed with emotion recognition evaluation tests. The purpose of this study is to investigate the link between different brain imaging modalities and cognitive status in Mild Cognitive Impairment (MCI) patients, with the goal of uncovering potential physiopathological mechanisms based on social cognition performance.

METHODS

The relationship between the Reading the Mind in the Eyes Test (RMET) and some clinical and biochemical variables ([18 F]FDG PET-CT and anatomical MR parameters, neuropsychological evaluation, and CSF biomarkers) was studied in 166 patients with MCI by using a correlational approach.

RESULTS

The RMET correlated with neuropsychological variables, as well as with structural and functional brain parameters obtained from the MR and FDG-PET imaging evaluation. However, significant correlations between the RMET and CSF biomarkers were not found.

DISCUSSION

Different neuroimaging parameters were found to be related to an emotion recognition task in MCI. This analysis identified potential minimally-invasive biomarkers providing some knowledge about the physiopathological mechanisms in MCI.

The role of emotion recognition in the intergenerational transmission of child maltreatment: A multigenerational family study

BACKGROUND

Understanding how child maltreatment is passed down from one generation to the next is crucial for the development of intervention and prevention strategies that may break the cycle of child maltreatment. Changes in emotion recognition due to childhood maltreatment have repeatedly been found, and may underly the intergenerational transmission of child maltreatment.

OBJECTIVE

In this study we, therefore, examined whether the ability to recognize emotions plays a role in the intergenerational transmission of child abuse and neglect.

PARTICIPANTS AND SETTING

A total of 250 parents (104 males, 146 females) were included that participated in a three-generation family study.

METHOD

Participants completed an emotion recognition task in which they were presented with series of photographs that depicted the unfolding of facial expressions from neutrality to the peak emotions anger, fear, happiness, and sadness. Multi-informant measures were used to examine experienced and perpetrated child maltreatment.

RESULTS

A history of abuse, but not neglect, predicted a shorter reaction time to identify fear and anger. In addition, parents who showed higher levels of neglectful behavior made more errors in identifying fear, whereas parents who showed higher levels of abusive behavior made more errors in identifying anger. Emotion recognition did not mediate the association between experienced and perpetrated child maltreatment.

CONCLUSIONS

Findings highlight the importance of distinguishing between abuse and neglect when investigating the precursors and sequalae of child maltreatment. In addition, the effectiveness of interventions that aim to break the cycle of abuse and neglect could be improved by better addressing the specific problems with emotion processing of abusive and neglectful parents.

MGFKD: A semi-supervised multi-source domain adaptation algorithm for cross-subject EEG emotion recognition

Currently, most models rarely consider the negative transfer problem in the research field of cross-subject EEG emotion recognition. To solve this problem, this paper proposes a semi-supervised domain adaptive algorithm based on few labeled samples of target subject, which called multi-domain geodesic flow kernel dynamic distribution alignment (MGFKD). It consists of three modules: 1) GFK common feature extractor: projects the feature distribution of source and target subjects to the Grassmann manifold space, and obtains the latent common features of the two feature distributions through GFK method. 2) Source domain selector: obtains pseudo-labels of the target subject through weak classifier, finds "golden source subjects" by using few known labels of target subjects. 3) Label corrector: uses a dynamic distribution balance strategy to correct the pseudo-labels of the target subject. We conducted comparison experiments on the SEED and SEED-IV datasets, and the results show that MGFKD outperforms unsupervised and semi-supervised domain adaptation algorithms, achieving an average accuracy of 87.51±7.68% and 68.79±8.25% on the SEED and SEED-IV datasets with only one labeled sample per video for target subject. Especially when the number of source domains is set as 6 and the number of known labels is set as 5, the accuracy increase to 90.20±7.57% and 69.99±7.38%, respectively. The above results prove that our proposed algorithm can efficiently and quickly improve the cross-subject EEG emotion classification performance. Since it only need a small number of labeled samples of new subjects, making it has strong application value in future EEG-based emotion recognition applications.

Emotion Recognition Deficits in Children and Adolescents with Psychopathic Traits: A Systematic Review

Children and adolescents with psychopathic traits show deficits in emotion recognition, but there is no consensus as to the extent of their generalizability or about the variables that may be moderating the process. The present Systematic Review brings together the existing scientific corpus on the subject and attempts to answer these questions through an exhaustive review of the existing literature according to PRISMA 2020 statement. Results confirmed the existence of pervasive deficits in emotion recognition and, more specifically, on distress emotions (e.g., fear), a deficit that transcends all modalities of emotion presentation and all emotional stimuli used. Moreover, they supported the key role of attention to relevant areas that provide emotional cues (e.g., eye-region) and point out differences according to the presence of disruptive behavior and based on the psychopathy dimension examined. This evidence could advance the current knowledge on developmental models of psychopathic traits. Yet, homogenization of the conditions of research in this area should be prioritized to be able to draw more robust and generalizable conclusions.

The pictorial set of Emotional Social Interactive Scenarios between Chinese Adults (ESISCA): Development and validation

Affective picture databases with a single facial expression or body posture in one image have been widely applied to investigate emotion. However, to date, there was no standardized database containing the stimuli which involve multiple emotional signals in social interactive scenarios. The current study thus developed a pictorial set comprising 274 images depicting two Chinese adults' interactive scenarios conveying emotions of happiness, anger, sadness, fear, disgust, and neutral. The data of the valence and arousal ratings of the scenes and the emotional categories of the scenes and the faces in the images were provided in the present study. Analyses of the data collected from 70 undergraduate students suggested high reliabilities of the valence and arousal ratings of the scenes and high judgmental agreements in categorizing the scene and facial emotions. The findings suggested that the present dataset is well constructed and could be useful for future studies to investigate the emotion recognition or empathy in social interactions in both healthy and clinical (e.g., ASD) populations.

[Research on emotion recognition method based on IWOA-ELM algorithm for electroencephalogram]

Emotion is a crucial physiological attribute in humans, and emotion recognition technology can significantly assist individuals in self-awareness. Addressing the challenge of significant differences in electroencephalogram (EEG) signals among different subjects, we introduce a novel mechanism in the traditional whale optimization algorithm (WOA) to expedite the optimization and convergence of the algorithm. Furthermore, the improved whale optimization algorithm (IWOA) was applied to search for the optimal training solution in the extreme learning machine (ELM) model, encompassing the best feature set, training parameters, and EEG channels. By testing 24 common EEG emotion features, we concluded that optimal EEG emotion features exhibited a certain level of specificity while also demonstrating some commonality among subjects. The proposed method achieved an average recognition accuracy of 92.19% in EEG emotion recognition, significantly reducing the manual tuning workload and offering higher accuracy with shorter training times compared to the control method. It outperformed existing methods, providing a superior performance and introducing a novel perspective for decoding EEG signals, thereby contributing to the field of emotion research from EEG signal.

Social anxiety is associated with personal distress and disrupted recognition of negative emotions

Past research investigating the relation between social anxiety (SA), empathy and emotion recognition is marked by conceptual and methodological issues. In the present study, we aim to overcome these limitations by examining whether individuals with high (n = 40) vs. low (n = 43) social anxiety differed across these two facets of empathy and whether this could be related to their recognition of emotions. We employed a naturalistic emotion recognition paradigm in which participants watched short videos of individuals (targets) sharing authentic emotional experiences. After each video, we measured self-reported empathic concern and distress, as well as their ability to recognize the emotions expressed by the targets in the videos. Our results show that individuals with high social anxiety recognized the targets' emotions less accurately. Furthermore, high socially anxious individuals reported more personal distress than low socially anxious individuals, whereas no significant difference was found for empathic concern. The findings suggest that reduced recognition of emotions among SA individuals can be better explained by the negative effects of social stress than by a general deficit in empathy.

Research on driver's anger recognition method based on multimodal data fusion

OBJECTIVES

This paper aims to address the challenge of low accuracy in single-modal driver anger recognition by introducing a multimodal driver anger recognition model. The primary objective is to develop a multimodal fusion recognition method for identifying driver anger, focusing on electrocardiographic (ECG) signals and driving behavior signals.

METHODS

Emotion-inducing experiments were performed employing a driving simulator to capture both ECG signals and driving behavioral signals from drivers experiencing both angry and calm moods. An analysis of characteristic relationships and feature extraction was conducted on ECG signals and driving behavior signals related to driving anger. Seventeen effective feature indicators for recognizing driving anger were chosen to construct a dataset for driver anger. A binary classification model for recognizing driving anger was developed utilizing the Support Vector Machine (SVM) algorithm.

RESULTS

Multimodal fusion demonstrated significant advantages over single-modal approaches in emotion recognition. The SVM-DS model using decision-level fusion had the highest accuracy of 84.75%. Compared with the driver anger emotion recognition model based on unimodal ECG features, unimodal driving behavior features, and multimodal feature layer fusion, the accuracy increased by 9.10%, 4.15%, and 0.8%, respectively.

CONCLUSIONS

The proposed multimodal recognition model, incorporating ECG and driving behavior signals, effectively identifies driving anger. The research results provide theoretical and technical support for the establishment of a driver anger system.

SENSES-ASD: a social-emotional nurturing and skill enhancement system for autism spectrum disorder

This article introduces the Social-Emotional Nurturing and Skill Enhancement System (SENSES-ASD) as an innovative method for assisting individuals with autism spectrum disorder (ASD). Leveraging deep learning technologies, specifically convolutional neural networks (CNN), our approach promotes facial emotion recognition, enhancing social interactions and communication. The methodology involves the use of the Xception CNN model trained on the FER-2013 dataset. The designed system accepts a variety of media inputs, successfully classifying and predicting seven primary emotional states. Results show that our system achieved a peak accuracy rate of 71% on the training dataset and 66% on the validation dataset. The novelty of our work lies in the intricate combination of deep learning methods specifically tailored for high-functioning autistic adults and the development of a user interface that caters to their unique cognitive and sensory sensitivities. This offers a novel perspective on utilising technological advances for ASD intervention, especially in the domain of emotion recognition.

Facial emotion recognition in individuals with mild cognitive impairment: An exploratory study

Understanding facial emotions is fundamental to interact in social environments and modify behavior accordingly. Neurodegenerative processes can progressively transform affective responses and affect social competence. This exploratory study examined the neurocognitive correlates of face recognition, in individuals with two mild cognitive impairment (MCI) etiologies (prodromal to dementia - MCI, or consequent to Parkinson's disease - PD-MCI). Performance on the identification and memorization of neutral and emotional facial expressions was assessed in 31 individuals with MCI, 26 with PD-MCI, and 30 healthy controls (HC). Individuals with MCI exhibited selective impairment in recognizing faces expressing fear, along with difficulties in remembering both neutral and emotional faces. Conversely, individuals with PD-MCI showed no differences compared with the HC in either emotion recognition or memory. In MCI, no significant association emerged between the memory for facial expressions and cognitive difficulties. In PD-MCI, regression analyses showed significant associations with higher-level cognitive functions in the emotional memory task, suggesting the presence of compensatory mechanisms. In a subset of participants, voxel-based morphometry revealed that the performance on emotional tasks correlated with regional changes in gray matter volume. The performance in the matching of negative expressions was predicted by volumetric changes in brain areas engaged in face and emotional processing, in particular increased volume in thalamic nuclei and atrophy in the right parietal cortex. Future studies should leverage on neuroimaging data to determine whether differences in emotional recognition are mediated by pathology-specific atrophic patterns.

An emotion recognition method based on EWT-3D-CNN-BiLSTM-GRU-AT model

This has become a significant study area in recent years because of its use in brain-machine interaction (BMI). The robustness problem of emotion classification is one of the most basic approaches for improving the quality of emotion recognition systems. One of the two main branches of these approaches deals with the problem by extracting the features using manual engineering and the other is the famous artificial intelligence approach, which infers features of EEG data. This study proposes a novel method that considers the characteristic behavior of EEG recordings and based on the artificial intelligence method. The EEG signal is a noisy signal with a non-stationary and non-linear form. Using the Empirical Wavelet Transform (EWT) signal decomposition method, the signal's frequency components are obtained. Then, frequency-based features, linear and non-linear features are extracted. The resulting frequency-based, linear, and nonlinear features are mapped to the 2-D axis according to the positions of the EEG electrodes. By merging this 2-D images, 3-D images are constructed. In this way, the multichannel brain frequency of EEG recordings, spatial and temporal relationship are combined. Lastly, 3-D deep learning framework was constructed, which was combined with convolutional neural network (CNN), bidirectional long-short term memory (BiLSTM) and gated recurrent unit (GRU) with self-attention (AT). This model is named EWT-3D-CNN-BiLSTM-GRU-AT. As a result, we have created framework comprising handcrafted features generated and cascaded from state-of-the-art deep learning models. The framework is evaluated on the DEAP recordings based on the person-independent approach. The experimental findings demonstrate that the developed model can achieve classification accuracies of 90.57 % and 90.59 % for valence and arousal axes, respectively, for the DEAP database. Compared with existing cutting-edge emotion classification models, the proposed framework exhibits superior results for classifying human emotions.

Alexithymia and facial emotion recognition in patients with functional neurological disorder

OBJECTIVES

Patients with functional neurological disorder (FND) are known to have difficulties recognizing and processing emotions. Problems recognizing internal emotional states (alexithymia) are common in FND, but little is known about recognizing emotions expressed by other people. This study investigates whether patients with FND have higher levels of alexithymia and reduced facial emotion recognition compared to healthy controls.

METHODS

Patients with FND (n = 31, mean age=42.7 [SD=14.8] years, 54.8% women) were compared to healthy controls (n = 33, mean age=45.1 [SD=16.2] years, 63.6% women). The Bermond-Vorst Alexithymia Questionnaire (BVAQ) was used for the assessment of alexithymia and the Ekman 60 Faces Test (EFT) for facial emotion recognition.

RESULTS

Patients with FND had higher levels of alexithymia than healthy controls (BVAQ=71.8 [SD=19.8] versus 59.3 [SD=20.3], p = .02, Cohen's d=0.62). Facial emotion recognition did not significantly differ between FND patients and controls (EFT total score FND: 46.1 [SD=5.9], Controls: 47.5 [SD=5.5], p = .34, Cohen's d=0.24). Only recognition of surprise differed between patients and controls (FND: 8.4 [SD=1.8], Controls: 9.2 [SD=1.0), p = .03, Cohen's d= 0.56). Higher levels of alexithymia were associated with poorer facial emotion recognition, but this relationship was not statistically significant (FND: β= -0.20, p = .28; Controls: β=-0.03; p = .87).

CONCLUSIONS

The current data confirm prior observations that patients with FND have higher alexithymia levels than controls without FND. Difficulties recognizing emotions among patients with FND primarily involves recognition of internal emotional states rather than recognition of facially expressed emotions by others. These findings require replication in a larger and more divers sample.

EEG based emotion recognition by hierarchical bayesian spectral regression framework

Spectral regression (SR), a graph-based learning regression model, can be used to extract features from graphs to realize efficient dimensionality reduction. However, due to the SR method remains a regularized least squares problem and being defined in L2-norm space, the effect of artifacts in EEG signals cannot be efficiently resisted. In this work, to further improve the robustness of the graph-based regression models, we propose to utilize the prior distribution estimation in the Bayesian framework and develop a robust hierarchical Bayesian spectral regression framework (named HB-SR), which is designed with the hierarchical Bayesian ensemble strategies. In the proposed HB-SR, the impact of noises can be effectively reduced by the adaptive adjustment approach in model parameters with the data-driven manner. Specifically, in the current work, three different distributions have been elaborately designed to enhance the universality of the proposed HB-SR, i.e., Gaussian distribution, Laplace distribution, and Student-t distribution. To objectively evaluate the performance of the HB-SR framework, we conducted both simulation studies and emotion recognition experiments based on emotional EEG signals. Experimental results have consistently indicated that compared with other existing spectral regression methods, the proposed HB-SR can effectively suppress the influence of noises and achieve robust EEG emotion recognition.

Instance-representation transfer method based on joint distribution and deep adaptation for EEG emotion recognition

Electroencephalogram (EEG) emotion recognition technology is essential for improving human-computer interaction. However, the practical application of emotion recognition technology is limited due to the variety of subjects and sessions. Transfer learning has been applied to address this issue and has received extensive research and application. Studies mainly concentrate on either instance transfer or representation transfer methods. This paper proposes an emotion recognition method called Joint Distributed Instances Represent Transfer (JD-IRT), which includes two core components: Joint Distribution Deep Adaptation (JDDA) and Instance-Representation Transfer (I-RT). Specifically, JDDA is different from common representation transfer methods in transfer learning. It bridges the discrepancies of marginal and conditional distributions simultaneously and combines multiple adaptive layers and kernels for deep domain adaptation. On the other hand, I-RT utilizes instance transfer to select source domain data for better representation transfer. We performed experiments and compared them with other representative methods in the SEED, SEED-IV, and SEED-V datasets. In cross-subject experiments, our approach achieved an average accuracy of 83.21% in SEED, 52.12% in SEED-IV, and 60.17% in SEED-V. Similarly, in cross-session experiments, the accuracy was 91.29% in SEED, 59.02% in SEED-IV, and 65.91% in SEED-V. These results demonstrate the improvement in the accuracy of EEG emotion recognition using the proposed approach.

Graph theoretical brain connectivity measures to investigate neural correlates of music rhythms associated with fear and anger

The present study tests the hypothesis that emotions of fear and anger are associated with distinct psychophysiological and neural circuitry according to discrete emotion model due to contrasting neurotransmitter activities, despite being included in the same affective group in many studies due to similar arousal-valance scores of them in emotion models. EEG data is downloaded from OpenNeuro platform with access number of ds002721. Brain connectivity estimations are obtained by using both functional and effective connectivity estimators in analysis of short (2 sec) and long (6 sec) EEG segments across the cortex. In tests, discrete emotions and resting-states are identified by frequency band specific brain network measures and then contrasting emotional states are deep classified with 5-fold cross-validated Long Short Term Memory Networks. Logistic regression modeling has also been examined to provide robust performance criteria. Commonly, the best results are obtained by using Partial Directed Coherence in Gamma (31.5 - 60.5 H z ) sub-bands of short EEG segments. In particular, Fear and Anger have been classified with accuracy of 91.79%. Thus, our hypothesis is supported by overall results. In conclusion, Anger is found to be characterized by increased transitivity and decreased local efficiency in addition to lower modularity in Gamma-band in comparison to fear. Local efficiency refers functional brain segregation originated from the ability of the brain to exchange information locally. Transitivity refer the overall probability for the brain having adjacent neural populations interconnected, thus revealing the existence of tightly connected cortical regions. Modularity quantifies how well the brain can be partitioned into functional cortical regions. In conclusion, PDC is proposed to graph theoretical analysis of short EEG epochs in presenting robust emotional indicators sensitive to perception of affective sounds.