Enhanced performance of EEG-based brain-computer interfaces by joint sample and feature importance assessment

Electroencephalograph (EEG) has been a reliable data source for building brain-computer interface (BCI) systems; however, it is not reasonable to use the feature vector extracted from multiple EEG channels and frequency bands to perform recognition directly due to the two deficiencies. One is that EEG data is weak and non-stationary, which easily causes different EEG samples to have different quality. The other is that different feature dimensions corresponding to different brain regions and frequency bands have different correlations to a certain mental task, which is not sufficiently investigated. To this end, a Joint Sample and Feature importance Assessment (JSFA) model was proposed to simultaneously explore the different impacts of EEG samples and features in mental state recognition, in which the former is based on the self-paced learning technique while the latter is completed by the feature self-weighting technique. The efficacy of JSFA is extensively evaluated on two EEG data sets, i.e., SEED-IV and SEED-VIG. One is a classification task for emotion recognition and the other is a regression task for driving fatigue detection. Experimental results demonstrate that JSFA can effectively identify the importance of different EEG samples and features, leading to enhanced recognition performance of corresponding BCI systems.

The Social and Cognitive Online Training (SCOT) project: A digital randomized controlled trial to promote socio-cognitive well-being in older adults

OBJECTIVES

Effective prevention programs targeting risk factors for cognitive decline in the elderly are recommended given the progressive increase in the aging of the general population. The Social and Cognitive Online Training (SCOT) project is a randomized, controlled, parallel clinical trial designed to prevent the age-related decline in executive and social functions.

METHODS

The study included 60 cognitively healthy older adults (age = 71.8±5.3, education = 12.3±3.7, MoCA = 25.1±2.4). Participants underwent a baseline clinical and neuropsychological assessment and were then assigned to either an experimental group (SCOT) or a non-specific cognitive training group (CON). Both 8-week digital interventions included two individual cognitive training sessions and one group meeting per week. Post-intervention assessment evaluated the efficacy of the training on specific outcome measures: the Tower of London for executive functioning, the Ekman-60 Faces test, and the Mini-Social cognition & Emotional Assessment battery for social cognition. A measure of loneliness was included as an exploratory outcome.

RESULTS

Baseline demographic and neuropsychological characteristics were balanced between SCOT (n = 29) and CON (n = 28) groups. Pre-post-intervention analyses showed improvements in executive functioning and social cognition in both groups, without significant interaction effects. Exploratory post-hoc analyses stratifying the SCOT group by training performance showed significant post-training improvements in executive functioning, emotion recognition, and cognitive theory of mind for high-performing participants.

DISCUSSION

Results provide preliminary evidence for the beneficial effects of SCOT training, particularly for those who performed best during the training. The SCOT training could represent a new intervention to promote socio-cognitive well-being in the context of active ageing and dementia prevention.

Atypical Facial Expressivity in Young Children with Problematic Peer Relationships

Peer problems are frequently associated with difficulties in recognizing and appraising the emotions of others. It has been argued that facial responsiveness to others' emotions-or motor empathy-is a precursor of emotion processing and affective empathy. Although mimicry impairments have been observed in studies of young people with conduct problems, to our knowledge no study has examined facial responsiveness to others' expressions in young children and examined how this relates to peer relationship problems. Four- to 7-year-old children (n = 91) with or without teacher-reported peer relationship problems (Strength and Difficulties Questionnaire) viewed three dynamic film clips depicting a sad, happy, or scared child, while their spontaneous facial emotional responses were assessed using iMotions software that codes the movement of facial muscles. Children displayed facial expressivity that was congruent with the emotional expressions in the clips. Groups with and without peer problems did not differ in their responses to seeing a happy child. However, children with peer problems exhibited reduced or atypical facial emotional responses to the negative emotional clips. Decreased or atypical facial expressivity to negative emotions was also associated with severity of peer problems; atypical facial responsivity to sadness and reduced facial responsivity to fear predicted peer problems independently of one another. We conclude that reduced or atypical facial expressiveness in response to other children's dynamic facial expressions is associated with problematic peer relations in young children. The implications for early identification and interventions to support prosocial development are discussed.

Face and emotion recognition in individuals diagnosed with schizophrenia, ultra-high risk for psychosis, unaffected siblings, and healthy controls in a sample from Turkey

Face and emotion recognition are crucial components of social cognition. We aimed to compare them in patients diagnosed with schizophrenia (SCZ), ultra-high risk for psychosis (UHR), unaffected siblings of schizophrenia patients (SIB), and healthy controls (HC).

METHODS

One hundred sixty-six participants (45 SCZ, 14 UHR, 45 SIB, and 62 HC) were interviewed with the Structured Clinical Interview for DSM-5 (SCID-5). Positive and Negative syndrome scale (PANSS), PennCNB Facial Memory (CPF), and Emotion Recognition Task (ER40) were applied.

RESULTS

In CPF, SCZ performed significantly lower than SIB and HC. SIB was also significantly lower than HC for total correct responses. The sample size of the UHR group was small, and the statistical comparisons did not reach a significance, however, a trend towards decreased performance between the SCZ and SIB was found. In ER40, SCZ performed significantly lower than HC and SIB in all domains, except for the insignificant findings for angry ER between SIB and SCZ. SIB also performed significantly lower than HC for angry, negative, and total ER. UHR was similar to SCZ for happy and sad ER and performed significantly lower than HC for happy ER. The effect of SCZ diagnosis on the efficiency of CPF and ER40 was significant when corrected for age and education. For SCZ, PANSS also significantly affected the CPF and ER40.

CONCLUSION

Our findings suggest varying levels of face and emotion recognition deficits in individuals with SCZ, UHR, and SIB. Face and emotion recognition deficits are promising schizophrenia endophenotypes related to social cognition.

Synthesizing affective neurophysiological signals using generative models: A review paper

The integration of emotional intelligence in machines is an important step in advancing human-computer interaction. This demands the development of reliable end-to-end emotion recognition systems. However, the scarcity of public affective datasets presents a challenge. In this literature review, we emphasize the use of generative models to address this issue in neurophysiological signals, particularly Electroencephalogram (EEG) and Functional Near-Infrared Spectroscopy (fNIRS). We provide a comprehensive analysis of different generative models used in the field, examining their input formulation, deployment strategies, and methodologies for evaluating the quality of synthesized data. This review serves as a comprehensive overview, offering insights into the advantages, challenges, and promising future directions in the application of generative models in emotion recognition systems. Through this review, we aim to facilitate the progression of neurophysiological data augmentation, thereby supporting the development of more efficient and reliable emotion recognition systems.

Hearts and Minds: Emotion Recognition and Mentalizing in Parkinson's Disease and Progressive Supranuclear Palsy

OBJECTIVE

There are scarce data comparing Parkinson's disease (PD) and Progressive Supranuclear Palsy (PSP) in social cognition (SC). We aimed to compare patients with PSP and PD in SC.

METHODS

We included three groups: PD (n = 18), PSP (n = 20) and controls (n = 23). Participants underwent neuropsychological exams, including the mini-version of the Social and Emotional Assessment, which is composed of the facial emotion recognition test (FERT) and the modified faux-pas (mFP) test, which assesses Theory of Mind (ToM).

RESULTS

Patients with PD scored lower than controls in the FERT, but not in the mFP test. Patients with PSP performed worse than controls in both the mFP and FERT. PD and PSP groups did not differ in the FERT, but PSP performed worse than PD in the mFP test. The mFP test distinguished PSP from PD with 89% accuracy.

CONCLUSION

The assessment of ToM may contribute to the differentiation between PD and PSP.

Emotion recognition in amyotrophic lateral sclerosis in a dynamic environment

OBJECTIVE

The aim of our study was to measure the ability of ALS patients to process dynamic facial expressions as compared to a control group of healthy subjects and to correlate this ability in ALS patients with neuropsychological, clinical and neurological measures of the disease.

METHODS

Sixty-three ALS patients and 47 healthy controls were recruited. All the ALS patients also underwent i) the Geneva Emotion Recognition Test (GERT) in which ten actors express 14 types of dynamic emotions in brief video clips with audio, ii) the Edimburgh Cognitive and Behavioral ALS Screen (ECAS) test; iii) the ALS Functional Rating Scale Revised (ALSFRS-R) and iv) the Medical Research Council (MRC) for the evaluation of muscle strength. All the healthy subjects enrolled in the study underwent the GERT.

RESULTS

The recognition of irritation and pleasure was significantly different between ALS patients and the control group. The amusement, despair, irritation, joy, sadness and surprise had been falsely recognized differently between the two groups. Specific ALS cognitive impairment was associated with bulbar-onset phenotype (OR = 14,3889; 95%CI = 3,96-52,16). No association was observed between false emotion recognition and cognitive impairment (F(1,60)=,56,971, p=,45,333). The number of categorical errors was significantly higher in the ALS patients than in the control group (27,66 ± 7,28 vs 17,72 ± 5,29; t = 8723; p = 0.001).

CONCLUSIONS

ALS patients show deficits in the dynamic processing of a wide range of emotions. These deficits are not necessarily associated with a decline in higher cognitive functions: this could therefore lead to an underestimation of the phenomenon.

Multi-scale 3D-CRU for EEG emotion recognition

In this paper, we propose a novel multi-scale 3D-CRU model, with the goal of extracting more discriminative emotion feature from EEG signals. By concurrently exploiting the relative electrode locations and different frequency subbands of EEG signals, a three-dimensional feature representation is reconstructed wherein the Delta (δ) frequency pattern is included. We employ a multi-scale approach, termed 3D-CRU, to concurrently extract frequency and spatial features at varying levels of granularity within each time segment. In the proposed 3D-CRU, we introduce a multi-scale 3D Convolutional Neural Network (3D-CNN) to effectively capture discriminative information embedded within the 3D feature representation. To model the temporal dynamics across consecutive time segments, we incorporate a Gated Recurrent Unit (GRU) module to extract temporal representations from the time series of combined frequency-spatial features. Ultimately, the 3D-CRU model yields a global feature representation, encompassing comprehensive information across time, frequency, and spatial domains. Numerous experimental assessments conducted on publicly available DEAP and SEED databases provide empirical evidence supporting the enhanced performance of our proposed model in the domain of emotion recognition. These findings underscore the efficacy of the features extracted by the proposed multi-scale 3D-GRU model, particularly with the incorporation of the Delta (δ) frequency pattern. Specifically, on the DEAP dataset, the accuracy of Valence and Arousal are 93.12% and 94.31%, respectively, while on the SEED dataset, the accuracy is 92.25%.

Peripheral inflammation is associated with impaired sadness recognition in euthymic bipolar patients

BACKGROUND

Inflammation impairs cognitive function in healthy individuals and people with psychiatric disorders, such as bipolar disorder (BD). This effect may also impact emotion recognition, a fundamental element of social cognition. Our study aimed to investigate the relationships between pro-inflammatory cytokines and emotion recognition in euthymic BD patients and healthy controls (HCs).

METHODS

We recruited forty-four euthymic BD patients and forty healthy controls (HCs) and measured their inflammatory markers, including high-sensitivity C-reactive protein (hs-CRP), interleukin-6 (IL-6), and TNF-α. We applied validated cognitive tasks, the Wisconsin Card-Sorting Test (WCST) and Continuous Performance Test (CPT), and a social cognitive task for emotion recognition, Diagnostic Analyses of Nonverbal Accuracy, Taiwanese Version (DANVA-2-TW). We analyzed the relationships between cytokines and cognition and then explored possible predictive factors of sadness recognition accuracy.

RESULTS

Regarding pro-inflammatory cytokines, TNF-α was elevated in euthymic BD patients relative to HCs. In euthymic BD patients only, higher TNF-α levels were associated with lower accuracy of sadness recognition. Regression analysis revealed that TNF-α was an independent predictive factor of sadness recognition in patients with euthymic BD when neurocognition was controlled for.

CONCLUSIONS

We demonstrated that enhanced inflammation, indicated by increased TNF-α, was an independent predictive factor of impaired sadness recognition in BD patients but not in HCs. Our findings suggested a direct influence of TNF-α on sadness recognition and indicated vulnerability to depression in euthymic BD patients with chronic inflammation.

Enhancing cross-subject EEG emotion recognition through multi-source manifold metric transfer learning

Transfer learning (TL) has demonstrated its efficacy in addressing the cross-subject domain adaptation challenges in affective brain-computer interfaces (aBCI). However, previous TL methods usually use a stationary distance, such as Euclidean distance, to quantify the distribution dissimilarity between two domains, overlooking the inherent links among similar samples, potentially leading to suboptimal feature mapping. In this study, we introduced a novel algorithm called multi-source manifold metric transfer learning (MSMMTL) to enhance the efficacy of conventional TL. Specifically, we first selected the source domain based on Mahalanobis distance to enhance the quality of the source domains and then used manifold feature mapping approach to map the source and target domains on the Grassmann manifold to mitigate data drift between domains. In this newly established shared space, we optimized the Mahalanobis metric by maximizing the inter-class distances while minimizing the intra-class distances in the target domain. Recognizing that significant distribution discrepancies might persist across different domains even on the manifold, to ensure similar distributions between the source and target domains, we further imposed constraints on both domains under the Mahalanobis metric. This approach aims to reduce distributional disparities and enhance the electroencephalogram (EEG) emotion recognition performance. In cross-subject experiments, the MSMMTL model exhibits average classification accuracies of 88.83 % and 65.04 % for SEED and DEAP, respectively, underscoring the superiority of our proposed MSMMTL over other state-of-the-art methods. MSMMTL can effectively solve the problem of individual differences in EEG-based affective computing.

The relationship between the resting state functional connectivity and social cognition in schizophrenia: Results from the Italian Network for Research on Psychoses

Deficits in social cognition (SC) interfere with recovery in schizophrenia (SZ) and may be related to resting state brain connectivity. This study aimed at assessing the alterations in the relationship between resting state functional connectivity and the social-cognitive abilities of patients with SZ compared to healthy subjects. We divided the brain into 246 regions of interest (ROI) following the Human Healthy Volunteers Brainnetome Atlas. For each participant, we calculated the resting-state functional connectivity (rsFC) in terms of degree centrality (DC), which evaluates the total strength of the most powerful coactivations of every ROI with all other ROIs during rest. The rs-DC of the ROIs was correlated with five measures of SC assessing emotion processing and mentalizing in 45 healthy volunteers (HVs) chosen as a normative sample. Then, controlling for symptoms severity, we verified whether these significant associations were altered, i.e., absent or of opposite sign, in 55 patients with SZ. We found five significant differences between SZ patients and HVs: in the patients' group, the correlations between emotion recognition tasks and rsFC of the right entorhinal cortex (R-EC), left superior parietal lobule (L-SPL), right caudal hippocampus (R-c-Hipp), and the right caudal (R-c) and left rostral (L-r) middle temporal gyri (MTG) were lost. An altered resting state functional connectivity of the L-SPL, R-EC, R-c-Hipp, and bilateral MTG in patients with SZ may be associated with impaired emotion recognition. If confirmed, these results may enhance the development of non-invasive brain stimulation interventions targeting those cerebral regions to reduce SC deficit in SZ.

Spatio-Temporal Image-Based Encoded Atlases for EEG Emotion Recognition

Emotion recognition plays an essential role in human-human interaction since it is a key to understanding the emotional states and reactions of human beings when they are subject to events and engagements in everyday life. Moving towards human-computer interaction, the study of emotions becomes fundamental because it is at the basis of the design of advanced systems to support a broad spectrum of application areas, including forensic, rehabilitative, educational, and many others. An effective method for discriminating emotions is based on ElectroEncephaloGraphy (EEG) data analysis, which is used as input for classification systems. Collecting brain signals on several channels and for a wide range of emotions produces cumbersome datasets that are hard to manage, transmit, and use in varied applications. In this context, the paper introduces the Empátheia system, which explores a different EEG representation by encoding EEG signals into images prior to their classification. In particular, the proposed system extracts spatio-temporal image encodings, or atlases, from EEG data through the Processing and transfeR of Interaction States and Mappings through Image-based eNcoding (PRISMIN) framework, thus obtaining a compact representation of the input signals. The atlases are then classified through the Empátheia architecture, which comprises branches based on convolutional, recurrent, and transformer models designed and tuned to capture the spatial and temporal aspects of emotions. Extensive experiments were conducted on the Shanghai Jiao Tong University (SJTU) Emotion EEG Dataset (SEED) public dataset, where the proposed system significantly reduced its size while retaining high performance. The results obtained highlight the effectiveness of the proposed approach and suggest new avenues for data representation in emotion recognition from EEG signals.

An enhanced Coati Optimization Algorithm for global optimization and feature selection in EEG emotion recognition

Emotion recognition based on Electroencephalography (EEG) signals has garnered significant attention across diverse domains including healthcare, education, information sharing, and gaming, among others. Despite its potential, the absence of a standardized feature set poses a challenge in efficiently classifying various emotions. Addressing the issue of high dimensionality, this paper introduces an advanced variant of the Coati Optimization Algorithm (COA), called eCOA for global optimization and selecting the best subset of EEG features for emotion recognition. Specifically, COA suffers from local optima and imbalanced exploitation abilities as other metaheuristic methods. The proposed eCOA incorporates the COA and RUNge Kutta Optimizer (RUN) algorithms. The Scale Factor (SF) and Enhanced Solution Quality (ESQ) mechanism from RUN are applied to resolve the raised shortcomings of COA. The proposed eCOA algorithm has been extensively evaluated using the CEC'22 test suite and two EEG emotion recognition datasets, DEAP and DREAMER. Furthermore, the eCOA is applied for binary and multi-class classification of emotions in the dimensions of valence, arousal, and dominance using a multi-layer perceptron neural network (MLPNN). The experimental results revealed that the eCOA algorithm has more powerful search capabilities than the original COA and seven well-known counterpart methods related to statistical, convergence, and diversity measures. Furthermore, eCOA can efficiently support feature selection to find the best EEG features to maximize performance on four quadratic emotion classification problems compared to the methods of its counterparts. The suggested method obtains a classification accuracy of 85.17% and 95.21% in the binary classification of low and high arousal emotions in two public datasets: DEAP and DREAMER, respectively, which are 5.58% and 8.98% superior to existing approaches working on the same datasets for different subjects, respectively.

Reduced eye gaze fixation during emotion recognition among patients with temporal lobe epilepsy

OBJECTIVES

To investigate the facial scan patterns during emotion recognition (ER) through the dynamic facial expression task and the awareness of social interference test (TASIT) using eye tracking (ET) technology, and to find some ET indicators that can accurately depict the ER process, which is a beneficial supplement to existing ER assessment tools.

METHOD

Ninety-six patients with TLE and 88 healthy controls (HCs) were recruited. All participants watched the dynamic facial expression task and TASIT including a synchronized eye movement recording and recognized the emotion (anger, disgust, happiness, or sadness). The accuracy of ER was recorded. The first fixation time, first fixation duration, dwell time, and fixation count were selected and analyzed.

RESULTS

TLE patients exhibited ER impairment especially for disgust (Z =  - 3.391; p = 0.001) and sadness (Z =  - 3.145; p = 0.002). TLE patients fixated less on the face, as evidenced by the reduced fixation count (Z =  - 2.549; p = 0.011) of the face and a significant decrease in the fixation count rate (Z =  - 1.993; p = 0.046). During the dynamic facial expression task, TLE patients focused less on the eyes, as evidenced by the decreased first fixation duration (Z =  - 4.322; p = 0.000), dwell time (Z =  - 4.083; p = 0.000), and fixation count (Z =  - 3.699; p = 0.000) of the eyes.

CONCLUSION

TLE patients had ER impairment, especially regarding negative emotions, which may be attributable to their reduced fixation on the eyes during ER, and the increased fixation on the mouth could be a compensatory effect to improve ER performance. Eye-tracking technology could provide the process indicators of ER, and is a valuable supplement to traditional ER assessment tasks.

The Evolution of the Optimization of Cognitive and Social Functions in the Cerebellum and Thereby the Rise of Homo sapiens Through Cumulative Culture

The evolution of the prominent role of the cerebellum in the development of composite tools, and cumulative culture, leading to the rise of Homo sapiens is examined. Following Stout and Hecht's (2017) detailed description of stone-tool making, eight key repetitive involvements of the cerebellum are highlighted. These key cerebellar learning involvements include the following: (1) optimization of cognitive-social control, (2) prediction (3) focus of attention, (4) automaticity of smoothness, appropriateness, and speed of movement and cognition, (5) refined movement and social cognition, (6) learns models of extended practice, (7) learns models of Theory of Mind (ToM) of teachers, (8) is predominant in acquisition of novel behavior and cognition that accrues from the blending of cerebellar models sent to conscious working memory in the cerebral cortex. Within this context, the evolution of generalization and blending of cerebellar internal models toward optimization of social-cognitive learning is described. It is concluded that (1) repetition of movement and social cognition involving the optimization of internal models in the cerebellum during stone-tool making was the key selection factor toward social-cognitive and technological advancement, (2) observational learning during stone-tool making was the basis for both technological and social-cognitive evolution and, through an optimizing positive feedback loop between the cerebellum and cerebral cortex, the development of cumulative culture occurred, and (3) the generalization and blending of cerebellar internal models related to the unconscious forward control of the optimization of imagined future states in working memory was the most important brain adaptation leading to intertwined advances in stone-tool technology, cognitive-social processes behind cumulative culture (including the emergence of language and art) and, thereby, with the rise of Homo sapiens.

[Research progress on emotion recognition by combining virtual reality environment and electroencephalogram signals]

Emotion recognition refers to the process of determining and identifying an individual's current emotional state by analyzing various signals such as voice, facial expressions, and physiological indicators etc. Using electroencephalogram (EEG) signals and virtual reality (VR) technology for emotion recognition research helps to better understand human emotional changes, enabling applications in areas such as psychological therapy, education, and training to enhance people's quality of life. However, there is a lack of comprehensive review literature summarizing the combined researches of EEG signals and VR environments for emotion recognition. Therefore, this paper summarizes and synthesizes relevant research from the past five years. Firstly, it introduces the relevant theories of VR and EEG signal emotion recognition. Secondly, it focuses on the analysis of emotion induction, feature extraction, and classification methods in emotion recognition using EEG signals within VR environments. The article concludes by summarizing the research's application directions and providing an outlook on future development trends, aiming to serve as a reference for researchers in related fields.

Neural correlates of individual differences in multimodal emotion recognition ability

Studies have reported substantial variability in emotion recognition ability (ERA) - an important social skill - but possible neural underpinnings for such individual differences are not well understood. This functional magnetic resonance imaging (fMRI) study investigated neural responses during emotion recognition in young adults (N = 49) who were selected for inclusion based on their performance (high or low) during previous testing of ERA. Participants were asked to judge brief video recordings in a forced-choice emotion recognition task, wherein stimuli were presented in visual, auditory and multimodal (audiovisual) blocks. Emotion recognition rates during brain scanning confirmed that individuals with high (vs low) ERA received higher accuracy for all presentation blocks. fMRI-analyses focused on key regions of interest (ROIs) involved in the processing of multimodal emotion expressions, based on previous meta-analyses. In neural response to emotional stimuli contrasted with neutral stimuli, individuals with high (vs low) ERA showed higher activation in the following ROIs during the multimodal condition: right middle superior temporal gyrus (mSTG), right posterior superior temporal sulcus (PSTS), and right inferior frontal cortex (IFC). Overall, results suggest that individual variability in ERA may be reflected across several stages of decisional processing, including extraction (mSTG), integration (PSTS) and evaluation (IFC) of emotional information.

E-MFNN: an emotion-multimodal fusion neural network framework for emotion recognition

Emotional recognition is a pivotal research domain in computer and cognitive science. Recent advancements have led to various emotion recognition methods, leveraging data from diverse sources like speech, facial expressions, electroencephalogram (EEG), electrocardiogram, and eye tracking (ET). This article introduces a novel emotion recognition framework, primarily targeting the analysis of users' psychological reactions and stimuli. It is important to note that the stimuli eliciting emotional responses are as critical as the responses themselves. Hence, our approach synergizes stimulus data with physical and physiological signals, pioneering a multimodal method for emotional cognition. Our proposed framework unites stimulus source data with physiological signals, aiming to enhance the accuracy and robustness of emotion recognition through data integration. We initiated an emotional cognition experiment to gather EEG and ET data alongside recording emotional responses. Building on this, we developed the Emotion-Multimodal Fusion Neural Network (E-MFNN), optimized for multimodal data fusion to process both stimulus and physiological data. We conducted extensive comparisons between our framework's outcomes and those from existing models, also assessing various algorithmic approaches within our framework. This comparison underscores our framework's efficacy in multimodal emotion recognition. The source code is publicly available at https://figshare.com/s/8833d837871c78542b29.

Duration of face mask exposure matters: evidence from Swiss and Brazilian kindergartners' ability to recognise emotions

Wearing facial masks became a common practice worldwide during the COVID-19 pandemic. This study investigated (1) whether facial masks that cover adult faces affect 4- to 6-year-old children's recognition of emotions in those faces and (2) whether the duration of children's exposure to masks is associated with emotion recognition. We tested children from Switzerland (N = 38) and Brazil (N = 41). Brazil represented longer mask exposure due to a stricter mandate during COVID-19. Children had to choose a face displaying a specific emotion (happy, angry, or sad) when the face wore either no cover, a facial mask, or sunglasses. The longer hours of mask exposure were associated with better emotion recognition. Controlling for the hours of exposure, children were less likely to recognise emotions in partially hideen faces. Moreover, Brazilian children were more accurate in recognising happy faces than Swiss children. Overall, facial masks may negatively impact children's emotion recognition. However, prolonged exposure appears to buffer the lack of facial cues from the nose and mouth. In conclusion, restricting facial cues due to masks may impair kindergarten children's emotion recognition in the short run. However, it may facilitate their broader reading of facial emotional cues in the long run.

Supraphysiological testosterone levels from anabolic steroid use and reduced sensitivity to negative facial expressions in men

RATIONALE

Anabolic-androgenic steroids (AAS) are used to improve physical performance and appearance, but have been associated with deficits in social cognitive functioning. Approximately 30% of people who use AAS develop a dependence, increasing the risk for undesired effects.

OBJECTIVES

To assess the relationship between AAS use (current/previous), AAS dependence, and the ability to recognize emotional facial expressions, and investigate the potential mediating role of hormone levels.

METHODS

In total 156 male weightlifters, including those with current (n = 45) or previous (n = 34) AAS use and never-using controls (n = 77), completed a facial Emotion Recognition Task (ERT). Participants were presented with faces expressing one out of six emotions (sadness, happiness, fear, anger, disgust, and surprise) and were instructed to indicate which of the six emotions each face displayed. ERT accuracy and response time were recorded and evaluated for association with AAS use status, AAS dependence, and serum reproductive hormone levels. Mediation models were used to evaluate the mediating role of androgens in the relationship between AAS use and ERT performance.

RESULTS

Compared to never-using controls, men currently using AAS exhibited lower recognition accuracy for facial emotional expressions, particularly anger (Cohen's d = -0.57, pFDR = 0.03) and disgust (d = -0.51, pFDR = 0.05). Those with AAS dependence (n = 47) demonstrated worse recognition of fear relative to men without dependence (d = 0.58, p = 0.03). Recognition of disgust was negatively correlated with serum free testosterone index (FTI); however, FTI did not significantly mediate the association between AAS use and recognition of disgust.

CONCLUSIONS

Our findings demonstrate impaired facial emotion recognition among men currently using AAS compared to controls. While further studies are needed to investigate potential mechanisms, our analysis did not support a simple mediation effect of serum FTI.

Decoding emotion with phase-amplitude fusion features of EEG functional connectivity network

Decoding emotional neural representations from the electroencephalographic (EEG)-based functional connectivity network (FCN) is of great scientific importance for uncovering emotional cognition mechanisms and developing harmonious human-computer interactions. However, existing methods mainly rely on phase-based FCN measures (e.g., phase locking value [PLV]) to capture dynamic interactions between brain oscillations in emotional states, which fail to reflect the energy fluctuation of cortical oscillations over time. In this study, we initially examined the efficacy of amplitude-based functional networks (e.g., amplitude envelope correlation [AEC]) in representing emotional states. Subsequently, we proposed an efficient phase-amplitude fusion framework (PAF) to fuse PLV and AEC and used common spatial pattern (CSP) to extract fused spatial topological features from PAF for multi-class emotion recognition. We conducted extensive experiments on the DEAP and MAHNOB-HCI datasets. The results showed that: (1) AEC-derived discriminative spatial network topological features possess the ability to characterize emotional states, and the differential network patterns of AEC reflect dynamic interactions in brain regions associated with emotional cognition. (2) The proposed fusion features outperformed other state-of-the-art methods in terms of classification accuracy for both datasets. Moreover, the spatial filter learned from PAF is separable and interpretable, enabling a description of affective activation patterns from both phase and amplitude perspectives.

EEG-based emotion recognition using a temporal-difference minimizing neural network

Electroencephalogram (EEG) emotion recognition plays an important role in human-computer interaction. An increasing number of algorithms for emotion recognition have been proposed recently. However, it is still challenging to make efficient use of emotional activity knowledge. In this paper, based on prior knowledge that emotion varies slowly across time, we propose a temporal-difference minimizing neural network (TDMNN) for EEG emotion recognition. We use maximum mean discrepancy (MMD) technology to evaluate the difference in EEG features across time and minimize the difference by a multibranch convolutional recurrent network. State-of-the-art performances are achieved using the proposed method on the SEED, SEED-IV, DEAP and DREAMER datasets, demonstrating the effectiveness of including prior knowledge in EEG emotion recognition.

Assessing multimodal emotion recognition in multiple sclerosis with a clinically accessible measure

BACKGROUND

Multiple sclerosis (MS) negatively impacts cognition and has been associated with deficits in social cognition, including emotion recognition. There is a lack of research examining emotion recognition from multiple modalities in MS. The present study aimed to employ a clinically available measure to assess multimodal emotion recognition abilities among individuals with MS.

METHOD

Thirty-one people with MS and 21 control participants completed the Advanced Clinical Solutions Social Perceptions Subtest (ACS-SP), BICAMS, and measures of premorbid functioning, mood, and fatigue. ANCOVAs examined group differences in all outcomes while controlling for education. Correlational analyses examined potential correlates of emotion recognition in both groups.

RESULTS

The MS group performed significantly worse on the ACS-SP than the control group, F(1, 49) = 5.32, p = .025. Significant relationships between emotion recognition and cognitive functions were found only in the MS group, namely for information processing speed (r = 0.59, p < .001), verbal learning (r = 0.52, p = .003) and memory (r = 0.65, p < 0.001), and visuospatial learning (r = 0.62, p < 0.001) and memory (r = 0.52, p = .003). Emotion recognition did not correlate with premorbid functioning, mood, or fatigue in either group.

CONCLUSIONS

This study was the first to employ the ACS-SP to assess emotion recognition in MS. The results suggest that emotion recognition is impacted in MS and is related to other cognitive processes, such as information processing speed. The results provide information for clinicians amidst calls to include social cognition measures in standard MS assessments.

A novel transformer autoencoder for multi-modal emotion recognition with incomplete data

Multi-modal signals have become essential data for emotion recognition since they can represent emotions more comprehensively. However, in real-world environments, it is often impossible to acquire complete data on multi-modal signals, and the problem of missing modalities causes severe performance degradation in emotion recognition. Therefore, this paper represents the first attempt to use a transformer-based architecture, aiming to fill the modality-incomplete data from partially observed data for multi-modal emotion recognition (MER). Concretely, this paper proposes a novel unified model called transformer autoencoder (TAE), comprising a modality-specific hybrid transformer encoder, an inter-modality transformer encoder, and a convolutional decoder. The modality-specific hybrid transformer encoder bridges a convolutional encoder and a transformer encoder, allowing the encoder to learn local and global context information within each particular modality. The inter-modality transformer encoder builds and aligns global cross-modal correlations and models long-range contextual information with different modalities. The convolutional decoder decodes the encoding features to produce more precise recognition. Besides, a regularization term is introduced into the convolutional decoder to force the decoder to fully leverage the complete and incomplete data for emotional recognition of missing data. 96.33%, 95.64%, and 92.69% accuracies are attained on the available data of the DEAP and SEED-IV datasets, and 93.25%, 92.23%, and 81.76% accuracies are obtained on the missing data. Particularly, the model acquires a 5.61% advantage with 70% missing data, demonstrating that the model outperforms some state-of-the-art approaches in incomplete multi-modal learning.

Suicide-related cognitions and emotional bias performance in a community sample

BACKGROUND

Suicide is theorized to be connected to social interactions and feelings of belongingness. Those with suicide-related cognitions (SRCs) demonstrate attentional bias toward negative or suicide-related words, which can lead to increased feelings of rejection or alienation. As social interactions employ both verbal and nonverbal cues, there exists a gap in understanding how perception of emotional expressions can contribute to the development or exacerbation of suicidal ideation.

METHODS

The current sample (N = 114, 60.5 % female, 74.6 % white) completed the Suicide Cognitions Scale-Revised (SCS-R) and Patient Health Questionnaire (PHQ-9) to assess SRCs and depression severity. The Emotional Bias Task (EBT) was used to assess emotional response latency.

RESULTS

Multiple regression analyses on EBT results showed that endorsement of SRCs and depression severity were not associated with any particular emotional response bias. However, presence of SRCs showed an association with longer latencies to identify ambiguous emotional expressions, even when controlling for depressive symptoms and age LIMITATIONS: Measures were self-completed online. Relative homogeneity of the sample and cross-sectional design limits interpretation of the results.

CONCLUSIONS

Those with more severe SRCs take longer to recognize positive, nonverbal cues. Irregular processing of positive emotional stimuli combined with bias toward negative verbal cues could worsen feelings of rejection or alienation in social interactions, therefore increasing risk of developing SI. This suggests that interventions focusing on allocation of attentional resources to process positive social cues may be beneficial for those with SRCs to reduce severity and risk of suicide.

Relationship between neuroimaging and emotion recognition in mild cognitive impairment patients

OBJECTIVE

Dementia is a major public health problem with high needs for early detection, efficient treatment, and prognosis evaluation. Social cognition impairment could be an early dementia indicator and can be assessed with emotion recognition evaluation tests. The purpose of this study is to investigate the link between different brain imaging modalities and cognitive status in Mild Cognitive Impairment (MCI) patients, with the goal of uncovering potential physiopathological mechanisms based on social cognition performance.

METHODS

The relationship between the Reading the Mind in the Eyes Test (RMET) and some clinical and biochemical variables ([18 F]FDG PET-CT and anatomical MR parameters, neuropsychological evaluation, and CSF biomarkers) was studied in 166 patients with MCI by using a correlational approach.

RESULTS

The RMET correlated with neuropsychological variables, as well as with structural and functional brain parameters obtained from the MR and FDG-PET imaging evaluation. However, significant correlations between the RMET and CSF biomarkers were not found.

DISCUSSION

Different neuroimaging parameters were found to be related to an emotion recognition task in MCI. This analysis identified potential minimally-invasive biomarkers providing some knowledge about the physiopathological mechanisms in MCI.

The role of emotion recognition in the intergenerational transmission of child maltreatment: A multigenerational family study

BACKGROUND

Understanding how child maltreatment is passed down from one generation to the next is crucial for the development of intervention and prevention strategies that may break the cycle of child maltreatment. Changes in emotion recognition due to childhood maltreatment have repeatedly been found, and may underly the intergenerational transmission of child maltreatment.

OBJECTIVE

In this study we, therefore, examined whether the ability to recognize emotions plays a role in the intergenerational transmission of child abuse and neglect.

PARTICIPANTS AND SETTING

A total of 250 parents (104 males, 146 females) were included that participated in a three-generation family study.

METHOD

Participants completed an emotion recognition task in which they were presented with series of photographs that depicted the unfolding of facial expressions from neutrality to the peak emotions anger, fear, happiness, and sadness. Multi-informant measures were used to examine experienced and perpetrated child maltreatment.

RESULTS

A history of abuse, but not neglect, predicted a shorter reaction time to identify fear and anger. In addition, parents who showed higher levels of neglectful behavior made more errors in identifying fear, whereas parents who showed higher levels of abusive behavior made more errors in identifying anger. Emotion recognition did not mediate the association between experienced and perpetrated child maltreatment.

CONCLUSIONS

Findings highlight the importance of distinguishing between abuse and neglect when investigating the precursors and sequalae of child maltreatment. In addition, the effectiveness of interventions that aim to break the cycle of abuse and neglect could be improved by better addressing the specific problems with emotion processing of abusive and neglectful parents.

MGFKD: A semi-supervised multi-source domain adaptation algorithm for cross-subject EEG emotion recognition

Currently, most models rarely consider the negative transfer problem in the research field of cross-subject EEG emotion recognition. To solve this problem, this paper proposes a semi-supervised domain adaptive algorithm based on few labeled samples of target subject, which called multi-domain geodesic flow kernel dynamic distribution alignment (MGFKD). It consists of three modules: 1) GFK common feature extractor: projects the feature distribution of source and target subjects to the Grassmann manifold space, and obtains the latent common features of the two feature distributions through GFK method. 2) Source domain selector: obtains pseudo-labels of the target subject through weak classifier, finds "golden source subjects" by using few known labels of target subjects. 3) Label corrector: uses a dynamic distribution balance strategy to correct the pseudo-labels of the target subject. We conducted comparison experiments on the SEED and SEED-IV datasets, and the results show that MGFKD outperforms unsupervised and semi-supervised domain adaptation algorithms, achieving an average accuracy of 87.51±7.68% and 68.79±8.25% on the SEED and SEED-IV datasets with only one labeled sample per video for target subject. Especially when the number of source domains is set as 6 and the number of known labels is set as 5, the accuracy increase to 90.20±7.57% and 69.99±7.38%, respectively. The above results prove that our proposed algorithm can efficiently and quickly improve the cross-subject EEG emotion classification performance. Since it only need a small number of labeled samples of new subjects, making it has strong application value in future EEG-based emotion recognition applications.

Emotion Recognition Deficits in Children and Adolescents with Psychopathic Traits: A Systematic Review

Children and adolescents with psychopathic traits show deficits in emotion recognition, but there is no consensus as to the extent of their generalizability or about the variables that may be moderating the process. The present Systematic Review brings together the existing scientific corpus on the subject and attempts to answer these questions through an exhaustive review of the existing literature according to PRISMA 2020 statement. Results confirmed the existence of pervasive deficits in emotion recognition and, more specifically, on distress emotions (e.g., fear), a deficit that transcends all modalities of emotion presentation and all emotional stimuli used. Moreover, they supported the key role of attention to relevant areas that provide emotional cues (e.g., eye-region) and point out differences according to the presence of disruptive behavior and based on the psychopathy dimension examined. This evidence could advance the current knowledge on developmental models of psychopathic traits. Yet, homogenization of the conditions of research in this area should be prioritized to be able to draw more robust and generalizable conclusions.

The pictorial set of Emotional Social Interactive Scenarios between Chinese Adults (ESISCA): Development and validation

Affective picture databases with a single facial expression or body posture in one image have been widely applied to investigate emotion. However, to date, there was no standardized database containing the stimuli which involve multiple emotional signals in social interactive scenarios. The current study thus developed a pictorial set comprising 274 images depicting two Chinese adults' interactive scenarios conveying emotions of happiness, anger, sadness, fear, disgust, and neutral. The data of the valence and arousal ratings of the scenes and the emotional categories of the scenes and the faces in the images were provided in the present study. Analyses of the data collected from 70 undergraduate students suggested high reliabilities of the valence and arousal ratings of the scenes and high judgmental agreements in categorizing the scene and facial emotions. The findings suggested that the present dataset is well constructed and could be useful for future studies to investigate the emotion recognition or empathy in social interactions in both healthy and clinical (e.g., ASD) populations.

An evaluation of emotion recognition, emotion reactivity, and emotion dysregulation as prospective predictors of 12-month trajectories of non-suicidal self-injury in an adolescent psychiatric inpatient sample

BACKGROUND

Little is known about trajectories of NSSI. We aimed to identify NSSI trajectories in adolescent psychiatric inpatients and emotional processes that differentiate between trajectories.

METHODS

Participants were 180 adolescents (71.7 % female; mean age of 14.89 years, SD = 1.35) from a psychiatric inpatient facility. NSSI was assessed at their index hospitalization, as well as 6, and 12 months after discharge. Emotion recognition, emotion reactivity, and emotion dysregulation were assessed at baseline. Latent class mixture modeling was used to identify different NSSI trajectories and ANOVAs were used to evaluate predictors of the trajectories.

RESULTS

Analyses yielded three NSSI trajectories. These included a stable low-frequency class (90.53 % of sample), a stable moderate-frequency class, and a class characterized by high-frequency NSSI at baseline but that largely resolves by 6-month follow-up. After adjustments for multiple comparisons were made, only emotion regulation at baseline differentiated between the trajectories, with greater overall emotion dysregulation and greater emotional non-acceptance (a facet of emotion dysregulation) characterizing the initially high-frequency class and the stable moderate-frequency class more than the stable low-frequency class (ps < .05). Difficulties engaging in goal-directed behavior when distressed characterized the stable moderate-frequency NSSI class more than the stable low-frequency class (p < .05). Limitations The study sample consists predominantly of female and White adolescents and thus may not generalize to other demographic groups.

CONCLUSIONS

The current findings suggest that interventions involving emotion regulation with adolescents who engage in NSSI would particularly benefit from a focus on increasing acceptance of emotional experiences.

[Research on emotion recognition method based on IWOA-ELM algorithm for electroencephalogram]

Emotion is a crucial physiological attribute in humans, and emotion recognition technology can significantly assist individuals in self-awareness. Addressing the challenge of significant differences in electroencephalogram (EEG) signals among different subjects, we introduce a novel mechanism in the traditional whale optimization algorithm (WOA) to expedite the optimization and convergence of the algorithm. Furthermore, the improved whale optimization algorithm (IWOA) was applied to search for the optimal training solution in the extreme learning machine (ELM) model, encompassing the best feature set, training parameters, and EEG channels. By testing 24 common EEG emotion features, we concluded that optimal EEG emotion features exhibited a certain level of specificity while also demonstrating some commonality among subjects. The proposed method achieved an average recognition accuracy of 92.19% in EEG emotion recognition, significantly reducing the manual tuning workload and offering higher accuracy with shorter training times compared to the control method. It outperformed existing methods, providing a superior performance and introducing a novel perspective for decoding EEG signals, thereby contributing to the field of emotion research from EEG signal.

Social anxiety is associated with personal distress and disrupted recognition of negative emotions

Past research investigating the relation between social anxiety (SA), empathy and emotion recognition is marked by conceptual and methodological issues. In the present study, we aim to overcome these limitations by examining whether individuals with high (n = 40) vs. low (n = 43) social anxiety differed across these two facets of empathy and whether this could be related to their recognition of emotions. We employed a naturalistic emotion recognition paradigm in which participants watched short videos of individuals (targets) sharing authentic emotional experiences. After each video, we measured self-reported empathic concern and distress, as well as their ability to recognize the emotions expressed by the targets in the videos. Our results show that individuals with high social anxiety recognized the targets' emotions less accurately. Furthermore, high socially anxious individuals reported more personal distress than low socially anxious individuals, whereas no significant difference was found for empathic concern. The findings suggest that reduced recognition of emotions among SA individuals can be better explained by the negative effects of social stress than by a general deficit in empathy.

Research on driver's anger recognition method based on multimodal data fusion

OBJECTIVES

This paper aims to address the challenge of low accuracy in single-modal driver anger recognition by introducing a multimodal driver anger recognition model. The primary objective is to develop a multimodal fusion recognition method for identifying driver anger, focusing on electrocardiographic (ECG) signals and driving behavior signals.

METHODS

Emotion-inducing experiments were performed employing a driving simulator to capture both ECG signals and driving behavioral signals from drivers experiencing both angry and calm moods. An analysis of characteristic relationships and feature extraction was conducted on ECG signals and driving behavior signals related to driving anger. Seventeen effective feature indicators for recognizing driving anger were chosen to construct a dataset for driver anger. A binary classification model for recognizing driving anger was developed utilizing the Support Vector Machine (SVM) algorithm.

RESULTS

Multimodal fusion demonstrated significant advantages over single-modal approaches in emotion recognition. The SVM-DS model using decision-level fusion had the highest accuracy of 84.75%. Compared with the driver anger emotion recognition model based on unimodal ECG features, unimodal driving behavior features, and multimodal feature layer fusion, the accuracy increased by 9.10%, 4.15%, and 0.8%, respectively.

CONCLUSIONS

The proposed multimodal recognition model, incorporating ECG and driving behavior signals, effectively identifies driving anger. The research results provide theoretical and technical support for the establishment of a driver anger system.

SENSES-ASD: a social-emotional nurturing and skill enhancement system for autism spectrum disorder

This article introduces the Social-Emotional Nurturing and Skill Enhancement System (SENSES-ASD) as an innovative method for assisting individuals with autism spectrum disorder (ASD). Leveraging deep learning technologies, specifically convolutional neural networks (CNN), our approach promotes facial emotion recognition, enhancing social interactions and communication. The methodology involves the use of the Xception CNN model trained on the FER-2013 dataset. The designed system accepts a variety of media inputs, successfully classifying and predicting seven primary emotional states. Results show that our system achieved a peak accuracy rate of 71% on the training dataset and 66% on the validation dataset. The novelty of our work lies in the intricate combination of deep learning methods specifically tailored for high-functioning autistic adults and the development of a user interface that caters to their unique cognitive and sensory sensitivities. This offers a novel perspective on utilising technological advances for ASD intervention, especially in the domain of emotion recognition.

Facial emotion recognition in individuals with mild cognitive impairment: An exploratory study

Understanding facial emotions is fundamental to interact in social environments and modify behavior accordingly. Neurodegenerative processes can progressively transform affective responses and affect social competence. This exploratory study examined the neurocognitive correlates of face recognition, in individuals with two mild cognitive impairment (MCI) etiologies (prodromal to dementia - MCI, or consequent to Parkinson's disease - PD-MCI). Performance on the identification and memorization of neutral and emotional facial expressions was assessed in 31 individuals with MCI, 26 with PD-MCI, and 30 healthy controls (HC). Individuals with MCI exhibited selective impairment in recognizing faces expressing fear, along with difficulties in remembering both neutral and emotional faces. Conversely, individuals with PD-MCI showed no differences compared with the HC in either emotion recognition or memory. In MCI, no significant association emerged between the memory for facial expressions and cognitive difficulties. In PD-MCI, regression analyses showed significant associations with higher-level cognitive functions in the emotional memory task, suggesting the presence of compensatory mechanisms. In a subset of participants, voxel-based morphometry revealed that the performance on emotional tasks correlated with regional changes in gray matter volume. The performance in the matching of negative expressions was predicted by volumetric changes in brain areas engaged in face and emotional processing, in particular increased volume in thalamic nuclei and atrophy in the right parietal cortex. Future studies should leverage on neuroimaging data to determine whether differences in emotional recognition are mediated by pathology-specific atrophic patterns.

An emotion recognition method based on EWT-3D-CNN-BiLSTM-GRU-AT model

This has become a significant study area in recent years because of its use in brain-machine interaction (BMI). The robustness problem of emotion classification is one of the most basic approaches for improving the quality of emotion recognition systems. One of the two main branches of these approaches deals with the problem by extracting the features using manual engineering and the other is the famous artificial intelligence approach, which infers features of EEG data. This study proposes a novel method that considers the characteristic behavior of EEG recordings and based on the artificial intelligence method. The EEG signal is a noisy signal with a non-stationary and non-linear form. Using the Empirical Wavelet Transform (EWT) signal decomposition method, the signal's frequency components are obtained. Then, frequency-based features, linear and non-linear features are extracted. The resulting frequency-based, linear, and nonlinear features are mapped to the 2-D axis according to the positions of the EEG electrodes. By merging this 2-D images, 3-D images are constructed. In this way, the multichannel brain frequency of EEG recordings, spatial and temporal relationship are combined. Lastly, 3-D deep learning framework was constructed, which was combined with convolutional neural network (CNN), bidirectional long-short term memory (BiLSTM) and gated recurrent unit (GRU) with self-attention (AT). This model is named EWT-3D-CNN-BiLSTM-GRU-AT. As a result, we have created framework comprising handcrafted features generated and cascaded from state-of-the-art deep learning models. The framework is evaluated on the DEAP recordings based on the person-independent approach. The experimental findings demonstrate that the developed model can achieve classification accuracies of 90.57 % and 90.59 % for valence and arousal axes, respectively, for the DEAP database. Compared with existing cutting-edge emotion classification models, the proposed framework exhibits superior results for classifying human emotions.

Alexithymia and facial emotion recognition in patients with functional neurological disorder

OBJECTIVES

Patients with functional neurological disorder (FND) are known to have difficulties recognizing and processing emotions. Problems recognizing internal emotional states (alexithymia) are common in FND, but little is known about recognizing emotions expressed by other people. This study investigates whether patients with FND have higher levels of alexithymia and reduced facial emotion recognition compared to healthy controls.

METHODS

Patients with FND (n = 31, mean age=42.7 [SD=14.8] years, 54.8% women) were compared to healthy controls (n = 33, mean age=45.1 [SD=16.2] years, 63.6% women). The Bermond-Vorst Alexithymia Questionnaire (BVAQ) was used for the assessment of alexithymia and the Ekman 60 Faces Test (EFT) for facial emotion recognition.

RESULTS

Patients with FND had higher levels of alexithymia than healthy controls (BVAQ=71.8 [SD=19.8] versus 59.3 [SD=20.3], p = .02, Cohen's d=0.62). Facial emotion recognition did not significantly differ between FND patients and controls (EFT total score FND: 46.1 [SD=5.9], Controls: 47.5 [SD=5.5], p = .34, Cohen's d=0.24). Only recognition of surprise differed between patients and controls (FND: 8.4 [SD=1.8], Controls: 9.2 [SD=1.0), p = .03, Cohen's d= 0.56). Higher levels of alexithymia were associated with poorer facial emotion recognition, but this relationship was not statistically significant (FND: β= -0.20, p = .28; Controls: β=-0.03; p = .87).

CONCLUSIONS

The current data confirm prior observations that patients with FND have higher alexithymia levels than controls without FND. Difficulties recognizing emotions among patients with FND primarily involves recognition of internal emotional states rather than recognition of facially expressed emotions by others. These findings require replication in a larger and more divers sample.

EEG based emotion recognition by hierarchical bayesian spectral regression framework

Spectral regression (SR), a graph-based learning regression model, can be used to extract features from graphs to realize efficient dimensionality reduction. However, due to the SR method remains a regularized least squares problem and being defined in L2-norm space, the effect of artifacts in EEG signals cannot be efficiently resisted. In this work, to further improve the robustness of the graph-based regression models, we propose to utilize the prior distribution estimation in the Bayesian framework and develop a robust hierarchical Bayesian spectral regression framework (named HB-SR), which is designed with the hierarchical Bayesian ensemble strategies. In the proposed HB-SR, the impact of noises can be effectively reduced by the adaptive adjustment approach in model parameters with the data-driven manner. Specifically, in the current work, three different distributions have been elaborately designed to enhance the universality of the proposed HB-SR, i.e., Gaussian distribution, Laplace distribution, and Student-t distribution. To objectively evaluate the performance of the HB-SR framework, we conducted both simulation studies and emotion recognition experiments based on emotional EEG signals. Experimental results have consistently indicated that compared with other existing spectral regression methods, the proposed HB-SR can effectively suppress the influence of noises and achieve robust EEG emotion recognition.

Instance-representation transfer method based on joint distribution and deep adaptation for EEG emotion recognition

Electroencephalogram (EEG) emotion recognition technology is essential for improving human-computer interaction. However, the practical application of emotion recognition technology is limited due to the variety of subjects and sessions. Transfer learning has been applied to address this issue and has received extensive research and application. Studies mainly concentrate on either instance transfer or representation transfer methods. This paper proposes an emotion recognition method called Joint Distributed Instances Represent Transfer (JD-IRT), which includes two core components: Joint Distribution Deep Adaptation (JDDA) and Instance-Representation Transfer (I-RT). Specifically, JDDA is different from common representation transfer methods in transfer learning. It bridges the discrepancies of marginal and conditional distributions simultaneously and combines multiple adaptive layers and kernels for deep domain adaptation. On the other hand, I-RT utilizes instance transfer to select source domain data for better representation transfer. We performed experiments and compared them with other representative methods in the SEED, SEED-IV, and SEED-V datasets. In cross-subject experiments, our approach achieved an average accuracy of 83.21% in SEED, 52.12% in SEED-IV, and 60.17% in SEED-V. Similarly, in cross-session experiments, the accuracy was 91.29% in SEED, 59.02% in SEED-IV, and 65.91% in SEED-V. These results demonstrate the improvement in the accuracy of EEG emotion recognition using the proposed approach.

Graph theoretical brain connectivity measures to investigate neural correlates of music rhythms associated with fear and anger

The present study tests the hypothesis that emotions of fear and anger are associated with distinct psychophysiological and neural circuitry according to discrete emotion model due to contrasting neurotransmitter activities, despite being included in the same affective group in many studies due to similar arousal-valance scores of them in emotion models. EEG data is downloaded from OpenNeuro platform with access number of ds002721. Brain connectivity estimations are obtained by using both functional and effective connectivity estimators in analysis of short (2 sec) and long (6 sec) EEG segments across the cortex. In tests, discrete emotions and resting-states are identified by frequency band specific brain network measures and then contrasting emotional states are deep classified with 5-fold cross-validated Long Short Term Memory Networks. Logistic regression modeling has also been examined to provide robust performance criteria. Commonly, the best results are obtained by using Partial Directed Coherence in Gamma (31.5 - 60.5 H z ) sub-bands of short EEG segments. In particular, Fear and Anger have been classified with accuracy of 91.79%. Thus, our hypothesis is supported by overall results. In conclusion, Anger is found to be characterized by increased transitivity and decreased local efficiency in addition to lower modularity in Gamma-band in comparison to fear. Local efficiency refers functional brain segregation originated from the ability of the brain to exchange information locally. Transitivity refer the overall probability for the brain having adjacent neural populations interconnected, thus revealing the existence of tightly connected cortical regions. Modularity quantifies how well the brain can be partitioned into functional cortical regions. In conclusion, PDC is proposed to graph theoretical analysis of short EEG epochs in presenting robust emotional indicators sensitive to perception of affective sounds.

Healthy and Happy? An Ethical Investigation of Emotion Recognition and Regulation Technologies (ERR) within Ambient Assisted Living (AAL)

Ambient Assisted Living (AAL) refers to technologies that track daily activities of persons in need of care to enhance their autonomy and minimise their need for assistance. New technological developments show an increasing effort to integrate automated emotion recognition and regulation (ERR) into AAL systems. These technologies aim to recognise emotions via different sensors and, eventually, to regulate emotions defined as "negative" via different forms of intervention. Although these technologies are already implemented in other areas, AAL stands out by its tendency to enable an inconspicuous 24-hour surveillance in the private living space of users who rely on the technology to maintain a certain degree of independence in their daily activities. The combination of both technologies represents a new dimension of emotion recognition in a potentially vulnerable group of users. Our paper aims to provide an ethical contextualisation of the novel combination of both technologies. We discuss different concepts of emotions, namely Basic Emotion Theory (BET) and the Circumplex Model of Affect (CMA), that form the basis of ERR and provide an overview over the current technological developments in AAL. We highlight four ethical issues that specifically arise in the context of ERR in AAL systems, namely concerns regarding (1) the reductionist view of emotions, (2) solutionism as an underlying assumption of these technologies, (3) the privacy and autonomy of users and their emotions, (4) the tendency of machine learning techniques to normalise and generalise human behaviour and emotional reactions.

Cerebral and cerebellar correlates of social cognitive impairment in schizophrenia

Social cognition is a broad construct encompassing the ways in which individuals perceive, process, and use information about other people. Social cognition involves both lower- and higher-level processes such as emotion recognition and theory of mind, respectively. Social cognitive impairments have been repeatedly demonstrated in schizophrenia spectrum illnesses and, crucially, are related to functional outcomes. In this review, we summarize the literature investigating the brain networks implicated in social cognitive impairments in schizophrenia spectrum illnesses. In addition to cortical and limbic loci and networks, we also discuss evidence for cerebellar contributions to social cognitive impairment in this population. We conclude by synthesizing these two literatures, with an emphasis on current knowledge gaps, particularly in regard to cerebellar influences, and future directions.

Sneaky emotions: impact of data partitions in affective computing experiments with brain-computer interfacing

Brain-Computer Interfacing (BCI) has shown promise in Machine Learning (ML) for emotion recognition. Unfortunately, how data are partitioned in training/test splits is often overlooked, which makes it difficult to attribute research findings to actual modeling improvements or to partitioning issues. We introduce the "data transfer rate" construct (i.e., how much data of the test samples are seen during training) and use it to examine data partitioning effects under several conditions. As a use case, we consider emotion recognition in videos using electroencephalogram (EEG) signals. Three data splits are considered, each representing a relevant BCI task: subject-independent (affective decoding), video-independent (affective annotation), and time-based (feature extraction). Model performance may change significantly (ranging e.g. from 50% to 90%) depending on how data is partitioned, in classification accuracy. This was evidenced in all experimental conditions tested. Our results show that (1) for affective decoding, it is hard to achieve performance above the baseline case (random classification) unless some data of the test subjects are considered in the training partition; (2) for affective annotation, having data from the same subject in training and test partitions, even though they correspond to different videos, also increases performance; and (3) later signal segments are generally more discriminative, but it is the number of segments (data points) what matters the most. Our findings not only have implications in how brain data are managed, but also in how experimental conditions and results are reported.

DSE-Mixer: A pure multilayer perceptron network for emotion recognition from EEG feature maps

BACKGROUND

Decoding emotions from brain maps is a challenging task. Convolutional Neural Network (CNN) is commonly used for EEG feature map. However, due to its local bias, CNN is unable to efficiently utilize the global spatial information of EEG signals which limits the accuracy of emotion recognition.

NEW METHODS

We design the Dual-scal EEG-Mixer(DSE-Mixer) model for EEG feature map processing. Its brain region mixer layer and electrode mixer layer are designed to fuse EEG information at different spatial scales. For each mixer layer, the structure of alternating mixing of rows and columns of the input table enables cross-regional and cross-Mchannel communication of EEG information. In addition, a channel attention mechanism is introduced to adaptively learn the importance of each channel.

RESULTS

On the DEAP dataset, the DSE-Mixer model achieved a binary classification accuracy of 95.19% for arousal and 95.22% for valence. For the four-class classification across valence and arousal, the accuracies were HVHA: 92.12%, HVLA: 89.77%, LVLA: 93.35%, and LVHA: 92.63%. On the SEED dataset, the average recognition accuracy for the three emotions (positive, negative, and neutral) is 93.69%.

COMPARISON WITH EXISTING METHODS

In the emotion recognition research based on the DEAP and SEED datasets, DSE-Mixer achieved a high ranking performance. Compared to the two commonly used model in computer vision field, CNN and Vision Transformer(VIT), DSE-Mixer achieved significantly higher classification accuracy while requiring much less computational complexity.

CONCLUSIONS

DSE-Mixer provides a novel brain map processing model with a small size, demonstrating outstanding performance in emotion recognition.

The Association Between Emotion Recognition and Internalizing Problems in Children and Adolescents: A Three-Level Meta-Analysis

Numerous studies have explored the link between how well youth recognize emotions and their internalizing problems, but a consensus remains elusive. This study used a three-level meta-analysis model to quantitatively synthesize the findings of existing studies to assess the relationship. A moderation analysis was also conducted to explore the sources of research heterogeneity. Through a systematic literature search, a total of 42 studies with 201 effect sizes were retrieved for the current meta-analysis, and 7579 participants were included. Emotion recognition was negatively correlated with internalizing problems. Children and adolescents with weaker emotion recognition skills were more likely to have internalizing problems. In addition, this meta-analysis found that publication year had a significant moderating effect. The correlation between emotion recognition and internalizing problems decreased over time. The degree of internalizing problems was also found to be a significant moderator. The correlation between emotion recognition and internalizing disorders was higher than the correlation between emotion recognition and internalizing symptoms. Deficits in emotion recognition might be relevant for the development and/or maintenance of internalizing problems in children and adolescents. The overall effect was small and future research should explore the clinical relevance of the association.

Mentalizing can be Impaired in Patients with Meningiomas Originating in the Anterior Skull Base

OBJECTIVE

Mentalizing is an essential function of our social lives. Impairment of mentalizing due to meningiomas has not received attention because most patients return to their social lives after surgical treatment. We investigated the influence of meningiomas and their surgical resection on mentalizing.

METHODS

Low- and high-level mentalizing were retrospectively examined in 61 patients with meningiomas and 14 healthy volunteers. Mentalizing was assessed using the facial expression recognition test and picture arrangement test of the Wechsler Adult Intelligence Scale, third edition, before and after surgery. We examined the influence of tumor localization on mentalizing and recovery from mentalizing disorders after tumor resection. Voxel-based lesion-symptom mapping was performed to investigate the relationship between impairments in mentalizing and tumor location.

RESULTS

Before surgery, mentalizing was impaired significantly in patients with meningiomas compared to those in the control group (low-level: P = 0.015, high-level: P = 0.011). This impairment was associated with contact between the tumor and frontal lobe (low-level: P = 0.036, high-level: P = 0.047) and was severe in patients with tumors arising in the anterior skull base (low-level: P = 0.0045, high-level: P = 0.043). Voxel-based lesion-symptom mapping revealed that when the basal cortex of the frontal lobe was compressed by the tumor, the risk of impaired mentalizing was high. The region responsible for high-level mentalizing was located deeper than that responsible for low-level mentalizing. After the surgical removal of the tumor, the test scores significantly improved (low-level: P = 0.035, high-level: P = 0.045).

CONCLUSIONS

Mentalizing was impaired by meningiomas arising from the anterior skull base, but it can improve after surgical resection of the tumors.

S-LSTM-ATT: a hybrid deep learning approach with optimized features for emotion recognition in electroencephalogram

Purpose

Human emotion recognition using electroencephalograms (EEG) is a critical area of research in human-machine interfaces. Furthermore, EEG data are convoluted and diverse; thus, acquiring consistent results from these signals remains challenging. As such, the authors felt compelled to investigate EEG signals to identify different emotions.

Methods

A novel deep learning (DL) model stacked long short-term memory with attention (S-LSTM-ATT) model is proposed for emotion recognition (ER) in EEG signals. Long Short-Term Memory (LSTM) and attention networks effectively handle time-series EEG data and recognise intrinsic connections and patterns. Therefore, the model combined the strengths of the LSTM model and incorporated an attention network to enhance its effectiveness. Optimal features were extracted from the metaheuristic-based firefly optimisation algorithm (FFOA) to identify different emotions efficiently.

Results

The proposed approach recognised emotions in two publicly available standard datasets: SEED and EEG Brainwave. An outstanding accuracy of 97.83% in the SEED and 98.36% in the EEG Brainwave datasets were obtained for three emotion indices: positive, neutral and negative. Aside from accuracy, a comprehensive comparison of the proposed model's precision, recall, F1 score and kappa score was performed to determine the model's applicability. When applied to the SEED and EEG Brainwave datasets, the proposed S-LSTM-ATT achieved superior results to baseline models such as Convolutional Neural Networks (CNN), Gated Recurrent Unit (GRU) and LSTM.

Conclusion

Combining an FFOA-based feature selection (FS) and an S-LSTM-ATT-based classification model demonstrated promising results with high accuracy. Other metrics like precision, recall, F1 score and kappa score proved the suitability of the proposed model for ER in EEG signals.

STSNet: a novel spatio-temporal-spectral network for subject-independent EEG-based emotion recognition

How to use the characteristics of EEG signals to obtain more complementary and discriminative data representation is an issue in EEG-based emotion recognition. Many studies have tried spatio-temporal or spatio-spectral feature fusion to obtain higher-level representations of EEG data. However, these studies ignored the complementarity between spatial, temporal and spectral domains of EEG signals, thus limiting the classification ability of models. This study proposed an end-to-end network based on ManifoldNet and BiLSTM networks, named STSNet. The STSNet first constructed a 4-D spatio-temporal-spectral data representation and a spatio-temporal data representation based on EEG signals in manifold space. After that, they were fed into the ManifoldNet network and the BiLSTM network respectively to calculate higher-level features and achieve spatio-temporal-spectral feature fusion. Finally, extensive comparative experiments were performed on two public datasets, DEAP and DREAMER, using the subject-independent leave-one-subject-out cross-validation strategy. On the DEAP dataset, the average accuracy of the valence and arousal are 69.38% and 71.88%, respectively; on the DREAMER dataset, the average accuracy of the valence and arousal are 78.26% and 82.37%, respectively. Experimental results show that the STSNet model has good emotion recognition performance.