Neural signals evoked by stimuli of increasing social scene complexity are detectable at the single-trial level and right lateralized

From Neural Networks to Emotional Networks: A Systematic Review of EEG-Based Emotion Recognition in Cognitive Neuroscience and Real-World Applications

BACKGROUND/OBJECTIVES

This systematic review presents how neural and emotional networks are integrated into EEG-based emotion recognition, bridging the gap between cognitive neuroscience and practical applications.

METHODS

Following PRISMA, 64 studies were reviewed that outlined the latest feature extraction and classification developments using deep learning models such as CNNs and RNNs.

RESULTS

Indeed, the findings showed that the multimodal approaches were practical, especially the combinations involving EEG with physiological signals, thus improving the accuracy of classification, even surpassing 90% in some studies. Key signal processing techniques used during this process include spectral features, connectivity analysis, and frontal asymmetry detection, which helped enhance the performance of recognition. Despite these advances, challenges remain more significant in real-time EEG processing, where a trade-off between accuracy and computational efficiency limits practical implementation. High computational cost is prohibitive to the use of deep learning models in real-world applications, therefore indicating a need for the development and application of optimization techniques. Aside from this, the significant obstacles are inconsistency in labeling emotions, variation in experimental protocols, and the use of non-standardized datasets regarding the generalizability of EEG-based emotion recognition systems.

DISCUSSION

These challenges include developing adaptive, real-time processing algorithms, integrating EEG with other inputs like facial expressions and physiological sensors, and a need for standardized protocols for emotion elicitation and classification. Further, related ethical issues with respect to privacy, data security, and machine learning model biases need to be much more proclaimed to responsibly apply research on emotions to areas such as healthcare, human-computer interaction, and marketing.

CONCLUSIONS

This review provides critical insight into and suggestions for further development in the field of EEG-based emotion recognition toward more robust, scalable, and ethical applications by consolidating current methodologies and identifying their key limitations.

A neurophysiological signature of dynamic emotion recognition associated with social communication skills and cortical gamma-aminobutyric acid levels in children

Introduction

Emotion recognition is a core feature of social perception. In particular, perception of dynamic facial emotional expressions is a major feature of the third visual pathway. However, the classical N170 visual evoked signal does not provide a pure correlate of such processing. Indeed, independent component analysis has demonstrated that the N170 component is already active at the time of the P100, and is therefore distorted by early components. Here we implemented, a dynamic face emotional paradigm to isolate a more pure face expression selective N170. We searched for a neural correlate of perception of dynamic facial emotional expressions, by starting with a face baseline from which a facial expression evolved. This allowed for a specific facial expression contrast signal which we aimed to relate with social communication abilities and cortical gamma-aminobutyric acid (GABA) levels.

Methods

We recorded event-related potentials (ERPs) and Magnetic Resonance (MRS) measures in 35 typically developing (TD) children, (10-16 years) sex-matched, during emotion recognition of an avatar morphing/unmorphing from neutral to happy/sad expressions. This task allowed for the elimination of the contribution low-level visual components, in particular the P100, by morphing baseline isoluminant neutral faces into specific expressions, isolating dynamic emotion recognition. Therefore, it was possible to isolate a dynamic face sensitive N170 devoid of interactions with earlier components.

Results

We found delayed N170 and P300, with a hysteresis type of dependence on stimulus trajectory (morphing/unmorphing), with hemispheric lateralization. The delayed N170 is generated by an extrastriate source, which can be related to the third visual pathway specialized in biological motion processing. GABA levels in visual cortex were related with N170 amplitude and latency and predictive of worse social communication performance (SCQ scores). N170 latencies reflected delayed processing speed of emotional expressions and related to worse social communication scores.

Discussion

In sum, we found a specific N170 electrophysiological signature of dynamic face processing related to social communication abilities and cortical GABA levels. These findings have potential clinical significance supporting the hypothesis of a spectrum of social communication abilities and the identification of a specific face-expression sensitive N170 which can potentially be used in the development of diagnostic and intervention tools.

Enhanced Mirror Neuron Network Activity and Effective Connectivity during Live Interaction Among Female Subjects

Facial expressions are indispensable in daily human communication. Previous neuroimaging studies investigating facial expression processing have presented pre-recorded stimuli and lacked live face-to-face interaction. Our paradigm alternated between presentations of real-time model performance and pre-recorded videos of dynamic facial expressions to participants. Simultaneous functional magnetic resonance imaging (fMRI) and facial electromyography activity recordings, as well as post-scan valence and arousal ratings were acquired from 44 female participants. Live facial expressions enhanced the subjective valence and arousal ratings as well as facial muscular responses. Live performances showed greater engagement of the right posterior superior temporal sulcus (pSTS), right inferior frontal gyrus (IFG), right amygdala and right fusiform gyrus, and modulated the effective connectivity within the right mirror neuron system (IFG, pSTS, and right inferior parietal lobule). A support vector machine algorithm could classify multivoxel activation patterns in brain regions involved in dynamic facial expression processing in the mentalizing networks (anterior and posterior cingulate cortex). These results indicate that live social interaction modulates the activity and connectivity of the right mirror neuron system and enhances spontaneous mimicry, further facilitating emotional contagion.

Event-Related Potentials during Verbal Recognition of Naturalistic Neutral-to-Emotional Dynamic Facial Expressions

Event-related potentials during facial emotion recognition have been studied for more than twenty years. Nowadays, there has been a growing interest in the use of naturalistic stimuli. This research was aimed, therefore, at studying event-related potentials (ERP) during recognition of dynamic facial neutral-to-emotional expressions, more ecologically valid than static faces. We recorded the ERP of 112 participants who watched 144 dynamic morphs depicting a gradual change from a neutral expression to a basic emotional expression (anger, disgust, fear, happiness, sadness and surprise) and labelled those emotions verbally. We revealed some typical ERP, like N170, P2, EPN and LPP. Participants with lower accuracy exhibited a larger posterior P2. Participants with faster correct responses exhibited a larger amplitude of P2 and LPP. We also conducted a classification analysis that yielded the accuracy of 76% for prediction of participants who recognise emotions quickly on the basis of the amplitude of posterior P2 and LPP. These results extend data from previous research about the electroencephalographic correlates of facial emotion recognition.

A Bayesian-optimized design for an interpretable convolutional neural network to decode and analyze the P300 response in autism

OBJECTIVE

P300 can be analyzed in autism spectrum disorder (ASD) to derive biomarkers and can be decoded in BCIs to reinforce ASD impaired skills. Convolutional neural networks (CNNs) have been proposed for P300 decoding, outperforming traditional algorithms but they i) do not investigate optimal designs in different training conditions; ii) lack in interpretability. To overcome these limitations, an interpretable CNN (ICNN), that we recently proposed for motor decoding, has been modified and adopted here, with its optimal design searched via Bayesian optimization.

APPROACH

The ICNN provides a straightforward interpretation of spectral and spatial features learned to decode P300. The Bayesian-optimized (BO) ICNN design was investigated separately for different training strategies (within-subject, within-session, and cross-subject) and BO models were used for the subsequent analyses. Specifically, transfer learning (TL) potentialities were investigated by assessing how pretrained cross-subject BO models performed on a new subject vs. random-initialized models. Furthermore, within-subject BO-derived models were combined with an Explanation Technique (ICNN+ET) to analyze P300 spectral and spatial features.

MAIN RESULTS

The ICNN resulted comparable or even outperformed existing CNNs, at the same time being lighter. Bayesian-optimized ICNN designs differed depending on the training strategy, needing more capacity as the training set variability increased. Furthermore, TL provided higher performance than networks trained from scratch. The ICNN+ET analysis suggested the frequency range [2, 5.8] Hz as the most relevant, and spatial features showed a right-hemispheric parietal asymmetry. The ICNN+ET-derived features, but not ERP-derived features, resulted significantly and highly correlated to ADOS clinical scores.

SIGNIFICANCE

This study substantiates the idea that a CNN can be designed both accurate and interpretable for P300 decoding, with an optimized design depending on the training condition. The novel ICNN-based analysis tool was able to better capture ASD neural signatures than traditional ERP analysis, possibly paving the way for identifying novel biomarkers.

Feedback processing is enhanced following exploration in continuous environments

Decision-making is typically studied by presenting participants with a small set of options. However, real-world behaviour, like foraging, often occurs in continuous environments. The degree to which human decision-making in discrete tasks generalizes to continuous tasks is questionable. For example, successful foraging comprises both exploration (learning about the environment) and exploitation (taking advantage of what is known). Although progress has been made in understanding the neural processes related to this trade-off in discrete tasks, it is currently unclear how, or whether, the same processes are involved in continuous tasks. To address this, we recorded electroencephalographic data while participants "dug for gold" by selecting locations on a map. Participants were cued beforehand that the map contained either a single patch of gold, or many patches of gold. We then used a computational model to classify participant responses as either exploitations, which were driven by previous reward locations and amounts, or explorations. Our participants were able to adjust their strategy based on reward distribution, exploring more in multi-patch environments and less in single-patch environments. We observed an enhancement of the feedback-locked P300, a neural signal previously linked to exploration in discrete tasks, which suggests the presence of a general neural system for managing the explore-exploit trade-off. Furthermore, the P300 was accompanied by an exploration-related enhancement of the late positive potential that was greatest in the multi-patch environment, suggesting a role for motivational processes during exploration.

Stimulus dependent neural oscillatory patterns show reliable statistical identification of autism spectrum disorder in a face perceptual decision task

OBJECTIVE

Electroencephalographic biomarkers have been widely investigated in autism, in the search for diagnostic, prognostic and therapeutic outcome measures. Here we took advantage of the information available in temporal oscillatory patterns evoked by simple perceptual decisions to investigate whether stimulus dependent oscillatory signatures can be used as potential biomarkers in autism spectrum disorder (ASD).

METHODS

We studied an extensive set of stimuli (9 categories of faces) and performed data driven classification (Support vector machine, SVM) of ASD vs. Controls with features based on the EEG power responses. We carried out an extensive time-frequency and synchrony analysis of distinct face categories requiring different processing mechanisms in terms of non-holistic vs. holistic processing.

RESULTS

We found that the neuronal oscillatory responses of low gamma frequency band, locked to photographic and abstract two-tone (Mooney) face stimulus presentation are decreased in ASD vs. the control group. We also found decreased time-frequency (TF) responses in the beta band in ASD after 350 ms, possibly related to motor preparation. On the other hand, synchrony in the 30-45 Hz band showed a distinct spatial pattern in ASD. These power changes enabled accurate classification of ASD with an SVM approach. SVM accuracy was approximately 85%. ROC curves showed about 94% AUC (area under the curve). Combination of Mooney and Photographic face stimuli evoked features enabled a better separation between groups, reaching an AUC of 98.6%.

CONCLUSION

We identified a relative decrease in EEG responses to face stimuli in ASD in the beta (15-30 Hz; >350 ms) and gamma (30-45 Hz; 55-80 Hz; 50-350 ms) frequency ranges. These can be used as input of a machine learning approach to separate between groups with high accuracy.

SIGNIFICANCE

Future studies can use EEG time-frequency patterns evoked by particular types of faces as a diagnostic biomarker and potentially as outcome measures in therapeutic trials.