The Effects of Dynamic and Static Emotional Facial Expressions of Humans and Their Avatars on the EEG: An ERP and ERD/ERS Study

Spatiotemporal processing of real faces is modified by visual sensing

Live human faces, when engaged as visual stimuli, recruit unique and extensive patterns of neural activity. However, the underlying neural mechanisms that underly these live face-to-face processes are not known. We hypothesized that the neural correlates for live face processes are modulated by both spatial and temporal features of the live faces as well as visual sensing parameters. Hemodynamic signals detected by functional near infrared spectroscopy (fNIRS) were acquired concurrently with co-activated electroencephalographic (EEG) and eye-tracking signals during interactive gaze at a live human face or gaze at a human-like robot face. Regression of the fNIRS signals with two eye-gaze variables, fixation duration and dwell time, revealed separate regions of neural correlates, right supramarginal gyrus (lateral visual stream) and right inferior parietal sulcus (dorsal visual stream), respectively. These two areas served as the regions of interest for the EEG analysis. Standardized low-resolution brain electromagnetic tomography (sLORETA) was applied to determine theta (4 - 7 Hz) and alpha (8-13 Hz) oscillatory activity in these regions. Variations in oscillatory patterns corresponding to the neural correlates of the visual sensing parameters suggest an increase in spatial binding for the dorsal relative to the lateral regions of interest during live face-to-face visual stimulation.

Motor imagery EEG decoding based on TS-former for spinal cord injury patients

Traditional machine learning methods struggle with efficiency when processing large-scale data, while deep learning approaches, such as convolutional neural networks (CNN) and long short-term memory networks (LSTM), exhibit certain limitations when handling long-duration sequences. The choice of convolutional kernel size needs to be determined after several experiments, and LSTM has difficulty capturing effective information from long-time sequences. In this paper, we propose a transfer learning (TL) method based on Transformer, which constructs a new network architecture for feature extraction and classification of electroencephalogram (EEG) signals in the time-space domain, named TS-former. The frequency and spatial domain information of EEG signals is extracted using the Filter Bank Common Spatial Pattern (FBCSP), and the resulting features are subsequently processed by the Transformer to capture temporal patterns. The input features are processed by the Transformer using a multi-head attention mechanism, and the final classification outputs are generated through a fully connected layer. A classification model is pre-trained using fine-tuning techniques. When performing a new classification task, only some layers of the model are modified to adapt it to the new data and achieve good classification results. The experiments are conducted on a motor imagery (MI) EEG dataset from 16 spinal cord injury (SCI) patients. After training the model using a ten-time ten-fold cross-validation method, the average classification accuracy reached 95.09 %. Our experimental results confirm a new approach to build a brain-computer interface (BCI) system for rehabilitation training of SCI patients.

Subthalamic nucleus oscillations during facial emotion processing and apathy in Parkinson's disease

BACKGROUND

Parkinson's disease (PD) is primarily characterized by motor symptoms, but patients also experience a relatively high prevalence of non-motor symptoms, including emotional and cognitive impairments. While the subthalamic nucleus (STN) is a common target for deep brain stimulation to treat motor symptoms in PD, its role in emotion processing is still under investigation. This study examines the subthalamic neural oscillatory activities during facial emotion processing and its association with affective characteristics.

METHODS

Twenty PD patients who underwent subthalamic deep brain stimulation surgery performed a facial-expression-recognition task while STN local field potential (LFP) and frontal electroencephalography (EEG) were recorded. The facial-emotion-induced time-frequency decomposition of the STN-LFP and the frontal EEG, as well as the LFP-EEG coherence, were analyzed. Furthermore, the correlation between STN activities and affective characteristics was examined.

RESULTS

Facial expressions elicited increased delta-theta-band and decreased alpha-beta-band activities in STN-LFP. Reduced alpha-beta-band LFP desynchronization was correlated with the severity of apathy. Increased theta-band and decreased alpha-beta-band EEG activities responded to facial emotion. Notably, lower coherence between STN-LFP and frontal EEG in delta-theta-band activity and alpha-band activity correlated with the degree of anhedonia.

CONCLUSION

These results indicate that subthalamic activities during facial emotion processing are associated with apathy and anhedonia, emphasizing the cognitive-limbic function of STN and its role as a physiological target for apathy neuromodulation in PD.

The neuronal signature of surprised facial expression processing under different attentional focuses: A time-domain and time-frequency study

Previous studies have shown that high-arousal positive and negative facial expressions influence event-related potential (ERP) and time-frequency responses depending on attentional focuses. However, little is known about how relevant neural responses are influenced by surprised facial expressions, which are also high in arousal but ambiguous in valence. To address the issue, 38 participants were presented with surprised, happy, angry and neutral facial expressions. Attention was manipulated to focus on facial emotional attributes, facial non-emotional attributes, non-facial attributes, or was free to the participants. ERP results showed larger N170 responses to surprised compared to neutral facial expressions when attention focused on facial attributes and to surprised compared to angry and neutral facial expressions when attention focused on facial non-emotional attributes. Time-frequency analyses revealed reduced power of early occipital theta to surprised compared to happy and angry expressions when attention focused on facial emotions and to surprised compared to angry expressions when attention focused on non-facial stimuli. Parietal delta power was smaller for surprised facial expressions than for angry facial expressions when attention focused on facial emotions and for surprised facial expressions than for angry and neutral facial expressions when attention was directed to non-facial stimuli. These findings might suggest that neural responses to surprised facial expressions are modulated by attentional focus.

Impact of virtual agent facial emotions and attention on N170 ERP amplitude: comparative study

Introduction

It is known from the literature that face perception of virtual agents affects the amplitude and latency of the ERP components. However, sensitivity of the N170 component to virtual agent facial emotions, and level of attention to facial emotional expressions were not investigated in the virtual reality environment by now, which was the aim of our study.

Methods

EEG recording, 2D and 3D visual testing of the neutral, happy and disgusted facial emotions of virtual agents were used. The protocol consisted of three sessions in the attentional condition of participants to each facial emotion (passive, active, and active to neutral facial emotional expression). The amplitudes of the N170 ERP were also reflected in the comparative analysis between 2D and VR.

Results

In the context of virtual agent facial emotional expressions, we identified the following dynamics of the N170 amplitude: attention (passive/active) showed no signaling effect; active attention to neutral virtual agent facial emotions reduced the N170 amplitude; significant interactions were observed between the factors "emotion × attention" and "environment × attention," but no interaction was found among all three factors.

Conclusion

The immersive quality of the environment in which visual and emotional events are presented has a less pronounced effect on early-stage facial processing at N170 amplitude. Thus, our findings indicate that the N170 amplitude is primarily modulated by the emotional content and attention directed to virtual agent facial emotional expressions.

EEG correlates of static and dynamic face perception: The role of naturalistic motion

Much of our understanding of how the brain processes dynamic faces comes from research that compares static photographs to dynamic morphs, which exhibit simplified, computer-generated motion. By comparing static, video recorded, and dynamic morphed expressions, we aim to identify the neural correlates of naturalistic facial dynamism, using time-domain and time-frequency analysis. Dynamic morphs were made from the neutral and peak frames of video recorded transitions of happy and fearful expressions, which retained expression change and removed asynchronous and non-linear features of naturalistic facial motion. We found that dynamic morphs elicited increased N400 amplitudes and lower LPP amplitudes compared to other stimulus types. Video recordings elicited higher LPP amplitudes and greater frontal delta activity compared to other stimuli. Thematic analysis of participant interviews using a large language model revealed that participants found it difficult to assess the genuineness of morphed expressions, and easier to analyse the genuineness of happy compared to fearful expressions. Our findings suggest that animating real faces with artificial motion may violate expectations (N400) and reduce the social salience (LPP) of dynamic morphs. Results also suggest that delta oscillations in the frontal region may be involved with the perception of naturalistic facial motion in happy and fearful expressions. Overall, our findings highlight the sensitivity of neural mechanisms required for face perception to subtle changes in facial motion characteristics, which has important implications for neuroimaging research using faces with simplified motion.

The role of the somatosensory system in the feeling of emotions: a neurostimulation study

Emotional experiences deeply impact our bodily states, such as when we feel 'anger', our fists close and our face burns. Recent studies have shown that emotions can be mapped onto specific body areas, suggesting a possible role of the primary somatosensory system (S1) in emotion processing. To date, however, the causal role of S1 in emotion generation remains unclear. To address this question, we applied transcranial alternating current stimulation (tACS) on the S1 at different frequencies (beta, theta, and sham) while participants saw emotional stimuli with different degrees of pleasantness and levels of arousal. Results showed that modulation of S1 influenced subjective emotional ratings as a function of the frequency applied. While theta and beta-tACS made participants rate the emotional images as more pleasant (higher valence), only theta-tACS lowered the subjective arousal ratings (more calming). Skin conductance responses recorded throughout the experiment confirmed a different arousal for pleasant versus unpleasant stimuli. Our study revealed that S1 has a causal role in the feeling of emotions, adding new insight into the embodied nature of emotions. Importantly, we provided causal evidence that beta and theta frequencies contribute differently to the modulation of two dimensions of emotions-arousal and valence-corroborating the view of a dissociation between these two dimensions of emotions.

Evidence of bottom-up homeostatic modulation induced taVNS during emotional and Go/No-Go tasks

Bilateral transcutaneous auricular vagus nerve stimulation (taVNS) - a non-invasive neuromodulation technique - has been investigated as a safe and feasible technique to treat many neuropsychiatric conditions. such as epilepsy, depression, anxiety, and chronic pain. Our aim is to investigate the effect of taVNS on neurophysiological processes during emotional and Go/No-Go tasks, and changes in frontal alpha asymmetry. We performed a randomized, double-blind, sham-controlled trial with 44 healthy individuals who were allocated into two groups (the active taVNS group and the sham taVNS group). Subjects received one session of taVNS (active or sham) for 60 min. QEEG was recorded before and after the interventions, and the subjects were assessed while exposed to emotional conditions with sad and happy facial expressions, followed by a Go/No-Go trial. The results demonstrated a significant increase in N2 amplitude in the No-Go condition for the active taVNS post-intervention compared to the sham taVNS after adjusting by handedness, mood, and fatigue levels (p = 0.046), significantly reduced ERD during sad conditions after treatment (p = 0.037), and increased frontal alpha asymmetry towards the right frontal hemisphere during the emotional task condition (p = 0.046). Finally, we observed an interesting neural signature in this study that suggests a bottom-up modulation from brainstem/subcortical to cortical areas as characterized by improved lateralization of alpha oscillations towards the frontal right hemisphere, and changes in ERP during emotional and Go/No-Go tasks that suggests a better subcortical response to the tasks. Such bottom-up effects may mediate some of the clinical effects of taVNS.

Electrophysiological correlates of face and object perception: A comparative analysis of 2D laboratory and virtual reality conditions

Human face perception is a specialized visual process with inherent social significance. The neural mechanisms reflecting this intricate cognitive process have evolved in spatially complex and emotionally rich environments. Previous research using VR to transfer an established face perception paradigm to realistic conditions has shown that the functional properties of face-sensitive neural correlates typically observed in the laboratory are attenuated outside the original modality. The present study builds on these results by comparing the perception of persons and objects under conventional laboratory (PC) and realistic conditions in VR. Adhering to established paradigms, the PC- and VR modalities both featured images of persons and cars alongside standard control images. To investigate the individual stages of realistic face processing, response times, the typical face-sensitive N170 component, and relevant subsequent components (L1, L2; pre-, post-response) were analyzed within and between modalities. The between-modality comparison of response times and component latencies revealed generally faster processing under realistic conditions. However, the obtained N170 latency and amplitude differences showed reduced discriminative capacity under realistic conditions during this early stage. These findings suggest that the effects commonly observed in the lab are specific to monitor-based presentations. Analyses of later and response-locked components showed specific neural mechanisms for identification and evaluation are employed when perceiving the stimuli under realistic conditions, reflected in discernible amplitude differences in response to faces and objects beyond the basic perceptual features. Conversely, the results do not provide evidence for comparable stimulus-specific perceptual processing pathways when viewing pictures of the stimuli under conventional laboratory conditions.

Digital Health to Patient-Facing Artificial Intelligence: Ethical Implications and Threats to Dignity for Patients With Cancer

Ethical considerations for patient-facing AI for oncology: dignity, autonomy, safety, equity, inclusivity.

A neurophysiological signature of dynamic emotion recognition associated with social communication skills and cortical gamma-aminobutyric acid levels in children

Introduction

Emotion recognition is a core feature of social perception. In particular, perception of dynamic facial emotional expressions is a major feature of the third visual pathway. However, the classical N170 visual evoked signal does not provide a pure correlate of such processing. Indeed, independent component analysis has demonstrated that the N170 component is already active at the time of the P100, and is therefore distorted by early components. Here we implemented, a dynamic face emotional paradigm to isolate a more pure face expression selective N170. We searched for a neural correlate of perception of dynamic facial emotional expressions, by starting with a face baseline from which a facial expression evolved. This allowed for a specific facial expression contrast signal which we aimed to relate with social communication abilities and cortical gamma-aminobutyric acid (GABA) levels.

Methods

We recorded event-related potentials (ERPs) and Magnetic Resonance (MRS) measures in 35 typically developing (TD) children, (10-16 years) sex-matched, during emotion recognition of an avatar morphing/unmorphing from neutral to happy/sad expressions. This task allowed for the elimination of the contribution low-level visual components, in particular the P100, by morphing baseline isoluminant neutral faces into specific expressions, isolating dynamic emotion recognition. Therefore, it was possible to isolate a dynamic face sensitive N170 devoid of interactions with earlier components.

Results

We found delayed N170 and P300, with a hysteresis type of dependence on stimulus trajectory (morphing/unmorphing), with hemispheric lateralization. The delayed N170 is generated by an extrastriate source, which can be related to the third visual pathway specialized in biological motion processing. GABA levels in visual cortex were related with N170 amplitude and latency and predictive of worse social communication performance (SCQ scores). N170 latencies reflected delayed processing speed of emotional expressions and related to worse social communication scores.

Discussion

In sum, we found a specific N170 electrophysiological signature of dynamic face processing related to social communication abilities and cortical GABA levels. These findings have potential clinical significance supporting the hypothesis of a spectrum of social communication abilities and the identification of a specific face-expression sensitive N170 which can potentially be used in the development of diagnostic and intervention tools.

Alpha-band oscillations and emotion: A review of studies on picture perception

Although alpha-band activity has long been a focus of psychophysiological research, its modulation by emotional value during picture perception has only recently been studied systematically. Here, we review these studies and report that the most consistent alpha oscillatory pattern indexing emotional processing is an enhanced desynchronization (ERD) over posterior sensors when viewing emotional compared with neutral pictures. This enhanced alpha ERD is not specific to unpleasant picture content, as previously proposed for other measures of affective response, but has also been observed for pleasant stimuli. Evidence suggests that this effect is not confined to the alpha band but that it also involves a desynchronization of the lower beta frequencies (8-20 Hz). The emotional modulation of alpha ERD occurs even after massive stimulus repetition and when emotional cues serve as task-irrelevant distractors, consistent with the hypothesis that evaluative processes are mandatory in emotional picture processing. A similar enhanced ERD has been observed for other significant cues (e.g., conditioned aversive stimuli, or in anticipation of a potential threat), suggesting that it reflects cortical excitability associated with the engagement of the motivational systems.

The Reward-Related Shift of Emotional Contagion from the Observer's Perspective Correlates to Their Intimacy with the Expresser

Although previous studies have found a bidirectional relationship between emotional contagion and reward, there is insufficient research to prove the effect of reward on the social function of emotional contagion. To explore this issue, the current study used electroencephalography (EEG) and the interactive way in which the expresser played games to help participants obtain reward outcomes. The results demonstrated a significant correlation between changes in emotional contagion and closeness, indicating that emotional contagion has a social regulatory function. Regarding the impact of reward outcomes, the results showed that compared to the context of a loss, in the context of a win, participants' closeness toward the expresser shifted to a more intimate level, their emotional contagion changed in a more positive direction, and the activity of the late positive component (LPC) of the event-related potentials (ERPs) changed to a greater extent. Significantly, the mediation results demonstrated the effect of reward and indicated that changes in the LPC elicited while experiencing the expressers' emotion predicted the subsequent shifts in closeness through alterations in emotional contagion of the anger emotion in the winning context and the happy emotion in the loss context. This study provides empirical evidence regarding the social function of emotional contagion and proves for the first time that the reward context plays a role in it.

Autistic traits shape neuronal oscillations during emotion perception under attentional load modulation

Emotional content is particularly salient, but situational factors such as cognitive load may disturb the attentional prioritization towards affective stimuli and interfere with their processing. In this study, 31 autistic and 31 typically developed children volunteered to assess their perception of affective prosodies via event-related spectral perturbations of neuronal oscillations recorded by electroencephalography under attentional load modulations induced by Multiple Object Tracking or neutral images. Although intermediate load optimized emotion processing by typically developed children, load and emotion did not interplay in children with autism. Results also outlined impaired emotional integration emphasized in theta, alpha and beta oscillations at early and late stages, and lower attentional ability indexed by the tracking capacity. Furthermore, both tracking capacity and neuronal patterns of emotion perception during task were predicted by daily-life autistic behaviors. These findings highlight that intermediate load may encourage emotion processing in typically developed children. However, autism aligns with impaired affective processing and selective attention, both insensitive to load modulations. Results were discussed within a Bayesian perspective that suggests atypical updating in precision between sensations and hidden states, towards poor contextual evaluations. For the first time, implicit emotion perception assessed by neuronal markers was integrated with environmental demands to characterize autism.

A Smart Mirror for Emotion Monitoring in Home Environments

Smart mirrors are devices that can display any kind of information and can interact with the user using touch and voice commands. Different kinds of smart mirrors exist: general purpose, medical, fashion, and other task specific ones. General purpose smart mirrors are suitable for home environments but the exiting ones offer similar, limited functionalities. In this paper, we present a general-purpose smart mirror that integrates several functionalities, standard and advanced, to support users in their everyday life. Among the advanced functionalities are the capabilities of detecting a person’s emotions, the short- and long-term monitoring and analysis of the emotions, a double authentication protocol to preserve the privacy, and the integration of Alexa Skills to extend the applications of the smart mirrors. We exploit a deep learning technique to develop most of the smart functionalities. The effectiveness of the device is demonstrated by the performances of the implemented functionalities, and the evaluation in terms of its usability with real users.