Amygdala responds to direct gaze in real but not in computer-generated faces

Can deepfakes be used to study emotion perception? A comparison of dynamic face stimuli

Video recordings accurately capture facial expression movements; however, they are difficult for face perception researchers to standardise and manipulate. For this reason, dynamic morphs of photographs are often used, despite their lack of naturalistic facial motion. This study aimed to investigate how humans perceive emotions from faces using real videos and two different approaches to artificially generating dynamic expressions - dynamic morphs, and AI-synthesised deepfakes. Our participants perceived dynamic morphed expressions as less intense when compared with videos (all emotions) and deepfakes (fearful, happy, sad). Videos and deepfakes were perceived similarly. Additionally, they perceived morphed happiness and sadness, but not morphed anger or fear, as less genuine than other formats. Our findings support previous research indicating that social responses to morphed emotions are not representative of those to video recordings. The findings also suggest that deepfakes may offer a more suitable standardized stimulus type compared to morphs. Additionally, qualitative data were collected from participants and analysed using ChatGPT, a large language model. ChatGPT successfully identified themes in the data consistent with those identified by an independent human researcher. According to this analysis, our participants perceived dynamic morphs as less natural compared with videos and deepfakes. That participants perceived deepfakes and videos similarly suggests that deepfakes effectively replicate natural facial movements, making them a promising alternative for face perception research. The study contributes to the growing body of research exploring the usefulness of generative artificial intelligence for advancing the study of human perception.

Emotion-gaze interaction affects time-to-collision estimates, but not preferred interpersonal distance towards looming faces

Estimating the time until impending collision (time-to-collision, TTC) of approaching or looming individuals and maintaining a comfortable distance from others (interpersonal distance, IPD) are commonly required in daily life and contribute to survival and social goals. Despite accumulating evidence that facial expressions and gaze direction interactively influence face processing, it remains unclear how these facial features affect the spatiotemporal processing of looming faces. We examined whether facial expressions (fearful vs. neutral) and gaze direction (direct vs. averted) interact on the judgments of TTC and IPD for looming faces, based on the shared signal hypothesis that fear signals the existence of threats in the environment when coupled with averted gaze. Experiment 1 demonstrated that TTC estimates were reduced for fearful faces compared to neutral ones only when the concomitant gaze was averted. In Experiment 2, the emotion-gaze interaction was not observed in the IPD regulation, which is arguably sensitive to affective responses to faces. The results suggest that fearful-averted faces modulate the cognitive extrapolation process of looming motion by communicating environmental threats rather than by altering subjective fear or perceived emotional intensity of faces. The TTC-specific effect may reflect an enhanced defensive response to unseen threats implied by looming fearful-averted faces. Our findings provide insight into how the visual system processes facial features to ensure bodily safety and comfortable interpersonal communication in dynamic environments.

Amygdala fMRI-A Critical Appraisal of the Extant Literature

Even before the advent of fMRI, the amygdala occupied a central space in the affective neurosciences. Yet this amygdala-centred view on emotion processing gained even wider acceptance after the inception of fMRI in the early 1990s, a landmark that triggered a goldrush of fMRI studies targeting the amygdala in vivo. Initially, this amygdala fMRI research was mostly confined to task-activation studies measuring the magnitude of the amygdala's response to emotional stimuli. Later, interest began to shift more towards the study of the amygdala's resting-state functional connectivity and task-based psychophysiological interactions. Later still, the test-retest reliability of amygdala fMRI came under closer scrutiny, while at the same time, amygdala-based real-time fMRI neurofeedback gained widespread popularity. Each of these major subdomains of amygdala fMRI research has left its marks on the field of affective neuroscience at large. The purpose of this review is to provide a critical assessment of this literature. By integrating the insights garnered by these research branches, we aim to answer the question: What part (if any) can amygdala fMRI still play within the current landscape of affective neuroscience? Our findings show that serious questions can be raised with regard to both the reliability and validity of amygdala fMRI. These conclusions force us to cast doubt on the continued viability of amygdala fMRI as a core pilar of the affective neurosciences.

The Neuroscience of Human and Artificial Intelligence Presence

Two decades of social neuroscience and neuroeconomics research illustrate the brain mechanisms that are engaged when people consider human beings, often in comparison to considering artificial intelligence (AI) as a nonhuman control. AI as an experimental control preserves agency and facilitates social interactions but lacks a human presence, providing insight into brain mechanisms that are engaged by human presence and the presence of AI. Here, I review this literature to determine how the brain instantiates human and AI presence across social perception and decision-making paradigms commonly used to realize a social context. People behave toward humans differently than they do toward AI. Moreover, brain regions more engaged by humans compared to AI extend beyond the social cognition brain network to all parts of the brain, and the brain sometimes is engaged more by AI than by humans. Finally, I discuss gaps in the literature, limitations in current neuroscience approaches, and how an understanding of the brain correlates of human and AI presence can inform social science in the wild.

Biological postural oscillations during facial expression of pain in virtual characters modulate early and late ERP components associated with empathy: A pilot study

There is a surge in the use of virtual characters in cognitive sciences. However, their behavioural realism remains to be perfected in order to trigger more spontaneous and socially expected reactions in users. It was recently shown that biological postural oscillations (idle motion) were a key ingredient to enhance the empathic response to its facial pain expression. The objective of this study was to examine, using electroencephalography, whether idle motion would modulate the neural response associated with empathy when viewing a pain-expressing virtual character. Twenty healthy young adults were shown video clips of a virtual character displaying a facial expression of pain while its body was either static (Still condition) or animated with pre-recorded human postural oscillations (Idle condition). Participants rated the virtual human's facial expression of pain as significantly more intense in the Idle condition compared to the Still condition. Both the early (N2-N3) and the late (rLPP) event-related potentials (ERPs) associated with distinct dimensions of empathy, affective resonance and perspective-taking, respectively, were greater in the Idle condition compared to the Still condition. These findings confirm the potential of idle motion to increase empathy for pain expressed by virtual characters. They are discussed in line with contemporary empathy models in relation to human-machine interactions.

Should I trust you? Investigating trustworthiness judgements of painful facial expressions

Past research indicates that patients' reports of pain are often met with skepticism and that observers tend to underestimate patients' pain. The mechanisms behind these biases are not yet fully understood. One relevant domain of inquiry is the interaction between the emotional valence of a stranger's expression and the onlooker's trustworthiness judgment. The emotion overgeneralization hypothesis posits that when facial cues of valence are clear, individuals displaying negative expressions (e.g., disgust) are perceived as less trustworthy than those showing positive facial expressions (e.g., happiness). Accordingly, we hypothesized that facial expressions of pain (like disgust) would be judged more untrustworthy than facial expressions of happiness. In two separate studies, we measured trustworthiness judgments of four different facial expressions (i.e., neutral, happiness, pain, and disgust), displayed by both computer-generated and real faces, via both explicit self-reported ratings (Study 1) and implicit motor trajectories in a trustworthiness categorization task (Study 2). Ratings and categorization findings partly support our hypotheses. Our results reveal for the first time that when judging strangers' facial expressions, both negative expressions were perceived as more untrustworthy than happy expressions. They also indicate that facial expressions of pain are perceived as untrustworthy as disgust expressions, at least for computer-generated faces. These findings are relevant to the clinical setting because they highlight how overgeneralization of emotional facial expressions may subtend an early perceptual bias exerted by the patient's emotional facial cues onto the clinician's cognitive appraisal process.

Corneal reflections and skin contrast yield better memory of human and virtual faces

Virtual faces have been found to be rated less human-like and remembered worse than photographic images of humans. What it is in virtual faces that yields reduced memory has so far remained unclear. The current study investigated face memory in the context of virtual agent faces and human faces, real and manipulated, considering two factors of predicted influence, i.e., corneal reflections and skin contrast. Corneal reflections referred to the bright points in each eye that occur when the ambient light reflects from the surface of the cornea. Skin contrast referred to the degree to which skin surface is rough versus smooth. We conducted two memory experiments, one with high-quality virtual agent faces (Experiment 1) and the other with the photographs of human faces that were manipulated (Experiment 2). Experiment 1 showed better memory for virtual faces with increased corneal reflections and skin contrast (rougher rather than smoother skin). Experiment 2 replicated these findings, showing that removing the corneal reflections and smoothening the skin reduced memory recognition of manipulated faces, with a stronger effect exerted by the eyes than the skin. This study highlights specific features of the eyes and skin that can help explain memory discrepancies between real and virtual faces and in turn elucidates the factors that play a role in the cognitive processing of faces.

A systematic survey of face stimuli used in psychological research 2000-2020

For decades, psychology has relied on highly standardized images to understand how people respond to faces. Many of these stimuli are rigorously generated and supported by excellent normative data; as such, they have played an important role in the development of face science. However, there is now clear evidence that testing with ambient images (i.e., naturalistic images "in the wild") and including expressions that are spontaneous can lead to new and important insights. To precisely quantify the extent to which our current knowledge base has relied on standardized and posed stimuli, we systematically surveyed the face stimuli used in 12 key journals in this field across 2000-2020 (N = 3374 articles). Although a small number of posed expression databases continue to dominate the literature, the use of spontaneous expressions seems to be increasing. However, there has been no increase in the use of ambient or dynamic stimuli over time. The vast majority of articles have used highly standardized and nonmoving pictures of faces. An emerging trend is that virtual faces are being used as stand-ins for human faces in research. Overall, the results of the present survey highlight that there has been a significant imbalance in favor of standardized face stimuli. We argue that psychology would benefit from a more balanced approach because ambient and spontaneous stimuli have much to offer. We advocate a cognitive ethological approach that involves studying face processing in natural settings as well as the lab, incorporating more stimuli from "the wild".

The role of sex and emotion on emotion perception in artificial faces: An ERP study

Sex has a significant impact on the perception of emotional expressions. However, it remains unclear whether sex influences the perception of emotions in artificial faces, which are becoming popular in emotion research. We used an emotion recognition task with FaceGen faces portraying six basic emotions aiming to investigate the effect of sex and emotion on behavioural and electrophysiological parameters. 71 participants performed the task while EEG was recorded. The recognition of sadness was the poorest, however, females recognized sadness better than males. ERP results indicated that fear, disgust, and anger evoked higher amplitudes of late positive potential over the left parietal region compared to neutral expression. Females demonstrated higher values of global field power as compared to males. The interaction between sex and emotion on ERPs was not significant. The results of our study may be valuable for future therapies and research, as it emphasizes possibly distinct processing of emotions and potential sex differences in the recognition of emotional expressions in FaceGen faces.

The Neurobiological Correlates of Gaze Perception in Healthy Individuals and Neurologic Patients

The ability to adaptively follow conspecific eye movements is crucial for establishing shared attention and survival. Indeed, in humans, interacting with the gaze direction of others causes the reflexive orienting of attention and the faster object detection of the signaled spatial location. The behavioral evidence of this phenomenon is called gaze-cueing. Although this effect can be conceived as automatic and reflexive, gaze-cueing is often susceptible to context. In fact, gaze-cueing was shown to interact with other factors that characterize facial stimulus, such as the kind of cue that induces attention orienting (i.e., gaze or non-symbolic cues) or the emotional expression conveyed by the gaze cues. Here, we address neuroimaging evidence, investigating the neural bases of gaze-cueing and the perception of gaze direction and how contextual factors interact with the gaze shift of attention. Evidence from neuroimaging, as well as the fields of non-invasive brain stimulation and neurologic patients, highlights the involvement of the amygdala and the superior temporal lobe (especially the superior temporal sulcus (STS)) in gaze perception. However, in this review, we also emphasized the discrepancies of the attempts to characterize the distinct functional roles of the regions in the processing of gaze. Finally, we conclude by presenting the notion of invariant representation and underline its value as a conceptual framework for the future characterization of the perceptual processing of gaze within the STS.

Amygdala function in emotion, cognition, and behavior

The amygdala is a core structure in the anterior medial temporal lobe, with an important role in several brain functions involving memory, emotion, perception, social cognition, and even awareness. As a key brain structure for saliency detection, it triggers and controls widespread modulatory signals onto multiple areas of the brain, with a great impact on numerous aspects of adaptive behavior. Here we discuss the neural mechanisms underlying these functions, as established by animal and human research, including insights provided in both healthy and pathological conditions.

Angry facial expressions bias towards aversive actions

Social interaction requires fast and efficient processing of another person's intentions. In face-to-face interactions, aversive or appetitive actions typically co-occur with emotional expressions, allowing an observer to anticipate action intentions. In the present study, we investigated the influence of facial emotions on the processing of action intentions. Thirty-two participants were presented with video clips showing virtual agents displaying a facial emotion (angry vs. happy) while performing an action (punch vs. fist-bump) directed towards the observer. During each trial, video clips stopped at varying durations of the unfolding action, and participants had to recognize the presented action. Naturally, participants' recognition accuracy improved with increasing duration of the unfolding actions. Interestingly, while facial emotions did not influence accuracy, there was a significant influence on participants' action judgements. Participants were more likely to judge a presented action as a punch when agents showed an angry compared to a happy facial emotion. This effect was more pronounced in short video clips, showing only the beginning of an unfolding action, than in long video clips, showing near-complete actions. These results suggest that facial emotions influence anticipatory processing of action intentions allowing for fast and adaptive responses in social interactions.

Temporal lobe epilepsy alters neural responses to human and avatar facial expressions in the face perception network

BACKGROUND AND OBJECTIVE

Although avatars are now widely used in advertisement, entertainment, and business today, no study has investigated whether brain lesions in neurological patients interfere with brain activation in response to dynamic avatar facial expressions. The aim of our event-related fMRI study was to compare brain activation differences in people with epilepsy and controls during the processing of fearful and neutral dynamic expressions displayed by human or avatar faces.

METHODS

Using functional magnetic resonance imaging (fMRI), we examined brain responses to dynamic facial expressions of trained actors and their avatar look-alikes in 16 people with temporal lobe epilepsy (TLE) and 26 controls. The actors' fearful and neutral expressions were recorded on video and conveyed onto their avatar look-alikes by face tracking.

RESULTS

Our fMRI results show that people with TLE exhibited reduced response differences between fearful and neutral expressions displayed by humans in the right amygdala and the left superior temporal sulcus (STS). Further, TLE was associated with reduced response differences between human and avatar fearful expressions in the dorsal pathway of the face perception network (STS and inferior frontal gyrus) as well as in the medial prefrontal cortex.

CONCLUSIONS

Taken together, these findings suggest that brain responses to dynamic facial expressions are altered in people with TLE compared to neurologically healthy individuals-regardless of whether the face is human or computer-generated. In TLE, areas sensitive to dynamic facial features and associated with processes relating to the self and others are particularly affected when processing dynamic human and avatar expressions. Our findings highlight that the impact of TLE on facial emotion processing must be extended to artificial faces and should be considered when applying dynamic avatars in the context of neurological conditions.

Dynamic human and avatar facial expressions elicit differential brain responses

Computer-generated characters, so-called avatars, are widely used in advertising, entertainment, human–computer interaction or as research tools to investigate human emotion perception. However, brain responses to avatar and human faces have scarcely been studied to date. As such, it remains unclear whether dynamic facial expressions of avatars evoke different brain responses than dynamic facial expressions of humans. In this study, we designed anthropomorphic avatars animated with motion tracking and tested whether the human brain processes fearful and neutral expressions in human and avatar faces differently. Our fMRI results showed that fearful human expressions evoked stronger responses than fearful avatar expressions in the ventral anterior and posterior cingulate gyrus, the anterior insula, the anterior and posterior superior temporal sulcus, and the inferior frontal gyrus. Fearful expressions in human and avatar faces evoked similar responses in the amygdala. We did not find different responses to neutral human and avatar expressions. Our results highlight differences, but also similarities in the processing of fearful human expressions and fearful avatar expressions even if they are designed to be highly anthropomorphic and animated with motion tracking. This has important consequences for research using dynamic avatars, especially when processes are investigated that involve cortical and subcortical regions.

Meeting another's gaze shortens subjective time by capturing attention

Gaze directed at the observer (direct gaze) is an important and highly salient social signal with multiple effects on cognitive processes and behavior. It is disputed whether the effect of direct gaze is caused by attentional capture or increased arousal. Time estimation may provide an answer because attentional capture predicts an underestimation of time whereas arousal predicts an overestimation. In a temporal bisection task, observers were required to classify the duration of a stimulus as short or long. Stimulus duration was selected randomly between 988 and 1479 ms. When gaze was directed at the observer, participants underestimated stimulus duration, suggesting that effects of direct gaze are caused by attentional capture, not increased arousal. Critically, this effect was limited to dynamic stimuli where gaze appeared to move toward the participant. The underestimation was present with stimuli showing a full face, but also with stimuli showing only the eye region, inverted faces and high-contrast eye-like stimuli. However, it was absent with static pictures of full faces and dynamic nonfigurative stimuli. Because the effect of direct gaze depended on motion, which is common in naturalistic scenes, more consideration needs to be given to the ecological validity of stimuli in the study of social attention.

In your phase: neural phase synchronisation underlies visual imagery of faces

Mental imagery is the process through which we retrieve and recombine information from our memory to elicit the subjective impression of “seeing with the mind’s eye”. In the social domain, we imagine other individuals while recalling our encounters with them or modelling alternative social interactions in future. Many studies using imaging and neurophysiological techniques have shown several similarities in brain activity between visual imagery and visual perception, and have identified frontoparietal, occipital and temporal neural components of visual imagery. However, the neural connectivity between these regions during visual imagery of socially relevant stimuli has not been studied. Here we used electroencephalography to investigate neural connectivity and its dynamics between frontal, parietal, occipital and temporal electrodes during visual imagery of faces. We found that voluntary visual imagery of faces is associated with long-range phase synchronisation in the gamma frequency range between frontoparietal electrode pairs and between occipitoparietal electrode pairs. In contrast, no effect of imagery was observed in the connectivity between occipitotemporal electrode pairs. Gamma range synchronisation between occipitoparietal electrode pairs predicted subjective ratings of the contour definition of imagined faces. Furthermore, we found that visual imagery of faces is associated with an increase of short-range frontal synchronisation in the theta frequency range, which temporally preceded the long-range increase in the gamma synchronisation. We speculate that the local frontal synchrony in the theta frequency range might be associated with an effortful top-down mnemonic reactivation of faces. In contrast, the long-range connectivity in the gamma frequency range along the fronto-parieto-occipital axis might be related to the endogenous binding and subjective clarity of facial visual features.

Emotional appraisal processing of computer-generated facial expressions: an functional near-infrared spectroscopy study

OBJECTIVE

The current study aims to investigate whether computer generated (CG) expressions of emotion evoke similar emotional appraisal processing in the lateral orbitofrontal cortex (lOFC) compared to real human expressions, as well as how speech cues would influence the processing.

METHODS

Functional near-infrared spectroscopy was used to measure the neural activations in the prefrontal cortex during emotion recognition task. Thirty normal participants were asked to view videos of dynamic facial expressions and selected the emotions that were best matches with the expressions.

RESULTS

CG expressions evoked less activation in the lOFC comparing to real human expressions. Furthermore, speech cues increased the activation in the lOFC for CG expressions but not real expressions.

CONCLUSION

Comparing to real expressions, CG expressions evoked less appraisal processing related to motivational values although this disadvantage can be compensated to some extent by presenting the expressions with speech cues.

A Multidimensional View on Social and Non-Social Rewards

Social rewards are a broad and heterogeneous set of stimuli including for instance smiling faces, gestures, or praise. They have been widely investigated in cognitive and social neuroscience as well as psychology. Research often contrasts the neural processing of social rewards with non-social ones, with the aim to demonstrate the privileged and unique nature of social rewards or to examine shared neural processing underlying them. However, such comparisons mostly neglect other important dimensions of rewards that are conflated in those types of rewards: primacy, temporal proximity, duration, familiarity, source, tangibility, naturalness, and magnitude. We identify how commonly used rewards in both social and non-social domains may differ in respect to these dimensions and how their interaction calls for careful consideration of alternative interpretations of observed effects. Additionally, we propose potential solutions on how to adapt the multidimensional view to experimental research. Altogether, these methodological considerations aim to inform and improve future experimental designs in research utilizing rewarding stimuli, especially in the social domain.