The differential contribution of facial expressions, prosody, and speech content to empathy

Machine Learning Approach to Identifying Empathy Using the Vocals of Mental Health Helpline Counselors: Algorithm Development and Validation

Background

This research study aimed to detect the vocal features immersed in empathic counselor speech using samples of calls to a mental health helpline service.

Objective

This study aimed to produce an algorithm for the identification of empathy from these features, which could act as a training guide for counselors and conversational agents who need to transmit empathy in their vocals.

Methods

Two annotators with a psychology background and English heritage provided empathy ratings for 57 calls involving female counselors, as well as multiple short call segments within each of these calls. These ratings were found to be well-correlated between the 2 raters in a sample of 6 common calls. Using vocal feature extraction from call segments and statistical variable selection methods, such as L1 penalized LASSO (Least Absolute Shrinkage and Selection Operator) and forward selection, a total of 14 significant vocal features were associated with empathic speech. Generalized additive mixed models (GAMM), binary logistics regression with splines, and random forest models were used to obtain an algorithm that differentiated between high- and low-empathy call segments.

Results

The binary logistics regression model reported higher predictive accuracies of empathy (area under the curve [AUC]=0.617, 95% CI 0.613-0.622) compared to the GAMM (AUC=0.605, 95% CI 0.601-0.609) and the random forest model (AUC=0.600, 95% CI 0.595-0.604). This difference was statistically significant, as evidenced by the nonoverlapping 95% CIs obtained for AUC. The DeLong test further validated these results, showing a significant difference in the binary logistic model compared to the random forest (D=6.443, df=186283, P<.001) and GAMM (Z=5.846, P<.001). These findings confirm that the binary logistic regression model outperforms the other 2 models concerning predictive accuracy for empathy classification.

Conclusions

This study suggests that the identification of empathy from vocal features alone is challenging, and further research involving multimodal models (eg, models incorporating facial expression, words used, and vocal features) are encouraged for detecting empathy in the future. This study has several limitations, including a relatively small sample of calls and only 2 empathy raters. Future research should focus on accommodating multiple raters with varied backgrounds to explore these effects on perceptions of empathy. Additionally, considering counselor vocals from larger, more heterogeneous populations, including mixed-gender samples, will allow an exploration of the factors influencing the level of empathy projected in counselor voices more generally.

Early affective empathy, emotion contagion, and empathic concern in borderline personality disorder: A systematic review and meta-analysis

BACKGROUND

Individuals with Borderline Personality Disorder (BPD) are theorized to experience lower cognitive empathy but heightened affective empathy. Despite meta-analyses addressing cognitive empathy, affective empathy remains unexplored. This pre-registered systematic review and meta-analysis investigated affective empathy in individuals with BPD or high BPD traits relative to healthy comparisons, using a multidimensional approach including, early affective empathy, emotion contagion, and empathic concern.

METHODS

Systematic search of SCOPUS, PubMed, Medline COMPLETE, and PsycINFO (June 27, 2022, May 14, 2023, and July 1, 2024) was completed. Included studies compared affective empathy in those with BPD/high BPD traits with healthy comparisons, utilized experimental or self-report designs, and were peer-reviewed or PhD theses. Risk of bias was assessed using the Newcastle-Ottawa Scale.

RESULTS

Among 22 eligible studies identified, results revealed individuals with BPD/high BPD traits showed significantly higher emotion contagion (Npooled = 1797, g = -1.10, 95 % CI [-1.57, -0.62]). No significant differences were found in empathic concern (Npooled = 1545, g = 0.06, 95 % CI [-0.10, 0.22]), or early affective empathy for anger (Npooled = 245, g = 0.28, 95 % CI [-0.0.53, 1.09]) and happiness, (Npooled = 189, g = 0.34, 95 % CI [-0.1.50, 2.18]).

LIMITATIONS

Few included studies for early affective empathy, methodological shortcomings in the broader literature and study heterogeneity suggest caution when interpreting these effects, emphasizing the need for targeted research.

CONCLUSIONS

While individuals with BPD/high BPD traits are more likely to subjectively experience others' distress through emotion contagion, no differences were found in early affective empathy or ability to direct sympathy and concern towards others.

Empathy in schizophrenia: neural alterations during emotion recognition and affective sharing

Introduction

Deficits in emotion recognition and processing are characteristic for patients with schizophrenia [SCZ].

Methods

We targeted both emotion recognition and affective sharing, one in static and one in dynamic facial stimuli, during functional magnetic resonance imaging [fMRI] in 22 SCZ patients and 22 matched healthy controls [HC]. Current symptomatology and cognitive deficits were assessed as potential influencing factors.

Results

Behaviorally, patients only showed a prolonged response time in age-discrimination trials. For emotion-processing trials, patients showed a difference in neural response, without an observable behavioral correlate. During emotion and age recognition in static stimuli, a reduced activation of the bilateral anterior cingulate cortex [ACC] and the right anterior insula [AI] emerged. In the affective sharing task, patients showed a reduced activation in the left and right caudate nucleus, right AI and inferior frontal gyrus [IFG], right cerebellum, and left thalamus, key areas of empathy.

Discussion

We conclude that patients have deficits in complex visual information processing regardless of emotional content on a behavioral level and that these deficits coincide with aberrant neural activation patterns in emotion processing networks. The right AI as an integrator of these networks plays a key role in these aberrant neural activation patterns and, thus, is a promising candidate area for neurofeedback approaches.

Dynamic and/or multimodal assessments for social cognition in neuropsychology: Results from a systematic literature review

Objective: Despite the prevalence of socio-cognitive disturbances, and their important diagnostic/therapeutic implications, the assessment of these disturbances remains scarce. This systematic review aims to identify available social cognition tools for adult assessment that use multimodal and/or dynamic social cues, specifying their strengths and limitations (e.g. from a methodological, psychometric, ecological, and clinical perspective). Method: An electronic search was conducted in Pubmed, PsychINFO, Embase and Scopus databases for articles published up to the 3th of January 2023 and the first 200 Google Scholar results on the same date. The PRISMA methodology was applied, 3884 studies were screened based on title and abstract and 329 full texts were screened. Articles using pseudo-dynamic methodologies (e.g. morphing), reported only subjective or self-reported measures, or investigated only physiological or brain activity responses were excluded. Results: In total, 149 works were included in this review, representing 65 assessment tools (i.e. 48% studying emotion recognition (n = 31), 32% Theory of Mind (n = 21), 5% empathy (n = 3), 1.5% moral cognition/social reasoning (n = 1), and 14% being multimodal (n = 9)). For each study, the tool's main characteristics, psychometric properties, ecological validity indicators and available norms are reported. The tools are presented according to social-cognitive process assessed and communication channels used. Conclusions: This study highlights the lack of validated and standardized tools. A few tools appear to partially meet some clinical needs. The development of methodologies using a first-person paradigm and taking into account the multidimensional nature of social cognition seems a relevant research endeavour for greater ecological validity.

The vocal side of empathy: neural correlates of pain perception in spoken complaints

In the extensive neuroimaging literature on empathy for pain, few studies have investigated how this phenomenon may relate to everyday social situations such as spoken interactions. The present study used functional Magnetic Resonance Imaging (fMRI) to assess how complaints, as vocal expressions of pain, are empathically processed by listeners and how these empathic responses may vary based on speakers' vocal expression and cultural identity. Twenty-four French participants listened to short utterances describing a painful event, which were either produced in a neutral-sounding or complaining voice by both in-group (French) and out-group (French Canadian) speakers. Results suggest that the perception of suffering from a complaining voice increased activity in the emotional voice areas, composed of voice-sensitive temporal regions interacting with prefrontal cortices and the amygdala. The Salience and Theory of Mind networks, associated with affective and cognitive aspects of empathy, also showed prosody-related activity and specifically correlated with behavioral evaluations of suffering by listeners. Complaints produced by in- vs out-group speakers elicited sensorimotor and default mode activity, respectively, suggesting accent-based changes in empathic perspective. These results, while reaffirming the role of key networks in tasks involving empathy, highlight the importance of vocal expression information and social categorization processes when perceiving another's suffering during social interactions.

Mood congruency affects physiological synchrony but not empathic accuracy in a naturalistic empathy task

Empathy is a crucial aspect of our daily lives, as it enhances our wellbeing and is a proxy for prosocial behavior. It encompasses two related but partially distinct components: cognitive and affective empathy. Both are susceptible to context, biases and an individual's physiological state. Few studies have explored the effects of a person's mood on these empathy components, and results are mixed. The current study takes advantage of an ecological, naturalistic empathy task - the empathic accuracy (EA) task - in combination with physiological measurements to examine and differentiate between the effects of one's mood on both empathy components. Participants were induced with positive or negative mood and presented videos of targets narrating autobiographical negative stories, selected from a Chinese empathy dataset that we developed (now publicly available). The stories were conveyed in audio-only, visual-only and full-video formats. Participants rated the target's emotional state while watching or listening to their stories, and physiological measures were taken throughout the process. Importantly, similar measures were taken from the targets when they narrated the stories, allowing a comparison between participants' and targets' measures. We found that in audio-only and visual-only conditions, participants whose moods were congruent with the target showed higher physiological synchrony than those with incongruent mood, implying a mood-congruency effect on affective empathy. However, there was no mood effect on empathic accuracy (reflecting cognitive empathy), suggesting a different influence of mood on the two empathy components.

Expressive Prosody in Patients With Focal Anterior Temporal Neurodegeneration

BACKGROUND AND OBJECTIVES

Progressive focal anterior temporal lobe (ATL) neurodegeneration has been historically called semantic dementia. More recently, semantic variant primary progressive aphasia (svPPA) and semantic behavioral variant frontotemporal dementia (sbvFTD) have been linked with predominant left and right ATL neurodegeneration, respectively. Nonetheless, clinical tools for an accurate diagnosis of sbvFTD are still lacking. Expressive prosody refers to the modulation of pitch, loudness, tempo, and quality of voice used to convey emotional and linguistic information and has been linked to bilateral but right-predominant frontotemporal functioning. Changes in expressive prosody can be detected with semiautomated methods and could represent a useful diagnostic marker of socioemotional functioning in sbvFTD.

METHODS

Participants underwent a comprehensive neuropsychological and language evaluation and a 3T MRI at the University of California San Francisco. Each participant provided a verbal description of the picnic scene from the Western Aphasia Battery. The fundamental frequency (f0) range, an acoustic measure of pitch variability, was extracted for each participant. We compared the f0 range between groups and investigated associations with an informant-rated measure of empathy, a facial emotion labeling task, and gray matter (GM) volumes using voxel-based morphometry.

RESULTS

Twenty-eight patients with svPPA, 18 with sbvFTD, and 18 healthy controls (HCs) were included. f0 range was significantly different across groups: patients with sbvFTD showed reduced f0 range in comparison with both patients with svPPA (mean difference of -1.4 ± 2.4 semitones; 95% CI -2.4 to -0.4]; p < 0.005) and HCs (mean difference of -1.9 ± 3.0 semitones; 95% CI -3.0 to -0.7]; p < 0.001). A higher f0 range was correlated with a greater informant-rated empathy (r = 0.355; p ≤ 0.05), but not facial emotion labeling. Finally, the lower f0 range was correlated with lower GM volume in the right superior temporal gyrus, encompassing anterior and posterior portions (p < 0.05 FWE cluster corrected).

DISCUSSION

Expressive prosody may be a useful clinical marker of sbvFTD. Reduced empathy is a core symptom in sbvFTD; the present results extend this to prosody, a core component of social interaction, at the intersection of speech and emotion. They also inform the long-standing debate on the lateralization of expressive prosody in the brain, highlighting the critical role of the right superior temporal lobe.

Function without feeling: neural reactivity and intercommunication during flexible motor adjustments evoked by emotional and neutral stimuli

Motor conflicts arise when we need to quickly overwrite prepotent behavior. It has been proposed that affective stimuli modulate the neural processing of motor conflicts. However, previous studies have come to inconsistent conclusions regarding the neural impact of affective information on conflict processing. We employed functional magnetic resonance imaging during a Go/Change-Go task, where motor conflicts were either evoked by neutral or emotionally negative stimuli. Dynamic causal modeling was used to investigate how motor conflicts modulate the intercommunication between the anterior cingulate cortex (ACC) and the anterior insula (AI) as 2 central regions for cognitive control. Conflicts compared to standard actions were associated with increased BOLD activation in several brain areas, including the dorsal ACC and anterior insula. There were no differences in neural activity between emotional and non-emotional conflict stimuli. Conflicts compared to standard actions lowered neural self-inhibition of the ACC and AI and led to increased effective connectivity from the ACC to AI contralateral to the acting hand. Thus, our study indicates that neural conflict processing is primarily driven by the functional relevance of action-related stimuli, not their inherent affective meaning. Furthermore, it sheds light on the role of interconnectivity between ACC and AI for the implementation of flexible behavioral change.

Eyes on you: Ensuring empathic accuracy or signalling empathy?

The eye region is thought to play an important role in the ability to accurately infer others' feelings, or empathic accuracy (EA), which is an important skill for social interaction. However, most past studies used static pictures, including only visual information, and knowledge about the contribution of the eye region to EA when visual information is presented together with verbal content is lacking. We therefore examined whether eye gazing contributes to EA during videos of emotional autobiographical stories including both visual and verbal content. One hundred seven perceivers watched videos of targets talking about positive and negative life events and continuously rated the targets' feelings during the videos. Simultaneously, perceivers' eyes were tracked. After each video, perceivers reported on their feelings and the extent to which they empathized with and took the perspective of the targets. In contrast to studies using static pictures, we found that gazing to the eyes of targets during the videos did not significantly contribute to EA. At the same time, results on the association between the amount of gaze towards the eye region of targets and perceivers' state and trait empathy ratings suggest that eye gazing might signal empathy and social engagement to others.

Listen to my feelings! How prosody and accent drive the empathic relevance of complaining speech

Interpersonal communication often involves sharing our feelings with others; complaining, for example, aims to elicit empathy in listeners by vocally expressing a speaker's suffering. Despite the growing neuroscientific interest in the phenomenon of empathy, few have investigated how it is elicited in real time by vocal signals (prosody), and how this might be affected by interpersonal factors, such as a speaker's cultural background (based on their accent). To investigate the neural processes at play when hearing spoken complaints, twenty-six French participants listened to complaining and neutral utterances produced by in-group French and out-group Québécois (i.e., French-Canadian) speakers. Participants rated how hurt the speaker felt while their cerebral activity was monitored with electroencephalography (EEG). Principal Component Analysis of Event-Related Potentials (ERPs) taken at utterance onset showed culture-dependent time courses of emotive prosody processing. The high motivational relevance of ingroup complaints increased the P200 response compared to all other utterance types; in contrast, outgroup complaints selectively elicited an early posterior negativity in the same time window, followed by an increased N400 (due to ongoing effort to derive affective meaning from outgroup voices). Ingroup neutral utterances evoked a late negativity which may reflect re-analysis of emotively less salient, but culturally relevant ingroup speech. Results highlight the time-course of neurocognitive responses that contribute to emotive speech processing for complaints, establishing the critical role of prosody as well as social-relational factors (i.e., cultural identity) on how listeners are likely to "empathize" with a speaker.

Neural and Affective Responses to Prolonged Eye Contact with One's Own Adolescent Child and Unfamiliar Others

Eye contact is crucial for the formation and maintenance of social relationships, and plays a key role in facilitating a strong parent-child bond. However, the precise neural and affective mechanisms through which eye contact impacts on parent-child relationships remain elusive. We introduce a task to assess parents' neural and affective responses to prolonged direct and averted gaze coming from their own child, and an unfamiliar child and adult. While in the scanner, 79 parents (n = 44 mothers and n = 35 fathers) were presented with prolonged (16-38 s) videos of their own child, an unfamiliar child, an unfamiliar adult, and themselves (i.e., targets), facing the camera with a direct or an averted gaze. We measured BOLD-responses, tracked parents' eye movements during the videos, and asked them to report on their mood and feelings of connectedness with the targets after each video. Parents reported improved mood and increased feelings of connectedness after prolonged exposure to direct versus averted gaze and these effects were amplified for unfamiliar targets compared to their own child, due to high affect and connectedness ratings after videos of their own child. Neuroimaging results showed that the sight of one's own child was associated with increased activity in middle occipital gyrus, fusiform gyrus and inferior frontal gyrus relative to seeing an unfamiliar child or adult. While we found no robust evidence of specific neural correlates of eye contact (i.e., contrast direct > averted gaze), an exploratory parametric analysis showed that dorsomedial prefrontal cortex (dmPFC) activity increased linearly with duration of eye contact (collapsed across all "other" targets). Eye contact-related dmPFC activity correlated positively with increases in feelings of connectedness, suggesting that this region may drive feelings of connectedness during prolonged eye contact with others. These results underline the importance of prolonged eye contact for affiliative processes and provide first insights into its neural correlates. This may pave the way for new research in individuals or pairs in whom affiliative processes are disrupted.

Robust Multimodal Emotion Recognition from Conversation with Transformer-Based Crossmodality Fusion

Decades of scientific research have been conducted on developing and evaluating methods for automated emotion recognition. With exponentially growing technology, there is a wide range of emerging applications that require emotional state recognition of the user. This paper investigates a robust approach for multimodal emotion recognition during a conversation. Three separate models for audio, video and text modalities are structured and fine-tuned on the MELD. In this paper, a transformer-based crossmodality fusion with the EmbraceNet architecture is employed to estimate the emotion. The proposed multimodal network architecture can achieve up to 65% accuracy, which significantly surpasses any of the unimodal models. We provide multiple evaluation techniques applied to our work to show that our model is robust and can even outperform the state-of-the-art models on the MELD.

Emotivity in the Voice: Prosodic, Lexical, and Cultural Appraisal of Complaining Speech

Emotive speech is a social act in which a speaker displays emotional signals with a specific intention; in the case of third-party complaints, this intention is to elicit empathy in the listener. The present study assessed how the emotivity of complaints was perceived in various conditions. Participants listened to short statements describing painful or neutral situations, spoken with a complaining or neutral prosody, and evaluated how complaining the speaker sounded. In addition to manipulating features of the message, social-affiliative factors which could influence complaint perception were varied by adopting a cross-cultural design: participants were either Québécois (French Canadian) or French and listened to utterances expressed by both cultural groups. The presence of a complaining tone of voice had the largest effect on participant evaluations, while the nature of statements had a significant, but smaller influence. Marginal effects of culture on explicit evaluation of complaints were found. A multiple mediation analysis suggested that mean fundamental frequency was the main prosodic signal that participants relied on to detect complaints, though most of the prosody effect could not be linearly explained by acoustic parameters. These results highlight a tacit agreement between speaker and listener: what characterizes a complaint is how it is said (i.e., the tone of voice), more than what it is about or who produces it. More generally, the study emphasizes the central importance of prosody in expressive speech acts such as complaints, which are designed to strengthen social bonds and supportive responses in interactive behavior. This intentional and interpersonal aspect in the communication of emotions needs to be further considered in research on affect and communication.

The Influence of Empathy Trait and Gender on Empathic Responses. A Study With Dynamic Emotional Stimulus and Eye Movement Recordings

Previous studies have suggested that empathic process involve several components such as cognitive empathy, affective empathy, and prosocial concern. It has also been reported that gender and empathy trait can influence empathic responses such as emotional recognition, which requires an appropriate scanning of faces. However, the degree to which these factors influence the empathic responses, which include emotion recognition, affective empathy, and cognitive empathy, has not yet been specified. Aim: The aim of the present study was to identify the differences between individuals with high and low level of empathy trait, as well as differences between men and women, in an explicit task in order to evaluate the empathic responses. Methods: With this goal in mind, we recorded eye movements during the presentation of dynamic emotional stimuli (joy, anger, fear, and neutral videos). After watching each video, participants had to rate the valence and arousal dimensions of emotional content and explicit empathy responses were assessed. Thirty participants (15 women) were included in a High Empathy group (HE; mean age = 21.0) and 30 participants (16 women) in the Low Empathy group (LE; mean age = 21.2), according to their scores in the Interpersonal Reactivity Index (IRI) scale. Results: As expected, the HE group showed higher scores than the LE group in the explicit empathy responses. These differences, based on global scores, were mainly explained by affective empathy and cognitive empathy responses but not by emotional recognition one. No differences were observed by gender in these measures. Regarding eye movements in the dynamic emotional stimuli, HE group had longer fixation duration on the eyes area than LE group. In addition, women spent more time on the eyes area in comparison to men. Discussion: Our findings suggest that both men and women with high empathy trait are more accurate to empathizing but not on the basis of the emotional recognition response. The fact that women spent more time on the eyes area did not seem to affect the empathic responses to the dynamic emotional stimulus. Overall, empathic responses of both men and women are modulated by their empathic trait. In addition, empathic trait and gender seem to impact strategies to deal with emotional facial information.

Dysfunction of Empathy and Related Processes in Borderline Personality Disorder: A Systematic Review

After participating in this activity, learners should be better able to:• Assess differences between adult patients with the diagnosis of borderline personality disorder (BPD) and healthy control subjects in terms of empathy and related processes• Evaluate the effects of empathy or related processes as factors contributing to abnormal social functioning in BPD ABSTRACT: We reviewed 45 original research studies, published between 2000 and 2019, to assess differences between adult patients with the diagnosis of borderline personality disorder (BPD) and healthy control subjects in terms of empathy and related processes (i.e., theory of mind, mentalizing, social cognition, and emotional intelligence). Thirty-six studies reported deficits of empathy or related processes in patients with BPD. Enhanced emotional empathy in BPD was also reported in eight studies, all of which revealed that patients had increased scores of personal distress on the Interpersonal Reactivity Index self-report questionnaire. Six studies did not find significant differences between patients with BPD and healthy control subjects in terms of empathy or related processes. No study reported enhanced cognitive empathy, social cognition, or emotional intelligence in patients with BPD. We postulate that deficits of empathy or related processes contribute to preempting the formation of stable interpersonal relationships, whereas enhanced emotional empathy might lead to personal (and interpersonal) distress, further contributing to abnormal social functioning in BPD.

Top-down attention modulation on the perception of others' vocal pain: An event-related potential study

Pain is typically expressed through various sensory (e.g., visual and auditory) modalities: the human voice conveys information about social and affective communication. While the empathic responses to others' pain in the visual modality are modulated by top-down attention constraints, it remains unclear whether empathy for such expressions in the auditory modality also involves such top-down modulation mechanisms. Therefore, the present study investigates how neural correlates of empathic processes to others' vocal pain are modulated by the task-instructed attention manipulations. Each participant completed the following three tasks: (1) Pain Judgment Task, in which participants were instructed to pay attention to pain cues in vocal stimuli, (2) Gender Judgment Task, in which participants were instructed to pay attention to non-pain cues in vocal stimuli; (3) Passive Listening Task, a control task in which participants were instructed to passively listen to the vocal stimuli without any required response. The earlier frontal-central N1 response to either others' painful or neutral voice was greater in the Pain Judgment Task than in the other two tasks, suggesting a general attention modulation on the bottom-up sensory processing of vocal stimuli. The frontal-central P2 responses to others' painful voices was greater in the Pain Judgment Task than in the other two tasks, but not to others' neutral voices, thus suggesting selective attention modulation on the P2 response to others' pain. Late positive complex (LPC) to others' painful and neutral voices differed significantly regardless of task manipulations, thus suggesting empathic pain modulation on LPC response. All these results demonstrated top-down attention modulation on affective sharing responses others' vocal pain, but not on cognitive appraisal process of others' vocal pain.

Emotional Recognition and Empathy both in Deaf and Blind Adults

Studies addressing the recognition of emotions in blind or deaf participants have been carried out only with children and adolescents. Due to these age limits, such studies do not clarify the long-term effects of vision and hearing disabilities on emotion recognition in adults. We assessed the ability to recognize basic emotions in 15 deaf adults (aged 32.4 ± 8.1 yrs) and in 15 blind adults (48.3 ± 10.5 yrs). Auditory and visual stimuli expressing six basic emotional states were presented to participants (Florida Affect Battery). Participants also performed an empathy test. Deaf participants showed difficulties in emotion recognition tasks compared to the typical hearing participants; however, differences were only statistically reliable for Facial Emotion Discrimination and Naming tasks (specifically, naming expressions of fear). Deaf participants also revealed inferior levels of cognitive empathy. Concerning blind participants, their performance was lower than the controls' only when the task required the evaluation of emotional prosody while ignoring the semantic content of the sentence. Overall, although deaf and blind participants performed reasonably well on tasks requiring recognition of basic emotions, sensory loss may hinder their social perception skills when processing subtle emotions or when the extraction of simultaneous prosodic and semantic information is required.

The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English

The RAVDESS is a validated multimodal database of emotional speech and song. The database is gender balanced consisting of 24 professional actors, vocalizing lexically-matched statements in a neutral North American accent. Speech includes calm, happy, sad, angry, fearful, surprise, and disgust expressions, and song contains calm, happy, sad, angry, and fearful emotions. Each expression is produced at two levels of emotional intensity, with an additional neutral expression. All conditions are available in face-and-voice, face-only, and voice-only formats. The set of 7356 recordings were each rated 10 times on emotional validity, intensity, and genuineness. Ratings were provided by 247 individuals who were characteristic of untrained research participants from North America. A further set of 72 participants provided test-retest data. High levels of emotional validity and test-retest intrarater reliability were reported. Corrected accuracy and composite "goodness" measures are presented to assist researchers in the selection of stimuli. All recordings are made freely available under a Creative Commons license and can be downloaded at https://doi.org/10.5281/zenodo.1188976.

Hot Speech and Exploding Bombs: Autonomic Arousal During Emotion Classification of Prosodic Utterances and Affective Sounds

Emotional expressions provide strong signals in social interactions and can function as emotion inducers in a perceiver. Although speech provides one of the most important channels for human communication, its physiological correlates, such as activations of the autonomous nervous system (ANS) while listening to spoken utterances, have received far less attention than in other domains of emotion processing. Our study aimed at filling this gap by investigating autonomic activation in response to spoken utterances that were embedded into larger semantic contexts. Emotional salience was manipulated by providing information on alleged speaker similarity. We compared these autonomic responses to activations triggered by affective sounds, such as exploding bombs, and applause. These sounds had been rated and validated as being either positive, negative, or neutral. As physiological markers of ANS activity, we recorded skin conductance responses (SCRs) and changes of pupil size while participants classified both prosodic and sound stimuli according to their hedonic valence. As expected, affective sounds elicited increased arousal in the receiver, as reflected in increased SCR and pupil size. In contrast, SCRs to angry and joyful prosodic expressions did not differ from responses to neutral ones. Pupil size, however, was modulated by affective prosodic utterances, with increased dilations for angry and joyful compared to neutral prosody, although the similarity manipulation had no effect. These results indicate that cues provided by emotional prosody in spoken semantically neutral utterances might be too subtle to trigger SCR, although variation in pupil size indicated the salience of stimulus variation. Our findings further demonstrate a functional dissociation between pupil dilation and skin conductance that presumably origins from their differential innervation.

Neural measures of the role of affective prosody in empathy for pain

Emotional communication often needs the integration of affective prosodic and semantic components from speech and the speaker’s facial expression. Affective prosody may have a special role by virtue of its dual-nature; pre-verbal on one side and accompanying semantic content on the other. This consideration led us to hypothesize that it could act transversely, encompassing a wide temporal window involving the processing of facial expressions and semantic content expressed by the speaker. This would allow powerful communication in contexts of potential urgency such as witnessing the speaker’s physical pain. Seventeen participants were shown with faces preceded by verbal reports of pain. Facial expressions, intelligibility of the semantic content of the report (i.e., participants’ mother tongue vs. fictional language) and the affective prosody of the report (neutral vs. painful) were manipulated. We monitored event-related potentials (ERPs) time-locked to the onset of the faces as a function of semantic content intelligibility and affective prosody of the verbal reports. We found that affective prosody may interact with facial expressions and semantic content in two successive temporal windows, supporting its role as a transverse communication cue.

Experimental investigation of cognitive and affective empathy in borderline personality disorder: Effects of ambiguity in multimodal social information processing

Borderline personality disorder (BPD) is characterized by affective instability and interpersonal problems. In the context of social interaction, impairments in empathy are proposed to result in inadequate social behavior. In contrast to findings of reduced cognitive empathy, some authors suggested enhanced emotional empathy in BPD. It was investigated whether ambiguity leads to decreased cognitive or emotional empathy in BPD. Thirty-four patients with BPD and thirty-two healthy controls were presented with video clips, which were presented through prosody, facial expression, and speech content. Experimental conditions were designed to induce ambiguity by presenting neutral valence in one of these communication channels. Subjects were asked to indicate the actors' emotional valence, their decision confidence, and their own emotional state. BPD patients showed increased emotional empathy when neutral stories comprised nonverbally expressed emotions. In contrast, when all channels were emotional, patients showed lower emotional empathy than healthy controls. Regarding cognitive empathy, there were no significant differences between BPD patients and healthy control subjects in recognition accuracy, but reduced decision confidence in BPD. These results suggest that patients with BPD show altered emotional empathy, experiencing higher rates of emotional contagion when emotions are expressed nonverbally. The latter may contribute to misunderstandings and inadequate social behavior.

Facing the Problem: Impaired Emotion Recognition During Multimodal Social Information Processing in Borderline Personality Disorder

Previous research has revealed alterations and deficits in facial emotion recognition in patients with borderline personality disorder (BPD). During interpersonal communication in daily life, social signals such as speech content, variation in prosody, and facial expression need to be considered simultaneously. We hypothesized that deficits in higher level integration of social stimuli contribute to difficulties in emotion recognition in BPD, and heightened arousal might explain this effect. Thirty-one patients with BPD and thirty-one healthy controls were asked to identify emotions in short video clips, which were designed to represent different combinations of the three communication channels: facial expression, speech content, and prosody. Skin conductance was recorded as a measure of sympathetic arousal, while controlling for state dissociation. Patients with BPD showed lower mean accuracy scores than healthy control subjects in all conditions comprising emotional facial expressions. This was true for the condition with facial expression only, and for the combination of all three communication channels. Electrodermal responses were enhanced in BPD only in response to auditory stimuli. In line with the major body of facial emotion recognition studies, we conclude that deficits in the interpretation of facial expressions lead to the difficulties observed in multimodal emotion processing in BPD.

Computational Analysis and Simulation of Empathic Behaviors: a Survey of Empathy Modeling with Behavioral Signal Processing Framework

Empathy is an important psychological process that facilitates human communication and interaction. Enhancement of empathy has profound significance in a range of applications. In this paper, we review emerging directions of research on computational analysis of empathy expression and perception as well as empathic interactions, including their simulation. We summarize the work on empathic expression analysis by the targeted signal modalities (e.g., text, audio, and facial expressions). We categorize empathy simulation studies into theory-based emotion space modeling or application-driven user and context modeling. We summarize challenges in computational study of empathy including conceptual framing and understanding of empathy, data availability, appropriate use and validation of machine learning techniques, and behavior signal processing. Finally, we propose a unified view of empathy computation and offer a series of open problems for future research.

Brain responds to another person's eye blinks in a natural setting-the more empathetic the viewer the stronger the responses

An observer's brain is known to respond to another person's small nonverbal signals, such as gaze shifts and eye blinks. Here we aimed to find out how an observer's brain reacts to a speaker's eye blinks in the presence of other audiovisual information. Magnetoencephalographic brain responses along with eye gaze were recorded from 13 adults who watched a video of a person telling a story. The video was presented first without sound (visual), then with sound (audiovisual), and finally the audio story was presented with a still-frame picture on the screen (audio control). The viewers mainly gazed at the eye region of the speaker. Their saccades were suppressed at about 180 ms after the start of the speaker's blinks, a subsequent increase of saccade occurence to the base level, or higher, at around 340 ms. The suppression occurred in visual and audiovisual conditions but not during the control audio presentation. Prominent brain responses to blinks peaked in the viewer's occipital cortex at about 250 ms, with no differences in mean peak amplitudes or latencies between visual and audiovisual conditions. During the audiovisual, but not visual-only, presentation, the responses were the stronger the more empathetic the subject was according to the Empathic Concern score of the Interpersonal Reactivity Index questionnaire (Spearman's rank correlation, 0.73). The other person's eye blinks, nonverbal signs that often go unnoticed, thus elicited clear brain responses even in the presence of attention-attracting audiovisual information from the narrative, with stronger responses in people with higher empathy scores.

Emotional expressions beyond facial muscle actions. A call for studying autonomic signals and their impact on social perception

Humans are well adapted to quickly recognize and adequately respond to another’s emotions. Different theories propose that mimicry of emotional expressions (facial or otherwise) mechanistically underlies, or at least facilitates, these swift adaptive reactions. When people unconsciously mimic their interaction partner’s expressions of emotion, they come to feel reflections of those companions’ emotions, which in turn influence the observer’s own emotional and empathic behavior. The majority of research has focused on facial actions as expressions of emotion. However, the fact that emotions are not just expressed by facial muscles alone is often still ignored in emotion perception research. In this article, I therefore argue for a broader exploration of emotion signals from sources beyond the face muscles that are more automatic and difficult to control. Specifically, I will focus on the perception of implicit sources such as gaze and tears and autonomic responses such as pupil-dilation, eyeblinks and blushing that are subtle yet visible to observers and because they can hardly be controlled or regulated by the sender, provide important “veridical” information. Recently, more research is emerging about the mimicry of these subtle affective signals including pupil-mimicry. I will here review this literature and suggest avenues for future research that will eventually lead to a better comprehension of how these signals help in making social judgments and understand each other’s emotions.

Neural responses to dynamic multimodal stimuli and pathology-specific impairments of social cognition in schizophrenia and depression

BACKGROUND

Individuals with schizophrenia and people with depression both show abnormal behavioural and neural responses when perceiving and responding to emotional stimuli, but pathology-specific differences and commonalities remain mostly unclear.

AIMS

To directly compare empathic responses to dynamic multimodal emotional stimuli in a group with schizophrenia and a group with depression, and to investigate their neural correlates using functional magnetic resonance imaging (fMRI).

METHOD

The schizophrenia group (n = 20), the depression group (n = 24) and a control group (n = 24) were presented with portrait-shot video clips expressing emotion through three possible communication channels: facial expression, prosody and content. Participants rated their own and the actor's emotional state as an index of empathy.

RESULTS

Although no group differences were found in empathy ratings, characteristic differences emerged in the fMRI activation patterns. The schizophrenia group demonstrated aberrant activation patterns during the neutral speech content condition in regions implicated in multimodal integration and formation of semantic constructs. Those in the depression group were most affected during conditions with trimodal emotional and trimodal neutral stimuli, in key regions of the mentalising network.

CONCLUSIONS

Our findings reveal characteristic differences in patients with schizophrenia compared with those with depression in their cortical responses to dynamic affective stimuli. These differences indicate that impairments in responding to emotional stimuli may be caused by pathology-specific problems in social cognition.

The concepts of representation and information in explanatory theories of human behavior

Focusing in experimental study of human behavior, this article discusses the concepts of information and mental representation aiming the integration of their biological, computational, and semantic aspects. Assuming that the objective of any communication process is ultimately to modify the receiver's state, the term correlational information is proposed as a measure of how changes occurring in external world correlate with changes occurring inside an individual. Mental representations are conceptualized as a special case of information processing in which correlational information is received, recorded, but also modified by a complex emergent process of associating new elements. In humans, the acquisition of information and creation of mental representations occurs in a two-step process. First, a sufficiently complex brain structure is necessary to establishing internal states capable to co-vary with external events. Second, the validity or meaning of these representations must be gradually achieved by confronting them with the environment. This contextualization can be considered as part of the process of ascribing meaning to information and representations. The hypothesis introduced here is that the sophisticated psychological constructs classically associated with the concept of mental representation are essentially of the same nature of simple interactive behaviors. The capacity of generating elaborated mental phenomena like beliefs and desires emerges gradually during evolution and, in a given individual, by learning and social interaction.

Patterns of empathy as embodied practice in clinical conversation-a musical dimension

Cognitive linguistics and conversation analysis (a) converge in the analysis of category bound activities and (b) in viewing thinking and talking as embodied activities. The first aim of this paper is to outline these powerful theories as useful tools for the analysis of enacting empathy. The second aim is to outline these theories as useful tools for the analysis of how empathy is co-enacted in clinical conversation documented in transcripts. Cognitive Linguistics and Conversation Analysis converge in detecting patterns of I-You-relationships with roots in early preverbal embodied protoconversation continuing to more symbolic conversational level. The paper proposes to describe this continuity of empathic conversation in musical metaphors like balance, rhythm and resonance. In a first section transcripts from therapeutic sessions are presented. In a second section linguistic and other research data are presented in order to bring empirical data to this new conception of how empathy can be understood, how it is done and how two participants cooperate to enact empathy. Ideas for further research are outlined.

How can we explain physician accuracy in assessing patient distress? A multilevel analysis in patients with advanced cancer

OBJECTIVE

To examine the determinants of the accuracy with which physicians assess metastatic cancer patient distress, also referred to as their empathic accuracy (EA). Hypothesized determinants were physician empathic attitude, self-efficacy in empathic skills, physician-perceived rapport with the patient, patient distress and patient expressive suppression.

METHODS

Twenty-eight physicians assessed their patients' distress level on the distress thermometer, while patients (N=201) independently rated their distress level on the same tool. EA was the difference between both scores in absolute value. Hypothesized determinants were assessed using self-reported questionnaires. Multilevel analyses were carried out.

RESULTS

Little of the variance in EA was explained by physician variables. EA was higher with higher levels of patient distress. Physician-perceived quality of rapport was positively associated with EA. However, for highly distressed patients, good rapport was associated with lower EA. Patient expressive suppression was also related to lower EA.

CONCLUSION

This study adds to the understanding of EA in oncological settings, particularly in challenging the common assumption that EA depends largely on physician characteristics or that better rapport would always favor higher EA.

PRACTICE IMPLICATIONS

Physicians should ask patients for feedback regarding their emotions. In parallel, patients should be prompted to express their concerns.

Empathic competencies in violent offenders

Violent offending has often been associated with a lack of empathy, but experimental investigations are rare. The present study aimed at clarifying whether violent offenders show a general empathy deficit or specific deficits regarding the separate subcomponents. To this end, we assessed three core components of empathy (emotion recognition, perspective taking, affective responsiveness) as well as skin conductance response (SCR) in a sample of 30 male violent offenders and 30 healthy male controls. Data analysis revealed reduced accuracy in violent offenders compared to healthy controls only in emotion recognition, and that a high number of violent assaults was associated with decreased accuracy in perspective taking for angry scenes. SCR data showed reduced physiological responses in the offender group specifically for fear and disgust stimuli during emotion recognition and perspective taking. In addition, higher psychopathy scores in the violent offender group were associated with reduced accuracy in affective responsiveness. This is the first study to show that mainly emotion recognition is deficient in violent offenders whereas the other components of empathy are rather unaffected. This divergent impact of violent offending on the subcomponents of empathy suggests that all three empathy components can be targeted by therapeutic interventions separately.

Multisensory integration mechanisms during aging

The rapid demographical shift occurring in our society implies that understanding of healthy aging and age-related diseases is one of our major future challenges. Sensory impairments have an enormous impact on our lives and are closely linked to cognitive functioning. Due to the inherent complexity of sensory perceptions, we are commonly presented with a complex multisensory stimulation and the brain integrates the information from the individual sensory channels into a unique and holistic percept. The cerebral processes involved are essential for our perception of sensory stimuli and becomes especially important during the perception of emotional content. Despite ongoing deterioration of the individual sensory systems during aging, there is evidence for an increase in, or maintenance of, multisensory integration processing in aging individuals. Within this comprehensive literature review on multisensory integration we aim to highlight basic mechanisms and potential compensatory strategies the human brain utilizes to help maintain multisensory integration capabilities during healthy aging to facilitate a broader understanding of age-related pathological conditions. Further our goal was to identify where further research is needed.

Connecting multimodality in human communication

A successful reciprocal evaluation of social signals serves as a prerequisite for social coherence and empathy. In a previous fMRI study we studied naturalistic communication situations by presenting video clips to our participants and recording their behavioral responses regarding empathy and its components. In two conditions, all three channels transported congruent emotional or neutral information, respectively. Three conditions selectively presented two emotional channels and one neutral channel and were thus bimodally emotional. We reported channel-specific emotional contributions in modality-related areas, elicited by dynamic video clips with varying combinations of emotionality in facial expressions, prosody, and speech content. However, to better understand the underlying mechanisms accompanying a naturalistically displayed human social interaction in some key regions that presumably serve as specific processing hubs for facial expressions, prosody, and speech content, we pursued a reanalysis of the data. Here, we focused on two different descriptions of temporal characteristics within these three modality-related regions [right fusiform gyrus (FFG), left auditory cortex (AC), left angular gyrus (AG) and left dorsomedial prefrontal cortex (dmPFC)]. By means of a finite impulse response (FIR) analysis within each of the three regions we examined the post-stimulus time-courses as a description of the temporal characteristics of the BOLD response during the video clips. Second, effective connectivity between these areas and the left dmPFC was analyzed using dynamic causal modeling (DCM) in order to describe condition-related modulatory influences on the coupling between these regions. The FIR analysis showed initially diminished activation in bimodally emotional conditions but stronger activation than that observed in neutral videos toward the end of the stimuli, possibly by bottom-up processes in order to compensate for a lack of emotional information. The DCM analysis instead showed a pronounced top-down control. Remarkably, all connections from the dmPFC to the three other regions were modulated by the experimental conditions. This observation is in line with the presumed role of the dmPFC in the allocation of attention. In contrary, all incoming connections to the AG were modulated, indicating its key role in integrating multimodal information and supporting comprehension. Notably, the input from the FFG to the AG was enhanced when facial expressions conveyed emotional information. These findings serve as preliminary results in understanding network dynamics in human emotional communication and empathy.

Evidence for gender-specific endophenotypes in high-functioning autism spectrum disorder during empathy

Despite remarkable behavioral gender differences in patients with autism spectrum disorder (ASD), and growing evidence for a diminished male : female ratio for the putative "male disorder" ASD, aspects of gender are not addressed accordingly in ASD research. Our study aims at filling this gap by exploring empathy abilities in a group of 28 patients with high-functioning ASD and 28 gender-, age- and education-matched non-autistic subjects, for the first time by means of functional neuroimaging (fMRI). In an event-related fMRI paradigm, emotional ("E") and neutral ("N") video clips presented actors telling self-related short stories. After each clip, participants were asked to indicate their own emotion and its intensity as well as the emotion and intensity perceived for the actor. Behaviorally, we found significantly less empathic responses in the overall ASD group compared with non-autistic subjects, and inadequate emotion recognition for the neutral clips in the female ASD group compared with healthy women. Neurally, increased activation of the bilateral medial frontal gyrus was found in male patients compared with female patients, a pattern which was not present in the non-autistic group. Additionally, autistic women exhibited decreased activation of midbrain and limbic regions compared with non-autistic women, whereas there was no significant difference within the male group. While we did not find a fundamental empathic deficit in autistic patients, our data propose different ways of processing empathy in autistic men and women, suggesting stronger impairments in cognitive aspects of empathy/theory of mind for men, and alterations of social reciprocity for women.

The psychological well-being of couples experiencing a chronic illness: A matter of personal and partner illness cognitions and the role of marital quality

This study aimed to examine the effects of 75 cardiac patients’ and their spouses’ illness cognitions of acceptance and helplessness on their own and their partner’s psychological symptoms, as well as the impact of marital quality. Dyadic responses were examined with the Actor–Partner Interdependence Model, while PROCESS was used to examine possible conditional indirect effects. Patients’ and spouses’ cognitions were related to their partners’ symptoms through the partners’ cognitions. These relationships were not statistically significant at the higher levels of marital quality. Thus, there seems to be a flow of information between partners, while marital quality may impact self-regulation processes.

Emotional speech processing at the intersection of prosody and semantics

The ability to accurately perceive emotions is crucial for effective social interaction. Many questions remain regarding how different sources of emotional cues in speech (e.g., prosody, semantic information) are processed during emotional communication. Using a cross-modal emotional priming paradigm (Facial affect decision task), we compared the relative contributions of processing utterances with single-channel (prosody-only) versus multi-channel (prosody and semantic) cues on the perception of happy, sad, and angry emotional expressions. Our data show that emotional speech cues produce robust congruency effects on decisions about an emotionally related face target, although no processing advantage occurred when prime stimuli contained multi-channel as opposed to single-channel speech cues. Our data suggest that utterances with prosodic cues alone and utterances with combined prosody and semantic cues both activate knowledge that leads to emotional congruency (priming) effects, but that the convergence of these two information sources does not always heighten access to this knowledge during emotional speech processing.