Comparing supervised and unsupervised approaches to emotion categorization in the human brain, body, and subjective experience

Fear-related psychophysiological patterns are situation and individual dependent: A Bayesian model comparison approach

Is there a universal mapping of physiology to emotion, or do these mappings vary substantially by person or situation? Psychologists, philosophers, and neuroscientists have debated this question for decades. Most previous studies have focused on differentiating emotions on the basis of accompanying autonomic responses using analytical approaches that often assume within-category homogeneity. In the present study, we took an alternative approach to this question. We determined the extent to which the relationship between subjective experience and autonomic reactivity generalizes across, or depends upon, the individual and situation for instances of a single emotion category, specifically, fear. Electrodermal activity and cardiac activity-two autonomic measures that are often assumed to show robust relationships with instances of fear-were recorded while participants reported fear experience in response to dozens of fear-evoking videos related to three distinct situations: spiders, heights, and social encounters. We formally translated assumptions from diverse theoretical models into a common framework for model comparison analyses. Results exceedingly favored a model that assumed situation-dependency in the relationship between fear experience and autonomic reactivity, with subject variance also significant but constrained by situation. Models that assumed generalization across situations and/or individuals performed much worse by comparison. These results call into question the assumption of generalizability of autonomic-subjective mappings across instances of fear, as required in translational research from nonhuman animals to humans, and advance a situated approach to understanding the autonomic correlates of fear experience. (PsycInfo Database Record (c) 2024 APA, all rights reserved).

Inferring Causal Factors of Core Affect Dynamics on Social Participation through the Lens of the Observer

A core endeavour in current affective computing and social signal processing research is the construction of datasets embedding suitable ground truths to foster machine learning methods. This practice brings up hitherto overlooked intricacies. In this paper, we consider causal factors potentially arising when human raters evaluate the affect fluctuations of subjects involved in dyadic interactions and subsequently categorise them in terms of social participation traits. To gauge such factors, we propose an emulator as a statistical approximation of the human rater, and we first discuss the motivations and the rationale behind the approach.The emulator is laid down in the next section as a phenomenological model where the core affect stochastic dynamics as perceived by the rater are captured through an Ornstein-Uhlenbeck process; its parameters are then exploited to infer potential causal effects in the attribution of social traits. Following that, by resorting to a publicly available dataset, the adequacy of the model is evaluated in terms of both human raters' emulation and machine learning predictive capabilities. We then present the results, which are followed by a general discussion concerning findings and their implications, together with advantages and potential applications of the approach.

Improving the study of brain-behavior relationships by revisiting basic assumptions

Neuroimaging research has been at the forefront of concerns regarding the failure of experimental findings to replicate. In the study of brain-behavior relationships, past failures to find replicable and robust effects have been attributed to methodological shortcomings. Methodological rigor is important, but there are other overlooked possibilities: most published studies share three foundational assumptions, often implicitly, that may be faulty. In this paper, we consider the empirical evidence from human brain imaging and the study of non-human animals that calls each foundational assumption into question. We then consider the opportunities for a robust science of brain-behavior relationships that await if scientists ground their research efforts in revised assumptions supported by current empirical evidence.

Searching, Navigating, and Recommending Movies through Emotions: A Scoping Review

Movies offer viewers a broad range of emotional experiences, providing entertainment, and meaning. Following the PRISMA-ScR guidelines, we reviewed the literature on digital systems designed to help users search and browse movie libraries and offer recommendations based on emotional content. Our search yielded 83 eligible documents (published between 2000 and 2021). We identified 22 case studies, 34 empirical studies, 26 proof of concept, and one theoretical paper. User transactions (e.g., ratings, tags) were the preferred source of information. The documents examined approached emotions from both a categorical ( $n=35$ ) and dimensional ( $n=18$ ) perspectives, and nine documents offer a combination of both approaches. Although there are several authors mentioned, the references used are frequently dated, and 12 documents do not mention author or model used. We identified 61 words related to emotion or affect. Documents presented on average 1.36 positive terms and 2.64 negative terms. Sentiment analysis ( $n=31$ ) is frequently used for emotion identification, followed by subjective evaluations ( $n=15$ ), movie low-level audio and visual features (n = 11), and face recognition technologies ( $n=8$ ). We discuss limitations and offer a brief review of current emotion models and research.

Context reconsidered: Complex signal ensembles, relational meaning, and population thinking in psychological science

This article considers the status and study of "context" in psychological science through the lens of research on emotional expressions. The article begins by updating three well-trod methodological debates on the role of context in emotional expressions to reconsider several fundamental assumptions lurking within the field's dominant methodological tradition: namely, that certain expressive movements have biologically prepared, inherent emotional meanings that issue from singular, universal processes which are independent of but interact with contextual influences. The second part of this article considers the scientific opportunities that await if we set aside this traditional understanding of "context" as a moderator of signals with inherent psychological meaning and instead consider the possibility that psychological events emerge in ecosystems of signal ensembles, such that the psychological meaning of any individual signal is entirely relational. Such a fundamental shift has radical implications not only for the science of emotion but for psychological science more generally. It offers opportunities to improve the validity and trustworthiness of psychological science beyond what can be achieved with improvements to methodological rigor alone. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

A new science of emotion: implications for functional neurological disorder

Functional neurological disorder reflects impairments in brain networks leading to distressing motor, sensory and/or cognitive symptoms that demonstrate positive clinical signs on examination incongruent with other conditions. A central issue in historical and contemporary formulations of functional neurological disorder has been the mechanistic and aetiological role of emotions. However, the debate has mostly omitted fundamental questions about the nature of emotions in the first place. In this perspective article, we first outline a set of relevant working principles of the brain (e.g. allostasis, predictive processing, interoception and affect), followed by a focused review of the theory of constructed emotion to introduce a new understanding of what emotions are. Building on this theoretical framework, we formulate how altered emotion category construction can be an integral component of the pathophysiology of functional neurological disorder and related functional somatic symptoms. In doing so, we address several themes for the functional neurological disorder field including: (i) how energy regulation and the process of emotion category construction relate to symptom generation, including revisiting alexithymia, 'panic attack without panic', dissociation, insecure attachment and the influential role of life experiences; (ii) re-interpret select neurobiological research findings in functional neurological disorder cohorts through the lens of the theory of constructed emotion to illustrate its potential mechanistic relevance; and (iii) discuss therapeutic implications. While we continue to support that functional neurological disorder is mechanistically and aetiologically heterogenous, consideration of how the theory of constructed emotion relates to the generation and maintenance of functional neurological and functional somatic symptoms offers an integrated viewpoint that cuts across neurology, psychiatry, psychology and cognitive-affective neuroscience.

Neural signatures of individual variability in context-dependent perception of ambiguous facial expression

How do we incorporate contextual information to infer others' emotional state? Here we employed a naturalistic context-dependent facial expression estimation task where participants estimated pleasantness levels of others' ambiguous expression faces when sniffing different contextual cues (e.g., urine, fish, water, and rose). Based on their pleasantness rating data, we placed participants on a context-dependency continuum and mapped the individual variability in the context-dependency onto the neural representation using a representational similarity analysis. We found that the individual variability in the context-dependency of facial expression estimation correlated with the activity level of the pregenual anterior cingulate cortex (pgACC) and the amygdala and was also decoded by the neural representation of the ventral anterior insula (vAI). A dynamic causal modeling revealed that those with higher context-dependency exhibited a greater degree of the modulation from vAI to the pgACC. These findings provide novel insights into the neural circuitry associated with the individual variability in context-dependent facial expression estimation and the first empirical evidence for individual variability in the predictive accounts of affective states.

Unsupervised classification reveals consistency and degeneracy in neural network patterns of emotion

In the present study, we used an unsupervised classification algorithm to reveal both consistency and degeneracy in neural network connectivity during anger and anxiety. Degeneracy refers to the ability of different biological pathways to produce the same outcomes. Previous research is suggestive of degeneracy in emotion, but little research has explicitly examined whether degenerate functional connectivity patterns exist for emotion categories such as anger and anxiety. Twenty-four subjects underwent functional magnetic resonance imaging (fMRI) while listening to unpleasant music and self-generating experiences of anger and anxiety. A data-driven model building algorithm with unsupervised classification (subgrouping Group Iterative Multiple Model Estimation) identified patterns of connectivity among 11 intrinsic networks that were associated with anger vs anxiety. As predicted, degenerate functional connectivity patterns existed within these overarching consistent patterns. Degenerate patterns were not attributable to differences in emotional experience or other individual-level factors. These findings are consistent with the constructionist account that emotions emerge from flexible functional neuronal assemblies and that emotion categories such as anger and anxiety each describe populations of highly variable instances.

The Temporal Dynamics of Spontaneous Emotional Brain States and Their Implications for Mental Health

Temporal processes play an important role in elaborating and regulating emotional responding during routine mind wandering. However, it is unknown whether the human brain reliably transitions among multiple emotional states at rest and how psychopathology alters these affect dynamics. Here, we combined pattern classification and stochastic process modeling to investigate the chronometry of spontaneous brain activity indicative of six emotions (anger, contentment, fear, happiness, sadness, and surprise) and a neutral state. We modeled the dynamic emergence of these brain states during resting-state fMRI and validated the results across two population cohorts-the Duke Neurogenetics Study and the Nathan Kline Institute Rockland Sample. Our findings indicate that intrinsic emotional brain dynamics are effectively characterized as a discrete-time Markov process, with affective states organized around a neutral hub. The centrality of this network hub is disrupted in individuals with psychopathology, whose brain state transitions exhibit greater inertia and less frequent resetting from emotional to neutral states. These results yield novel insights into how the brain signals spontaneous emotions and how alterations in their temporal dynamics contribute to compromised mental health.

Automatic Decision-Making Style Recognition Method Using Kinect Technology

In recent years, somatosensory interaction technology, represented by Microsoft's Kinect hardware platform, has been widely used in various fields, such as entertainment, education, and medicine. Kinect technology can easily capture and record behavioral data, which provides new opportunities for behavioral and psychological correlation analysis research. In this paper, an automatic decision-style recognition method is proposed. Experiments involving 240 subjects were conducted to obtain face data and individual decision-making style score. The face data was obtained using the Kinect camera, and the decision-style score were obtained via a questionnaire. To realize automatic recognition of an individual decision-making style, machine learning was employed to establish the mapping relationship between the face data and a scaled evaluation of the decision-making style score. This study adopts a variety of classical machine learning algorithms, including Linear regression, Support vector machine regression, Ridge regression, and Bayesian ridge regression. The experimental results show that the linear regression model returns the best results. The correlation coefficient between the linear regression model evaluation results and the scale evaluation results was 0.6, which represents a medium and higher correlation. The results verify the feasibility of automatic decision-making style recognition method based on facial analysis.

A Computational Neural Model for Mapping Degenerate Neural Architectures

Degeneracy in biological systems refers to a many-to-one mapping between physical structures and their functional (including psychological) outcomes. Despite the ubiquity of the phenomenon, traditional analytical tools for modeling degeneracy in neuroscience are extremely limited. In this study, we generated synthetic datasets to describe three situations of degeneracy in fMRI data to demonstrate the limitations of the current univariate approach. We describe a novel computational approach for the analysis referred to as neural topographic factor analysis (NTFA). NTFA is designed to capture variations in neural activity across task conditions and participants. The advantage of this discovery-oriented approach is to reveal whether and how experimental trials and participants cluster into task conditions and participant groups. We applied NTFA on simulated data, revealing the appropriate degeneracy assumption in all three situations and demonstrating NTFA's utility in uncovering degeneracy. Lastly, we discussed the importance of testing degeneracy in fMRI data and the implications of applying NTFA to do so.

Comparing supervised and unsupervised approaches to multimodal emotion recognition

We investigated emotion classification from brief video recordings from the GEMEP database wherein actors portrayed 18 emotions. Vocal features consisted of acoustic parameters related to frequency, intensity, spectral distribution, and durations. Facial features consisted of facial action units. We first performed a series of person-independent supervised classification experiments. Best performance (AUC = 0.88) was obtained by merging the output from the best unimodal vocal (Elastic Net, AUC = 0.82) and facial (Random Forest, AUC = 0.80) classifiers using a late fusion approach and the product rule method. All 18 emotions were recognized with above-chance recall, although recognition rates varied widely across emotions (e.g., high for amusement, anger, and disgust; and low for shame). Multimodal feature patterns for each emotion are described in terms of the vocal and facial features that contributed most to classifier performance. Next, a series of exploratory unsupervised classification experiments were performed to gain more insight into how emotion expressions are organized. Solutions from traditional clustering techniques were interpreted using decision trees in order to explore which features underlie clustering. Another approach utilized various dimensionality reduction techniques paired with inspection of data visualizations. Unsupervised methods did not cluster stimuli in terms of emotion categories, but several explanatory patterns were observed. Some could be interpreted in terms of valence and arousal, but actor and gender specific aspects also contributed to clustering. Identifying explanatory patterns holds great potential as a meta-heuristic when unsupervised methods are used in complex classification tasks.

Variation is the Norm: Brain State Dynamics Evoked By Emotional Video Clips

For the last several decades, emotion research has attempted to identify a "biomarker" or consistent pattern of brain activity to characterize a single category of emotion (e.g., fear) that will remain consistent across all instances of that category, regardless of individual and context. In this study, we investigated variation rather than consistency during emotional experiences while people watched video clips chosen to evoke instances of specific emotion categories. Specifically, we developed a sequential probabilistic approach to model the temporal dynamics in a participant's brain activity during video viewing. We characterized brain states during these clips as distinct state occupancy periods between state transitions in blood oxygen level dependent (BOLD) signal patterns. We found substantial variation in the state occupancy probability distributions across individuals watching the same video, supporting the hypothesis that when it comes to the brain correlates of emotional experience, variation may indeed be the norm.