When's a Smile a Smile? Or how to Detect a Message by Digitizing the Signal

A Pilot Trial of Dopamine Replacement for Dynamic Facial Expressions in Parkinson's Disease

Background

Current conflict exists regarding the potential beneficial effects of dopamine medications on facial expressivity in Parkinson's disease. Via digital video analysis software, we previously found reduced facial movement (entropy) and slower time to reach peak entropy in individuals with Parkinson's disease compared to controls.

Objectives

We aimed to determine whether levodopa medications improved parameters of dynamic facial expressions (amplitude, speed).

Methods

A total of 34 individuals with idiopathic Parkinson's disease were videotaped making voluntary facial expressions (happy, fear, anger, disgust) when "on" and "off" levodopa. Participants were 52 to 80 years old, early to mid-stage disease, non-demented, and included more men (65%). Expressions were digitized and analyzed using software that extracted three variables: two indices of movement change (total entropy, percent entropy change) and time to reach peak expression.

Results

Indices of facial movement (total entropy, peak entropy) and timing were significantly improved when patients were "on" vs "off" medication (all F's ≥ 3.00, P < 0.05). For total movement and time to reach peak entropy, levodopa-related improvements were emotion nonspecific. Levodopa-related improvement for peak entropy was driven primarily by happy expressions. There was no relationship between quantitative indices and clinical measures of mood (depression, anxiety) and motor disease severity.

Conclusion

The effects of levodopa on Parkinson's disease voluntary facial movement and on timing were robust and consistent with those of levodopa on other intentional movements in Parkinson's disease. This improvement possibly occurred because of levodopa enhanced activation of face representation areas in fronto-cortical regions or because of less movement-based suppression.

Laterality of motor symptom onset and facial expressivity in Parkinson disease using face digitization

The onset of motor symptoms in Parkinson disease (PD) is typically unilateral. Previous work has suggested that laterality of motor symptoms may also influence non-motor symptoms including cognition and emotion perception. In line with hemispheric differences in emotion processing, we tested whether left side/right brain motor onset was associated with worse expression of facial affect when compared to right side/left brain motor onset. We evaluated movement changes associated with facial affect in 30 patients with idiopathic PD (15 left-sided motor onset, 15 right-sided motor onset) and 20 healthy controls. Participants were videotaped while posing three facial expressions: fear, anger, and happiness. Expressions were digitized and analyzed using software that extracted three variables: two measures of dynamic movement change (total entropy and entropy percent change) and a measure of time to initiate facial expression (latency). The groups did not differ in overall amount of movement change or percentchange. However, left-sided onset PD patients were significantly slower in initiating anger and happiness facial expressions than were right-sided onset PD patients and controls. Our results indicated PD patients with left-sided symptom onset had greater latency in initiating two of three facial expressions, which may reflect laterality effects in intentional behaviour.

Faces, people and the brain: The 45th Sir Frederic Bartlett Lecture

The fact that the face is a source of diverse social signals allows us to use face and person perception as a model system for asking important psychological questions about how our brains are organised. A key issue concerns whether we rely primarily on some form of generic representation of the common physical source of these social signals (the face) to interpret them, or instead create multiple representations by assigning different aspects of the task to different specialist components. Variants of the specialist components hypothesis have formed the dominant theoretical perspective on face perception for more than three decades, but despite this dominance of formally and informally expressed theories, the underlying principles and extent of any division of labour remain uncertain. Here, I discuss three important sources of constraint: first, the evolved structure of the brain; second, the need to optimise responses to different everyday tasks; and third, the statistical structure of faces in the perceiver’s environment. I show how these constraints interact to determine the underlying functional organisation of face and person perception.

Recognition of facial expressions during step-function stroboscopic presentation

We studied the perception of human facial emotional expressions during step-function stroboscopic presentation of changing mimics. Consecutive stages of each of the six basic facial expressions were pre sented to the participants: neutral face (300 ms) — expression of medium intensity (10—40 ms) — intense expression (30—120 ms) — expression of medium intensity (10—40 ms) — neutral face (100 ms). Alternative forced choice task was used to categorize the facial expressions. The results were compared to previous studies (Barabanschikov, Korolkova, Lobodinskaya, 2015; 2016), conducted using the same paradigm but with boxcar-function change of the expression: neutral face — intense expression — neutral face. We found that the dynamics of facial expression recognition, as well as errors and recognition time are almost identical in conditions of boxcar- and step-function presentation. One factor influencing the recognition rate is the proportion of presentation time of static (neutral) and changing (facial expression) aspects of the stimulus. In suboptimal conditions of facial expression perception (minimal presentation time of 10+30+10 ms and reduced intensity of expressions) we revealed stroboscopic sensibilization — a previously described phenomenon of enhanced recognition rate of low-attractive expressions (disgust, sadness, fear and anger), which has been previously found in conditions of boxcar-function presentation of expressions. We confirmed the similarity of influence of real and apparent motion on the recognition of basic facial emotional expressions.

Why Smiles Generate Leniency

The authors investigated the social significance of human smiles, specifically the penchant for transgressors who smile to be judged more leniently than those who do not. Of particular interest was whether different types of smiles generate different degrees of leniency and what mediated the effect. Subjects judged a case of possible academic misconduct. Materials included a photograph of a female target displaying a neutral expression, felt smile, false smile, or miserable smile. Smiling targets received more leniency than nonsmiling targets, although they were not seen as less guilty. The type of smile did not significantly moderate the amount of leniency shown. Of the variables evaluated for mediating the smile-leniency effect, such as perceiving the target as more likable, submissive, or diplomatic, the one that best accounted for the effect was perceiving the target as a trustworthy person. The implication seems to be—if in trouble, smile.

Effects of Dynamic Aspects of Facial Expressions: A Review

A key feature of facial behavior is its dynamic quality. However, most previous research has been limited to the use of static images of prototypical expressive patterns. This article explores the role of facial dynamics in the perception of emotions, reviewing relevant empirical evidence demonstrating that dynamic information improves coherence in the identification of affect (particularly for degraded and subtle stimuli), leads to higher emotion judgments (i.e., intensity and arousal), and helps to differentiate between genuine and fake expressions. The findings underline that using static expressions not only poses problems of ecological validity, but also limits our understanding of what facial activity does. Implications for future research on facial activity, particularly for social neuroscience and affective computing, are discussed.

Comparison of Deliberate and Spontaneous Facial Movement in Smiles and Eyebrow Raises

We investigated movement differences between deliberately posed and spontaneously occurring smiles and eyebrow raises during a videotaped interview that included a facial movement assessment. Using automated facial image analysis, we quantified lip corner and eyebrow movement during periods of visible smiles and eyebrow raises and compared facial movement within participants. As in an earlier study, maximum speed of movement onset was greater in deliberate smiles. Maximum speed and amplitude were greater and duration shorter in deliberate compared to spontaneous eyebrow raises. Asymmetry of movement did not differ within participants. Similar patterns contrasting deliberate and spontaneous movement in both smiles and eyebrow raises suggest a common pattern of signaling for spontaneous facial displays.

The thrill of victory and the agony of defeat: spontaneous expressions of medal winners of the 2004 Athens Olympic Games

Facial behaviors of medal winners of the judo competition at the 2004 Athens Olympic Games were coded with P. Ekman and W. V. Friesen's (1978) Facial Affect Coding System (FACS) and interpreted using their Emotion FACS dictionary. Winners' spontaneous expressions were captured immediately when they completed medal matches, when they received their medal from a dignitary, and when they posed on the podium. The 84 athletes who contributed expressions came from 35 countries. The findings strongly supported the notion that expressions occur in relation to emotionally evocative contexts in people of all cultures, that these expressions correspond to the facial expressions of emotion considered to be universal, that expressions provide information that can reliably differentiate the antecedent situations that produced them, and that expressions that occur without inhibition are different than those that occur in social and interactive settings.

Faces of emotion in Parkinsons disease: micro-expressivity and bradykinesia during voluntary facial expressions

In humans, the neural circuitry underlying facial expressions differs, depending on whether facial expressions are spontaneously (i.e., limbic, subcortical) or voluntarily initiated (i.e., frontal cortex). Previous investigators have suggested that the "masked face" of Parkinson's disease involves spontaneous, but not intentional, facial expressions. In contrast, we hypothesized that intentional facial expressions may be slowed (bradykinetic) and involve less movement, in much the same way that other intentional movements are affected by Parkinson's disease. To test this hypothesis, we used sophisticated computer imaging techniques to quantify dynamic facial movement. Relative to controls, Parkinson patients had reduced facial movement (entropy) and were significantly slowed in reaching a peak expression (i.e., bradykinesia). These findings are consistent with the view that the basal ganglia play a role in affecting intentional facial movements. This possibly occurs because of diminished efficiency and/or activation of face representation areas in the frontal cortical regions (i.e., motor, premotor, and supplementary motor area) or because of movement-based suppression secondary to dopaminergic reduction in frontostriatal pathways. Taken together, the characterization of Parkinson's disease as a model system for the neuroanatomic dissociation between voluntary and spontaneous expressions may be unjustified.

Long-term habituation of the smile response with deep brain stimulation

Human and animal research has shown that the ventral striatum, including the nucleus accumbens, may play a critical role in mediating positive emotions. Recently we described a subject with obsessive-compulsive disorder who intra-operatively exhibited the acute onset of an asymmetric smile and acute positive emotional change with contralateral deep brain stimulation (DBS) in either the right or left nucleus accumbens and anterior limb of the internal capsule region. The purpose of the present study was to examine the stability of the stimulation-induced smile(s) over a 12-month period. Custom computer software objectively quantified left and right facial movement during DBS. Although stimulation-induced smiles were elicited at one and two months post-surgery, they were no longer present from 3-12 months following chronic high frequency DBS. The smiles could not be elicited even with long washout periods. These findings imply potential long-term habituation and changes in the neural chemistry (possibly neuroplasticity) induced by chronic DBS.

What's in a "smile?" Intra-operative observations of contralateral smiles induced by deep brain stimulation

OBJECTIVE

To describe smiling and euphoria induced by deep brain stimulation (DBS).

BACKGROUND AND SIGNIFICANCE

The brain systems inducing emotional experiences and displays are not entirely known, but the ventral striatum including the nucleus accumbens has been posited to play a critical role in mediating emotions with positive valence. DBS has been successfully employed for the treatment of movement disorders, and most recently obsessive compulsive disorder (OCD). The purpose of this report is to describe the emotional changes associated with stimulation of the ventral striatum.

METHODS

A single patient with intractable OCD had electrode arrays placed in the right and left anterior limbs of the internal capsule and region of the nucleus accumbens. Changes in facial movement during stimulation were quantified by video recording. Ten video segments, time locked to the onset of stimulation, were digitized and changes in pixel intensity that occurred over both sides of the lower face, on a frame by frame basis, following stimulation onset were computed. These summed changes in pixel intensity represented the dependent variable of "entropy" and directly corresponded to changes in light reflectance that occur during facial movement.

RESULTS

During stimulation on both the right and left side, the patient consistently developed a half smile on the side of the face contralateral to the stimulating electrode, and also became euphoric. The effect ceased when DBS was discontinued.

CONCLUSIONS

DBS in the region of the nucleus accumbens produced smile and euphoria suggesting that alterations in the ventral striatum may result in emotional experience and displays. We hypothesize the existence of a limbic-motor network responsible for such changes. This observation suggests that DBS may be useful as a therapy for mood disorders.

Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles

The assumption that the smile is an evolved facial display suggests that there may be universal features of smiling in addition to the basic facial configuration. We show that smiles include not only a stable configuration of features, but also temporally consistent movement patterns. In spontaneous smiles from two social contexts, duration of lip corner movement during the onset phase was independent of social context and the presence of other facial movements, including dampening. These additional movements produced variation in both peak and offset duration. Both onsets and offsets had dynamic properties similar to automatically controlled movements, with a consistent relation between maximum velocity and amplitude of lip corner movement in smiles from two distinct contexts. Despite the effects of individual and social factors on facial expression timing overall, consistency in onset and offset phases suggests that portions of the smile display are relatively stereotyped and may be automatically produced.

Dissociable neural pathways are involved in the recognition of emotion in static and dynamic facial expressions

Facial expressions of emotion powerfully influence social behavior. The distributed network of brain regions thought to decode these social signals has been empirically defined using static, usually photographic, displays of such expressions. Facial emotional expressions are however highly dynamic signals that encode the emotion message in facial action patterns. This study sought to determine whether the encoding of facial expressions of emotion by static or dynamic displays is associated with different neural correlates for their decoding. We used positron emission tomography to compare patterns of brain activity in healthy men and women during the explicit judgment of emotion intensity in static and dynamic facial expressions of anger and happiness. Compared to judgments of spatial orientation for moving neutral facial expressions, the judgment of anger in dynamic expressions was associated with increased right-lateralized activity in the medial, superior, middle, and inferior frontal cortex and cerebellum, while judgments of happiness were associated with relative activation of the cuneus, temporal cortex, and the middle, medial, and superior frontal cortex. In contrast, the perception of anger or happiness in static facial expressions activated a motor, prefrontal, and parietal cortical network previously shown to be involved in motor imagery. The direct contrast of dynamic and static expressions indicated differential activation of visual area V5, superior temporal sulcus, periamygdaloid cortex, and cerebellum for dynamic angry expressions and differential activation of area V5, extrastriate cortex, brain stem, and middle temporal cortical activations for dynamic happy expressions. Thus, a distribution of neural activations is related to the analysis of emotion messages in the nearly constant biological motion of the face and differ for angry and happy expressions. Static displays of facial emotional expression may represent noncanonical stimuli that are processed for emotion content by mental strategies and neural events distinct from their more ecologically relevant dynamic counterparts.

Digitizing the moving face during dynamic displays of emotion

Humans typically decode facial signals during dynamic interactions in which the face moves. In this study, we digitized real time video signals in order to examine movement asymmetries across the face during emotional and nonemotional expressions. Forty dextral males were tested. For each expression, a 400 ms video segment was analyzed for changes in signal value (pixel intensity) over consecutive frames. The upper and lower face regions were examined separately due to differences in the cortical enervation of facial muscles in the upper (bilateral) vs lower face (contralateral). Results revealed distinctly different movement asymmetries over the lower and upper hemiface. In the upper face, more movement occurred over the right side for most facial expressions, regardless of emotionality. The latter finding questions the assumption that muscles of the upper face are symmetrical and/or bilaterally enervated in a symmetrical manner. In the lower face, negative expressions linked to fight-flight emotions (i.e. fear, anger) were associated with greater left sided movement, whereas happiness tended to be associated with more right sided movement. No consistent pattern of movement asymmetry occurred for nonemotional expressions. Although the valence-related movement asymmetries in the lower face are consistent with neuropsychological models of emotional expressivity, it remains unclear whether they reflect activation or inhibitory hemispheric mechanisms. Taken together, these data suggest that multiple factors may contribute to expressive movement asymmetries of the face.

Evolution and facial action in reflex, social motive, and paralanguage

Based upon current evolutionary theory and recent laboratory and field data, this paper introduces a behavioral-ecology view of human facial displays that contrasts with previous views of faces as innate, prototypic, "iconic" expressions of fundamental emotions. First, I detail the criteria for establishing genetic and epigenetic contributions to facial behavior. Under these criteria, cross-cultural communality in canonical facial displays implies neither their genetic control nor their fundamental relation to emotion. New findings reestablish phylogenetic continuity by showing that human facial displays and vigilance for them, like their nonhuman counterparts, coevolve in the service of social motives; faces issued in solitude typically reflect imaginary or implicit interaction. Two evolutionary trends are probably involved in shaping facial actions--emancipation of reflexes, and automatization of instrumental acts. Modern conceptions of genetic and cultural evolution, and knowledge about animal signaling, suggest new hypotheses regarding the origins and functions of facial displays. In so doing, they clarify how facial displays relate to reflexion, motive and intention, emotion and psychophysiology, and language and paralanguage.