Cortical processing of head- and eye-gaze cues guiding joint social attention

Look past the cooperative eye hypothesis: reconsidering the evolution of human eye appearance

The external appearance of the human eye has been prominently linked to the evolution of complex sociocognitive functions in our species. The cooperative eye hypothesis (CEH) proposes that human eyeballs, with their weakly expressed conjunctival and scleral pigmentation, are uniquely conspicuous and evolved under selective pressures to behave cooperatively, therefore signalling attentiveness to conspecifics. Non-human primates are instead assumed to display less-salient eye morphologies that help mask their gaze to facilitate competitive, rather than cooperative actions. Here, we argue that the CEH, although continuing to be influential, lacks robust empirical support. Over the past two decades, multidisciplinary research has undermined its original rationale and central premises: human eye pigmentation does not uniquely stand out among primates, it is not uniform at species level and the available evidence does not conclusively suggest that it facilitates gaze following to notable extents. Hence, the CEH currently provides a theoretical framework that risks confusing, rather than informing, inferences about the evolution of human external eye appearance and its selective drivers. In a call to move past it, we review alternative hypotheses with the potential to elucidate the emergence of the human ocular phenotype from the considerable spectrum of diversity found within the primate order.

Investigating the impact of schizophrenia traits on attention: the role of the theta band in a modified Posner cueing paradigm

Joint attention is an indispensable tool for daily communication. Abnormalities in joint attention may be a key reason underlying social impairment in schizophrenia spectrum disorders. In this study, we aimed to explore the attentional orientation mechanism related to schizotypal traits in a social situation. Here, we employed a Posner cueing paradigm with social attentional cues. Subjects needed to detect the location of a target that is cued by gaze and head orientation. The power in the theta frequency band was used to examine the attentional process in the schizophrenia spectrum. There were four main findings. First, a significant association was found between schizotypal traits and attention orientation in response to invalid gaze cues. Second, individuals with schizotypal traits exhibited significant activation of neural oscillations and synchrony in the theta band, which correlated with their schizotypal tendencies. Third, neural oscillations and synchrony demonstrated a synergistic effect during social tasks, particularly when processing gaze cues. Finally, the relationship between schizotypal traits and attention orientation was mediated by neural oscillations and synchrony in the theta frequency band. These findings deepen our understanding of the impact of theta activity in schizotypal traits on joint attention and offer new insights for future intervention strategies.

When "more for others, less for self" leads to co-benefits: A tri-MRI dyad-hyperscanning study

Unselfishness is admired, especially when collaborations between groups of various scales are urgently needed. However, its neural mechanisms remain elusive. In a tri-MRI dyad-hyperscanning experiment involving 26 groups, each containing 4 participants as two rotating pairs in a coordination game, we sought to achieve reciprocity, or "winning in turn by the two interacting players," as the precursor to unselfishness. Due to its critical role in social processing, the right temporal-parietal junction (rTPJ) was the seed for both time domain (connectivity) and frequency domain (i.e., coherence) analyses. For the former, negative connectivity between the rTPJ and the mentalizing network areas (e.g., the right inferior parietal lobule, rIPL) was identified, and such connectivity was further negatively correlated with the individual's final gain, supporting our task design that "rewarded" the reciprocal participants. For the latter, cerebral coherences of the rTPJs emerged between the interacting pairs (i.e., within-group interacting pairs), and the coupling between the rTPJ and the right superior temporal gyrus (rSTG) between the players who were not interacting with each other (i.e., within-group noninteracting pairs). These coherences reinforce the hypotheses that the rTPJ-rTPJ coupling tracks the collaboration processes and the rTPJ-rSTG coupling for the emergence of decontextualized shared meaning. Our results underpin two social roles (inferring others' behavior and interpreting social outcomes) subserved by the rTPJ-related network and highlight its interaction with other-self/other-concerning brain areas in reaching co-benefits among unselfish players.

Gaze and Arrows: Does the Gaze-Following Patch in the Posterior Temporal Cortex Differentiate Social and Symbolic Spatial Cues?

The gaze-following patch (GFP) is located in the posterior temporal cortex and has been described as a cortical module dedicated to processing other people's gaze-direction in a domain-specific manner. Thus, it appears to be the neural correlate of Baron-Cohen's eye direction detector (EDD) which is one of the core modules in his mindreading system-a neurocognitive model for the theory of mind concept. Inspired by Jerry Fodor's ideas on the modularity of the mind, Baron-Cohen proposed that, among other things, the individual modules are domain specific. In the case of the EDD, this means that it exclusively processes eye-like stimuli to extract gaze-direction and that other stimuli, which may carry directional information as well, are processed elsewhere. If the GFP is indeed EDD's neural correlate, it must meet this expectation. To test this, we compared the GFP's BOLD activity during gaze-direction following with the activity during arrow-direction following in the present human fMRI study. Contrary to the expectation based on the assumption of domain specificity, we did not find a differentiation between gaze- and arrow-direction following. In fact, we were not able to reproduce the GFP as presented in the previous studies. A possible explanation is that in the present study-unlike the previous work-the gaze stimuli did not contain an obvious change of direction that represented a visual motion. Hence, the critical stimulus component responsible for the identification of the GFP in the previous experiments might have been visual motion.

A Trained Humanoid Robot can Perform Human-Like Crossmodal Social Attention and Conflict Resolution

To enhance human-robot social interaction, it is essential for robots to process multiple social cues in a complex real-world environment. However, incongruency of input information across modalities is inevitable and could be challenging for robots to process. To tackle this challenge, our study adopted the neurorobotic paradigm of crossmodal conflict resolution to make a robot express human-like social attention. A behavioural experiment was conducted on 37 participants for the human study. We designed a round-table meeting scenario with three animated avatars to improve ecological validity. Each avatar wore a medical mask to obscure the facial cues of the nose, mouth, and jaw. The central avatar shifted its eye gaze while the peripheral avatars generated sound. Gaze direction and sound locations were either spatially congruent or incongruent. We observed that the central avatar's dynamic gaze could trigger crossmodal social attention responses. In particular, human performance was better under the congruent audio-visual condition than the incongruent condition. Our saliency prediction model was trained to detect social cues, predict audio-visual saliency, and attend selectively for the robot study. After mounting the trained model on the iCub, the robot was exposed to laboratory conditions similar to the human experiment. While the human performance was overall superior, our trained model demonstrated that it could replicate attention responses similar to humans.

The role of temporal cortex in the control of attention

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

The roles of gaze and head orientation in face categorization during rapid serial visual presentation

Little is known about how perceived gaze direction and head orientation may influence human categorization of visual stimuli as faces. To address this question, a sequence of unsegmented natural images, each containing a random face or a non-face object, was presented in rapid succession (stimulus duration: 91.7 ms per image) during which human observers were instructed to respond immediately to every face presentation. Faces differed in gaze and head orientation in 7 combinations - full-front views with perceived gaze (1) directed to the observer, (2) averted to the left, or (3) averted to the right, left ¾ side views with (4) direct gaze or (5) averted gaze, and right ¾ side views with (6) direct gaze or (7) averted gaze - were presented randomly throughout the sequence. We found highly accurate and rapid behavioural responses to all kinds of faces. Crucially, both perceived gaze direction and head orientation had comparable, non-interactive effects on response times, where direct gaze was responded faster than averted gaze by 48 ms and full-front view faster than ¾ side view also by 48 ms on average. Presentations of full-front faces with direct gaze led to an additive speed advantage of 96 ms to ¾ faces with averted gaze. The results reveal that the effects of perceived gaze direction and head orientation on the speed of face categorization probably depend on the degree of social relevance of the face to the viewer.

Frontal, Parietal, and Temporal Brain Areas Are Differentially Activated When Disambiguating Potential Objects of Joint Attention

Humans establish joint attention with others by following the other's gaze. Previous work has suggested that a cortical patch (gaze-following patch, GFP) close to the posterior superior temporal sulcus (pSTS) may serve as a link between the extraction of the other's gaze direction and the resulting shifts of attention, mediated by human lateral intraparietal area (hLIP). However, it is not clear how the brain copes with situations in which information on gaze direction alone is insufficient to identify the target object because more than one may lie along the gaze vector. In this fMRI study, we tested human subjects on a paradigm that allowed the identification of a target object based on the integration of the other's gaze direction and information provided by an auditory cue on the relevant object category. Whereas the GFP activity turned out to be fully determined by the use of gaze direction, activity in hLIP reflected the total information needed to pinpoint the target. Moreover, in an exploratory analysis, we found that a region in the inferior frontal junction (IFJ) was sensitive to the total information on the target. An examination of the BOLD time courses in the three identified areas suggests functionally complementary roles. Although the GFP seems to primarily process directional information stemming from the other's gaze, the IFJ may help to analyze the scene when gaze direction and auditory information are not sufficient to pinpoint the target. Finally, hLIP integrates both streams of information to shift attention to distinct spatial locations.

Decoding of the other's focus of attention by a temporal cortex module

Faces attract the observer's attention toward objects and locations of interest for the other, thereby allowing the two agents to establish joint attention. Previous work has delineated a network of cortical "patches" in the macaque cortex, processing faces, eventually also extracting information on the other's gaze direction. Yet, the neural mechanism that links information on gaze direction, guiding the observer's attention to the relevant object, has remained elusive. Here we present electrophysiological evidence for the existence of a distinct "gaze-following patch" (GFP) with neurons that establish this linkage in a highly flexible manner. The other's gaze and the object, singled out by the gaze, are linked only if this linkage is pertinent within the prevailing social context. The properties of these neurons establish the GFP as a key switch in controlling social interactions based on the other's gaze.

Controlling attention to gaze and arrows in attention deficit hyperactivity disorder

The aim of this research was to assess implicit processing of social and non-social distracting cues in children with ADHD. Young people with ADHD and matched controls were asked to classify target words (LEFT/RIGHT) which were accompanied by a distracter eye-gaze or arrow. Typically developing participants showed evidence of interference effects from both eye-gaze and arrow distracters. In contrast, the ADHD group showed evidence of interference effects from arrow but failed to show interference from eye-gaze. This absence of interference effects from eye-gaze observed in the participants with ADHD may reflect an attentional impairment in attending to socially relevant information.

Following Eye Gaze Activates a Patch in the Posterior Temporal Cortex That Is not Part of the Human "Face Patch" System

Humans follow another person’s eye gaze to objects of interest to the other, thereby establishing joint attention, a first step toward developing a theory of the other’s mind. Previous functional MRI studies agree that a “gaze-following patch” (GFP) of cortex close to the posterior superior temporal sulcus (STS) is specifically implicated in eye gaze-following. The location of the GFP is in the vicinity of the posterior members of the core face-processing system that consists of distinct patches in ventral visual cortex, the STS, and frontal cortex, also involved in processing information on the eyes. To test whether the GFP might correspond to one of the posterior face patches, we compared the pattern of blood oxygenation level–dependent (BOLD) imaging contrasts reflecting the passive vision of static faces with the one evoked by shifts of attention guided by the eye gaze of others. The viewing of static faces revealed the face patch system. On the other hand, eye gaze-following activated a cortical patch (the GFP) with its activation maximum separated by more than 24 mm in the right and 19 mm in the left hemisphere from the nearest face patch, the STS face area (FA). This segregation supports a distinct function of the GFP, different from the elementary processing of facial information.

Heads First: Visual Aftereffects Reveal Hierarchical Integration of Cues to Social Attention

Determining where another person is attending is an important skill for social interaction that relies on various visual cues, including the turning direction of the head and body. This study reports a novel high-level visual aftereffect that addresses the important question of how these sources of information are combined in gauging social attention. We show that adapting to images of heads turned 25° to the right or left produces a perceptual bias in judging the turning direction of subsequently presented bodies. In contrast, little to no change in the judgment of head orientation occurs after adapting to extremely oriented bodies. The unidirectional nature of the aftereffect suggests that cues from the human body signaling social attention are combined in a hierarchical fashion and is consistent with evidence from single-cell recording studies in nonhuman primates showing that information about head orientation can override information about body posture when both are visible.

Information flow between interacting human brains: Identification, validation, and relationship to social expertise

Social interactions are fundamental for human behavior, but the quantification of their neural underpinnings remains challenging. Here, we used hyperscanning functional MRI (fMRI) to study information flow between brains of human dyads during real-time social interaction in a joint attention paradigm. In a hardware setup enabling immersive audiovisual interaction of subjects in linked fMRI scanners, we characterize cross-brain connectivity components that are unique to interacting individuals, identifying information flow between the sender's and receiver's temporoparietal junction. We replicate these findings in an independent sample and validate our methods by demonstrating that cross-brain connectivity relates to a key real-world measure of social behavior. Together, our findings support a central role of human-specific cortical areas in the brain dynamics of dyadic interactions and provide an approach for the noninvasive examination of the neural basis of healthy and disturbed human social behavior with minimal a priori assumptions.

Gorillas with white sclera: A naturally occurring variation in a morphological trait linked to social cognitive functions

Human eye morphology is considered unique among the primates in that humans possess larger width/height ratios (WHR), expose a greater amount of visible sclera (SSI; width of exposed eyeball/width of visible iris), and critically, have a white sclera due to a lack of pigmentation. White sclera in humans amplifies gaze direction, whereas the all-dark eyes of apes are hypothesized to conceal gaze from others. This study examines WHR and SSI in humans (N = 13) and gorillas (N = 85) engaged in direct and averted gazes and introduces a qualitative assessment of sclera color to evaluate variations in sclera pigmentation. The results confirm previous findings that humans possess a larger WHR than gorillas but indicate that humans and gorillas display similar amounts of visible sclera. Additionally, 72% (N = 124) of gorilla eyes in this sample deviated from the assumed all-dark eye condition. This questions whether gaze camouflage is the primary function of darkened sclera in non-human primates or whether other functional roles can be ascribed to the sclera, light or dark. We argue that white sclera evolved to amplify direct gazes in humans, which would have played a significant role in the development of ostensive communication, which is communication that both shows something and shows the intention to show something. We conclude that the horizontal elongation of the human eye, rather than sclera color, more reliably distinguishes human from great ape eyes, represented here by gorillas.

Neural signals evoked by stimuli of increasing social scene complexity are detectable at the single-trial level and right lateralized

Classification of neural signals at the single-trial level and the study of their relevance in affective and cognitive neuroscience are still in their infancy. Here we investigated the neurophysiological correlates of conditions of increasing social scene complexity using 3D human models as targets of attention, which may also be important in autism research. Challenging single-trial statistical classification of EEG neural signals was attempted for detection of oddball stimuli with increasing social scene complexity. Stimuli had an oddball structure and were as follows: 1) flashed schematic eyes, 2) simple 3D faces flashed between averted and non-averted gaze (only eye position changing), 3) simple 3D faces flashed between averted and non-averted gaze (head and eye position changing), 4) animated avatar alternated its gaze direction to the left and to the right (head and eye position), 5) environment with 4 animated avatars all of which change gaze and one of which is the target of attention. We found a late (> 300 ms) neurophysiological oddball correlate for all conditions irrespective of their complexity as assessed by repeated measures ANOVA. We attempted single-trial detection of this signal with automatic classifiers and obtained a significant balanced accuracy classification of around 79%, which is noteworthy given the amount of scene complexity. Lateralization analysis showed a specific right lateralization only for more complex realistic social scenes. In sum, complex ecological animations with social content elicit neurophysiological events which can be characterized even at the single-trial level. These signals are right lateralized. These finding paves the way for neuroscientific studies in affective neuroscience based on complex social scenes, and given the detectability at the single trial level this suggests the feasibility of brain computer interfaces that can be applied to social cognition disorders such as autism.

The Mona Lisa effect: neural correlates of centered and off-centered gaze

The Mona Lisa effect describes the phenomenon when the eyes of a portrait appear to look at the observer regardless of the observer's position. Recently, the metaphor of a cone of gaze has been proposed to describe the range of gaze directions within which a person feels looked at. The width of the gaze cone is about five degrees of visual angle to either side of a given gaze direction. We used functional magnetic resonance imaging to investigate how the brain regions involved in gaze direction discrimination would differ between centered and decentered presentation positions of a portrait exhibiting eye contact. Subjects observed a given portrait's eyes. By presenting portraits with varying gaze directions-eye contact (0°), gaze at the edge of the gaze cone (5°), and clearly averted gaze (10°), we revealed that brain response to gaze at the edge of the gaze cone was similar to that produced by eye contact and different from that produced by averted gaze. Right fusiform gyrus and right superior temporal sulcus showed stronger activation when the gaze was averted as compared to eye contact. Gaze sensitive areas, however, were not affected by the portrait's presentation location. In sum, although the brain clearly distinguishes averted from centered gaze, a substantial change of vantage point does not alter neural activity, thus providing a possible explanation why the feeling of eye contact is upheld even in decentered stimulus positions.

Impaired reflexive orienting to social cues in attention deficit hyperactivity disorder

The present study investigated whether another person's social attention, specifically the direction of their eye gaze, and non-social directional cues triggered reflexive orienting in individuals with Attention Deficit Hyperactivity Disorder (ADHD) and age-matched controls. A choice reaction time and a detection tasks were used in which eye gaze, arrow and peripheral cues correctly (congruent) or incorrectly (incongruent) signalled target location. Independently of the type of the task, differences between groups were specific to the cue condition. Typically developing individuals shifted attention to the location cued by both social and non-social cues, whereas ADHD group showed evidence of reflexive orienting only to locations previously cued by non-social stimuli (arrow and peripheral cues) but failed to show such orienting effect in response to social eye gaze cues. The absence of reflexive orienting effect for eye gaze cues observed in the participants with ADHD may reflect an attentional impairment in responding to socially relevant information.

Disparate substrates for head gaze following and face perception in the monkey superior temporal sulcus

Primates use gaze cues to follow peer gaze to an object of joint attention. Gaze following of monkeys is largely determined by head or face orientation. We used fMRI in rhesus monkeys to identify brain regions underlying head gaze following and to assess their relationship to the 'face patch' system, the latter being the likely source of information on face orientation. We trained monkeys to locate targets by either following head gaze or using a learned association of face identity with the same targets. Head gaze following activated a distinct region in the posterior STS, close to-albeit not overlapping with-the medial face patch delineated by passive viewing of faces. This 'gaze following patch' may be the substrate of the geometrical calculations needed to translate information on head orientation from the face patches into precise shifts of attention, taking the spatial relationship of the two interacting agents into account.

Social learning in humans and other animals

Decisions made by individuals can be influenced by what others think and do. Social learning includes a wide array of behaviors such as imitation, observational learning of novel foraging techniques, peer or parental influences on individual preferences, as well as outright teaching. These processes are believed to underlie an important part of cultural variation among human populations and may also explain intraspecific variation in behavior between geographically distinct populations of animals. Recent neurobiological studies have begun to uncover the neural basis of social learning. Here we review experimental evidence from the past few decades showing that social learning is a widespread set of skills present in multiple animal species. In mammals, the temporoparietal junction, the dorsomedial, and dorsolateral prefrontal cortex, as well as the anterior cingulate gyrus, appear to play critical roles in social learning. Birds, fish, and insects also learn from others, but the underlying neural mechanisms remain poorly understood. We discuss the evolutionary implications of these findings and highlight the importance of emerging animal models that permit precise modification of neural circuit function for elucidating the neural basis of social learning.

Eye gaze metrics reflect a shared motor representation for action observation and movement imagery

Action observation (AO) and movement imagery (MI) have been reported to share similar neural networks. This study investigated the congruency between AO and MI using the eye gaze metrics, dwell time and fixation number. A simple reach-grasp-place arm movement was observed and, in a second condition, imagined where the movement was presented from the first person perspective (1PP) and the third person perspective (3PP). Dwell time and number of fixations were calculated for whole scene and regions of interest (ROIs). For whole scene, no significant differences were found in the number of fixations for condition (AO, MI) or perspective. Dwell time, however, was significantly longer in AO than MI. For ROIs, the number of fixations was significantly greater in 1PP than 3PP. The data provide support for congruence between motor simulation states but also indicate some functional differences.

A head view-invariant representation of gaze direction in anterior superior temporal sulcus

Humans show a remarkable ability to discriminate others' gaze direction, even though a given direction can be conveyed by many physically dissimilar configurations of different eye positions and head views. For example, eye contact can be signaled by a rightward glance in a left-turned head or by direct gaze in a front-facing head. Such acute gaze discrimination implies considerable perceptual invariance. Previous human research found that superior temporal sulcus (STS) responds preferentially to gaze shifts [1], but the underlying representation that supports such general responsiveness remains poorly understood. Using multivariate pattern analysis (MVPA) of human functional magnetic resonance imaging (fMRI) data, we tested whether STS contains a higher-order, head view-invariant code for gaze direction. The results revealed a finely graded gaze direction code in right anterior STS that was invariant to head view and physical image features. Further analyses revealed similar gaze effects in left anterior STS and precuneus. Our results suggest that anterior STS codes the direction of another's attention regardless of how this information is conveyed and demonstrate how high-level face areas carry out fine-grained, perceptually relevant discrimination through invariance to other face features.

Social communication in young children with traumatic brain injury: relations with corpus callosum morphometry

The purpose of the present investigation was to characterize the relations of specific social communication behaviors, including joint attention, gestures, and verbalization, with surface area of midsagittal corpus callosum (CC) subregions in children who sustained traumatic brain injury (TBI) before 7 years of age. Participants sustained mild (n=10) or moderate-severe (n=26) noninflicted TBI. The mean age at injury was 33.6 months; mean age at MRI was 44.4 months. The CC was divided into seven subregions. Relative to young children with mild TBI, those with moderate-severe TBI had smaller surface area of the isthmus. A semi-structured sequence of social interactions between the child and an examiner was videotaped and coded for specific social initiation and response behaviors. Social responses were similar across severity groups. Even though the complexity of their language was similar, children with moderate-severe TBI used more gestures than those with mild TBI to initiate social overtures; this may indicate a developmental lag or deficit as the use of gestural communication typically diminishes after age 2. After controlling for age at scan and for total brain volume, the correlation of social interaction response and initiation scores with the midsagittal surface area of the CC regions was examined. For the total group, responding to a social overture using joint attention was significantly and positively correlated with surface area of all regions, except the rostrum. Initiating joint attention was specifically and negatively correlated with surface area of the anterior midbody. Use of gestures to initiate a social interaction correlated significantly and positively with surface area of the anterior and posterior midbody. Social response and initiation behaviors were selectively related to regional callosal surface areas in young children with TBI. Specific brainbehavior relations indicate early regional specialization of anterior and posterior CC for social communication.