A naturalistic paradigm simulating gaze-based social interactions for the investigation of social agency

Neural mechanisms underlying the interactive exchange of facial emotional expressions

Facial emotional expressions are crucial in face-to-face social interactions, and recent findings have highlighted their interactive nature. However, the underlying neural mechanisms remain unclear. This electroencephalography study investigated whether the interactive exchange of facial expressions modulates socio-emotional processing. Participants (N = 41) displayed a facial emotional expression (angry, neutral, or happy) toward a virtual agent, and the agent then responded with a further emotional expression (angry or happy) or remained neutral (control condition). We assessed subjective experience (valence, arousal), facial EMG (Zygomaticus, Corrugator), and event-related potentials (EPN, LPP) elicited by the agent's response. Replicating previous findings, we found that an agent's happy facial expression was experienced as more pleasant and elicited increased Zygomaticus activity when participants had initiated the interaction with a happy compared to an angry expression. At the neural level, angry expressions resulted in a greater LPP than happy expressions, but only when participants directed an angry or happy, but not a neutral, expression at the agent. These findings suggest that sending an emotional expression increases salience and enhances the processing of received emotional expressions, indicating that an interactive setting alters brain responses to social stimuli.

Sense of agency at a temporally-delayed gaze-contingent display

The subjective feeling of being the author of one's actions and the subsequent consequences is referred to as a sense of agency. Such a feeling is crucial for usability in human-computer interactions, where eye movement has been adopted, yet this area has been scarcely investigated. We examined how the temporal action-feedback discrepancy affects the sense of agency concerning eye movement. Participants conducted a visual search for an array of nine Chinese characters within a temporally-delayed gaze-contingent display, blurring the peripheral view. The relative delay between each eye movement and the subsequent window movement varied from 0 to 4,000 ms. In the control condition, the window played a recorded gaze behavior. The mean authorship rating and the proportion of "self" responses in the categorical authorship report ("self," "delayed self," and "other") gradually decreased as the temporal discrepancy increased, with "other" being rarely reported, except in the control condition. These results generally mirror those of prior studies on hand actions, suggesting that sense of agency extends beyond the effector body parts to other modalities, and two different types of sense of agency that have different temporal characteristics are simultaneously operating. The mode of fixation duration shifted as the delay increased under 200-ms delays and was divided into two modes at 200-500 ms delays. The frequency of 0-1.5° saccades exhibited an increasing trend as the delay increased. These results demonstrate the influence of perceived action-effect discrepancy on action refinement and task strategy.

Sense of agency at a gaze-contingent display with jittery temporal delay

Introduction

Inconsistent jittery temporal delays between action and subsequent feedback, prevalent in network-based human-computer interaction (HCI), have been insufficiently explored, particularly regarding their impact on the sense of agency (SoA). This study investigates the SoA in the context of eye-gaze HCI under jittery delay conditions.

Methods

Participants performed a visual search for Chinese characters using a biresolutional gaze-contingent display, which displayed a high-resolution image in the central vision and a low-resolution in the periphery. We manipulated the delay between eye movements and display updates using a truncated normal distribution (μ to μ + 2 σ) with μ ranging from 0 to 400 ms and σ fixed at 50 ms. Playback of recorded gaze data provided a non-controllable condition.

Results

The study revealed that both reported authorship and controllability scores, as well as the fixation count per second, decreased as μ increased, aligning with trends observed under constant delay conditions. The subjective authorship weakened significantly at a μ of 94 ms. Notably, the comparison between jittery and constant delays indicated the minimum value (μ) of the distribution as a critical parameter influencing both authorship perception and visual search time efficiency.

Discussion

This finding underscores the importance of the shortest delay in modulating SoA. Further examining the relative distribution for fixation duration and saccade amplitude suggests an adaptation in action planning and attention distribution in response to delay. By providing a systematic examination of the statistical attributes of jittery delays that most significantly affect SoA, this research offers valuable implications for the design of efficient, delay-tolerant eye-gaze HCI, expanding our understanding of SoA in technologically mediated interactions. Moreover, our findings highlight the significance of considering both constant and variable delay impacts in HCI usability design, marking a novel contribution to the field.

Impaired recognition of interactive intentions in adults with autism spectrum disorder not attributable to differences in visual attention or coordination via eye contact and joint attention

Altered nonverbal communication patterns especially with regard to gaze interactions are commonly reported for persons with autism spectrum disorder (ASD). In this study we investigate and differentiate for the first time the interplay of attention allocation, the establishment of shared focus (eye contact and joint attention) and the recognition of intentions in gaze interactions in adults with ASD compared to control persons. Participants interacted via gaze with a virtual character (VC), who they believed was controlled by another person. Participants were instructed to ascertain whether their partner was trying to interact with them. In fact, the VC was fully algorithm-controlled and showed either interactive or non-interactive gaze behavior. Participants with ASD were specifically impaired in ascertaining whether their partner was trying to interact with them or not as compared to participants without ASD whereas neither the allocation of attention nor the ability to establish a shared focus were affected. Thus, perception and production of gaze cues seem preserved while the evaluation of gaze cues appeared to be impaired. An additional exploratory analysis suggests that especially the interpretation of contingencies between the interactants' actions are altered in ASD and should be investigated more closely.

Eye did this! Sense of agency in eye movements

This study investigates the sense of agency (SoA) for saccades with implicit and explicit agency measures. In two eye tracking experiments, participants moved their eyes towards on-screen stimuli that subsequently changed color. Participants then either reproduced the temporal interval between saccade and color-change (Experiment 1) or reported the time points of these events with an auditory Libet clock (Experiment 2) to measure temporal binding effects as implicit indices of SoA. Participants were either made to believe to exert control over the color change or not (agency manipulation). Explicit ratings indicated that the manipulation of causal beliefs and hence agency was successful. However, temporal binding was only evident for caused effects, and only when a sufficiently sensitive procedure was used (auditory Libet clock). This suggests a feebler connection between temporal binding and SoA than previously proposed. The results also provide evidence for a relatively fast acquisition of sense of agency for previously never experienced types of action-effect associations. This indicates that the underlying processes of action control may be rooted in more intricate and adaptable cognitive models than previously thought. Oculomotor SoA as addressed in the present study presumably represents an important cognitive foundation of gaze-based social interaction (social sense of agency) or gaze-based human-machine interaction scenarios. PUBLIC SIGNIFICANCE STATEMENT: In this study, sense of agency for eye movements in the non-social domain is investigated in detail, using both explicit and implicit measures. Therefore, it offers novel and specific insights into comprehending sense of agency concerning effects induced by eye movements, as well as broader insights into agency pertaining to entirely newly acquired types of action-effect associations. Oculomotor sense of agency presumably represents an important cognitive foundation of gaze-based social interaction (social agency) or gaze-based human-machine interaction scenarios. Due to peculiarities of the oculomotor domain such as the varying degree of volitional control, eye movements could provide new information regarding more general theories of sense of agency in future research.

Motivation from Agency and Reward in Typical Development and Autism: Narrative Review of Behavioral and Neural Evidence

Our ability to perform voluntary actions and make choices is shaped by the motivation from having control over the resulting effects (agency) and positive outcomes (reward). We offer an overview of distinct and common behavioral and neural signatures of agency and reward. We discuss their typical and atypical developmental trajectories, focusing on autism spectrum disorder (ASD), which is characterized by neurodiverse processes underlying action selection. We propose that reduced sensitivity to agency and reward in ASD may be related to atypical multisensory processes and motor planning, with potential for understanding restricted and repetitive behaviors. We emphasize the limitations of the existing literature, and prospects for future research. Understanding the neurocognitive processes that shape the way people with ASD select actions and perceive outcomes is essential to support not only learning, but also volition and self-determination.

Language Dysfunction in Schizophrenia: Assessing Neural Tracking to Characterize the Underlying Disorder(s)?

Deficits in language production and comprehension are characteristic of schizophrenia. To date, it remains unclear whether these deficits arise from dysfunctional linguistic knowledge, or dysfunctional predictions derived from the linguistic context. Alternatively, the deficits could be a result of dysfunctional neural tracking of auditory information resulting in decreased auditory information fidelity and even distorted information. Here, we discuss possible ways for clinical neuroscientists to employ neural tracking methodology to independently characterize deficiencies on the auditory-sensory and abstract linguistic levels. This might lead to a mechanistic understanding of the deficits underlying language related disorder(s) in schizophrenia. We propose to combine naturalistic stimulation, measures of speech-brain synchronization, and computational modeling of abstract linguistic knowledge and predictions. These independent but likely interacting assessments may be exploited for an objective and differential diagnosis of schizophrenia, as well as a better understanding of the disorder on the functional level-illustrating the potential of neural tracking methodology as translational tool in a range of psychotic populations.

Implying social interaction and its influence on gaze behavior to the eyes

Researchers have increasingly focused on how the potential for social interaction modulates basic processes of visual attention and gaze behavior. In this study, we investigated why people may experience social interaction and what factors contributed to their subjective experience. We furthermore investigated whether implying social interaction modulated gaze behavior to people’s faces, specifically the eyes. To imply the potential for interaction, participants received either one of two instructions: 1) they would be presented with a person via a ‘live’ video-feed, or 2) they would be presented with a pre-recorded video clip of a person. Prior to the presentation, a confederate walked into a separate room to suggest to participants that (s)he was being positioned behind a webcam. In fact, all participants were presented with a pre-recorded clip. During the presentation, we measured participants’ gaze behavior with an eye tracker, and after the presentation, participants were asked whether they believed that the confederate was ‘live’ or not, and, why they thought so. Participants varied greatly in their judgements about whether the confederate was ‘live’ or not. Analyses of gaze behavior revealed that a large subset of participants who received the live-instruction gazed less at the eyes of confederates compared with participants who received the pre-recorded-instruction. However, for both the live-instruction group and the pre-recorded instruction group, another subset of participants gazed predominantly at the eyes. The current findings may contribute to the development of experimental designs aimed to capture the interactive aspects of social cognition and visual attention.