A model of face selection in viewing video stories

Emergence of human-like attention and distinct head clusters in self-supervised vision transformers: A comparative eye-tracking study

Visual attention models aim to predict human gaze behavior, yet traditional saliency models and deep gaze prediction networks face limitations. Saliency models rely on handcrafted low-level visual features, often failing to capture human gaze dynamics, while deep learning-based gaze prediction models lack biological plausibility. Vision Transformers (ViTs), which use self-attention mechanisms, offer an alternative, but when trained with conventional supervised learning, their attention patterns tend to be dispersed and unfocused. This study demonstrates that ViTs trained with self-supervised DINO (self-Distillation with NO labels) develop structured attention that closely aligns with human gaze behavior when viewing videos. Our analysis reveals that self-attention heads in later layers of DINO-trained ViTs autonomously differentiate into three distinct clusters: (1) G1 heads (20%), which focus on key points within figures (e.g., the eyes of the main character) and resemble human gaze; (2) G2 heads (60%), which distribute attention over entire figures with sharp contours (e.g., the bodies of all characters); and (3) G3 heads (20%), which primarily attend to the background. These findings provide insights into how human overt attention and figure-ground segregation emerge in visual perception. Our work suggests that self-supervised learning enables ViTs to develop attention mechanisms that are more aligned with biological vision than traditional supervised training.

Fear lies in the eyes of the beholder-Robust evidence for reduced gaze dispersion upon avoidable threat

A rapid detection and processing of relevant information in our environment is crucial for survival. The human eyes are drawn to social or threatening stimuli as they may carry essential information on how to behave appropriately in a given context. Recent studies further showed a centralization of gaze that reminded of freezing behaviors in rodents. Probably constituting a component of an adaptive defense mode, centralized eye movements predicted the speed of motor actions. Here we conducted two experiments to examine if and how these presumably survival-relevant gaze patterns interact. Subjects viewed images including social, that is, faces (Experiment 1, N = 50) or threatening stimuli, that is, snakes or spiders (Experiment 2, N = 50) while awaiting an inevitable (shock), no (safety), or an avoidable shock (flight) they could escape from by a fast button press. The social and threatening cues within the scenes differed in their distance from the image center and we acquired eye-tracking and autonomic physiological data. Although we observed an initial orienting toward social and threatening stimulus aspects, this exploration pattern vanished towards the end of flight trials when a pronounced centralization of gaze emerged. Replicating previous findings, the amount of this center bias predicted the speed of motor reactions, and we observed a concurrent activation of the sympathetic and parasympathetic nervous system. Taken together, the exploration of potentially relevant cues seems to be part of a reflexive-orienting response regardless of contextual valence. However, centralization of gaze may be a threat-specific action-preparatory response that occurs across a wide range of stimulus contexts.

Orienting towards social features in naturalistic scenes is reflexive

Saliency-based models of visual attention postulate that, when a scene is freely viewed, attention is predominantly allocated to those elements that stand out in terms of their physical properties. However, eye-tracking studies have shown that saliency models fail to predict gaze behavior accurately when social information is included in an image. Notably, gaze pattern analyses revealed that depictions of human beings are heavily prioritized independent of their low-level physical saliency. What remains unknown, however, is whether the prioritization of such social features is a reflexive or a voluntary process. To investigate the early stages of social attention in more detail, participants viewed photographs of naturalistic scenes with and without social features (i.e., human heads or bodies) for 200 ms while their eye movements were being recorded. We observed significantly more first eye movements to regions containing social features than would be expected from a chance level distribution of saccades. Additionally, a generalized linear mixed model analysis revealed that the social content of a region better predicted first saccade direction than its saliency suggesting that social features partially override the impact of low-level physical saliency on gaze patterns. Given the brief image presentation time that precluded visual exploration, our results provide compelling evidence for a reflexive component in social attention. Moreover, the present study emphasizes the importance of considering social influences for a more coherent understanding of human attentional selection.

Preferential Processing of Social Features and Their Interplay with Physical Saliency in Complex Naturalistic Scenes

According to so-called saliency-based attention models, attention during free viewing of visual scenes is particularly allocated to physically salient image regions. In the present study, we assumed that social features in complex naturalistic scenes would be processed preferentially irrespective of their physical saliency. Therefore, we expected worse prediction of gazing behavior by saliency-based attention models when social information is present in the visual field. To test this hypothesis, participants freely viewed color photographs of complex naturalistic social (e.g., including heads, bodies) and non-social (e.g., including landscapes, objects) scenes while their eye movements were recorded. In agreement with our hypothesis, we found that social features (especially heads) were heavily prioritized during visual exploration. Correspondingly, the presence of social information weakened the influence of low-level saliency on gazing behavior. Importantly, this pattern was most pronounced for the earliest fixations indicating automatic attentional processes. These findings were further corroborated by a linear mixed model approach showing that social features (especially heads) add substantially to the prediction of fixations beyond physical saliency. Taken together, the current study indicates gazing behavior for naturalistic scenes to be better predicted by the interplay of social and physically salient features than by low-level saliency alone. These findings strongly challenge the generalizability of saliency-based attention models and demonstrate the importance of considering social influences when investigating the driving factors of human visual attention.