Differences of monkey and human overt attention under natural conditions

Using Natural Scenes to Enhance our Understanding of the Cerebral Cortex's Role in Visual Search

Using natural scenes is an approach to studying the visual and eye movement systems approximating how these systems function in everyday life. This review examines the results from behavioral and neurophysiological studies using natural scene viewing in humans and monkeys. The use of natural scenes for the study of cerebral cortical activity is relatively new and presents challenges for data analysis. Methods and results from the use of natural scenes for the study of the visual and eye movement cortex are presented, with emphasis on new insights that this method provides enhancing what is known about these cortical regions from the use of conventional methods.

Dimension of visual information interacts with working memory in monkeys and humans

Humans demonstrate behavioural advantages (biases) towards particular dimensions (colour or shape of visual objects), but such biases are significantly altered in neuropsychological disorders. Recent studies have shown that lesions in the prefrontal cortex do not abolish dimensional biases, and therefore suggest that such biases might not depend on top-down prefrontal-mediated attention and instead emerge as bottom-up processing advantages. We hypothesised that if dimensional biases merely emerge from an enhancement of object features, the presence of visual objects would be necessary for the manifestation of dimensional biases. In a specifically-designed working memory task, in which macaque monkeys and humans performed matching based on the object memory rather than the actual object, we found significant dimensional biases in both species, which appeared as a shorter response time and higher accuracy in the preferred dimension (colour and shape dimension in humans and monkeys, respectively). Moreover, the mnemonic demands of the task influenced the magnitude of dimensional bias. Our findings in two primate species indicate that the dichotomy of top-down and bottom-up processing does not fully explain the emergence of dimensional biases. Instead, dimensional biases may emerge when processed information regarding visual object features interact with mnemonic and executive functions to guide goal-directed behaviour.

Unsupervised foveal vision neural architecture with top-down attention

Deep learning architectures are an extremely powerful tool for recognizing and classifying images. However, they require supervised learning and normally work on vectors of the size of image pixels and produce the best results when trained on millions of object images. To help mitigate these issues, we propose an end-to-end architecture that fuses bottom-up saliency and top-down attention with an object recognition module to focus on relevant data and learn important features that can later be fine-tuned for a specific task, employing only unsupervised learning. In addition, by utilizing a virtual fovea that focuses on relevant portions of the data, the training speed can be greatly improved. We test the performance of the proposed Gamma saliency technique on the Toronto and CAT 2000 databases, and the foveated vision in the large Street View House Numbers (SVHN) database. The results with foveated vision show that Gamma saliency performs at the same level as the best alternative algorithms while being computationally faster. The results in SVHN show that our unsupervised cognitive architecture is comparable to fully supervised methods and that saliency also improves CNN performance if desired. Finally, we develop and test a top-down attention mechanism based on the Gamma saliency applied to the top layer of CNNs to facilitate scene understanding in multi-object cluttered images. We show that the extra information from top-down saliency is capable of speeding up the extraction of digits in the cluttered multidigit MNIST data set, corroborating the important role of top down attention.

The Role of Primate Prefrontal Cortex in Bias and Shift Between Visual Dimensions

Imaging and neural activity recording studies have shown activation in the primate prefrontal cortex when shifting attention between visual dimensions is necessary to achieve goals. A fundamental unanswered question is whether representations of these dimensions emerge from top-down attentional processes mediated by prefrontal regions or from bottom-up processes within visual cortical regions. We hypothesized a causative link between prefrontal cortical regions and dimension-based behavior. In large cohorts of humans and macaque monkeys, performing the same attention shifting task, we found that both species successfully shifted between visual dimensions, but both species also showed a significant behavioral advantage/bias to a particular dimension; however, these biases were in opposite directions in humans (bias to color) versus monkeys (bias to shape). Monkeys' bias remained after selective bilateral lesions within the anterior cingulate cortex (ACC), frontopolar cortex, dorsolateral prefrontal cortex (DLPFC), orbitofrontal cortex (OFC), or superior, lateral prefrontal cortex. However, lesions within certain regions (ACC, DLPFC, or OFC) impaired monkeys' ability to shift between these dimensions. We conclude that goal-directed processing of a particular dimension for the executive control of behavior depends on the integrity of prefrontal cortex; however, representation of competing dimensions and bias toward them does not depend on top-down prefrontal-mediated processes.

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

Proto-Object Based Saliency Model With Texture Detection Channel

The amount of visual information projected from the retina to the brain exceeds the information processing capacity of the latter. Attention, therefore, functions as a filter to highlight important information at multiple stages of the visual pathway that requires further and more detailed analysis. Among other functions, this determines where to fixate since only the fovea allows for high resolution imaging. Visual saliency modeling, i.e. understanding how the brain selects important information to analyze further and to determine where to fixate next, is an important research topic in computational neuroscience and computer vision. Most existing bottom-up saliency models use low-level features such as intensity and color, while some models employ high-level features, like faces. However, little consideration has been given to mid-level features, such as texture, for visual saliency models. In this paper, we extend a biologically plausible proto-object based saliency model by adding simple texture channels which employ nonlinear operations that mimic the processing performed by primate visual cortex. The extended model shows statistically significant improved performance in predicting human fixations compared to the previous model. Comparing the performance of our model with others on publicly available benchmarking datasets, we find that our biologically plausible model matches the performance of other models, even though those were designed entirely for maximal performance with little regard to biological realism.

Human and monkey infant attention to dynamic social and nonsocial stimuli

The present study explored behavioral norms for infant social attention in typically developing human and nonhuman primate infants. We examined the normative development of attention to dynamic social and nonsocial stimuli longitudinally in macaques (Macaca mulatta) at 1, 3, and 5 months of age (N = 75) and humans at 2, 4, 6, 8, and 13 months of age (N = 69) using eye tracking. All infants viewed concurrently played silent videos-one social video and one nonsocial video. Both macaque and human infants were faster to look to the social than the nonsocial stimulus, and both species grew faster to orient to the social stimulus with age. Further, macaque infants' social attention increased linearly from 1 to 5 months. In contrast, human infants displayed a nonlinear pattern of social interest, with initially greater attention to the social stimulus, followed by a period of greater interest in the nonsocial stimulus, and then a rise in social interest from 6 to 13 months. Overall, human infants looked longer than macaque infants, suggesting humans have more sustained attention in the first year of life. These findings highlight potential species similarities and differences, and reflect a first step in establishing baseline patterns of early social attention development.

Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments

Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.

Gaze Control During Simulator Driving in Adolescents With and Without Attention Deficit Hyperactivity Disorder

Importance: Attention deficit hyperactivity disorder (ADHD) is associated with driving deficits. Visual standards for driving define minimum qualifications for safe driving, including acuity and field of vision, but they do not consider the ability to explore the environment efficiently by shifting the gaze, which is a critical element of safe driving.

Objective: To examine visual exploration during simulated driving in adolescents with and without ADHD.

Design: Adolescents with and without ADHD drove a driving simulator for approximately 10 min while their gaze was monitored. They then completed a battery of questionnaires.

Setting: University lab.

Participants: Participants with (n = 16) and without (n = 15) ADHD were included. Participants had a history of neurological disorders other than ADHD and normal or corrected-to-normal vision. Control participants reported not having a diagnosis of ADHD. Participants with ADHD had been previously diagnosed by a qualified professional.

Outcomes and Measures: We compared the following measures between ADHD and non-ADHD groups: dashboard dwell times, fixation variance, entropy, and fixation duration.

Results: Findings showed that participants with ADHD were more restricted in their patterns of exploration than control group participants. They spent considerably more time gazing at the dashboard and had longer periods of fixation with lower variability and randomness.

Conclusions and Relevance: The results support the hypothesis that adolescents with ADHD engage in less active exploration during simulated driving.

What This Article Adds: This study raises concerns regarding the driving competence of people with ADHD and opens up new directions for potential training programs that focus on exploratory gaze control.

Neural Mechanisms of High-Level Vision

The last three decades have seen major strides in our understanding of neural mechanisms of high-level vision, or visual cognition of the world around us. Vision has also served as a model system for the study of brain function. Several broad insights, as yet incomplete, have recently emerged. First, visual perception is best understood not as an end unto itself, but as a sensory process that subserves the animal's behavioral goal at hand. Visual perception is likely to be simply a side effect that reflects the readout of visual information processing that leads to behavior. Second, the brain is essentially a probabilistic computational system that produces behaviors by collectively evaluating, not necessarily consciously or always optimally, the available information about the outside world received from the senses, the behavioral goals, prior knowledge about the world, and possible risks and benefits of a given behavior. Vision plays a prominent role in the overall functioning of the brain providing the lion's share of information about the outside world. Third, the visual system does not function in isolation, but rather interacts actively and reciprocally with other brain systems, including other sensory faculties. Finally, various regions of the visual system process information not in a strict hierarchical manner, but as parts of various dynamic brain-wide networks, collectively referred to as the "connectome." Thus, a full understanding of vision will ultimately entail understanding, in granular, quantitative detail, various aspects of dynamic brain networks that use visual sensory information to produce behavior under real-world conditions. © 2017 American Physiological Society. Compr Physiol 8:903-953, 2018.

Primate social attention: Species differences and effects of individual experience in humans, great apes, and macaques

When viewing social scenes, humans and nonhuman primates focus on particular features, such as the models' eyes, mouth, and action targets. Previous studies reported that such viewing patterns vary significantly across individuals in humans, and also across closely-related primate species. However, the nature of these individual and species differences remains unclear, particularly among nonhuman primates. In large samples of human and nonhuman primates, we examined species differences and the effects of experience on patterns of gaze toward social movies. Experiment 1 examined the species differences across rhesus macaques, nonhuman apes (bonobos, chimpanzees, and orangutans), and humans while they viewed movies of various animals' species-typical behaviors. We found that each species had distinct viewing patterns of the models' faces, eyes, mouths, and action targets. Experiment 2 tested the effect of individuals' experience on chimpanzee and human viewing patterns. We presented movies depicting natural behaviors of chimpanzees to three groups of chimpanzees (individuals from a zoo, a sanctuary, and a research institute) differing in their early social and physical experiences. We also presented the same movies to human adults and children differing in their expertise with chimpanzees (experts vs. novices) or movie-viewing generally (adults vs. preschoolers). Individuals varied within each species in their patterns of gaze toward models' faces, eyes, mouths, and action targets depending on their unique individual experiences. We thus found that the viewing patterns for social stimuli are both individual- and species-specific in these closely-related primates. Such individual/species-specificities are likely related to both individual experience and species-typical temperament, suggesting that primate individuals acquire their unique attentional biases through both ontogeny and evolution. Such unique attentional biases may help them learn efficiently about their particular social environments.

Switch from ambient to focal processing mode explains the dynamics of free viewing eye movements

Previous studies have reported that humans employ ambient and focal modes of visual exploration while they freely view natural scenes. These two modes have been characterized based on eye movement parameters such as saccade amplitude and fixation duration, but not by any visual features of the viewed scenes. Here we propose a new characterization of eye movements during free viewing based on how eyes are moved from and to objects in a visual scene. We applied this characterization to data obtained from freely-viewing macaque monkeys. We show that the analysis based on this characterization gives a direct indication of a behavioral shift from ambient to focal processing mode along the course of free viewing exploration. We further propose a stochastic model of saccade sequence generation incorporating a switch between the two processing modes, which quantitatively reproduces the behavioral features observed in the data.

Predicting rhesus monkey eye movements during natural-image search

There are three prominent factors that can predict human visual-search behavior in natural scenes: the distinctiveness of a location (salience), similarity to the target (relevance), and features of the environment that predict where the object might be (context). We do not currently know how well these factors are able to predict macaque visual search, which matters because it is arguably the most popular model for asking how the brain controls eye movements. Here we trained monkeys to perform the pedestrian search task previously used for human subjects. Salience, relevance, and context models were all predictive of monkey eye fixations and jointly about as precise as for humans. We attempted to disrupt the influence of scene context on search by testing the monkeys with an inverted set of the same images. Surprisingly, the monkeys were able to locate the pedestrian at a rate similar to that for upright images. The best predictions of monkey fixations in searching inverted images were obtained by rotating the results of the model predictions for the original image. The fact that the same models can predict human and monkey search behavior suggests that the monkey can be used as a good model for understanding how the human brain enables natural-scene search.

An extensive dataset of eye movements during viewing of complex images

We present a dataset of free-viewing eye-movement recordings that contains more than 2.7 million fixation locations from 949 observers on more than 1000 images from different categories. This dataset aggregates and harmonizes data from 23 different studies conducted at the Institute of Cognitive Science at Osnabrück University and the University Medical Center in Hamburg-Eppendorf. Trained personnel recorded all studies under standard conditions with homogeneous equipment and parameter settings. All studies allowed for free eye-movements, and differed in the age range of participants (~7-80 years), stimulus sizes, stimulus modifications (phase scrambled, spatial filtering, mirrored), and stimuli categories (natural and urban scenes, web sites, fractal, pink-noise, and ambiguous artistic figures). The size and variability of viewing behavior within this dataset presents a strong opportunity for evaluating and comparing computational models of overt attention, and furthermore, for thoroughly quantifying strategies of viewing behavior. This also makes the dataset a good starting point for investigating whether viewing strategies change in patient groups.

Differential Contribution of Low- and High-level Image Content to Eye Movements in Monkeys and Humans

Oculomotor selection exerts a fundamental impact on our experience of the environment. To better understand the underlying principles, researchers typically rely on behavioral data from humans, and electrophysiological recordings in macaque monkeys. This approach rests on the assumption that the same selection processes are at play in both species. To test this assumption, we compared the viewing behavior of 106 humans and 11 macaques in an unconstrained free-viewing task. Our data-driven clustering analyses revealed distinct human and macaque clusters, indicating species-specific selection strategies. Yet, cross-species predictions were found to be above chance, indicating some level of shared behavior. Analyses relying on computational models of visual saliency indicate that such cross-species commonalities in free viewing are largely due to similar low-level selection mechanisms, with only a small contribution by shared higher level selection mechanisms and with consistent viewing behavior of monkeys being a subset of the consistent viewing behavior of humans.

Primate cognition: attention, episodic memory, prospective memory, self-control, and metacognition as examples of cognitive control in nonhuman primates

Primate Cognition is the study of cognitive processes, which represent internal mental processes involved in discriminations, decisions, and behaviors of humans and other primate species. Cognitive control involves executive and regulatory processes that allocate attention, manipulate and evaluate available information (and, when necessary, seek additional information), remember past experiences to plan future behaviors, and deal with distraction and impulsivity when they are threats to goal achievement. Areas of research that relate to cognitive control as it is assessed across species include executive attention, episodic memory, prospective memory, metacognition, and self-control. Executive attention refers to the ability to control what sensory stimuli one attends to and how one regulates responses to those stimuli, especially in cases of conflict. Episodic memory refers to memory for personally experienced, autobiographical events. Prospective memory refers to the formation and implementation of future-intended actions, such as remembering what needs to be done later. Metacognition consists of control and monitoring processes that allow individuals to assess what information they have and what information they still need, and then if necessary to seek information. Self-control is a regulatory process whereby individuals forego more immediate or easier to obtain rewards for more delayed or harder to obtain rewards that are objectively more valuable. The behavioral complexity shown by nonhuman primates when given tests to assess these capacities indicates psychological continuities with human cognitive control capacities. However, more research is needed to clarify the proper interpretation of these behaviors with regard to possible cognitive constructs that may underlie such behaviors. WIREs Cogn Sci 2016, 7:294-316. doi: 10.1002/wcs.1397 For further resources related to this article, please visit the WIREs website.

Modeling Visual Exploration in Rhesus Macaques with Bottom-Up Salience and Oculomotor Statistics

There is a growing interest in studying biological systems in natural settings, in which experimental stimuli are less artificial and behavior is less controlled. In primate vision research, free viewing of complex images has elucidated novel neural responses, and free viewing in humans has helped discover attentional and behavioral impairments in patients with neurological disorders. In order to fully interpret data collected from free viewing of complex scenes, it is critical to better understand what aspects of the stimuli guide viewing behavior. To this end, we have developed a novel viewing behavior model called a Biased Correlated Random Walk (BCRW) to describe free viewing behavior during the exploration of complex scenes in monkeys. The BCRW can predict fixation locations better than bottom-up salience. Additionally, we show that the BCRW can be used to test hypotheses regarding specific attentional mechanisms. For example, we used the BCRW to examine the source of the central bias in fixation locations. Our analyses suggest that the central bias may be caused by a natural tendency to reorient the eyes toward the center of the stimulus, rather than a photographer's bias to center salient items in a scene. Taken together these data suggest that the BCRW can be used to further our understanding of viewing behavior and attention, and could be useful in optimizing stimulus and task design.

Population coding in area V4 during rapid shape detections

While previous studies have suggested that neuronal correlations are common in visual cortex over a range of timescales, the effect of correlations on rapid visually based decisions has received little attention. We trained Macaca mulatta to saccade to a peripherally presented shape embedded in dynamic noise as soon as the shape appeared. While the monkeys performed the task, we recorded from neuronal populations (5-29 cells) using a microelectrode array implanted in area V4, a visual area thought to be involved in form perception. While modest correlations were present between cells during visual stimulation, their magnitude did not change significantly subsequent to the appearance of a shape. We quantified the reliability and temporal precision with which neuronal populations signaled the appearance of the shape and predicted the animals' choices using mutual information analyses. To study the impact of correlations, we shuffled the activity from each cell across observations while retaining stimulus-dependent modulations in firing rate. We found that removing correlations by shuffling across trials minimally affected the reliability or timing with which pairs, or larger groups of cells, signaled the presence of a shape. To assess the downstream impact of correlations, we also studied how shuffling affected the ability of V4 populations to predict behavioral choices. Surprisingly, shuffling created a modest increase in the accuracy of such predictions, suggesting that the reliability of downstream neurons is slightly compromised by activity correlations. Our findings are consistent with neuronal correlations having a minimal effect on the reliability and timing of rapid perceptual decisions.

Saliency and saccade encoding in the frontal eye field during natural scene search

The frontal eye field (FEF) plays a central role in saccade selection and execution. Using artificial stimuli, many studies have shown that the activity of neurons in the FEF is affected by both visually salient stimuli in a neuron's receptive field and upcoming saccades in a certain direction. However, the extent to which visual and motor information is represented in the FEF in the context of the cluttered natural scenes we encounter during everyday life has not been explored. Here, we model the activities of neurons in the FEF, recorded while monkeys were searching natural scenes, using both visual and saccade information. We compare the contribution of bottom-up visual saliency (based on low-level features such as brightness, orientation, and color) and saccade direction. We find that, while saliency is correlated with the activities of some neurons, this relationship is ultimately driven by activities related to movement. Although bottom-up visual saliency contributes to the choice of saccade targets, it does not appear that FEF neurons actively encode the kind of saliency posited by popular saliency map theories. Instead, our results emphasize the FEF's role in the stages of saccade planning directly related to movement generation.

Rapid shape detection signals in area V4

Vision in foveate animals is an active process that requires rapid and constant decision-making. For example, when a new object appears in the visual field, we can quickly decide to inspect it by directing our eyes to the object's location. We studied the contribution of primate area V4 to these types of rapid foveation decisions. Animals performed a reaction time task that required them to report when any shape appeared within a peripherally-located noisy stimulus by making a saccade to the stimulus location. We found that about half of the randomly sampled V4 neurons not only rapidly and precisely represented the appearance of this shape, but they were also predictive of the animal's saccades. A neuron's ability to predict the animal's saccades was not related to the specificity with which the cell represented a single type of shape but rather to its ability to signal whether any shape was present. This relationship between sensory sensitivity and behavioral predictiveness was not due to global effects such as alertness, as it was equally likely to be observed for cells with increases and decreases in firing rate. Careful analysis of the timescales of reliability in these neurons implies that they reflect both feedforward and feedback shape detecting processes. In approximately 7% of our recorded sample, individual neurons were able to predict both the delay and precision of the animal's shape detection performance. This suggests that a subset of V4 neurons may have been directly and causally contributing to task performance and that area V4 likely plays a critical role in guiding rapid, form-based foveation decisions.

Extrafoveal preview benefit during free-viewing visual search in the monkey

Previous studies have shown that subjects require less time to process a stimulus at the fovea after a saccade if they have viewed the same stimulus in the periphery immediately prior to the saccade. This extrafoveal preview benefit indicates that information about the visual form of an extrafoveally viewed stimulus can be transferred across a saccade. Here, we extend these findings by demonstrating and characterizing a similar extrafoveal preview benefit in monkeys during a free-viewing visual search task. We trained two monkeys to report the orientation of a target among distractors by releasing one of two bars with their hand; monkeys were free to move their eyes during the task. Both monkeys took less time to indicate the orientation of the target after foveating it, when the target lay closer to the fovea during the previous fixation. An extrafoveal preview benefit emerged even if there was more than one intervening saccade between the preview and the target fixation, indicating that information about target identity could be transferred across more than one saccade and could be obtained even if the search target was not the goal of the next saccade. An extrafoveal preview benefit was also found for distractor stimuli. These results aid future physiological investigations of the extrafoveal preview benefit.

A MOTION ATTENTION MODEL BASED ON RARITY WEIGHTING AND MOTION CUES IN DYNAMIC SCENES

Nowadays, motion attention model is a controversial topic in the biological computer vision area. The computational attention model can be decomposed into a set of features via predefined channels. Here we designed a bio-inspired vision attention model, and added the rarity measurement onto it. The priority of rarity is emphasized under the assumption of weighting effect upon the features logic fusion. At this stage, a final saliency map at each frame is adjusted by the spatiotemporal and rarity values. By doing this, the process of mimicking human vision attention becomes more realistic and logical to the real circumstance. The experiments are conducted on the benchmark dataset of static images and video sequences. We simulated the attention shift based on several dataset. Most importantly, our dynamic scenes are mostly selected from the objects moving on the highway and dynamic scenes. The former one can be developed on the detection of car collision and will be a useful tool for further application in robotics. We also conduct experiment on the other video clips to prove the rationality of rarity factor and feature cues fusion methods. Finally, the evaluation results indicate our visual attention model outperforms several state-of-the-art motion attention models.

Evidence that emotion mediates social attention in rhesus macaques

BACKGROUND

Recent work on non-human primates indicates that the allocation of social attention is mediated by characteristics of the attending animal, such as social status and genotype, as well as by the value of the target to which attention is directed. Studies of humans indicate that an individual's emotion state also plays a crucial role in mediating their social attention; for example, individuals look for longer towards aggressive faces when they are feeling more anxious, and this bias leads to increased negative arousal and distraction from other ongoing tasks. To our knowledge, no studies have tested for an effect of emotion state on allocation of social attention in any non-human species.

METHODOLOGY

We presented captive adult male rhesus macaques with pairs of adult male conspecific face images - one with an aggressive expression, one with a neutral expression - and recorded gaze towards these images. Each animal was tested twice, once during a putatively stressful condition (i.e. following a veterinary health check), and once during a neutral (or potentially positive) condition (i.e. a period of environmental enrichment). Initial analyses revealed that behavioural indicators of anxiety and stress were significantly higher after the health check than during enrichment, indicating that the former caused a negative shift in emotional state.

PRINCIPLE FINDINGS

The macaques showed initial vigilance for aggressive faces across both conditions, but subsequent responses differed between conditions. Following the health check, initial vigilance was followed by rapid and sustained avoidance of aggressive faces. By contrast, during the period of enrichment, the macaques showed sustained attention towards the same aggressive faces.

CONCLUSIONS/SIGNIFICANCE

These data provide, to our knowledge, the first evidence that shifts in emotion state mediate social attention towards and away from facial cues of emotion in a non-human animal. This work provides novel insights into the evolution of emotion-attention interactions in humans, and mechanisms of social behaviour in non-human primates, and may have important implications for understanding animal psychological wellbeing.

Attentional effects on gaze preference for salient loci in traffic scenes

Alerting drivers for self-regulation of attention might decrease crash risks attributable to absent-minded driving. However, no reliable method exists for monitoring driver attention. Therefore, we examined attentional effects on gaze preference for salient loci (GPS) in traffic scenes. In an active viewing (AV) condition requiring endogenous attention for traffic scene comprehension, participants identified appropriate speeds for driving in presented traffic scene images. In a passive viewing (PV) condition requiring no endogenous attention, participants passively viewed traffic scene images. GPS was quantified by the mean saliency value averaged across fixation locations. Results show that GPS was less during AV than during PV. Additionally, gaze dwell time on signboards was shorter for AV than for PV. These results suggest that, in the absence of endogenous attention for traffic scene comprehension, gaze tends to concentrate on irrelevant salient loci in a traffic environment. Therefore, increased GPS can indicate absent-minded driving.

PRACTITIONER SUMMARY

The present study demonstrated that, without endogenous attention for traffic scene comprehension, gaze tends to concentrate on irrelevant salient loci in a traffic environment. This result suggests that increased gaze preference for salient loci indicates absent-minded driving, which is otherwise difficult to detect.

Overt attention toward oriented objects in free-viewing barn owls

Visual saliency based on orientation contrast is a perceptual product attributed to the functional organization of the mammalian brain. We examined this visual phenomenon in barn owls by mounting a wireless video microcamera on the owls' heads and confronting them with visual scenes that contained one differently oriented target among similarly oriented distracters. Without being confined by any particular task, the owls looked significantly longer, more often, and earlier at the target, thus exhibiting visual search strategies so far demonstrated in similar conditions only in primates. Given the considerable differences in phylogeny and the structure of visual pathways between owls and humans, these findings suggest that orientation saliency has computational optimality in a wide variety of ecological contexts, and thus constitutes a universal building block for efficient visual information processing in general.

Perceptual mechanism underlying gaze guidance in chimpanzees and humans

Previous studies comparing eye movements between humans and their closest relatives, chimpanzees, have revealed similarities and differences between the species in terms of where individuals fixate their gaze during free viewing of a naturalistic scene, including social stimuli (e.g. body and face). However, those results were somewhat confounded by the fact that gaze behavior is influenced by low-level stimulus properties (e.g., color and form) and by high-level processes such as social sensitivity and knowledge about the scene. Given the known perceptual and cognitive similarities between chimpanzees and humans, it is expected that such low-level effects do not play a critical role in explaining the high-level similarities and differences between the species. However, there is no quantitative evidence to support this assumption. To estimate the effect of local stimulus saliency on such eye-movement patterns, this study used a well-established bottom-up saliency model. In addition, to elucidate the cues that the viewers use to guide their gaze, we presented scenes in which we had manipulated various stimulus properties. As expected, the saliency model did not fully predict the fixation patterns actually observed in chimpanzees and humans. In addition, both species used multiple cues to fixate socially significant areas such as the face. There was no evidence suggesting any differences between chimpanzees and humans in their responses to low-level saliency. Therefore, this study found a substantial amount of similarity in the perceptual mechanisms underlying gaze guidance in chimpanzees and humans and thereby offers a foundation for direct comparisons between them.

Influence of low-level stimulus features, task dependent factors, and spatial biases on overt visual attention

Visual attention is thought to be driven by the interplay between low-level visual features and task dependent information content of local image regions, as well as by spatial viewing biases. Though dependent on experimental paradigms and model assumptions, this idea has given rise to varying claims that either bottom-up or top-down mechanisms dominate visual attention. To contribute toward a resolution of this discussion, here we quantify the influence of these factors and their relative importance in a set of classification tasks. Our stimuli consist of individual image patches (bubbles). For each bubble we derive three measures: a measure of salience based on low-level stimulus features, a measure of salience based on the task dependent information content derived from our subjects' classification responses and a measure of salience based on spatial viewing biases. Furthermore, we measure the empirical salience of each bubble based on our subjects' measured eye gazes thus characterizing the overt visual attention each bubble receives. A multivariate linear model relates the three salience measures to overt visual attention. It reveals that all three salience measures contribute significantly. The effect of spatial viewing biases is highest and rather constant in different tasks. The contribution of task dependent information is a close runner-up. Specifically, in a standardized task of judging facial expressions it scores highly. The contribution of low-level features is, on average, somewhat lower. However, in a prototypical search task, without an available template, it makes a strong contribution on par with the two other measures. Finally, the contributions of the three factors are only slightly redundant, and the semi-partial correlation coefficients are only slightly lower than the coefficients for full correlations. These data provide evidence that all three measures make significant and independent contributions and that none can be neglected in a model of human overt visual attention.

Human-monkey gaze correlations reveal convergent and divergent patterns of movie viewing

The neuroanatomical organization of the visual system is largely similar across primate species, predicting similar visual behaviors and perceptions. Although responses to trial-by-trial presentation of static images suggest that primates share visual orienting strategies, these reduced stimuli fail to capture key elements of the naturalistic, dynamic visual world in which we evolved. Here, we compared the gaze behavior of humans and macaques when they viewed three different 3-minute movie clips. We found significant intersubject and interspecies gaze correlations, suggesting that both species attend a common set of events in each scene. Comparing human and monkey gaze behavior with a computational saliency model revealed that interspecies gaze correlations were driven by biologically relevant social stimuli overlooked by low-level saliency models. Additionally, humans, but not monkeys, tended to gaze toward the targets of viewed individual's actions or gaze. Together, these data suggest that human and monkey gaze behavior comprises converging and diverging informational strategies, driven by both scene content and context; they are not fully described by simple low-level visual models.

Overt and covert visual search in primates: reaction times and gaze shift strategies

In order to investigate the search performance and strategies of nonhuman primates, two macaque monkeys were trained to search for a target template among differently oriented distractors in both free-gaze and fixed-gaze viewing conditions (overt and covert search). In free-gaze search, reaction times (RT) and eye movements revealed the theoretically predicted characteristics of exhaustive and self-terminating serial search, with certain exceptions that are also observed in humans. RT was linearly related to the number of fixations but not necessarily to the number of items on display. Animals scanned the scenes in a nonrandom manner spending notably more time on targets and items inspected last (just before reaction). The characteristics of free-gaze search were then compared with search performance under fixed gaze (covert search) and with the performance of four human subjects tested in similar experiments. By and large the performance characteristics of both groups were similar; monkeys were slightly faster, and humans more accurate. Both species produced shorter RT in fixed-gaze than in free-gaze search. But while RT slopes of the human subjects still showed the theoretically predicted difference between hits and rejections, slopes of the two monkeys appeared to collapse. Despite considerable priming and short-term learning when similar tests were continuously repeated, no substantial long-term training effects were seen when test conditions and set sizes were frequently varied. Altogether, the data reveal many similarities between human and monkey search behavior but indicate that search is not necessarily restricted to exclusively serial processes.

Saliency on a natural scene background: effects of color and luminance contrast add linearly

In natural vision, shifts in spatial attention are associated with shifts of gaze. Computational models of such overt attention typically use the concept of a saliency map: Normalized maps of center-surround differences are computed for individual stimulus features and added linearly to obtain the saliency map. Although the predictions of such models correlate with fixated locations better than chance, their mechanistic assumptions are less well investigated. Here, we tested one key assumption: Do the effects of different features add linearly or according to a max-type of interaction? We measured the eye position of observers viewing natural stimuli whose luminance contrast and/or color contrast (saturation) increased gradually toward one side. We found that these feature gradients biased fixations toward regions of high contrasts. When two contrast gradients (color and luminance) were superimposed, linear summation of their individual effects predicted their combined effect. This demonstrated that the interaction of color and luminance contrast with respect to human overt attention is--irrespective of the precise model--consistent with the assumption of linearity, but not with a max-type interaction of these features.

Bayesian surprise attracts human attention

We propose a formal Bayesian definition of surprise to capture subjective aspects of sensory information. Surprise measures how data affects an observer, in terms of differences between posterior and prior beliefs about the world. Only data observations which substantially affect the observer's beliefs yield surprise, irrespectively of how rare or informative in Shannon's sense these observations are. We test the framework by quantifying the extent to which humans may orient attention and gaze towards surprising events or items while watching television. To this end, we implement a simple computational model where a low-level, sensory form of surprise is computed by simple simulated early visual neurons. Bayesian surprise is a strong attractor of human attention, with 72% of all gaze shifts directed towards locations more surprising than the average, a figure rising to 84% when focusing the analysis onto regions simultaneously selected by all observers. The proposed theory of surprise is applicable across different spatio-temporal scales, modalities, and levels of abstraction.

SUN: A Bayesian framework for saliency using natural statistics

We propose a definition of saliency by considering what the visual system is trying to optimize when directing attention. The resulting model is a Bayesian framework from which bottom-up saliency emerges naturally as the self-information of visual features, and overall saliency (incorporating top-down information with bottom-up saliency) emerges as the pointwise mutual information between the features and the target when searching for a target. An implementation of our framework demonstrates that our model's bottom-up saliency maps perform as well as or better than existing algorithms in predicting people's fixations in free viewing. Unlike existing saliency measures, which depend on the statistics of the particular image being viewed, our measure of saliency is derived from natural image statistics, obtained in advance from a collection of natural images. For this reason, we call our model SUN (Saliency Using Natural statistics). A measure of saliency based on natural image statistics, rather than based on a single test image, provides a straightforward explanation for many search asymmetries observed in humans; the statistics of a single test image lead to predictions that are not consistent with these asymmetries. In our model, saliency is computed locally, which is consistent with the neuroanatomy of the early visual system and results in an efficient algorithm with few free parameters.

Through a barn owl's eyes: interactions between scene content and visual attention

In this study we investigated visual attention properties of freely behaving barn owls, using a miniature wireless camera attached to their heads. The tubular eye structure of barn owls makes them ideal subjects for this research since it limits their eye movements. Video sequences recorded from the owl's point of view capture part of the visual scene as seen by the owl. Automated analysis of video sequences revealed that during an active search task, owls repeatedly and consistently direct their gaze in a way that brings objects of interest to a specific retinal location (retinal fixation area). Using a projective model that captures the geometry between the eye and the camera, we recovered the corresponding location in the recorded images (image fixation area). Recording in various types of environments (aviary, office, outdoors) revealed significant statistical differences of low level image properties at the image fixation area compared to values extracted at random image patches. These differences are in agreement with results obtained in primates in similar studies. To investigate the role of saliency and its contribution to drawing the owl's attention, we used a popular bottom-up computational model. Saliency values at the image fixation area were typically greater than at random patches, yet were only 20% out of the maximal saliency value, suggesting a top-down modulation of gaze control.

Fixations in natural scenes: interaction of image structure and image content

Explorative eye movements specifically target some parts of a scene while ignoring others. Here, we investigate how local image structure--defined by spatial frequency contrast--and informative image content--defined by higher order image statistics-are weighted for the selection of fixation points. We measured eye movements of macaque monkeys freely viewing a set of natural and manipulated images outside a particular task. To probe the effect of scene content, we locally introduced patches of pink noise into natural images, and to probe the interaction with image structure, we altered the contrast of the noise. We found that fixations specifically targeted the natural image parts and spared the uninformative noise patches. However, both increasing and decreasing the contrast of the noise attracted more fixations, and, in the extreme cases, compensated the effect of missing content. Introducing patches from another natural image led to similar results. In all paradigms tested, the interaction between scene structure and informative scene content was the same in any of the first six fixations on an image, demonstrating that the weighting of these factors is constant during viewing of an image. These results question theories, which suggest that initial fixations are driven by stimulus structure whereas later fixations are determined by informative scene content.