Visual attention: the where, what, how and why of saliency

Application of 3D neural networks and explainable AI to classify ICDAS detection system on mandibular molars

STATEMENT OF PROBLEM

Considerable variations exist in cavity preparation methods and approaches. Whether the extent and depth of cavity preparation because of the extent of caries affects the overall accuracy of training deep learning models remains unexplored.

PURPOSE

The purpose of this study was to investigate the difference in 3-dimensionsal (3D) model cavity preparations after International Caries Detection and Assessment System (ICDAS) classification performed by different practitioners and the subsequent influence on the ability of a deep learning model to predict cavity classification.

MATERIAL AND METHODS

Two operators prepared 56 restorative cavities on simulated mandibular first molars according to 4 ICDAS classifications, followed by 3D scanning and computer-aided design processing. The surface area, virtual volume, Hausdorff distance (HD), and Dice Similarity Coefficients were computed. Multivariate analysis of variance was used to assess cavity size and operator proficiency interactions, and 1-way ANOVA was used to evaluate HD differences across 4 cavity classifications (α=.05). The 3D convolutional neural network (CNN) predicted the ICDAS class, and Saliency Maps explained the decisions of the models.

RESULTS

Operator 1 exhibited a cavity preparation surface area of 360.55 ±15.39 mm2, and operator 2 recorded 355.24 ±10.79 mm2. Volumetric differences showed operator 1 with 440.41 ±35.29 mm3 and operator 2 with 441.01 ±35.37 mm3. Significant interactions (F=2.31, P=.01) between cavity size and operator proficiency were observed. A minimal 0.13 ±0.097 mm variation was noted in overlapping preparations by the 2 operators. The 3D CNN model achieved an accuracy of 94.44% in classifying the ICDAS classes with a 66.67% accuracy when differentiating cavities prepared by the 2 operators.

CONCLUSIONS

Operator performance discrepancies were evident in the occlusal cavity floor, primarily due to varying cavity depths. Deep learning effectively classified cavity depths from 3D intraoral scans and was less affected by preparation quality or operator skills.

Emotion-Guided Attention Impacts Deliberate Multi-Evidence Emotion-Related Perceptual Decision-Making

Emotion-guided endogenous attention (e.g., attending to fear) may play a crucial role in determining how humans integrate emotional evidence from various sources when assessing the general emotional tenor of the environment. For instance, what emotion a presenter focuses on can shape their perception of the overall emotion of the room. While there is an increasing interest in understanding how endogenous attention affects emotion perception, existing studies have largely focused on single-stimulus perception. There is limited understanding of how endogenous attention influences emotion evidence integration across multiple sources. To investigate this question, human participants (N = 40) were invited to judge the average emotion across an array of faces ranging from fearful to happy. Endogenous attention was manipulated by instructing participants to decide whether the face array was "fearful or not" (fear attention), "happy or not" (happy attention). Eye movement results revealed an endogenous attention-induced sampling bias such that participants paid more attention to extreme emotional evidence congruent with the target emotion. Computational modeling revealed that endogenous attention shifted the decision criterion to be more conservative, leading to reduced target-category decisions. These findings unraveled the cognitive and computational mechanisms of how endogenous attention impacts the way we gather emotional evidence and make integrative decisions, shedding light on emotion-related decision-making.

How is emotional evidence from multiple sources used in perceptual decision making?

Judging the emotional nature of a scene requires us to deliberately integrate pieces of evidence with varying intensities of emotion. Our existing knowledge about emotion-related perceptual decision-making is largely based on paradigms using single stimulus and, when involving multiple stimuli, rapid decisions. Consequently, it remains unclear how we sample and integrate multiple pieces of emotional evidence deliberately to form an overall judgment. Findings from non-emotion rapid decision-making studies show humans down-sample and downweight extreme evidence. However, deliberate decision-making may rely on a different attention mode than in rapid decision-making; and extreme emotional stimuli are inherently salient. Given these critical differences, it is imperative to directly examine the deliberate decision-making process about multiple emotional stimuli. In the current study, human participants (N = 33) viewed arrays of faces with expressions ranging from extremely fearful to extremely happy freely with their eye movement tracked. They then decided whether the faces were more fearful or happier on average. In contrast to conclusions drawn from non-emotion and rapid decision-making studies, eye movement measures revealed that participants attentionally sampled extreme emotional evidence more than less extreme evidence. Computational modeling results indicated that even though participants exhibited biased attention distribution, they weighted various emotional evidence equally. These findings provide novel insights into how people sample and integrate multiple pieces of emotional evidence, contribute to a more comprehensive understanding of emotion-related decision-making, and shed light on the mechanisms of pathological affective decisions.

Martinez L, Criado-Boado F, Müeller J, Clark A. A computational approach to selective attention in embodied approaches to cognitive archaeology

This article proposes a novel computational approach to embodied approaches in cognitive archaeology called computational cognitive archaeology (CCA). We argue that cognitive archaeology, understood as the study of the human mind based on archaeological findings such as artefacts and material remains excavated and interpreted in the present, can benefit from the integration of novel methods in computational neuroscience interested in modelling the way the brain, the body and the environment are coupled and parameterized to allow for adaptive behaviour. We discuss the kind of tasks that CCA may engage in with a narrative example of how one can model the cumulative cultural evolution of the material and cognitive components of technologies, focusing on the case of knapping technology. This article thus provides a novel theoretical framework to formalize research in cognitive archaeology using recent developments in computational neuroscience.

Selective and prolonged attention to emotional scenes in humans and bonobos

Perceiving emotions in others is at the foundation of higher-order social cognition. The importance of emotions is evidenced by the fact that they receive prioritized attention at early stages of processing the environment in humans and some other primates. Nevertheless, we do not fully understand how emotion modulates attention over longer durations in primates, particularly in great apes. Bonobos, one of our closest relatives, stand out in emotion processing and regulation among great apes. This makes them an interesting comparison species and a valuable model for studying the evolution of emotion perception in hominids. We investigated how bonobos and humans spontaneously attend to emotionally valent scenes in a preferential looking task using eye-tracking. With Bayesian mixed modelling, we found that bonobos and humans generally looked longer at emotional scenes, mainly of conspecifics. Moreover, while bonobos did not have a bias toward emotional human scenes, humans sustained their attention toward bonobos playing, grooming and having sex. Furthermore, when exploring an immediate bias for emotions, humans showed a bias toward affiliative human scenes, and bonobos showed a bias away from bonobos-in-distress scenes. These findings suggest that emotions modulate attention at early and later attentional stages in bonobos, similar to humans.

Sensory Drive Modifies Brain Dynamics and the Temporal Integration Window

Perception is suggested to occur in discrete temporal windows, clocked by cycles of neural oscillations. An important testable prediction of this theory is that individuals' peak frequencies of oscillations should correlate with their ability to segregate the appearance of two successive stimuli. An influential study tested this prediction and showed that individual peak frequency of spontaneously occurring alpha (8-12 Hz) correlated with the temporal segregation threshold between two successive flashes of light [Samaha, J., & Postle, B. R. The speed of alpha-band oscillations predicts the temporal resolution of visual perception. Current Biology, 25, 2985-2990, 2015]. However, these findings were recently challenged [Buergers, S., & Noppeney, U. The role of alpha oscillations in temporal binding within and across the senses. Nature Human Behaviour, 6, 732-742, 2022]. To advance our understanding of the link between oscillations and temporal segregation, we devised a novel experimental approach. Rather than relying entirely on spontaneous brain dynamics, we presented a visual grating before the flash stimuli that is known to induce continuous oscillations in the gamma band (45-65 Hz). By manipulating the contrast of the grating, we found that high contrast induces a stronger gamma response and a shorter temporal segregation threshold, compared to low-contrast trials. In addition, we used a novel tool to characterize sustained oscillations and found that, for half of the participants, both the low- and high-contrast gratings were accompanied by a sustained and phase-locked alpha oscillation. These participants tended to have longer temporal segregation thresholds. Our results suggest that visual stimulus drive, reflected by oscillations in specific bands, is related to the temporal resolution of visual perception.

Representational maps in the brain: concepts, approaches, and applications

Neural systems have evolved to process sensory stimuli in a way that allows for efficient and adaptive behavior in a complex environment. Recent technological advances enable us to investigate sensory processing in animal models by simultaneously recording the activity of large populations of neurons with single-cell resolution, yielding high-dimensional datasets. In this review, we discuss concepts and approaches for assessing the population-level representation of sensory stimuli in the form of a representational map. In such a map, not only are the identities of stimuli distinctly represented, but their relational similarity is also mapped onto the space of neuronal activity. We highlight example studies in which the structure of representational maps in the brain are estimated from recordings in humans as well as animals and compare their methodological approaches. Finally, we integrate these aspects and provide an outlook for how the concept of representational maps could be applied to various fields in basic and clinical neuroscience.

Paying attention to natural scenes in area V1

Natural scene responses in the primary visual cortex are modulated simultaneously by attention and by contextual signals about scene statistics stored across the connectivity of the visual processing hierarchy. We hypothesized that attentional and contextual signals interact in V1 in a manner that primarily benefits the representation of natural stimuli, rich in high-order statistical structure. Recording from two macaques engaged in a spatial attention task, we found that attention enhanced the decodability of stimulus identity from population responses evoked by natural scenes, but not by synthetic stimuli lacking higher-order statistical regularities. Population analysis revealed that neuronal responses converged to a low-dimensional subspace only for natural stimuli. Critically, we determined that the attentional enhancement in stimulus decodability was captured by the natural-scene subspace, indicating an alignment between the attentional and natural stimulus variance. These results suggest that attentional and contextual signals interact in V1 in a manner optimized for natural vision.

Pathway-selective optogenetics reveals the functional anatomy of top-down attentional modulation in the macaque visual cortex

Spatial attention represents a powerful top-down influence on sensory responses in primate visual cortical areas. The frontal eye field (FEF) has emerged as a key candidate area for the source of this modulation. However, it is unclear whether the FEF exerts its effects via its direct axonal projections to visual areas or indirectly through other brain areas and whether the FEF affects both the enhancement of attended and the suppression of unattended sensory responses. We used pathway-selective optogenetics in rhesus macaques performing a spatial attention task to inhibit the direct input from the FEF to area MT, an area along the dorsal visual pathway specialized for the processing of visual motion information. Our results show that the optogenetic inhibition of the FEF input specifically reduces attentional modulation in MT by about a third without affecting the neurons' sensory response component. We find that the direct FEF-to-MT pathway contributes to both the enhanced processing of target stimuli and the suppression of distractors. The FEF, thus, selectively modulates firing rates in visual area MT, and it does so via its direct axonal projections.

Are acoustics enough? Semantic effects on auditory salience in natural scenes

Auditory salience is a fundamental property of a sound that allows it to grab a listener's attention regardless of their attentional state or behavioral goals. While previous research has shed light on acoustic factors influencing auditory salience, the semantic dimensions of this phenomenon have remained relatively unexplored owing both to the complexity of measuring salience in audition as well as limited focus on complex natural scenes. In this study, we examine the relationship between acoustic, contextual, and semantic attributes and their impact on the auditory salience of natural audio scenes using a dichotic listening paradigm. The experiments present acoustic scenes in forward and backward directions; the latter allows to diminish semantic effects, providing a counterpoint to the effects observed in forward scenes. The behavioral data collected from a crowd-sourced platform reveal a striking convergence in temporal salience maps for certain sound events, while marked disparities emerge in others. Our main hypothesis posits that differences in the perceptual salience of events are predominantly driven by semantic and contextual cues, particularly evident in those cases displaying substantial disparities between forward and backward presentations. Conversely, events exhibiting a high degree of alignment can largely be attributed to low-level acoustic attributes. To evaluate this hypothesis, we employ analytical techniques that combine rich low-level mappings from acoustic profiles with high-level embeddings extracted from a deep neural network. This integrated approach captures both acoustic and semantic attributes of acoustic scenes along with their temporal trajectories. The results demonstrate that perceptual salience is a careful interplay between low-level and high-level attributes that shapes which moments stand out in a natural soundscape. Furthermore, our findings underscore the important role of longer-term context as a critical component of auditory salience, enabling us to discern and adapt to temporal regularities within an acoustic scene. The experimental and model-based validation of semantic factors of salience paves the way for a complete understanding of auditory salience. Ultimately, the empirical and computational analyses have implications for developing large-scale models for auditory salience and audio analytics.

Responses to Raven matrices: Governed by visual complexity and centrality

Raven matrices are widely considered a pure test of cognitive abilities. Previous research has examined the extent to which cognitive strategies are predictive of the number of correct responses to Raven items. This study examined whether response times can be explained directly from the centrality and visual complexity of the matrix cells (edge density and perceived complexity). A total of 159 participants completed a 12-item version of the Raven Advanced Progressive Matrices. In addition to item number (an index of item difficulty), the findings demonstrated a positive correlation between the visual complexity of Raven items and both the mean response time and the number of fixations on the matrix (a strong correlate of response time). Moreover, more centrally placed cells as well as more complex cells received more fixations. It is concluded that response times on Raven matrices are impacted by low-level stimulus attributes, namely, visual complexity and eccentricity.

Modeling Eye Movements During Decision Making: A Review

This article reviews recent advances in the psychometric and econometric modeling of eye-movements during decision making. Eye movements offer a unique window on unobserved perceptual, cognitive, and evaluative processes of people who are engaged in decision making tasks. They provide new insights into these processes, which are not easily available otherwise, allow for explanations of fundamental search and choice phenomena, and enable predictions of future decisions. We propose a theoretical framework of the search and choice tasks that people commonly engage in and of the underlying cognitive processes involved in those tasks. We discuss how these processes drive specific eye-movement patterns. Our framework emphasizes the central role of task and strategy switching for complex goal attainment. We place the extant literature within that framework, highlight recent advances in modeling eye-movement behaviors during search and choice, discuss limitations, challenges, and open problems. An agenda for further psychometric modeling of eye movements during decision making concludes the review.

Co-Occurrence, Extension, and Social Salience: The Emergence of Indexicality in an Artificial Language

We investigated the emergence of sociolinguistic indexicality using an artificial-language-learning paradigm. Sociolinguistic indexicality involves the association of linguistic variants with nonlinguistic social or contextual features. Any linguistic variant can acquire "constellations" of such indexical meanings, though they also exhibit an ordering, with first-order indices associated with particular speaker groups and higher-order indices targeting stereotypical attributes of those speakers. Much natural-language research has been conducted on this phenomenon, but little experimental work has focused on how indexicality emerges. Here, we present three miniature artificial-language experiments designed to break ground on this question. Results show ready formation of first-order indexicality based on co-occurrence alone, with higher-order indexicality emerging as a result of extension to new speaker groups, modulated by the perceived practical importance of the indexed social feature.

Revealing the stimulus-driven component of attention through modulations of auditory salience by timbre attributes

Attention allows the listener to select relevant information from their environment, and disregard what is irrelevant. However, irrelevant stimuli sometimes manage to capture it and stand out from a scene because of bottom-up processes driven by salient stimuli. This attentional capture effect was observed using an implicit approach based on the additional singleton paradigm. In the auditory domain, it was shown that sound attributes such as intensity and frequency tend to capture attention during auditory search (cost to performance) for targets defined on a different dimension such as duration. In the present study, the authors examined whether a similar phenomenon occurs for attributes of timbre such as brightness (related to the spectral centroid) and roughness (related the amplitude modulation depth). More specifically, we revealed the relationship between the variations of these attributes and the magnitude of the attentional capture effect. In experiment 1, the occurrence of a brighter sound (higher spectral centroid) embedded in sequences of successive tones produced significant search costs. In experiments 2 and 3, different values of brightness and roughness confirmed that attention capture is monotonically driven by the sound features. In experiment 4, the effect was found to be symmetrical: positive or negative, the same difference in brightness had the same negative effect on performance. Experiment 5 suggested that the effect produced by the variations of the two attributes is additive. This work provides a methodology for quantifying the bottom-up component of attention and brings new insights on attention capture and auditory salience.

Reason-Based Recommendations From a Developmental Systems Approach for Students With Needs Across Functional Domains

PURPOSE

This tutorial aims to introduce school-based speech-language pathologists (SLPs) to developmental systems theory as a framework for considering interactions across functional domains, such as language, vision, and motor, for students with complex needs.

METHOD

This tutorial summarizes the current literature on developmental systems theory in its application to working with students who have needs in multiple domains of functioning in addition to communication. A hypothetical case of a student, James, with cerebral palsy, cortical visual impairment, and complex communication needs, is presented to illustrate the primary tenets of the theory.

RESULTS

Specific reason-based recommendations are presented that SLPs can put to practice with their own caseload in direct response to the three tenets of developmental systems theory.

CONCLUSIONS

A developmental systems approach will be useful in expanding SLP knowledge of where to begin and how to best serve children with language, motor, vision, and other concomitant needs. The tenets, including sampling, context dependency, and interdependency, and the application of developmental systems theory can be instrumental in providing a way forward for SLPs struggling with the assessment and intervention of students with complex needs.

Dementia in Convolutional Neural Networks: Using Deep Learning Models to Simulate Neurodegeneration of the Visual System

Although current research aims to improve deep learning networks by applying knowledge about the healthy human brain and vice versa, the potential of using such networks to model and study neurodegenerative diseases remains largely unexplored. In this work, we present an in-depth feasibility study modeling progressive dementia in silico with deep convolutional neural networks. Therefore, networks were trained to perform visual object recognition and then progressively injured by applying neuronal as well as synaptic injury. After each iteration of injury, network object recognition accuracy, saliency map similarity between the intact and injured networks, and internal activations of the degenerating models were evaluated. The evaluation revealed that cognitive function of the network progressively decreased with increasing injury load whereas this effect was much more pronounced for synaptic damage. The effects of neurodegeneration found for the in silico model are especially similar to the loss of visual cognition seen in patients with posterior cortical atrophy.

Do our hands see what our eyes see? Investigating spatial and haptic abilities

Spatial abilities (SAs) are cognitive resources used to mentally manipulate representations of objects to solve problems. Haptic abilities (HAs) represent tactile interactions with real-world objects transforming somatic information into mental representations. Both are proposed to be factors in anatomy education, yet relationships between SAs and HAs remain unknown. The objective of the current study was to explore SA-HA interactions. A haptic ability test (HAT) was developed based on the mental rotations test (MRT) with three-dimensional (3D) objects. The HAT was undertaken in three sensory conditions: (1) sighted, (2) sighted with haptics, and (3) haptics. Participants (n = 22; 13 females, 9 males) completed the MRT and were categorized into high spatial abilities (HSAs) (n = 12, mean± standard deviation: 13.7 ± 3.0) and low spatial abilities (LSAs) (n = 10, 5.6 ± 2.0) based on score distributions about the overall mean. Each SA group's HAT scores were compared across the three sensory conditions. Spearman's correlation coefficients between MRT and HAT scores indicated a statistically significant correlation in sighted condition (r = 0.553, p = 0.015) but were not significant in the sighted with haptics (r = 0.0.078, p = 0.212) and haptics (r = 0.043, p = 0.279) conditions. These data suggest HAs appear unrelated to SAs. With haptic exploration, LSA HAT scores were compensated; comparing HSA with LSA: sighted with haptics [median (lower and upper quartiles): 12 (12,13) vs. 12 (11,13), p = 0.254], and haptics [12 (11,13) vs. 12 (10,12), p = 0.381] conditions. Migrations to online anatomy teaching may unwittingly remove important sensory modalities from the learner. Understanding learner behaviors and performance when haptic inputs are removed from the learning environment represents valuable insight informing future anatomy curriculum and resource development.

Frequency modulation of cortical rhythmicity governs behavioral variability, excitability and synchrony of neurons in the visual cortex

Research in cognitive neuroscience has renewed the idea that brain oscillations are a core organization implicated in fundamental brain functions. Growing evidence reveals that the characteristic features of these oscillations, including power, phase and frequency, are highly non-stationary, fluctuating alongside alternations in sensation, cognition and behavior. However, there is little consensus on the functional implications of the instantaneous frequency variation in cortical excitability and concomitant behavior. Here, we capitalized on intracortical electrophysiology in the macaque monkey's visual area MT performing a visuospatial discrimination task with visual cues. We observed that the instantaneous frequency of the theta-alpha oscillations (4-13 Hz) is modulated among specific neurons whose RFs overlap with the cued stimulus location. Interestingly, we found that such frequency modulation is causally correlated with MT excitability at both scales of individual and ensemble of neurons. Moreover, studying the functional relevance of frequency variations indicated that the average theta-alpha frequencies foreshadow the monkey's reaction time. Our results also revealed that the neural synchronization strength alters with the average frequency shift in theta-alpha oscillations, suggesting frequency modulation is critical for mutually adjusting MTs' rhythms. Overall, our findings propose that theta-alpha frequency variations modulate MT's excitability, regulate mutual neurons' rhythmicity and indicate variability in behavior.

Facial mask disturbs ocular exploration but not pupil reactivity

INTRODUCTION

The COVID-19 pandemic has imposed to wear a face mask that may have negative consequences for social interactions despite its health benefits. A lot of recent studies focused on emotion recognition of masked faces, as the mouth is, with the eyes, essential to convey emotional content. However, none have studied neurobehavioral and neurophysiological markers of masked faces perception, such as ocular exploration and pupil reactivity. The purpose of this eye tracking study was to quantify how wearing a facial accessory, and in particular a face mask, affected the ocular and pupillary response to a face, emotional or not.

METHODS

We used videos of actors wearing a facial accessory to characterize the visual exploration and pupillary response in several occlusion (no accessory, sunglasses, scarf, and mask) and emotional conditions (neutral, happy, and sad) in a population of 44 adults.

RESULTS

We showed that ocular exploration differed for face covered with an accessory, and in particular a mask, compared to the classical visual scanning pattern of a non-covered face. The covered areas of the face were less explored. Pupil reactivity seemed only slightly affected by the mask, while its sensitivity to emotions was observed even in the presence of a facial accessory.

DISCUSSION

These results suggest a mixed impact of the mask on attentional capture and physiological adjustment, which does not seem to be reconcilable with its strong effect on behavioral emotional recognition previously described.

Selective particle attention: Rapidly and flexibly selecting features for deep reinforcement learning

Deep Reinforcement Learning (RL) is often criticised for being data inefficient and inflexible to changes in task structure. Part of the reason for these issues is that Deep RL typically learns end-to-end using backpropagation, which results in task-specific representations. One approach for circumventing these problems is to apply Deep RL to existing representations that have been learned in a more task-agnostic fashion. However, this only partially solves the problem as the Deep RL algorithm learns a function of all pre-existing representations and is therefore still susceptible to data inefficiency and a lack of flexibility. Biological agents appear to solve this problem by forming internal representations over many tasks and only selecting a subset of these features for decision-making based on the task at hand; a process commonly referred to as selective attention. We take inspiration from selective attention in biological agents and propose a novel algorithm called Selective Particle Attention (SPA), which selects subsets of existing representations for Deep RL. Crucially, these subsets are not learned through backpropagation, which is slow and prone to overfitting, but instead via a particle filter that rapidly and flexibly identifies key subsets of features using only reward feedback. We evaluate SPA on two tasks that involve raw pixel input and dynamic changes to the task structure, and show that it greatly increases the efficiency and flexibility of downstream Deep RL algorithms.

LPMP: A Bio-Inspired Model for Visual Localization in Challenging Environments

Autonomous vehicles require precise and reliable self-localization to cope with dynamic environments. The field of visual place recognition (VPR) aims to solve this challenge by relying on the visual modality to recognize a place despite changes in the appearance of the perceived visual scene. In this paper, we propose to tackle the VPR problem following a neuro-cybernetic approach. To this end, the Log-Polar Max-Pi (LPMP) model is introduced. This bio-inspired neural network allows building a neural representation of the environment via an unsupervised one-shot learning. Inspired by the spatial cognition of mammals, visual information in the LPMP model are processed through two distinct pathways: a "what" pathway that extracts and learns the local visual signatures (landmarks) of a visual scene and a "where" pathway that computes their azimuth. These two pieces of information are then merged to build a visuospatial code that is characteristic of the place where the visual scene was perceived. Three main contributions are presented in this article: 1) the LPMP model is studied and compared with NetVLAD and CoHog, two state-of-the-art VPR models; 2) a test benchmark for the evaluation of VPR models according to the type of environment traveled is proposed based on the Oxford car dataset; and 3) the impact of the use of a novel detector leading to an uneven paving of an environment is evaluated in terms of the localization performance and compared to a regular paving. Our experiments show that the LPMP model can achieve comparable or better localization performance than NetVLAD and CoHog.

The time course of salience: not entirely caused by salience

Visual salience is a key component of attentional selection, the process that guards the scarce resources needed for conscious recognition and perception. In previous works, we proposed a measure of visual salience based on a formal theory of visual selection. However, the strength of visual salience depends on the time course as well as local physical contrasts. Evidence from multiple experimental designs in the literature suggests that the strength of salience rises initially and declines after approximately 150 ms. The present article amends the theory-based salience measure beyond local physical contrasts to the time course of salience. It does so through a first experiment which reveals that-contrary to expectations-salience is not reduced during the first 150 ms after onset. Instead, the overall visual processing capacity is severely reduced, which corresponds to a reduced processing speed of all stimuli in the visual field. A second experiment confirms this conclusion by replicating the result. We argue that the slower stimulus processing may have been overlooked previously because the attentional selection mechanism had not yet been modeled in studies on the time course of salience.

Do Children With Developmental Language Disorder Activate Scene Knowledge to Guide Visual Attention? Effect of Object-Scene Inconsistencies on Gaze Allocation

Our visual environment is highly predictable in terms of where and in which locations objects can be found. Based on visual experience, children extract rules about visual scene configurations, allowing them to generate scene knowledge. Similarly, children extract the linguistic rules from relatively predictable linguistic contexts. It has been proposed that the capacity of extracting rules from both domains might share some underlying cognitive mechanisms. In the present study, we investigated the link between language and scene knowledge development. To do so, we assessed whether preschool children (age range = 5;4–6;6) with Developmental Language Disorder (DLD), who present several difficulties in the linguistic domain, are equally attracted to object-scene inconsistencies in a visual free-viewing task in comparison with age-matched children with Typical Language Development (TLD). All children explored visual scenes containing semantic (e.g., soap on a breakfast table), syntactic (e.g., bread on the chair back), or both inconsistencies (e.g., soap on the chair back). Since scene knowledge interacts with image properties (i.e., saliency) to guide gaze allocation during visual exploration from the early stages of development, we also included the objects’ saliency rank in the analysis. The results showed that children with DLD were less attracted to semantic and syntactic inconsistencies than children with TLD. In addition, saliency modulated syntactic effect only in the group of children with TLD. Our findings indicate that children with DLD do not activate scene knowledge to guide visual attention as efficiently as children with TLD, especially at the syntactic level, suggesting a link between scene knowledge and language development.

Statistical Learning of Frequent Distractor Locations in Visual Search Involves Regional Signal Suppression in Early Visual Cortex

Observers can learn locations where salient distractors appear frequently to reduce potential interference-an effect attributed to better suppression of distractors at frequent locations. But how distractor suppression is implemented in the visual cortex and within the frontoparietal attention networks remains unclear. We used fMRI and a regional distractor-location learning paradigm with two types of distractors defined in either the same (orientation) or a different (color) dimension to the target to investigate this issue. fMRI results showed that BOLD signals in early visual cortex were significantly reduced for distractors (as well as targets) occurring at the frequent versus rare locations, mirroring behavioral patterns. This reduction was more robust with same-dimension distractors. Crucially, behavioral interference was correlated with distractor-evoked visual activity only for same- (but not different-) dimension distractors. Moreover, with different- (but not same-) dimension distractors, a color-processing area within the fusiform gyrus was activated more when a distractor was present in the rare region versus being absent and more with a distractor in the rare versus frequent locations. These results support statistical learning of frequent distractor locations involving regional suppression in early visual cortex and point to differential neural mechanisms of distractor handling with different- versus same-dimension distractors.

Decoding Brain Representations by Multimodal Learning of Neural Activity and Visual Features

This work presents a novel method of exploring human brain-visual representations, with a view towards replicating these processes in machines. The core idea is to learn plausible computational and biological representations by correlating human neural activity and natural images. Thus, we first propose a model, EEG-ChannelNet, to learn a brain manifold for EEG classification. After verifying that visual information can be extracted from EEG data, we introduce a multimodal approach that uses deep image and EEG encoders, trained in a siamese configuration, for learning a joint manifold that maximizes a compatibility measure between visual features and brain representations. We then carry out image classification and saliency detection on the learned manifold. Performance analyses show that our approach satisfactorily decodes visual information from neural signals. This, in turn, can be used to effectively supervise the training of deep learning models, as demonstrated by the high performance of image classification and saliency detection on out-of-training classes. The obtained results show that the learned brain-visual features lead to improved performance and simultaneously bring deep models more in line with cognitive neuroscience work related to visual perception and attention.

Semantic priming and neurobiology in schizophrenia: A theoretical review

In this theoretical review we bridge the cognitive and neurobiological sciences to shed light on the neurocognitive foundations of the semantic priming effect in schizophrenia. We review and theoretically evaluate the neurotransmitter systems (dopaminergic, GABAergic and glutamatergic) and neurobiological underpinnings of behavioural and electrophysiological (N400) semantic priming in the pathology, and the main hypotheses on their geneses: a disinhibition of the semantic spread of activation, a disorganised semantic storage or noisy lexical-semantic associations, a psychomotor artefact, an artefact of relatedness proportions, or an inability to mobilise contextual information. We further assess the literature on the endophenotype of Formal Thought Disorder from multiple standpoints, ranging from neurophysiology to cognition: considerations are weaved on neuronal (PV basket cell, SST, VIP) and receptor deficits (DRD1, NMDA), neurotransmitter imbalances (dopamine), cortical and dopaminergic lateralisation, inter alia. In conclusion, we put forth novel postulates on the underlying causes of controlled hypopriming, automatic hyperpriming, N400 reversals (larger amplitudes for close associations), indirect versus direct hyperpriming, and the endophenotype of lexical-semantic disturbances in schizophrenia.

Client-Server Approach for Managing Visual Attention, Integrated in a Cognitive Architecture for a Social Robot

This paper proposes a novel system for managing visual attention in social robots. This system is based on a client/server approach that allows integration with a cognitive architecture controlling the robot. The core of this architecture is a distributed knowledge graph, in which the perceptual needs are expressed by the presence of arcs to stimuli that need to be perceived. The attention server sends motion commands to the actuators of the robot, while the attention clients send requests through the common knowledge representation. The common knowledge graph is shared by all levels of the architecture. This system has been implemented on ROS and tested on a social robot to verify the validity of the approach and was used to solve the tests proposed in RoboCup @ Home and SciROc robotic competitions. The tests have been used to quantitatively compare the proposal to traditional visual attention mechanisms.

Experimental control of conflict in a predictive visual probe task: Highly reliable bias scores related to anxiety

Concerns have been raised about the low reliability of measurements of spatial attentional bias via RT differences in dot-probe tasks. The anticipatory form of the bias, directed towards predicted future stimuli, appears to have relatively good reliability, reaching around 0.70. However, studies thus far have not attempted to experimentally control task-related influence on bias, which could further improve reliability. Evoking top-down versus bottom-up conflict may furthermore reveal associations with individual differences related to mental health. In the current study, a sample of 143 participants performed a predictive Visual Probe Task (predVPT) with angry and neutral face stimuli online. In this task, an automatic bias is induced via visually neutral cues that predict the location of an upcoming angry face. A task-relevant bias was induced via blockwise shifts in the likely location of target stimuli. The bias score resulting from these factors was calculated as RTs to target stimuli at locations of predicted but not actually presented angry versus neutral faces. Correlations were tested with anxiety, depression, self-esteem and aggression scales. An overall bias towards threat was found with a split-half reliability of 0.90, and 0.89 after outlier removal. Avoidance of threat in blocks with a task-relevant bias away from threat was correlated with anxiety, with correction for multiple testing. The same relationship was nominally significant for depression and low self-esteem. In conclusion, we showed high reliability of spatial attentional bias that was related to anxiety.

Evaluating the evidence for expectation suppression in the visual system

Reports of expectation suppression have shaped the development of influential predictive coding-based theories of visual perception. However recent work has highlighted confounding factors that may mimic or inflate expectation suppression effects. In this review, we describe four confounds that are prevalent across experiments that tested for expectation suppression: effects of surprise, attention, stimulus repetition and adaptation, and stimulus novelty. With these confounds in mind we then critically review the evidence for expectation suppression across probabilistic cueing, statistical learning, oddball, action-outcome learning and apparent motion designs. We found evidence for expectation suppression within a specific subset of statistical learning designs that involved weeks of sequence learning prior to neural activity measurement. Across other experimental contexts, whereby stimulus appearance probabilities were learned within one or two testing sessions, there was inconsistent evidence for genuine expectation suppression. We discuss how an absence of expectation suppression could inform models of predictive processing, repetition suppression and perceptual decision-making. We also provide suggestions for designing experiments that may better test for expectation suppression in future work.

Food-related attentional bias and its associations with appetitive motivation and body weight: A systematic review and meta-analysis

Theoretical models suggest that food-related visual attentional bias (AB) may be related to appetitive motivational states and individual differences in body weight; however, findings in this area are equivocal. We conducted a systematic review and series of meta-analyses to determine if there is a positive association between food-related AB and: (1.) body mass index (BMI) (number of effect sizes (k) = 110), (2.) hunger (k = 98), (3.) subjective craving for food (k = 35), and (4.) food intake (k = 44). Food-related AB was robustly associated with craving (r = 0.134 (95% CI 0.061, 0.208); p < .001), food intake (r = 0.085 (95% CI 0.038, 0.132); p < .001), and hunger (r = 0.048 (95% CI 0.016, 0.079); p = .003), but these correlations were small. Food-related AB was unrelated to BMI (r = 0.008 (95% CI -0.020, 0.035); p = .583) and this result was not moderated by type of food stimuli, method of AB assessment, or the subcomponent of AB that was examined. Furthermore, in a between-groups analysis (k = 22) which directly compared participants with overweight/obesity to healthy-weight control groups, there was no evidence for an effect of weight status on food-related AB (Hedge's g = 0.104, (95% CI -0.050, 0.258); p = .186). Taken together, these findings suggest that food-related AB is sensitive to changes in the motivational value of food, but is unrelated to individual differences in body weight. Our findings question the traditional view of AB as a trait-like index of preoccupation with food and have implications for novel theoretical perspectives on the role of food AB in appetite control and obesity.

A Generalized Encoding System for Alpha Oscillations Through Visual Saliency Analysis

By learning how the brain reacts to external visual stimuli and examining possible triggered brain statuses, we conduct a systematic study on an encoding problem that estimates ongoing EEG dynamics from visual information. A novel generalized system is proposed to encode the alpha oscillations modulated during video viewing by employing the visual saliency involved in the presented natural video stimuli. Focusing on the parietal and occipital lobes, the encoding effects at different alpha frequency bins and brain locations are examined by a real-valued genetic algorithm (GA), and possible links between alpha features and saliency patterns are constructed. The robustness and reliability of the proposed system are demonstrated in a 10-fold cross-validation. The results show that stimuli with different saliency levels can induce significant changes in occipito-parietal alpha oscillations and that alpha at higher frequency bins responded the most in involuntary attention related to bottom-up-based visual processing. This study provides a novel approach to understand the processing of involuntary attention in the brain dynamics and would further be beneficial to the development of brain-computer interfaces and visual design.

Evidence of predictive selective attention in fiddler crabs during escape in the natural environment

Selective attention is of fundamental relevance to animals for performing a diversity of tasks such as mating, feeding, predation and avoiding predators. Within natural environments, prey animals are often exposed to multiple, simultaneous threats, which significantly complicates the decision-making process. However, selective attention is rarely studied in complex, natural environments or in the context of escape responses. We therefore asked how relatively simple animals integrate the information from multiple, concurrent threatening events. Do they identify and respond only to what they perceive as the most dangerous threat, or do they respond to multiple stimuli at the same time? Do simultaneous threats evoke an earlier or stronger response than single threats? We investigated these questions by conducting field experiments and compared escape responses of the fiddler crab Gelasimus dampieri when faced with either a single or two simultaneously approaching dummy predators. We used the dummies' approach trajectories to manipulate the threat level; a directly approaching dummy indicated higher risk while a tangentially approaching dummy that passed the crabs at a distance represented a lower risk. The crabs responded later, but on average more often, when approached more directly. However, when confronted with the two dummies simultaneously, the crabs responded as if approached only by the directly approaching dummy. This suggests that the crabs are able to predict how close the dummy's trajectory is to a collision course and selectively suppress their normally earlier response to the less dangerous dummy. We thus provide evidence of predictive selective attention within a natural environment.

ST-MTL: Spatio-Temporal multitask learning model to predict scanpath while tracking instruments in robotic surgery

Representation learning of the task-oriented attention while tracking instrument holds vast potential in image-guided robotic surgery. Incorporating cognitive ability to automate the camera control enables the surgeon to concentrate more on dealing with surgical instruments. The objective is to reduce the operation time and facilitate the surgery for both surgeons and patients. We propose an end-to-end trainable Spatio-Temporal Multi-Task Learning (ST-MTL) model with a shared encoder and spatio-temporal decoders for the real-time surgical instrument segmentation and task-oriented saliency detection. In the MTL model of shared-parameters, optimizing multiple loss functions into a convergence point is still an open challenge. We tackle the problem with a novel asynchronous spatio-temporal optimization (ASTO) technique by calculating independent gradients for each decoder. We also design a competitive squeeze and excitation unit by casting a skip connection that retains weak features, excites strong features, and performs dynamic spatial and channel-wise feature recalibration. To capture better long term spatio-temporal dependencies, we enhance the long-short term memory (LSTM) module by concatenating high-level encoder features of consecutive frames. We also introduce Sinkhorn regularized loss to enhance task-oriented saliency detection by preserving computational efficiency. We generate the task-aware saliency maps and scanpath of the instruments on the dataset of the MICCAI 2017 robotic instrument segmentation challenge. Compared to the state-of-the-art segmentation and saliency methods, our model outperforms most of the evaluation metrics and produces an outstanding performance in the challenge.

Can we accurately predict where we look at paintings?

The objective of this study is to investigate and to simulate the gaze deployment of observers on paintings. For that purpose, we built a large eye tracking dataset composed of 150 paintings belonging to 5 art movements. We observed that the gaze deployment over the proposed paintings was very similar to the gaze deployment over natural scenes. Therefore, we evaluate existing saliency models and propose a new one which significantly outperforms the most recent deep-based saliency models. Thanks to this new saliency model, we can predict very accurately what are the salient areas of a painting. This opens new avenues for many image-based applications such as animation of paintings or transformation of a still painting into a video clip.

Predictive eye movements when hitting a bouncing ball

Predictive eye movements targeted toward the direction of ball bounce are a feature of gaze behavior when intercepting a target soon after it has bounced. However, there is conjecture over the exact location toward which these predictive eye movements are directed, and whether gaze during this period is moving or instead "lies in wait" for the ball to arrive. Therefore, the aim of this study was to further examine the location toward which predictive eye movements are made when hitting a bouncing ball. We tracked the eye and head movements of 23 novice participants who attempted to hit approaching tennis balls in a virtual environment. The balls differed in time from bounce to contact (300, 550, and 800 ms). Results revealed that participants made predictive saccades shortly before the ball bounced in two-thirds of all trials. These saccades were directed several degrees above the position at which the ball bounced, rather than toward the position at which it bounced or toward a position the ball would occupy shortly after the bounce. After the saccade, a separation of roles for the eyes and head ensured that gaze continued to change so that it was as close as possible to the ball soon after bounce. Smooth head movements were responsible for the immediate and ongoing changes in gaze to align it with the ball in the lateral direction, while eye movements realigned gaze with the ball in the vertical direction from approximately 100 ms after the ball changed its direction of motion after bounce. We conclude that predictive saccades direct gaze above the location at which the ball will bounce, presumably in order to facilitate ball tracking after the bounce.

Temporal attention boosts perceptual effects of spatial attention and feature-based attention

Temporal attention, that is, the process of anticipating the occurrence of a stimulus at a given time point, has been shown to improve perceptual processing of visual stimuli. In the present study, we investigated whether and how temporal attention interacts with spatial attention and feature-based attention in visual selection. To monitor the influence of the three different attention dimensions on perceptual processing, we measured event-related potentials (ERPs). Our participants performed a visual search task, in which a colored singleton was presented amongst homogenous distractors. We manipulated spatial and feature-based attention by requiring participants to respond only to target singletons in a particular color and at a to-be-attended spatial location. We manipulated temporal attention by means of an explicit temporal cue that announced either validly or invalidly the occurrence of the search display. We obtained early ERP effects of spatial attention and feature-based attention at the validly cued but not at the invalidly cued time point. Taken together, our results suggest that temporal attention boosts early effects of spatial and feature-based attention.

Blink Rate Patterns Provide a Reliable Measure of Individual Engagement with Scene Content

Eye-blinking has emerged as a promising means of measuring viewer engagement with visual content. This method capitalizes on the fact that although we remain largely unaware of our eye-blinking in everyday situations, eye-blinks are inhibited at precise moments in time so as to minimize the loss of visual information that occurs during a blink. Probabilistically, the more important the visual information is to the viewer, the more likely he or she will be to inhibit blinking. In the present study, viewer engagement was experimentally manipulated in order to: (1) replicate past studies suggesting that a group of viewers will blink less often when watching content that they perceive as more important or relevant; (2) test the reliability of the measure by investigating constraints on the timescale over which blink rate patterns can be used to accurately quantify viewer engagement; and (3) examine whether blink rate patterns can be used to quantify what an individual - as opposed to a group of viewers-perceives as engaging. Results demonstrate that blink rate patterns can be used to measure changes in individual and group engagement that unfold over relatively short (1 second) and long (60 second) timescales. However, for individuals with lower blink rates, blink rate patterns may provide less optimal measures when engagement shifts rapidly (at intervals of 1 second or less). Findings support the use of eye-blink measures in future studies investigating a person's subjective perception of how engaging a stimulus is.

Effect of a visual tracking intervention on attention and behavior of children with Attention Deficit Hyperactivity Disorder

Attention deficit hyperactivity disorder is characterized by several cognitive and behavioral problems such as inattention and impulsivity, abnormal control of eye movements and relocation, visual fixation and visuospatial perception. There is a link between core motor functions such as oculomotor function and cognition to the extent that the oculomotor system acts as a mediator between the motor and cognitive functions. Therefore, the effects of eye-tracking intervention were investigated on attention in these children. Thirty - nine boys with ADHD, 6 to 10 years of age were recruited and randomized to receive current occupational therapy (control group), or occupational therapy accompanied with eye-tracking exercises (experimental group). They were evaluated using the Conner's Parent Rating Scale, the Continuous Performance Task-2, and the Test of Visual-Motor Skills-Revised before and after the intervention. Significant improvements in the mean scores of cognitive problems (F=9/22), coping behavior (F=6.03) and hyperactivity (F=9.77) were detected in the posttest between the two groups (p<0.05). Furthermore, in the Continuous Performance Test scores, detectability (F=5.68), omission errors (F=17.89), commission errors (F=19.45), reaction time (F=8.95), variability (F=7.07), and preservation (F=6.33) showed significant differences between control and experimental groups (p<0.01). It appears that eye-tracking interventions designed based on the isolation of neck and eye movement might have an important role in improving cognitive function and coping behaviors in these children. It seems that these exercises could increase eye movement control; improve cognitive function and response inhibition.

Push-pull competition between bottom-up and top-down auditory attention to natural soundscapes

In everyday social environments, demands on attentional resources dynamically shift to balance our attention to targets of interest while alerting us to important objects in our surrounds. The current study uses electroencephalography to explore how the push-pull interaction between top-down and bottom-up attention manifests itself in dynamic auditory scenes. Using natural soundscapes as distractors while subjects attend to a controlled rhythmic sound sequence, we find that salient events in background scenes significantly suppress phase-locking and gamma responses to the attended sequence, countering enhancement effects observed for attended targets. In line with a hypothesis of limited attentional resources, the modulation of neural activity by bottom-up attention is graded by degree of salience of ambient events. The study also provides insights into the interplay between endogenous and exogenous attention during natural soundscapes, with both forms of attention engaging a common fronto-parietal network at different time lags.

Emergence of Visual Center-Periphery Spatial Organization in Deep Convolutional Neural Networks

Research at the intersection of computer vision and neuroscience has revealed hierarchical correspondence between layers of deep convolutional neural networks (DCNNs) and cascade of regions along human ventral visual cortex. Recently, studies have uncovered emergence of human interpretable concepts within DCNNs layers trained to identify visual objects and scenes. Here, we asked whether an artificial neural network (with convolutional structure) trained for visual categorization would demonstrate spatial correspondences with human brain regions showing central/peripheral biases. Using representational similarity analysis, we compared activations of convolutional layers of a DCNN trained for object and scene categorization with neural representations in human brain visual regions. Results reveal a brain-like topographical organization in the layers of the DCNN, such that activations of layer-units with central-bias were associated with brain regions with foveal tendencies (e.g. fusiform gyrus), and activations of layer-units with selectivity for image backgrounds were associated with cortical regions showing peripheral preference (e.g. parahippocampal cortex). The emergence of a categorical topographical correspondence between DCNNs and brain regions suggests these models are a good approximation of the perceptual representation generated by biological neural networks.

The lateral prefrontal cortex of primates encodes stimulus colors and their behavioral relevance during a match-to-sample task

The lateral prefrontal cortex of primates (lPFC) plays a central role in complex cognitive behavior, in decision-making as well as in guiding top-down attention. However, how and where in lPFC such behaviorally relevant signals are computed is poorly understood. We analyzed neural recordings from chronic microelectrode arrays implanted in lPFC region 8Av/45 of two rhesus macaques. The animals performed a feature match-to-sample task requiring them to match both motion and color information in a test stimulus. This task allowed to separate the encoding of stimulus motion and color from their current behavioral relevance on a trial-by-trial basis. We found that upcoming motor behavior can be robustly predicted from lPFC activity. In addition, we show that 8Av/45 encodes the color of a visual stimulus, regardless of its behavioral relevance. Most notably, whether a color matches the searched-for color can be decoded independent of a trial's motor outcome and while subjects detect unique feature conjunctions of color and motion. Thus, macaque area 8Av/45 computes, among other task-relevant information, the behavioral relevance of visual color features. Such a signal is most critical for both the selection of responses as well as the deployment of top-down modulatory signals, like feature-based attention.

The Value of Homework: Exposure to Odors in the Home Cage Enhances Odor-Discrimination Learning in Mice

Perceptual learning is an enhancement in discriminability of similar stimuli following experience with those stimuli. Here, we examined the efficacy of adding additional active training following a standard training session, compared with additional stimulus exposure in the absence of associated task performance. Mice were trained daily in an odor-discrimination task, and then, several hours later each day, received 1 of 3 different manipulations: 1) a second active-training session, 2) non-task-related odor exposure in the home cage, or 3) no second session. For home-cage exposure, odorants were presented in small tubes that mice could sniff and investigate for a similar period of time as in the active discrimination task each day. The results demonstrate that daily home-cage exposure was equivalent to active odor training in supporting improved odor discrimination. Daily home-cage exposure to odorants that did not match those used in the active task did not improve learning, yielding outcomes similar to those obtained with no second session. Piriform cortical local field potential recordings revealed that both sampling in the active learning task and investigation in the home cage evoked similar beta band oscillatory activity. Together the results suggest that odor-discrimination learning can be significantly enhanced by addition of odor exposure outside of the active training task, potentially because of the robust activity evoked in the olfactory system by both exposure paradigms. They further suggest that odorant exposure alone could enhance or maintain odor-discrimination abilities in conditions associated with olfactory impairment, such as aging or dementia.

Distinguishing between parallel and serial processing in visual attention from neurobiological data

Serial and parallel processing in visual search have been long debated in psychology, but the processing mechanism remains an open issue. Serial processing allows only one object at a time to be processed, whereas parallel processing assumes that various objects are processed simultaneously. Here, we present novel neural models for the two types of processing mechanisms based on analysis of simultaneously recorded spike trains using electrophysiological data from prefrontal cortex of rhesus monkeys while processing task-relevant visual displays. We combine mathematical models describing neuronal attention and point process models for spike trains. The same model can explain both serial and parallel processing by adopting different parameter regimes. We present statistical methods to distinguish between serial and parallel processing based on both maximum likelihood estimates and decoding the momentary focus of attention when two stimuli are presented simultaneously. Results show that both processing mechanisms are in play for the simultaneously recorded neurons, but neurons tend to follow parallel processing in the beginning after the onset of the stimulus pair, whereas they tend to serial processing later on.

Auditory Mismatch Negativity Under Predictive Coding Framework and Its Role in Psychotic Disorders

Traditional neuroscience sees sensory perception as a simple feedforward process. This view is challenged by the predictive coding model in recent years due to the robust evidence researchers had found on how our prediction could influence perception. In the first half of this article, we reviewed the concept of predictive brain and some empirical evidence of sensory prediction in visual and auditory processing. The predictive function along the auditory pathway was mainly studied by mismatch negativity (MMN)-a brain response to an unexpected disruption of regularity. We summarized a range of MMN paradigms and discussed how they could contribute to the theoretical development of the predictive coding neural network by the mechanism of adaptation and deviance detection. Such methodological and conceptual evolution sharpen MMN as a tool to better understand the structural and functional brain abnormality for neuropsychiatric disorder such as schizophrenia.

Neural correlates of goal-directed enhancement and suppression of visual stimuli in the absence of conscious perception

An observer's current goals can influence the processing of visual stimuli. Such influences can work to enhance goal-relevant stimuli and suppress goal-irrelevant stimuli. Here, we combined behavioral testing and electroencephalography (EEG) to examine whether such enhancement and suppression effects arise even when the stimuli are masked from awareness. We used a feature-based spatial cueing paradigm, in which participants searched four-item arrays for a target in a specific color. Immediately before the target array, a nonpredictive cue display was presented in which a cue matched or mismatched the searched-for target color, and appeared either at the target location (spatially valid) or another location (spatially invalid). Cue displays were masked using continuous flash suppression. The EEG data revealed that target-colored cues produced robust N2pc and N_T responses-both signatures of spatial orienting-and distractor-colored cues produced a robust P_D-a signature of suppression. Critically, the cueing effects occurred for both conscious and unconscious cues. The N2pc and N_T were larger in the aware versus unaware cue condition, but the P_D was roughly equivalent in magnitude across the two conditions. Our findings suggest that top-down control settings for task-relevant features elicit selective enhancement and suppression even in the absence of conscious perception. We conclude that conscious perception modulates selective enhancement of visual features, but suppression of those features is largely independent of awareness.

A point-process matched filter for event detection and decoding from population spike trains

OBJECTIVE

Information encoding in neurons can be described through their response fields. The spatial response field of a neuron is the region of space in which a sensory stimulus or a behavioral event causes that neuron to fire. Neurons can also exhibit temporal response fields (TRFs), which characterize a transient response to stimulus or behavioral event onsets. These neurons can thus be described by a spatio-temporal response field (STRF). The activity of neurons with STRFs can be well-described with point process models that characterize binary spike trains with an instantaneous firing rate that is a function of both time and space. However, developing decoders for point process models of neurons that exhibit TRFs is challenging because it requires prior knowledge of event onset times, which are unknown. Indeed, point process filters (PPF) to date have largely focused on decoding neuronal activity without considering TRFs. Also, neural classifiers have required data to be behavior- or stimulus-aligned, i.e. event times to be known, which is often not possible in real-world applications. Our objective in this work is to develop a viable decoder for neurons with STRFs when event times are unknown.

APPROACH

To enable decoding of neurons with STRFs, we develop a novel point-process matched filter (PPMF) that can detect events and estimate their onset times from population spike trains. We also devise a PPF for neurons with transient responses as characterized by STRFs. When neurons exhibit STRFs and event times are unknown, the PPMF can be combined with the PPF or with discrete classifiers for continuous and discrete brain state decoding, respectively.

MAIN RESULTS

We validate our algorithm on two datasets: simulated spikes from neurons that encode visual saliency in response to stimuli, and prefrontal spikes recorded in a monkey performing a delayed-saccade task. We show that the PPMF can estimate the stimulus times and saccade times accurately. Further, the PPMF combined with the PPF can decode visual saliency maps without knowing the stimulus times. Similarly, the PPMF combined with a point process classifier can decode the saccade direction without knowing the saccade times.

SIGNIFICANCE

These event detection and decoding algorithms can help develop neurotechnologies to decode cognitive states from neural responses that exhibit STRFs.

Eye Tracking in the Wild: Piloting a Real-Life Assessment Paradigm for Older Adults

Previous research showed associations between personality traits and eye movements of young adults in the laboratory. However, less is known about these associations in real life and in older age. Primarily, there seems to be no paradigm to assess eye movements of older adults in real life. The present feasibility study thus aimed to test grocery shopping as a real-life assessment paradigm with older adults. Additionally, possible links between personality traits and eye movements were explored. The sample consisted of 38 older individuals (M = 72.85 years). Participants did their grocery shopping in a supermarket while wearing an eye tracker. Three key feasibility issues were examined, that is (1) wearability of the eye tracker during grocery shopping, (2) recording, and (3) evaluation of eye movements in a real-life context. Our real-life assessment paradigm showed to be feasible to implement and acceptable to older adults. This feasibility study provides specific practical recommendations which may be useful for future studies that plan to innovatively expand the traditional methods repertoire of personality science and aging research by using eye tracking in real life.

Local and Global Influences of Visual Spatial Selection and Locomotion in Mouse Primary Visual Cortex

Sensory selection and movement locally and globally modulate neural responses in seemingly similar ways. For example, locomotion enhances visual responses in mouse primary visual cortex (V1), resembling the effects of spatial attention on primate visual cortical activity. However, interactions between these local and global mechanisms and the resulting effects on perceptual behavior remain largely unknown. Here, we describe a novel mouse visual spatial selection task in which animals either monitor one of two locations for a contrast change ("selective mice") or monitor both ("non-selective mice") and can run at will. Selective mice perform well only when their selected stimulus changes, giving rise to local electrophysiological changes in the corresponding hemisphere of V1 including decreased noise correlations and increased visual information. Non-selective mice perform well when either stimulus changes, giving rise to global changes across both hemispheres of V1. During locomotion, selective mice have worse behavioral performance, increased noise correlations in V1, and decreased visual information, while non-selective mice have decreased noise correlations in V1 but no change in performance or visual information. Our findings demonstrate that mice can locally or globally enhance visual information, but the interaction of the global effect of locomotion with local selection impairs behavioral performance. Moving forward, this mouse model will facilitate future studies of local and global sensory modulatory mechanisms and their effects on behavior.

The Role of Saliency in Learning First Words

In word learning, one key accomplishment is the reference, that is, the linking of a word to its referent. According to classical theories, the term reference captures a mental event: A person uses a word to mentally recall a concept of an entity (an object or event) in order to bring it into the mental focus of an interaction. The developmental literature proposes different approaches regarding how children accomplish this link. Although researchers agree that multiple processes (within and across phonological, lexical, and semantic areas) are responsible for word learning, recent research has highlighted the role of saliency and perception as crucial factors in the early phases of word learning. Generally speaking, whereas some approaches to solving the reference problem attribute a greater role to the referent's properties being salient, others emphasize the social context that is needed to select the appropriate referent. In this review, we aim to systematize terminology and propose that the reason why assessments of the impact of saliency on word learning are controversial is that definitions of the term saliency reveal different weightings of the importance that either perceptual or social stimuli have for the learning process. We propose that defining early word learning in terms of paying attention to salient stimuli is too narrow. Instead, we emphasize that a new link between a word and its referent will succeed if a stimulus is relevant for the child.