Predictive coding as a model of the V1 saliency map hypothesis

Cellular psychology: relating cognition to context-sensitive pyramidal cells

'Cellular psychology' is a new field of inquiry that studies dendritic mechanisms for adapting mental events to the current context, thus increasing their coherence, flexibility, effectiveness, and comprehensibility. Apical dendrites of neocortical pyramidal cells have a crucial role in cognition - those dendrites receive input from diverse sources, including feedback, and can amplify the cell's feedforward transmission if relevant in that context. Specialized subsets of inhibitory interneurons regulate this cooperative context-sensitive processing by increasing or decreasing amplification. Apical input has different effects on cellular output depending on whether we are awake, deeply asleep, or dreaming. Furthermore, wakeful thought and imagery may depend on apical input. High-resolution neuroimaging in humans supports and complements evidence on these cellular mechanisms from other mammals.

Attention-Based Variational Autoencoder Models for Human-Human Interaction Recognition via Generation

The remarkable human ability to predict others' intent during physical interactions develops at a very early age and is crucial for development. Intent prediction, defined as the simultaneous recognition and generation of human-human interactions, has many applications such as in assistive robotics, human-robot interaction, video and robotic surveillance, and autonomous driving. However, models for solving the problem are scarce. This paper proposes two attention-based agent models to predict the intent of interacting 3D skeletons by sampling them via a sequence of glimpses. The novelty of these agent models is that they are inherently multimodal, consisting of perceptual and proprioceptive pathways. The action (attention) is driven by the agent's generation error, and not by reinforcement. At each sampling instant, the agent completes the partially observed skeletal motion and infers the interaction class. It learns where and what to sample by minimizing the generation and classification errors. Extensive evaluation of our models is carried out on benchmark datasets and in comparison to a state-of-the-art model for intent prediction, which reveals that classification and generation accuracies of one of the proposed models are comparable to those of the state of the art even though our model contains fewer trainable parameters. The insights gained from our model designs can inform the development of efficient agents, the future of artificial intelligence (AI).

The cognitive penetrability of perception: A blocked debate and a tentative solution

Despite the extensive body of psychological findings suggesting that cognition influences perception, the debate between defenders and detractors of the cognitive penetrability of perception persists. While detractors demand more strictness in psychological experiments, proponents consider that empirical studies show that cognitive penetrability occurs. These considerations have led some theorists to propose that the debate has reached a dead end. The issue about where perception ends and cognition begins is, I argue, one of the reasons why the debate is cornered. Another reason is the inability of psychological studies to present uncontroversial interpretations of the results obtained. To dive into other kinds of empirical sources is, therefore, required to clarify the debate. In this paper, I explain where the debate is blocked, and suggest that neuroscientific evidence together with the predictive coding account, might decant the discussion on the side of the penetrability thesis.

Predicting Human Saccadic Scanpaths Based on Iterative Representation Learning

Visual attention is a dynamic process of scene exploration and information acquisition. However, existing research on attention modeling has concentrated on estimating static salient locations. In contrast, dynamic attributes presented by saccade have not been well explored in previous attention models. In this paper, we address the problem of saccadic scanpath prediction by introducing an iterative representation learning framework. Within the framework, saccade can be interpreted as an iterative process of predicting one fixation according to the current representation and updating the representation based on the gaze shift. In the predicting phase, we propose a Bayesian definition of saccade to combine the influence of perceptual residual and spatial location on the selection of fixations. In implementation, we compute the representation error of an autoencoder-based network to measure perceptual residuals of each area. Simultaneously, we integrate saccade amplitude and center-weighted mechanism to model the influence of spatial location. Based on estimating the influence of two parts, the final fixation is defined as the point with the largest posterior probability of gaze shift. In the updating phase, we update the representation pattern for the subsequent calculation by retraining the network with samples extracted around the current fixation. In the experiments, the proposed model can replicate the fundamental properties of psychophysics in visual search. In addition, it can achieve superior performance on several benchmark eye-tracking data sets.

Psychophysical evaluation of individual low-level feature influences on visual attention

In this study we provide the analysis of eye movement behavior elicited by low-level feature distinctiveness with a dataset of synthetically-generated image patterns. Design of visual stimuli was inspired by the ones used in previous psychophysical experiments, namely in free-viewing and visual searching tasks, to provide a total of 15 types of stimuli, divided according to the task and feature to be analyzed. Our interest is to analyze the influences of low-level feature contrast between a salient region and the rest of distractors, providing fixation localization characteristics and reaction time of landing inside the salient region. Eye-tracking data was collected from 34 participants during the viewing of a 230 images dataset. Results show that saliency is predominantly and distinctively influenced by: 1. feature type, 2. feature contrast, 3. temporality of fixations, 4. task difficulty and 5. center bias. This experimentation proposes a new psychophysical basis for saliency model evaluation using synthetic images.

Bottom-Up Visual Saliency Estimation With Deep Autoencoder-Based Sparse Reconstruction

Research on visual perception indicates that the human visual system is sensitive to center-surround (C-S) contrast in the bottom-up saliency-driven attention process. Different from the traditional contrast computation of feature difference, models based on reconstruction have emerged to estimate saliency by starting from original images themselves instead of seeking for certain ad hoc features. However, in the existing reconstruction-based methods, the reconstruction parameters of each area are calculated independently without taking their global correlation into account. In this paper, inspired by the powerful feature learning and data reconstruction ability of deep autoencoders, we construct a deep C-S inference network and train it with the data sampled randomly from the entire image to obtain a unified reconstruction pattern for the current image. In this way, global competition in sampling and learning processes can be integrated into the nonlocal reconstruction and saliency estimation of each pixel, which can achieve better detection results than the models with separate consideration on local and global rarity. Moreover, by learning from the current scene, the proposed model can achieve the feature extraction and interaction simultaneously in an adaptive way, which can form a better generalization ability to handle more types of stimuli. Experimental results show that in accordance with different inputs, the network can learn distinct basic features for saliency modeling in its code layer. Furthermore, in a comprehensive evaluation on several benchmark data sets, the proposed method can outperform the existing state-of-the-art algorithms.

Repetition priming results in sensitivity attenuation

Repetition priming refers to the change in the ability to perform a task on a stimulus as a consequence of a former encounter with that very same item. Usually, repetition results in faster and more accurate performance. In the present study, we used a contrast discrimination protocol to assess perceptual sensitivity and response bias of Gabor gratings that are either repeated (same orientation) or alternated (different orientation). We observed that contrast discrimination performance is worse, not better, for repeated than for alternated stimuli. In a second experiment, we varied the probability of stimulus repetition, thus testing whether the repetition effect is due to bottom-up or top-down factors. We found that it is top-down expectation that determines the effect. We discuss the implication of these findings for repetition priming and related phenomena as sensory attenuation. This article is part of a Special Issue entitled SI: Prediction and Attention.

Primary visual cortex represents the difference between past and present

The visual system is confronted with rapidly changing stimuli in everyday life. It is not well understood how information in such a stream of input is updated within the brain. We performed voltage-sensitive dye imaging across the primary visual cortex (V1) to capture responses to sequences of natural scene contours. We presented vertically and horizontally filtered natural images, and their superpositions, at 10 or 33 Hz. At low frequency, the encoding was found to represent not the currently presented images, but differences in orientation between consecutive images. This was in sharp contrast to more rapid sequences for which we found an ongoing representation of current input, consistent with earlier studies. Our finding that for slower image sequences, V1 does no longer report actual features but represents their relative difference in time counteracts the view that the first cortical processing stage must always transfer complete information. Instead, we show its capacities for change detection with a new emphasis on the role of automatic computation evolving in the 100-ms range, inevitably affecting information transmission further downstream.

Asymmetric effect of automatic deviant detection: The effect of familiarity in visual mismatch negativity

The visual mismatch negativity (vMMN) component is regarded as a prediction error signal elicited by events violating the sequential regularities of environmental stimulation. The aim of the study was to investigate the effect of familiarity on the vMMN. Stimuli were patterns comprised of familiar (N) or unfamiliar (И) letters. In a passive oddball paradigm, letters (N and И) were presented as either standard or deviant in separate conditions. VMMNs emerged in both conditions; peak latency of vMMN was shorter to the И deviant compared to the vMMN elicited by the N deviant. To test the orientation-specific effect of the oblique lines on the vMMN, we introduced a control experiment. In the control experiment, the patterns were constructed solely from oblique lines, identical to the oblique lines of the N and И stimuli. Contrary to the first experiment, there was no significant difference between the vMNNs elicited by the two orientations. Therefore, the differences in vMMNs to И and N deviants are not attributable to the physical difference between the И and N stimuli. Consequently, the vMMN is sensitive to the familiarity of the stimuli. This article is part of a Special Issue entitled SI: Prediction and Attention.

Activation of parallel fiber feedback by spatially diffuse stimuli reduces signal and noise correlations via independent mechanisms in a cerebellum-like structure

Correlations between the activities of neighboring neurons are observed ubiquitously across systems and species and are dynamically regulated by several factors such as the stimulus' spatiotemporal extent as well as by the brain's internal state. Using the electrosensory system of gymnotiform weakly electric fish, we recorded the activities of pyramidal cell pairs within the electrosensory lateral line lobe (ELL) under spatially localized and diffuse stimulation. We found that both signal and noise correlations were markedly reduced (>40%) under the latter stimulation. Through a network model incorporating key anatomical features of the ELL, we reveal how activation of diffuse parallel fiber feedback from granule cells by spatially diffuse stimulation can explain both the reduction in signal as well as the reduction in noise correlations seen experimentally through independent mechanisms. First, we show that burst-timing dependent plasticity, which leads to a negative image of the stimulus and thereby reduces single neuron responses, decreases signal but not noise correlations. Second, we show trial-to-trial variability in the responses of single granule cells to sensory input reduces noise but not signal correlations. Thus, our model predicts that the same feedback pathway can simultaneously reduce both signal and noise correlations through independent mechanisms. To test this prediction experimentally, we pharmacologically inactivated parallel fiber feedback onto ELL pyramidal cells. In agreement with modeling predictions, we found that inactivation increased both signal and noise correlations but that there was no significant relationship between magnitude of the increase in signal correlations and the magnitude of the increase in noise correlations. The mechanisms reported in this study are expected to be generally applicable to the cerebellum as well as other cerebellum-like structures. We further discuss the implications of such decorrelation on the neural coding strategies used by the electrosensory and by other systems to process natural stimuli.

Correlated activity is observed ubiquitously in the CNS but how activation of specific neural circuits affects correlated activity under behaviorally relevant contexts is poorly understood. Here, through a combination of electrophysiology, pharmacology, and mathematical modeling, we show that activation of the same parallel fiber feedback pathway leads to simultaneous reductions in both signal and noise correlations via independent mechanisms. Specifically, we show that feedback in the form of a negative image of the stimulus is necessary in order to attenuate signal but not noise correlations. Moreover, we show that trial-to-trial variability in the spiking responses of neurons providing this feedback is necessary to attenuate noise but not signal correlations. Our model thus predicts that activation of the same feedback pathway can simultaneously reduce both signal and noise correlations through independent mechanisms. In agreement with modeling prediction, pharmacological inactivation led to a strong increase in both signal and noise correlations but the magnitude of the change in signal correlation was not related to the magnitude of the change in noise correlations. Our proposed mechanism for simultaneous control of both signal and noise correlations is generic and is thus likely to be applicable to the cerebellum and to other cerebellar-like structures.

Asymmetry of automatic change detection shown by the visual mismatch negativity: an additional feature is identified faster than missing features

In two experiments, we demonstrated that an asymmetric effect of the brain electric activity that is elicited by nonattended visual stimuli is similar to the one found in responses observed in the performance of visual search tasks. The automatic detection of violated sequential regularities was investigated by measuring the visual mismatch negativity (vMMN) component of event-related brain potentials (ERPs). In Experiment 1, within a sequence of stimulus displays with O characters, infrequently presented Q characters elicited an earlier vMMN than did infrequent O characters within a sequence of Q characters. In Experiment 2, similar asymmetric results emerged if only 16% of the characters were different within an infrequent display. In both experiments, these stimuli were irrelevant; during the stimulus sequences, participants performed a demanding videogame. We suggest that the underlying match/mismatch and decision processes are similar in the vMMN and in the attention-related visual search paradigm, at least in the case of the stimuli in the present experiments.

Classification using sparse representations: a biologically plausible approach

Representing signals as linear combinations of basis vectors sparsely selected from an overcomplete dictionary has proven to be advantageous for many applications in pattern recognition, machine learning, signal processing, and computer vision. While this approach was originally inspired by insights into cortical information processing, biologically plausible approaches have been limited to exploring the functionality of early sensory processing in the brain, while more practical applications have employed non-biologically plausible sparse coding algorithms. Here, a biologically plausible algorithm is proposed that can be applied to practical problems. This algorithm is evaluated using standard benchmark tasks in the domain of pattern classification, and its performance is compared to a wide range of alternative algorithms that are widely used in signal and image processing. The results show that for the classification tasks performed here, the proposed method is competitive with the best of the alternative algorithms that have been evaluated. This demonstrates that classification using sparse representations can be performed in a neurally plausible manner, and hence, that this mechanism of classification might be exploited by the brain.

Attention and predictions: control of spatial attention beyond the endogenous-exogenous dichotomy

The mechanisms of attention control have been extensively studied with a variety of methodologies in animals and in humans. Human studies using non-invasive imaging techniques highlighted a remarkable difference between the pattern of responses in dorsal fronto-parietal regions vs. ventral fronto-parietal (vFP) regions, primarily lateralized to the right hemisphere. Initially, this distinction at the neuro-physiological level has been related to the distinction between cognitive processes associated with strategic/endogenous vs. stimulus-driven/exogenous of attention control. Nonetheless, quite soon it has become evident that, in almost any situation, attention control entails a complex combination of factors related to both the current sensory input and endogenous aspects associated with the experimental context. Here, we review several of these aspects first discussing the joint contribution of endogenous and stimulus-driven factors during spatial orienting in complex environments and, then, turning to the role of expectations and predictions in spatial re-orienting. We emphasize that strategic factors play a pivotal role for the activation of the ventral system during stimulus-driven control, and that the dorsal system makes use of stimulus-driven signals for top-down control. We conclude that both the dorsal and the vFP networks integrate endogenous and exogenous signals during spatial attention control and that future investigations should manipulate both these factors concurrently, so as to reveal to full extent of these interactions.

Visual nonclassical receptive field effects emerge from sparse coding in a dynamical system

Extensive electrophysiology studies have shown that many V1 simple cells have nonlinear response properties to stimuli within their classical receptive field (CRF) and receive contextual influence from stimuli outside the CRF modulating the cell's response. Models seeking to explain these non-classical receptive field (nCRF) effects in terms of circuit mechanisms, input-output descriptions, or individual visual tasks provide limited insight into the functional significance of these response properties, because they do not connect the full range of nCRF effects to optimal sensory coding strategies. The (population) sparse coding hypothesis conjectures an optimal sensory coding approach where a neural population uses as few active units as possible to represent a stimulus. We demonstrate that a wide variety of nCRF effects are emergent properties of a single sparse coding model implemented in a neurally plausible network structure (requiring no parameter tuning to produce different effects). Specifically, we replicate a wide variety of nCRF electrophysiology experiments (e.g., end-stopping, surround suppression, contrast invariance of orientation tuning, cross-orientation suppression, etc.) on a dynamical system implementing sparse coding, showing that this model produces individual units that reproduce the canonical nCRF effects. Furthermore, when the population diversity of an nCRF effect has also been reported in the literature, we show that this model produces many of the same population characteristics. These results show that the sparse coding hypothesis, when coupled with a biophysically plausible implementation, can provide a unified high-level functional interpretation to many response properties that have generally been viewed through distinct mechanistic or phenomenological models.

Simple cells in the primary visual cortex (V1) demonstrate many response properties that are either nonlinear or involve response modulations (i.e., stimuli that do not cause a response in isolation alter the cell's response to other stimuli). These non-classical receptive field (nCRF) effects are generally modeled individually and their collective role in biological vision is not well understood. Previous work has shown that classical receptive field (CRF) properties of V1 cells (i.e., the spatial structure of the visual field responsive to stimuli) could be explained by the sparse coding hypothesis, which is an optimal coding model that conjectures a neural population should use the fewest number of cells simultaneously to represent each stimulus. In this paper, we have performed extensive simulated physiology experiments to show that many nCRF response properties are simply emergent effects of a dynamical system implementing this same sparse coding model. These results suggest that rather than representing disparate information processing operations themselves, these nCRF effects could be consequences of an optimal sensory coding strategy that attempts to represent each stimulus most efficiently. This interpretation provides a potentially unifying high-level functional interpretation to many response properties that have generally been viewed through distinct models.