Neuronal Mechanisms of Visual Attention

Spike-based dynamic computing with asynchronous sensing-computing neuromorphic chip

By mimicking the neurons and synapses of the human brain and employing spiking neural networks on neuromorphic chips, neuromorphic computing offers a promising energy-efficient machine intelligence. How to borrow high-level brain dynamic mechanisms to help neuromorphic computing achieve energy advantages is a fundamental issue. This work presents an application-oriented algorithm-software-hardware co-designed neuromorphic system for this issue. First, we design and fabricate an asynchronous chip called "Speck", a sensing-computing neuromorphic system on chip. With the low processor resting power of 0.42mW, Speck can satisfy the hardware requirements of dynamic computing: no-input consumes no energy. Second, we uncover the "dynamic imbalance" in spiking neural networks and develop an attention-based framework for achieving the algorithmic requirements of dynamic computing: varied inputs consume energy with large variance. Together, we demonstrate a neuromorphic system with real-time power as low as 0.70mW. This work exhibits the promising potentials of neuromorphic computing with its asynchronous event-driven, sparse, and dynamic nature.

Distinct feedforward and feedback pathways for cell-type specific attention effects

Selective attention is thought to depend on enhanced firing activity in extrastriate areas. Theories suggest that this enhancement depends on selective inter-areal communication via gamma (30-80 Hz) phase-locking. To test this, we simultaneously recorded from different cell types and cortical layers of macaque V1 and V4. We find that while V1-V4 gamma phase-locking between local field potentials increases with attention, the V1 gamma rhythm does not engage V4 excitatory-neurons, but only fast-spiking interneurons in L4 of V4. By contrast, attention enhances V4 spike-rates in both excitatory and inhibitory cells, most strongly in L2/3. The rate increase in L2/3 of V4 precedes V1 in time. These findings suggest enhanced signal transmission with attention does not depend on inter-areal gamma phase-locking and show that the endogenous gamma rhythm has cell-type- and layer-specific effects on downstream target areas. Similar findings were made in the mouse visual system, based on opto-tagging of identified interneurons.

Spatial attention alters visual cortical representation during target anticipation

Attention enables us to efficiently and flexibly interact with the environment by prioritizing some image features in preparation for responding to a stimulus. Using a concurrent psychophysics- fMRI experiment, we investigated how covert spatial attention affects responses in human visual cortex prior to target onset, and how it affects subsequent behavioral performance. Performance improved at cued locations and worsened at uncued locations, relative to distributed attention, demonstrating a selective tradeoff in processing. Pre-target BOLD responses in cortical visual field maps changed in two ways: First, there was a stimulus-independent baseline shift, positive in map locations near the cued location and negative elsewhere, paralleling the behavioral results. Second, population receptive field centers shifted toward the attended location. Both effects increased in higher visual areas. Together, the results show that spatial attention has large effects on visual cortex prior to target appearance, altering neural response properties throughout and across multiple visual field maps.

Auditory change detection and visual selective attention: association between MMN and N2pc

While the auditory and visual systems each provide distinct information to our brain, they also work together to process and prioritize input to address ever-changing conditions. Previous studies highlighted the trade-off between auditory change detection and visual selective attention; however, the relationship between them is still unclear. Here, we recorded electroencephalography signals from 106 healthy adults in three experiments. Our findings revealed a positive correlation at the population level between the amplitudes of event-related potential indices associated with auditory change detection (mismatch negativity) and visual selective attention (posterior contralateral N2) when elicited in separate tasks. This correlation persisted even when participants performed a visual task while disregarding simultaneous auditory stimuli. Interestingly, as visual attention demand increased, participants whose posterior contralateral N2 amplitude increased the most exhibited the largest reduction in mismatch negativity, suggesting a within-subject trade-off between the two processes. Taken together, our results suggest an intimate relationship and potential shared mechanism between auditory change detection and visual selective attention. We liken this to a total capacity limit that varies between individuals, which could drive correlated individual differences in auditory change detection and visual selective attention, and also within-subject competition between the two, with task-based modulation of visual attention causing within-participant decrease in auditory change detection sensitivity.

Are neuronal mechanisms of attention universal across human sensory and motor brain maps?

One's experience of shifting attention from the color to the smell to the act of picking a flower seems like a unitary process applied, at will, to one modality after another. Yet, the unique and separable experiences of sight versus smell versus movement might suggest that the neural mechanisms of attention have been separately optimized to employ each modality to its greatest advantage. Moreover, addressing the issue of universality can be particularly difficult due to a paucity of existing cross-modal comparisons and a dearth of neurophysiological methods that can be applied equally well across disparate modalities. Here we outline some of the conceptual and methodological issues related to this problem and present an instructive example of an experimental approach that can be applied widely throughout the human brain to permit detailed, quantitative comparison of attentional mechanisms across modalities. The ultimate goal is to spur efforts across disciplines to provide a large and varied database of empirical observations that will either support the notion of a universal neural substrate for attention or more clearly identify the degree to which attentional mechanisms are specialized for each modality.

We know what attention is!

Attention is one of the most thoroughly investigated psychological phenomena, yet skepticism about attention is widespread: we do not know what it is, it is too many things, there is no such thing. The deficiencies highlighted are not about experimental work but the adequacy of the scientific theory of attention. Combining common scientific claims about attention into a single theory leads to internal inconsistency. This paper demonstrates that a specific functional conception of attention is incorporated into the tasks used in standard experimental paradigms. In accepting these paradigms as valid probes of attention, we commit to this common conception. The conception unifies work at multiple levels of analysis into a coherent scientific explanation of attention. Thus, we all know what attention is.

Vision and microsaccades: Time to pay attention!

Visual perception of exceedingly small and highly detailed spatial regions depends on coordinated patterns of small shifts of the line of sight ('microsaccades') aided by pre-saccadic shifts of spatial attention directed precisely to the intended target of the saccade.

Disinhibition by VIP interneurons is orthogonal to cross-modal attentional modulation in primary visual cortex

Attentional modulation of sensory processing is a key feature of cognition; however, its neural circuit basis is poorly understood. A candidate mechanism is the disinhibition of pyramidal cells through vasoactive intestinal peptide (VIP) and somatostatin (SOM)-positive interneurons. However, the interaction of attentional modulation and VIP-SOM disinhibition has never been directly tested. We used all-optical methods to bi-directionally manipulate VIP interneuron activity as mice performed a cross-modal attention-switching task. We measured the activities of VIP, SOM, and parvalbumin (PV)-positive interneurons and pyramidal neurons identified in the same tissue and found that although activity in all cell classes was modulated by both attention and VIP manipulation, their effects were orthogonal. Attention and VIP-SOM disinhibition relied on distinct patterns of changes in activity and reorganization of interactions between inhibitory and excitatory cells. Circuit modeling revealed a precise network architecture consistent with multiplexing strong yet non-interacting modulations in the same neural population.

Emergent human-like covert attention in feedforward convolutional neural networks

Covert attention allows the selection of locations or features of the visual scene without moving the eyes. Cues and contexts predictive of a target's location orient covert attention and improve perceptual performance. The performance benefits are widely attributed to theories of covert attention as a limited resource, zoom, spotlight, or weighting of visual information. However, such concepts are difficult to map to neuronal populations. We show that a feedforward convolutional neural network (CNN) trained on images to optimize target detection accuracy and with no explicit incorporation of an attention mechanism, a limited resource, or feedback connections learns to utilize cues and contexts in the three most prominent covert attention tasks (Posner cueing, set size effects in search, and contextual cueing) and predicts the cue/context influences on human accuracy. The CNN's cueing/context effects generalize across network training schemes, to peripheral and central pre-cues, discrimination tasks, and reaction time measures, and critically do not vary with reductions in network resources (size). The CNN shows comparable cueing/context effects to a model that optimally uses image information to make decisions (Bayesian ideal observer) but generalizes these effects to cue instances unseen during training. Together, the findings suggest that human-like behavioral signatures of covert attention in the three landmark paradigms might be an emergent property of task accuracy optimization in neuronal populations without positing limited attentional resources. The findings might explain recent behavioral results showing cueing and context effects across a variety of simple organisms with no neocortex, from archerfish to fruit flies.

Dissociable components of attention exhibit distinct neuronal signatures in primate visual cortex

Attention can be deployed in multiple forms and facilitates behavior by influencing perceptual sensitivity and choice bias. Attention is also associated with a myriad of changes in sensory neural activity. Yet, the relationship between the behavioral components of attention and the accompanying changes in neural activity remains largely unresolved. We examined this relationship by quantifying sensitivity and bias in monkeys performing a task that dissociated eye movement responses from the focus of covert attention. Unexpectedly, bias, not sensitivity, increased at the focus of covert attention, whereas sensitivity increased at the location of planned eye movements. Furthermore, neuronal activity within visual area V4 varied robustly with bias, but not sensitivity, at the focus of covert attention. In contrast, correlated variability between neuronal pairs was lowest at the location of planned eye movements, and varied with sensitivity, but not bias. Thus, dissociable behavioral components of attention exhibit distinct neuronal signatures within the visual cortex.

Resting-state functional magnetic resonance imaging indices are related to electrophysiological dysfunction in degenerative cervical myelopathy

The age-related degenerative pathologies of the cervical spinal column that comprise degenerative cervical myelopathy (DCM) cause myelopathy due spinal cord compression. Functional neurological assessment of DCM can potentially reveal the severity and pathological mechanism of DCM. However, functional assessment by conventional MRI remains difficult. This study used resting-state functional MRI (rs-fMRI) to investigate the relationship between functional connectivity (FC) strength and neurophysiological indices and examined the feasibility of functional assessment by FC for DCM. Preoperatively, 34 patients with DCM underwent rs-fMRI scans. Preoperative central motor conduction time (CMCT) reflecting motor functional disability and intraoperative somatosensory evoked potentials (SEP) reflecting sensory functional disability were recorded as electrophysiological indices of severity of the cervical spinal cord impairment. We performed seed-to-voxel FC analysis and correlation analyses between FC strength and the two electrophysiological indices. We found that FC strength between the primary motor cortex and the precuneus correlated significantly positively with CMCT, and that between the lateral part of the sensorimotor cortex and the lateral occipital cortex also showed a significantly positive correlation with SEP amplitudes. These results suggest that we can evaluate neurological and electrophysiological severity in patients with DCM by analyzing FC strengths between certain brain regions.

Pathway-selective optogenetics reveals the functional anatomy of top-down attentional modulation in the macaque visual cortex

Spatial attention represents a powerful top-down influence on sensory responses in primate visual cortical areas. The frontal eye field (FEF) has emerged as a key candidate area for the source of this modulation. However, it is unclear whether the FEF exerts its effects via its direct axonal projections to visual areas or indirectly through other brain areas and whether the FEF affects both the enhancement of attended and the suppression of unattended sensory responses. We used pathway-selective optogenetics in rhesus macaques performing a spatial attention task to inhibit the direct input from the FEF to area MT, an area along the dorsal visual pathway specialized for the processing of visual motion information. Our results show that the optogenetic inhibition of the FEF input specifically reduces attentional modulation in MT by about a third without affecting the neurons' sensory response component. We find that the direct FEF-to-MT pathway contributes to both the enhanced processing of target stimuli and the suppression of distractors. The FEF, thus, selectively modulates firing rates in visual area MT, and it does so via its direct axonal projections.

Trying harder: how cognitive effort sculpts neural representations during working memory

The neural mechanisms by which motivational factors influence cognition remain unknown. Using fMRI, we tested how cognitive effort impacts working memory (WM). Participants were precued whether WM difficulty would be hard or easy. Hard trials demanded more effort as a later decision required finer mnemonic precision. Behaviorally, pupil size was larger and response times were slower on hard trials suggesting our manipulation of effort succeeded. Neurally, we observed robust persistent activity in prefrontal cortex, especially during hard trials. We found strong decoding of location in visual cortex, where accuracy was higher on hard trials. Connecting these across-region effects, we found that the amplitude of delay period activity in frontal cortex predicted decoded accuracy in visual cortex on a trial-wise basis. We conclude that the gain of persistent activity in frontal cortex may be the source of effort-related feedback signals that improve the quality of WM representations stored in visual cortex.

Activity in primate visual cortex is minimally driven by spontaneous movements

Organisms process sensory information in the context of their own moving bodies, an idea referred to as embodiment. This idea is important for developmental neuroscience, robotics and systems neuroscience. The mechanisms supporting embodiment are unknown, but a manifestation could be the observation in mice of brain-wide neuromodulation, including in the primary visual cortex, driven by task-irrelevant spontaneous body movements. We tested this hypothesis in macaque monkeys (Macaca mulatta), a primate model for human vision, by simultaneously recording visual cortex activity and facial and body movements. We also sought a direct comparison using an analogous approach to those used in mouse studies. Here we found that activity in the primate visual cortex (V1, V2 and V3/V3A) was associated with the animals' own movements, but this modulation was largely explained by the impact of the movements on the retinal image, that is, by changes in visual input. These results indicate that visual cortex in primates is minimally driven by spontaneous movements and may reflect species-specific sensorimotor strategies.

Computational assessment of visual coding across mouse brain areas and behavioural states

Introduction

Our brain is bombarded by a diverse range of visual stimuli, which are converted into corresponding neuronal responses and processed throughout the visual system. The neural activity patterns that result from these external stimuli vary depending on the object or scene being observed, but they also change as a result of internal or behavioural states. This raises the question of to what extent it is possible to predict the presented visual stimuli from neural activity across behavioural states, and how this varies in different brain regions.

Methods

To address this question, we assessed the computational capacity of decoders to extract visual information in awake behaving mice, by analysing publicly available standardised datasets from the Allen Brain Institute. We evaluated how natural movie frames can be distinguished based on the activity of units recorded in distinct brain regions and under different behavioural states. This analysis revealed the spectrum of visual information present in different brain regions in response to binary and multiclass classification tasks.

Results

Visual cortical areas showed highest classification accuracies, followed by thalamic and midbrain regions, with hippocampal regions showing close to chance accuracy. In addition, we found that behavioural variability led to a decrease in decoding accuracy, whereby large behavioural changes between train and test sessions reduced the classification performance of the decoders. A generalised linear model analysis suggested that this deterioration in classification might be due to an independent modulation of neural activity by stimulus and behaviour. Finally, we reconstructed the natural movie frames from optimal linear classifiers, and observed a strong similarity between reconstructed and actual movie frames. However, the similarity was significantly higher when the decoders were trained and tested on sessions with similar behavioural states.

Conclusion

Our analysis provides a systematic assessment of visual coding in the mouse brain, and sheds light on the spectrum of visual information present across brain areas and behavioural states.

Human-like scene interpretation by a guided counterstream processing

In modeling vision, there has been a remarkable progress in recognizing a range of scene components, but the problem of analyzing full scenes, an ultimate goal of visual perception, is still largely open. To deal with complete scenes, recent work focused on the training of models for extracting the full graph-like structure of a scene. In contrast with scene graphs, humans' scene perception focuses on selected structures in the scene, starting with a limited interpretation and evolving sequentially in a goal-directed manner [G. L. Malcolm, I. I. A. Groen, C. I. Baker, Trends. Cogn. Sci. 20, 843-856 (2016)]. Guidance is crucial throughout scene interpretation since the extraction of full scene representation is often infeasible. Here, we present a model that performs human-like guided scene interpretation, using an iterative bottom-up, top-down processing, in a "counterstream" structure motivated by cortical circuitry. The process proceeds by the sequential application of top-down instructions that guide the interpretation process. The results show how scene structures of interest to the viewer are extracted by an automatically selected sequence of top-down instructions. The model shows two further benefits. One is an inherent capability to deal well with the problem of combinatorial generalization-generalizing broadly to unseen scene configurations, which is limited in current network models [B. Lake, M. Baroni, 35th International Conference on Machine Learning, ICML 2018 (2018)]. The second is the ability to combine visual with nonvisual information at each cycle of the interpretation process, which is a key aspect for modeling human perception as well as advancing AI vision systems.

Context-dependent attentional spotlight in pulvinar-V1 interaction

Spatial attention is often described as a mental spotlight that enhances information processing at the attended location. Using fMRI, we investigated background connectivity between the pulvinar and V1 in relation to focused versus diffused attention allocation, in weak and strong crowding contexts. Our findings revealed that focused attention led to enhanced correlations between the pulvinar and V1. Notably, this modulation was initiated by the pulvinar, and the strength of the modulation was dependent on the saliency of the target. These findings suggest that the pulvinar initiates information reweighting to V1, which underlies attentional selection in cluttered scenes.

Fundamental units of numerosity estimation

Humans can approximately enumerate a large number of objects at a single glance. While several mechanisms have been proposed to account for this ability, the fundamental units over which they operate remain unclear. Previous studies have argued that estimation mechanisms act only on topologically distinct units or on units formed by spatial grouping cues such as proximity and connectivity, but not on units grouped by similarity. Over four experiments, we tested this claim by systematically assessing and demonstrating that similarity grouping leads to underestimation, just as spatial grouping does. Ungrouped objects with the same low-level properties as grouped objects did not cause underestimation. Further, the underestimation caused by spatial and similarity grouping was additive, suggesting that these grouping processes operate independently. These findings argue against the proposal that estimation mechanisms operate solely on topological units. Instead, we conclude that estimation processes act on representations constructed after Gestalt grouping principles, whether similarity based or spatial, have organised incoming visual input.

The Architecture of Object-Based Attention

The allocation of attention to objects raises several intriguing questions: What are objects, how does attention access them, what anatomical regions are involved? Here, we review recent progress in the field to determine the mechanisms underlying object-based attention. First, findings from unconscious priming and cueing suggest that the preattentive targets of object-based attention can be fully developed object representations that have reached the level of identity. Next, the control of object-based attention appears to come from ventral visual areas specialized in object analysis that project downward to early visual areas. How feedback from object areas can accurately target the object's specific locations and features is unknown but recent work in autoencoding has made this plausible. Finally, we suggest that the three classic modes of attention may not be as independent as is commonly considered, and instead could all rely on object-based attention. Specifically, studies show that attention can be allocated to the separated members of a group-without affecting the space between them-matching the defining property of feature-based attention. At the same time, object-based attention directed to a single small item has the properties of space-based attention. We outline the architecture of object-based attention, the novel predictions it brings, and discuss how it works in parallel with other attention pathways.

Common population codes produce extremely nonlinear neural manifolds

Populations of neurons represent sensory, motor, and cognitive variables via patterns of activity distributed across the population. The size of the population used to encode a variable is typically much greater than the dimension of the variable itself, and thus, the corresponding neural population activity occupies lower-dimensional subsets of the full set of possible activity states. Given population activity data with such lower-dimensional structure, a fundamental question asks how close the low-dimensional data lie to a linear subspace. The linearity or nonlinearity of the low-dimensional structure reflects important computational features of the encoding, such as robustness and generalizability. Moreover, identifying such linear structure underlies common data analysis methods such as Principal Component Analysis (PCA). Here, we show that for data drawn from many common population codes the resulting point clouds and manifolds are exceedingly nonlinear, with the dimension of the best-fitting linear subspace growing at least exponentially with the true dimension of the data. Consequently, linear methods like PCA fail dramatically at identifying the true underlying structure, even in the limit of arbitrarily many data points and no noise.

Rodent attention: Probing the mouse mind with reverse correlation

A novel approach to studying attention in mice reveals processes similar to those in humans and lays out an efficient way to explore its neuronal correlates in a genetically tractable animal model.

Visual attention to features and space in mice using reverse correlation

Visual attention allows the brain to evoke behaviors based on the most important visual features. Mouse models offer immense potential to gain a circuit-level understanding of this phenomenon, yet how mice distribute attention across features and locations is not well understood. Here, we describe a new approach to address this limitation by training mice to detect weak vertical bars in a background of dynamic noise while spatial cues manipulate their attention. By adapting a reverse-correlation method from human studies, we linked behavioral decisions to stimulus features and locations. We show that mice deployed attention to a small rostral region of the visual field. Within this region, mice attended to multiple features (orientation, spatial frequency, contrast) that indicated the presence of weak vertical bars. This attentional tuning grew with training, multiplicatively scaled behavioral sensitivity, approached that of an ideal observer, and resembled the effects of attention in humans. Taken together, we demonstrate that mice can simultaneously attend to multiple features and locations of a visual stimulus.

Dissociable roles of human frontal eye fields and early visual cortex in presaccadic attention

Shortly before saccadic eye movements, visual sensitivity at the saccade target is enhanced, at the expense of sensitivity elsewhere. Some behavioral and neural correlates of this presaccadic shift of attention resemble those of covert attention, deployed during fixation. Microstimulation in non-human primates has shown that presaccadic attention modulates perception via feedback from oculomotor to visual areas. This mechanism also seems plausible in humans, as both oculomotor and visual areas are active during saccade planning. We investigated this hypothesis by applying TMS to frontal or visual areas during saccade preparation. By simultaneously measuring perceptual performance, we show their causal and differential roles in contralateral presaccadic attention effects: Whereas rFEF+ stimulation enhanced sensitivity opposite the saccade target throughout saccade preparation, V1/V2 stimulation reduced sensitivity at the saccade target only shortly before saccade onset. These findings are consistent with presaccadic attention modulating perception through cortico-cortical feedback and further dissociate presaccadic and covert attention.

A dynamic 1/f noise protocol to assess visual attention without biasing perceptual processing

Psychophysical paradigms measure visual attention via localized test items to which observers must react or whose features have to be discriminated. These items, however, potentially interfere with the intended measurement, as they bias observers' spatial and temporal attention to their location and presentation time. Furthermore, visual sensitivity for conventional test items naturally decreases with retinal eccentricity, which prevents direct comparison of central and peripheral attention assessments. We developed a stimulus that overcomes these limitations. A brief oriented discrimination signal is seamlessly embedded into a continuously changing 1/f noise field, such that observers cannot anticipate potential test locations or times. Using our new protocol, we demonstrate that local orientation discrimination accuracy for 1/f filtered signals is largely independent of retinal eccentricity. Moreover, we show that items present in the visual field indeed shape the distribution of visual attention, suggesting that classical studies investigating the spatiotemporal dynamics of visual attention via localized test items may have obtained a biased measure. We recommend our protocol as an efficient method to evaluate the behavioral and neurophysiological correlates of attentional orienting across space and time.

Topological insights into the neural basis of flexible behavior

It is widely accepted that there is an inextricable link between neural computations, biological mechanisms, and behavior, but it is challenging to simultaneously relate all three. Here, we show that topological data analysis (TDA) provides an important bridge between these approaches to studying how brains mediate behavior. We demonstrate that cognitive processes change the topological description of the shared activity of populations of visual neurons. These topological changes constrain and distinguish between competing mechanistic models, are connected to subjects' performance on a visual change detection task, and, via a link with network control theory, reveal a tradeoff between improving sensitivity to subtle visual stimulus changes and increasing the chance that the subject will stray off task. These connections provide a blueprint for using TDA to uncover the biological and computational mechanisms by which cognition affects behavior in health and disease.

Tasks and their role in visual neuroscience

Vision is widely used as a model system to gain insights into how sensory inputs are processed and interpreted by the brain. Historically, careful quantification and control of visual stimuli have served as the backbone of visual neuroscience. There has been less emphasis, however, on how an observer's task influences the processing of sensory inputs. Motivated by diverse observations of task-dependent activity in the visual system, we propose a framework for thinking about tasks, their role in sensory processing, and how we might formally incorporate tasks into our models of vision.

Polar angle asymmetries in visual perception and neural architecture

Human visual performance changes with visual field location. It is best at the center of gaze and declines with eccentricity, and also varies markedly with polar angle. These perceptual polar angle asymmetries are linked to asymmetries in the organization of the visual system. We review and integrate research quantifying how performance changes with visual field location and how this relates to neural organization at multiple stages of the visual system. We first briefly review how performance varies with eccentricity and the neural foundations of this effect. We then focus on perceptual polar angle asymmetries and their neural foundations. Characterizing perceptual and neural variations across and around the visual field contributes to our understanding of how the brain translates visual signals into neural representations which form the basis of visual perception.

The visible gorilla: Unexpected fast-not physically salient-Objects are noticeable

It is widely believed that observers can fail to notice clearly visible unattended objects, even if they are moving. Here, we created parametric tasks to test this belief and report the results of three high-powered experiments (total n = 4,493) indicating that this effect is strongly modulated by the speed of the unattended object. Specifically, fast-but not slow-objects are readily noticeable, whether they are attended or not. These results suggest that fast motion serves as a potent exogenous cue that overrides task-focused attention, showing that fast speeds, not long exposure duration or physical salience, strongly diminish inattentional blindness effects.

Multimodal deep neural decoding reveals highly resolved spatiotemporal profile of visual object representation in humans

Perception and categorization of objects in a visual scene are essential to grasp the surrounding situation. Recently, neural decoding schemes, such as machine learning in functional magnetic resonance imaging (fMRI), has been employed to elucidate the underlying neural mechanisms. However, it remains unclear as to how spatially distributed brain regions temporally represent visual object categories and sub-categories. One promising strategy to address this issue is neural decoding with concurrently obtained neural response data of high spatial and temporal resolution. In this study, we explored the spatial and temporal organization of visual object representations using concurrent fMRI and electroencephalography (EEG), combined with neural decoding using deep neural networks (DNNs). We hypothesized that neural decoding by multimodal neural data with DNN would show high classification performance in visual object categorization (faces or non-face objects) and sub-categorization within faces and objects. Visualization of the fMRI DNN was more sensitive than that in the univariate approach and revealed that visual categorization occurred in brain-wide regions. Interestingly, the EEG DNN valued the earlier phase of neural responses for categorization and the later phase of neural responses for sub-categorization. Combination of the two DNNs improved the classification performance for both categorization and sub-categorization compared with fMRI DNN or EEG DNN alone. These deep learning-based results demonstrate a categorization principle in which visual objects are represented in a spatially organized and coarse-to-fine manner, and provide strong evidence of the ability of multimodal deep learning to uncover spatiotemporal neural machinery in sensory processing.

Spatial Attention Modulates Neuronal Interactions between Simple and Complex Cells in V1

Visual perception is profoundly modulated by spatial attention, which can selectively prioritize goal-related information. Previous studies found spatial attention facilitated the efficacy of neuronal communication between visual cortices with hierarchical organizations. In the primary visual cortex (V1), there is also a hierarchical connection between simple (S) and complex (C) cells. We wonder whether and how spatial attention modulates neuronal communication within V1, especially for neuronal pairs with heterogeneous visual input. We simultaneously recorded the pairs' activity from macaque monkeys when they performed a spatial-attention-involved task, then applied likelihood-based Granger causality analysis to explore attentional modulation of neuronal interactions. First, a significant attention-related decrease in Granger causality was found in S-C pairs, which primarily displayed in the S-to-C feedforward connection. Second, the interaction strength of the feedforward connection was significantly higher than that of the feedback under attend toward (AT) conditions. Although information flow did not alter as the attentional focus shifted, the strength of communications between target- and distractor-stimuli-covered neurons differed only when attending to complex cells' receptive fields (RFs). Furthermore, pairs' communications depended on the attentional modulation of neurons' firing rates. Our findings demonstrate spatial attention does not induce specific information flow but rather amplifies directed communication within V1.

Independent spatiotemporal effects of spatial attention and background clutter on human object location representations

Spatial attention helps us to efficiently localize objects in cluttered environments. However, the processing stage at which spatial attention modulates object location representations remains unclear. Here we investigated this question identifying processing stages in time and space in an EEG and fMRI experiment respectively. As both object location representations and attentional effects have been shown to depend on the background on which objects appear, we included object background as an experimental factor. During the experiments, human participants viewed images of objects appearing in different locations on blank or cluttered backgrounds while either performing a task on fixation or on the periphery to direct their covert spatial attention away or towards the objects. We used multivariate classification to assess object location information. Consistent across the EEG and fMRI experiment, we show that spatial attention modulated location representations during late processing stages (>150 ms, in middle and high ventral visual stream areas) independent of background condition. Our results clarify the processing stage at which attention modulates object location representations in the ventral visual stream and show that attentional modulation is a cognitive process separate from recurrent processes related to the processing of objects on cluttered backgrounds.

The role of the parietal lobe in task-irrelevant suppression during learning

BACKGROUND

Attention optimizes the selection of visual information, while suppressing irrelevant visual input through cortical mechanisms that are still unclear. We set to investigate these processes using an attention task with an embedded to-be-ignored interfering visual input.

OBJECTIVE

We delivered electrical stimulation to attention-related brain areas to modulate these facilitatory/inhibitory attentional mechanisms. We asked whether overtly training on a task while being covertly exposed to visual features from a visually identical but different task tested at baseline might influence post-training performance on the baseline task.

METHODS

In Experiment one, at baseline subjects performed an orientation discrimination (OD) task using a pair of gratings presented at individual's psychophysical threshold. We then trained participants over three-day separate sessions on a temporal order judgment task (TOJ), using the exact same gratings but presented with different time offsets. On the last post-training session we re-tested OD. We coupled training with transcranial random noise stimulation (tRNS) over the parietal cortex, the human middle temporal area or sham, in three separate groups. In Experiment two, subjects performed the same OD task at baseline and post-training, while tRNS was delivered at rest during the same sessions and stimulation conditions as in Experiment one.

RESULTS

Results showed that tRNS over parietal cortex facilitated learning of the trained TOJ task. Moreover, we found a detrimental effect on the untrained OD task when subjects received parietal tRNS coupled with training (Experiment one), but a benefit on OD when subjects received stimulation while at rest (Experiment two).

CONCLUSIONS

These results clearly indicate that task-irrelevant information is actively suppressed during learning, and that this prioritization mechanism of selection likely resides in the parietal cortex.

Spike rates of frontal eye field neurons predict reaction times in a spatial attention task

Which neuronal signal(s) predict reaction times when subjects respond to a target at covertly attended locations? Although recent studies showed that spike rates are not predictive, it remains a highly contested question. Therefore, we record single-unit activity from frontal eye field (FEF) neurons while macaques are performing a covert spatial attention task. We find that the attentional modulation of spike rates of FEF neurons is strongly correlated with behavioral reaction times. Moreover, this correlation already emerges 1 s before target dimming, which triggers the behavioral responses. This prediction of reaction times by spike rates is found in neurons showing attention-dependent enhanced and suppressed activity for targets and distractors, respectively, yet in varying degrees across subjects. Thus, spike rates of FEF neurons can predict reaction times persistently and well before the operant behavior during selective attention tasks. Such long prediction windows will be useful for developing spike-based brain-machine interfaces.

TransFNN: A Novel Overtemperature Prediction Method for HVDC Converter Valves Based on an Improved Transformer and the F-NN Algorithm

Appropriate cooling of the converter valve in a high-voltage direct current (HVDC) transmission system is highly significant for the safety, stability, and economical operation of a power grid. The proper adjustment of cooling measures is based on the accurate perception of the valve's future overtemperature state, which is characterized by the valve's cooling water temperature. However, very few previous studies have focused on this need, and the existing Transformer model, which excels in time-series predictions, cannot be directly applied to forecast the valve overtemperature state. In this study, we modified the Transformer and present a hybrid Transformer-FCM-NN (TransFNN) model to predict the future overtemperature state of the converter valve. The TransFNN model decouples the forecast process into two stages: (i) The modified Transformer is used to obtain the future values of the independent parameters; (ii) the relation between the valve cooling water temperature and the six independent operating parameters is fit, and the output of the Transformer is used to calculate the future values of the cooling water temperature. The results of the quantitative experiments showed that the proposed TransFNN model outperformed other models with which it was compared; with TransFNN being applied to predict the overtemperature state of the converter valves, the forecast accuracy was 91.81%, which was improved by 6.85% compared with that of the original Transformer model. Our work provides a novel approach to predicting the valve overtemperature state and acts as a data-driven tool for operation and maintenance personnel to use to adjust valve cooling measures punctually, effectively, and economically.

Engaging in word recognition elicits highly specific modulations in visual cortex

A person's cognitive state determines how their brain responds to visual stimuli. The most common such effect is a response enhancement when stimuli are task relevant and attended rather than ignored. In this fMRI study, we report a surprising twist on such attention effects in the visual word form area (VWFA), a region that plays a key role in reading. We presented participants with strings of letters and visually similar shapes, which were either relevant for a specific task (lexical decision or gap localization) or ignored (during a fixation dot color task). In the VWFA, the enhancement of responses to attended stimuli occurred only for letter strings, whereas non-letter shapes evoked smaller responses when attended than when ignored. The enhancement of VWFA activity was accompanied by strengthened functional connectivity with higher-level language regions. These task-dependent modulations of response magnitude and functional connectivity were specific to the VWFA and absent in the rest of visual cortex. We suggest that language regions send targeted excitatory feedback into the VWFA only when the observer is trying to read. This feedback enables the discrimination of familiar and nonsense words and is distinct from generic effects of visual attention.

Attention in flux

Selective attention comprises essential infrastructural functions supporting cognition-anticipating, prioritizing, selecting, routing, integrating, and preparing signals to guide adaptive behavior. Most studies have examined its consequences, systems, and mechanisms in a static way, but attention is at the confluence of multiple sources of flux. The world advances, we operate within it, our minds change, and all resulting signals progress through multiple pathways within the dynamic networks of our brains. Our aim in this review is to raise awareness of and interest in three important facets of how timing impacts our understanding of attention. These include the challenges posed to attention by the timing of neural processing and psychological functions, the opportunities conferred to attention by various temporal structures in the environment, and how tracking the time courses of neural and behavioral modulations with continuous measures yields surprising insights into the workings and principles of attention.

A cortical zoom-in operation underlies covert shifts of visual spatial attention

Shifting the focus of attention without moving the eyes poses challenges for signal coding in visual cortex in terms of spatial resolution, signal routing, and cross-talk. Little is known how these problems are solved during focus shifts. Here, we analyze the spatiotemporal dynamic of neuromagnetic activity in human visual cortex as a function of the size and number of focus shifts in visual search. We find that large shifts elicit activity modulations progressing from highest (IT) through mid-level (V4) to lowest hierarchical levels (V1). Smaller shifts cause those modulations to start at lower levels in the hierarchy. Successive shifts involve repeated backward progressions through the hierarchy. We conclude that covert focus shifts arise from a cortical coarse-to-fine process progressing from retinotopic areas with larger toward areas with smaller receptive fields. This process localizes the target and increases the spatial resolution of selection, which resolves the above issues of cortical coding.

Dissociable roles of human frontal eye fields and early visual cortex in presaccadic attention - evidence from TMS

Shortly before each saccadic eye movement, presaccadic attention improves visual sensitivity at the saccade target 1-5 at the expense of lowered sensitivity at non-target locations 6-11 . Some behavioral and neural correlates of presaccadic attention and covert attention -which likewise enhances sensitivity, but during fixation 12 -are similar 13 . This resemblance has led to the debatable 13-18 notion that presaccadic and covert attention are functionally equivalent and rely on the same neural circuitry 19-21 . At a broad scale, oculomotor brain structures (e.g., FEF) are also modulated during covert attention 22-24 - yet by distinct neuronal subpopulations 25-28 . Perceptual benefits of presaccadic attention rely on feedback from oculomotor structures to visual cortices 29,30 ( Fig. 1a ); micro-stimulation of FEF in non-human primates affects activity in visual cortex 31-34 and enhances visual sensitivity at the movement field of the stimulated neurons 35-37 . Similar feedback projections seem to exist in humans: FEF+ activation precedes occipital activation during saccade preparation 38,39 and FEF TMS modulates activity in visual cortex 40-42 and enhances perceived contrast in the contralateral hemifield 40 . We investigated presaccadic feedback in humans by applying TMS to frontal or visual areas during saccade preparation. By simultaneously measuring perceptual performance, we show the causal and differential roles of these brain regions in contralateral presaccadic benefits at the saccade target and costs at non-targets: Whereas rFEF+ stimulation reduced presaccadic costs throughout saccade preparation, V1/V2 stimulation reduced benefits only shortly before saccade onset. These effects provide causal evidence that presaccadic attention modulates perception through cortico-cortical feedback and further dissociate presaccadic and covert attention.

The cost of divided attention for detection of simple visual features primarily reflects limits in post-perceptual processing

Covert spatial attention allows us to prioritize processing at relevant locations. Perception is generally poorer when attention is distributed across multiple locations than when attention is focused on a single location. However, while divided attention typically impairs performance, recent work suggests that divided attention does not seem to impair detection of simple visual features. Here, we re-examined this possibility. In two experiments, observers detected a simple target (a vertical Gabor), and we manipulated whether attention was focused at one location (focal-cue condition) or distributed across two locations (distributed-cue condition). In Experiment 1, targets could appear independently at each location, such that observers needed to judge target presence for each location separately in the distributed-cue condition. Under these conditions, we found a robust cost of dividing attention. Next, we further probed what stage of processing gave rise to this cost. In Experiment 1, the cost of dividing attention could reflect a limit in the ability to make concurrent judgments about target presence. In Experiment 2, we simplified the task to test whether this was the case: just one target could appear on each trial, such that observers made a single judgment ("was a target present?") in both the focal-cue and distributed-cue conditions. Here, we found a marginal cost of dividing attention that was weaker than the cost in Experiment 1. Together, our results suggest that divided attention does impair detection of simple visual features, but that this cost is primarily due to a limit in post-perceptual processes.

Seeing an Auditory Object: Pupillary Light Response Reflects Covert Attention to Auditory Space and Object

Attention to the relevant object and space is the brain's strategy to effectively process the information of interest in complex environments with limited neural resources. Numerous studies have documented how attention is allocated in the visual domain, whereas the nature of attention in the auditory domain has been much less explored. Here, we show that the pupillary light response can serve as a physiological index of auditory attentional shift and can be used to probe the relationship between space-based and object-based attention as well. Experiments demonstrated that the pupillary response corresponds to the luminance condition where the attended auditory object (e.g., spoken sentence) was located, regardless of whether attention was directed by a spatial (left or right) or nonspatial (e.g., the gender of the talker) cue and regardless of whether the sound was presented via headphones or loudspeakers. These effects on the pupillary light response could not be accounted for as a consequence of small (although observable) biases in gaze position drifting. The overall results imply a unified audiovisual representation of spatial attention. Auditory object-based attention contains the space representation of the attended auditory object, even when the object is oriented without explicit spatial guidance.

Tuning curves vs. population responses, and perceptual consequences of receptive-field remapping

Sensory processing is often studied by examining how a given neuron responds to a parameterized set of stimuli (tuning curve) or how a given stimulus evokes responses from a parameterized set of neurons (population response). Although tuning curves and the corresponding population responses contain the same information, they can have different properties. These differences are known to be important because the perception of a stimulus should be decoded from its population response, not from any single tuning curve. The differences are less studied in the spatial domain where a cell's spatial tuning curve is simply its receptive field (RF) profile. Here, we focus on evaluating the common belief that perrisaccadic forward and convergent RF shifts lead to forward (translational) and convergent (compressive) perceptual mislocalization, respectively, and investigate the effects of three related factors: decoders' awareness of RF shifts, changes of cells' covering density near attentional locus (the saccade target), and attentional response modulation. We find that RF shifts alone produce either no shift or an opposite shift of the population responses depending on whether or not decoders are aware of the RF shifts. Thus, forward RF shifts do not predict forward mislocalization. However, convergent RF shifts change cells' covering density for aware decoders (but not for unaware decoders) which may predict convergent mislocalization. Finally, attentional modulation adds a convergent component to population responses for stimuli near the target. We simulate the combined effects of these factors and discuss the results with extant mislocalization data. We speculate that perisaccadic mislocalization might be the flash-lag effect unrelated to perisaccadic RF remapping but to resolve the issue, one has to address the question of whether or not perceptual decoders are aware of RF shifts.

A blinking focal pattern of re-entrant activity in the avian tectum

Re-entrant connections are inherent to nervous system organization; however, a comprehensive understanding of their operation is still lacking. In birds, topographically organized re-entrant signals, carried by axons from the nucleus-isthmi-parvocellularis (Ipc), are distinctly recorded as bursting discharges across the optic tectum (TeO). Here, we used up to 48 microelectrodes regularly spaced on the superficial tectal layers of anesthetized pigeons to characterize the spatial-temporal pattern of this axonal re-entrant activity in response to different visual stimulation. We found that a brief luminous spot triggered repetitive waves of bursting discharges that, appearing from initial sources, propagated horizontally to areas representing up to 28° of visual space, widely exceeding the area activated by the retinal fibers. In response to visual motion, successive burst waves started along and around the stimulated tectal path, tracking the stimulus in discontinuous steps. When two stimuli were presented, the burst-wave sources alternated between the activated tectal loci, as if only one source could be active at any given time. Because these re-entrant signals boost the retinal input to higher visual areas, their peculiar dynamics mimic a blinking "spotlight," similar to the internal searching mechanism classically used to explain spatial attention. Tectal re-entry from Ipc is thus highly structured and intrinsically discontinuous, and higher tectofugal areas, which lack retinotopic organization, will thus receive incoming visual activity in a sequential and piecemeal fashion. We anticipate that analogous re-entrant patterns, perhaps hidden in less bi-dimensionally organized topographies, may organize the flow of neural activity in other parts of the brain as well.

What is attention? Adverbialist theories

This article presents theories of attention that attempt to derive their answer to the question of what attention is from their answers to the question of what it is for some activity to be done attentively. Such theories provide a distinctive account of the difficulties that are faced by the attempt to locate processes in the brain by which the phenomena of attention can be explained. Their account does not share the pessimism of theories suggesting that the concept of attention is defective. Instead it reconstrues the explanatory relationship between attention and the processes that constitute it, in a way that is illustrated here by considering the relationship between attention and the processes that are identified by the biased competition theory. After considering some of the ways in which an adverbialist approach might be developed, the article concludes by suggesting some possible solutions to a problem concerning distraction, by which prominent adverbialist theories of attention have been dogged. This article is categorized under: Psychology > Attention Philosophy > Metaphysics Philosophy > Foundations of Cognitive Science.

Tracking momentary fluctuations in human attention with a cognitive brain-machine interface

Selective attention produces systematic effects on neural states. It is unclear whether, conversely, momentary fluctuations in neural states have behavioral significance for attention. We investigated this question in the human brain with a cognitive brain-machine interface (cBMI) for tracking electrophysiological steady-state visually evoked potentials (SSVEPs) in real-time. Discrimination accuracy (d') was significantly higher when target stimuli were triggered at high, versus low, SSVEP power states. Target and distractor SSVEP power was uncorrelated across the hemifields, and target d' was unaffected by distractor SSVEP power states. Next, we trained participants on an auditory neurofeedback paradigm to generate biased, cross-hemispheric competitive interactions between target and distractor SSVEPs. The strongest behavioral effects emerged when competitive SSVEP dynamics unfolded at a timescale corresponding to the deployment of endogenous attention. In sum, SSVEP power dynamics provide a reliable readout of attentional state, a result with critical implications for tracking and training human attention.

Reward salience but not spatial attention dominates the value representation in the orbitofrontal cortex

The orbitofrontal cortex (OFC) encodes value and plays a key role in value-based decision-making. However, the attentional modulation of the OFC's value encoding is poorly understood. We trained two monkeys to detect a luminance change at a cued location between a pair of visual stimuli, which were over-trained pictures associated with different amounts of juice reward and, thus, different reward salience. Both the monkeys' behavior and the dorsolateral prefrontal cortex neuronal activities indicated that the monkeys actively directed their spatial attention toward the cued stimulus during the task. However, the OFC's neuronal responses were dominated by the stimulus with higher reward salience and encoded its value. The value of the less salient stimulus was only weakly represented regardless of spatial attention. The results demonstrate that reward and spatial attention are distinctly represented in the prefrontal cortex and the OFC maintains a stable representation of reward salience minimally affected by attention.

Microstimulation of visual area V4 improves visual stimulus detection

Neuronal activity in visual area V4 is well known to be modulated by selective attention, and there are reports on V4 lesions leading to attentional deficits. However, it remains unclear whether V4 microstimulation can elicit attentional benefits. To test this hypothesis, we performed local microstimulation in area V4 and explored its spatial and time dynamics in two macaque monkeys performing a visual detection task. Microstimulation was delivered via chronically implanted multi-electrode arrays. We found that microstimulation increases average performance by 35% and reduces luminance detection thresholds by −30%. This benefit critically depends on the onset of microstimulation relative to the stimulus, consistent with known dynamics of endogenous attention. These results show that local microstimulation of V4 can improve behavior and highlight the critical role of V4 for attention.

•

Microstimulation of visual area V4 improves visual stimulus detection

•

Effects of V4 microstimulation extend to the other hemifield

•

Microstimulation effects are time dependent and consistent with attention dynamics

Neuronal congruency effects in macaque prefrontal cortex

The interplay between task-relevant and task-irrelevant information may induce conflicts that impair behavioral performance, a.k.a. behavioral congruency effects. The neuronal mechanisms underlying behavioral congruency effects, however, are poorly understood. We recorded single unit activity in monkey prefrontal cortex using a task-switching paradigm and discovered a neuronal congruency effect (NCE) that is carried by target and distractor neurons which process target and distractor-related information, respectively. The former neurons provide more signal, the latter less noise in congruent compared to incongruent conditions, resulting in a better target representation. Such NCE is dominated by the level of congruency, and is not determined by the task rules the subjects used, their reaction times (RT), the length of the delay period, nor the response levels of the neurons. We propose that this NCE can explain behavioral congruency effects in general, as well as previous fMRI and EEG results in various conflict paradigms.

Stimulus-induced conflicts impair behavior in conflict tasks resulting in a phenomenon known as the behavioral congruency effect. Here, the authors investigate the neural underpinnings of this phenomenon and report a neuronal congruency effect in macaque prefrontal cortex to explain this impairment.

When We Study the Ability to Attend, What Exactly Are We Trying to Understand?

When we study the human ability to attend, what exactly do we seek to understand? It is not clear what the answer might be to this question. There is still so much to know, while acknowledging the tremendous progress of past decades of research. It is as if each new study adds a tile to the mosaic that, when viewed from a distance, we hope will reveal the big picture of attention. However, there is no map as to how each tile might be placed nor any guide as to what the overall picture might be. It is like digging up bits of mosaic tile at an ancient archeological site with no key as to where to look and then not only having to decide which picture it belongs to but also where exactly in that puzzle it should be placed. I argue that, although the unearthing of puzzle pieces is very important, so is their placement, but this seems much less emphasized. We have mostly unearthed a treasure trove of puzzle pieces but they are all waiting for cleaning and reassembly. It is an activity that is scientifically far riskier, but with great risk comes a greater reward. Here, I will look into two areas of broad agreement, specifically regarding visual attention, and dig deeper into their more nuanced meanings, in the hope of sketching a starting point for the guide to the attention mosaic. The goal is to situate visual attention as a purely computational problem and not as a data explanation task; it may become easier to place the puzzle pieces once you understand why they exist in the first place.

Neuroscience and architecture: Modulating behavior through sensorimotor responses to the built environment

As we move through the world, natural and built environments implicitly guide behavior by appealing to certain sensory and motor dynamics. This process can be motivated by automatic attention to environmental features that resonate with specific sensorimotor responses. This review aims at providing a psychobiological framework describing how environmental features can lead to automated sensorimotor responses through defined neurophysiological mechanisms underlying attention. Through the use of automated processes in subsets of cortical structures, the goal of this framework is to describe on a neuronal level the functional link between the designed environment and sensorimotor responses. By distinguishing between environmental features and sensorimotor responses we elaborate on how automatic behavior employs the environment for sensorimotor adaptation. This is realized through a thalamo-cortical network integrating environmental features with motor aspects of behavior. We highlight the underlying transthalamic transmission from an Enactive and predictive perspective and review recent studies that effectively modulated behavior by systematically manipulating environmental features. We end by suggesting a promising combination of neuroimaging and computational analysis for future studies.