Perisaccadic Receptive Field Expansion in the Lateral Intraparietal Area

How forward remapping predicts peri-saccadic biphasic mislocalization

Neurons in many visual and oculomotor "priority-map" brain areas display forward receptive field (RF) remapping: they respond to stimuli appearing before a saccade at the spatial location that their RF will occupy after the saccade. Concurrently, psychophysical studies have shown that flashes around saccade onset are systematically mislocalized in various patterns. One prominent pattern is a biphasic pattern, where flashes right before a saccade are mislocalized in the saccade direction (forward) and flashes right after a saccade are mislocalized opposite to the saccade direction (backward). Although forward RF remapping and biphasic mislocalization have been suspected to be linked, how this works has never been explained. Here, we show how persistent flash-evoked activity and decoding of the flash position after the saccade combine to produce this biphasic mislocalization pattern. We implement a rate model, consistent with the essential properties of RF remapping, and show that biphasic mislocalization results from insufficient remapping before the saccade, and residual/inappropriate remapping after the saccade. Less remapping before the saccade produces larger forward mislocalization of pre-saccadic flashes, and less remapping after the saccade produces smaller backward mislocalization of post-saccadic flashes. Forward RF remapping thus captures a biphasic peri-saccadic flash-mislocalization pattern consistent with behavioral data.

A circuit model for transsaccadic space updating and mislocalization

We perceive a stable, continuous world despite drastic changes of retinal images across saccades. However, while persistent objects in daily life appear stable across saccades, stimuli flashed around saccades can be grossly mislocalized. We address this puzzle with our recently proposed circuit model for perisaccadic receptive-field (RF) remapping in the lateral interparietal area (LIP) and frontal eye fields (FEF). The model uses center-excitation/surround-inhibition connections to store a relevant stimulus' retinal location in memory as a population activity. This activity profile is updated across each saccade by directional connections gated by the corollary discharge (CD) of the saccade command. The updating is a continuous backward (against the saccade) shift of the population activity (equivalent to continuous forward remapping of the RFs), whose cumulative effect across the saccade is a subtraction of the saccade vector. The model that correctly updates persistent stimuli, and flashes well before and after saccades, produces the observed forward and backward translational mislocalization for flashes around the saccade onset and offset, respectively, because of insufficient and unnecessary cumulative updating after the saccade, caused by visual response latency and sluggish CD time course. We confirm the model prediction that for perisaccadic RFs measured with flashes before the saccades, the forward remapping magnitudes across the saccades are smaller for later flashes. Our work suggests that transsaccadic perception is stable because the presaccadic retinal position of an object is updated to match the postsaccadic (reafferent) retinal position of the same object, and that the brain uses "unaware" decoders which do not distinguish between different origins of neurons' activities.

Perisaccadic perceptual mislocalization strength depends on the visual appearance of saccade targets

We normally perceive a stable visual environment despite eye movements. To achieve such stability, visual processing integrates information across a given saccade, and laboratory hallmarks of such integration are robustly observed by presenting brief perisaccadic visual probes. In one classic phenomenon, probe locations are grossly mislocalized. This mislocalization is believed to depend, at least in part, on corollary discharge associated with saccade-related neuronal movement commands. However, we recently found that superior colliculus motor bursts, a known source of corollary discharge, can be different for different image appearances of the saccade target. Therefore, here we investigated whether perisaccadic mislocalization also depends on saccade target appearance. We asked human participants to generate saccades to either low (0.5 cycles/°) or high (5 cycles/°) spatial frequency gratings. We always placed a high-contrast target spot at grating center, to ensure matched saccades across image types. We presented a single, brief perisaccadic probe, which was high in contrast to avoid saccadic suppression, and the subjects pointed (via mouse cursor) at the seen probe location. We observed stronger perisaccadic mislocalization for low-spatial frequency saccade targets and for upper visual field probe locations. This was despite matched saccade metrics and kinematics across conditions, and it was also despite matched probe visibility for the different saccade target images (low vs. high spatial frequency). Assuming that perisaccadic visual mislocalization depends on corollary discharge, our results suggest that such discharge might relay more than just spatial saccade vectors to the visual system; saccade target visual features can also be transmitted.NEW & NOTEWORTHY Brief visual probes are grossly mislocalized when presented in the temporal vicinity of saccades. Although the mechanisms of such mislocalization are still under investigation, one component of them could derive from corollary discharge signals associated with saccade movement commands. Here, we were motivated by the observation that superior colliculus movement bursts, one source of corollary discharge, vary with saccade target image appearance. If so, then perisaccadic mislocalization should also do so, which we confirmed.

A circuit model for transsaccadic space updating and mislocalization

We perceive a stable, continuous world despite drastic changes of retinal images across saccades. However, while persistent objects in daily life appear stable across saccades, stimuli flashed around saccades can be grossly mislocalized. We address this puzzle with our recently proposed circuit model for perisaccadic receptive-field (RF) remapping in LIP and FEF. The model uses center/surround connections to store a relevant stimulus' retinal location in memory as a population activity. This activity profile is updated across each saccade by directional connections gated by the corollary discharge (CD) of the saccade command. The updating is a continuous backward (against the saccade) shift of the population activity (equivalent to continuous forward remapping of the RFs), whose cumulative effect across the saccade is a subtraction of the saccade vector. The model explains forward and backward translational mislocalization for stimuli flashed around the saccade onset and offset, respectively, as insufficient and unnecessary cumulative updating after the saccade, caused by the sluggish CD time course and visual response latency. We confirm the model prediction that for perisaccadic RFs measured with flashes before the saccades, the final forward remapping magnitudes after the saccades are smaller for later flashes. We discuss the possibility that compressive mislocalization results from a brief reduction of attentional remapping and repulsion. Although many models of RF remapping, transsaccadic updating, and perisaccadic mislocalization have been proposed, our work unifies them into a single circuit mechanism and suggests that the brain uses "unaware" decoders which do not distinguish between different origins of neurons' activities.

Presaccadic preview shapes postsaccadic processing more where perception is poor

The presaccadic preview of a peripheral target enhances the efficiency of its postsaccadic processing, termed the extrafoveal preview effect. Peripheral visual performance-and thus the quality of the preview-varies around the visual field, even at isoeccentric locations: It is better along the horizontal than vertical meridian and along the lower than upper vertical meridian. To investigate whether these polar angle asymmetries influence the preview effect, we asked human participants to preview four tilted gratings at the cardinals, until a central cue indicated which one to saccade to. During the saccade, the target orientation either remained or slightly changed (valid/invalid preview). After saccade landing, participants discriminated the orientation of the (briefly presented) second grating. Stimulus contrast was titrated with adaptive staircases to assess visual performance. Expectedly, valid previews increased participants' postsaccadic contrast sensitivity. This preview benefit, however, was inversely related to polar angle perceptual asymmetries; largest at the upper, and smallest at the horizontal meridian. This finding reveals that the visual system compensates for peripheral asymmetries when integrating information across saccades, by selectively assigning higher weights to the less-well perceived preview information. Our study supports the recent line of evidence showing that perceptual dynamics around saccades vary with eye movement direction.

Widespread receptive field remapping in early primate visual cortex

Predictive remapping of receptive fields (RFs) is thought to be one of the critical mechanisms for enforcing perceptual stability during eye movements. While RF remapping has been observed in several cortical areas, its role in early visual cortex and its consequences on the tuning properties of neurons have been poorly understood. Here, we track remapping RFs in hundreds of neurons from visual area V2 while subjects perform a cued saccade task. We find that remapping is widespread in area V2 across neurons from all recorded cortical layers and cell types. Furthermore, our results suggest that remapping RFs not only maintain but also transiently enhance their feature selectivity due to untuned suppression. Taken together, these findings shed light on the dynamics and prevalence of remapping in the early visual cortex, forcing us to revise current models of perceptual stability during saccadic eye movements.

Pre-saccadic Preview Shapes Post-Saccadic Processing More Where Perception is Poor

The pre-saccadic preview of a peripheral target enhances the efficiency of its post-saccadic processing, termed the extrafoveal preview effect. Peripheral visual performance -and thus the quality of the preview- varies around the visual field, even at iso-eccentric locations: it is better along the horizontal than vertical meridian and along the lower than upper vertical meridian. To investigate whether these polar angle asymmetries influence the preview effect, we asked human participants (to preview four tilted gratings at the cardinals, until a central cue indicated to which one to saccade. During the saccade, the target orientation either remained or slightly changed (valid/invalid preview). After saccade landing, participants discriminated the orientation of the (briefly presented) second grating. Stimulus contrast was titrated with adaptive staircases to assess visual performance. Expectedly, valid previews increased participants' post-saccadic contrast sensitivity. This preview benefit, however, was inversely related to polar angle perceptual asymmetries; largest at the upper, and smallest at the horizontal meridian. This finding reveals that the visual system compensates for peripheral asymmetries when integrating information across saccades, by selectively assigning higher weights to the less-well perceived preview information. Our study supports the recent line of evidence showing that perceptual dynamics around saccades vary with eye movement direction.

Feature-selective responses in macaque visual cortex follow eye movements during natural vision

In natural vision, primates actively move their eyes several times per second via saccades. It remains unclear whether, during this active looking, visual neurons exhibit classical retinotopic properties, anticipate gaze shifts or mirror the stable quality of perception, especially in complex natural scenes. Here, we let 13 monkeys freely view thousands of natural images across 4.6 million fixations, recorded 883 h of neuronal responses in six areas spanning primary visual to anterior inferior temporal cortex and analyzed spatial, temporal and featural selectivity in these responses. Face neurons tracked their receptive field contents, indicated by category-selective responses. Self-consistency analysis showed that general feature-selective responses also followed eye movements and remained gaze-dependent over seconds of viewing the same image. Computational models of feature-selective responses located retinotopic receptive fields during free viewing. We found limited evidence for feature-selective predictive remapping and no viewing-history integration. Thus, ventral visual neurons represent the world in a predominantly eye-centered reference frame during natural vision.

Perisaccadic and attentional remapping of receptive fields in lateral intraparietal area and frontal eye fields

The nature and function of perisaccadic receptive field (RF) remapping have been controversial. We use a delayed saccade task to reduce previous confounds and examine the remapping time course in the lateral intraparietal area and frontal eye fields. In the delay period, the RF shift direction turns from the initial fixation to the saccade target. In the perisaccadic period, RFs first shift toward the target (convergent remapping), but around the time of saccade onset/offset, the shifts become predominantly toward the post-saccadic RF locations (forward remapping). Thus, unlike forward remapping that depends on the corollary discharge (CD) of the saccade command, convergent remapping appears to follow attention from the initial fixation to the target. We model the data with attention-modulated and CD-gated connections and show that both sets of connections emerge automatically in neural networks trained to update stimulus retinal locations across saccades. Our work thus unifies previous findings into a mechanism for transsaccadic visual stability.

Perisaccadic and Attentional Remapping of Receptive Fields in Lateral Intraparietal Area and Frontal Eye Fields

The nature and function of perisaccadic receptive-field (RF) remapping have been controversial. We used a delayed saccade task to reduce previous confounds and examined the remapping time course in areas LIP and FEF. In the delay period, the RF shift direction turned from the initial fixation to the saccade target. In the perisaccadic period, RFs first shifted toward the target (convergent remapping) but around the time of saccade onset/offset, the shifts became predominantly toward the post-saccadic RF locations (forward remapping). Thus, unlike forward remapping that depends on the corollary discharge (CD) of the saccade command, convergent remapping appeared to follow attention from the initial fixation to the target. We modelled the data with attention-modulated and CD-gated connections, and showed that both sets of connections emerged automatically in neural networks trained to update stimulus retinal locations across saccades. Our work thus unifies previous findings into a mechanism for transsaccadic visual stability.

Neural population dynamics of human working memory

The activity of neurons in macaque prefrontal cortex (PFC) persists during working memory (WM) delays, providing a mechanism for memory.1,2,3,4,5,6,7,8,9,10,11 Although theory,11,12 including formal network models,13,14 assumes that WM codes are stable over time, PFC neurons exhibit dynamics inconsistent with these assumptions.15,16,17,18,19 Recently, multivariate reanalyses revealed the coexistence of both stable and dynamic WM codes in macaque PFC.20,21,22,23 Human EEG studies also suggest that WM might contain dynamics.24,25 Nonetheless, how WM dynamics vary across the cortical hierarchy and which factors drive dynamics remain unknown. To elucidate WM dynamics in humans, we decoded WM content from fMRI responses across multiple cortical visual field maps.26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48 We found coexisting stable and dynamic neural representations of WM during a memory-guided saccade task. Geometric analyses of neural subspaces revealed that early visual cortex exhibited stronger dynamics than high-level visual and frontoparietal cortex. Leveraging models of population receptive fields, we visualized and made the neural dynamics interpretable. We found that during WM delays, V1 population initially encoded a narrowly tuned bump of activation centered on the peripheral memory target. Remarkably, this bump then spread inward toward foveal locations, forming a vector along the trajectory of the forthcoming memory-guided saccade. In other words, the neural code transformed into an abstraction of the stimulus more proximal to memory-guided behavior. Therefore, theories of WM must consider both sensory features and their task-relevant abstractions because changes in the format of memoranda naturally drive neural dynamics.

Tuning curves vs. population responses, and perceptual consequences of receptive-field remapping

Sensory processing is often studied by examining how a given neuron responds to a parameterized set of stimuli (tuning curve) or how a given stimulus evokes responses from a parameterized set of neurons (population response). Although tuning curves and the corresponding population responses contain the same information, they can have different properties. These differences are known to be important because the perception of a stimulus should be decoded from its population response, not from any single tuning curve. The differences are less studied in the spatial domain where a cell's spatial tuning curve is simply its receptive field (RF) profile. Here, we focus on evaluating the common belief that perrisaccadic forward and convergent RF shifts lead to forward (translational) and convergent (compressive) perceptual mislocalization, respectively, and investigate the effects of three related factors: decoders' awareness of RF shifts, changes of cells' covering density near attentional locus (the saccade target), and attentional response modulation. We find that RF shifts alone produce either no shift or an opposite shift of the population responses depending on whether or not decoders are aware of the RF shifts. Thus, forward RF shifts do not predict forward mislocalization. However, convergent RF shifts change cells' covering density for aware decoders (but not for unaware decoders) which may predict convergent mislocalization. Finally, attentional modulation adds a convergent component to population responses for stimuli near the target. We simulate the combined effects of these factors and discuss the results with extant mislocalization data. We speculate that perisaccadic mislocalization might be the flash-lag effect unrelated to perisaccadic RF remapping but to resolve the issue, one has to address the question of whether or not perceptual decoders are aware of RF shifts.

Distinct networks coupled with parietal cortex for spatial representations inside and outside the visual field

Our mental representation of egocentric space is influenced by the disproportionate sensory perception of the body. Previous studies have focused on the neural architecture for egocentric representations within the visual field. However, the space representation underlying the body is still unclear. To address this problem, we applied both functional Magnitude Resonance Imaging (fMRI) and Magnetoencephalography (MEG) to a spatial-memory paradigm by using a virtual environment in which human participants remembered a target location left, right, or back relative to their own body. Both experiments showed larger involvement of the frontoparietal network in representing a retrieved target on the left/right side than on the back. Conversely, the medial temporal lobe (MTL)-parietal network was more involved in retrieving a target behind the participants. The MEG data showed an earlier activation of the MTL-parietal network than that of the frontoparietal network during retrieval of a target location. These findings suggest that the parietal cortex may represent the entire space around the self-body by coordinating two distinct brain networks.

Cointegration of single-transistor neurons and synapses by nanoscale CMOS fabrication for highly scalable neuromorphic hardware

Cointegration of multistate single-transistor neurons and synapses was demonstrated for highly scalable neuromorphic hardware, using nanoscale complementary metal-oxide semiconductor (CMOS) fabrication. The neurons and synapses were integrated on the same plane with the same process because they have the same structure of a metal-oxide semiconductor field-effect transistor with different functions such as homotype. By virtue of 100% CMOS compatibility, it was also realized to cointegrate the neurons and synapses with additional CMOS circuits. Such cointegration can enhance packing density, reduce chip cost, and simplify fabrication procedures. The multistate single-transistor neuron that can control neuronal inhibition and the firing threshold voltage was achieved for an energy-efficient and reliable neural network. Spatiotemporal neuronal functionalities are demonstrated with fabricated single-transistor neurons and synapses. Image processing for letter pattern recognition and face image recognition is performed using experimental-based neuromorphic simulation.

Perisaccadic encoding of temporal information in macaque area V4

The accurate processing of temporal information is of critical importance in everyday life. Yet, psychophysical studies in humans have shown that the perception of time is distorted around saccadic eye movements. The neural correlates of this misperception are still poorly understood. Behavioral and neural evidence suggest that it is tightly linked to other known perisaccadic modulations of visual perception. To further our understanding of how temporal processing is affected by saccades, we studied the representations of brief visual time intervals during fixation and saccades in area V4 of two awake macaques. We presented random sequences of vertical bar stimuli and extracted neural responses to double-pulse stimulation at varying interstimulus intervals. Our results show that temporal information about very brief intervals of as brief as 20 ms is reliably represented in the multiunit activity in area V4. Response latencies were not systematically modulated by the saccade. However, a general increase in perisaccadic activity altered the ratio of response amplitudes within stimulus pairs compared with fixation. In line with previous studies showing that the perception of brief time intervals is partly based on response levels, this may be seen as a possible correlate of the perisaccadic misperception of time.NEW & NOTEWORTHY We investigated for the first time how temporal information on very brief timescales is represented in area V4 around the time of saccadic eye movements. Overall, the responses showed an unexpectedly precise representation of time intervals. Our finding of a perisaccadic modulation of relative response amplitudes introduces a new possible correlate of saccade-related perceptual distortions of time.

Perisaccadic remapping: What? How? Why?

About 25 years ago, the discovery of receptive field (RF) remapping in the parietal cortex of nonhuman primates revealed that visual RFs, widely assumed to have a fixed retinotopic organization, can change position before every saccade. Measuring such changes can be deceptively difficult. As a result, studies that followed have generated a fascinating but somewhat confusing picture of the phenomenon. In this review, we describe how observations of RF remapping depend on the spatial and temporal sampling of visual RFs and saccade directions. Further, we summarize some of the theories of how remapping might occur in neural circuitry. Finally, based on neurophysiological and psychophysical observations, we discuss the ways in which remapping information might facilitate computations in downstream brain areas.

Spiking neurons with spatiotemporal dynamics and gain modulation for monolithically integrated memristive neural networks

As a key building block of biological cortex, neurons are powerful information processing units and can achieve highly complex nonlinear computations even in individual cells. Hardware implementation of artificial neurons with similar capability is of great significance for the construction of intelligent, neuromorphic systems. Here, we demonstrate an artificial neuron based on NbO_x volatile memristor that not only realizes traditional all-or-nothing, threshold-driven spiking and spatiotemporal integration, but also enables dynamic logic including XOR function that is not linearly separable and multiplicative gain modulation among different dendritic inputs, therefore surpassing neuronal functions described by a simple point neuron model. A monolithically integrated 4 × 4 fully memristive neural network consisting of volatile NbO_x memristor based neurons and nonvolatile TaO_x memristor based synapses in a single crossbar array is experimentally demonstrated, showing capability in pattern recognition through online learning using a simplified δ-rule and coincidence detection, which paves the way for bio-inspired intelligent systems.

Designing energy efficient and scalable artificial networks for neuromorphic computing remains a challenge. Here, the authors demonstrate online learning in a monolithically integrated 4 × 4 fully memristive neural network consisting of volatile NbOx memristor neurons and nonvolatile TaOx memristor synapses.

Corollary Discharge Versus Efference Copy: Distinct Neural Signals in Speech Preparation Differentially Modulate Auditory Responses

Actions influence sensory processing in a complex way to shape behavior. For example, during actions, a copy of motor signals-termed "corollary discharge" (CD) or "efference copy" (EC)-can be transmitted to sensory regions and modulate perception. However, the sole inhibitory function of the motor copies is challenged by mixed empirical observations as well as multifaceted computational demands for behaviors. We hypothesized that the content in the motor signals available at distinct stages of actions determined the nature of signals (CD vs. EC) and constrained their modulatory functions on perceptual processing. We tested this hypothesis using speech in which we could precisely control and quantify the course of action. In three electroencephalography (EEG) experiments using a novel delayed articulation paradigm, we found that preparation without linguistic contents suppressed auditory responses to all speech sounds, whereas preparing to speak a syllable selectively enhanced the auditory responses to the prepared syllable. A computational model demonstrated that a bifurcation of motor signals could be a potential algorithm and neural implementation to achieve the distinct functions in the motor-to-sensory transformation. These results suggest that distinct motor signals are generated in the motor-to-sensory transformation and integrated with sensory input to modulate perception.

Saccade-induced changes in ocular torsion reveal predictive orientation perception

Natural orienting of gaze often results in a retinal image that is rotated relative to space due to ocular torsion. However, we perceive neither this rotation nor a moving world despite visual rotational motion on the retina. This perceptual stability is often attributed to the phenomenon known as predictive remapping, but the current remapping literature ignores this torsional component. In addition, studies often simply measure remapping across either space or features (e.g., orientation) but in natural circumstances, both components are bound together for stable perception. One natural circumstance in which the perceptual system must account for the current and future eye orientation to correctly interpret the orientation of external stimuli occurs during movements to or from oblique eye orientations (i.e., eye orientations with both a horizontal and vertical angular component relative to the primary position). Here we took advantage of oblique eye orientation-induced ocular torsion to examine perisaccadic orientation perception. First, we found that orientation perception was largely predicted by the rotated retinal image. Second, we observed a presaccadic remapping of orientation perception consistent with maintaining a stable (but spatially inaccurate) retinocentric perception throughout the saccade. These findings strongly suggest that our seamless perceptual stability relies on retinocentric signals that are predictively remapped in all three ocular dimensions with each saccade.

Neuronal Representation of the Saccadic Timing Signals in Macaque Lateral Intraparietal Area

Primates frequently make saccades direct fovea on interesting objects to receive acute visual information. However, saccade displaces the images on retina and disrupts the visual constancy. One possible mechanism to retain visual constancy is by integrating the presaccadic and postsaccadic visual information right at the time of saccade, which makes the timing of saccade crucial. So far, the saccadic timing signals have been found only in the subcortical regions, for example, the cerebellum and superior colliculus, but not in the neocortex. Here we report 2 types of saccadic timing signals in macaque lateral intraparietal area (LIP). First, many presaccadic response neurons started to decline activity either right around the start (saccade-on-decay) or the end (saccade-off-decay) of saccades. Notably, the time difference between saccade-off-decay and saccade-on-decay was highly correlated with the mean duration of saccades but not with the individual ones, and both saccade-off-decay and saccade-on-decay were better aligned with saccade end than saccade start-reflecting prediction. Second, the peak activity plateau of a group of postsaccadic response neurons was highly correlated with the actual duration of saccade-reflecting reality. While the predicted timing signals might facilitate the integration of visual information across saccades in LIP, the actual duration signals might calibrate the prediction errors.

Corollary Discharge Contributions to Perceptual Continuity Across Saccades

Our vision depends upon shifting our high-resolution fovea to objects of interest in the visual field. Each saccade displaces the image on the retina, which should produce a chaotic scene with jerks occurring several times per second. It does not. This review examines how an internal signal in the primate brain (a corollary discharge) contributes to visual continuity across saccades. The article begins with a review of evidence for a corollary discharge in the monkey and evidence from inactivation experiments that it contributes to perception. The next section examines a specific neuronal mechanism for visual continuity, based on corollary discharge that is referred to as visual remapping. Both the basic characteristics of this anticipatory remapping and the factors that control it are enumerated. The last section considers hypotheses relating remapping to the perceived visual continuity across saccades, including remapping's contribution to perceived visual stability across saccades.

Time course of spatiotopic updating across saccades

Humans move their eyes several times per second, yet we perceive the outside world as continuous despite the sudden disruptions created by each eye movement. To date, the mechanism that the brain employs to achieve visual continuity across eye movements remains unclear. While it has been proposed that the oculomotor system quickly updates and informs the visual system about the upcoming eye movement, behavioral studies investigating the time course of this updating suggest the involvement of a slow mechanism, estimated to take more than 500 ms to operate effectively. This is a surprisingly slow estimate, because both the visual system and the oculomotor system process information faster. If spatiotopic updating is indeed this slow, it cannot contribute to perceptual continuity, because it is outside the temporal regime of typical oculomotor behavior. Here, we argue that the behavioral paradigms that have been used previously are suboptimal to measure the speed of spatiotopic updating. In this study, we used a fast gaze-contingent paradigm, using high phi as a continuous stimulus across eye movements. We observed fast spatiotopic updating within 150 ms after stimulus onset. The results suggest the involvement of a fast updating mechanism that predictively influences visual perception after an eye movement. The temporal characteristics of this mechanism are compatible with the rate at which saccadic eye movements are typically observed in natural viewing.

Pre-saccadic remapping relies on dynamics of spatial attention

Each saccade shifts the projections of the visual scene on the retina. It has been proposed that the receptive fields of neurons in oculomotor areas are predictively remapped to account for these shifts. While remapping of the whole visual scene seems prohibitively complex, selection by attention may limit these processes to a subset of attended locations. Because attentional selection consumes time, remapping of attended locations should evolve in time, too. In our study, we cued a spatial location by presenting an attention-capturing cue at different times before a saccade and constructed maps of attentional allocation across the visual field. We observed no remapping of attention when the cue appeared shortly before saccade. In contrast, when the cue appeared sufficiently early before saccade, attentional resources were reallocated precisely to the remapped location. Our results show that pre-saccadic remapping takes time to develop suggesting that it relies on the spatial and temporal dynamics of spatial attention.

Vision During Saccadic Eye Movements

The perceptual consequences of eye movements are manifold: Each large saccade is accompanied by a drop of sensitivity to luminance-contrast, low-frequency stimuli, impacting both conscious vision and involuntary responses, including pupillary constrictions. They also produce transient distortions of space, time, and number, which cannot be attributed to the mere motion on the retinae. All these are signs that the visual system evokes active processes to predict and counteract the consequences of saccades. We propose that a key mechanism is the reorganization of spatiotemporal visual fields, which transiently increases the temporal and spatial uncertainty of visual representations just before and during saccades. On one hand, this accounts for the spatiotemporal distortions of visual perception; on the other hand, it implements a mechanism for fusing pre- and postsaccadic stimuli. This, together with the active suppression of motion signals, ensures the stability and continuity of our visual experience.

Dissonant Representations of Visual Space in Prefrontal Cortex during Eye Movements

We used local field potentials (LFPs) and spikes to investigate representations of visual space in prefrontal cortex and the dynamics of those representations during eye movements. Spatial information contained in LFPs of the frontal eye field (FEF) was differentially distributed across frequencies, with a majority of that information being carried in alpha and high-gamma bands and minimal signal in the low-gamma band. During fixation, spatial information from alpha and high-gamma bands and spiking activity was robust across cortical layers. Receptive fields (RFs) derived from alpha and high-gamma bands were retinocentrically organized, and they were spatially correlated both with each other and with spiking RFs. However, alpha and high-gamma RFs probed before eye movements were dissociated. Whereas high-gamma and spiking RFs immediately converged toward the movement goal, alpha RFs remained largely unchanged during the initial probe response, but they converged later. These observations reveal possible mechanisms of dynamic spatial representations that underlie visual perception during eye movements.

Learning Peri-saccadic Remapping of Receptive Field from Experience in Lateral Intraparietal Area

Our eyes move constantly at a frequency of 3-5 times per second. These movements, called saccades, induce the sweeping of visual images on the retina, yet we perceive the world as stable. It has been suggested that the brain achieves this visual stability via predictive remapping of neuronal receptive field (RF). A recent experimental study disclosed details of this remapping process in the lateral intraparietal area (LIP), that is, about the time of the saccade, the neuronal RF expands along the saccadic trajectory temporally, covering the current RF (CRF), the future RF (FRF), and the region the eye will sweep through during the saccade. A cortical wave (CW) model was also proposed, which attributes the RF remapping as a consequence of neural activity propagating in the cortex, triggered jointly by a visual stimulus and the corollary discharge (CD) signal responsible for the saccade. In this study, we investigate how this CW model is learned naturally from visual experiences at the development of the brain. We build a two-layer network, with one layer consisting of LIP neurons and the other superior colliculus (SC) neurons. Initially, neuronal connections are random and non-selective. A saccade will cause a static visual image to sweep through the retina passively, creating the effect of the visual stimulus moving in the opposite direction of the saccade. According to the spiking-time-dependent-plasticity rule, the connection path in the opposite direction of the saccade between LIP neurons and the connection path from SC to LIP are enhanced. Over many such visual experiences, the CW model is developed, which generates the peri-saccadic RF remapping in LIP as observed in the experiment.

Heading representations in primates are compressed by saccades

Perceptual illusions help to understand how sensory signals are decoded in the brain. Here we report that the opposite approach is also applicable, i.e., results from decoding neural activity from monkey extrastriate visual cortex correctly predict a hitherto unknown perceptual illusion in humans. We record neural activity from monkey medial superior temporal (MST) and ventral intraparietal (VIP) area during presentation of self-motion stimuli and concurrent reflexive eye movements. A heading-decoder performs veridically during slow eye movements. During fast eye movements (saccades), however, the decoder erroneously reports compression of heading toward straight ahead. Functional equivalents of macaque areas MST and VIP have been identified in humans, implying a perceptual correlate (illusion) of this perisaccadic decoding error. Indeed, a behavioral experiment in humans shows that perceived heading is perisaccadically compressed toward the direction of gaze. Response properties of primate areas MST and VIP are consistent with being the substrate of the newly described visual illusion.Macaque higher visual areas MST and VIP encode heading direction based on self-motion stimuli. Here the authors show that, while making saccades, the heading direction decoded from the neural responses is compressed toward straight-ahead, and independently demonstrate a perceptual illusion in humans based on this perisaccadic decoding error.

Coherent alpha oscillations link current and future receptive fields during saccades

Oscillations are ubiquitous in the brain, and they can powerfully influence neural coding. In particular, when oscillations at distinct sites are coherent, they provide a means of gating the flow of neural signals between different cortical regions. Coherent oscillations also occur within individual brain regions, although the purpose of this coherence is not well understood. Here, we report that within a single brain region, coherent alpha oscillations link stimulus representations as they change in space and time. Specifically, in primate cortical area V4, alpha coherence links sites that encode the retinal location of a visual stimulus before and after a saccade. These coherence changes exhibit properties similar to those of receptive field remapping, a phenomenon in which individual neurons change their receptive fields according to the metrics of each saccade. In particular, alpha coherence, like remapping, is highly dependent on the saccade vector and the spatial arrangement of current and future receptive fields. Moreover, although visual stimulation plays a modulatory role, it is neither necessary nor sufficient to elicit alpha coherence. Indeed, a similar pattern of coherence is observed even when saccades are made in darkness. Together, these results show that the pattern of alpha coherence across the retinotopic map in V4 matches many of the properties of receptive field remapping. Thus, oscillatory coherence might play a role in constructing the stable representation of visual space that is an essential aspect of conscious perception.

Thalamic control of functional cortical connectivity

The thalamus is an evolutionarily conserved structure with extensive reciprocal connections to cortical regions. While its role in transmitting sensory signals is well-studied, its broader engagement in cognition is unclear. In this review, we discuss evidence that the thalamus regulates functional connectivity within and between cortical regions, determining how a cognitive process is implemented across distributed cortical microcircuits. Within this framework, thalamic circuits do not necessarily determine the categorical content of a cognitive process (e.g., sensory details in feature-based attention), but rather provide a route by which task-relevant cortical representations are sustained and coordinated. Additionally, thalamic control of cortical connectivity bridges general arousal to the specific processing of categorical content, providing an intermediate level of cognitive and circuit description that will facilitate mapping neural computations onto thought and behavior.

Circuits for presaccadic visual remapping

Saccadic eye movements rapidly displace the image of the world that is projected onto the retinas. In anticipation of each saccade, many neurons in the visual system shift their receptive fields. This presaccadic change in visual sensitivity, known as remapping, was first documented in the parietal cortex and has been studied in many other brain regions. Remapping requires information about upcoming saccades via corollary discharge. Analyses of neurons in a corollary discharge pathway that targets the frontal eye field (FEF) suggest that remapping may be assembled in the FEF's local microcircuitry. Complementary data from reversible inactivation, neural recording, and modeling studies provide evidence that remapping contributes to transsaccadic continuity of action and perception. Multiple forms of remapping have been reported in the FEF and other brain areas, however, and questions remain about the reasons for these differences. In this review of recent progress, we identify three hypotheses that may help to guide further investigations into the structure and function of circuits for remapping.

Corollary Discharge and Oculomotor Proprioception: Cortical Mechanisms for Spatially Accurate Vision

A classic problem in psychology is understanding how the brain creates a stable and accurate representation of space for perception and action despite a constantly moving eye. Two mechanisms have been proposed to solve this problem: Herman von Helmholtz's idea that the brain uses a corollary discharge of the motor command that moves the eye to adjust the visual representation, and Sir Charles Sherrington's idea that the brain measures eye position to calculate a spatial representation. Here, we discuss the cognitive, neuropsychological, and physiological mechanisms that support each of these ideas. We propose that both are correct: A rapid corollary discharge signal remaps the visual representation before an impending saccade, computing accurate movement vectors; and an oculomotor proprioceptive signal enables the brain to construct a more accurate craniotopic representation of space that develops slowly after the saccade.

Neural Network Evidence for the Coupling of Presaccadic Visual Remapping to Predictive Eye Position Updating

As we look around a scene, we perceive it as continuous and stable even though each saccadic eye movement changes the visual input to the retinas. How the brain achieves this perceptual stabilization is unknown, but a major hypothesis is that it relies on presaccadic remapping, a process in which neurons shift their visual sensitivity to a new location in the scene just before each saccade. This hypothesis is difficult to test in vivo because complete, selective inactivation of remapping is currently intractable. We tested it in silico with a hierarchical, sheet-based neural network model of the visual and oculomotor system. The model generated saccadic commands to move a video camera abruptly. Visual input from the camera and internal copies of the saccadic movement commands, or corollary discharge, converged at a map-level simulation of the frontal eye field (FEF), a primate brain area known to receive such inputs. FEF output was combined with eye position signals to yield a suitable coordinate frame for guiding arm movements of a robot. Our operational definition of perceptual stability was "useful stability," quantified as continuously accurate pointing to a visual object despite camera saccades. During training, the emergence of useful stability was correlated tightly with the emergence of presaccadic remapping in the FEF. Remapping depended on corollary discharge but its timing was synchronized to the updating of eye position. When coupled to predictive eye position signals, remapping served to stabilize the target representation for continuously accurate pointing. Graded inactivations of pathways in the model replicated, and helped to interpret, previous in vivo experiments. The results support the hypothesis that visual stability requires presaccadic remapping, provide explanations for the function and timing of remapping, and offer testable hypotheses for in vivo studies. We conclude that remapping allows for seamless coordinate frame transformations and quick actions despite visual afferent lags. With visual remapping in place for behavior, it may be exploited for perceptual continuity.