Transsaccadic integration of visual features in a line intersection task

Occipital and parietal cortex participate in a cortical network for transsaccadic discrimination of object shape and orientation

Saccades change eye position and interrupt vision several times per second, necessitating neural mechanisms for continuous perception of object identity, orientation, and location. Neuroimaging studies suggest that occipital and parietal cortex play complementary roles for transsaccadic perception of intrinsic versus extrinsic spatial properties, e.g., dorsomedial occipital cortex (cuneus) is sensitive to changes in spatial frequency, whereas the supramarginal gyrus (SMG) is modulated by changes in object orientation. Based on this, we hypothesized that both structures would be recruited to simultaneously monitor object identity and orientation across saccades. To test this, we merged two previous neuroimaging protocols: 21 participants viewed a 2D object and then, after sustained fixation or a saccade, judged whether the shape or orientation of the re-presented object changed. We, then, performed a bilateral region-of-interest analysis on identified cuneus and SMG sites. As hypothesized, cuneus showed both saccade and feature (i.e., object orientation vs. shape change) modulations, and right SMG showed saccade-feature interactions. Further, the cuneus activity time course correlated with several other cortical saccade/visual areas, suggesting a 'functional network' for feature discrimination. These results confirm the involvement of occipital/parietal cortex in transsaccadic vision and support complementary roles in spatial versus identity updating.

Dynamic saccade context triggers more stable object-location binding

Our visual systems rapidly perceive and integrate information about object identities and locations. There is long-standing debate about how we achieve world-centered (spatiotopic) object representations across eye movements, with many studies reporting persistent retinotopic (eye-centered) effects even for higher-level object-location binding. But these studies are generally conducted in fairly static experimental contexts. Might spatiotopic object-location binding only emerge in more dynamic saccade contexts? In the present study, we investigated this using the Spatial Congruency Bias paradigm in healthy adults. In the static (single saccade) context, we found purely retinotopic binding, as before. However, robust spatiotopic binding emerged in the dynamic (multiple frequent saccades) context. We further isolated specific factors that modulate retinotopic and spatiotopic binding. Our results provide strong evidence that dynamic saccade context can trigger more stable object-location binding in ecologically-relevant spatiotopic coordinates, perhaps via a more flexible brain state which accommodates improved visual stability in the dynamic world.

The spatial properties of adaptation-induced distance compression

Exposure to a dynamic texture reduces the perceived separation between objects, altering the mapping between physical relations in the environment and their neural representations. Here we investigated the spatial tuning and spatial frame of reference of this aftereffect to understand the stage(s) of processing where adaptation-induced changes occur. In Experiment 1, we measured apparent separation at different positions relative to the adapted area, revealing a strong but tightly tuned compression effect. We next tested the spatial frame of reference of the effect, either by introducing a gaze shift between adaptation and test phase (Experiment 2) or by decoupling the spatial selectivity of adaptation in retinotopic and world-centered coordinates (Experiment 3). Results across the two experiments indicated that both retinotopic and world-centered adaptation effects can occur independently. Spatial attention to the location of the adaptor alone could not account for the world-centered transfer we observed, and retinotopic adaptation did not transfer to world-centered coordinates after a saccade (Experiment 4). Finally, we found that aftereffects in different reference frames have a similar, narrow spatial tuning profile (Experiment 5). Together, our results suggest that the neural representation of local separation resides early in the visual cortex, but it can also be modulated by activity in higher visual areas.

Spatiotopic and retinotopic memory in the context of natural images

Neural responses throughout the visual cortex encode stimulus location in a retinotopic (i.e., eye-centered) reference frame, and memory for stimulus position is most precise in retinal coordinates. Yet visual perception is spatiotopic: objects are perceived as stationary, even though eye movements cause frequent displacement of their location on the retina. Previous studies found that, after a single saccade, memory of retinotopic locations is more accurate than memory of spatiotopic locations. However, it is not known whether various aspects of natural viewing affect the retinotopic reference frame advantage. We found that the retinotopic advantage may in part depend on a retinal afterimage, which can be effectively nullified through backwards masking. Moreover, in the presence of natural scenes, spatiotopic memory is more accurate than retinotopic memory, but only when subjects are provided sufficient time to process the scene before the eye movement. Our results demonstrate that retinotopic memory is not always more accurate than spatiotopic memory and that the fidelity of memory traces in both reference frames are sensitive to the presence of contextual cues.

Is the n-back task a measure of unstructured working memory capacity? Towards understanding its connection to other working memory tasks

Working memory is fundamental to human cognitive functioning, and it is often measured with the n-back task. However, it is not clear whether the n-back task is a valid measure of working memory. Importantly, previous studies have found poor correlations with measures of complex span, whereas a recent study (Frost et al., 2019) showed that n-back performance was correlated with a transsaccadic memory task but dissociated from performance on the change detection task, a well-accepted measure of working memory capacity. To test whether capacity is involved in the n-back task we correlated a spatial version of the test with different versions of the change detection task. Experiment 1 introduced perceptual and cognitive disruptions to the change detection task. This impacted task performance, however, all versions of the change detection task remained highly correlated with one another whereas there was no significant correlation with the n-back task. Experiment 2 removed spatial and non-spatial context from the change detection task. This produced a correlation with n-back. Our results indicate that the n-back task is supported by faculties similar to those that support change detection, but that this commonality is hidden when contextual information is available to be exploited in a change detection task such that structured representations can form. We suggest that n-back might be a valid measure of working memory, and that the ability to exploit contextual information is an important faculty captured by some versions of the change detection task.

Transsaccadic perception is affected by saccade landing point deviations after saccadic adaptation

Whenever we move our eyes, some visual information obtained before a saccade is combined with the visual information obtained after a saccade. Interestingly, saccades rarely land exactly on the saccade target, which may pose a problem for transsaccadic perception as it could affect the quality of postsaccadic input. Recently, however, we showed that transsaccadic feature integration is actually unaffected by deviations of saccade landing points. Possibly, transsaccadic integration remains unaffected because the presaccadic shift of attention follows the intended saccade target and not the actual saccade landing point during regular saccades. Here, we investigated whether saccade landing point errors can in fact alter transsaccadic perception when the presaccadic shift of attention follows the saccade landing point deviation. Given that saccadic adaptation not only changes the saccade vector, but also the presaccadic shift of attention, we combined a feature report paradigm with saccadic adaptation. Observers reported the color of the saccade target, which occasionally changed slightly during a saccade to the target. This task was performed before and after saccadic adaptation. The results showed that, after adaptation, presaccadic color information became less precise and transsaccadic perception had a stronger reliance on the postsaccadic color estimate. Therefore, although previous studies have shown that transsaccadic perception is generally unaffected by saccade landing point deviations, our results reveal that this cannot be considered a general property of the visual system. When presaccadic shifts of attention follow altered saccade landing points, transsaccadic perception is affected, suggesting that transsaccadic feature perception might be dependent on visual spatial attention.

Occipital cortex is modulated by transsaccadic changes in spatial frequency: an fMRI study

Previous neuroimaging studies have shown that inferior parietal and ventral occipital cortex are involved in the transsaccadic processing of visual object orientation. Here, we investigated whether the same areas are also involved in transsaccadic processing of a different feature, namely, spatial frequency. We employed a functional magnetic resonance imaging paradigm where participants briefly viewed a grating stimulus with a specific spatial frequency that later reappeared with the same or different frequency, after a saccade or continuous fixation. First, using a whole-brain Saccade > Fixation contrast, we localized two frontal (left precentral sulcus and right medial superior frontal gyrus), four parietal (bilateral superior parietal lobule and precuneus), and four occipital (bilateral cuneus and lingual gyri) regions. Whereas the frontoparietal sites showed task specificity, the occipital sites were also modulated in a saccade control task. Only occipital cortex showed transsaccadic feature modulations, with significant repetition enhancement in right cuneus. These observations (parietal task specificity, occipital enhancement, right lateralization) are consistent with previous transsaccadic studies. However, the specific regions differed (ventrolateral for orientation, dorsomedial for spatial frequency). Overall, this study supports a general role for occipital and parietal cortex in transsaccadic vision, with a specific role for cuneus in spatial frequency processing.

Stronger saccadic suppression of displacement and blanking effect in children

Humans do not notice small displacements to objects that occur during saccades, termed saccadic suppression of displacement (SSD), and this effect is reduced when a blank is introduced between the pre- and postsaccadic stimulus (Bridgeman, Hendry, & Stark, 1975; Deubel, Schneider, & Bridgeman, 1996). While these effects have been studied extensively in adults, it is unclear how these phenomena are characterized in children. A potentially related mechanism, saccadic suppression of contrast sensitivity-a prerequisite to achieve a stable percept-is stronger for children (Bruno, Brambati, Perani, & Morrone, 2006). However, the evidence for how transsaccadic stimulus displacements may be suppressed or integrated is mixed. While they can integrate basic visual feature information from an early age, they cannot integrate multisensory information (Gori, Viva, Sandini, & Burr, 2008; Nardini, Jones, Bedford, & Braddick, 2008), suggesting a failure in the ability to integrate more complex sensory information. We tested children 7 to 12 years old and adults 19 to 23 years old on their ability to perceive intrasaccadic stimulus displacements, with and without a postsaccadic blank. Results showed that children had stronger SSD than adults and a larger blanking effect. Children also had larger undershoots and more variability in their initial saccade endpoints, indicating greater intrinsic uncertainty, and they were faster in executing corrective saccades to account for these errors. Together, these results suggest that children may have a greater internal expectation or prediction of saccade error than adults; thus, the stronger SSD in children may be due to higher intrinsic uncertainty in target localization or saccade execution.

Spatiotopic and saccade-specific transsaccadic memory for object detail

The content and nature of transsaccadic memory are still a matter of debate. Brief postsaccadic target blanking was demonstrated to recover transsaccadic memory and defeat saccadic suppression of displacement. We examined whether blanking would also support transsaccadic transfer of detailed form information. Observers saccaded to a peripheral, checkerboard-like stimulus and reported whether an intrasaccadic change had occurred in its upper or lower half. On half of the trials, the stimulus was blanked for 200 ms with saccade onset. In a fixation condition, observers kept fixation but the stimulus was displaced from periphery to fixation, mimicking the retinal events of the saccade condition. Results show that stimulus blanking improves transsaccadic change detection, with performance being far superior to the retinally equivalent fixation condition. Our findings argue in favor of a remapped memory trace that can be accessed only in the blanking condition, when not being overwritten by the salient postsaccadic stimulus.

Transsaccadic integration is dominated by early, independent noise

Humans are able to integrate pre- and postsaccadic percepts of an object across saccades to maintain perceptual stability. Previous studies have used Maximum Likelihood Estimation (MLE) to determine that integration occurs in a near-optimal manner. Here, we compared three different models to investigate the mechanism of integration in more detail: an early noise model, where noise is added to the pre- and postsaccadic signals before integration occurs; a late-noise model, where noise is added to the integrated signal after integration occurs; and a temporal summation model, where integration benefits arise from the longer transsaccadic presentation duration compared to pre- and postsaccadic presentation only. We also measured spatiotemporal aspects of integration to determine whether integration can occur for very brief stimulus durations, across two hemifields, and in spatiotopic and retinotopic coordinates. Pre-, post-, and transsaccadic performance was measured at different stimulus presentation durations, both at the saccade target and a location where the pre- and postsaccadic stimuli were presented in different hemifields across the saccade. Results showed that for both within- and between-hemifields conditions, integration could occur when pre- and postsaccadic stimuli were presented only briefly, and that the pattern of integration followed an early noise model. Whereas integration occurred when the pre- and post-saccadic stimuli were presented in the same spatiotopic coordinates, there was no integration when they were presented in the same retinotopic coordinates. This contrast suggests that transsaccadic integration is limited by early, independent, sensory noise acting separately on pre- and postsaccadic signals.

The spatial and temporal properties of attentional selectivity for saccades and reaches

The preparation and execution of saccades and goal-directed movements elicits an accompanying shift in attention at the locus of the impending movement. However, some key aspects of the spatiotemporal profile of this attentional shift between eye and hand movements are not resolved. While there is evidence that attention is improved at the target location when making a reach, it is not clear how attention shifts over space and time around the movement target as a saccade and a reach are made to that target. Determining this spread of attention is an important aspect in understanding how attentional resources are used in relation to movement planning and guidance in real world tasks. We compared performance on a perceptual discrimination paradigm during a saccade-alone task, reach-alone task, and a saccade-plus-reach task to map the temporal profile of the premotor attentional shift at the goal of the movement and at three surrounding locations. We measured performance relative to a valid baseline level to determine whether motor planning induces additional attentional facilitation compared to mere covert attention. Sensitivity increased relative to movement onset at the target and at the surrounding locations, for both the saccade-alone and saccade-plus-reach conditions. The results suggest that the temporal profile of the attentional shift is similar for the two tasks involving saccades (saccade-alone and saccade-plus-reach tasks), but is very different when the influence of the saccade is removed. In this case, performance in the saccade-plus-reach task reflects the lower sensitivity observed when a reach-alone task is being conducted. In addition, the spatial profile of this spread of attention is not symmetrical around the target. This suggests that when a saccade and reach are being planned together, the saccade drives the attentional shift, and the reach-alone carries little attentional weight.

Working memory in action: inspecting the systematic and unsystematic errors of spatial memory across saccades

Our ability to interact with the world depends on memory buffers that flexibly store and process information for short periods of time. Current working memory research, however, mainly uses tasks that avoid eye movements, whereas in daily life we need to remember information across saccades. Because saccades disrupt perception and attention, the brain might use special transsaccadic memory systems. Therefore, to compare working memory systems between and across saccades, the current study devised transsaccadic memory tasks that evaluated the influence of memory load on several kinds of systematic and unsystematic spatial errors, and tested whether these measures predicted performance in more established working memory paradigms. Experiment 1 used a line intersection task that had people integrate lines shown before and after saccades, and it administered a 2-back task. Experiments 2 and 3 asked people to point at one of several locations within a memory array flashed before an eye movement, and we tested change detection and 2-back performance. We found that unsystematic transsaccadic errors increased with memory load and were correlated with 2-back performance. Systematic errors produced similar results, although effects varied as a function of the geometric layout of the memory arrays. Surprisingly, transsaccadic errors did not predict change detection performance despite the latter being a widely accepted measure of working memory capacity. Our results suggest that working memory systems between and across saccades share, in part, similar neural resources. Nevertheless, our data highlight the importance of investigating working memory across saccades.

Transsaccadic integration benefits are not limited to the saccade target

Across saccades, humans can integrate the low-resolution presaccadic information of an upcoming saccade target with the high-resolution postsaccadic information. There is converging evidence to suggest that transsaccadic integration occurs at the saccade target. However, given divergent evidence on the spatial specificity of related mechanisms such as attention, visual working memory, and remapping, it is unclear whether integration is also possible at locations other than the saccade target. We tested the spatial profile of transsaccadic integration, by testing perceptual performance at six locations around the saccade target and between the saccade target and initial fixation. Results show that integration benefits do not differ between the saccade target and surrounding locations. Transsaccadic integration benefits are not specific to the saccade target and can occur at other locations when they are behaviorally relevant, although there is a trend for worse performance for the location above initial fixation compared with those in the direction of the saccade. This suggests that transsaccadic integration may be a more general mechanism used to reconcile task-relevant pre- and postsaccadic information at attended locations other than the saccade target.

NEW & NOTEWORTHY This study shows that integration of pre- and postsaccadic information across saccades is not restricted to the saccade target. We found performance benefits of transsaccadic integration at attended locations other than the saccade target, and these benefits did not differ from those found at the saccade target. This suggests that transsaccadic integration may be a more general mechanism used to reconcile pre- and postsaccadic information at task-relevant locations.

Predictive remapping of visual features beyond saccadic targets

Visual stability is thought to be mediated by predictive remapping of the relevant object information from its current, presaccadic location to its future, postsaccadic location on the retina. However, it is heavily debated whether and what feature information is predictively remapped during the presaccadic interval. Here we examined the spatial and featural properties of predictive remapping in a set of three psychophysical studies. We made use of an orientation-adaptation paradigm, in which we induced a tilt aftereffect by prolonged exposure to an oriented adaptor stimulus. Following this adaptation phase, a test stimulus was presented shortly before saccade onset. We found strong evidence for predictive remapping of the features of this test stimulus presented shortly before saccade onset, evidenced by a large tilt aftereffect elicited when the adaptor was positioned at the postsaccadic retinal location of the test stimulus. Conversely, the adaptation state itself, caused by the exposure to the adaptor stimulus, was not predictively remapped. Furthermore, we establish that predictive remapping also occurs for stimuli that are not saccade targets, pointing toward a forward remapping process operating across the whole visual field. Together, our findings suggest that predictive feature remapping of object information plays an important role in mediating visual stability.

Calibration of peripheral perception of shape with and without saccadic eye movements

The cortical representations of a visual object differ radically across saccades. Several studies claim that the visual system adapts the peripheral percept to better match the subsequent foveal view. Recently, Herwig, Weiß, and Schneider (2015, Annals of the New York Academy of Sciences, 1339(1), 97-105) found that the perception of shape demonstrates a saccade-dependent learning effect. Here, we ask whether this learning actually requires saccades. We replicated Herwig et al.'s (2015) study and introduced a fixation condition. In a learning phase, participants were exposed to objects whose shape systematically changed during a saccade, or during a displacement from peripheral to foveal vision (without a saccade). In a subsequent test, objects were perceived as less (more) curved if they previously changed from more circular (triangular) in the periphery to more triangular (circular) in the fovea. Importantly, this pattern was seen both with and without saccades. We then tested whether a variable delay between the presentations of the peripheral and foveal objects would affect their association-hypothetically weakening it at longer delays. Again, we found that shape judgments depended on the changes experienced during the learning phase and that they were similar in both the saccade and fixation conditions. Surprisingly, they were not affected by the delay between the peripheral and foveal presentations over the range we tested. These results suggest that a general associative process, independent of saccade execution, contributes to the perception of shape across viewpoints.

Functions of Memory Across Saccadic Eye Movements

Several times per second, humans make rapid eye movements called saccades which redirect their gaze to sample new regions of external space. Saccades present unique challenges to both perceptual and motor systems. During the movement, the visual input is smeared across the retina and severely degraded. Once completed, the projection of the world onto the retina has undergone a large-scale spatial transformation. The vector of this transformation, and the new orientation of the eye in the external world, is uncertain. Memory for the pre-saccadic visual input is thought to play a central role in compensating for the disruption caused by saccades. Here, we review evidence that memory contributes to (1) detecting and identifying changes in the world that occur during a saccade, (2) bridging the gap in input so that visual processing does not have to start anew, and (3) correcting saccade errors and recalibrating the oculomotor system to ensure accuracy of future saccades. We argue that visual working memory (VWM) is the most likely candidate system to underlie these behaviours and assess the consequences of VWM's strict resource limitations for transsaccadic processing. We conclude that a full understanding of these processes will require progress on broader unsolved problems in psychology and neuroscience, in particular how the brain solves the object correspondence problem, to what extent prior beliefs influence visual perception, and how disparate signals arriving with different delays are integrated.

Optimal trans-saccadic integration relies on visual working memory

Saccadic eye movements alter the visual processing of objects of interest by bringing them from the periphery, where there is only low-resolution vision, to the high-resolution fovea. Evidence suggests that people are able to achieve trans-saccadic integration in a near-optimal manner; however the mechanisms underlying integration are still unclear. Visual working memory (VWM) is sustained across a saccade, and it has been suggested that this memory resource is used to store and compare the pre- and post- saccadic percepts. This study directly tested the hypothesis that VWM is necessary for optimal trans-saccadic integration, by introducing memory load during a saccade, and testing subsequent integration performance on feature similar and dissimilar stimuli. Results show that integration performance was impaired when there was an additional memory task. Additionally, performance on the memory task was affected by feature-specific integration stimuli. Our results suggest that VWM supports the integration of pre- and post- saccadic stimuli because integration performance is impaired under VWM load.

Decoding Trans-Saccadic Memory

We examine whether peripheral information at a planned saccade target affects immediate postsaccadic processing at the fovea on saccade landing. Current neuroimaging research suggests that presaccadic stimulation has a late effect on postsaccadic processing, in contrast to the early effect seen in behavioral studies. Human participants (both male and female) were instructed to saccade toward a face or a house that, on different trials, remained the same, changed, or disappeared during the saccade. We used a multivariate pattern analysis of electroencephalography data to decode face versus house processing directly after the saccade. The classifier was trained on separate trials without a saccade, where a house or face was presented at the fovea. When the saccade target remained the same across the saccade, we could reliably decode the target 123 ms after saccade offset. In contrast, when the target was changed during the saccade, the new target was decoded at a later time-point, 151 ms after saccade offset. The "same" condition advantage suggests that congruent presaccadic information facilitates processing of the postsaccadic stimulus compared with incongruent information. Finally, the saccade target could be decoded above chance even when it had been removed during the saccade, albeit with a slower time course (162 ms) and poorer signal strength. These findings indicate that information about the (peripheral) presaccadic stimulus is transferred across the saccade so that it becomes quickly available and influences processing at its expected new retinal position (the fovea).SIGNIFICANCE STATEMENT Here we provide neural evidence for early information transfer across saccades. Specifically, we examined the effect of presaccadic sensory information on the initial neuronal processing of a postsaccadic stimuli. Using electroencephalography and multivariate pattern analysis, we found the following: (1) that the identity of the presaccadic stimulus modulated the postsaccadic latency of stimulus relevant information; and (2) that a saccadic neural marker for a saccade target stimulus could be detected even when the stimulus had been removed during saccade. These results demonstrate that information about the peripheral presaccadic stimulus was transferred across the saccade and influenced processing at a new retinal position (the fovea) directly after the saccade landed.

Attention modulates trans-saccadic integration

With every saccade, humans must reconcile the low resolution peripheral information available before a saccade, with the high resolution foveal information acquired after the saccade. While research has shown that we are able to integrate peripheral and foveal vision in a near-optimal manner, it is still unclear which mechanisms may underpin this important perceptual process. One potential mechanism that may moderate this integration process is visual attention. Pre-saccadic attention is a well documented phenomenon, whereby visual attention shifts to the location of an upcoming saccade before the saccade is executed. While it plays an important role in other peri-saccadic processes such as predictive remapping, the role of attention in the integration process is as yet unknown. This study aimed to determine whether the presentation of an attentional distractor during a saccade impaired trans-saccadic integration, and to measure the time-course of this impairment. Results showed that presenting an attentional distractor impaired integration performance both before saccade onset, and during the saccade, in selected subjects who showed integration in the absence of a distractor. This suggests that visual attention may be a mechanism that facilitates trans-saccadic integration.

Object-location binding across a saccade: A retinotopic spatial congruency bias

Despite frequent eye movements that rapidly shift the locations of objects on our retinas, our visual system creates a stable perception of the world. To do this, it must convert eye-centered (retinotopic) input to world-centered (spatiotopic) percepts. Moreover, for successful behavior we must also incorporate information about object features/identities during this updating - a fundamental challenge that remains to be understood. Here we adapted a recent behavioral paradigm, the "spatial congruency bias," to investigate object-location binding across an eye movement. In two initial baseline experiments, we showed that the spatial congruency bias was present for both gabor and face stimuli in addition to the object stimuli used in the original paradigm. Then, across three main experiments, we found the bias was preserved across an eye movement, but only in retinotopic coordinates: Subjects were more likely to perceive two stimuli as having the same features/identity when they were presented in the same retinotopic location. Strikingly, there was no evidence of location binding in the more ecologically relevant spatiotopic (world-centered) coordinates; the reference frame did not update to spatiotopic even at longer post-saccade delays, nor did it transition to spatiotopic with more complex stimuli (gabors, shapes, and faces all showed a retinotopic congruency bias). Our results suggest that object-location binding may be tied to retinotopic coordinates, and that it may need to be re-established following each eye movement rather than being automatically updated to spatiotopic coordinates.

Spatiotopic updating across saccades revealed by spatially-specific fMRI adaptation

Brain representations of visual space are predominantly eye-centred (retinotopic) yet our experience of the world is largely world-centred (spatiotopic). A long-standing question is how the brain creates continuity between these reference frames across successive eye movements (saccades). Here we use functional magnetic resonance imaging (fMRI) to address whether spatially specific repetition suppression (RS) is evident during trans-saccadic perception. We presented two successive Gabor patches (S1 and S2) in either the upper or lower visual field, left or right of fixation. Spatial congruency was manipulated by having S1 and S2 occur in the same or different upper/lower visual field. On half the trials, a saccade was cued between S1 and S2, placing spatiotopic and retinotopic reference frames in opposition. Equivalent RS was observed in the posterior parietal cortex and frontal eye fields when S1-S2 were spatiotopically congruent, irrespective of whether retinotopic and spatiotopic coordinates were in accord or were placed in opposition by a saccade. Additionally the post-saccadic response to S2 demonstrated spatially-specific RS in retinotopic visual regions, with stronger RS in extrastriate than striate cortex. Collectively, these results are consistent with a robust trans-saccadic spatial updating mechanism for object position that directly influences even the earliest levels of visual processing.

Trans-saccadic interactions in human parietal and occipital cortex during the retention and comparison of object orientation

The cortical sites for the trans-saccadic storage and integration of visual object features are unknown. Here, we used a variant of fMRI-Adaptation where subjects fixated to the left or right of a briefly presented visual grating, maintained fixation or saccaded to the opposite side, then judged whether a re-presented grating had the same or different orientation. fMRI analysis revealed trans-saccadic interactions (different > same orientation) in a visual field-insensitive cluster within right supramarginal gyrus. This cluster was located at the anterolateral pole of the parietal eye field (identified in a localizer task). We also observed gaze centered, field-specific interactions (same > different orientation) in an extrastriate cluster overlapping with putative 'V4'. Based on these data and our literature review, we conclude that these supramarginal and extrastriate areas are involved in the retention, spatial updating, and evaluation of object orientation information across saccades.

No Evidence for Automatic Remapping of Stimulus Features or Location Found with fMRI

The input to our visual system shifts every time we move our eyes. To maintain a stable percept of the world, visual representations must be updated with each saccade. Near the time of a saccade, neurons in several visual areas become sensitive to the regions of visual space that their receptive fields occupy after the saccade. This process, known as remapping, transfers information from one set of neurons to another, and may provide a mechanism for visual stability. However, it is not clear whether remapping transfers information about stimulus features in addition to information about stimulus location. To investigate this issue, we recorded blood-oxygen-level dependent (BOLD) functional magnetic resonance imaging (fMRI) responses while human subjects viewed images of faces and houses (two visual categories with many feature differences). Immediately after some image presentations, subjects made a saccade that moved the previously stimulated location to the opposite side of the visual field. We then used a combination of univariate analyses and multivariate pattern analyses to test whether information about stimulus location and stimulus features were remapped to the ipsilateral hemisphere after the saccades. We found no reliable indication of stimulus feature remapping in any region. However, we also found no reliable indication of stimulus location remapping, despite the fact that our paradigm was highly similar to previous fMRI studies of remapping. The absence of location remapping in our study precludes strong conclusions regarding feature remapping. However, these results also suggest that measurement of location remapping with fMRI depends strongly on the details of the experimental paradigm used. We highlight differences in our approach from the original fMRI studies of remapping, discuss potential reasons for the failure to generalize prior location remapping results, and suggest directions for future research.

Nonretinotopic visual processing in the brain

A basic principle in visual neuroscience is the retinotopic organization of neural receptive fields. Here, we review behavioral, neurophysiological, and neuroimaging evidence for nonretinotopic processing of visual stimuli. A number of behavioral studies have shown perception depending on object or external-space coordinate systems, in addition to retinal coordinates. Both single-cell neurophysiology and neuroimaging have provided evidence for the modulation of neural firing by gaze position and processing of visual information based on craniotopic or spatiotopic coordinates. Transient remapping of the spatial and temporal properties of neurons contingent on saccadic eye movements has been demonstrated in visual cortex, as well as frontal and parietal areas involved in saliency/priority maps, and is a good candidate to mediate some of the spatial invariance demonstrated by perception. Recent studies suggest that spatiotopic selectivity depends on a low spatial resolution system of maps that operates over a longer time frame than retinotopic processing and is strongly modulated by high-level cognitive factors such as attention. The interaction of an initial and rapid retinotopic processing stage, tied to new fixations, and a longer lasting but less precise nonretinotopic level of visual representation could underlie the perception of both a detailed and a stable visual world across saccadic eye movements.

The relationship between visual attention and visual working memory encoding: A dissociation between covert and overt orienting

There is substantial debate over whether visual working memory (VWM) and visual attention constitute a single system for the selection of task-relevant perceptual information or whether they are distinct systems that can be dissociated when their representational demands diverge. In the present study, we focused on the relationship between visual attention and the encoding of objects into VWM. Participants performed a color change-detection task. During the retention interval, a secondary object, irrelevant to the memory task, was presented. Participants were instructed either to execute an overt shift of gaze to this object (Experiments 1-3) or to attend it covertly (Experiments 4 and 5). Our goal was to determine whether these overt and covert shifts of attention disrupted the information held in VWM. We hypothesized that saccades, which typically introduce a memorial demand to bridge perceptual disruption, would lead to automatic encoding of the secondary object. However, purely covert shifts of attention, which introduce no such demand, would not result in automatic memory encoding. The results supported these predictions. Saccades to the secondary object produced substantial interference with VWM performance, but covert shifts of attention to this object produced no interference with VWM performance. These results challenge prevailing theories that consider attention and VWM to reflect a common mechanism. In addition, they indicate that the relationship between attention and VWM is dependent on the memorial demands of the orienting behavior. (PsycINFO Database Record

Transsaccadic processing: stability, integration, and the potential role of remapping

While our frequent saccades allow us to sample the complex visual environment in a highly efficient manner, they also raise certain challenges for interpreting and acting upon visual input. In the present, selective review, we discuss key findings from the domains of cognitive psychology, visual perception, and neuroscience concerning two such challenges: (1) maintaining the phenomenal experience of visual stability despite our rapidly shifting gaze, and (2) integrating visual information across discrete fixations. In the first two sections of the article, we focus primarily on behavioral findings. Next, we examine the possibility that a neural phenomenon known as predictive remapping may provide an explanation for aspects of transsaccadic processing. In this section of the article, we delineate and critically evaluate multiple proposals about the potential role of predictive remapping in light of both theoretical principles and empirical findings.

The effects of TMS over dorsolateral prefrontal cortex on trans-saccadic memory of multiple objects

Humans typically make several rapid eye movements (saccades) per second. It is thought that visual working memory can retain and spatially integrate three to four objects or features across each saccade but little is known about this neural mechanism. Previously we showed that transcranial magnetic stimulation (TMS) to the posterior parietal cortex and frontal eye fields degrade trans-saccadic memory of multiple object features (Prime, Vesia, & Crawford, 2008, Journal of Neuroscience, 28(27), 6938-6949; Prime, Vesia, & Crawford, 2010, Cerebral Cortex, 20(4), 759-772.). Here, we used a similar protocol to investigate whether dorsolateral prefrontal cortex (DLPFC), an area involved in spatial working memory, is also involved in trans-saccadic memory. Subjects were required to report changes in stimulus orientation with (saccade task) or without (fixation task) an eye movement in the intervening memory interval. We applied single-pulse TMS to left and right DLPFC during the memory delay, timed at three intervals to arrive approximately 100 ms before, 100 ms after, or at saccade onset. In the fixation task, left DLPFC TMS produced inconsistent results, whereas right DLPFC TMS disrupted performance at all three intervals (significantly for presaccadic TMS). In contrast, in the saccade task, TMS consistently facilitated performance (significantly for left DLPFC/perisaccadic TMS and right DLPFC/postsaccadic TMS) suggesting a dis-inhibition of trans-saccadic processing. These results are consistent with a neural circuit of trans-saccadic memory that overlaps and interacts with, but is partially separate from the circuit for visual working memory during sustained fixation.

Shape selectivity and remapping in dorsal stream visual area LIP

We explore the visual world by making rapid eye movements (saccades) to focus on objects and locations of interest. Despite abrupt retinal image shifts, we see the world as stable. Remapping contributes to visual stability by updating the internal image with every saccade. Neurons in macaque lateral intraparietal cortex (LIP) and other brain areas update information about salient locations around the time of a saccade. The depth of information transfer remains to be thoroughly investigated. Area LIP, as part of the dorsal visual stream, is regarded as a spatially selective area, yet there is evidence that LIP neurons also encode object features. We sought to determine whether LIP remaps shape information. This knowledge is important for understanding what information is retained from each glance. We identified 82 remapping neurons. First, we presented shapes within the receptive field and tested for shape selectivity in a fixation task. Among the remapping neurons, 28 neurons (34%) were selective for shape. Second, we presented the same shapes in the future location of the receptive field around the time of the saccade and tested for shape selectivity during remapping. Thirty-one (38%) neurons were selective for shape. Of 11 neurons that were shape selective in both tasks, 5 showed significant correlation between shape selectivity in the two tasks. Across the population, there was a weak but significant correlation between responses to shape in the two tasks. Our results provide neurophysiological evidence that remapped responses in area LIP can encode shape information as well as spatial information.

The parietal cortex in sensemaking: the dissociation of multiple types of spatial information

According to the data-frame theory, sensemaking is a macrocognitive process in which people try to make sense of or explain their observations by processing a number of explanatory structures called frames until the observations and frames become congruent. During the sensemaking process, the parietal cortex has been implicated in various cognitive tasks for the functions related to spatial and temporal information processing, mathematical thinking, and spatial attention. In particular, the parietal cortex plays important roles by extracting multiple representations of magnitudes at the early stages of perceptual analysis. By a series of neural network simulations, we demonstrate that the dissociation of different types of spatial information can start early with a rather similar structure (i.e., sensitivity on a common metric), but accurate representations require specific goal-directed top-down controls due to the interference in selective attention. Our results suggest that the roles of the parietal cortex rely on the hierarchical organization of multiple spatial representations and their interactions. The dissociation and interference between different types of spatial information are essentially the result of the competition at different levels of abstraction.

Right-hemispheric dominance for visual remapping in humans

We review evidence showing a right-hemispheric dominance for visuo-spatial processing and representation in humans. Accordingly, visual disorganization symptoms (intuitively related to remapping impairments) are observed in both neglect and constructional apraxia. More specifically, we review findings from the intervening saccade paradigm in humans--and present additional original data--which suggest a specific role of the asymmetrical network at the temporo-parietal junction (TPJ) in the right hemisphere in visual remapping: following damage to the right dorsal posterior parietal cortex (PPC) as well as part of the corpus callosum connecting the PPC to the frontal lobes, patient OK in a double-step saccadic task exhibited an impairment when the second saccade had to be directed rightward. This singular and lateralized deficit cannot result solely from the patient's cortical lesion and, therefore, we propose that it is due to his callosal lesion that may specifically interrupt the interhemispheric transfer of information necessary to execute accurate rightward saccades towards a remapped target location. This suggests a specialized right-hemispheric network for visuo-spatial remapping that subsequently transfers target location information to downstream planning regions, which are symmetrically organized.

Spatial constancy mechanisms in motor control

The success of the human species in interacting with the environment depends on the ability to maintain spatial stability despite the continuous changes in sensory and motor inputs owing to movements of eyes, head and body. In this paper, I will review recent advances in the understanding of how the brain deals with the dynamic flow of sensory and motor information in order to maintain spatial constancy of movement goals. The first part summarizes studies in the saccadic system, showing that spatial constancy is governed by a dynamic feed-forward process, by gaze-centred remapping of target representations in anticipation of and across eye movements. The subsequent sections relate to other oculomotor behaviour, such as eye-head gaze shifts, smooth pursuit and vergence eye movements, and their implications for feed-forward mechanisms for spatial constancy. Work that studied the geometric complexities in spatial constancy and saccadic guidance across head and body movements, distinguishing between self-generated and passively induced motion, indicates that both feed-forward and sensory feedback processing play a role in spatial updating of movement goals. The paper ends with a discussion of the behavioural mechanisms of spatial constancy for arm motor control and their physiological implications for the brain. Taken together, the emerging picture is that the brain computes an evolving representation of three-dimensional action space, whose internal metric is updated in a nonlinear way, by optimally integrating noisy and ambiguous afferent and efferent signals.

Cortical mechanisms for trans-saccadic memory and integration of multiple object features

Constructing an internal representation of the world from successive visual fixations, i.e. separated by saccadic eye movements, is known as trans-saccadic perception. Research on trans-saccadic perception (TSP) has been traditionally aimed at resolving the problems of memory capacity and visual integration across saccades. In this paper, we review this literature on TSP with a focus on research showing that egocentric measures of the saccadic eye movement can be used to integrate simple object features across saccades, and that the memory capacity for items retained across saccades, like visual working memory, is restricted to about three to four items. We also review recent transcranial magnetic stimulation experiments which suggest that the right parietal eye field and frontal eye fields play a key functional role in spatial updating of objects in TSP. We conclude by speculating on possible cortical mechanisms for governing egocentric spatial updating of multiple objects in TSP.

Visual stability

Our vision remains stable even though the movements of our eyes, head and bodies create a motion pattern on the retina. One of the most important, yet basic, feats of the visual system is to correctly determine whether this retinal motion is owing to real movement in the world or rather our own self-movement. This problem has occupied many great thinkers, such as Descartes and Helmholtz, at least since the time of Alhazen. This theme issue brings together leading researchers from animal neurophysiology, clinical neurology, psychophysics and cognitive neuroscience to summarize the state of the art in the study of visual stability. Recently, there has been significant progress in understanding the limits of visual stability in humans and in identifying many of the brain circuits involved in maintaining a stable percept of the world. Clinical studies and new experimental methods, such as transcranial magnetic stimulation, now make it possible to test the causal role of different brain regions in creating visual stability and also allow us to measure the consequences when the mechanisms of visual stability break down.

Motion and tilt aftereffects occur largely in retinal, not in object, coordinates in the Ternus-Pikler display

Recent studies have shown that a variety of aftereffects occurs in a non-retinotopic frame of reference. These findings have been taken as strong evidence that remapping of visual information occurs in a hierarchic manner in the human cortex with an increasing magnitude from early to higher levels. Other studies, however, failed to find non-retinotopic aftereffects. These experiments all relied on paradigms involving eye movements. Recently, we have developed a new paradigm, based on the Ternus-Pikler display, which tests retinotopic vs. non-retinotopic processing without the involvement of eye movements. Using this paradigm, we found strong evidence that attention, form, and motion processing can occur in a non-retinotopic frame of reference. Here, we show that motion and tilt aftereffects are largely retinotopic.

Spatiotopic coding and remapping in humans

How our perceptual experience of the world remains stable and continuous in the face of continuous rapid eye movements still remains a mystery. This review discusses some recent progress towards understanding the neural and psychophysical processes that accompany these eye movements. We firstly report recent evidence from imaging studies in humans showing that many brain regions are tuned in spatiotopic coordinates, but only for items that are actively attended. We then describe a series of experiments measuring the spatial and temporal phenomena that occur around the time of saccades, and discuss how these could be related to visual stability. Finally, we introduce the concept of the spatio-temporal receptive field to describe the local spatiotopicity exhibited by many neurons when the eyes move.

Interactions between gaze-centered and allocentric representations of reach target location in the presence of spatial updating

Numerous studies have investigated the phenomenon of egocentric spatial updating in gaze-centered coordinates, and some have studied the use of allocentric cues in visually-guided movement, but it is not known how these two mechanisms interact. Here, we tested whether gaze-centered and allocentric information combine at the time of viewing the target, or if the brain waits until the last possible moment. To do this, we took advantage of the well-known fact that pointing and reaching movements show gaze-centered 'retinal magnification' errors (RME) that update across saccades. During gaze fixation, we found that visual landmarks, and hence allocentric information, reduces RME for targets in the left visual hemifield but not in the right. When a saccade was made between viewing and reaching, this landmark-induced reduction in RME only depended on gaze at reach, not at encoding. Based on this finding, we argue that egocentric-allocentric combination occurs after the intervening saccade. This is consistent with previous findings in healthy and brain damaged subjects suggesting that the brain updates early spatial representations during eye movement and combines them at the time of action.

Summation of visual motion across eye movements reflects a nonspatial decision mechanism

Human vision remains perceptually stable even though retinal inputs change rapidly with each eye movement. Although the neural basis of visual stability remains unknown, a recent psychophysical study pointed to the existence of visual feature-representations anchored in environmental rather than retinal coordinates (e.g., "spatiotopic" receptive fields; Melcher and Morrone, 2003). In that study, sensitivity to a moving stimulus presented after a saccadic eye movement was enhanced when preceded by another moving stimulus at the same spatial location before the saccade. The finding is consistent with spatiotopic sensory integration, but it could also have arisen from a probabilistic improvement in performance due to the presence of more than one motion signal for the perceptual decision. Here we show that this statistical advantage accounts completely for summation effects in this task. We first demonstrate that measurements of summation are confounded by noise related to an observer's uncertainty about motion onset times. When this uncertainty is minimized, comparable summation is observed regardless of whether two motion signals occupy the same or different locations in space, and whether they contain the same or opposite directions of motion. These results are incompatible with the tuning properties of motion-sensitive sensory neurons and provide no evidence for a spatiotopic representation of visual motion. Instead, summation in this context reflects a decision mechanism that uses abstract representations of sensory events to optimize choice behavior.

Cue Reliability and a Landmark Stability Heuristic Determine Relative Weighting Between Egocentric and Allocentric Visual Information in Memory-Guided Reach

It is not known how egocentric visual information (location of a target relative to the self) and allocentric visual information (location of a target relative to external landmarks) are integrated to form reach plans. Based on behavioral data from rodents and humans we hypothesized that the degree of stability in visual landmarks would influence the relative weighting. Furthermore, based on numerous cue-combination studies we hypothesized that the reach system would act like a maximum-likelihood estimator (MLE), where the reliability of both cues determines their relative weighting. To predict how these factors might interact we developed an MLE model that weighs egocentric and allocentric information based on their respective reliabilities, and also on an additional stability heuristic. We tested the predictions of this model in 10 human subjects by manipulating landmark stability and reliability (via variable amplitude vibration of the landmarks and variable amplitude gaze shifts) in three reach-to-touch tasks: an egocentric control (reaching without landmarks), an allocentric control (reaching relative to landmarks), and a cue-conflict task (involving a subtle landmark “shift” during the memory interval). Variability from all three experiments was used to derive parameters for the MLE model, which was then used to simulate egocentric–allocentric weighting in the cue-conflict experiment. As predicted by the model, landmark vibration—despite its lack of influence on pointing variability (and thus allocentric reliability) in the control experiment—had a strong influence on egocentric–allocentric weighting. A reduced model without the stability heuristic was unable to reproduce this effect. These results suggest heuristics for extrinsic cue stability are at least as important as reliability for determining cue weighting in memory-guided reaching.

TMS over human frontal eye fields disrupts trans-saccadic memory of multiple objects

We recently showed that transcranial magnetic stimulation (TMS) over the right parietal eye fields disrupts memory of object features and locations across saccades. We applied TMS over the frontal eye fields (FEF) as subjects compared the feature details of visual targets presented either within a single eye fixation (Fixation Task) or across a saccade (Saccade Task). TMS pulses were randomly delivered at one of 3 time intervals around the time of the saccade, or at equivalent times in the Fixation Task. A No-TMS control confirmed that subjects could normally retain approximately 3 visual features. TMS in the Fixation Task had no effect compared with No-TMS, but differences among TMS times were found during right FEF stimulation. TMS over either the right or left FEF disrupted memory performance in the Saccade Task when stimulation coincided most closely with the saccade. The capacity to compare pre-and postsaccadic features was reduced to 1-2 objects, as expected if the spatial aspect of memory was disrupted. These findings suggest that the FEF plays a role in the spatial processing involved in trans-saccadic memory of visual features. We propose that this process employs saccade-related feedback signals similar to those observed in spatial updating.

The role of peripheral vision in saccade planning: learning from people with tunnel vision

Both visually salient and top-down information are important in eye movement control, but their relative roles in the planning of daily saccades are unclear. We investigated the effect of peripheral vision loss on saccadic behaviors in patients with tunnel vision (visual field diameters 7 degrees-16 degrees) in visual search and real-world walking experiments. The patients made up to two saccades per second to their pre-saccadic blind areas, about half of which had no overlap between the post- and pre-saccadic views. In the visual search experiment, visual field size and the background (blank or picture) did not affect the saccade sizes and direction of patients (n = 9). In the walking experiment, the patients (n = 5) and normal controls (n = 3) had similar distributions of saccade sizes and directions. These findings might provide a clue about the large extent of the top-down mechanism influence on eye movement control.

Spatial updating and the maintenance of visual constancy

Spatial updating is the means by which we keep track of the locations of objects in space even as we move. Four decades of research have shown that humans and non-human primates can take the amplitude and direction of intervening movements into account, including saccades (both head-fixed and head-free), pursuit, whole-body rotations and translations. At the neuronal level, spatial updating is thought to be maintained by receptive field locations that shift with changes in gaze, and evidence for such shifts has been shown in several cortical areas. These regions receive information about the intervening movement from several sources including motor efference copies when a voluntary movement is made and vestibular/somatosensory signals when the body is in motion. Many of these updating signals arise from brainstem regions that monitor our ongoing movements and subsequently transmit this information to the cortex via pathways that likely include the thalamus. Several issues of debate include (1) the relative contribution of extra-retinal sensory and efference copy signals to spatial updating, (2) the source of an updating signal for real life, three-dimensional motion that cannot arise from brain areas encoding only two-dimensional commands, and (3) the reference frames used by the brain to integrate updating signals from various sources. This review highlights the relevant spatial updating studies and provides a summary of the field today. We find that spatial constancy is maintained by a highly evolved neural mechanism that keeps track of our movements, transmits this information to relevant brain regions, and then uses this information to change the way in which single neurons respond. In this way, we are able to keep track of relevant objects in the outside world and interact with them in meaningful ways.

Transcranial magnetic stimulation over posterior parietal cortex disrupts transsaccadic memory of multiple objects

The posterior parietal cortex (PPC) plays a role in spatial updating of goals for eye and arm movements across saccades, but less is known about its role in updating perceptual memory. We reported previously that transsaccadic memory has a capacity for storing the orientations of three to four Gabor patches either within a single fixation (fixation task) or between separate fixations (saccade task). Here, we tested the role of the PPC in transsaccadic memory in eight subjects by simultaneously applying single-pulse transcranial magnetic stimulation (TMS) over the right and left PPC, over several control sites, and comparing these to behavioral controls with no TMS. In TMS trials, we randomly delivered pulses at one of three different time intervals around the time of the saccade, or at an equivalent time in the fixation task. Controls confirmed that subjects could normally retain at least three visual features. TMS over the left PPC and a control site had no significant effect on this performance. However, TMS over the right PPC disrupted memory performance in both tasks. This TMS-induced effect was most disruptive in the saccade task, in particular when stimulation coincided more closely with saccade timing. Here, the capacity to compare presaccadic and postsaccadic features was reduced to one object, as expected if the spatial aspect of memory was disrupted. This finding suggests that right PPC plays a role in the spatial processing involved in transsaccadic memory of visual features. We propose that this process uses saccade-related feedback signals similar to those observed in spatial updating.

Attention and active vision

Visual perception results from the interaction of incoming sensory signals and top down cognitive and motor signals. Here we focus on the representation of attended locations in parietal cortex and in earlier visual cortical areas. We review evidence that these spatial representations are modulated not only by selective attention but also by the intention to move the eyes. We describe recent experiments in monkey and human that elucidate the mechanisms and circuitry involved in updating, or remapping, the representations of salient stimuli. Two central ideas emerge. First, selective attention and remapping are closely intertwined, and together contribute to the percept of spatial stability. Second, remapping is accomplished not by a single area but by the participation of parietal, frontal and extrastriate cortex as well as subcortical structures. This neural circuitry is distinguished by significant redundancy and plasticity, suggesting that the updating of salient stimuli is fundamental for spatial stability and visuospatial behavior. We conclude that multiple processes and pathways contribute to active vision in the primate brain.

Visual memory capacity in transsaccadic integration

How we perceive the visual world as stable and unified suggests the existence of transsaccadic integration that retains and integrates visual information from one eye fixation to another eye fixation across saccadic eye movements. However, the capacity of transsaccadic integration is still a subject of controversy. We tested our subjects' memory capacity of two basic visual features, i.e. luminance (Experiment 1) and orientation (Experiment 2), both within a single fixation (i.e. visual working memory) and between separate fixations (i.e. transsaccadic memory). Experiment 2 was repeated, but attention allocation was manipulated using attentional cues at either the target or distracter (Experiment 3). Subjects were able to retain 3-4 objects in transsaccadic memory for luminance and orientation; errors generally increased as saccade size increased; and, subjects were more accurate when attention was allocated to the same location as the impending target. These results were modelled by inputting a noisy extra-retinal signal into an eye-centered feature map. Our results suggest that transsaccadic memory has a similar capacity for storing simple visual features as basic visual memory, but this capacity is dependent both on the metrics of the saccade and allocation of attention.

Gaze fixation patterns for negotiating complex ground terrain

We constantly encounter different ground terrain in our environment that we must safely traverse. The visual system is unique, as it is the only sensory system that can provide accurate and precise information about the environment at a distance through a series of fixations directed to salient objects and/or surfaces. However, how the nervous system utilizes visual information regarding complex ground terrain to guide safe foot placement is not known. We had individuals walk across a walkway with varying ground terrain while gaze fixations were monitored. Several findings emerged. First, gaze fixations were highly task-relevant in that they were predominantly made to areas eventually stepped on and their patterns tended to depend on the task instructions. Second, fixations were frequently directed to a transition region between different surfaces in addition to fixations directed to an actual surface. These results suggest that fixations are directed to regions that maximize the amount of information which the nervous system can integrate in order to facilitate safe foot placement. And third, spatial information of the upcoming ground terrain was sampled sequentially in small sections and continuously updated as the individual traversed the challenging ground terrain. This is suggestive of on-line control and may be beneficial to ensure one is able to adapt to stability concerns, unexpected changes in terrain, or sudden changes in the path taken.

Optimal inference explains dimension-specific contractions of spatial perception

It is known that people misperceive scenes they see during rapid eye movements called saccades. It has been suggested that some of these misperceptions could be an artifact of neurophysiological processes related to the internal remapping of spatial coordinates during saccades. Alternatively, we have recently suggested, based on a computational model, that transsaccadic misperceptions result from optimal inference. As one of the properties of the model, sudden object displacements that occur in sync with a saccade should be perceived as contracted in a non-linear fashion. To explore this model property, here we use computer simulations and psychophysical methods first to test how robust the model is to close-to-optimal approximations and second to test two model predictions: (a) contracted transsaccadic perception should be dimension-specific with more contraction for jumps parallel to the saccade than orthogonal to it, and (b) contraction should rise as a function of visuomotor noise. Our results are consistent with these predictions. They support the idea that human transsaccadic integration is governed by close-to-optimal inference.

Remapping in human visual cortex

With each eye movement, stationary objects in the world change position on the retina, yet we perceive the world as stable. Spatial updating, or remapping, is one neural mechanism by which the brain compensates for shifts in the retinal image caused by voluntary eye movements. Remapping of a visual representation is believed to arise from a widespread neural circuit including parietal and frontal cortex. The current experiment tests the hypothesis that extrastriate visual areas in human cortex have access to remapped spatial information. We tested this hypothesis using functional magnetic resonance imaging (fMRI). We first identified the borders of several occipital lobe visual areas using standard retinotopic techniques. We then tested subjects while they performed a single-step saccade task analogous to the task used in neurophysiological studies in monkeys, and two conditions that control for visual and motor effects. We analyzed the fMRI time series data with a nonlinear, fully Bayesian hierarchical statistical model. We identified remapping as activity in the single-step task that could not be attributed to purely visual or oculomotor effects. The strength of remapping was roughly monotonic with position in the visual hierarchy: remapped responses were largest in areas V3A and hV4 and smallest in V1 and V2. These results demonstrate that updated visual representations are present in cortical areas that are directly linked to visual perception.