Remapping attention in multiple object tracking

Attentional tracking takes place over perceived rather than veridical positions

Illusions can induce striking differences between perception and retinal input. For instance, a static Gabor with a moving internal texture appears to be shifted in the direction of its internal motion, a shift that increases dramatically when the Gabor itself is also in motion. Here, we ask whether attention operates on the perceptual or physical location of this stimulus. To do so, we generated an attentional tracking task where participants (N = 15) had to keep track of a single target among three Gabors that rotated around a common center in the periphery. During tracking, the illusion was used to make three Gabors appear either shifted away from or toward one another while maintaining the same physical separation. Because tracking performance depends in part on target to distractor spacing, if attention selects targets from perceived positions, performance should be better when the Gabors appear further apart and worse when they appear closer together. We find that tracking performance is superior with greater perceived separation, implying that attentional tracking operates over perceived rather than physical positions.

Effect of bilingualism on visual tracking attention and resistance to distraction

Speaking more than one language has been associated with enhanced cognitive capacities. Here we evaluated whether bilingual individuals have advantages in visual tracking attention. Adult bilingual (n = 35) and monolingual (n = 35) participants were tested in the Multiple Object Tracking task (MOT). In one condition, the MOT was performed by itself establishing the baseline performance of each group, and in the other condition, it was performed while participants counted backward out loud in their mother tongue. At baseline, the average speed tracking threshold of bilinguals was not better than that of the monolinguals. Importantly, for bilinguals, counting backward decreased their threshold by only 15%, but, for monolinguals, it decreased it three times as much. This result suggests that bilingualism confers advantages to visual tracking attention when dual tasking is required, extending the evidence that bilingualism affords cognitive benefits beyond verbal communication.

The coordinate system of endogenous spatial attention during smooth pursuit

A central question in vision is whether spatial attention is represented in an eye-centered (retinotopic) or world-centered (spatiotopic) reference-frame. Most previous studies on this question focused on how coordinates are modulated across saccades. In the present study, we investigated the reference-frame of attention across smooth pursuit eye-movements using a goal-directed saccade task. In two experiments, participants were asked to pursue a moving target while attending to one or two grating stimuli. On each trial, one stimulus was constant in its retinal position and the other was constant in its spatial position. Upon detection of a slight change in stimulus orientation, participants were asked to stop pursuing and perform a fast saccade toward the modified stimulus. In the focused attention condition, they attended one, predefined, stimulus, and in the divided attention condition they attended both. In Experiment 1 the angle of the orientation change marking the target event was constant across participants and conditions. In Experiment 2, the angle was individually adapted to equate performance across participants and conditions. Findings of the two experiments were consistent and showed that the enhancement of mean visual sensitivity in the focused relative to the divided attention condition was similar in magnitude for both retinotopic and spatiotopic targets. This indicates that during smooth pursuit, endogenous attention was proportionally divided between targets in retinotopic and spatiotopic frames of reference.

Decoding Trans-Saccadic Memory

We examine whether peripheral information at a planned saccade target affects immediate postsaccadic processing at the fovea on saccade landing. Current neuroimaging research suggests that presaccadic stimulation has a late effect on postsaccadic processing, in contrast to the early effect seen in behavioral studies. Human participants (both male and female) were instructed to saccade toward a face or a house that, on different trials, remained the same, changed, or disappeared during the saccade. We used a multivariate pattern analysis of electroencephalography data to decode face versus house processing directly after the saccade. The classifier was trained on separate trials without a saccade, where a house or face was presented at the fovea. When the saccade target remained the same across the saccade, we could reliably decode the target 123 ms after saccade offset. In contrast, when the target was changed during the saccade, the new target was decoded at a later time-point, 151 ms after saccade offset. The "same" condition advantage suggests that congruent presaccadic information facilitates processing of the postsaccadic stimulus compared with incongruent information. Finally, the saccade target could be decoded above chance even when it had been removed during the saccade, albeit with a slower time course (162 ms) and poorer signal strength. These findings indicate that information about the (peripheral) presaccadic stimulus is transferred across the saccade so that it becomes quickly available and influences processing at its expected new retinal position (the fovea).SIGNIFICANCE STATEMENT Here we provide neural evidence for early information transfer across saccades. Specifically, we examined the effect of presaccadic sensory information on the initial neuronal processing of a postsaccadic stimuli. Using electroencephalography and multivariate pattern analysis, we found the following: (1) that the identity of the presaccadic stimulus modulated the postsaccadic latency of stimulus relevant information; and (2) that a saccadic neural marker for a saccade target stimulus could be detected even when the stimulus had been removed during saccade. These results demonstrate that information about the peripheral presaccadic stimulus was transferred across the saccade and influenced processing at a new retinal position (the fovea) directly after the saccade landed.

Spatiotopic coding during dynamic head tilt

Humans maintain a stable representation of the visual world effortlessly, despite constant movements of the eyes, head, and body, across multiple planes. Whereas visual stability in the face of saccadic eye movements has been intensely researched, fewer studies have investigated retinal image transformations induced by head movements, especially in the frontal plane. Unlike head rotations in the horizontal and sagittal planes, tilting the head in the frontal plane is only partially counteracted by torsional eye movements and consequently induces a distortion of the retinal image to which we seem to be completely oblivious. One possible mechanism aiding perceptual stability is an active reconstruction of a spatiotopic map of the visual world, anchored in allocentric coordinates. To explore this possibility, we measured the positional motion aftereffect (PMAE; the apparent change in position after adaptation to motion) with head tilts of ∼42° between adaptation and test (to dissociate retinal from allocentric coordinates). The aftereffect was shown to have both a retinotopic and spatiotopic component. When tested with unpatterned Gaussian blobs rather than sinusoidal grating stimuli, the retinotopic component was greatly reduced, whereas the spatiotopic component remained. The results suggest that perceptual stability may be maintained at least partially through mechanisms involving spatiotopic coding.NEW & NOTEWORTHY Given that spatiotopic coding could play a key role in maintaining visual stability, we look for evidence of spatiotopic coding after retinal image transformations caused by head tilt. To this end, we measure the strength of the positional motion aftereffect (PMAE; previously shown to be largely spatiotopic after saccades) after large head tilts. We find that, as with eye movements, the spatial selectivity of the PMAE has a large spatiotopic component after head rotation.

Functional connectivity indicates differential roles for the intraparietal sulcus and the superior parietal lobule in multiple object tracking

Attentive tracking requires sustained object-based attention, rather than passive vigilance or rapid attentional shifts to brief events. Several theories of tracking suggest a mechanism of indexing objects that allows for attentional resources to be directed toward the moving targets. Imaging studies have shown that cortical areas belonging to the dorsal frontoparietal attention network increase BOLD-signal during multiple object tracking (MOT). Among these areas, some studies have assigned IPS a particular role in object indexing, but the neuroimaging evidence has been sparse. In the present study, we tested participants on a continuous version of the MOT task in order to investigate how cortical areas engage in functional networks during attentional tracking. Specifically, we analyzed the data using eigenvector centrality mapping (ECM) analysis, which provides estimates of individual voxels' connectedness with hub-like parts of the functional network. The results obtained using permutation based voxel-wise statistics support the proposed role for the IPS in object indexing as this region displayed increased centrality during tracking as well as increased functional connectivity with both prefrontal and visual perceptual cortices. In contrast, the opposite pattern was observed for the SPL, with decreasing centrality, as well as reduced functional connectivity with the visual and frontal cortices, in agreement with a hypothesized role for SPL in attentional shifts. These findings provide novel evidence that IPS and SPL serve different functional roles during MOT, while at the same time being highly engaged during tracking as measured by BOLD-signal changes.

Attentional trade-offs maintain the tracking of moving objects across saccades

In many situations like playing sports or driving a car, we keep track of moving objects, despite the frequent eye movements that drastically interrupt their retinal motion trajectory. Here we report evidence that transsaccadic tracking relies on trade-offs of attentional resources from a tracked object's motion path to its remapped location. While participants covertly tracked a moving object, we presented pulses of coherent motion at different locations to probe the allocation of spatial attention along the object's entire motion path. Changes in the sensitivity for these pulses showed that during fixation attention shifted smoothly in anticipation of the tracked object's displacement. However, just before a saccade, attentional resources were withdrawn from the object's current motion path and reflexively drawn to the retinal location the object would have after saccade. This finding demonstrates the predictive choice the visual system makes to maintain the tracking of moving objects across saccades.

Buildup of spatial information over time and across eye-movements

To interact rapidly and effectively with our environment, our brain needs access to a neural representation of the spatial layout of the external world. However, the construction of such a map poses major challenges, as the images on our retinae depend on where the eyes are looking, and shift each time we move our eyes, head and body to explore the world. Research from many laboratories including our own suggests that the visual system does compute spatial maps that are anchored to real-world coordinates. However, the construction of these maps takes time (up to 500ms) and also attentional resources. We discuss research investigating how retinotopic reference frames are transformed into spatiotopic reference-frames, and how this transformation takes time to complete. These results have implications for theories about visual space coordinates and particularly for the current debate about the existence of spatiotopic representations.

Constructing stable spatial maps of the world

To interact rapidly and effectively with our environment, our brain needs access to a neural representation--or map--of the spatial layout of the external world. However, the construction of such a map poses major challenges to the visual system, given that the images on our retinae depend on where the eyes are looking, and shift each time we move our eyes, head, and body to explore the world. Much research has been devoted to how the stability is achieved, with the debate often polarized between the utility of spatiotopic maps (that remain solid in external coordinates), as opposed to transiently updated retinotopic maps. Our research suggests that the visual system uses both strategies to maintain stability. fMRI, motion-adaptation, and saccade-adaptation studies demonstrate and characterize spatiotopic neural maps within the dorsal visual stream that remain solid in external rather than retinal coordinates. However, the construction of these maps takes time (up to 500 ms) and attentional resources. To solve the immediate problems created by individual saccades, we postulate the existence of a separate system to bridge each saccade with neural units that are 'transiently craniotopic'. These units prepare for the effects of saccades with a shift of their receptive fields before the saccade starts, then relaxing back into their standard position during the saccade, compensating for its action. Psychophysical studies investigating localization of stimuli flashed briefly around the time of saccades provide strong support for these neural mechanisms, and show quantitatively how they integrate information across saccades. This transient system cooperates with the spatiotopic mechanism to provide a useful map to guide interactions with our environment: one rapid and transitory, bringing into play the high-resolution visual areas; the other slow, long-lasting, and low-resolution, useful for interacting with the world.

A retinotopic attentional trace after saccadic eye movements: evidence from event-related potentials

Saccadic eye movements are a major source of disruption to visual stability, yet we experience little of this disruption. We can keep track of the same object across multiple saccades. It is generally assumed that visual stability is due to the process of remapping, in which retinotopically organized maps are updated to compensate for the retinal shifts caused by eye movements. Recent behavioral and ERP evidence suggests that visual attention is also remapped, but that it may still leave a residual retinotopic trace immediately after a saccade. The current study was designed to further examine electrophysiological evidence for such a retinotopic trace by recording ERPs elicited by stimuli that were presented immediately after a saccade (80 msec SOA). Participants were required to maintain attention at a specific location (and to memorize this location) while making a saccadic eye movement. Immediately after the saccade, a visual stimulus was briefly presented at either the attended location (the same spatiotopic location), a location that matched the attended location retinotopically (the same retinotopic location), or one of two control locations. ERP data revealed an enhanced P1 amplitude for the stimulus presented at the retinotopically matched location, but a significant attenuation for probes presented at the original attended location. These results are consistent with the hypothesis that visuospatial attention lingers in retinotopic coordinates immediately following gaze shifts.

Temporal dynamics of divided spatial attention

In naturalistic settings, observers often have to monitor multiple objects dispersed throughout the visual scene. However, the degree to which spatial attention can be divided across spatially noncontiguous objects has long been debated, particularly when those objects are in close proximity. Moreover, the temporal dynamics of divided attention are unclear: is the process of dividing spatial attention gradual and continuous, or does it onset in a discrete manner? To address these issues, we recorded steady-state visual evoked potentials (SSVEPs) as subjects covertly monitored two flickering targets while ignoring an intervening distractor that flickered at a different frequency. All three stimuli were clustered within either the lower left or the lower right quadrant, and our dependent measure was SSVEP power at the target and distractor frequencies measured over time. In two experiments, we observed a temporally discrete increase in power for target- vs. distractor-evoked SSVEPs extending from ∼350 to 150 ms prior to correct (but not incorrect) responses. The divergence in SSVEP power immediately prior to a correct response suggests that spatial attention can be divided across noncontiguous locations, even when the targets are closely spaced within a single quadrant. In addition, the division of spatial attention appears to be relatively discrete, as opposed to slow and continuous. Finally, the predictive relationship between SSVEP power and behavior demonstrates that these neurophysiological measures of divided attention are meaningfully related to cognitive function.

Behavioral dynamics and neural grounding of a dynamic field theory of multi-object tracking

The ability to dynamically track moving objects in the environment is crucial for efficient interaction with the local surrounds. Here, we examined this ability in the context of the multi-object tracking (MOT) task. Several theories have been proposed to explain how people track moving objects; however, only one of these previous theories is implemented in a real-time process model, and there has been no direct contact between theories of object tracking and the growing neural literature using ERPs and fMRI. Here, we present a neural process model of object tracking that builds from a Dynamic Field Theory of spatial cognition. Simulations reveal that our dynamic field model captures recent behavioral data examining the impact of speed and tracking duration on MOT performance. Moreover, we show that the same model with the same trajectories and parameters can shed light on recent ERP results probing how people distribute attentional resources to targets vs. distractors. We conclude by comparing this new theory of object tracking to other recent accounts, and discuss how the neural grounding of the theory might be effectively explored in future work.

Paying Attention through Eye Movements: A Computational Investigation of the Premotor Theory of Spatial Attention

Growing evidence indicates that planning eye movements and orienting visuospatial attention share overlapping brain mechanisms. A tight link between endogenous attention and eye movements is maintained by the premotor theory, in contrast to other accounts that postulate the existence of specific attention mechanisms that modulate the activity of information processing systems. The strong assumption of equivalence between attention and eye movements, however, is challenged by demonstrations that human observers are able to keep attention on a specific location while moving the eyes elsewhere. Here we investigate whether a recurrent model of saccadic planning can account for attentional effects without requiring additional or specific mechanisms separate from the circuits that perform sensorimotor transformations for eye movements. The model builds on the basis function approach and includes a circuit that performs spatial remapping using an “internal forward model” of how visual inputs are modified as a result of saccadic movements. Simulations show that the latter circuit is crucial to account for dissociations between attention and eye movements that may be invoked to disprove the premotor theory. The model provides new insights into how spatial remapping may be implemented in parietal cortex and offers a computational framework for recent proposals that link visual stability with remapping of attention pointers.

Spatial Reference in Multiple Object Tracking

Spatial reference in multiple object tracking is available from configurations of dynamic objects and static reference objects. In three experiments, we studied the use of spatial reference in tracking and in relocating targets after abrupt scene rotations. Observers tracked 1, 2, 3, 4, and 6 targets in 3D scenes, in which white balls moved on a square floor plane. The floor plane was either visible thus providing static spatial reference or it was invisible. Without scene rotations, the configuration of dynamic objects provided sufficient spatial reference and static spatial reference was not advantageous. In contrast, with abrupt scene rotations of 20°, static spatial reference supported in relocating targets. A wireframe floor plane lacking local visual detail was as effective as a checkerboard. Individually colored geometric forms as static reference objects provided no additional benefit either, even if targets were centered on these forms at the abrupt scene rotation. Individualizing the dynamic objects themselves by color for a brief interval around the abrupt scene rotation, however, did improve performance. We conclude that attentional tracking of moving targets proceeds within dynamic configurations but detached from static local background.