Urgent Decision Making: Resolving Visuomotor Interactions at High Temporal Resolution

Location- and feature-based selection histories make independent, qualitatively distinct contributions to urgent visuomotor performance

Attention mechanisms guide visuomotor behavior by weighing physical salience and internal goals to prioritize stimuli as choices for action. Although less well studied, selection history, which reflects multiple facets of experience with recent events, is increasingly recognized as a distinct source of attentional bias. To examine how selection history impacts saccadic choices, we trained two macaque monkeys to perform an urgent version of an oddball search task in which a red target appeared among three green distracters, or vice versa. By imposing urgency, performance could be tracked continuously as it transitioned from uninformed guesses to informed choices as a function of processing time. This, in turn, permitted assessment of attentional control as manifest in motor biases, processing speed, and asymptotic accuracy. Here, we found that the probability of making a correct choice was strongly modulated by the histories of preceding target locations and target colors. Crucially, although both effects were gated by success (or reward), their dynamics were clearly distinct: whereas location history promoted a motor bias, color history modulated perceptual sensitivity, and these influences acted independently. Thus, combined selection histories can give rise to enormous swings in visuomotor performance even in simple tasks with highly discriminable stimuli.

Dissociable Effects of Urgency and Evidence Accumulation during Reaching Revealed by Dynamic Multisensory Integration

When making perceptual decisions, humans combine information across sensory modalities dependent on their respective uncertainties. However, it remains unknown how the brain integrates multisensory feedback during movement and which factors besides sensory uncertainty influence sensory contributions. We performed two reaching experiments on healthy adults to investigate whether movement corrections to combined visual and mechanical perturbations scale with visual uncertainty. To describe the dynamics of multimodal feedback responses, we further varied movement time and visual feedback duration during the movement. The results of our first experiment show that the contribution of visual feedback decreased with uncertainty. Additionally, we observed a transient phase during which visual feedback responses were stronger during faster movements. In a follow-up experiment, we found that the contribution of vision increased more quickly during slow movements when we presented the visual feedback for a longer time. Muscle activity corresponding to these visual responses exhibited modulations with sensory uncertainty and movement speed ca. 100 ms following the onset of the visual feedback. Using an optimal feedback control model, we show that the increased response to visual feedback during fast movements can be explained by an urgency-dependent increase in control gains. Further, the fact that a longer viewing duration increased the visual contributions suggests that the brain accumulates sensory information over time to estimate the state of the arm during reaching. Our results provide additional evidence concerning the link between reaching control and decision-making, both of which appear to be influenced by sensory evidence accumulation and response urgency.

Rapid eye and hand responses in an interception task are differentially modulated by context-dependent predictability

When catching a falling ball or avoiding a collision with traffic, humans can quickly generate eye and limb responses to unpredictable changes in their environment. Mechanisms of limb and oculomotor control when responding to sudden changes in the environment have mostly been investigated independently. Here, we investigated eye-hand coordination in a rapid interception task where human participants used a virtual paddle to intercept a moving target. The target moved vertically down a computer screen and could suddenly jump to the left or right. In high-certainty blocks, the target always jumped; in low-certainty blocks, the target only jumped in a portion of the trials. Further, we manipulated response urgency by varying the time of target jumps, with early jumps requiring less urgent responses and late jumps requiring more urgent responses. Our results highlight differential effects of certainty and urgency on eye-hand coordination. Participants initiated both eye and hand responses earlier for high-certainty compared with low-certainty blocks. Hand reaction times decreased and response vigor increased with increasing urgency levels. However, eye reaction times were lowest for medium-urgency levels and eye vigor was unaffected by urgency. Across all trials, we found a weak positive correlation between eye and hand responses. Taken together, these results suggest that the limb and oculomotor systems use similar early sensorimotor processing; however, rapid responses are modulated differentially to attain system-specific sensorimotor goals.

Coupling of saccade plans to endogenous attention during urgent choices

The neural mechanisms that willfully direct attention to specific locations in space are closely related to those for generating targeting eye movements (saccades). However, the degree to which the voluntary deployment of attention to a location necessarily activates a corresponding saccade plan remains unclear. One problem is that attention and saccades are both automatically driven by salient sensory events; another is that the underlying processes unfold within tens of milliseconds only. Here, we use an urgent task design to resolve the evolution of a visuomotor choice on a moment-by-moment basis while independently controlling the endogenous (goal-driven) and exogenous (salience-driven) contributions to performance. Human participants saw a peripheral cue and, depending on its color, either looked at it (prosaccade) or looked at a diametrically opposite, uninformative non-cue (antisaccade). By varying the luminance of the stimuli, the exogenous contributions could be cleanly dissociated from the endogenous process guiding the choice over time. According to the measured time courses, generating a correct antisaccade requires about 30 ms more processing time than generating a correct prosaccade based on the same perceptual signal. The results indicate that saccade plans elaborated during fixation are biased toward the location where attention is endogenously deployed, but the coupling is weak and can be willfully overridden very rapidly.

Coupling of saccade plans to endogenous attention during urgent choices

The neural mechanisms that willfully direct attention to specific locations in space are closely related to those for generating targeting eye movements (saccades). However, the degree to which the voluntary deployment of attention to a location necessarily activates a corresponding saccade plan remains unclear. One problem is that attention and saccades are both automatically driven by salient sensory events; another is that the underlying processes unfold within tens of milliseconds only. Here, we use an urgent task design to resolve the evolution of a visuomotor choice on a moment-by-moment basis while independently controlling the endogenous (goal-driven) and exogenous (salience-driven) contributions to performance. Human participants saw a peripheral cue and, depending on its color, either looked at it (prosaccade) or looked at a diametrically opposite, uninformative non-cue (antisaccade). By varying the luminance of the stimuli, the exogenous contributions could be cleanly dissociated from the endogenous process guiding the choice over time. According to the measured timecourses, generating a correct antisaccade requires about 30 ms more processing time than generating a correct prosaccade based on the same perceptual signal. The results indicate that saccade plans elaborated during fixation are biased toward the location where attention is endogenously deployed, but the coupling is weak and can be willfully overridden very rapidly.

Always expect the unexpected: eye position modulates visual cortex excitability in a stimulus-free environment

Stimuli that potentially require a rapid defensive or avoidance action can appear from the periphery at any time in natural environments. de Wit et al. (Cortex 127: 120-130, 2020) recently reported novel evidence suggestive of a fundamental neural mechanism that allows organisms to effectively deal with such situations. In the absence of any task, motor cortex excitability was found to be greater whenever gaze was directed away from either hand. If modulation of cortical excitability as a function of gaze location is a fundamental principle of brain organization, then one would expect its operation to be present outside of motor cortex, including brain regions involved in perception. To test this hypothesis, we applied single-pulse transcranial magnetic stimulation (TMS) to the right lateral occipital lobe while participants directed their eyes to the left, straight-ahead, or to the right, and reported the presence or absence of a phosphene. No external stimuli were presented. Cortical excitability as reflected by the proportion of trials on which phosphenes were elicited from stimulation of the right visual cortex was greater with eyes deviated to the right as compared with the left. In conjunction with our previous findings of change in motor cortex excitability when gaze and effector are not aligned, this eye position-driven change in visual cortex excitability presumably serves to facilitate the detection of stimuli and subsequent readiness to act in nonfoveated regions of space. The existence of this brain-wide mechanism has clear adaptive value given the unpredictable nature of natural environments in which human beings are situated and have evolved.NEW & NOTEWORTHY For many complex tasks, humans focus attention on the site relevant to the task at hand. Humans evolved and live in dangerous environments, however, in which threats arise from outside the attended site; this fact necessitates a process by which the periphery is monitored. Using single-pulse transcranial magnetic stimulation (TMS), we demonstrated for the first time that eye position modulates visual cortex excitability. We argue that this underlies at least in part what we term "surveillance attention."

Stimulus salience conflicts and colludes with endogenous goals during urgent choices

Selecting where to look next depends on both the salience of objects and current goals (what we are looking for), but discerning their relative contributions over the time frame of typical visuomotor decisions (200-250 ms) has been difficult. Here we investigate this problem using an urgent choice task with which the two contributions can be dissociated and tracked moment by moment. Behavioral data from three monkeys corresponded with model-based predictions: when salience favored the target, perceptual performance evolved rapidly and steadily toward an asymptotic level; when salience favored the distracter, many rapid errors were produced and the rise in performance took more time-effects analogous to oculomotor and attentional capture. The results show that salience has a brief (∼50 ms) but inexorable impact that leads to exogenous, involuntary capture, and this can either help or hinder performance, depending on the alignment between salience and ongoing internal goals.

Exogenous capture accounts for fundamental differences between pro- and antisaccade performance

To generate the next eye movement, oculomotor circuits take into consideration the physical salience of objects in view and current behavioral goals, exogenous and endogenous influences, respectively. However, the interactions between exogenous and endogenous mechanisms and their dynamic contributions to target selection have been difficult to resolve because they evolve extremely rapidly. In a recent study (Salinas et al., 2019), we achieved the necessary temporal precision using an urgent variant of the antisaccade task wherein motor plans are initiated early and choice accuracy depends sharply on when exactly the visual cue information becomes available. Empirical and modeling results indicated that the exogenous signal arrives ∼80 ms after cue onset and rapidly accelerates the (incorrect) plan toward the cue, whereas the informed endogenous signal arrives ∼25 ms later to favor the (correct) plan away from the cue. Here, we scrutinize a key mechanistic hypothesis about this dynamic, that the exogenous and endogenous signals act at different times and independently of each other. We test quantitative model predictions by comparing the performance of human participants instructed to look toward a visual cue or away from it under high urgency. We find that, indeed, the exogenous response is largely impervious to task instructions; it simply flips its sign relative to the correct choice, and this largely explains the drastic differences in psychometric performance between the two tasks. Thus, saccadic choices are strongly dictated by the alignment between salience and behavioral goals.

Eye Movements as a Window into Decision-Making

For over 100 years, eye movements have been studied and used as indicators of human sensory and cognitive functions. This review evaluates how eye movements contribute to our understanding of the processes that underlie decision-making. Eye movement metrics signify the visual and task contexts in which information is accumulated and weighed. They indicate the efficiency with which we evaluate the instructions for decision tasks, the timing and duration of decision formation, the expected reward associated with a decision, the accuracy of the decision outcome, and our ability to predict and feel confident about a decision. Because of their continuous nature, eye movements provide an exciting opportunity to probe decision processes noninvasively in real time. Expected final online publication date for the Annual Review of Vision Science, Volume 8 is September 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Urgency forces stimulus-driven action by overcoming cognitive control

Intelligent behavior requires to act directed by goals despite competing action tendencies triggered by stimuli in the environment. For eye movements, it has recently been discovered that this ability is briefly reduced in urgent situations (Salinas et al., 2019). In a time-window before an urgent response, participants could not help but look at a suddenly appearing visual stimulus, even though their goal was to look away from it. Urgency seemed to provoke a new visual-oculomotor phenomenon: A period in which saccadic eye movements are dominated by external stimuli, and uncontrollable by current goals. This period was assumed to arise from brain mechanisms controlling eye movements and spatial attention, such as those of the frontal eye field. Here, we show that the phenomenon is more general than previously thought. We found that also in well-investigated manual tasks, urgency made goal-conflicting stimulus features dominate behavioral responses. This dominance of behavior followed established trial-to-trial signatures of cognitive control mechanisms that replicate across a variety of tasks. Thus together, these findings reveal that urgency temporarily forces stimulus-driven action by overcoming cognitive control in general, not only at brain mechanisms controlling eye movements.

Under time pressure, the exogenous modulation of saccade plans is ubiquitous, intricate, and lawful

The choice of where to look next is determined by both exogenous (bottom-up) and endogenous (top-down) factors, but details of their interaction and distinct contributions to target selection have remained elusive. Recent experiments with urgent choice tasks, in which stimuli are evaluated while motor plans are already advancing, have greatly clarified these contributions. Specifically, exogenous modulations associated with stimulus detection act rapidly and briefly (∼25 ms) to automatically halt and/or boost ongoing motor plans as per spatial congruence rules. These stereotypical modulations explain, in quantitative detail, characteristic features of many saccadic tasks (e.g. antisaccade, countermanding, saccadic-inhibition, gap, and double-step). Thus, the same low-level visuomotor interactions contribute to diverse oculomotor phenomena traditionally attributed to different neural mechanisms.