Continuous updating of visuospatial memory in superior colliculus during slow eye movements

Updating the remembered position of targets following passive lateral translation

Spatial updating, the ability to track the egocentric position of surrounding objects during self-motion, is fundamental to navigating around the world. However, people make systematic errors when updating the position of objects after linear self-motion. To determine the source of these errors, we measured errors in remembered target position with or without passive lateral translations. Self-motion was presented both visually (simulated in virtual reality) and physically (on a 6-DOF motion platform). People underestimated targets' eccentricity in general even when just asked to remember them for a few seconds (5-7 seconds), with larger underestimations of more eccentric targets. We hypothesized that updating errors would depend on target eccentricity, which was manifested as errors depending not only on target eccentricity but also the observer's movement range. When updating the position of targets within the range of movement (such that their actual locations crossed the viewer's midline), people overestimated their change in position relative to their head/body compared to when judging the location of objects that were outside the range of movement and therefore did not cross the midline. We interpret these results as revealing changes in the efficacy of spatial updating depending on participant's perception of self-motion and the perceptual consequences for targets represented initially in one half of the visual field having to be reconstructed in the opposite hemifield.

Multiunit Frontal Eye Field Activity Codes the Visuomotor Transformation, But Not Gaze Prediction or Retrospective Target Memory, in a Delayed Saccade Task

Single-unit (SU) activity-action potentials isolated from one neuron-has traditionally been employed to relate neuronal activity to behavior. However, recent investigations have shown that multiunit (MU) activity-ensemble neural activity recorded within the vicinity of one microelectrode-may also contain accurate estimations of task-related neural population dynamics. Here, using an established model-fitting approach, we compared the spatial codes of SU response fields with corresponding MU response fields recorded from the frontal eye fields (FEFs) in head-unrestrained monkeys (Macaca mulatta) during a memory-guided saccade task. Overall, both SU and MU populations showed a simple visuomotor transformation: the visual response coded target-in-eye coordinates, transitioning progressively during the delay toward a future gaze-in-eye code in the saccade motor response. However, the SU population showed additional secondary codes, including a predictive gaze code in the visual response and retention of a target code in the motor response. Further, when SUs were separated into regular/fast spiking neurons, these cell types showed different spatial code progressions during the late delay period, only converging toward gaze coding during the final saccade motor response. Finally, reconstructing MU populations (by summing SU data within the same sites) failed to replicate either the SU or MU pattern. These results confirm the theoretical and practical potential of MU activity recordings as a biomarker for fundamental sensorimotor transformations (e.g., target-to-gaze coding in the oculomotor system), while also highlighting the importance of SU activity for coding more subtle (e.g., predictive/memory) aspects of sensorimotor behavior.

Perceptual-Cognitive Integration for Goal-Directed Action in Naturalistic Environments

Real-world actions require one to simultaneously perceive, think, and act on the surrounding world, requiring the integration of (bottom-up) sensory information and (top-down) cognitive and motor signals. Studying these processes involves the intellectual challenge of cutting across traditional neuroscience silos, and the technical challenge of recording data in uncontrolled natural environments. However, recent advances in techniques, such as neuroimaging, virtual reality, and motion tracking, allow one to address these issues in naturalistic environments for both healthy participants and clinical populations. In this review, we survey six topics in which naturalistic approaches have advanced both our fundamental understanding of brain function and how neurologic deficits influence goal-directed, coordinated action in naturalistic environments. The first part conveys fundamental neuroscience mechanisms related to visuospatial coding for action, adaptive eye-hand coordination, and visuomotor integration for manual interception. The second part discusses applications of such knowledge to neurologic deficits, specifically, steering in the presence of cortical blindness, impact of stroke on visual-proprioceptive integration, and impact of visual search and working memory deficits. This translational approach-extending knowledge from lab to rehab-provides new insights into the complex interplay between perceptual, motor, and cognitive control in naturalistic tasks that are relevant for both basic and clinical research.

Working memory control predicts fixation duration in scene-viewing

When viewing scenes, observers differ in how long they linger at each fixation location and how far they move their eyes between fixations. What factors drive these differences in eye-movement behaviors? Previous work suggests individual differences in working memory capacity may influence fixation durations and saccade amplitudes. In the present study, participants (N = 98) performed two scene-viewing tasks, aesthetic judgment and memorization, while viewing 100 photographs of real-world scenes. Working memory capacity, working memory processing ability, and fluid intelligence were assessed with an operation span task, a memory updating task, and Raven's Advanced Progressive Matrices, respectively. Across participants, we found significant effects of task on both fixation durations and saccade amplitudes. At the level of each individual participant, we also found a significant relationship between memory updating task performance and participants' fixation duration distributions. However, we found no effect of fluid intelligence and no effect of working memory capacity on fixation duration or saccade amplitude distributions, inconsistent with previous findings. These results suggest that the ability to flexibly maintain and update working memory is strongly related to fixation duration behavior.

Mnemonic representations in human lateral geniculate nucleus

There is a growing appreciation for the role of the thalamus in high-level cognition. Motivated by findings that internal cognitive state drives activity in feedback layers of primary visual cortex (V1) that target the lateral geniculate nucleus (LGN), we investigated the role of LGN in working memory (WM). Specifically, we leveraged model-based neuroimaging approaches to test the hypothesis that human LGN encodes information about spatial locations temporarily encoded in WM. First, we localized and derived a detailed topographic organization in LGN that accords well with previous findings in humans and non-human primates. Next, we used models constructed on the spatial preferences of LGN populations in order to reconstruct spatial locations stored in WM as subjects performed modified memory-guided saccade tasks. We found that population LGN activity faithfully encoded the spatial locations held in memory in all subjects. Importantly, our tasks and models allowed us to dissociate the locations of retinal stimulation and the motor metrics of memory-guided saccades from the maintained spatial locations, thus confirming that human LGN represents true WM information. These findings add LGN to the growing list of subcortical regions involved in WM, and suggest a key pathway by which memories may influence incoming processing at the earliest levels of the visual hierarchy.

Persistent Activity During Working Memory From Front to Back

Working memory (WM) extends the duration over which information is available for processing. Given its importance in supporting a wide-array of high level cognitive abilities, uncovering the neural mechanisms that underlie WM has been a primary goal of neuroscience research over the past century. Here, we critically review what we consider the two major "arcs" of inquiry, with a specific focus on findings that were theoretically transformative. For the first arc, we briefly review classic studies that led to the canonical WM theory that cast the prefrontal cortex (PFC) as a central player utilizing persistent activity of neurons as a mechanism for memory storage. We then consider recent challenges to the theory regarding the role of persistent neural activity. The second arc, which evolved over the last decade, stemmed from sophisticated computational neuroimaging approaches enabling researchers to decode the contents of WM from the patterns of neural activity in many parts of the brain including early visual cortex. We summarize key findings from these studies, their implications for WM theory, and finally the challenges these findings pose. Our goal in doing so is to identify barriers to developing a comprehensive theory of WM that will require a unification of these two "arcs" of research.

Confidence in predicted position error explains saccadic decisions during pursuit

A fundamental problem in motor control is the coordination of complementary movement types to achieve a common goal. As a common example, humans view moving objects through coordinated pursuit and saccadic eye movements. Pursuit is initiated and continuously controlled by retinal image velocity. During pursuit, eye position may lag behind the target. This can be compensated by the discrete execution of a catch-up saccade. The decision to trigger a saccade is influenced by both position and velocity errors, and the timing of saccades can be highly variable. The observed distributions of saccade frequency and trigger time remain poorly understood, and this decision process remains imprecisely quantified. Here, we propose a predictive, probabilistic model explaining the decision to trigger saccades during pursuit to foveate moving targets. In this model, expected position error and its associated uncertainty are predicted through Bayesian inference across noisy, delayed sensory observations (Kalman filtering). This probabilistic prediction is used to estimate the confidence that a saccade is needed (quantified through log-probability ratio), triggering a saccade upon accumulating to a fixed threshold. The model qualitatively explains behavioral observations on the frequency and trigger time distributions of saccades during pursuit over a range of target motion trajectories. Furthermore, this model makes novel predictions that saccade decisions are highly sensitive to uncertainty for small predicted position errors, but this influence diminishes as the magnitude of predicted position error increases. We suggest that this predictive, confidence-based decision-making strategy represents a fundamental principle for the probabilistic neural control of coordinated movements.NEW & NOTEWORTHY This is the first stochastic dynamical systems model of pursuit-saccade coordination accounting for noise and delays in the sensorimotor system. The model uses Bayesian inference to predictively estimate visual motion, triggering saccades when confidence in predicted position error accumulates to a threshold. This model explains saccade frequency and trigger time distributions across target trajectories and makes novel predictions about the influence of sensory uncertainty in saccade decisions during pursuit.

Spatially Specific Working Memory Activity in the Human Superior Colliculus

Theoretically, working memory (WM) representations are encoded by population activity of neurons with distributed tuning across the stored feature. Here, we leverage computational neuroimaging approaches to map the topographic organization of human superior colliculus (SC) and model how population activity in SC encodes WM representations. We first modeled receptive field properties of voxels in SC, deriving a detailed topographic organization resembling that of the primate SC. Neural activity within human (5 male and 1 female) SC persisted throughout a retention interval of several types of modified memory-guided saccade tasks. Assuming an underlying neural architecture of the SC based on its retinotopic organization, we used an encoding model to show that the pattern of activity in human SC represents locations stored in WM. Our tasks and models allowed us to dissociate the locations of visual targets and the motor metrics of memory-guided saccades from the spatial locations stored in WM, thus confirming that human SC represents true WM information. These data have several important implications. They add the SC to a growing number of cortical and subcortical brain areas that form distributed networks supporting WM functions. Moreover, they specify a clear neural mechanism by which topographically organized SC encodes WM representations.

SIGNIFICANCE STATEMENT Using computational neuroimaging approaches, we mapped the topographic organization of human superior colliculus (SC) and modeled how population activity in SC encodes working memory (WM) representations, rather than simpler visual or motor properties that have been traditionally associated with the laminar maps in the primate SC. Together, these data both position the human SC into a distributed network of brain areas supporting WM and elucidate the neural mechanisms by which the SC supports WM.

Spatiotemporal transformations for gaze control

Sensorimotor transformations require spatiotemporal coordination of signals, that is, through both time and space. For example, the gaze control system employs signals that are time-locked to various sensorimotor events, but the spatial content of these signals is difficult to assess during ordinary gaze shifts. In this review, we describe the various models and methods that have been devised to test this question, and their limitations. We then describe a new method that can (a) simultaneously test between all of these models during natural, head-unrestrained conditions, and (b) track the evolving spatial continuum from target (T) to future gaze coding (G, including errors) through time. We then summarize some applications of this technique, comparing spatiotemporal coding in the primate frontal eye field (FEF) and superior colliculus (SC). The results confirm that these areas preferentially encode eye-centered, effector-independent parameters, and show-for the first time in ordinary gaze shifts-a spatial transformation between visual and motor responses from T to G coding. We introduce a new set of spatial models (T-G continuum) that revealed task-dependent timing of this transformation: progressive during a memory delay between vision and action, and almost immediate without such a delay. We synthesize the results from our studies and supplement it with previous knowledge of anatomy and physiology to propose a conceptual model where cumulative transformation noise is realized as inaccuracies in gaze behavior. We conclude that the spatiotemporal transformation for gaze is both local (observed within and across neurons in a given area) and distributed (with common signals shared across remote but interconnected structures).

Timing Determines Tuning: A Rapid Spatial Transformation in Superior Colliculus Neurons during Reactive Gaze Shifts

Gaze saccades, rapid shifts of the eyes and head toward a goal, have provided fundamental insights into the neural control of movement. For example, it has been shown that the superior colliculus (SC) transforms a visual target (T) code to future gaze (G) location commands after a memory delay. However, this transformation has not been observed in "reactive" saccades made directly to a stimulus, so its contribution to normal gaze behavior is unclear. Here, we tested this using a quantitative measure of the intermediate codes between T and G, based on variable errors in gaze endpoints. We demonstrate that a rapid spatial transformation occurs within the primate's SC (Macaca mulatta) during reactive saccades, involving a shift in coding from T, through intermediate codes, to G. This spatial shift progressed continuously both across and within cell populations [visual, visuomotor (VM), motor], rather than relaying discretely between populations with fixed spatial codes. These results suggest that the SC produces a rapid, noisy, and distributed transformation that contributes to variable errors in reactive gaze shifts.

Superior colliculus encodes visual saliency during smooth pursuit eye movements

The saliency map has played a long-standing role in models and theories of visual attention, and it is now supported by neurobiological evidence from several cortical and subcortical brain areas. While visual saliency is computed during moments of active fixation, it is not known whether the same is true while engaged in smooth pursuit of a moving stimulus, which is very common in real-world vision. Here, we examined extrafoveal saliency coding in the superior colliculus, a midbrain area associated with attention and gaze, during smooth pursuit eye movements. We found that SC neurons from the superficial visual layers showed a robust representation of peripheral saliency evoked by a conspicuous stimulus embedded in a wide-field array of goal-irrelevant stimuli. In contrast, visuomotor neurons from the intermediate saccade-related layers showed a poor saliency representation, even though most of these neurons were visually responsive during smooth pursuit. These results confirm and extend previous findings that place the SCs in a unique role as a saliency map that monitors peripheral vision during foveation of stationary and now moving objects.

Autism Pathogenesis: The Superior Colliculus

After been exposed to the visual input, in the first year of life, the brain experiences subtle but massive changes apparently crucial for communicative/emotional and social human development. Its lack could be the explanation of the very high prevalence of autism in children with total congenital blindness. The present theory postulates that the superior colliculus is the key structure for such changes for several reasons: it dominates visual behavior during the first months of life; it is ready at birth for complex visual tasks; it has a significant influence on several hemispheric regions; it is the main brain hub that permanently integrates visual and non-visual, external and internal information (bottom-up and top-down respectively); and it owns the enigmatic ability to take non-conscious decisions about where to focus attention. It is also a sentinel that triggers the subcortical mechanisms which drive social motivation to follow faces from birth and to react automatically to emotional stimuli. Through indirect connections it also activates simultaneously several cortical structures necessary to develop social cognition and to accomplish the multiattentional task required for conscious social interaction in real life settings. Genetic or non-genetic prenatal or early postnatal factors could disrupt the SC functions resulting in autism. The timing of postnatal biological disruption matches the timing of clinical autism manifestations. Astonishing coincidences between etiologies, clinical manifestations, cognitive and pathogenic autism theories on one side and SC functions on the other are disclosed in this review. Although the visual system dependent of the SC is usually considered as accessory of the LGN canonical pathway, its imprinting gives the brain a qualitatively specific functions not supplied by any other brain structure.

Optogenetic auditory fMRI reveals the effects of visual cortical inputs on auditory midbrain response

Sensory cortices contain extensive descending (corticofugal) pathways, yet their impact on brainstem processing - particularly across sensory systems - remains poorly understood. In the auditory system, the inferior colliculus (IC) in the midbrain receives cross-modal inputs from the visual cortex (VC). However, the influences from VC on auditory midbrain processing are unclear. To investigate whether and how visual cortical inputs affect IC auditory responses, the present study combines auditory blood-oxygenation-level-dependent (BOLD) functional MRI (fMRI) with cell-type specific optogenetic manipulation of visual cortex. The results show that predominant optogenetic excitation of the excitatory pyramidal neurons in the infragranular layers of the primary VC enhances the noise-evoked BOLD fMRI responses within the IC. This finding reveals that inputs from VC influence and facilitate basic sound processing in the auditory midbrain. Such combined optogenetic and auditory fMRI approach can shed light on the large-scale modulatory effects of corticofugal pathways and guide detailed electrophysiological studies in the future.

Spatial localization of sound elicits early responses from occipital visual cortex in humans

Much evidence points to an interaction between vision and audition at early cortical sites. However, the functional role of these interactions is not yet understood. Here we show an early response of the occipital cortex to sound that it is strongly linked to the spatial localization task performed by the observer. The early occipital response to a sound, usually absent, increased by more than 10-fold when presented during a space localization task, but not during a time localization task. The response amplification was not only specific to the task, but surprisingly also to the position of the stimulus in the two hemifields. We suggest that early occipital processing of sound is linked to the construction of an audio spatial map that may utilize the visual map of the occipital cortex.

Object files across eye movements: Previous fixations affect the latencies of corrective saccades

One of the factors contributing to a seamless visual experience is object correspondence-that is, the integration of pre- and postsaccadic visual object information into one representation. Previous research had suggested that before the execution of a saccade, a target object is loaded into visual working memory and subsequently is used to locate the target object after the saccade. Until now, studies on object correspondence have not taken previous fixations into account. In the present study, we investigated the influence of previously fixated information on object correspondence. To this end, we adapted a gaze correction paradigm in which a saccade was executed toward either a previously fixated or a novel target. During the saccade, the stimuli were displaced such that the participant's gaze landed between the target stimulus and a distractor. Participants then executed a corrective saccade to the target. The results indicated that these corrective saccades had lower latencies toward previously fixated than toward nonfixated targets, indicating object-specific facilitation. In two follow-up experiments, we showed that presaccadic spatial and object (surface feature) information can contribute separately to the execution of a corrective saccade, as well as in conjunction. Whereas the execution of a corrective saccade to a previously fixated target object at a previously fixated location is slowed down (i.e., inhibition of return), corrective saccades toward either a previously fixated target object or a previously fixated location are facilitated. We concluded that corrective saccades are executed on the basis of object files rather than of unintegrated feature information.

Neonatal imitation in context: Sensorimotor development in the perinatal period

More than 35 years ago, Meltzoff and Moore (1977) published their famous article, “Imitation of facial and manual gestures by human neonates.” Their central conclusion, that neonates can imitate, was and continues to be controversial. Here, we focus on an often-neglected aspect of this debate, namely, neonatal spontaneous behaviors themselves. We present a case study of a paradigmatic orofacial “gesture,” namely tongue protrusion and retraction (TP/R). Against the background of new research on mammalian aerodigestive development, we ask: How does the human aerodigestive system develop, and what role does TP/R play in the neonate's emerging system of aerodigestion? We show that mammalian aerodigestion develops in two phases: (1) from the onset of isolated orofacial movementsin uteroto the postnatal mastery of suckling at 4 months after birth; and (2) thereafter, from preparation to the mastery of mastication and deglutition of solid foods. Like other orofacial stereotypies, TP/R emerges in the first phase and vanishes prior to the second. Based upon recent advances in activity-driven early neural development, we suggest a sequence of three developmental events in which TP/R might participate: the acquisition of tongue control, the integration of the central pattern generator (CPG) for TP/R with other aerodigestive CPGs, and the formation of connections within the cortical maps of S1 and M1. If correct, orofacial stereotypies are crucial to the maturation of aerodigestion in the neonatal period but also unlikely to co-occur with imitative behavior.

A State Space Model for Spatial Updating of Remembered Visual Targets during Eye Movements

In the oculomotor system, spatial updating is the ability to aim a saccade toward a remembered visual target position despite intervening eye movements. Although this has been the subject of extensive experimental investigation, there is still no unifying theoretical framework to explain the neural mechanism for this phenomenon, and how it influences visual signals in the brain. Here, we propose a unified state-space model (SSM) to account for the dynamics of spatial updating during two types of eye movement; saccades and smooth pursuit. Our proposed model is a non-linear SSM and implemented through a recurrent radial-basis-function neural network in a dual Extended Kalman filter (EKF) structure. The model parameters and internal states (remembered target position) are estimated sequentially using the EKF method. The proposed model replicates two fundamental experimental observations: continuous gaze-centered updating of visual memory-related activity during smooth pursuit, and predictive remapping of visual memory activity before and during saccades. Moreover, our model makes the new prediction that, when uncertainty of input signals is incorporated in the model, neural population activity and receptive fields expand just before and during saccades. These results suggest that visual remapping and motor updating are part of a common visuomotor mechanism, and that subjective perceptual constancy arises in part from training the visual system on motor tasks.

Superior Colliculus Responses to Attended, Unattended, and Remembered Saccade Targets during Smooth Pursuit Eye Movements

In realistic environments, keeping track of multiple visual targets during eye movements likely involves an interaction between vision, top-down spatial attention, memory, and self-motion information. Recently we found that the superior colliculus (SC) visual memory response is attention-sensitive and continuously updated relative to gaze direction. In that study, animals were trained to remember the location of a saccade target across an intervening smooth pursuit (SP) eye movement (Dash et al., 2015). Here, we modified this paradigm to directly compare the properties of visual and memory updating responses to attended and unattended targets. Our analysis shows that during SP, active SC visual vs. memory updating responses share similar gaze-centered spatio-temporal profiles (suggesting a common mechanism), but updating was weaker by ~25%, delayed by ~55 ms, and far more dependent on attention. Further, during SP the sum of passive visual responses (to distracter stimuli) and memory updating responses (to saccade targets) closely resembled the responses for active attentional tracking of visible saccade targets. These results suggest that SP updating signals provide a damped, delayed estimate of attended location that contributes to the gaze-centered tracking of both remembered and visible saccade targets.

Transition from Target to Gaze Coding in Primate Frontal Eye Field during Memory Delay and Memory-Motor Transformation

The frontal eye fields (FEFs) participate in both working memory and sensorimotor transformations for saccades, but their role in integrating these functions through time remains unclear. Here, we tracked FEF spatial codes through time using a novel analytic method applied to the classic memory-delay saccade task. Three-dimensional recordings of head-unrestrained gaze shifts were made in two monkeys trained to make gaze shifts toward briefly flashed targets after a variable delay (450-1500 ms). A preliminary analysis of visual and motor response fields in 74 FEF neurons eliminated most potential models for spatial coding at the neuron population level, as in our previous study (Sajad et al., 2015). We then focused on the spatiotemporal transition from an eye-centered target code (T; preferred in the visual response) to an eye-centered intended gaze position code (G; preferred in the movement response) during the memory delay interval. We treated neural population codes as a continuous spatiotemporal variable by dividing the space spanning T and G into intermediate T–G models and dividing the task into discrete steps through time. We found that FEF delay activity, especially in visuomovement cells, progressively transitions from T through intermediate T–G codes that approach, but do not reach, G. This was followed by a final discrete transition from these intermediate T–G delay codes to a “pure” G code in movement cells without delay activity. These results demonstrate that FEF activity undergoes a series of sensory–memory–motor transformations, including a dynamically evolving spatial memory signal and an imperfect memory-to-motor transformation.

Parallel updating and weighting of multiple spatial maps for visual stability during whole body motion

It is known that the brain uses multiple reference frames to code spatial information, including eye-centered and body-centered frames. When we move our body in space, these internal representations are no longer in register with external space, unless they are actively updated. Whether the brain updates multiple spatial representations in parallel, or whether it restricts its updating mechanisms to a single reference frame from which other representations are constructed, remains an open question. We developed an optimal integration model to simulate the updating of visual space across body motion in multiple or single reference frames. To test this model, we designed an experiment in which participants had to remember the location of a briefly presented target while being translated sideways. The behavioral responses were in agreement with a model that uses a combination of eye- and body-centered representations, weighted according to the reliability in which the target location is stored and updated in each reference frame. Our findings suggest that the brain simultaneously updates multiple spatial representations across body motion. Because both representations are kept in sync, they can be optimally combined to provide a more precise estimate of visual locations in space than based on single-frame updating mechanisms.