Closed loop motor-sensory dynamics in human vision

Sub-cone visual resolution by active, adaptive sampling in the human foveola

The foveated architecture of the human retina and the eye's mobility enables prime spatial vision, yet the interplay between photoreceptor cell topography and the constant motion of the eye during fixation remains unexplored. With in vivo foveal cone-resolved imaging and simultaneous microscopic photo stimulation, we examined visual acuity in both eyes of 16 participants while precisely recording the stimulus path on the retina. We find that resolution thresholds were correlated with the individual retina's sampling capacity, and exceeded what static sampling limits would predict by 18%, on average. The length and direction of fixational drift motion, previously thought to be primarily random, played a key role in achieving this sub-cone diameter resolution. The oculomotor system finely adjusts drift behavior towards retinal areas with higher cone densities within only a few hundred milliseconds to enhance retinal sampling.

Consequences of eye movements for spatial selectivity

What determines spatial tuning in the visual system? Standard views rely on the assumption that spatial information is directly inherited from the relative position of photoreceptors and shaped by neuronal connectivity.1,2 However, human eyes are always in motion during fixation,3,4,5,6 so retinal neurons receive temporal modulations that depend on the interaction of the spatial structure of the stimulus with eye movements. It has long been hypothesized that these modulations might contribute to spatial encoding,7,8,9,10,11,12 a proposal supported by several recent observations.13,14,15,16 A fundamental, yet untested, consequence of this encoding strategy is that spatial tuning is not hard-wired in the visual system but critically depends on how the fixational motion of the eye shapes the temporal structure of the signals impinging onto the retina. Here we used high-resolution techniques for eye-tracking17 and gaze-contingent display control18 to quantitatively test this distinctive prediction. We examined how contrast sensitivity, a hallmark of spatial vision, is influenced by fixational motion, both during normal active fixation and when the spatiotemporal stimulus on the retina is altered to mimic changes in fixational control. We showed that visual sensitivity closely follows the strength of the luminance modulations delivered within a narrow temporal bandwidth, so changes in fixational motion have opposite visual effects at low and high spatial frequencies. By identifying a key role for oculomotor activity in spatial selectivity, these findings have important implications for the perceptual consequences of abnormal eye movements, the sources of perceptual variability, and the function of oculomotor control.

Cognitive influences on fixational eye movements

We perceive the world based on visual information acquired via oculomotor control,1 an activity intertwined with ongoing cognitive processes.2,3,4 Cognitive influences have been primarily studied in the context of macroscopic movements, like saccades and smooth pursuits. However, our eyes are never still, even during periods of fixation. One of the fixational eye movements, ocular drifts, shifts the stimulus over hundreds of receptors on the retina, a motion that has been argued to enhance the processing of spatial detail by translating spatial into temporal information.5 Despite their apparent randomness, ocular drifts are under neural control.6,7,8 However little is known about the control of drift beyond the brainstem circuitry of the vestibulo-ocular reflex.9,10 Here, we investigated the cognitive control of ocular drifts with a letter discrimination task. The experiment was designed to reveal open-loop effects, i.e., cognitive oculomotor control driven by specific prior knowledge of the task, independent of incoming sensory information. Open-loop influences were isolated by randomly presenting pure noise fields (no letters) while subjects engaged in discriminating specific letter pairs. Our results show open-loop control of drift direction in human observers.

Active vision: How you look reflects what you are looking for

While we fixate an object, our eyes are never stationary but constantly drifting, with miniature movements traditionally thought to be random and involuntary. A new study shows that the orientation of such drift in humans is actually not random but is influenced by the task demands to improve performance.

Inferring visual space from ultra-fine extra-retinal knowledge of gaze position

It has long been debated how humans resolve fine details and perceive a stable visual world despite the incessant fixational motion of their eyes. Current theories assume these processes to rely solely on the visual input to the retina, without contributions from motor and/or proprioceptive sources. Here we show that contrary to this widespread assumption, the visual system has access to high-resolution extra-retinal knowledge of fixational eye motion and uses it to deduce spatial relations. Building on recent advances in gaze-contingent display control, we created a spatial discrimination task in which the stimulus configuration was entirely determined by oculomotor activity. Our results show that humans correctly infer geometrical relations in the absence of spatial information on the retina and accurately combine high-resolution extraretinal monitoring of gaze displacement with retinal signals. These findings reveal a sensory-motor strategy for encoding space, in which fine oculomotor knowledge is used to interpret the fixational input to the retina.

Idiosyncratic selection of active touch for shape perception

Hand movements are essential for tactile perception of objects. However, the specific functions served by active touch strategies, and their dependence on physiological parameters, are unclear and understudied. Focusing on planar shape perception, we tracked at high resolution the hands of 11 participants during shape recognition task. Two dominant hand movement strategies were identified: contour following and scanning. Contour following movements were either tangential to the contour or oscillating perpendicular to it. Scanning movements crossed between distant parts of the shapes’ contour. Both strategies exhibited non-uniform coverage of the shapes’ contours. Idiosyncratic movement patterns were specific to the sensed object. In a second experiment, we have measured the participants’ spatial and temporal tactile thresholds. Significant portions of the variations in hand speed and in oscillation patterns could be explained by the idiosyncratic thresholds. Using data-driven simulations, we show how specific strategy choices may affect receptors activation. These results suggest that motion strategies of active touch adapt to both the sensed object and to the perceiver’s physiological parameters.

Oculo-retinal dynamics can explain the perception of minimal recognizable configurations

Natural vision is a dynamic and continuous process. Under natural conditions, visual object recognition typically involves continuous interactions between ocular motion and visual contrasts, resulting in dynamic retinal activations. In order to identify the dynamic variables that participate in this process and are relevant for image recognition, we used a set of images that are just above and below the human recognition threshold and whose recognition typically requires >2 s of viewing. We recorded eye movements of participants while attempting to recognize these images within trials lasting 3 s. We then assessed the activation dynamics of retinal ganglion cells resulting from ocular dynamics using a computational model. We found that while the saccadic rate was similar between recognized and unrecognized trials, the fixational ocular speed was significantly larger for unrecognized trials. Interestingly, however, retinal activation level was significantly lower during these unrecognized trials. We used retinal activation patterns and oculomotor parameters of each fixation to train a binary classifier, classifying recognized from unrecognized trials. Only retinal activation patterns could predict recognition, reaching 80% correct classifications on the fourth fixation (on average, ∼2.5 s from trial onset). We thus conclude that the information that is relevant for visual perception is embedded in the dynamic interactions between the oculomotor sequence and the image. Hence, our results suggest that ocular dynamics play an important role in recognition and that understanding the dynamics of retinal activation is crucial for understanding natural vision.

Active sensory substitution allows fast learning via effective motor-sensory strategies

We examined the development of new sensing abilities in adults by training participants to perceive remote objects through their fingers. Using an Active-Sensing based sensory Substitution device (ASenSub), participants quickly learned to perceive fast via the new modality and preserved their high performance for more than 20 months. Both sighted and blind participants exhibited almost complete transfer of performance from 2D images to novel 3D physical objects. Perceptual accuracy and speed using the ASenSub were, on average, 300% and 600% better than previous reports for 2D images and 3D objects. This improvement is attributed to the ability of the participants to employ their own motor-sensory strategies. Sighted participants dominant strategy was based on motor-sensory convergence on the most informative regions of objects, similarly to fixation patterns in vision. Congenitally, blind participants did not show such a tendency, and many of their exploratory procedures resembled those observed with natural touch.