Optic flow and eye movements

Retinal optic flow during natural locomotion

We examine the structure of the visual motion projected on the retina during natural locomotion in real world environments. Bipedal gait generates a complex, rhythmic pattern of head translation and rotation in space, so without gaze stabilization mechanisms such as the vestibular-ocular-reflex (VOR) a walker’s visually specified heading would vary dramatically throughout the gait cycle. The act of fixation on stable points in the environment nulls image motion at the fovea, resulting in stable patterns of outflow on the retinae centered on the point of fixation. These outflowing patterns retain a higher order structure that is informative about the stabilized trajectory of the eye through space. We measure this structure by applying the curl and divergence operations on the retinal flow velocity vector fields and found features that may be valuable for the control of locomotion. In particular, the sign and magnitude of foveal curl in retinal flow specifies the body’s trajectory relative to the gaze point, while the point of maximum divergence in the retinal flow field specifies the walker’s instantaneous overground velocity/momentum vector in retinotopic coordinates. Assuming that walkers can determine the body position relative to gaze direction, these time-varying retinotopic cues for the body’s momentum could provide a visual control signal for locomotion over complex terrain. In contrast, the temporal variation of the eye-movement-free, head-centered flow fields is large enough to be problematic for use in steering towards a goal. Consideration of optic flow in the context of real-world locomotion therefore suggests a re-evaluation of the role of optic flow in the control of action during natural behavior.

We recorded the full body kinematics and binocular gaze of humans walking through real-world natural environment and estimated visual motion (optic flow) using both computational video analysis and geometric simulation. Contrary to the established theories of the role of optic flow in the control of locomotion, we found that eye-movement-free, head-centric optic flow is highly unstable due to the complex phasic trajectory of the head during natural locomotion, rendering it an unlikely candidate for heading perception. In contrast, retina-centered optic flow consisted of a regular pattern of outflowing motion centered on the fovea. Retinal optic flow contained highly consistent patterns that specified the walker’s trajectory relative to the point of fixation, which may provide powerful, retinotopic cues that may be used for the visual control of locomotion in natural environments. This examination of optic flow in real-world contexts suggest a need to re-evaluate existing theories of the role of optic flow in the visual control of action during natural behavior.

On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping

Visual attention is often predictive for future actions in humans. In manipulation tasks, the eyes tend to fixate an object of interest even before the reach-to-grasp is initiated. Some recent studies have proposed to exploit this anticipatory gaze behavior to improve the control of dexterous upper limb prostheses. This requires a detailed understanding of visuomotor coordination to determine in which temporal window gaze may provide helpful information. In this paper, we verify and quantify the gaze and motor behavior of 14 transradial amputees who were asked to grasp and manipulate common household objects with their missing limb. For comparison, we also include data from 30 able-bodied subjects who executed the same protocol with their right arm. The dataset contains gaze, first person video, angular velocities of the head, and electromyography and accelerometry of the forearm. To analyze the large amount of video, we developed a procedure based on recent deep learning methods to automatically detect and segment all objects of interest. This allowed us to accurately determine the pixel distances between the gaze point, the target object, and the limb in each individual frame. Our analysis shows a clear coordination between the eyes and the limb in the reach-to-grasp phase, confirming that both intact and amputated subjects precede the grasp with their eyes by more than 500 ms. Furthermore, we note that the gaze behavior of amputees was remarkably similar to that of the able-bodied control group, despite their inability to physically manipulate the objects.

Visually fixating or tracking another person decreases balance control in young and older females walking in a real-world scenario

Balance control during overground walking was assessed in 10 young (23.6 ± 3.4) and 10 older (71.0 ± 5.5 years) healthy females during free gaze, and when fixating or tracking another person in an everyday use waiting room. Balance control was characterised by medial/lateral sacrum acceleration dispersion, and gaze fixations were simultaneously assessed with eye tracking equipment. The results showed decreased balance control when fixating a stationary (p = 0.003, g_av = 0.19) and tracking a walking (p = 0.027, g_av = 0.16) person compared to free gaze. The older adults exhibited reduced baseline stability throughout, but the decrease caused by the visual tasks was not more profound than the younger adults. The decreased balance control when fixating on or tracking the observed person was likely due to more challenging conditions for interpreting retinal flow, which facilitated less reliable estimates of self-motion through vision. The older adults either processed retinal flow during the tasks as effectively as the young adults, or they adopted a more rigid posture to facilitate visual stability, which masked any ageing effect of the visual tasks. The decrease in balance control, the first to be shown in this context, may warrant further investigation in those with ocular or vestibular dysfunction.

Smooth pursuits decrease balance control during locomotion in young and older healthy females

Dynamic balance control-characterised as movement of the trunk and lower limbs-was assessed during fixation of a fixed target, smooth pursuits and saccadic eye movements in ten young (22.9 ± 1.5 years) and ten older (72.1 ± 8.2 years) healthy females walking overground. Participants were presented with visual stimuli to initiate eye movements, and posture and gaze were assessed with motion analysis and eye tracking equipment. The results showed an increase in medial/lateral (ML) trunk movement (C7: p = 0.012; sacrum: p = 0.009) and step-width variability (p = 0.052) during smooth pursuits compared to a fixed target, with no changes for saccades compared to a fixed target. The elders demonstrated greater ML trunk movement (sacrum: p = 0.037) and step-width variability (p = 0.037) than the younger adults throughout, although this did not interact with the eye movements. The findings showed that smooth pursuits decreased balance control in young and older adults similarly, which was likely a consequence of more complicated retinal flow. Since healthy elders are typically already at a postural disadvantage, further decreases in balance caused by smooth pursuits are undesirable.

Measuring dwell time percentage from head-mounted eye-tracking data--comparison of a frame-by-frame and a fixation-by-fixation analysis

Although analysing software for eye-tracking data has significantly improved in the past decades, the analysis of gaze behaviour recorded with head-mounted devices is still challenging and time-consuming. Therefore, new methods have to be tested to reduce the analysis workload while maintaining accuracy and reliability. In this article, dwell time percentages to six areas of interest (AOIs), of six participants cycling on four different roads, were analysed both frame-by-frame and in a 'fixation-by-fixation' manner. The fixation-based method is similar to the classic frame-by-frame method but instead of assigning frames, fixations are assigned to one of the AOIs. Although some considerable differences were found between the two methods, a Pearson correlation of 0.930 points out a good validity of the fixation-by-fixation method. For the analysis of gaze behaviour over an extended period of time, the fixation-based approach is a valuable and time-saving alternative for the classic frame-by-frame analysis.

The visual control of bicycle steering: The effects of speed and path width

Although cycling is a widespread form of transportation, little is known about the visual behaviour of bicycle users. This study examined whether the visual behaviour of cyclists can be explained by the two-level model of steering described for car driving, and how it is influenced by cycling speed and lane width. In addition, this study investigated whether travel fixations, described during walking, can also be found during a cycling task. Twelve adult participants were asked to cycle three 15m long cycling lanes of 10, 25 and 40cm wide at three different self-selected speeds (i.e., slow, preferred and fast). Participants' gaze behaviour was recorded at 50Hz using a head mounted eye tracker and the resulting scene video with overlay gaze cursor was analysed frame by frame. Four types of fixations were distinguished: (1) travel fixations, (2) fixations inside the cycling lane (path), (3) fixations to the final metre of the lane (goal), and (4) fixations outside of the cycling lane (external). Participants were found to mainly watch the path (41%) and goal (40%) region while very few travel fixations were made (<5%). Instead of travel fixations, an OptoKinetic Nystagmus was revealed when looking at the near path. Large variability between subjects in fixation location suggests that different strategies were used. Wider lanes resulted in a shift of gaze towards the end of the lane and to external regions, whereas higher cycling speeds resulted in a more distant gaze behaviour and more travel fixations. To conclude, the two-level model of steering as described for car driving is not fully in line with our findings during cycling, but the assumption that both the near and the far region is necessary for efficient steering seems valid. A new model for visual behaviour during goal directed locomotion is presented.

Impact and sources of neuronal variability in the fly's motion vision pathway

Nervous systems encode information about dynamically changing sensory input by changes in neuronal activity. Neuronal activity changes, however, also arise from noise sources within and outside the nervous system or from changes of the animal's behavioral state. The resulting variability of neuronal responses in representing sensory stimuli limits the reliability with which animals can respond to stimuli and may thus even affect the chances for survival in certain situations. Relevant sources of noise arising at different stages along the motion vision pathway have been investigated from the sensory input to the initiation of behavioral reactions. Here, we concentrate on the reliability of processing visual motion information in flies. Flies rely on visual motion information to guide their locomotion. They are among the best established model systems for the processing of visual motion information allowing us to bridge the gap between behavioral performance and underlying neuronal computations. It has been possible to directly assess the consequences of noise at major stages of the fly's visual motion processing system on the reliability of neuronal signals. Responses of motion sensitive neurons and their variability have been related to optomotor movements as indicators for the overall performance of visual motion computation. We address whether and how noise already inherent in the stimulus, e.g. photon noise for the visual system, influences later processing stages and to what extent variability at the output level of the sensory system limits behavioral performance. Recent advances in circuit analysis and the progress in monitoring neuronal activity in behaving animals should now be applied to understand how the animal meets the requirements of fast and reliable manoeuvres in naturalistic situations.

Visual estimation of travel distance during walking

The optic flow generated in the eyes during self-motion provides an important control signal for direction and speed of self-motion, and can be used to track the distance that has been traveled. The use of vision for these behavioral tasks can be studied in isolation in virtual reality setups, in which self-motion is merely simulated, and in which the visual motion can be controlled independently of other sensory cues. In such experiments it was found that the estimation of the travel distance of a simulated movement shows characteristic errors, sometimes overestimating and sometimes underestimating the true travel distance. These errors can be explained by a leaky path integration model. To test whether this model also holds for actual self-motion in the real world we studied walking distance perception in an open field with tasks similar to those previously used in virtual environments. We show that similar errors occur in the estimation of travel distance in the real world as in virtual environment, and that they are consistent with the leaky integration model.

Efficient encoding of natural optic flow

Statistically efficient processing schemes focus the resources of a signal processing system on the range of statistically probable signals. Relying on the statistical properties of retinal motion signals during ego-motion we propose a nonlinear processing scheme for retinal flow. It maximizes the mutual information between the visual input and its neural representation, and distributes the processing load uniformly over the neural resources. We derive predictions for the receptive fields of motion sensitive neurons in the velocity space. The properties of the receptive fields are tightly connected to their position in the visual field, and to their preferred retinal velocity. The velocity tuning properties show characteristics of properties of neurons in the motion processing pathway of the primate brain.

Radial motion bias in macaque frontal eye field

The visual responsiveness and spatial tuning of frontal eye field (FEF) neurons were determined using a delayed memory saccade task. Neurons with visual responses were then tested for direction selectivity using moving random dot patterns centered in the visual receptive field. The preferred axis of motion showed a significant tendency to be aligned with the receptive-field location so as to favor motion toward or away from the center of gaze. Centrifugal (outward) motion was preferred over centripetal motion. Motion-sensitive neurons in FEF thus appear to have a direction bias at the population level. This bias may facilitate the detection or discrimination of expanding optic flow patterns. The direction bias is similar to that seen in visual area MT and in posterior parietal cortex, from which FEF receives afferent projections. The outward motion bias may explain asymmetries in saccades made to moving targets. A representation of optic flow in FEF might be useful for planning eye movements during navigation.

Movement-induced motion signal distributions in outdoor scenes

The movement of an observer generates a characteristic field of velocity vectors on the retina (Gibson 1950). Because such optic flow-fields are useful for navigation, many theoretical, psychophysical and physiological studies have addressed the question how ego-motion parameters such as direction of heading can be estimated from optic flow. Little is known, however, about the structure of optic flow under natural conditions. To address this issue, we recorded sequences of panoramic images along accurately defined paths in a variety of outdoor locations and used these sequences as input to a two-dimensional array of correlation-based motion detectors (2DMD). We find that (a) motion signal distributions are sparse and noisy with respect to local motion directions; (b) motion signal distributions contain patches (motion streaks) which are systematically oriented along the principal flow-field directions; (c) motion signal distributions show a distinct, dorso-ventral topography, reflecting the distance anisotropy of terrestrial environments; (d) the spatiotemporal tuning of the local motion detector we used has little influence on the structure of motion signal distributions, at least for the range of conditions we tested; and (e) environmental motion is locally noisy throughout the visual field, with little spatial or temporal correlation; it can therefore be removed by temporal averaging and is largely over-ridden by image motion caused by observer movement. Our results suggest that spatial or temporal integration is important to retrieve reliable information on the local direction and size of motion vectors, because the structure of optic flow is clearly detectable in the temporal average of motion signal distributions. Ego-motion parameters can be reliably retrieved from such averaged distributions under a range of environmental conditions. These observations raise a number of questions about the role of specific environmental and computational constraints in the processing of natural optic flow.

Interhemispheric connections between primary visual areas: beyond the midline rule

In the last five years, a number of detailed anatomical, electrophysiological, optical imaging and simulation studies performed in a variety of non-human species have revealed that the functional organization of callosal connections between primary visual areas is more elaborate than previously thought. Callosal cell bodies and terminals are clustered in columns whose correspondence to features mapped in the visual cortex, such as orientation and ocularity, are starting to be understood. Callosal connections are not restricted to the vertical midline representation nor do they establish merely point-to-point retinotopic correspondences across the hemispheres, as traditionally believed. In addition, anatomical studies have revealed the existence of an ipsilateral component of callosal axons. The aim of this short review is to propose how these new data can be integrated into an updated scheme of the circuits responsible for assembling the primary visual field map.

Visual-vestibular interactive responses in the macaque ventral intraparietal area (VIP)

Self-motion detection requires the interaction of a number of sensory systems for correct perceptual interpretation of a given movement and an eventual motor response. Parietal cortical areas are thought to play an important role in this function, and we have thus studied the encoding of multimodal signals and their spatiotemporal interactions in the ventral intraparietal area of macaque monkeys. Thereby, we have identified for the first time the presence of vestibular sensory input to this area and described its interaction with somatosensory and visual signals, via extracellular single-cell recordings in awake head-fixed animals. Visual responses were driven by large field stimuli that simulated either backward or forward self-motion (contraction or expansion stimuli, respectively), or movement in the frontoparallel plane (visual increments moving simultaneously in the same direction). While the dominant sensory modality in most neurons was visual, about one third of all recorded neurons responded to horizontal rotation. These vestibular responses were typically in phase with head velocity, but in some cases they could signal acceleration or even showed integration to position. The associated visual responses were always codirectional with the vestibular on-direction, i.e. noncomplementary. Somatosensory responses were in register with the visual preferred direction, either in the same or in the opposite direction, thus signalling translation or rotation in the horizontal plane. These results, taken together with data on responses to optic flow stimuli obtained in a parallel study, strongly suggest an involvement of area VIP in the analysis and the encoding of self-motion.