The perception of spatial relations in a patient with visual form agnosia

Integration of Eye-Centered and Landmark-Centered Codes in Frontal Eye Field Gaze Responses

The visual system is thought to separate egocentric and allocentric representations, but behavioral experiments show that these codes are optimally integrated to influence goal-directed movements. To test if frontal cortex participates in this integration, we recorded primate frontal eye field activity during a cue-conflict memory delay saccade task. To dissociate egocentric and allocentric coordinates, we surreptitiously shifted a visual landmark during the delay period, causing saccades to deviate by 37% in the same direction. To assess the cellular mechanisms, we fit neural response fields against an egocentric (eye-centered target-to-gaze) continuum, and an allocentric shift (eye-to-landmark-centered) continuum. Initial visual responses best-fit target position. Motor responses (after the landmark shift) predicted future gaze position but embedded within the motor code was a 29% shift toward allocentric coordinates. This shift appeared transiently in memory-related visuomotor activity, and then reappeared in motor activity before saccades. Notably, fits along the egocentric and allocentric shift continua were initially independent, but became correlated across neurons just before the motor burst. Overall, these results implicate frontal cortex in the integration of egocentric and allocentric visual information for goal-directed action, and demonstrate the cell-specific, temporal progression of signal multiplexing for this process in the gaze system.

How do the two visual streams interact with each other?

The current consensus divides primate cortical visual processing into two broad networks or "streams" composed of highly interconnected areas (Milner and Goodale 2006, 2008; Goodale 2014). The ventral stream, passing from primary visual cortex (V1) through to inferior parts of the temporal lobe, is considered to mediate the transformation of the contents of the visual signal into the mental furniture that guides memory, recognition and conscious perception. In contrast the dorsal stream, passing from V1 through to various areas in the posterior parietal lobe, is generally considered to mediate the visual guidance of action, primarily in real time. The brain, however, does not work through mutually insulated subsystems, and indeed there are well-documented interconnections between the two streams. Evidence for contributions from ventral stream systems to the dorsal stream comes from human neuropsychological and neuroimaging research, and indicates a crucial role in mediating complex and flexible visuomotor skills. Complementary evidence points to a role for posterior dorsal-stream visual analysis in certain aspects of 3-D perceptual function in the ventral stream. A series of studies of a patient with visual form agnosia has been instrumental in shaping our knowledge of what each stream can achieve in isolation; but it has also helped us to tease apart the relative dependence of parietal visuomotor systems on direct bottom-up visual inputs versus inputs redirected via perceptual systems within the ventral stream.

Posterior cortical atrophy: visuomotor deficits in reaching and grasping

Posterior Cortical Atrophy (PCA) is a rare clinical syndrome characterized by the predominance of higher-order visual disturbances such as optic ataxia, a characteristic of Balint's syndrome. Deficits result from progressive neurodegeneration of occipito-temporal and occipito-parietal cortices. The current study sought to explore the visuomotor functioning of four individuals with PCA by testing their ability to reach out and grasp real objects under various viewing conditions. Experiment 1 had participants reach out and grasp simple, rectangular blocks under visually- and memory-guided conditions. Experiment 2 explored participants' abilities to accurately reach for objects located in their visual periphery. This investigation revealed that PCA patients demonstrate many of the same deficits that have been previously reported in other individuals with optic ataxia, such as “magnetic misreaching”—a pathological reaching bias toward the point of visual fixation when grasping peripheral targets. Unlike many other individuals with optic ataxia, however, the patients in the current study also show symptoms indicative of damage to the more perceptual stream of visual processing, including abolished grip scaling during memory-guided grasping and deficits in face and object identification. These investigations are the first to perform a quantitative analysis of the visuomotor deficits exhibited by patients with PCA. Critically, this study helps characterize common symptoms of PCA, a vital first step for generating effective diagnostic criteria and therapeutic strategies for this understudied neurodegenerative disorder.

A Bayesian framework for active artificial perception

In this paper, we present a Bayesian framework for the active multimodal perception of 3-D structure and motion. The design of this framework finds its inspiration in the role of the dorsal perceptual pathway of the human brain. Its composing models build upon a common egocentric spatial configuration that is naturally fitting for the integration of readings from multiple sensors using a Bayesian approach. In the process, we will contribute with efficient and robust probabilistic solutions for cyclopean geometry-based stereovision and auditory perception based only on binaural cues, modeled using a consistent formalization that allows their hierarchical use as building blocks for the multimodal sensor fusion framework. We will explicitly or implicitly address the most important challenges of sensor fusion using this framework, for vision, audition, and vestibular sensing. Moreover, interaction and navigation require maximal awareness of spatial surroundings, which, in turn, is obtained through active attentional and behavioral exploration of the environment. The computational models described in this paper will support the construction of a simultaneously flexible and powerful robotic implementation of multimodal active perception to be used in real-world applications, such as human-machine interaction or mobile robot navigation.

The relationship between allocentric and egocentric frames of reference and categorical and coordinate spatial information processing

We report two experiments on the relationship between allocentric/egocentric frames of reference and categorical/coordinate spatial relations. Jager and Postma (2003) suggest two theoretical possibilities about their relationship: categorical judgements are better when combined with an allocentric reference frame and coordinate judgements with an egocentric reference frame (interaction hypothesis); allocentric/egocentric and categorical/coordinate form independent dimensions (independence hypothesis). Participants saw stimuli comprising two vertical bars (targets), one above and the other below a horizontal bar. They had to judge whether the targets appeared on the same side (categorical) or at the same distance (coordinate) with respect either to their body-midline (egocentric) or to the centre of the horizontal bar (allocentric). The results from Experiment 1 showed a facilitation in the allocentric and categorical conditions. In line with the independence hypothesis, no interaction effect emerged. To see whether the results were affected by the visual salience of the stimuli, in Experiment 2 the luminance of the horizontal bar was reduced. As a consequence, a significant interaction effect emerged indicating that categorical judgements were more accurate than coordinate ones, and especially so in the allocentric condition. Furthermore, egocentric judgements were as accurate as allocentric ones with a specific improvement when combined with coordinate spatial relations. The data from Experiment 2 showed that the visual salience of stimuli affected the relationship between allocentric/egocentric and categorical/coordinate dimensions. This suggests that the emergence of a selective interaction between the two dimensions may be modulated by the characteristics of the task.

Pointing to two imaginary targets at the same time: bimanual allocentric and egocentric localization in visual form agnosic D.F

We have previously shown the visual form agnosic patient D.F. has spared sensorimotor utilization of visual information relative to her poor perceptual processing of the same stimulus attributes. Her visuomotor skills are, however, only normal when egocentric visual coding can be used in the task. In other words, her egocentric sensorimotor processing is intact while her 'allocentric' coding of spatial position is impaired. The current investigation extends these previous observations by comparing D.F.'s performance in bimanual pointing to pairs of stimuli directly (the egocentric task) versus pointing to the homologous positions on an adjacent workspace (pantomimed reaching, the allocentric task). The results showed greatly superior pointing accuracy in direct pointing compared to pantomimed pointing. The mechanisms supporting her limited but remaining sensitivity to spatial relationships during pantomimed pointing remain unknown. These residual skills may reflect partially spared categorical coding and/or internal sensorimotor self-cueing.

Impaired distance perception and size constancy following bilateral occipitoparietal damage

Accurate distance perception depends on the processing and integration of a variety of monocular and binocular cues. Dorsal stream lesions can impair this process, but details of this neurocognitive relationship remain unclear. Here, we tested a patient with bilateral occipitoparietal damage and severely impaired stereopsis. We addressed four related questions: (1) Can distance and size perception survive limitations in perceiving monocular and binocular cues? (2) Are egocentric (self-referential) and allocentric (object-referential) distance judgments similarly impaired? (3) Are distance measurements equally impaired in peripersonal and extrapersonal space? (4) Are size judgments possible when distance processing is impaired? The results demonstrate that the patient's lesions impaired both her distance and size perception, but not uniformly. Her performance when using an egocentric reference frame was more impaired than her performance when using an allocentric reference frame. Likewise, her distance judgments in peripersonal space were more impaired than those in extrapersonal space. The patient showed partial preservation in size processing of novel objects even when familiar size cues were removed.

Dissociation of egocentric and allocentric coding of space in visual search after right middle cerebral artery stroke

Spatial representations rely on different frames of reference. Patients with unilateral neglect may behave as suffering from either egocentric or allocentric deficiency. The neural substrates representing these reference frames are still under discussion. Here we used a visual search paradigm to distinguish between egocentric and allocentric deficits in patients with right hemisphere cortical lesions. An attention demanding search task served to divide patients according to egocentric versus allocentric deficits. The results indicate that egocentric impairment was associated with damage in premotor cortex involving the frontal eye fields. Allocentric impairment on the other hand was linked to lesions in more ventral regions near the parahippocampal gyrus (PHG).

Somatosensory processing subserving perception and action: Dissociations, interactions, and integration

The commentaries have raised important points regarding different aspects of our model. Some have queried the nature of the proposed dissociations, whereas others have requested and provided further details regarding aspects we had glossed over. Here we suggest that our approach to identify major processing streams based on the processing goal does not preclude interactions between them. We further specify details regarding body representations, haptic object recognition, and crossmodal processing, but are also aware that several features of the model require further filling in.

Pointing to places and spaces in a patient with visual form agnosia

Previous investigations of visuospatial abilities in the visual form agnosic patient D.F. suggest that her egocentric sensorimotor processing is intact while her 'allocentric' judgments of spatial position are impaired. The current investigation extends these previous observations by comparing D.F.'s performance at pointing to a set of spatially distributed stimuli, either directly or by 'pantomiming' the responses in an adjacent homologous workspace. The results showed accurate sensorimotor localization when D.F. pointed directly to single targets or to sequences of targets, presumably as she could use egocentric visual coding. In spite of making relatively spared spatial judgments about the arrays, however, D.F. performed quite poorly when copying them and on the pantomimed pointing task. In this latter task good performance presumably depends on an ability to represent both the categorical and coordinate properties of the array (as does copying them), and to translate these into the effector-based coordinates required for accurate action. D.F.'s pantomimed pointing was similar to her copies of target arrays, as in both tasks there was evidence of spared (although somewhat degraded) appreciation of the relative spatial positions of the stimuli. Remarkably, her accuracy in this allocentric task was not worsened by longer pointing sequences. It is possible that D.F.'s degraded performance reflects a relative (though not complete) preservation of categorical coding within the ventral stream, despite a loss of coordinate coding there.

Grasping what is graspable: evidence from visual form agnosia

Patient DF has profound visual form agnosia. Despite this, she has no problem adjusting her finger-thumb grip aperture to the width of objects when reaching to grasp them. In a previous study, however, she was found to have great difficulty in scaling her grip aperture when attempting to grasp a transparent disc through two holes cut into it. This problem was attributed to a putative difference between the visual processing of size and distance in the brain, whereby DF retained the capacity for processing object size but not the separation between distinct elements such as holes. In the present study we have tested this idea more directly, and found no evidence to support such a distinction. Nonetheless, we replicated our earlier finding that DF is unable to produce normal prehension movements when attempting to grasp transparent stimuli by placing her digits into holes. We suggest that, whilst some simple objects offer themselves directly to the dorsal stream for grasping, an intact ventral stream is required to respond appropriately to more complex stimuli.

Delayed reaching and grasping in patients with optic ataxia

A series of experiments documenting the reaching and grasping of two patients with optic ataxia is presented. We compare their immediate responses with their behavior when required to delay for a few seconds before responding. When the delayed response is 'pantomimed', i.e. made in the absence of the target object, their performance typically improves. This pattern was predicted from a two-visual-systems model in which the cortical dorsal stream mediates normal visually guided actions while the ventral stream deals with visual information that has to be held in memory. We further found that when a 'preview' task was used in which the patients could use memorized information to guide a response to a still-present target object, they did so in preference to using the visual information facing them.

Dissociating 'what' and 'how' in visual form agnosia: a computational investigation

Patients with visual form agnosia exhibit a profound impairment in shape perception (what an object is) coupled with intact visuomotor functions (how to act on an object), demonstrating a dissociation between visual perception and action. How can these patients act on objects that they cannot perceive? Although two explanations of this 'what-how' dissociation have been offered, each explanation has shortcomings. A 'pathway information' account of the 'what-how' dissociation is presented in this paper. This account hypothesizes that 'where' and 'how' tasks require less information than 'what' tasks, thereby allowing 'where/how' to remain relatively spared in the face of neurological damage. Simulations with a neural network model test the predictions of the pathway information account. Following damage to an input layer common to the 'what' and 'where/how' pathways, the model performs object identification more poorly than spatial localization. Thus, the model offers a parsimonious explanation of differential 'what-how' performance in visual form agnosia. The simulation results are discussed in terms of their implications for visual form agnosia and other neuropsychological syndromes.

Grasping the past. delay can improve visuomotor performance

"Optic ataxia" is caused by damage to the human posterior parietal cortex (PPC). It disrupts all components of a visually guided prehension movement, not only the transport of the hand toward an object's location, but also the in-flight finger movements pretailored to the metric properties of the object. Like previous cases, our patient (I.G.) was quite unable to open her handgrip appropriately when directly reaching out to pick up objects of different sizes. When first tested, she failed to do this even when she had previewed the target object 5 s earlier. Yet despite this deficit in "real" grasping, we found, counterintuitively, that I.G. showed good grip scaling when "pantomiming" a grasp for an object seen earlier but no longer present. We then found that, after practice, I.G. became able to scale her handgrip when grasping a real target object that she had previewed earlier. By interposing catch trials in which a different object was covertly substituted for the original object during the delay between preview and grasp, we found that I.G. was now using memorized visual information to calibrate her real grasping movements. These results provide new evidence that "off-line" visuomotor guidance can be provided by networks independent of the PPC.

Do action systems resist visual illusions?

Arguments about the relative independence of visual modules in the primate brain are not new. Recently, though, these debates have resurfaced in the form of arguments about the extent to which visuomotor reaching and grasping systems are insensitive to visual illusions that dramatically bias visual perception. The first wave of studies of illusory effects on perception and action have supported the idea of independence of motor systems, but recent findings have been more critical. In this article, I review several of these studies, most of which (but not all) can be reconciled with the two-visual-systems model.

A paradoxical improvement of misreaching in optic ataxia: new evidence for two separate neural systems for visual localization

We tested a patient (A. T.) with bilateral brain damage to the parietal lobes, whose resulting 'optic ataxia' causes her to make large pointing errors when asked to locate single light emitting diodes presented in her visual field. We report here that, unlike normal individuals, A. T.'s pointing accuracy improved when she was required to wait for 5 s before responding. This counter-intuitive result is interpreted as reflecting the very brief time-scale on which visuomotor control systems in the superior parietal lobe operate. When an immediate response was required, A. T.'s damaged visuomotor system caused her to make large errors; but when a delay was required, a different, more flexible, visuospatial coding system--presumably relatively intact in her brain--came into play, resulting in much more accurate responses. The data are consistent with a dual processing theory whereby motor responses made directly to visual stimuli are guided by a dedicated system in the superior parietal and premotor cortices, while responses to remembered stimuli depend on perceptual processing and may thus crucially involve processing within the temporal neocortex.

Vision for perception and vision for action in the primate brain

Visual systems first evolved not to enable animals to see, but to provide distal sensory control of their movements. Vision as 'sight' is a relative newcomer to the evolutionary landscape, but its emergence has enabled animals to carry out complex cognitive operations on perceptual representations of the world. The two streams of visual processing that have been identified in the primate cerebral cortex are a reflection of these two functions of vision. The dorsal 'action' stream projecting from primary visual cortex to the posterior parietal cortex provides flexible control of more ancient subcortical visuomotor modules for the production of motor acts. The ventral 'perceptual' stream projecting from the primary visual cortex to the temporal lobe provides the rich and detailed representation of the world required for cognitive operations. Both streams process information about the structure of objects and about their spatial locations--and both are subject to the modulatory influences of attention. Each stream, however, uses visual information in different ways. Transformations carried out in the ventral stream permit the formation of perceptual representations that embody the enduring characteristics of objects and their relations; those carried out in the dorsal stream which utilize moment-to-moment information about objects within egocentric frames of reference, mediate the control of skilled actions. Both streams work together in the production of goal-directed behaviour.