The physiology of stereopsis

Simulating the cortical 3D visuomotor transformation of reach depth

We effortlessly perform reach movements to objects in different directions and depths. However, how networks of cortical neurons compute reach depth from binocular visual inputs remains largely unknown. To bridge the gap between behavior and neurophysiology, we trained a feed-forward artificial neural network to uncover potential mechanisms that might underlie the 3D transformation of reach depth. Our physiologically-inspired 4-layer network receives distributed 3D visual inputs (1^st layer) along with eye, head and vergence signals. The desired motor plan was coded in a population (3^rd layer) that we read out (4^th layer) using an optimal linear estimator. After training, our network was able to reproduce all known single-unit recording evidence on depth coding in the parietal cortex. Network analyses predict the presence of eye/head and vergence changes of depth tuning, pointing towards a gain-modulation mechanism of depth transformation. In addition, reach depth was computed directly from eye-centered (relative) visual distances, without explicit absolute depth coding. We suggest that these effects should be observable in parietal and pre-motor areas.

Spatially invariant computations in stereoscopic vision

PERCEPTION OF STEREOSCOPIC DEPTH REQUIRES THAT VISUAL SYSTEMS SOLVE A CORRESPONDENCE PROBLEM: find parts of the left-eye view of the visual scene that correspond to parts of the right-eye view. The standard model of binocular matching implies that similarity of left and right images is computed by inter-ocular correlation. But the left and right images of the same object are normally distorted relative to one another by the binocular projection, in particular when slanted surfaces are viewed from close distance. Correlation often fails to detect correct correspondences between such image parts. We investigate a measure of inter-ocular similarity that takes advantage of spatially invariant computations similar to the computations performed by complex cells in biological visual systems. This measure tolerates distortions of corresponding image parts and yields excellent performance over a much larger range of surface slants than the standard model. The results suggest that, rather than serving as disparity detectors, multiple binocular complex cells take part in the computation of inter-ocular similarity, and that visual systems are likely to postpone commitment to particular binocular disparities until later stages in the visual process.

HEAD-CENTRIC DISPARITY AND EPIPOLAR GEOMETRY ESTIMATION FROM A POPULATION OF BINOCULAR ENERGY NEURONS

We present a hybrid neural network architecture that supports the estimation of binocular disparity in a cyclopean, head-centric coordinate system without explicitly establishing retinal correspondences. Instead the responses of binocular energy neurons are gain-modulated by oculomotor signals. The network can handle the full six degrees of freedom of binocular gaze and operates directly on image pairs of possibly varying contrast. Furthermore, we show that in the absence of an oculomotor signal the same architecture is capable of estimating the epipolar geometry directly from the population response. The increased complexity of the scenarios considered in this work provides an important step towards the application of computational models centered on gain modulation mechanisms in real-world robotic applications. The proposed network is shown to outperform a standard computer vision technique on a disparity estimation task involving real-world stereo images.

Contributions of excitation and suppression in shaping spatial frequency selectivity of V1 neurons as revealed by binocular measurements

Neurons in the early visual cortex are generally highly sensitive to stimuli presented to the two eyes. However, the majority of studies on spatial and temporal aspects of neural responses were based on monocular measurements. To study neurons under more natural, i.e., binocular, conditions, we presented sinusoidal gratings of a variety of spatial frequencies (SF) dichoptically in rapid sequential flashes and analyzed the data using a binocular reverse correlation technique for neurons in cat area 17. The resulting set of data represents a frequency-domain binocular receptive field from which detailed selectivities, both monocular and binocular, could be obtained. Consistent with previous studies, the responses could generally be explained by linear summation of inputs from the two eyes. Suppressive responses were also observed and were delayed typically by 5–15 ms relative to excitatory responses. However, we have found more diverse nature of suppressive responses than those reported previously. The optimal suppressive frequency could be either higher or lower than that of the excitatory responses. The bandwidth of SF tuning of the suppressive responses was usually broader than that of the excitatory responses. Cells with lower optimal SFs for suppression tended to show high optimal SFs and sharp tuning curves. The dynamic shift of optimal SF from low to high SF was accompanied by suppression with earlier onset and higher peak SF or later onset and lower peak SF than excitation. These results suggest that the suppression plays an essential role in generating the temporal dynamics of SF selectivity.

Pose estimation through cue integration: a neuroscience-inspired approach

The aim of this paper is to improve the skills of robotic systems in their interaction with nearby objects. The basic idea is to enhance visual estimation of objects in the world through the merging of different visual estimators of the same stimuli. A neuroscience-inspired model of stereoptic and perspective orientation estimators, merged according to different criteria, is implemented on a robotic setup and tested in different conditions. Experimental results suggest that the integration of multiple monocular and binocular cues can make robot sensory systems more reliable and versatile. The same results, compared with simulations and data from human studies, show that the model is able to reproduce some well-recognized neuropsychological effects.

Bridging the gap: global disparity processing in the human visual cortex

The human stereoscopic system is remarkable in its ability to utilize widely separated features as references to support fine depth discrimination. In a search for possible neural substrates of this ability, we recorded high-density EEG and used a distributed inverse technique to estimate population-level disparity responses in five regions of interest (ROIs): V1, V3A, hMT+, V4, and lateral occipital complex (LOC). The stimulus was a central modulating disk surrounded by a correlated "reference" annulus presented in the fixation plane. We varied a gap separating the disk from the annulus parametrically from 0 to 5.5° as a test of long-range disparity integration. In the V1, LOC, and hMT+ ROIs, the responses with gaps >0.5° were equal to those obtained in a control condition where the surround was composed of uncorrelated noise (no reference). By contrast, in the V4 and V3A ROIs, responses with gaps as large as 5.5° were still significantly higher than the control. As a test of the spatial distribution of the disparity reference information, we manipulated the properties of the stimulus by placing noise between the center and the surround or throughout the surround. The V3A ROI was particularly sensitive to disparity noise between the center and annulus regions, suggesting an important contribution of disparity edge detectors in this ROI.

Selectivity for three-dimensional contours and surfaces in the anterior intraparietal area

The macaque anterior intraparietal area (AIP) is crucial for visually guided grasping. AIP neurons respond during the visual presentation of real-world objects and encode the depth profile of disparity-defined curved surfaces. We investigated the neural representation of curved surfaces in AIP using a stimulus-reduction approach. The stimuli consisted of three-dimensional (3-D) shapes curved along the horizontal axis, the vertical axis, or both the horizontal and the vertical axes of the shape. The depth profile was defined solely by binocular disparity that varied along either the boundary or the surface of the shape or along both the boundary and the surface of the shape. The majority of AIP neurons were selective for curved boundaries along the horizontal or the vertical axis, and neural selectivity emerged at short latencies. Stimuli in which disparity varied only along the surface of the shape (with zero disparity on the boundaries) evoked selectivity in a smaller proportion of AIP neurons and at considerably longer latencies. AIP neurons were not selective for 3-D surfaces composed of anticorrelated disparities. Thus the neural selectivity for object depth profile in AIP is present when only the boundary is curved in depth, but not for disparity in anticorrelated stereograms.

Estimation of 3D shape from image orientations

One of the main functions of vision is to estimate the 3D shape of objects in our environment. Many different visual cues, such as stereopsis, motion parallax, and shading, are thought to be involved. One important cue that remains poorly understood comes from surface texture markings. When a textured surface is slanted in 3D relative to the observer, the surface patterns appear compressed in the retinal image, providing potentially important information about 3D shape. What is not known, however, is how the brain actually measures this information from the retinal image. Here, we explain how the key information could be extracted by populations of cells tuned to different orientations and spatial frequencies, like those found in the primary visual cortex. To test this theory, we created stimuli that selectively stimulate such cell populations, by "smearing" (filtering) images of 2D random noise into specific oriented patterns. We find that the resulting patterns appear vividly 3D, and that increasing the strength of the orientation signals progressively increases the sense of 3D shape, even though the filtering we apply is physically inconsistent with what would occur with a real object. This finding suggests we have isolated key mechanisms used by the brain to estimate shape from texture. Crucially, we also find that adapting the visual system's orientation detectors to orthogonal patterns causes unoriented random noise to look like a specific 3D shape. Together these findings demonstrate a crucial role of orientation detectors in the perception of 3D shape.

Understanding attentional modulation of binocular rivalry: a framework based on biased competition

Starting from early scientific explorations of binocular rivalry, researchers have wondered about the degree to which an observer can exert voluntary attentional control over rivalry dynamics. The answer to this question would not only reveal the extent to which we may determine our own conscious visual experience, but also advance our understanding of the neural mechanisms underlying binocular rivalry. Classic studies, intriguingly, reached contradictory conclusions, ranging from an absence of attentional control, as advocated by Breese, to nearly complete control of rivalry dynamics, as reported by Helmholtz. Recent investigations have revisited this question, but the results have continued to echo the conflicting findings of earlier studies, seemingly precluding a comprehensive understanding of attentional effects on rivalry. Here, we review both classic and modern studies, and propose a unifying framework derived from the biased competition theory of attention. The key assumption of this theory is that the nature of stimulus conflict determines the limits of attentional modulation. For example, a condition in which unresolved stimulus conflict transpires through many levels of visual processing should be very susceptible to attentional control. When applied to binocular rivalry, this framework predicts strong attentional modulations under conditions of unresolved stimulus conflict (e.g., initial selection) and conditions where conflict is resolved at higher levels of visual processing (e.g., stimulus rivalry). Additionally, the efficacy of attentional control over rivalry can be increased by utilization of demanding, behaviorally relevant tasks, and likely through perceptual training paradigms. We show that this framework can help facilitate the understanding and synthesis of a diverse set of results on attentional control over rivalry, and we propose several directions for future research on this interesting topic.

Three-dimensional transformations for goal-directed action

Much of the central nervous system is involved in visuomotor transformations for goal-directed gaze and reach movements. These transformations are often described in terms of stimulus location, gaze fixation, and reach endpoints, as viewed through the lens of translational geometry. Here, we argue that the intrinsic (primarily rotational) 3-D geometry of the eye-head-reach systems determines the spatial relationship between extrinsic goals and effector commands, and therefore the required transformations. This approach provides a common theoretical framework for understanding both gaze and reach control. Combined with an assessment of the behavioral, neurophysiological, imaging, and neuropsychological literature, this framework leads us to conclude that (a) the internal representation and updating of visual goals are dominated by gaze-centered mechanisms, but (b) these representations must then be transformed as a function of eye and head orientation signals into effector-specific 3-D movement commands.

Statistical modeling of 3-D natural scenes with application to Bayesian stereopsis

We studied the empirical distributions of luminance, range and disparity wavelet coefficients using a coregistered database of luminance and range images. The marginal distributions of range and disparity are observed to have high peaks and heavy tails, similar to the well-known properties of luminance wavelet coefficients. However, we found that the kurtosis of range and disparity coefficients is significantly larger than that of luminance coefficients. We used generalized Gaussian models to fit the empirical marginal distributions. We found that the marginal distribution of luminance coefficients have a shape parameter p between 0.6 and 0.8, while range and disparity coefficients have much smaller parameters p < 0.32, corresponding to a much higher peak. We also examined the conditional distributions of luminance, range and disparity coefficients. The magnitudes of luminance and range (disparity) coefficients show a clear positive correlation, which means, at a location with larger luminance variation, there is a higher probability of a larger range (disparity) variation. We also used generalized Gaussians to model the conditional distributions of luminance and range (disparity) coefficients. The values of the two shape parameters (p,s) reflect the observed luminance-range (disparity) dependency. As an example of the usefulness of luminance statistics conditioned on range statistics, we modified a well-known Bayesian stereo ranging algorithm using our natural scene statistics models, which improved its performance.

Visuomotor coordination is different for different directions in three-dimensional space

In most visuomotor tasks in which subjects have to reach to visual targets or move the hand along a particular trajectory, eye movements have been shown to lead hand movements. Because the dynamics of vergence eye movements is different from that of smooth pursuit and saccades, we have investigated the lead time of gaze relative to the hand for the depth component (vergence) and in the frontal plane (smooth pursuit and saccades) in a tracking task and in a tracing task in which human subjects were instructed to move the finger along a 3D path. For tracking, gaze leads finger position on average by 28 ± 6 ms (mean ± SE) for the components in the frontal plane but lags finger position by 95 ± 39 ms for the depth dimension. For tracing, gaze leads finger position by 151 ± 36 ms for the depth dimension. For the frontal plane, the mean lead time of gaze relative to the hand is 287 ± 13 ms. However, we found that the lead time in the frontal plane was inversely related to the tangential velocity of finger. This inverse relation for movements in the frontal plane could be explained by assuming that gaze leads the finger by a constant distance of ∼ 2.6 cm (range of 1.5-3.6 cm across subjects).

Coding of stereoscopic depth information in visual areas V3 and V3A

The process of stereoscopic depth perception is thought to begin with the analysis of absolute binocular disparity, the difference in position of corresponding features in the left and right eye images with respect to the points of fixation. Our sensitivity to depth, however, is greater when depth judgments are based on relative disparity, the difference between two absolute disparities, compared to when they are based on absolute disparity. Therefore, the visual system is thought to compute relative disparities for fine depth discrimination. Functional magnetic resonance imaging studies in humans and monkeys have suggested that visual areas V3 and V3A may be specialized for stereoscopic depth processing based on relative disparities. In this study, we measured absolute and relative disparity-tuning of neurons in V3 and V3A of alert fixating monkeys, and we compared their basic tuning properties with those published previously for other visual areas. We found that neurons in V3 and V3A predominantly encode absolute, not relative, disparities. We also found that basic parameters of disparity-tuning in V3 and V3A are similar to those from other extrastriate visual areas. Finally, by comparing single-unit activity with multi-unit activity measured at the same recording site, we demonstrate that neurons with similar disparity selectivity are clustered in both V3 and V3A. We conclude that areas V3 and V3A are not particularly specialized for processing stereoscopic depth information compared to other early visual areas, at least with respect to the tuning properties that we have examined.

Bayesian sampling in visual perception

It is well-established that some aspects of perception and action can be understood as probabilistic inferences over underlying probability distributions. In some situations, it would be advantageous for the nervous system to sample interpretations from a probability distribution rather than commit to a particular interpretation. In this study, we asked whether visual percepts correspond to samples from the probability distribution over image interpretations, a form of sampling that we refer to as Bayesian sampling. To test this idea, we manipulated pairs of sensory cues in a bistable display consisting of two superimposed moving drifting gratings, and we asked subjects to report their perceived changes in depth ordering. We report that the fractions of dominance of each percept follow the multiplicative rule predicted by Bayesian sampling. Furthermore, we show that attractor neural networks can sample probability distributions if input currents add linearly and encode probability distributions with probabilistic population codes.

On the road to invariant object recognition: how cortical area V2 transforms absolute to relative disparity during 3D vision

Invariant recognition of objects depends on a hierarchy of cortical stages that build invariance gradually. Binocular disparity computations are a key part of this transformation. Cortical area V1 computes absolute disparity, which is the horizontal difference in retinal location of an image in the left and right foveas. Many cells in cortical area V2 compute relative disparity, which is the difference in absolute disparity of two visible features. Relative, but not absolute, disparity is invariant under both a disparity change across a scene and vergence eye movements. A neural network model is introduced which predicts that shunting lateral inhibition of disparity-sensitive layer 4 cells in V2 causes a peak shift in cell responses that transforms absolute disparity from V1 into relative disparity in V2. This inhibitory circuit has previously been implicated in contrast gain control, divisive normalization, selection of perceptual groupings, and attentional focusing. The model hereby links relative disparity to other visual functions and thereby suggests new ways to test its mechanistic basis. Other brain circuits are reviewed wherein lateral inhibition causes a peak shift that influences behavioral responses.

The integration of disparity, shading and motion parallax cues for depth perception in humans and monkeys

A visual stimulus display was created that enabled us to examine how effectively the three depth cues of disparity, motion parallax and shading can be integrated in humans and monkeys. The display was designed to allow us to present these three depth cues separately and in various combinations. Depth was processed most effectively and most rapidly when all three cues were presented together indicating that these separate cues are integrated at yet unknown sites in the brain. Testing in humans and monkeys yielded similar results suggesting that monkeys are a good animal model for the study of the underlying neural mechanisms of depth perception.

Path dissociation of visual signals entering the cortex: disparity, contour orientation and position

Whether position and orientation shifts induced by monocular context also act as a disparity for purposes of stereoscopy was investigated experimentally in order to examine the extent to which lateral spatial localization and stereoscopic depth share circuitry. A monocular tilt illusion in a line does not lead to a commensurate depth tilt of that line in binocular view, nor does a position shift in a bisection task caused by a gap within monocular dynamic random noise produce the commensurate depth displacement. Interocular transfer of monocularly-induced shifts, which might explain such findings, was eliminated as a factor. The results can therefore be interpreted as indicators of channeling and ordering of spatial signals paths in the visual cortex and imply that two-dimensional contextual interactions operate at a processing level beyond where disparity has already been extracted.

Disparity-tuned population responses from human visual cortex

We used source imaging of visual evoked potentials to measure neural population responses over a wide range of horizontal disparities (0.5-64 arcmin). The stimulus was a central disk that moved back and forth across the fixation plane at 2 Hz, surrounded either by binocularly uncorrelated dots (disparity noise) or by correlated dots presented in the fixation plane. Both disk and surround were composed of dynamic random dots to remove coherent monocular information. Disparity tuning was measured in five visual regions of interest (ROIs) [V1, human middle temporal area (hMT+), V4, lateral occipital complex (LOC), and V3A], defined in separate functional magnetic resonance imaging scans. The disparity tuning functions peaked between 2 and 16 arcmin for both types of surround in each ROI. Disparity tuning in the V1 ROI was unaffected by the type of surround, but surround correlation altered both the amplitude and phase of the disparity responses in the other ROIs. Response amplitude increased when the disk was in front of the surround in the V3A and LOC ROIs, indicating that these areas encode figure-ground relationships and object convexity. The correlated surround produced a consistent phase lag at the second harmonic in the hMT+ and V4 ROIs without a change in amplitude, while in the V3A ROI, both phase and amplitude effects were observed. Sensitivity to disparity context is thus widespread in visual cortex, but the dynamics of these contextual interactions differ across regions.

Cortical dynamics underlying face completion in human visual system

In natural images, visual objects are typically occluded by other objects. A remarkable ability of our visual system is to complete occluded objects effortlessly and see whole, uninterrupted objects. How object completion is implemented in the visual system is still largely unknown. In this study, using a backward masking paradigm, we combined psychophysics and functional magnetic resonance imaging to investigate the temporal evolvement of face completion at different levels of the visual processing hierarchy. Human subjects were presented with two kinds of stimuli that were designed to elicit or not elicit the percept of a completed face, although they were physically very similar. By contrasting subjects' behavioral and blood oxygenation level-dependent (BOLD) responses to completed and noncompleted faces, we measured the psychophysical time course of the face completion and its underlying cortical dynamics. We found that face completion manifested its effect between 50 and 250 ms after stimulus onset. Relative to noncompleted faces, completed faces induced weaker BOLD response at early processing phases in retinotopic visual areas V1 and V2 and stronger BOLD response at late processing phases in occipital face area and fusiform face area. Attending away from the stimuli largely abolished these effects. These findings suggest that face completion consists of two synergetic phases: early suppression in lower visual areas and late enhancement in higher visual areas; moreover, attention is necessary to these neural events.

Complex cells in the cat striate cortex have multiple disparity detectors in the three-dimensional binocular receptive fields

Along the visual pathway, neurons generally become more specialized for signaling a limited subset of stimulus attributes and become more invariant to changes in the stimulus position within the receptive fields (RFs). One of the likely mechanisms underlying such invariance appears to be pooling of detectors located at different positions. Does such spatial pooling occur for disparity-selective neurons in primary visual cortex? To examine whether the three-dimensional (3D) binocular RFs are constructed by pooling detectors for binocular disparity, we investigated binocular interactions in the 3D space for neurons in the cat striate cortex. Approximately one-third of complex cells showed the spatial pooling of disparity detectors to a significant degree, whereas the majority of simple cells did not. The degree of spatial pooling of disparity detectors along the preferred orientation axis was generally larger than that along the axis orthogonal to the orientation axis. We then reconstructed 3D binocular RFs in their complete form and examined their structures. Disparity tuning curves were compared across positions along the orientation axis in the RFs. A small population of cells appeared to show a gradual shift of the preferred disparity along this axis, indicating that they can potentially signal inclination in the 3D space. However, the majority of cells exhibited a position-invariant disparity tuning. Finally, disparity tuning curves were examined for all oblique angles in addition to horizontal and vertical. Tunings were broadest along the orientation axis as the disparity energy model predicts.

Two independent mechanisms for motion-in-depth perception: evidence from individual differences

Our forward-facing eyes allow us the advantage of binocular visual information: using the tiny differences between right and left eye views to learn about depth and location in three dimensions. Our visual systems also contain specialized mechanisms to detect motion-in-depth from binocular vision, but the nature of these mechanisms remains controversial. Binocular motion-in-depth perception could theoretically be based on first detecting binocular disparity and then monitoring how it changes over time. The alternative is to monitor the motion in the right and left eye separately and then compare these motion signals. Here we used an individual differences approach to test whether the two sources of information are processed via dissociated mechanisms, and to measure the relative importance of those mechanisms. Our results suggest the existence of two distinct mechanisms, each contributing to the perception of motion-in-depth in most observers. Additionally, for the first time, we demonstrate the relative prevalence of the two mechanisms within a normal population. In general, visual systems appear to rely mostly on the mechanism sensitive to changing binocular disparity, but perception of motion-in-depth is augmented by the presence of a less sensitive mechanism that uses interocular velocity differences. Occasionally, we find observers with the opposite pattern of sensitivity. More generally this work showcases the power of the individual differences approach in studying the functional organization of cognitive systems.

Transfer of perceptual learning of depth discrimination between local and global stereograms

Several previous studies reported differences when stereothresholds are assessed with local-contour stereograms vs. complex random-dot stereograms (RDSs). Dissimilar thresholds may be due to differences in the properties of the stereograms (e.g. spatial frequency content, contrast, inter-element separation, area) or to different underlying processing mechanisms. This study examined the transfer of perceptual learning of depth discrimination between local and global RDSs with similar properties, and vice versa. If global and local stereograms are processed by separate neural mechanisms, then the magnitude and rate of training for the two types of stimuli are likely to differ, and the transfer of training from one stimulus type to the other should be minimal. Based on previous results, we chose RDSs with element densities of 0.17% and 28.3% to serve as the local and global stereograms, respectively. Fourteen inexperienced subjects with normal binocular vision were randomly assigned to either a local- or global- RDS training group. Stereothresholds for both stimulus types were measured before and after 7700 training trials distributed over 10 sessions. Stereothresholds for the trained condition improve for approximately 3000 trials, by an average of 0.36+/-0.08 for local and 0.29+/-0.10 for global RDSs, and level off thereafter. Neither the rate nor the magnitude of improvement differ statistically between the local- and global-training groups. Further, no significant difference exists in the amount of improvement on the trained vs. the untrained targets for either training group. These results are consistent with the operation of a single mechanism to process both local and global stereograms.

Neural modulation by binocular disparity greatest in human dorsal visual stream

Although cortical activation to binocular disparity can be demonstrated throughout occipital and parietal cortices, the relative contributions to depth perception made by different human cortical areas have not been established. To investigate whether different regions are optimized for specific disparity ranges, we have measured the responses of occipital and parietal areas to different magnitudes of binocular disparity. Using stimuli consisting of sinusoidal depth modulations, we measured cortical activation when the stimuli were located at pedestal disparities of 0, 0.1, 0.35, and 0.7 degrees from fixation. Across all areas, occipital and parietal, there was an increase in BOLD signal with increasing pedestal disparity, compared with a plane at zero disparity. However, the greatest modulation of response by the different pedestals was found in the dorsal visual areas and the parietal areas. These differences contrast with the response to the zero disparity plane, compared with fixation, which is greatest in the early visual areas, smaller in the ventral and dorsal visual areas, and absent in parietal areas. Using the simultaneously acquired psychophysical data we also measured a greater response to correct than to incorrect trials, an effect that increased with rising pedestal disparity and was greatest in dorsal visual and parietal areas. These results illustrate that the dorsal stream, along both its occipital and parietal branches, can reliably discriminate a large range of disparities.

Modeling stereopsis via Markov random field

Markov random field (MRF) and belief propagation have given birth to stereo vision algorithms with top performance. This article explores their biological plausibility. First, an MRF model guided by physiological and psychophysical facts was designed. Typically an MRF-based stereo vision algorithm employs a likelihood function that reflects the local similarity of two regions and a potential function that models the continuity constraint. In our model, the likelihood function is constructed on the basis of the disparity energy model because complex cells are considered as front-end disparity encoders in the visual pathway. Our likelihood function is also relevant to several psychological findings. The potential function in our model is constrained by the psychological finding that the strength of the cooperative interaction minimizing relative disparity decreases as the separation between stimuli increases. Our model is tested on three kinds of stereo images. In simulations on images with repetitive patterns, we demonstrate that our model could account for the human depth percepts that were previously explained by the second-order mechanism. In simulations on random dot stereograms and natural scene images, we demonstrate that false matches introduced by the disparity energy model can be reliably removed using our model. A comparison with the coarse-to-fine model shows that our model is able to compute the absolute disparity of small objects with larger relative disparity. We also relate our model to several physiological findings. The hypothesized neurons of the model are selective for absolute disparity and have facilitative extra receptive field. There are plenty of such neurons in the visual cortex. In conclusion, we think that stereopsis can be implemented by neural networks resembling MRF.

Cooperative and competitive interactions facilitate stereo computations in macaque primary visual cortex

Inferring depth from binocular disparities is a difficult problem for the visual system because local features in the left- and right-eye images must be matched correctly to solve this "stereo correspondence problem." Cortical architecture and computational studies suggest that lateral interactions among neurons could help resolve local uncertainty about disparity encoded in individual neurons by incorporating contextual constraints. We found that correlated activity among pairs of neurons in primary visual cortex depended both on disparity-tuning relationships and the stimuli displayed within the receptive fields of the neurons. Nearby pairs of neurons with distinct disparity tuning exhibited a decrease in spike correlation at competing disparities soon after response onset. Distant neuronal pairs of similar disparity tuning exhibited an increase in spike correlation at mutually preferred disparities. The observed correlated activity and response dynamics suggests that local competitive and distant cooperative interactions improve disparity tuning of individual neurons over time. Such interactions could represent a neural substrate for the principal constraints underlying cooperative stereo algorithms.

The neuroscience of vision-based grasping: a functional review for computational modeling and bio-inspired robotics

The topic of vision-based grasping is being widely studied in humans and in other primates using various techniques and with different goals. The fundamental related findings are reviewed in this paper, with the aim of providing researchers from different fields, including intelligent robotics and neural computation, a comprehensive but accessible view on the subject. A detailed description of the principal sensorimotor processes and the brain areas involved is provided following a functional perspective, in order to make this survey especially useful for computational modeling and bio-inspired robotic applications.

The third dimension in the primary visual cortex

Anatomical superposition of the cortical projections from the overlapping visual fields of the two eyes does not make it obvious how the disposition of objects in the third dimension is encoded. Hubel and Wiesel's demonstration that units in the primary visual cortex of the mammal respond preferentially to elongated contours of specific orientation encouraged the inquiry into whether binocular disparity might not similarly be represented as an attribute interdigitated within the orderly progression of position. When this was found to indeed be the case, this entrained a brisk research activity into the disparity of receptive fields of single units in the primary visual cortex and the influence on their response of the three-dimensional locations of outside world stimuli. That cells' preferred orientations covered the whole gamut whereas space perception required only horizontal disparity was an apparent paradox that needed resolution. A connection with an observer's stereoscopic performance was made by the discovery that cells in the primate primary visual cortex display good tuning to the disparity in random-dot stereograms. But a wide gap still remains between the properties of these cortical units and human stereo thresholds in simple target configurations, let alone depth judgments in which perceptual and cognitive factors enter. When the neural circuits in the primary visual cortex that are involved in processing depth are eventually traced in detail they will also need to have properties that allow for the plasticity in learning and experience.

From disparity to depth: how to make a grating and a plaid appear in the same depth plane

Even though binocular disparity is a very well-studied cue to depth, the function relating disparity and perceived depth has been characterized only for the case of horizontal disparities. We sought to determine the general relationship between disparity and depth for a particular set of stimuli. The horizontal disparity direction is a special case, albeit an especially important one. Non-horizontal disparities arise from a number of sources under natural viewing condition. Moreover, they are implicit in patterns that are one-dimensional, such as gratings, lines, and edges, and in one-dimensional components of two-dimensional patterns, where a stereo matching direction is not well-defined. What function describes perceived depth in these cases? To find out, we measured the phase disparities that produced depth matches between a reference stimulus and a test stimulus. The reference stimulus was two-dimensional, a plaid; the test stimulus was one-dimensional, a grating. We find that horizontal disparity is no more important than other disparity directions in determining depth matches between these two stimuli. As a result, a grating and a plaid appear equal in depth when their horizontal disparities are, in general, unequal. Depth matches are well predicted by a simple disparity vector calculation; they survive changes in component parameters that conserve these vector quantities. The disparity vector rule also describes how the disparities of 1-D components might contribute to the perceived depth of 2-D stimuli.

Disparity- and velocity-based signals for three-dimensional motion perception in human MT+

How does the primate visual system encode three-dimensional motion? The macaque middle temporal area (MT) and the human MT complex (MT+) have well-established sensitivity to two-dimensional frontoparallel motion and static disparity. However, evidence for sensitivity to three-dimensional motion has remained elusive. We found that human MT+ encodes two binocular cues to three-dimensional motion: changing disparities over time and interocular comparisons of retinal velocities. By varying important properties of moving dot displays, we distinguished these three-dimensional motion signals from their constituents, instantaneous binocular disparity and monocular retinal motion. An adaptation experiment confirmed direction selectivity for three-dimensional motion. Our results indicate that MT+ carries critical binocular signals for three-dimensional motion processing, revealing an important and previously overlooked role for this well-studied brain area.

Kinematics of pointing movements made in a virtual versus a physical 3-dimensional environment in healthy and stroke subjects

OBJECTIVE

To compare kinematics of 3-dimensional pointing movements performed in a virtual environment (VE) displayed through a head-mounted display with those made in a physical environment.

DESIGN

Observational study of movement in poststroke and healthy subjects.

SETTING

Motion analysis laboratory.

PARTICIPANTS

Adults (n=15; 4 women; 59+/-15.4y) with chronic poststroke hemiparesis were recruited. Participants had moderate upper-limb impairment with Chedoke-McMaster Arm Scores ranging from 3 to 6 out of 7. Twelve healthy subjects (6 women; 53.3+/-17.1y) were recruited from the community.

INTERVENTIONS

Not applicable.

MAIN OUTCOME MEASURES

Arm and trunk kinematics were recorded in similar virtual and physical environments with an Optotrak System (6 markers; 100Hz; 5s). Subjects pointed as quickly and as accurately as possible to 6 targets (12 trials/target in a randomized sequence) placed in arm workspace areas requiring different arm movement patterns and levels of difficulty. Movements were analyzed in terms of performance outcome measures (endpoint precision, trajectory, peak velocity) and arm and trunk movement patterns (elbow and shoulder ranges of motion, elbow/shoulder coordination, trunk displacement, rotation).

RESULTS

For healthy subjects, precision and trajectory straightness were higher in VE when pointing to contralateral targets, and movements were slower for all targets in VE. Stroke participants made less accurate and more curved movements in VE and used less trunk displacement. Elbow/shoulder coordination differed when pointing to the lower ipsilateral target. There were no group-by-environment interactions.

CONCLUSIONS

Movements in both environments were sufficiently similar to consider VE a valid environment for clinical interventions and motor control studies.

Image-size differences worsen stereopsis independent of eye position

With the eyes in forward gaze, stereo performance worsens when one eye's image is larger than the other's. Near, eccentric objects naturally create retinal images of different sizes. Does this mean that stereopsis exhibits deficits for such stimuli? Or does the visual system compensate for the predictable image-size differences? To answer this, we measured discrimination of a disparity-defined shape for different relative image sizes. We did so for different gaze directions, some compatible with the image-size difference and some not. Magnifications of 10-15% caused a clear worsening of stereo performance. The worsening was determined only by relative image size and not by eye position. This shows that no neural compensation for image-size differences accompanies eye-position changes, at least prior to disparity estimation. We also found that a local cross-correlation model for disparity estimation performs like humans in the same task, suggesting that the decrease in stereo performance due to image-size differences is a byproduct of the disparity-estimation method. Finally, we looked for compensation in an observer who has constantly different image sizes due to differing eye lengths. She performed best when the presented images were roughly the same size, indicating that she has compensated for the persistent image-size difference.

Limits of stereopsis explained by local cross-correlation

Human stereopsis has two well-known constraints: the disparity-gradient limit, which is the inability to perceive depth when the change in disparity within a region is too large, and the limit of stereoresolution, which is the inability to perceive spatial variations in disparity that occur at too fine a spatial scale. We propose that both limitations can be understood as byproducts of estimating disparity by cross-correlating the two eyes' images, the fundamental computation underlying the disparity-energy model. To test this proposal, we constructed a local cross-correlation model with biologically motivated properties. We then compared model and human behaviors in the same psychophysical tasks. The model and humans behaved quite similarly: they both exhibited a disparity-gradient limit and had similar stereoresolution thresholds. Performance was affected similarly by changes in a variety of stimulus parameters. By modeling the effects of stimulus blur and of using different sizes of image patches, we found evidence that the smallest neural mechanism humans use to estimate disparity is 3-6 arcmin in diameter. We conclude that the disparity-gradient limit and stereoresolution are indeed byproducts of using local cross-correlation to estimate disparity.

Structure and function come unglued in the visual cortex

In this issue of Neuron, Chowdhury and DeAngelis report that training monkeys to perform a fine depth discrimination abolishes the contribution of signals from area MT to the execution of a different, coarse depth discrimination. This result calls into question the principle of associating particular visual areas with particular visual functions, by showing that such associations are modifiable by experience.

Mechanisms underlying the transformation of disparity signals from V1 to V2 in the macaque

Stereo vision relies on cortical signals that encode binocular disparity. In V1, the disparity energy model explains many features of binocular interaction, but it overestimates the responses to anticorrelated images. Combining the outputs of two, or more, energy model-like subunits [two-subunit (2SU) model] can resolve this discrepancy and provides an alternative explanation for disparity signals previously thought to indicate phase disparity between the receptive fields (RFs) of each eye. The 2SU model naturally explains how "near/far" (odd-symmetric) tuning becomes dominant in extrastriate cortex. To compare the energy and the 2SU models, we used a broadband compound grating and applied a common interocular phase difference to all spatial frequency components (a stimulus phase disparity), combined with a common spatial displacement (a stimulus position disparity). This produces binocular images that never occur in natural viewing, for which the 2SU model and the energy model make distinctively different predictions. Responses of neurons recorded from both V1 and V2 of awake rhesus macaques systematically deviated from the predictions of the energy model, in accordance with the 2SU model. These deviations correlated with the symmetry of the tuning curve, indicating that the 2SU mechanism is exploited to produce odd symmetry. Nonetheless, individual subunits also contain RF phase disparity that contributes to odd symmetry. The results suggest that neurons in V2 probably inherit phase disparity signals from V1 neurons, but systematically combine input from V1 neurons with different position disparities, in a way that elaborates odd-symmetric tuning and extends the range of disparities encoded by single neurons.

Fine discrimination training alters the causal contribution of macaque area MT to depth perception

When a new perceptual task is learned, plasticity occurs in the brain to mediate improvements in performance with training. How do these changes affect the neural substrates of previously learned tasks? We addressed this question by examining the effect of fine discrimination training on the causal contribution of area MT to coarse depth discrimination. When monkeys are trained to discriminate between two coarse absolute disparities (near versus far) embedded in noise, reversible inactivation of area MT devastates performance. In contrast, after animals are trained to discriminate fine differences in relative disparity, MT inactivation no longer impairs coarse depth discrimination. This effect does not result from changes in the disparity tuning of MT neurons, suggesting plasticity in the flow of disparity signals to decision circuitry. These findings show that the contribution of particular brain area to task performance can change dramatically as a result of learning new tasks.

Representation of binocular surfaces by cortical neurons

Useful representations of the three-dimensional (3D) world go beyond assigning depth to individual points, building maps of surfaces and shapes. Studies in a wide range of extrastriate cortical areas have shown that single neurons show selective responses to 3D surfaces. The extent to which this advances the representation beyond that provided by the earliest binocular signals requires careful evaluation. We conclude that current data are not sufficient to identify distinctive contributions from different cortical areas to the binocular representation of 3D surfaces.

Cyclopean geometry of binocular vision

The geometry of binocular projection is analyzed in relation to the primate visual system. An oculomotor parameterization that includes the classical vergence and version angles is defined. It is shown that the epipolar geometry of the system is constrained by binocular coordination of the eyes. A local model of the scene is adopted in which depth is measured relative to a plane containing the fixation point. These constructions lead to an explicit parameterization of the binocular disparity field involving the gaze angles as well as the scene structure. The representation of visual direction and depth is discussed with reference to the relevant psychophysical and neurophysiological literature.

Postnatal development of disparity sensitivity in visual area 2 (v2) of macaque monkeys

Macaque monkeys do not reliably discriminate binocular depth cues until about 8 wk of age. The neural factors that limit the development of fine depth perception in primates are not known. In adults, binocular depth perception critically depends on detection of relative binocular disparities and the earliest site in the primate visual brain where a substantial proportion of neurons are capable of discriminating relative disparity is visual area 2 (V2). We examined the disparity sensitivity of V2 neurons during the first 8 wk of life in infant monkeys and compared the responses of V2 neurons to those of V1 neurons. We found that the magnitude of response modulation in V2 and V1 neurons as a function of interocular spatial phase disparity was adult-like as early as 2 wk of age. However, the optimal spatial frequency and binocular response rate of these disparity sensitive neurons were more than an octave lower in 2- and 4-wk-old infants than in adults. Consequently, despite the lower variability of neuronal firing in V2 and V1 neurons of infant monkeys, the ability of these neurons to discriminate fine disparity differences was significantly reduced compared with adults. This reduction in disparity sensitivity of V2 and V1 neurons is likely to limit binocular depth perception during the first several weeks of a monkey's life.

The relationship between subthreshold and suprathreshold ocular dominance in cat primary visual cortex

Primary visual cortex (V1) is the site at which right and left eye pathways are first integrated, leading to a representation of the visual world in depth. The ocular dominance (OD) of individual cortical neurons varies and may be changed by altering visual experience during the developmental critical period. Estimates of OD, commonly used to quantify the right and left eye synaptic inputs, have previously been based on spike rate. Membrane potential (V(m)), however, is more closely related to the synaptic inputs onto neurons and should therefore more closely reflect the degree of input from the two eyes. To determine the relationship between OD based on membrane potential and on spike rate, intracellular recordings were made from visual cortical neurons. OD based on spike rate was systematically more monocular than OD based on membrane potential. The discrepancy between membrane-potential OD and spike-rate OD may be accounted for by a simple model of V(m)-to-spike-rate transformation.

A map for horizontal disparity in monkey V2

The perception of visual depth is determined by integration of spatial disparities of inputs from the two eyes. Single cells in visual cortex of monkeys are known to respond to specific binocular disparities; however, little is known about their functional organization. We now show, using intrinsic signal optical imaging and single-unit physiology, that, in the thick stripe compartments of the second visual area (V2), there is a clustered organization of Near cells and Far cells, and moreover, there are topographic maps for Near to Far disparities within V2. Our findings suggest that maps for visual disparity are calculated in V2, and demonstrate parallels in functional organization between the thin, pale, and thick stripes of V2.

Updating Target Distance Across Eye Movements in Depth

We tested between two coding mechanisms that the brain may use to retain distance information about a target for a reaching movement across vergence eye movements. If the brain was to encode a retinal disparity representation (retinal model), i.e., target depth relative to the plane of fixation, each vergence eye movement would require an active update of this representation to preserve depth constancy. Alternatively, if the brain was to store an egocentric distance representation of the target by integrating retinal disparity and vergence signals at the moment of target presentation, this representation should remain stable across subsequent vergence shifts (nonretinal model). We tested between these schemes by measuring errors of human reaching movements ( n = 14 subjects) to remembered targets, briefly presented before a vergence eye movement. For comparison, we also tested their directional accuracy across version eye movements. With intervening vergence shifts, the memory-guided reaches showed an error pattern that was based on the new eye position and on the depth of the remembered target relative to that position. This suggests that target depth is recomputed after the gaze shift, supporting the retinal model. Our results also confirm earlier literature showing retinal updating of target direction. Furthermore, regression analyses revealed updating gains close to one for both target depth and direction, suggesting that the errors arise after the updating stage during the subsequent reference frame transformations that are involved in reaching.

Higher order visual processing in macaque extrastriate cortex

The extrastriate cortex of primates encompasses a substantial portion of the cerebral cortex and is devoted to the higher order processing of visual signals and their dispatch to other parts of the brain. A first step towards the understanding of the function of this cortical tissue is a description of the selectivities of the various neuronal populations for higher order aspects of the image. These selectivities present in the various extrastriate areas support many diverse representations of the scene before the subject. The list of the known selectivities includes that for pattern direction and speed gradients in middle temporal/V5 area; for heading in medial superior temporal visual area, dorsal part; for orientation of nonluminance contours in V2 and V4; for curved boundary fragments in V4 and shape parts in infero-temporal area (IT); and for curvature and orientation in depth from disparity in IT and CIP. The most common putative mechanism for generating such emergent selectivity is the pattern of excitatory and inhibitory linear inputs from the afferent area combined with nonlinear mechanisms in the afferent and receiving area.

Effect of vertical disparities on depth representation in macaque monkeys: MT physiology and behavior

Horizontal binocular disparities provide information about the distance of objects relative to the point of ocular fixation and must be combined with an estimate of viewing distance to recover the egocentric distance of an object. Vergence angle and the gradient of vertical disparities across the visual field are thought to provide independent sources of viewing distance information based on human behavioral studies. Although the effect of vergence angle on horizontal disparity selectivity in early visual cortex has been examined (with mixed results), the effect of the vertical disparity field has not been explored. We manipulated the vertical disparities in a large random-dot stimulus to simulate different viewing distances, and we examined the effect of this manipulation on both the responses of neurons in the middle temporal (MT) area and on the psychophysical performance of the animal in a curvature discrimination task. We report here that alterations to the vertical disparity field have no effect on the horizontal disparity tuning of MT neurons. However, the same manipulation strongly and systematically biases the monkey's judgments of curvature, consistent with previous human studies. We conclude that monkeys, like humans, make use of the vertical disparity field to estimate viewing distance, but that the physiological mechanisms for this effect occur either downstream of MT or in a different pathway.

Integrating motion and depth via parallel pathways

Processing of visual information is both parallel and hierarchical, with each visual area richly interconnected with other visual areas. An example of the parallel architecture of the primate visual system is the existence of two principal pathways providing input to the middle temporal visual area (MT): namely, a direct projection from striate cortex (V1), and a set of indirect projections that also originate in V1 but then relay through V2 and V3. Here we have reversibly inactivated the indirect pathways while recording from MT neurons and measuring eye movements in alert monkeys, a procedure that has enabled us to assess whether the two different input pathways are redundant or whether they carry different kinds of information. We find that this inactivation causes a disproportionate degradation of binocular disparity tuning relative to direction tuning in MT neurons, suggesting that the indirect pathways are important in the recovery of depth in three-dimensional scenes.

Adaptation to natural binocular disparities in primate V1 explained by a generalized energy model

Sensory processing in the brain is thought to have evolved to encode naturally occurring stimuli efficiently. We report an adaptation in binocular cortical neurons that reflects the tight constraints imposed by the geometry of 3D vision. We show that the widely used binocular energy model predicts that neurons dedicate part of their dynamic range to impossible combinations of left and right images. Approximately 42% of the neurons we record from V1 of awake monkeys behave in this way (a powerful confirmation of the model), while about 58% deviate from the model in a manner that concentrates more of their dynamic range on stimuli that obey the constraints of binocular geometry. We propose a simple extension of the energy model, using multiple subunits, that explains the adaptation we observe, as well as other properties of binocular neurons that have been hard to account for, such as the response to anti-correlated stereograms.