Stereopsis, vertical disparity and relief transformations

What's special about horizontal disparity

Horizontal disparity has been recognized as the primary signal driving stereoscopic depth since the invention of the stereoscope in the 1830s. It has a unique status in our understanding of binocular vision. The direction of offset of the eyes gives the disparities of corresponding image point locations across the two retinas a strong horizontal bias. Beyond the retina, other factors give shape to the effective disparity direction used by visual mechanisms. The influence of orientation is examined here. I argue that horizontal disparity is an inflection point along a continuum of effective directions, and its role in stereo vision can be reinterpreted. The pointwise geometric justification for its special status neglects the oriented structural elements of spatial vision, its physiological support is equivocal, and psychophysical support of its special status may partially reflect biased stimulus sampling. The literature shows that horizontal disparity plays no particular role in the processing of one-dimensional stimuli, a reflection of the stereo aperture problem. The resulting depth is non-veridical, even non-transitive. Although one-dimensional components contribute to the stereo depth of visual objects generally, two-dimensional stimuli appear not to inherit the aperture problem. However, a look at the two-dimensional stimuli that predominate in experimental studies shows regularities in orientation that give a new perspective on horizontal disparity.

Stereopsis without correspondence

Stereopsis has traditionally been considered a complex visual ability, restricted to large-brained animals. The discovery in the 1980s that insects, too, have stereopsis, therefore, challenged theories of stereopsis. How can such simple brains see in three dimensions? A likely answer is that insect stereopsis has evolved to produce simple behaviour, such as orienting towards the closer of two objects or triggering a strike when prey comes within range. Scientific thinking about stereopsis has been unduly anthropomorphic, for example assuming that stereopsis must require binocular fusion or a solution of the stereo correspondence problem. In fact, useful behaviour can be produced with very basic stereoscopic algorithms which make no attempt to achieve fusion or correspondence, or to produce even a coarse map of depth across the visual field. This may explain why some aspects of insect stereopsis seem poorly designed from an engineering point of view: for example, paying no attention to whether interocular contrast or velocities match. Such algorithms demonstrably work well enough in practice for their species, and may prove useful in particular autonomous applications. This article is part of a discussion meeting issue 'New approaches to 3D vision'.

Binocular Information Improves the Reliability and Consistency of Pictorial Relief

Binocular disparity is an important cue to three-dimensional shape. We assessed the contribution of this cue to the reliability and consistency of depth in stereoscopic photographs of natural scenes. Observers viewed photographs of cluttered scenes while adjusting a gauge figure to indicate the apparent three-dimensional orientation of the surfaces of objects. The gauge figure was positioned on the surfaces of objects at multiple points in the scene, and settings were made under monocular and binocular, stereoscopic viewing. Settings were used to create a depth relief map, indicating the apparent three-dimensional structure of the scene. We found that binocular cues increased the magnitude of apparent depth, the reliability of settings across repeated measures, and the consistency of perceived depth across participants. These results show that binocular cues make an important contribution to the precise and accurate perception of depth in natural scenes that contain multiple pictorial cues.

Stereoscopic depth constancy for physical objects and their virtual counterparts

Stereopsis plays an important role in depth perception; if so, disparity-defined depth should not vary with distance. However, studies of stereoscopic depth constancy often report systematic distortions in depth judgments over distance, particularly for virtual stimuli. Our aim was to understand how depth estimation is impacted by viewing distance and display-based cue conflicts by replicating physical objects in virtual counterparts. To this end, we measured perceived depth using virtual textured half-cylinders and identical three-dimensional (3D) printed versions at two viewing distances under monocular and binocular conditions. Virtual stimuli were viewed using a mirror stereoscope and an Oculus Rift head-mounted display (HMD), while physical stimuli were viewed in a controlled test environment. Depth judgments were similar in both virtual apparatuses, which suggests that variations in the viewing geometry and optics of the HMD have little impact on perceived depth. When viewing physical stimuli binocularly, judgments were accurate and exhibited stereoscopic depth constancy. However, in all cases, depth was underestimated for virtual stimuli and failed to achieve depth constancy. It is clear that depth constancy is only complete for cue-rich physical stimuli and that the failure of constancy in virtual stimuli is due to the presence of the vergence-accommodation conflict. Further, our post hoc analysis revealed that prior experience with virtual and physical environments had a strong effect on depth judgments. That is, performance in virtual environments was enhanced by limited exposure to a related task using physical objects.

An egocentric straight-ahead bias in primate's vision

As we plan to reach or manipulate objects, we generally orient our body so as to face them. Other objects occupying the same portion of space will likely represent potential obstacles for the intended action. Thus, either as targets or as obstacles, the objects located straight in front of us are often endowed with a special behavioral status. Here, we review a set of recent electrophysiological, imaging and behavioral studies bringing converging evidence that the objects which lie straight-ahead are subject to privileged visual processing. More precisely, these works collectively demonstrate that when gaze steers central vision away from the straight-ahead direction, the latter is still prioritized in peripheral vision. Straight-ahead objects evoke (1) stronger neuronal responses in macaque peripheral V1 neurons, (2) stronger EEG and fMRI activations across the human visual cortex and (3) faster reactive hand and eye movements. Here, we discuss the functional implications and underlying mechanisms behind this phenomenon. Notably, we propose that it can be considered as a new type of visuospatial attentional mechanism, distinct from the previously documented classes of endogenous and exogenous attention.

Stereoscopic depth constancy from a different direction

To calibrate stereoscopic depth from disparity our visual system must compensate for an object's egocentric location. Ideally, the perceived three-dimensional shape and size of objects in visual space should be invariant with their location such that rigid objects have a consistent identity and shape. These percepts should be accurate enough to support both perceptual judgments and visually-guided interaction. This theoretical note reviews the relationship of stereoscopic depth constancy to the geometry of stereoscopic space and seemingly esoteric concepts like the horopter. We argue that to encompass the full scope of stereoscopic depth constancy, researchers need to consider not just distance but also direction, that is 3D egocentric location in space. Judgements of surface orientation need to take into account the shape of the horopter and the computation of metric depth (when tasks depend on it) must compensate for direction as well as distance to calibrate disparities. We show that the concept of the horopter underlies these considerations and that the relationship between depth constancy and the horopter should be more explicit in the literature.

Stereoscopic 3D geometric distortions analyzed from the viewer's point of view

Stereoscopic 3D (S3D) geometric distortions can be introduced by mismatches among image capture, display, and viewing configurations. In previous work of S3D geometric models, geometric distortions have been analyzed from a third-person perspective based on the binocular depth cue (i.e., binocular disparity). A third-person perspective is different from what the viewer sees since monocular depth cues (e.g., linear perspective, occlusion, and shadows) from different perspectives are different. However, depth perception in a 3D space involves both monocular and binocular depth cues. Geometric distortions that are solely predicted by the binocular depth cue cannot describe what a viewer really perceives. In this paper, we combine geometric models and retinal disparity models to analyze geometric distortions from the viewer's perspective where both monocular and binocular depth cues are considered. Results show that binocular and monocular depth-cue conflicts in a geometrically distorted S3D space. Moreover, user-initiated head translations averting from the optimal viewing position in conventional S3D displays can also introduce geometric distortions, which are inconsistent with our natural 3D viewing condition. The inconsistency of depth cues in a dynamic scene may be a source of visually induced motions sickness.

Attentional selection in judgments of stereo depth

Stereoscopic depth is most useful when it comes from relative rather than absolute disparities. However, the depth perceived from relative disparities can vary with stimulus parameters that have no connection with depth or are irrelevant to the task. We investigated observers' ability to judge the stereo depth of task-relevant stimuli while ignoring irrelevant stimuli. The calculation of depth from disparity differs for 1-D and 2-D stimuli and we investigated the role this difference plays in observers' ability to selectively process relevant information. We show that the presence of irrelevant disparities affects perceived depth differently depending on stimulus dimensionality. Observers could not ignore disparities of irrelevant stimuli when they judged the relative depth between a 1-D stimulus (a grating) and a 2-D stimulus (a plaid). Yet these irrelevant disparities did not affect judgments of the relative depth between 2-D stimuli. Two processes contributing to stereo depth were identified, only one of which computes depth from a horizontal disparity metric and permits attentional selection. The other uses all stimuli, relevant and irrelevant, to calculate an effective disparity direction for comparing disparity magnitudes. These processes produce inseparable effects in most data sets. Using multiple disparity directions and comparing 1-D and 2-D stimuli can distinguish them.

Stereopsis in animals: evolution, function and mechanisms

Stereopsis is the computation of depth information from views acquired simultaneously from different points in space. For many years, stereopsis was thought to be confined to primates and other mammals with front-facing eyes. However, stereopsis has now been demonstrated in many other animals, including lateral-eyed prey mammals, birds, amphibians and invertebrates. The diversity of animals known to have stereo vision allows us to begin to investigate ideas about its evolution and the underlying selective pressures in different animals. It also further prompts the question of whether all animals have evolved essentially the same algorithms to implement stereopsis. If so, this must be the best way to do stereo vision, and should be implemented by engineers in machine stereopsis. Conversely, if animals have evolved a range of stereo algorithms in response to different pressures, that could inspire novel forms of machine stereopsis appropriate for distinct environments, tasks or constraints. As a first step towards addressing these ideas, we here review our current knowledge of stereo vision in animals, with a view towards outlining common principles about the evolution, function and mechanisms of stereo vision across the animal kingdom. We conclude by outlining avenues for future work, including research into possible new mechanisms of stereo vision, with implications for machine vision and the role of stereopsis in the evolution of camouflage.

Summary: Stereopsis has evolved independently in different animals. We review the various functions it serves and the variety of mechanisms that could underlie stereopsis in different species.

Early dynamics of stereoscopic surface slant perception

Surface orientation is an important visual primitive that can be estimated from monocular or binocular (stereoscopic) signals. Changes in motor planning occur within about 200 ms after either type of signal is perturbed, but the time it takes for apparent (perceived) slant to develop from stereoscopic cues is not known. Apparent slant sometimes develops very slowly (Gillam, Chambers, & Russo, 1988; van Ee & Erkelens, 1996). However, these long durations could reflect the time it takes for the visual system to resolve conflicts between slant cues that inevitably specify different slants in laboratory displays (Allison & Howard, 2000). We used a speed–accuracy tradeoff analysis to measure the time it takes to discriminate slant, allowing us to report psychometric functions as a function of response time. Observers reported which side of a slanted surface was farther, with a temporal deadline for responding that varied block-to-block. Stereoscopic slant discrimination rose above chance starting at 200 ms after stimulus onset. Unexpectedly, observers discriminated slant from binocular disparity faster than texture, and for stereoscopic whole-field stimuli faster than stereoscopic slant contrast stimuli. However, performance after the initial deviation from chance increased more rapidly for slant-contrast stimuli than whole-field stimuli. Discrimination latencies were similar for slants about the horizontal and vertical axes, but performance increased faster for slants about the vertical axis. Finally, slant from vertical disparity was somewhat slower than slant from horizontal disparity, which may reflect cue conflict. These results demonstrate, in contradiction with the previous literature, that the perception of slant from disparity happens very quickly—in fact, more quickly than the perception of slant from texture—and in comparable time to the simple perception of brightness from luminance.

Viewing geometry determines the contribution of binocular vision to the online control of grasping

Binocular vision is often assumed to make a specific, critical contribution to online visual control of grasping by providing precise information about the separation between digits and object. This account overlooks the ‘viewing geometry’ typically encountered in grasping, however. Separation of hand and object is rarely aligned precisely with the line of sight (the visual depth dimension), and analysis of the raw signals suggests that, for most other viewing angles, binocular feedback is less precise than monocular feedback. Thus, online grasp control relying selectively on binocular feedback would not be robust to natural changes in viewing geometry. Alternatively, sensory integration theory suggests that different signals contribute according to their relative precision, in which case the role of binocular feedback should depend on viewing geometry, rather than being ‘hard-wired’. We manipulated viewing geometry, and assessed the role of binocular feedback by measuring the effects on grasping of occluding one eye at movement onset. Loss of binocular feedback resulted in a significantly less extended final slow-movement phase when hand and object were separated primarily in the frontoparallel plane (where binocular information is relatively imprecise), compared to when they were separated primarily along the line of sight (where binocular information is relatively precise). Consistent with sensory integration theory, this suggests the role of binocular (and monocular) vision in online grasp control is not a fixed, ‘architectural’ property of the visuo-motor system, but arises instead from the interaction of viewer and situation, allowing robust online control across natural variations in viewing geometry.

The Role of Vertical Disparity in Distance and Depth Perception as Revealed by Different Stereo-Camera Configurations

Vertical binocular disparity is a source of distance information allowing the portrayal of the layout and 3D metrics of the visual space. The role of vertical disparity in the perception of depth, size, curvature, or slant of surfaces was revealed in several previous studies using cue conflict paradigms. In this study, we varied the configuration of stereo-cameras to investigate how changes in the horizontal and vertical disparity fields, conflicting with the vergence cue, affect perceived distance and depth. In four experiments, observers judged the distance of a cylinder displayed in front of a large fronto-parallel surface. Experiment 1 revealed that the presence of a background surface decreases the uncertainty in judgments of distance, suggesting that observers use the relative horizontal disparity between the target and the background as a cue to distance. Two other experiments showed that manipulating the pattern of vertical disparity affected both distance and depth perception. When vertical disparity specified a nearer distance than vergence (convergent cameras), perceived distance and depth were underestimated as compared with the condition where vertical disparity was congruent with vergence cues (parallel cameras). When vertical disparity specified a further distance than vergence, namely an infinite distance, distance and depth were overestimated. The removal of the vertical distortion lessened the effect on perceived distance. Overall, the results suggest that the vertical disparity introduced by the specific camera configuration is mainly responsible for the effect. These findings outline the role of vertical disparity in distance and depth perception and support the use of parallel cameras for designing stereograms.

Vertical Disparity Affects Shape and Size Judgments across Surfaces Separated in Depth

Vertical binocular disparity provides a useful source of information allowing three-dimensional (3-D) shape to be recovered from horizontal binocular disparity. In order to influence metric shape judgments, a large field of view is required, suggesting that vertical disparity may play a limited role in the perception of objects projecting small retinal images. This limitation could be overcome if vertical disparity information could be pooled over wide areas of 3-D space. This was investigated by assessing the effect of vertical disparity scaling of a large surround surface on the perceived size and 3-D shape of a small, central object. Observers adjusted the size and shape of a virtual, binocularly defined ellipsoid to match those of a real, hand-held tennis ball. The virtual ball was presented at three distances (200, 325, and 450 mm). Vertical disparities in a large surround surface were manipulated to be consistent with a distance of 160 mm or infinity. Both shape and size settings were influenced by this manipulation. This effect did not depend on presenting the surround and target objects at the same distance. These results suggest that the influence of vertical disparity on the perceived distance to a surface also affects the estimated distance of other visible surfaces. Vertical disparities are therefore important in the perception of metric depth, even for objects that in themselves subtend only small retinal images.

Stereopsis is adaptive for the natural environment

Humans and many animals have forward-facing eyes providing different views of the environment. Precise depth estimates can be derived from the resulting binocular disparities, but determining which parts of the two retinal images correspond to one another is computationally challenging. To aid the computation, the visual system focuses the search on a small range of disparities. We asked whether the disparities encountered in the natural environment match that range. We did this by simultaneously measuring binocular eye position and three-dimensional scene geometry during natural tasks. The natural distribution of disparities is indeed matched to the smaller range of correspondence search. Furthermore, the distribution explains the perception of some ambiguous stereograms. Finally, disparity preferences of macaque cortical neurons are consistent with the natural distribution.

Perceived depth in non-transitive stereo displays

The separation between the eyes shapes the distribution of binocular disparities and gives a special role to horizontal disparities. However, for one-dimensional stimuli, disparity direction, like motion direction, is linked to stimulus orientation. This makes the perceived depth of one-dimensional stimuli orientation dependent and generally non-veridical. It also allows perceived depth to violate transitivity. Three stimuli, A, B, and C, can be arranged such that A > B (stimulus A is seen as farther than stimulus B when they are presented together) and B > C, yet A ⩽ C. This study examines how the visual system handles the depth of A, B, and C when they are presented together, forming a pairwise inconsistent stereo display. Observers' depth judgments of displays containing a grating and two plaids resolved transitivity violations among the component stimulus pairs. However, these judgments were inconsistent with judgments of the same stimuli within depth-consistent displays containing no transitivity violations. To understand the contribution of individual disparity signals, observers were instructed in subsequent experiments to judge the depth of a subset of display stimuli. This attentional instruction was ineffective; relevant and irrelevant stimuli contributed equally to depth judgments. Thus, the perceived depth separating a pair of stimuli depended on the disparities of the other stimuli presented concurrently. This context dependence of stereo depth can be approximated by an obligatory pooling and comparison of the disparities of one- and two-dimensional stimuli along an axis defined locally by the stimuli.

Modeling depth from motion parallax with the motion/pursuit ratio

The perception of unambiguous scaled depth from motion parallax relies on both retinal image motion and an extra-retinal pursuit eye movement signal. The motion/pursuit ratio represents a dynamic geometric model linking these two proximal cues to the ratio of depth to viewing distance. An important step in understanding the visual mechanisms serving the perception of depth from motion parallax is to determine the relationship between these stimulus parameters and empirically determined perceived depth magnitude. Observers compared perceived depth magnitude of dynamic motion parallax stimuli to static binocular disparity comparison stimuli at three different viewing distances, in both head-moving and head-stationary conditions. A stereo-viewing system provided ocular separation for stereo stimuli and monocular viewing of parallax stimuli. For each motion parallax stimulus, a point of subjective equality (PSE) was estimated for the amount of binocular disparity that generates the equivalent magnitude of perceived depth from motion parallax. Similar to previous results, perceived depth from motion parallax had significant foreshortening. Head-moving conditions produced even greater foreshortening due to the differences in the compensatory eye movement signal. An empirical version of the motion/pursuit law, termed the empirical motion/pursuit ratio, which models perceived depth magnitude from these stimulus parameters, is proposed.

Vertical Size Disparity and the Correction of Stereo Correspondence

We examined the stage of vertical-disparity processing that produces a global stereoscopic slant. In two psychophysical experiments, we measured perceived slant about a vertical axis for two-dimensional stereoscopic patterns consisting of random dots, concentric lines, and radial lines. Binocular image differences were introduced into each pattern by vertically magnifying either the entire image for the right eye or that for the left eye. Because the continuous lines were geometrically ambiguous in local stereo correspondence, the three patterns differed from each other in the local horizontal disparity measured in retinal coordinates. The two experiments revealed that, despite the differences in the retinal horizontal disparity, the slant settings were generally similar for the three patterns, in both short and long viewing distances (25 cm and 120 cm, respectively). These results are consistent with the idea that the visual system uses vertical disparity at least when establishing local stereo correspondence. A Bayesian model is proposed to account for the results.

Stereoscopy and the Human Visual System

Stereoscopic displays have become important for many applications, including operation of remote devices, medical imaging, surgery, scientific visualization, and computer-assisted design. But the most significant and exciting development is the incorporation of stereo technology into entertainment: specifically, cinema, television, and video games. In these applications for stereo, three-dimensional (3D) imagery should create a faithful impression of the 3D structure of the scene being portrayed. In addition, the viewer should be comfortable and not leave the experience with eye fatigue or a headache. Finally, the presentation of the stereo images should not create temporal artifacts like flicker or motion judder. This paper reviews current research on stereo human vision and how it informs us about how best to create and present stereo 3D imagery. The paper is divided into four parts: (1) getting the geometry right, (2) depth cue interactions in stereo 3D media, (3) focusing and fixating on stereo images, and (4) how temporal presentation protocols affect flicker, motion artifacts, and depth distortion.

Perceived surface slant is systematically biased in the actively-generated optic flow

Humans make systematic errors in the 3D interpretation of the optic flow in both passive and active vision. These systematic distortions can be predicted by a biologically-inspired model which disregards self-motion information resulting from head movements (Caudek, Fantoni, & Domini 2011). Here, we tested two predictions of this model: (1) A plane that is stationary in an earth-fixed reference frame will be perceived as changing its slant if the movement of the observer's head causes a variation of the optic flow; (2) a surface that rotates in an earth-fixed reference frame will be perceived to be stationary, if the surface rotation is appropriately yoked to the head movement so as to generate a variation of the surface slant but not of the optic flow. Both predictions were corroborated by two experiments in which observers judged the perceived slant of a random-dot planar surface during egomotion. We found qualitatively similar biases for monocular and binocular viewing of the simulated surfaces, although, in principle, the simultaneous presence of disparity and motion cues allows for a veridical recovery of surface slant.

The precision of binocular and monocular depth judgments in natural settings

We measured binocular and monocular depth thresholds for objects presented in a real environment. Observers judged the depth separating a pair of metal rods presented either in relative isolation, or surrounded by other objects, including a textured surface. In the isolated setting, binocular thresholds were greatly superior to the monocular thresholds by as much as a factor of 18. The presence of adjacent objects and textures improved the monocular thresholds somewhat, but the superiority of binocular viewing remained substantial (roughly a factor of 10). To determine whether motion parallax would improve monocular sensitivity for the textured setting, we asked observers to move their heads laterally, so that the viewing eye was displaced by 8-10 cm; this motion produced little improvement in the monocular thresholds. We also compared disparity thresholds measured with the real rods to thresholds measured with virtual images in a standard mirror stereoscope. Surprisingly, for the two naive observers, the stereoscope thresholds were far worse than the thresholds for the real rods-a finding that indicates that stereoscope measurements for unpracticed observers should be treated with caution. With practice, the stereoscope thresholds for one observer improved to almost the precision of the thresholds for the real rods.

Stereo-vision: head-centric coding of retinal signals

Stereo-vision is generally considered to provide information about depth in a visual scene derived from disparities in the positions of an image on the two eyes; a new study has found evidence that retinal-image coding relative to the head is also important.

Inconsistency of perceived 3D shape

Internal consistency of local depth, slant, and curvature judgments was studied by asking participants to match two 3D surfaces rendered by different mixtures of 3D cues (velocity, texture, and shading). We found that perceptual judgments were not consistent with each other, with cue-specific distortions. Adding multiple cues did not eliminate the inconsistencies of the judgments. These results can be predicted by the Intrinsic Constraint (IC) model according to which the perceptual metric local estimates are a monotonically increasing function of the Signal-to-Noise Ratio of the optimal combination of direct information of 3D shape (Domini, Caudek, & Tassinari, 2006).

Vertical binocular disparity is encoded implicitly within a model neuronal population tuned to horizontal disparity and orientation

Primary visual cortex is often viewed as a "cyclopean retina", performing the initial encoding of binocular disparities between left and right images. Because the eyes are set apart horizontally in the head, binocular disparities are predominantly horizontal. Yet, especially in the visual periphery, a range of non-zero vertical disparities do occur and can influence perception. It has therefore been assumed that primary visual cortex must contain neurons tuned to a range of vertical disparities. Here, I show that this is not necessarily the case. Many disparity-selective neurons are most sensitive to changes in disparity orthogonal to their preferred orientation. That is, the disparity tuning surfaces, mapping their response to different two-dimensional (2D) disparities, are elongated along the cell's preferred orientation. Because of this, even if a neuron's optimal 2D disparity has zero vertical component, the neuron will still respond best to a non-zero vertical disparity when probed with a sub-optimal horizontal disparity. This property can be used to decode 2D disparity, even allowing for realistic levels of neuronal noise. Even if all V1 neurons at a particular retinotopic location are tuned to the expected vertical disparity there (for example, zero at the fovea), the brain could still decode the magnitude and sign of departures from that expected value. This provides an intriguing counter-example to the common wisdom that, in order for a neuronal population to encode a quantity, its members must be tuned to a range of values of that quantity. It demonstrates that populations of disparity-selective neurons encode much richer information than previously appreciated. It suggests a possible strategy for the brain to extract rarely-occurring stimulus values, while concentrating neuronal resources on the most commonly-occurring situations.

Latitude and longitude vertical disparities

The literature on vertical disparity is complicated by the fact that several different definitions of the term "vertical disparity" are in common use, often without a clear statement about which is intended or a widespread appreciation of the properties of the different definitions. Here, we examine two definitions of retinal vertical disparity: elevation-latitude and elevation-longitude disparities. Near the fixation point, these definitions become equivalent, but in general, they have quite different dependences on object distance and binocular eye posture, which have not previously been spelt out. We present analytical approximations for each type of vertical disparity, valid for more general conditions than previous derivations in the literature: we do not restrict ourselves to objects near the fixation point or near the plane of regard, and we allow for non-zero torsion, cyclovergence, and vertical misalignments of the eyes. We use these expressions to derive estimates of the latitude and longitude vertical disparities expected at each point in the visual field, averaged over all natural viewing. Finally, we present analytical expressions showing how binocular eye position-gaze direction, convergence, torsion, cyclovergence, and vertical misalignment-can be derived from the vertical disparity field and its derivatives at the fovea.

Probabilistic combination of slant information: weighted averaging and robustness as optimal percepts

Depth perception involves combining multiple, possibly conflicting, sensory measurements to estimate the 3D structure of the viewed scene. Previous work has shown that the perceptual system combines measurements using a statistically optimal weighted average. However, the system should only combine measurements when they come from the same source. We asked whether the brain avoids combining measurements when they differ from one another: that is, whether the system is robust to outliers. To do this, we investigated how two slant cues-binocular disparity and texture gradients-influence perceived slant as a function of the size of the conflict between the cues. When the conflict was small, we observed weighted averaging. When the conflict was large, we observed robust behavior: perceived slant was dictated solely by one cue, the other being rejected. Interestingly, the rejected cue was either disparity or texture, and was not necessarily the more variable cue. We modeled the data in a probabilistic framework, and showed that weighted averaging and robustness are predicted if the underlying likelihoods have heavier tails than Gaussians. We also asked whether observers had conscious access to the single-cue estimates when they exhibited robustness and found they did not, i.e. they completely fused despite the robust percepts.

3D surface orientation based on a novel representation of the orientation disparity field

The orientation disparity field from two orthographic views of an inclined planar surface patch (covered by straight lines) is analyzed, and a new tool to extract the patch orientation is provided: the function coupling the average orientation of each pair of corresponding surface contours with their orientation disparity. This function allows identifying the tilt of the surface, and two indeterminacy functions describing the set of surface inclinations (around the vertical and horizontal axes) over convergence angle values compatible with the orientation disparity field. Results of simulations show that the selection of inclination values matching the difference between the areas below the indeterminacy functions are consistent with some surface orientation effects found in psychophysical and computational experiments, like: the unbiased tilt vs. biased slant estimates, the slant underestimation, the surface orientation anisotropy, and the slant/tilt covariation.

Cyclopean geometry of binocular vision

The geometry of binocular projection is analyzed in relation to the primate visual system. An oculomotor parameterization that includes the classical vergence and version angles is defined. It is shown that the epipolar geometry of the system is constrained by binocular coordination of the eyes. A local model of the scene is adopted in which depth is measured relative to a plane containing the fixation point. These constructions lead to an explicit parameterization of the binocular disparity field involving the gaze angles as well as the scene structure. The representation of visual direction and depth is discussed with reference to the relevant psychophysical and neurophysiological literature.

The intrinsic constraint model for stereo-motion integration

How the visual system integrates the information provided by several depth cues is central for vision research. Here, we present a model for how the human visual system combines disparity and velocity information. The model provides a depth interpretation to a subspace defined by the covariation of the two signals. We show that human performance is consistent with the predictions of the model, and compare them with those of another theoretical approach, the modified weak-fusion model. We discuss the validity of each approach as a model for human perception of 3-D shape from multiple cues to depth.

Vergence-accommodation conflicts hinder visual performance and cause visual fatigue

Three-dimensional (3D) displays have become important for many applications including vision research, operation of remote devices, medical imaging, surgical training, scientific visualization, virtual prototyping, and more. In many of these applications, it is important for the graphic image to create a faithful impression of the 3D structure of the portrayed object or scene. Unfortunately, 3D displays often yield distortions in perceived 3D structure compared with the percepts of the real scenes the displays depict. A likely cause of such distortions is the fact that computer displays present images on one surface. Thus, focus cues-accommodation and blur in the retinal image-specify the depth of the display rather than the depths in the depicted scene. Additionally, the uncoupling of vergence and accommodation required by 3D displays frequently reduces one's ability to fuse the binocular stimulus and causes discomfort and fatigue for the viewer. We have developed a novel 3D display that presents focus cues that are correct or nearly correct for the depicted scene. We used this display to evaluate the influence of focus cues on perceptual distortions, fusion failures, and fatigue. We show that when focus cues are correct or nearly correct, (1) the time required to identify a stereoscopic stimulus is reduced, (2) stereoacuity in a time-limited task is increased, (3) distortions in perceived depth are reduced, and (4) viewer fatigue and discomfort are reduced. We discuss the implications of this work for vision research and the design and use of displays.

Misperceptions in Stereoscopic Displays: A Vision Science Perspective

3d shape and scene layout are often misperceived when viewing stereoscopic displays. For example, viewing from the wrong distance alters an object's perceived size and shape. It is crucial to understand the causes of such misperceptions so one can determine the best approaches for minimizing them. The standard model of misperception is geometric. The retinal images are calculated by projecting from the stereo images to the viewer's eyes. Rays are back-projected from corresponding retinal-image points into space and the ray intersections are determined. The intersections yield the coordinates of the predicted percept. We develop the mathematics of this model. In many cases its predictions are close to what viewers perceive. There are three important cases, however, in which the model fails: 1) when the viewer's head is rotated about a vertical axis relative to the stereo display (yaw rotation); 2) when the head is rotated about a forward axis (roll rotation); 3) when there is a mismatch between the camera convergence and the way in which the stereo images are displayed. In these cases, most rays from corresponding retinal-image points do not intersect, so the standard model cannot provide an estimate for the 3d percept. Nonetheless, viewers in these situations have coherent 3d percepts, so the visual system must use another method to estimate 3d structure. We show that the non-intersecting rays generate vertical disparities in the retinal images that do not arise otherwise. Findings in vision science show that such disparities are crucial signals in the visual system's interpretation of stereo images. We show that a model that incorporates vertical disparities predicts the percepts associated with improper viewing of stereoscopic displays. Improving the model of misperceptions will aid the design and presentation of 3d displays.

Direct extraction of curvature-based metric shape from stereo by view-modulated receptive fields

Any computation of metric surface structure from horizontal disparities depends on the viewing geometry, and analysing this dependence allows us to narrow down the choice of viable schemes. For example, all depth-based or slant-based schemes (i.e. nearly all existing models) are found to be unrealistically sensitive to natural errors in vergence. Curvature-based schemes avoid these problems and require only moderate, more robust view-dependent corrections to yield local object shape, without any depth coding. This fits the fact that humans are strikingly insensitive to global depth but accurate in discriminating surface curvature. The latter also excludes coding only affine structure. In view of new adaptation results, our goal becomes to directly extract retinotopic fields of metric surface curvatures (i.e. avoiding intermediate disparity curvature). To find a robust neural realisation, we combine new exact analysis with basic neural and psychophysical constraints. Systematic, step-by-step 'design' leads to neural operators which employ a novel family of 'dynamic' receptive fields (RFs), tuned to specific (bi-)local disparity structure. The required RF family is dictated by the non-Euclidean geometry that we identify as inherent in cyclopean vision. The dynamic RF-subfield patterns are controlled via gain modulation by binocular vergence and version, and parameterised by a cell-specific tuning to slant. Our full characterisation of the neural operators invites a range of new neurophysiological tests. Regarding shape perception, the model inverts widely accepted interpretations: It predicts the various types of errors that have often been mistaken for evidence against metric shape extraction.

Neural bases of stereopsis across visual field of the alert macaque monkey

Left and right retinal images of an object seen by the 2 eyes can occupy slightly disparate horizontal and/or vertical locations. The role of horizontal disparity (HD) in stereoscopic vision is well established, but the functional contribution of vertical disparity (VD) remains unclear. Various psychophysical studies have shown that HD and VD are used differently by the visual system depending on their location in the visual field, whether near the center of gaze or more peripheral. We show this horizontal/vertical distinction at the cellular level in monkey primary visual cortex (area V1). The range of VD encoding is reduced in central but not in the peripheral representation of the visual field. Moreover, neurons respond selectively to particular combinations of both types of disparities depending on the coded orientation as predicted by the disparity energy model. The preferred orientations of neurons near the fovea present a vertical bias that is well suited for stereopsis based on HD selectivity alone. In the periphery, instead, preferred orientations are radially biased, which allows a peripheral detector to convey the same depth signal based on either HD or VD. Such an organization has functional implications in both the perceptual and oculomotor domains.

The visual perception of plane tilt from motion in small field and large field: psychophysics and theory

Subjects indicated the tilt of dotted planes rotating in depth, in monocular viewing, under perspective projection. The responses depended on the FOV (field of view) and on the angle W between the tilt and frontal translation (orthogonal to the rotation axis). Response accuracy increased with the FOV, and decreased with W. Our results support the processing of the second-order optic flow in all cases, but indicate that this flow is quantitatively small in small-field, leading to tilt ambiguities. We examine computational models based on the affine components of the optic flow to interpret our results.

Stereo and motion information are not independently processed by the visual system

Many visual tasks are carried out by using multiple sources of sensory information to estimate environmental properties. In this paper, we present a model for how the visual system combines disparity and velocity information. We propose that, in a first stage of processing, the best possible estimate of the affine structure is obtained by computing a composite score from the disparity and velocity signals. In a second stage, a maximum likelihood Euclidean interpretation is assigned to the recovered affine structure. In two experiments, we show that human performance is consistent with the predictions of our model. The present results are also discussed in the framework of another theoretical approach of the depth cue combination process termed Modified Weak Fusion.

Depth aftereffects mediated by vertical disparities: Evidence for vertical disparity driven calibration of extraretinal signals during stereopsis

Perceptual adaptation often results in a repulsive aftereffect: stimuli are seen as biased away from the adaptation stimulus (). Here we report the absence of a repulsive aftereffect for a vertical gradient of vertical disparity (or vertical size ratio, VSR). We exposed observers to a binocular stimulus consisting of horizontal lines. This stimulus contains vertical, but not horizontal disparities. The visual system was able to measure the VSR of this stimulus: although the lines themselves always appeared unslanted, the VSR carried by the lines had a dramatic effect on the apparent slant of a horizontal row of dots, as predicted by recent accounts of Ogle's (1938) induced effect (e.g., Backus, Banks, van Ee, & Crowell, 1999). Yet we observed no repulsive aftereffect for the VSR signal: after adaptation to horizontal lines that were vertically larger in one eye, we found an attractive aftereffect, the magnitude of which was largest in stimuli that did not contain a VSR signal. We interpret these results as a case of recalibration: disagreement between extra-retinal eye position signals (EP) and VSR causes a recalibration in the use of EP as used in the stereoscopic perception of slant.

Disparity-defined objects moving in depth do not elicit three-dimensional shape constancy

Observers generally fail to recover three-dimensional shape accurately from binocular disparity. Typically, depth is overestimated at near distances and underestimated at far distances [Johnston, E. B. (1991). Systematic distortions of shape from stereopsis. Vision Research, 31, 1351-1360]. A simple prediction from this is that disparity-defined objects should appear to expand in depth when moving towards the observer, and compress in depth when moving away. However, additional information is provided when an object moves from which 3D Euclidean shape can be recovered, be this through the addition of structure from motion information [Richards, W. (1985). Structure from stereo and motion. Journal of the Optical Society of America A, 2, 343-349], or the use of non-generic strategies [Todd, J. T., & Norman, J. F. (2003). The visual perception of 3-D shape from multiple cues: Are observers capable of perceiving metric structure? Perception and Psychophysics, 65, 31-47]. Here, we investigated shape constancy for objects moving in depth. We found that to be perceived as constant in shape, objects needed to contract in depth when moving toward the observer, and expand in depth when moving away, countering the effects of incorrect distance scaling (Johnston, 1991). This is a striking example of the failure of shape constancy, but one that is predicted if observers neither accurately estimate object distance in order to recover Euclidean shape, nor are able to base their responses on a simpler processing strategy.

Focus cues affect perceived depth

Depth information from focus cues--accommodation and the gradient of retinal blur--is typically incorrect in three-dimensional (3-D) displays because the light comes from a planar display surface. If the visual system incorporates information from focus cues into its calculation of 3-D scene parameters, this could cause distortions in perceived depth even when the 2-D retinal images are geometrically correct. In Experiment 1 we measured the direct contribution of focus cues to perceived slant by varying independently the physical slant of the display surface and the slant of a simulated surface specified by binocular disparity (binocular viewing) or perspective/texture (monocular viewing). In the binocular condition, slant estimates were unaffected by display slant. In the monocular condition, display slant had a systematic effect on slant estimates. Estimates were consistent with a weighted average of slant from focus cues and slant from disparity/texture, where the cue weights are determined by the reliability of each cue. In Experiment 2, we examined whether focus cues also have an indirect effect on perceived slant via the distance estimate used in disparity scaling. We varied independently the simulated distance and the focal distance to a disparity-defined 3-D stimulus. Perceived slant was systematically affected by changes in focal distance. Accordingly, depth constancy (with respect to simulated distance) was significantly reduced when focal distance was held constant compared to when it varied appropriately with the simulated distance to the stimulus. The results of both experiments show that focus cues can contribute to estimates of 3-D scene parameters. Inappropriate focus cues in typical 3-D displays may therefore contribute to distortions in perceived space.

Vertical-disparity gradients are processed independently in different depth planes

We examined the effects of vertical-disparity gradients on apparent depth curvature of textured surfaces. In Experiment 1, vertical disparities induced expected curvatures when the surface had a horizontal disparity of < +/-40.34'. A central row of elements, lacking vertical disparities, ceased to have the same apparent curvature as the surface when the horizontal disparity between row and surface exceeded +/-5'. In Experiment 2, vertical disparities were not pooled between superimposed surfaces separated by horizontal disparities > +/-10'. Thus, vertical-disparity gradients are not pooled over depth for curvature perception. Our results suggest that vertical disparities are used to determine distances to surfaces directly, rather than to estimate vergence.

Localization of the plane of regard in space

When we fixate an object in space, the rotation centers of the eyes, together with the object, define a plane of regard. People perceive the elevation of objects relative to this plane accurately, irrespective of eye or head orientation (Poljac et al. (2004) Vision Res, in press). Yet, to create a correct representation of objects in space, the orientation of the plane of regard in space is required. Subjects pointed along an eccentric vertical line on a touch screen to the location where their plane of regard intersected the touch screen positioned on their right. The distance of the vertical line to the subject's eyes varied from 10 to 40 cm. Subjects were sitting upright and fixating one of the nine randomly presented directions ranging from 20 degrees left and down to 20 degrees right and up relative to their straight ahead. The eccentricity of fixations relative to the pointing location varied by up to 40 degrees . Subjects underestimated the elevation of their plane of regard (on average by 3.69 cm, SD=1.44 cm), regardless of the fixation direction or pointing distance. However, when the targets were shown on a display mounted in a table, to provide support of the subject's hand throughout the trial, subjects pointed accurately (average error 0.3 cm, SD=0.8 cm). In addition, head tilt 20 degrees to the left or right did not cause any change in accuracy. The bias observed in the first task could be caused by maintained tonus in arm muscles when the arm is raised, that might interfere with the transformation from visual to motor signals needed to perform the pointing movement. We conclude that the plane of regard is correctly localized in space. This may be a good starting point for representing objects in head-centric coordinates.

Evidence for implication of primate area V1 in neural 3-D spatial localization processing

We investigated the neural mechanisms underlying visual localization in 3-D space in area V1 of behaving monkeys. Three different sources of information, retinal disparity, viewing distance and gaze direction, that participate in these neural mechanisms are being reviewed. The way they interact with each other is studied by combining retinal and extraretinal signals. Interactions between retinal disparity and viewing distance have been shown in foveal V1; we have observed a strong modulation of the spontaneous activity and of the visual response of most V1 cells that was highly correlated with the vergence angle. As a consequence of these gain effects, neural horizontal disparity coding is favoured or refined for particular distances of fixation. Changing the gaze direction in the fronto-parallel plane also produces strong gains in the visual response of half of the cells in foveal V1. Cells tested for horizontal disparity and orientation selectivities show gain effects that occur coherently for the same spatial coordinates of the eyes. Shifts in preferred disparity also occurred in several neurons. Cells tested in calcarine V1 at retinal eccentricities larger than 10 degrees , show that horizontal disparity is encoded at least up to 20 degrees around both the horizontal and vertical meridians. At these large retinal eccentricities we found that vertical disparity is also encoded with tuning profiles similar to those of horizontal disparity coding. Combinations of horizontal and vertical disparity signals show that most cells encode both properties. In fact the expression of horizontal disparity coding depends on the vertical disparity signals that produce strong gain effects and frequent changes in peak selectivities. We conclude that the vertical disparity signal and the eye position signal serve to disambiguate the horizontal disparity signal to provide information on 3-D spatial coordinates in terms of distance, gaze direction and retinal eccentricity. We suggest that the relative weight among these different signals is the determining factor involved in the neural processing that gives information on 3-D spatial localization.

A physiological theory of depth perception from vertical disparity

It has been known since the time of Helmholtz that vertical differences between the two retinal images can generate depth perception. Although many ecologically and geometrically inspired theories have been proposed, the neural mechanisms underlying the phenomenon remain elusive. Here we propose a new theory for depth perception from vertical disparity based on the oriented binocular receptive fields of visual cortical cells and on the radial bias of the preferred-orientation distribution in the cortex. The theory suggests that oriented cells may treat a vertical disparity as a weaker, equivalent horizontal disparity. It explains the induced effect, and the quadrant and size dependence of vertical disparity. It predicts that horizontal and vertical disparities should locally enhance or cancel each other according to their depth signs, and that the effect of vertical disparity should be orientation dependent. These predictions were confirmed through psychophysical experiments.

Adaptation to vertical disparity induced-depth: implications for disparity processing

Depth aftereffects produced by prolonged inspection of an object in depth can be mediated by monocular and binocular depth cues. The adapting mechanisms responsible for such effects have not yet been fully determined. Theories of binocular depth aftereffects typically posit a role of an adaptive horizontal disparity sensitive mechanism, implying multiple cue-specific mechanisms for depth aftereffects. Here we examined whether binocular depth aftereffects can be attributed to such a cue-specific mechanism. In Experiment 1 we did so using a technique allowing us to maintain horizontal disparities and vergence constant for our adaptation stimuli, whilst manipulating simulated depth by virtue of a vertical disparity induced-depth effect. We found that depth aftereffects were almost identical to those produced by adaptation to stimuli of equivalent depth produced by conventional horizontal disparity modulations. In Experiment 2, we examined depth aftereffects following adaptation to apparently frontal surfaces produced by different combinations of horizontal and vertical disparity modulations. Aftereffects were close to zero. These results suggest that binocular depth aftereffects are not due to adaptation of a horizontal disparity sensitive mechanism, and we argue that adaptation occurs at the level of a 3D shape sensitive mechanism derived from multiple cues. Experiment 3 was a control to examine whether the two types of adaptation stimuli in Experiment 1 were indeed perceptually the same, since in theory they may differ if vertical disparities influenced metric depth scaling. We found no evidence of this, and concluded that the two classes of stimuli used in Experiment 1, though consisting of very different patterns of disparity, were perceptually equivalent.

Neurons in parafoveal areas V1 and V2 encode vertical and horizontal disparities

Stereoscopic vision mainly relies on binocular horizontal disparity (HD), and its cortical encoding is well established in the foveal representation of the visual field. The role of vertical disparity (VD) is more controversial. Thus far, in the monkey, very few studies have investigated the HD sensitivity beyond 5 degrees of retinal eccentricity and no evidence of a real encoding of VD exists in the parafoveal representation of areas V1 and V2. Using dynamic random dot stereograms, we have tested both HD and VD selectivities in the parafoveal representation of V1 (calcarine V1) and V2 (eccentricities > 10 degrees ) in a behaving monkey. HD and VD selectivities have been characterized using fitting with Gabor function. A large proportion of the tested cells were both HD and VD selective (47%) and, to a lesser extent, HD selective only (8%) or VD selective only (23%). We found a real encoding of VD, with the same diversity in the tuning profiles as described for HD, that cannot be assimilated to a simple perturbation of the HD matching process. Moreover, the VD encoding had a finer scale than the HD one, which is coherent with the smaller range of naturally occurring VD. For the HD encoding, both the percentage of selective cells and the tuning parameters were close to those reported in foveal V1. These results show that, at parafoveal eccentricities in V1 and V2, disparity detectors are tuned to both horizontal and vertical dimensions of the positional disparity existing between matched features in both retinas.