The computation of binocular edges

An Alternative Theory of Binocularity

The fact that seeing with two eyes is universal among vertebrates raises a problem that has long challenged vision scientists: how do animals with overlapping visual fields combine non-identical right and left eye images to achieve fusion and the perception of depth that follows? Most theories address this problem in terms of matching corresponding images on the right and left retinas. Here we suggest an alternative theory of binocular vision based on anatomical correspondence that circumvents the correspondence problem and provides a rationale for ocular dominance.

Location, location, location: development of spatiotemporal sequence learning in infancy

We investigated infants' sensitivity to spatiotemporal structure. In Experiment 1, circles appeared in a statistically defined spatial pattern. At test 11-month-olds, but not 8-month-olds, looked longer at a novel spatial sequence. Experiment 2 presented different color/shape stimuli, but only the location sequence was violated during test; 8-month-olds preferred the novel spatial structure, but 5-month-olds did not. In Experiment 3, the locations but not color/shape pairings were constant at test; 5-month-olds showed a novelty preference. Experiment 4 examined "online learning": We recorded eye movements of 8-month-olds watching a spatiotemporal sequence. Saccade latencies to predictable locations decreased. We argue that temporal order statistics involving informative spatial relations become available to infants during the first year after birth, assisted by multiple cues.

Detecting disparity in two-dimensional patterns

One can measure the disparities between two retinal images in several different ways. Experiments were conducted to identify the measure that is invariant at the threshold for detecting the disparity of two-dimensional patterns. The patterns used were stereo plaids, which permit a partial dissociation between the disparity of the pattern and the disparities of its one-dimensional components. For plaids with near-horizontal disparities, thresholds are limited by a disparity phase shift equal to the threshold phase shift for single gratings. For non-horizontal disparities, thresholds are elevated, yet are still phase-limited. In no disparity direction are thresholds for detecting disparity determined by the spatial extent of the plaids' disparity. Effects of the number and the orientation of components with task-relevant disparities indicate that plaid thresholds are limited by the disparity of the plaid's one-dimensional components. No evidence was found that these components form any higher-order pattern that can be used in detecting disparity. Oblique and near-vertical disparities generate elevated thresholds at a stage beyond component disparity detection. This second stage combines component disparities, which are ambiguous about depth, into pattern disparities capable of supporting veridical depth perception.

Macaque inferior temporal neurons are selective for three-dimensional boundaries and surfaces

The lower bank of the superior temporal sulcus (TEs), part of the inferior temporal cortex, contains neurons selective for disparity-defined three-dimensional (3-D) shape. The large majority of these TEs neurons respond to the spatial variation of disparity, i.e., are higher-order disparity selective. To determine whether curved boundaries or curved surfaces by themselves are sufficient to elicit 3-D shape selectivity, we recorded the responses of single higher-order disparity-selective TEs neurons to concave and convex 3-D shapes in which the disparity varied either along the boundary of the shape, or only along its surface. For a majority of neurons, a 3-D boundary was sufficient for 3-D shape selectivity. At least as many neurons responded selectively to 3-D surfaces, and a number of neurons exhibited both surface and boundary selectivity. The second aim of this study was to determine whether TEs neurons can represent differences in second-order disparities along the horizontal axis. The results revealed that TEs neurons can also be selective for horizontal 3-D shapes and can code the direction of curvature (vertical or horizontal). Thus, TEs neurons represent both boundaries and surfaces curved in depth and can signal the direction of curvature along a surface. These results show that TEs neurons use not only boundary but also surface information to encode 3-D shape properties.

Luminance spatial scale facilitates stereoscopic depth segmentation

Are differences in luminance spatial frequency between surfaces that overlap in depth useful for surface segmentation? We examined this question, using a novel stimulus termed a dual-surface disparity grating. The dual-surface grating was made from Gabor micropatterns and consisted of two superimposed sinusoidal disparity gratings of identical disparity-modulation spatial frequency and orientation but of opposite spatial phase. Corrugation amplitude thresholds for discrimination of the orientation of the dual-surface grating were obtained as a function of the difference in Gabor (luminance) spatial frequency between the two surfaces. When the Gabor micropatterns on the two surfaces were identical in spatial frequency, thresholds were very high and in some instances impossible to obtain. However, with as little as a 1-octave difference in spatial frequency between the surfaces, thresholds fell sharply to near-asymptotic levels. The fall in thresholds paralleled a change in the appearance of the stimulus from one of irregular depth to stereo transparency. The most parsimonious explanation for this finding is that the introduction of a between-surface luminance spatial-frequency difference reduces the number of spurious cross-surface binocular matches, thus helping to reveal the three-dimensional structure of the stimulus.

On the apparent collapse of stereopsis in random-dot-stereograms at isoluminance

We have investigated the apparent collapse of stereopsis obtained with random-dot-stereograms at isoluminance. Contrast thresholds for both depth and form discrimination of targets in random-dot- and figural stereograms were measured at a number of disparitics, using both isoluminant and isochromatic stimuli. All contrast thresholds for stereoscopic tasks were normalised to contrast thresholds for detecting the appropriate stimulus. We found that at isoluminance contrast thresholds for depth judgements were not higher for random-dot compared to figural stereograms, even when normalised to the same thresholds obtained with isochromatic stimuli. On the other hand contrast thresholds for three-dimensional form judgements were much higher than those for depth judgements in isoluminant, compared to isochromatic random-dot-stereograms. This specific impairment of stereoscopic form (as opposed to depth) processing at isoluminance was confirmed in a further experiment in which subjects were required to judge the presence and orientation of depth corrugations in a disparity-modulated random-dot-stereogram.

Fine-to-coarse scale disambiguation in stereopsis

Spatial frequency selectivity has been incorporated into various theories of stereo matching, along with spatial scale interactions operating from coarse-to-fine spatial scales. We concentrate here on the role of fine scale information in the stereo matching process and show that fine scale information is capable of disambiguating matches made at coarser scales. An ambiguous coarse scale stimulus was created by presenting a low frequency (2 c/deg) sine wave in anti-phase to the two eyes, whose endpoints betrayed no information about which way the sine waves should be matched. It could be seen with crossed or uncrossed disparity equally validly and at chance from trial to trial. To this was added a fine scale (8 c/deg) filtered random dot stimulus specifying unambiguously a certain disparity. Observers judged the apparent depth of the two stimuli as the disparity of the fine scale stimulus was varied. The sine wave was usually perceived to have the same sign disparity as the fine scale stimulus. Depth matching with the two superimposed stimuli confirmed that the coarse scale stimulus was actually disambiguated, and seen with disparities equal to half its spatial period. The results suggest the operation of a cross-spatial scale matching disambiguation process, which can operate in a fine-to-coarse fashion.

Size-disparity correlation in stereopsis at contrast threshold

Contrast thresholds for 75% correct depth identification in narrow-band filtered random dot stereograms were determined for different center spatial frequencies and binocular disparities. Rigorous control over vergence was maintained during testing, and a forced-choice procedure was used. The resulting contrast sensitivity function for stereopsis revealed sensitivity over a greater range of disparities at low than at high spatial frequencies. Sensitivity peaked for large disparities at low spatial frequencies and for small disparities at high spatial frequencies. When disparities were converted to effective binocular phase differences, the variation of contrast sensitivity with phase followed a consistent pattern across spatial frequencies, with peak sensitivity occurring mainly for binocular phases of between 90 degrees and 180 degrees. These results have implications for the extent of spatial integration at the input to the disparity sensing mechanism. A model postulating a spread of positional disparities independent of the spatial frequency selectivity of disparity-sensitive units cannot account for the results. But the size-disparity correlation strongly evident in our data is predicted by certain models of stereopsis, such as phase disparity encoding. An ideal observer analysis is developed that demonstrates that our results were not forced by the nature of the stimulus employed; rather, the quantum efficiency for stereopsis at contrast threshold follows the size-disparity correlation.

Neural models of stereoscopic vision

Human stereopsis remains an enigma: how does the brain match features between the left and right eye images and compute disparity between these matched features? Developments in computational neuroscience and machine vision have led to several models of human stereopsis that provide insight into possible mechanisms underlying this phenomenon. These models, reviewed in this paper, adopt one of three general strategies. One class of models employs cooperative interactions, whereby a unique solution to the matching problem emerges from excitatory and inhibitory interactions among binocular neural elements. A second class of models implements matching and disparity computation serially over multiple spatial scales. A third class relies on local, non-interacting computations performed in parallel to overcome speed limitations inherent in the other models. Considered together, these theoretical developments offer fresh insights concerning the actual neural concomitants of binocular stereopsis.

Structure from stereo by associative learning of the constraints

A computational model of structure from stereo that develops smoothness constraints naturally by associative learning of a large number of example mappings from disparity data to surface depth data is proposed. Banks of disparity-selective graded response units at all spatial locations in the visual field were the input data. These cells responded to matches of luminance change at convergent, divergent, or zero offsets in the left and right 'retina' samples. Surfaces were created by means of a pseudo-Markov process. From these surfaces, shaded marked and ummarked surfaces were created, along with random-dot versions of the same surfaces. Learning of these example shaded and shaded marked surfaces allowed the system to solve stereo mappings both for the surfaces it had learned and for surfaces it had not learned but which had been created by the same pseudo-Markov process. Further, the model was able to solve some random-dot versions of the surfaces when the surfaces had been learned as shaded marked surfaces.

Integration of stereo and texture cues in the formation of discontinuities during three-dimensional surface interpolation

A series of stereograms are presented which demonstrate that texture boundaries can strongly influence the perception of discontinuities between neighbouring three-dimensional (3-D) surfaces portrayed by means of stereo cues. In these demonstration figures, no stereo information is available in the immediate vicinity of the boundary between the two 3-D stereo surfaces because all texture in that region is removed in one eye's view. On the other hand, various forms of texture boundary information are provided in the resulting monocular region. This stimulus paradigm is used to explore the question: what influence does texture boundary information have on the nature of the perceived 3-D surface that is interpolated between two stimulus regions which carry stereo cues? It is shown that if a clear-cut texture boundary is present in the monocular region then this is used by the human visual system to fix the perceived location of 3-D crease and step surface discontinuities between the stereo regions. Collett (1985) explored this issue with a similar methodology and reported weak and unreliable assistance from monocular texture boundaries in helping shape 3-D stereo surface discontinuities. The strong and robust phenomena demonstrated here seem to rely on two main differences between the present stimuli and those of Collett. In the present stimuli, figurally continuous textures containing strong texture boundaries are used, together with a technique for minimising the complications, including binocular rivalry, that arise from the borders of the stimulus regions present in only one half of each stereogram.

Is edge information for stereoacuity spatially channeled?

Models of stereopsis generally assume that binocular correspondence is achieved through alignment of luminance edges in the two eyes. Yet the stimulus properties which constitute edge information for stereopsis have not been defined. Three experiments explored the nature of these stimulus properties. The first two experiments tested whether local luminance gradient and the relative phase of spatial components supply information about the position of edges which influences stereosensitivity. In Expt 1, stereothresholds were reduced with increased spatial frequency or contrast of sinusoidal luminance gratings, but no simple relationship between target luminance gradient and stereosensitivity was found. In Expt 2, stereothresholds were equivalent for targets having identical spatial frequency components, but differing in maximum luminance gradient and the relative spatial phase of their components. In addition, stereothresholds were lower for the target having the higher contrast in pairs of unequal-contrast targets having equal maximum luminance gradients. These results suggest that the properties of luminance gradient and relative spatial phase do not influence stereosensitivity independently of spatial frequency and contrast. Experiment 3 directly tested whether stereosensitivity depends on edge information whose disparity is detected independently at different spatial scales. Stereothresholds for IF + 5F compound targets were found to be equivalent to thresholds obtained separately with the more sensitive of the two components. Taken together with a compressive nonlinearity in the relationship between contrast and stereothreshold obtained by others (Halpern and Blake, 1989; Legge and Gu, 1989) and replicated in Expt 1, the results of Expt 3 indicate that, whatever the exact nature of the luminance discontinuity information utilized in disparity detection, it is processed independently at different spatial scales.

The role of monocular regions in stereoscopic displays

Random-dot stereograms of an object standing out from a background always contain a monocular region at the side of the foreground object. This is equivalent to the monocularly occluded part of the background in the real-life viewing of one object in front of another. The role of these monocular regions in the stereoscopic process has not been investigated previously, although it is generally assumed that they are a source of difficulty in stereoscopic resolution because of the unmatchable texture within them. The basis of the present study was a prediction that the presence of texture within these regions would facilitate rather than retard stereoscopic processing. This prediction follows from a hypothesis that stereoscopic processing is initially located at disparity discontinuities. Unmatched regions are only found at such discontinuities, and could serve to locate them.

PMF: a stereo correspondence algorithm using a disparity gradient limit

The advantages of solving the stereo correspondence problem by imposing a limit on the magnitude of allowable disparity gradients are examined. It is shown how the imposition of such a limit can provide a suitable balance between the twin requirements of disambiguating power and the ability to deal with a wide range of surfaces. Next, the design of a very simple stereo algorithm called PMF is described. In conjunction with certain other constraints used in many other stereo algorithms, PMF employs a limit on allowable disparity gradients of 1, a value that coincides with that reported for human stereoscopic vision. The excellent performance of PMF is illustrated on a series of natural and artificial stereograms. Finally, the differences between the theoretical justification for the use of disparity gradients for solving the stereo correspondence problems presented in the paper and others that exist in the stereo algorithm literature are discussed.

Does segregation of differently moving areas depend on relative or absolute displacement?

We have examined the occurrence of segregation in random dot kinematograms in which a central patch of dots, and the surrounding area, were each coherently displaced, either in the same or opposite directions (Fig. 1), by varying amounts. The limiting displacement for segregation to occur is determined primarily by the displacement of each region alone, rather than the relative displacement of neighbouring regions (Fig 2). We conclude that the "correspondence problem" is solved by means of a short range motion detection process acting on each region separately; segregation is achieved by comparing the results of this process for adjacent regions.

Surfaces with steep variations in depth pose difficulties for orientationally tuned disparity filters

Surfaces possessing steep variations in depth present severe difficulties for orientationally tuned filter models of stereopsis. These difficulties are discussed in connection with a random-dot stereogram depicting a surface with steep horizontal corrugations. As expected on theoretical grounds, we find that a vertical +/- 45 degrees orientationally filtered version of this stereogram cannot be fused. Moreover, it is demonstrated that a horizontal +/- 45 degrees filtered version can be fused only with difficulty and its stereo percept is poor compared to that of the unfiltered original. It is concluded that orientated filters seem ill-designed to mediate the extraction of disparity cues, at least in the cases under consideration.