Integration of stereo and texture cues in the formation of discontinuities during three-dimensional surface interpolation

Monocular and binocular edges enhance the perception of stereoscopic slant

Gradients of absolute binocular disparity across a slanted surface are often considered the basis for stereoscopic slant perception. However, perceived stereo slant around a vertical axis is usually slow and significantly under-estimated for isolated surfaces. Perceived slant is enhanced when surrounding surfaces provide a relative disparity gradient or depth step at the edges of the slanted surface, and also in the presence of monocular occlusion regions (sidebands). Here we investigate how different kinds of depth information at surface edges enhance stereo slant about a vertical axis. In Experiment 1, perceived slant decreased with increasing surface width, suggesting that the relative disparity between the left and right edges was used to judge slant. Adding monocular sidebands increased perceived slant for all surface widths. In Experiment 2, observers matched the slant of surfaces that were isolated or had a context of either monocular or binocular sidebands in the frontal plane. Both types of sidebands significantly increased perceived slant, but the effect was greater with binocular sidebands. These results were replicated in a second paradigm in which observers matched the depth of two probe dots positioned in front of slanted surfaces (Experiment 3). A large bias occurred for the surface without sidebands, yet this bias was reduced when monocular sidebands were present, and was nearly eliminated with binocular sidebands. Our results provide evidence for the importance of edges in stereo slant perception, and show that depth from monocular occlusion geometry and binocular disparity may interact to resolve complex 3D scenes.

Depth propagation and surface construction in 3-D vision

In stereo vision, regions with ambiguous or unspecified disparity can acquire perceived depth from unambiguous regions. This has been called stereo capture, depth interpolation or surface completion. We studied some striking induced depth effects suggesting that depth interpolation and surface completion are distinct stages of visual processing. An inducing texture (2-D Gaussian noise) had sinusoidal modulation of disparity, creating a smooth horizontal corrugation. The central region of this surface was replaced by various test patterns whose perceived corrugation was measured. When the test image was horizontal 1-D noise, shown to one eye or to both eyes without disparity, it appeared corrugated in much the same way as the disparity-modulated (DM) flanking regions. But when the test image was 2-D noise, or vertical 1-D noise, little or no depth was induced. This suggests that horizontal orientation was a key factor. For a horizontal sine-wave luminance grating, strong depth was induced, but for a square-wave grating, depth was induced only when its edges were aligned with the peaks and troughs of the DM flanking surface. These and related results suggest that disparity (or local depth) propagates along horizontal 1-D features, and then a 3-D surface is constructed from the depth samples acquired. The shape of the constructed surface can be different from the inducer, and so surface construction appears to operate on the results of a more local depth propagation process.

Spatial and temporal properties of stereoscopic surface interpolation

It is well established that under a wide range of conditions when a sparse collection of texture elements varies smoothly in depth, the spaces between the elements are assigned depth values. This disparity interpolation process has been studied in an effort to define some of its fundamental spatial and temporal constraints. To assess disparity interpolation we employed two tasks: a novel task that relies on the bisection of illusory boundaries created when subjective stereoscopic surfaces intersect, and one that relies on a 3-D shape discrimination. The results of both experiments show that there is no improvement in performance when texture density is increased from near 0.20 to 0.85 or when exposure duration is increased from 50-100 to 1000 ms. This lack of dependence on the addition of features that define the interpolated surface, along with the abrupt decline in performance below a critical value, is consistent with the view that surface interpolation is an important function of human stereoscopic vision.

Binocular depth perception from unpaired image points need not depend on scene organization

Dichoptic stimuli containing unmatched features can produce depth perception despite the absence of binocular disparity, a phenomenon known as da Vinci stereopsis. Unmatched points can arise from depth discontinuities and partial occlusion in the real world. It has been hypothesized that spatial organization of unmatched image features as dictated by the ecological optics of occlusion might determine perceived depth in da Vinci stereopsis. We tested this hypothesis by creating dichoptic stimuli containing unmatched points in which local cues and overall organization could be dissociated. For these stimuli, observers' perception of depth did not depend on the organization of the scene, but only on the local cues. This finding shows the perceived depth of unpaired points need not depend on reconstructing the spatial organization of depth discontinuities in real-world scenes.

Peak localization of sparsely sampled luminance patterns is based on interpolated 3D surface representation

Objects in the world are typically defined by contours and local features separated by extended featureless regions. Sparsely sampled profiles were therefore used to evaluate the cues involved in localizing objects defined by such separated features (as opposed to typical Vernier acuity or other line-based localization tasks). Objects, in the form of Gaussian blobs, were defined at the sample positions by luminance cues, binocular disparity cues or both together. Remarkably, the luminance information in the sampled profiles was unable to support localization for objects requiring interpolation when the perceived depth from the luminance cue was cancelled by a disparity cue. Disparity cues, on the other hand, improved localization substantially over that for luminance cues alone. These data indicate that it is only through the interpolated depth representation that the position of the sampled object can be recognized. The dominance of a depth representation in the performance of such tasks shows that the depth information is not just an overlay to the 2D sketch of the positional information, but a core process that must be completed before the position of the object can be recognized.

Surface textures improve the robustness of stereoscopic depth cues

This research develops design recommendations for surface textures (patterns of color on object surfaces) rendered with stereoscopic displays. In 3 method-of-adjustment procedure experiments, 8 participants matched the disparity of a circular probe and a planar stimulus rendered using a single visible edge. The experiments varied stimulus orientation and surface texture. Participants more accurately matched the depth of vertical stimuli than that of horizontal stimuli, consistent with previous studies and existing theory. Participants matched the depth of surfaces with large pixel-to-pixel luminance variations more accurately than they did surfaces with a small pixel-to-pixel luminance variation. Finally, they matched the depth of surfaces with vertical line patterns more accurately than they did surfaces with horizontal-striped texture patterns. These results suggest that designers can enhance depth perception in stereoscopic displays, and also reduce undesirable sensitivity to orientation, by rendering objects with surface textures using large pixel-to-pixel luminance variations.

Curvature biases in stereoscopic vision: a nasotemporal asymmetry

The reliability of curvature judgments for linear elements was studied, with stereograms that contained a binocular arc with curvature in depth, and either a binocular frontoparallel arc or a monocular one, on a background representing a hemiellipsoid. The subjects made about 15% errors on binocular arcs with curvature in depth, and 60%-80% of these occurred when both the hemiellipsoid and the arc were convex, the arc being perceived as concave, by transparency through the hemiellipsoid. There were also about 15%-30% errors on frontoparallel arcs, but spread among all situations, with a small prevalence of concave judgments. Curvature in depth was assigned to the monocular stimuli in more than 60% of the cases. There was a curvature bias when the monocular arcs were on the nasal side, and were viewed against a concave background. Assuming parallel viewing, nasal ingoing arcs were usually perceived as concave, and nasal outgoing arcs usually perceived as convex, in agreement with geometrical likelihood. Nasal-side elements captured by one eye are, in general, those with the highest likelihood of having matching elements in the other eye. Then the observed nasal bias effect suggests that the matching process in stereopsis could be driven from the nasal sides of the projections in the two cerebral hemispheres.

Depth capture by kinetic depth and by stereopsis

The perceived depth of regions within a stereogram lacking explicit disparity information can be captured by the surface structure of regions where disparity is explicit: stereo capture. In two experiments, observers estimated surface curvature/depth of an untextured object (a 'ribbon') superimposed on a cylinder textured with dots, the cylinder curvature being defined by disparity (stereo depth) or by motion parallax (kinetic depth: KD). With the stereo-defined cylinder, depth capture was obtained under conditions where the disparity of the ribbon was ambiguous; with the KD, cylinder depth capture was obtained under conditions where the motion flow of the cylinder was in a direction parallel to that of the ribbon. These results demonstrate yet another similarity between KD and stereopsis.

Anisotropy in Werner's binocular depth-contrast effect

We investigated Werner's binocular depth-contrast effect. Subjects viewed stereograms consisting of a test pattern and an inducing pattern. The half-images of the inducing pattern were either horizontally scaled or sheared relative to each other. Subjects judged the (induced) perceived slant of the test pattern. We were interested in what influence the spatial configuration of the test pattern and the inducing pattern had on the depth-contrast effect. We conclude that the depth-contrast effect is a global effect. In other words, it is not restricted to the location of the inducing pattern. The effect decreases with distance, however, in an anisotropic way. The depth-contrast effect was present most prominently when the test pattern was positioned in the direction along the slant (rotation) axis of the inducing pattern. We suggest that Werner's depth-contrast effect can be explained by the (previously reported) findings that: (1) stereopsis is relatively insensitive to whole-field horizontal scale and shear; and (2) stereopsis is very sensitive to horizontal scale and shear of two stimuli relative to each other.

On the accuracy of surface reconstruction from disparity interpolation

Observers viewed flashed random-dot stereograms depicting a pair of long, narrow, curved ribbons of textured surface defined by a Gabor function in disparity. Observers judged the location of the peak of the depth profile of one ribbon relative to that of the other. In one ribbon, disparity changed smoothly while in the other disparity was periodically sampled. Up to a limiting sampling period, disparity interpolation produced accurate surface reconstruction, but beyond that performance deteriorated rapidly. This interpolation limit depended on surface orientation (vertical vs horizontal) and disparity sign, but not Gabor spatial frequency.

Structure-from-motion: perceptual evidence for surface interpolation

Dynamic random-dot displays representing a rotating cylinder were used to investigate surface interpolation in the perception of structure-from-motion (SFM) in humans. Surface interpolation refers to a process in which a complete surface in depth is reconstructed from the object depth values extracted at the stimulus features. Surface interpolation will assign depth values even in parts of the object that contain no features. Such a "fill-in" process should make the detection of featureless stimulus areas ("holes") difficult. Indeed, we demonstrate that such holes in our rotating cylinder can be as wide as one-quarter of the stimulus before subjects can reliably detect their presence. Subjects were presented with a variation on the rotating cylinder in which all dots were oscillating either in synchrony or asynchronously. Subjects perceive a rigidly rotating cylinder even when such a percept is not in agreement with the physical stimulus. To reconcile this discrepancy between actual and perceived stimulus we propose that individual points contribute to a surface based object representation and that in this process the visual system looses access to the identity of the individual features that make up the surface. Finally we are able to explain a variety of previously documented perceptual peculiarities in the perception of structure-from-motion by arguing that the perceptual interpretation of the object's boundaries influences the surface interpolation process. These findings offer strong perceptual evidence for a process of surface interpolation and are also physiologically plausible given results from recordings in awake behaving monkey cortical areas V1 and MT. The companion paper demonstrates how such a surface interpolation process can be incorporated into a structure-from-motion algorithm and how object boundaries can influence the perception of structure-from-motion as has been demonstrated before and in this paper.

3-D vision and figure-ground separation by visual cortex

A neural network theory of three-dimensional (3-D) vision, called FACADE theory, is described. The theory proposes a solution of the classical figure-ground problem for biological vision. It does so by suggesting how boundary representations and surface representations are formed within a boundary contour system (BCS) and a feature contour system (FCS). The BCS and FCS interact reciprocally to form 3-D boundary and surface representations that are mutually consistent. Their interactions generate 3-D percepts wherein occluding and occluded object parts are separated, completed, and grouped. The theory clarifies how preattentive processes of 3-D perception and figure-ground separation interact reciprocally with attentive processes of spatial localization, object recognition, and visual search. A new theory of stereopsis is proposed that predicts how cells sensitive to multiple spatial frequencies, disparities, and orientations are combined by context-sensitive filtering, competition, and cooperation to form coherent BCS boundary segmentations. Several factors contribute to figure-ground pop-out, including: boundary contrast between spatially contiguous boundaries, whether due to scenic differences in luminance, color, spatial frequency, or disparity; partially ordered interactions from larger spatial scales and disparities to smaller scales and disparities; and surface filling-in restricted to regions surrounded by a connected boundary. Phenomena such as 3-D pop-out from a 2-D picture, Da Vinci stereopsis, 3-D neon color spreading, completion of partially occluded objects, and figure-ground reversals are analyzed. The BCS and FCS subsystems model aspects of how the two parvocellular cortical processing streams that join the lateral geniculate nucleus to prestriate cortical area V4 interact to generate a multiplexed representation of Form-And-Color-And-DEpth, or FACADE, within area V4. Area V4 is suggested to support figure-ground separation and to interact with cortical mechanisms of spatial attention, attentive object learning, and visual search. Adaptive resonance theory (ART) mechanisms model aspects of how prestriate visual cortex interacts reciprocally with a visual object recognition system in inferotemporal (IT) cortex for purposes of attentive object learning and categorization. Object attention mechanisms of the What cortical processing stream through IT cortex are distinguished from spatial attention mechanisms of the Where cortical processing stream through parietal cortex. Parvocellular BCS and FCS signals interact with the model What stream. Parvocellular FCS and magnocellular motion BCS signals interact with the model Where stream.(ABSTRACT TRUNCATED AT 400 WORDS)

Occlusion cues contribute to orientation judgments of occlusion-defined contours

Occlusion cues defining a contour in a 2-D stimulus pattern were shown to contribute to the accuracy of orientation judgments of that contour. The stimulus pattern was altered so that the occlusion cues became ambiguous, by introducing a textured background suggesting transparency of the stimulus pattern. Orientation judgments then became significantly less accurate. This finding shows that occlusion cues in 2-D patterns can be behaviorally relevant, in addition to generating the subjective percept commonly known as an illusory contour. The disruptive effect of the textured background on orientation judgments remained when no texture elements were present in the vicinity of the contour. This suggests that the generation of occlusion-defined contours relies as much on an evaluation of the surfaces at either side of the contour as being opaque as it does on local encoding of occlusion cues close to the contour. Finally, orientation sensitivity measured with contours defined by other than occlusion cues was not altered after the introduction of a textured background.

Shape from texture: ideal observers and human psychophysics

We describe an ideal observer model for estimating "shape from texture" which is derived from the principles of statistical information. For a given family of surface shapes, measures of statistical information can be computed for two different texture cues--density and orientation of texels. These measures can be used to predict lower bounds on the variance of shape judgements of "ideal" and human observers. They can also predict optimal weights for cue integration for the inference of shape from texture. These weights are directly proportional to the information carried by each cue. The ideal observer model therefore predicts that the variance of subjects' responses in a psychophysical shape judgement task should reflect the statistical importance of individual texture cues. Our results show that human performance in shape judgements for a one-parameter family of parabolic cylinders is often better than what an ideal observer achieves using a density cue alone. Therefore other information, for example the compression cue, must be used by human observers. For the first time, such results have been obtained without recourse to the unnatural cue conflict paradigms used in previous experiments. The model makes further predictions for the perception of planar slanted surfaces in the case of wide field of view.

Neural dynamics of motion perception: direction fields, apertures, and resonant grouping

A neural network model of global motion segmentation by visual cortex is described. Called the motion boundary contour system (BCS), the model clarifies how ambiguous local movements on a complex moving shape are actively reorganized into a coherent global motion signal. Unlike many previous researchers, we analyze how a coherent motion signal is imparted to all regions of a moving figure, not only to regions at which unambiguous motion signals exist. The model hereby suggests a solution to the global aperture problem. The motion BCS describes how preprocessing of motion signals by a motion oriented contrast (MOC) filter is joined to long-range cooperative grouping mechanisms in a motion cooperative-competitive (MOCC) loop to control phenomena such as motion capture. The motion BCS is computed in parallel with the static BCS of Grossberg and Mingolla (1985a, 1985b, 1987). Homologous properties of the motion BCS and the static BCS, specialized to process motion directions and static orientations, respectively, support a unified explanation of many data about static form perception and motion form perception that have heretofore been unexplained or treated separately. Predictions about microscopic computational differences of the parallel cortical streams V1-->MT and V1-->V2-->MT are made--notably, the magnocellular thick stripe and parvocellular interstripe streams. It is shown how the motion BCS can compute motion directions that may be synthesized from multiple orientations with opposite directions of contrast. Interactions of model simple cells, complex cells, hyper-complex cells, and bipole cells are described, with special emphasis given to new functional roles in direction disambiguation for endstopping at multiple processing stages and to the dynamic interplay of spatially short-range and long-range interactions.

On the coexistence of stereopsis and binocular rivalry

Dichoptically viewed complex texture stereograms with correlated spatial frequency information can yield stable depth perception, implying cooperative interaction between the two eyes. Dichoptically viewed dissimilar texture pairs may yield competition in the form of binocular rivalry. To study whether stereopsis and rivalry can spatially coexist when stimulus conditions for both are present, we had observers dichoptically view spatial frequency filtered random-dot patterns. The left eye viewed one half-image of an RDS; the right eye viewed the superimposition of the other RDS half-image (which when paired alone with the left-eye RDS yielded stereoscopic depth) and a noise target (which on its own engaged in rivalry with the right eye target). Observers judged the quality of depth and the rate of rivalry for these stereo-pairs. When the contrast of the noise component was low, observers experienced stereopsis and stable single vision that included the noise. At intermediate noise contrasts, local regions were seen either in rivalry or in stereoscopic depth, but rivalry and depth were not experienced at the same spatial location simultaneously. At high noise contrasts, the right eye target dominated almost exclusively, with little hint of stereopsis. Essentially the same pattern of results was obtained in forced-choice experiments in which observers judged the direction of stereoscopic tilt from vertical cosine gratings differing slightly in spatial frequency. Considered together, these results are inconsistent with theories positing that rivalry and stereopsis coexist at the same spatial location because they occur within independent, parallel pathways.