Stereopsis by harmonic analysis

What's special about horizontal disparity

Horizontal disparity has been recognized as the primary signal driving stereoscopic depth since the invention of the stereoscope in the 1830s. It has a unique status in our understanding of binocular vision. The direction of offset of the eyes gives the disparities of corresponding image point locations across the two retinas a strong horizontal bias. Beyond the retina, other factors give shape to the effective disparity direction used by visual mechanisms. The influence of orientation is examined here. I argue that horizontal disparity is an inflection point along a continuum of effective directions, and its role in stereo vision can be reinterpreted. The pointwise geometric justification for its special status neglects the oriented structural elements of spatial vision, its physiological support is equivocal, and psychophysical support of its special status may partially reflect biased stimulus sampling. The literature shows that horizontal disparity plays no particular role in the processing of one-dimensional stimuli, a reflection of the stereo aperture problem. The resulting depth is non-veridical, even non-transitive. Although one-dimensional components contribute to the stereo depth of visual objects generally, two-dimensional stimuli appear not to inherit the aperture problem. However, a look at the two-dimensional stimuli that predominate in experimental studies shows regularities in orientation that give a new perspective on horizontal disparity.

Short-latency disparity vergence in humans: evidence for early spatial filtering

Our study was concerned with the disparity detectors underlying the initial disparity vergence responses (DVRs) that are elicited at ultrashort latencies by binocular disparities applied to large images. DVRs were elicited in humans by applying horizontal disparity to vertical square-wave gratings lacking the fundamental (termed here, the "missing fundamental"). In the frequency domain, a pure square wave is composed of odd harmonics--first, third, fifth, seventh, etc.--such that the third, fifth, seventh, etc., have amplitudes that are one-third, one-fifth, one-seventh, etc., that of the first, and the missing fundamental lacks the first harmonic. The patterns seen by the two eyes have a phase difference of one-quarter wavelength, so the disparity of the features and 4n + 1 harmonics (where n = integer) has one sign (crossed or uncrossed), whereas the 4n - 1 harmonics--including the strongest Fourier component (the third harmonic)--has the opposite sign (uncrossed or crossed): spatial aliasing. The earliest DVRs, recorded with the search-coil technique, had minimum latencies of 70 to 80 ms and were generally in the direction of the third harmonic, that is, uncrossed disparities resulted in convergent eye movements. In other experiments on the DVRs, one eye saw a missing fundamental and the other saw a pure sine wave with the contrast and wavelength of the third harmonic but differing in phase by one-quarter wavelength. This resulted in short-latency vergence in accordance with matching of the third harmonic. These data all indicate the importance of the Fourier components, consistent with early spatial filtering prior to binocular matching.

Short-latency disparity vergence eye movements: a response to disparity energy

Vergence eye movements were elicited in human subjects by applying disparities to square-wave gratings lacking the fundamental ("missing fundamental", mf). Using a dichoptic arrangement, subjects viewed gratings that were identical at the two eyes except for a phase difference of 1/4 wavelength so that, based on the nearest-neighbor matches, the features and the 4n+1 harmonics (5th, 9th, etc.) all had binocular disparities of one sign, whereas the 4n-1 harmonics (3rd, 7th, etc.) all had disparities of the opposite sign. Further, the amplitude of the ith harmonic was proportional to 1/i. Using the electromagnetic search coil technique to record the positions of both eyes indicated that the earliest vergence eye movements elicited by these disparity stimuli had ultra-short latencies (minimum, <65 ms) and were always in the direction of the most prominent harmonic, the 3rd, but their magnitudes fell short of those elicited when the same disparities were applied to pure sinusoids whose spatial frequency and contrast matched those of the 3rd harmonic. This shortfall was evident in both the horizontal vergence responses recorded with vertical grating stimuli and the vertical vergence responses recorded with horizontal grating stimuli. When the next most prominent harmonic, the 5th, was removed from the mf stimulus (creating the "mf-5" stimulus) the vertical vergence responses showed almost no shortfall-indicating that it had been almost entirely due to that 5th harmonic-but the horizontal vergence responses still showed a small shortfall, at least with higher contrast stimuli. This small shortfall might represent a very minor contribution from higher harmonics and/or distortion products and/or a feature-based mechanism. We conclude that the earliest disparity vergence responses-especially vertical-were strongly dependent on the major Fourier components of the binocular images, consistent with early spatial filtering of the monocular visual inputs prior to their binocular combination as in the disparity-energy model of complex cells in striate cortex [Ohzawa, I., DeAngelis, G. C., & Freeman, R. D. (1990). Stereoscopic depth discrimination in the visual cortex: neurons ideally suited as disparity detectors. Science, 249, 1037-1041].

Detecting disparity in two-dimensional patterns

One can measure the disparities between two retinal images in several different ways. Experiments were conducted to identify the measure that is invariant at the threshold for detecting the disparity of two-dimensional patterns. The patterns used were stereo plaids, which permit a partial dissociation between the disparity of the pattern and the disparities of its one-dimensional components. For plaids with near-horizontal disparities, thresholds are limited by a disparity phase shift equal to the threshold phase shift for single gratings. For non-horizontal disparities, thresholds are elevated, yet are still phase-limited. In no disparity direction are thresholds for detecting disparity determined by the spatial extent of the plaids' disparity. Effects of the number and the orientation of components with task-relevant disparities indicate that plaid thresholds are limited by the disparity of the plaid's one-dimensional components. No evidence was found that these components form any higher-order pattern that can be used in detecting disparity. Oblique and near-vertical disparities generate elevated thresholds at a stage beyond component disparity detection. This second stage combines component disparities, which are ambiguous about depth, into pattern disparities capable of supporting veridical depth perception.

Stereopsis from interocular spatial frequency differences is not robust

Based on data obtained using one-dimensional noise patterns, Tyler & Sutter (1979). (Vision Research, 19, 859-865) concluded that stereoscopic tilt can result from an interocular spatial frequency difference in the absence of consistent horizontal disparity. We tested stereopsis using two-dimensional random-dot patterns that were bandpass filtered to contain 1.0 octave bands of spatial frequency with means that differed between the two eyes. With vertical, one-dimensional stimuli we replicated the results of Tyler and Sutter. However, stereoscopic tilt was not perceived based on spatial frequency differences alone when the monocular images contained as little as a +/- 14 deg range of orientation variation. In addition, model simulations demonstrate that the modest steroscopic performance produced by interocular spatial frequency differences in one-dimensional noise patterns are predicted by random disparity correlations at the pattern edges. The observations lead to the conclusion that stereopsis from frequency differences in the absence of pointwise disparity correlations does not reflect a special processing capability of human vision but is an artifact associated with one-dimensional stimuli. As such, it plays no role in steroscopic analysis of the natural environment.

Fine-to-coarse scale disambiguation in stereopsis

Spatial frequency selectivity has been incorporated into various theories of stereo matching, along with spatial scale interactions operating from coarse-to-fine spatial scales. We concentrate here on the role of fine scale information in the stereo matching process and show that fine scale information is capable of disambiguating matches made at coarser scales. An ambiguous coarse scale stimulus was created by presenting a low frequency (2 c/deg) sine wave in anti-phase to the two eyes, whose endpoints betrayed no information about which way the sine waves should be matched. It could be seen with crossed or uncrossed disparity equally validly and at chance from trial to trial. To this was added a fine scale (8 c/deg) filtered random dot stimulus specifying unambiguously a certain disparity. Observers judged the apparent depth of the two stimuli as the disparity of the fine scale stimulus was varied. The sine wave was usually perceived to have the same sign disparity as the fine scale stimulus. Depth matching with the two superimposed stimuli confirmed that the coarse scale stimulus was actually disambiguated, and seen with disparities equal to half its spatial period. The results suggest the operation of a cross-spatial scale matching disambiguation process, which can operate in a fine-to-coarse fashion.

The temporal range of motion sensing and motion perception

Apparent motion (AM) was studied using the missing-fundamental square-wave grating, displaced discretely over time. At inter-stimulus intervals (ISIs) greater than about 40 msec, AM was seen in the direction of displacement of the visible features of the pattern, while at shorter ISIs AM was seen in the reversed direction, following the displacement of the third harmonic spatial frequency component. This confirms (a) that the "long-range", feature-based process can bridge much greater time-gaps than the "short-range" motion sensors and (b) that the missing-fundamental pattern is a particularly useful tool for teasing the two processes apart.

A stereoscopic view of visual processing streams

Recent anatomical and physiological studies of the visual pathway suggest the existence of at least three parallel processing streams in the lateral geniculate/primary cortex structure--a magno/interblob stream for motion and transient information; a parvo/interblob stream for high spatial frequency, static information; and a parvo/blob stream for chromatic and low spatial frequency information. How does this functional typology relate to the processing for stereoscopic depth? Human stereopsis may be viewed as consisting of three distinct types of disparity processing: coarse, local stereopsis suitable for stereomovement processing by the magno/interblob stream; fine, global stereopsis suitable for the processing of complex random-dot stereograms by the parvo/interblob stream; and simple, protostereopsis for processing size differences between the two eyes by the parvo/blob stream. Extensive psychophysical evidence supports the identification of these three disparity processes with the three processing streams.

Is edge information for stereoacuity spatially channeled?

Models of stereopsis generally assume that binocular correspondence is achieved through alignment of luminance edges in the two eyes. Yet the stimulus properties which constitute edge information for stereopsis have not been defined. Three experiments explored the nature of these stimulus properties. The first two experiments tested whether local luminance gradient and the relative phase of spatial components supply information about the position of edges which influences stereosensitivity. In Expt 1, stereothresholds were reduced with increased spatial frequency or contrast of sinusoidal luminance gratings, but no simple relationship between target luminance gradient and stereosensitivity was found. In Expt 2, stereothresholds were equivalent for targets having identical spatial frequency components, but differing in maximum luminance gradient and the relative spatial phase of their components. In addition, stereothresholds were lower for the target having the higher contrast in pairs of unequal-contrast targets having equal maximum luminance gradients. These results suggest that the properties of luminance gradient and relative spatial phase do not influence stereosensitivity independently of spatial frequency and contrast. Experiment 3 directly tested whether stereosensitivity depends on edge information whose disparity is detected independently at different spatial scales. Stereothresholds for IF + 5F compound targets were found to be equivalent to thresholds obtained separately with the more sensitive of the two components. Taken together with a compressive nonlinearity in the relationship between contrast and stereothreshold obtained by others (Halpern and Blake, 1989; Legge and Gu, 1989) and replicated in Expt 1, the results of Expt 3 indicate that, whatever the exact nature of the luminance discontinuity information utilized in disparity detection, it is processed independently at different spatial scales.

Interocular differences in contrast and spatial frequency: effects on stereopsis and fusion

Anisometropia produces interocular differences in contrast and spatial frequency. The influence of these two parameters on Panum's fusional limit (PFL) and stereoscopic depth thresholds was investigated with sinusoidal gratings and one-dimensional band-pass-limited targets. Vertical fusion limits were unaffected by large interocular differences in contrast (40-10%) at two spatial frequencies (0.8 and 1.6 cpd). However, when tested with a low spatial frequency (0.8 cpd), stereothresholds increased 150% with an interocular difference in contrast as small as 50-25%. Stereoacuity was reduced less by differential contrast when tested with higher spatial frequencies (3.2 cpd). When tested with low spatial frequencies the stereothreshold was elevated more by reducing the contrast of one image than by equal contrast reductions of both ocular images. Stereothresholds appear to be elevated by binocular suppression evoked by interocular differences in contrast. Vertical as well as horizontal fusion limits decreased with increasing interocular size difference. Horizontal fusion limits fell off more gradually with increasing size difference than did vertical fusion limits, particularly at higher spatial frequencies (2.4 cpd). Similarly, stereothresholds increased with increasing interocular size differences. Changes in the fusion limit and stereothreshold that occur with interocular size differences are predicted from positional disparities between edge features rather than from differences in spatial frequency.

Apparent motion can be perceived between patterns with dissimilar spatial frequencies

Many recent models of movement processing in the human visual system predict that the perception of apparent motion requires stimuli that are similar in spatial frequency. The data presented here provide an example of the perception of apparent motion between patterns with nonoverlapping harmonic content. When patterns presented in alternate frames are dissimilar, motion can be perceived as long as the velocity is not too high.

Field processes in stereovision. A description of stereopsis appropriate to ophthalmology and visual perception

There is, as yet, no satisfactory theory of stereopsis, despite the fact that our overt knowledge of "solid seeing" is now about 150 years old, and that contributions to our understanding come today from many fields: ophthalmology, psychology, psychophysics, neurophysiology, computer modelling, and optical-TV display technology. We review herein, and demonstrate for the reader whenever possible, certain key perceptual properties of the stereoscopic event of which any general theory must take account: vector stereoscopy and the neural grid, depth in empty visual fields, the relationship between stereoscopic and cognitive contours, stereoscopic contour formation in the presence of blur (thus, at low levels of central visual acuity), the phenomenon of cortical locking and of neural grid evocation in the presence of either peripheral or central rivalry, certain unusual ranges of figural mismatch and the concept of the horopter in relation to modern single cell electroneurophysiology in animals and to the constancy of visual directions. Some comments are also made on the concept of disparity processing by single cortical neurons, together with a short discussion of the implications of certain views of the genetics of stereovision for the perception of novel random texture sine-wave stereograms. We conclude that any theory pertinent to ophthalmology and visual science must combine the global concepts of cortical integration, the neural lock and the neural grid, herein introduced, with the more classical concepts of particulate or local binocular cortical correspondence. Certain preliminary steps in this direction are presented.

What causes stereoscopic tilt from spatial frequency disparity

A controversy still exists concerning whether the tilt created with interocular spatial frequency disparity arises from a computation of spatial frequency differences or from cumulative positional disparity. In a first experiment, we examined the influence of positional disparity on tilt created with frequency disparity, reasoning that if tilt were computed from spatial frequency differences, the perceived angle should remain unaltered since adding a positional disparity does not change the harmonic content of the stimulus. The results indicated that positional disparity weakened perceived tilt. In a second experiment, we tested the idea that tilt results from the calculation of increasing positional disparity across the display, arguing if local matches of features in the two eyes are made in computing tilt, then the solution to binocular correspondence may be less ambiguous if the same number of cycles was displayed for both spatial frequencies. Perceived tilt increased when the number of cycles was equal, although the angle of tilt still decreased with positional disparity. In Experiment 3, we further reduced potential sources of ambiguity for the binocular matching process by employing D10s (the tenth derivative of a Gaussian) instead of grating patterns. Positional disparity exerted essentially no influence on the perceived angle of tilt of the D10s. Taken together, the results of these experiments suggest that tilt from frequency disparity can be explained solely on the basis of positional disparity.

Spatial tuning of static and dynamic local stereopsis

The range of spatial tuning for channels that process static and dynamic disparities was investigated in the central visual field by measuring stereoscopic thresholds as a function of the difference in size of spatially filtered bar-like patterns presented to the two eyes. Spatial tuning functions were revealed by an elevation of stereothreshold as the difference between the widths of bar patterns increased. Functions tuned to low spatial frequencies (0.075-2 c/deg) were classified as transient since their stereosensitivity was greater for dynamic (1 Hz) than static disparities. Functions tuned to high spatial frequencies (2.4-19 c/deg) were classified as sustained since their stereosensitivity was equal for dynamic and static disparities. When equal width patterns were presented to the two eyes, stereothreshold increased with spatial periods greater than 0.4 deg according to a constant 6 deg phase disparity. This size-disparity correlation suggests that large disparities are processed by spatial filters tuned to disparities proportional to their receptive field dimensions.

Spatial bandwidth of channels for slant estimated from complex gratings

Vertical sinusoidal gratings of slightly differing spatial frequencies presented to each eye lead to the perception of a slanted plane. If a complex pattern is created by presenting two different frequencies to one eye, while the other eye views an increment of one of those frequencies and a decrement of the other, then the stimulus is equivalent to two superimposed vertical planes slanted in opposite directions. Yet the observer generally sees only a single surface at a slant intermediate to the possible extremes. As the relative spatial-frequency content of the two opposing planes is varied, the observed slant is a measure of the strengths of the connecting interactions of the underlying stereomechanisms. From these data the bandwidth of the monocular input channels for the binocular slant mechanism can be estimated and is found to be about two octaves.