Spatial frequency channels in human vision as asymmetric (edge) mechanisms

Biological Basis and Computer Vision Applications of Image Phase Congruency: A Comprehensive Survey

The concept of Image Phase Congruency (IPC) is deeply rooted in the way the human visual system interprets and processes spatial frequency information. It plays an important role in visual perception, influencing our capacity to identify objects, recognize textures, and decipher spatial relationships in our environments. IPC is robust to changes in lighting, contrast, and other variables that might modify the amplitude of light waves yet leave their relative phase unchanged. This characteristic is vital for perceptual tasks as it ensures the consistent detection of features regardless of fluctuations in illumination or other environmental factors. It can also impact cognitive and emotional responses; cohesive phase information across elements fosters a perception of unity or harmony, while inconsistencies can engender a sense of discord or tension. In this survey, we begin by examining the evidence from biological vision studies suggesting that IPC is employed by the human perceptual system. We proceed to outline the typical mathematical representation and different computational approaches to IPC. We then summarize the extensive applications of IPC in computer vision, including denoise, image quality assessment, feature detection and description, image segmentation, image registration, image fusion, and object detection, among other uses, and illustrate its advantages with a number of examples. Finally, we discuss the current challenges associated with the practical applications of IPC and potential avenues for enhancement.

Suprathreshold length summation

To investigate the mechanisms underlying elongated spatial summation with a pattern-masking paradigm, we measured the contrast detection thresholds for elongated Gabor targets situated at 3° eccentricity to either the left or right of the fixation and elongated along an arc of the same radius to access homogeneous retinal sensitivity. The mask was a ring with a Gabor envelope of the same 3° center radius containing either a concentric (iso-orientation mask) or a radial (orthogonal mask) modulation. The task of the observer was to indicate whether the target in each trial was on the left or the right of the fixation. With orthogonal or low contrast iso-orientation masks, target thresholds first decreased with size with slope -1 on log-log coordinates until the target length reached 45' (specified as the half-height full-width of the Gabor envelope) and then further decreased according to a slope of -1/2, the latter being the signature of an ideal summation process. When the contrast of the iso-orientation mask was sufficiently high, however, the target thresholds, while still showing a -1 slope up to ∼10', asymptoted up to about 50' length, suggesting that the presence of the mask eliminated the ideal summation regime. Beyond about 50', the data approximated another -1 slope decrease in threshold, suggesting the existence of an extra-long channel that is not revealed by the conventional spatial summation paradigm. The full results could be explained by a divisive inhibition model, in which second-order filters sum responses across local oriented channels, combined with a single extra-long filter at least 300' in extent. In this model, the local filter response is given by the linear excitation of the local channels raised to a power, and scaled by divisive inhibition from all channels in the neighborhood. With the high-contrast iso-orientation masks, such divisive inhibition swamps the response to eliminate the ideal summation regime until the stimulus is long enough to activate the extra-long filter.

An image-driven model for pattern detection, resistant to Birdsall linearisation

If detection were governed by an isolated (and possibly nonlinear) transducer, then a linearisation of the psychometric function (d-prime vs target amplitude) must accompany any threshold elevation due to the addition of external noise. This is the Birdsall theorem. From the fact that noise can elevate threshold without linearising the psychometric function, we can safely infer that detection is not governed by an isolated transducer. Heretofore, image-driven models, which accept images or numerical descriptions thereof as input, have proven incompatible with this failure of Birdsall linearisation, unless they incorporate the principle of intrinsic uncertainty, which asserts that detection is governed by the maximum activity in several independent (noisy) sensors. One image-driven model incompatible with the failure of Birdsall linearisation is Watson and Solomon's (J. Opt. Soc. Am. A, 14 (1997), 2379) model of visual contrast gain control and pattern masking. Here I report a simple modification - pooling sensor outputs before, instead of after the comparison of input images - allowing that model to predict curved psychometric functions, even when external noise elevates threshold by more than 20 dB, without any detrimental effect to the quality of its fit to pattern-masking thresholds in the absence of noise. The failure of Birdsall linearisation, therefore, does not necessarily imply independent samples of performance-limiting noise in multiple visual sensors. Instead, performance-limiting noise may arise after the visual system combines output from mutually inhibitory sensors.

What is visible across the visual field?

It is sometimes claimed that because the resolution and sensitivity of visual perception are better in the fovea than in the periphery, peripheral vision cannot support the same kinds of colour and sharpness percepts as foveal vision. The fact that a scene nevertheless seems colourful and sharp throughout the visual field then poses a puzzle. In this study, I use a detailed model of human spatial vision to estimate the visibility of certain properties of natural scenes, including aspects of colourfulness, sharpness, and blurriness, across the visual field. The model is constructed to reproduce basic aspects of human contrast and colour sensitivity over a range of retinal eccentricities. I apply the model to colourful, complex natural scene images, and estimate the degree to which colour and edge information are present in the model's representation of the scenes. I find that, aside from the intrinsic drift in the spatial scale of the representation, there are not large qualitative differences between foveal and peripheral representations of 'colourfulness' and 'sharpness'.

Lateral effects in pattern vision

There is a large literature on lateral effects in pattern vision but no consensus about them or comprehensive model of them. This paper reviews the literature with a focus on the effects of parallel context in the central fovea. It describes seven experiments that measure detection and discrimination thresholds in annular and Gabor-pattern contexts at different separations. It presents a model of these effects, which is an elaboration of Foley's (1994) model. The model describes the results well, and it shows that lateral context affects the response to the target by both multiplicative excitation and additive inhibition. Both lateral effects extend for several wavelengths beyond the target. They vary in relative strength, producing near suppression and far enhancement of the response to the target. The model describes the detection and discrimination results well, and it also describes the results of experiments on lateral effects on perceived contrast. The model is consistent with the physiology of V1 cells.

Objective assessment of visual acuity: a refined model for analyzing the sweep VEP

PURPOSE

The aim of this study was to develop a simple and reliable method for the objective assessment of visual acuity by optimizing the stimulus used in commercially available systems and by improving the methods of evaluation using a nonlinear function, the modified Ricker model.

METHODS

Subjective visual acuity in the normal subjects was measured with Snellen targets, best-corrected, and in some cases also uncorrected and with plus lenses (+ 1 D, + 2 D, + 3 D). In patients, subjective visual acuity was measured best-corrected using the Freiburg Visual Acuity Test. Sweep VEP recordings to 11 spatial frequencies, with check sizes in logarithmically equidistant steps (0.6, 0.9, 1.4, 2.1, 3.3, 4.9, 7.3, 10.4, 18.2, 24.4, and 36.5 cpd), were obtained from 56 healthy subjects aged between 17 and 69 years (mean 42.5 ± 15.3 SD years) and 20 patients with diseases of the lens (n = 6), retina (n = 8) or optic nerve (n = 6). The results were fit by a multiple linear regression (2nd-order polynomial) or a nonlinear regression (modified Ricker model) and parameters compared (limiting spatial frequency (sf_limiting) and the spatial frequency of the vertex (sf_vertex) of the parabola for the 2nd-order polynomial fitting, and the maximal spatial frequency (sf_max), and the spatial frequency where the amplitude is 2 dB higher than the level of noise (sf_threshold) for the modified Ricker model.

RESULTS

Recording with 11 spatial frequencies allows a more accurate determination of acuities above 1.0 logMAR. Tuning curves fitted to the results show that compared to the normal 2nd-order polynomial analysis, the modified Ricker model is able to describe closely the amplitudes of the sweep VEP in relation to the spatial frequencies of the presented checkerboards. In patients with a visual acuity better than about 0.5 (decimal), the predicted acuities based on the different parameters show a good match of the predicted visual acuities based on the models established in healthy volunteers to the subjective visual acuities. However, for lower visual acuities, both models tend to overestimate the visual acuity (up to ~ 0.4 logMAR), especially in patients suffering from AMD.

CONCLUSIONS

Both models, the 2nd-order polynomial and the modified Ricker model performed equally well in the prediction of the visual acuity based on the amplitudes recorded using the sweep VEP. However, the modified Ricker model does not require the exclusion of data points from the fit, as necessary when fitting the 2nd-order polynomial model making it more reliable and robust against outliers, and, in addition, provides a measure for the noise of the recorded results.

Estimates of edge detection filters in human vision

Edge detection is widely believed to be an important early stage in human visual processing. However, there have been relatively few attempts to map human edge detection filters. In this study, observers had to locate a randomly placed step edge in brown noise (the integral of white noise) with a 1/f² power spectrum. Their responses were modelled by assuming the probability the observer chose an edge location depended on the response of their own edge detection filter to that location. The observer's edge detection filter was then estimated by maximum likelihood methods. The filters obtained were odd-symmetric and similar to a derivative of Gaussian, with a peak-to-trough width of 0.1-0.15 degrees. These filters are compared with previous estimates of edge detectors in humans, and with neurophysiological receptive fields and theoretical edge detectors.

A spatial frequency spectral peakedness model predicts discrimination performance of regularity in dot patterns

Subjective assessments of spatial regularity are common in everyday life and also in science, for example in developmental biology. It has recently been shown that regularity is an adaptable visual dimension. It was proposed that regularity is coded via the peakedness of the distribution of neural responses across receptive field size. Here, we test this proposal for jittered square lattices of dots. We examine whether discriminability correlates with a simple peakedness measure across different presentation conditions (dot number, size, and average spacing). Using a filter-rectify-filter model, we determined responses across scale. Consistently, two peaks are present: a lower frequency peak corresponding to the dot spacing of the regular pattern and a higher frequency peak corresponding to the pattern element (dot). We define the "peakedness" of a particular presentation condition as the relative heights of these two peaks for a perfectly regular pattern constructed using the corresponding dot size, number and spacing. We conducted two psychophysical experiments in which observers judged relative regularity in a 2-alternative forced-choice task. In the first experiment we used a single reference pattern of intermediate regularity and, in the second, Thurstonian scaling of patterns covering the entire range of regularity. In both experiments discriminability was highly correlated with peakedness for a wide range of presentation conditions. This supports the hypothesis that regularity is coded via peakedness of the distribution of responses across scale.

Binocular contrast-gain control for natural scenes: Image structure and phase alignment

In the context of natural scenes, we applied the pattern-masking paradigm to investigate how image structure and phase alignment affect contrast-gain control in binocular vision. We measured the discrimination thresholds of bandpass-filtered natural-scene images (targets) under various types of pedestals. Our first experiment had four pedestal types: bandpass-filtered pedestals, unfiltered pedestals, notch-filtered pedestals (which enabled removal of the spatial frequency), and misaligned pedestals (which involved rotation of unfiltered pedestals). Our second experiment featured six types of pedestals: bandpass-filtered, unfiltered, and notch-filtered pedestals, and the corresponding phase-scrambled pedestals. The thresholds were compared for monocular, binocular, and dichoptic viewing configurations. The bandpass-filtered pedestal and unfiltered pedestals showed classic dipper shapes; the dipper shapes of the notch-filtered, misaligned, and phase-scrambled pedestals were weak. We adopted a two-stage binocular contrast-gain control model to describe our results. We deduced that the phase-alignment information influenced the contrast-gain control mechanism before the binocular summation stage and that the phase-alignment information and structural misalignment information caused relatively strong divisive inhibition in the monocular and interocular suppression stages. When the pedestals were phase-scrambled, the elimination of the interocular suppression processing was the most convincing explanation of the results. Thus, our results indicated that both phase-alignment information and similar image structures cause strong interocular suppression.

Fixed versus variable internal noise in contrast transduction: The significance of Whittle's data

A longstanding issue in vision research concerns whether the internal noise involved in contrast transduction is fixed or variable in relation to contrast magnitude. Previous attempts to resolve the issue have focused on the analysis of contrast discrimination data, despite the fact that the effects of internal noise on thresholds are necessarily compounded by the shape of the underlying transducer function. An alternative approach is to compare data obtained from a particular class of scaling experiment - one based on a comparison of perceived contrast differences - with data from discrimination experiments gathered across the full range of contrast. Data from two studies by the late Paul Whittle provide the basis for such an analysis, pointing to the conclusion that contrast internal noise is fixed not variable.

Monocular perceptual learning of contrast detection facilitates binocular combination in adults with anisometropic amblyopia

Perceptual learning in contrast detection improves monocular visual function in adults with anisometropic amblyopia; however, its effect on binocular combination remains unknown. Given that the amblyopic visual system suffers from pronounced binocular functional loss, it is important to address how the amblyopic visual system responds to such training strategies under binocular viewing conditions. Anisometropic amblyopes (n = 13) were asked to complete two psychophysical supra-threshold binocular summation tasks: (1) binocular phase combination and (2) dichoptic global motion coherence before and after monocular training to investigate this question. We showed that these participants benefited from monocular training in terms of binocular combination. More importantly, the improvements observed with the area under log CSF (AULCSF) were found to be correlated with the improvements in binocular phase combination.

A case for human systems neuroscience

Can the human brain itself serve as a model for a systems neuroscience approach to understanding the human brain? After all, how the brain is able to create the richness and complexity of human behavior is still largely mysterious. What better choice to study that complexity than to study it in humans? However, measurements of brain activity typically need to be made non-invasively which puts severe constraints on what can be learned about the internal workings of the brain. Our approach has been to use a combination of psychophysics in which we can use human behavioral flexibility to make quantitative measurements of behavior and link those through computational models to measurements of cortical activity through magnetic resonance imaging. In particular, we have tested various computational hypotheses about what neural mechanisms could account for behavioral enhancement with spatial attention (Pestilli et al., 2011). Resting both on quantitative measurements and considerations of what is known through animal models, we concluded that weighting of sensory signals by the magnitude of their response is a neural mechanism for efficient selection of sensory signals and consequent improvements in behavioral performance with attention. While animal models have many technical advantages over studying the brain in humans, we believe that human systems neuroscience should endeavor to validate, replicate and extend basic knowledge learned from animal model systems and thus form a bridge to understanding how the brain creates the complex and rich cognitive capacities of humans.

Sensitivity to gaze-contingent contrast increments in naturalistic movies: An exploratory report and model comparison

Sensitivity to luminance contrast is a prerequisite for all but the simplest visual systems. To examine contrast increment detection performance in a way that approximates the natural environmental input of the human visual system, we presented contrast increments gaze-contingently within naturalistic video freely viewed by observers. A band-limited contrast increment was applied to a local region of the video relative to the observer's current gaze point, and the observer made a forced-choice response to the location of the target (≈25,000 trials across five observers). We present exploratory analyses showing that performance improved as a function of the magnitude of the increment and depended on the direction of eye movements relative to the target location, the timing of eye movements relative to target presentation, and the spatiotemporal image structure at the target location. Contrast discrimination performance can be modeled by assuming that the underlying contrast response is an accelerating nonlinearity (arising from a nonlinear transducer or gain control). We implemented one such model and examined the posterior over model parameters, estimated using Markov-chain Monte Carlo methods. The parameters were poorly constrained by our data; parameters constrained using strong priors taken from previous research showed poor cross-validated prediction performance. Atheoretical logistic regression models were better constrained and provided similar prediction performance to the nonlinear transducer model. Finally, we explored the properties of an extended logistic regression that incorporates both eye movement and image content features. Models of contrast transduction may be better constrained by incorporating data from both artificial and natural contrast perception settings.

Broad bandwidth of perceptual learning in second-order contrast modulation detection

Comparing characteristics of learning in first- and second-order systems might inform us about different neural plasticity in the two systems. In the current study, we aim to determine the properties of perceptual learning in second-order contrast modulation detection in normal adults. We trained nine observers to detect second-order gratings at an envelope modulation spatial frequency of 8 cycles/° with their nondominant eyes. We found that, although training generated the largest improvements around the trained frequency, contrast sensitivity over a broad range of spatial frequencies also improved, with a 4.09-octave bandwidth of perceptual learning, exhibiting specificity to the trained spatial frequency as well as a relatively large degree of generalization. The improvements in the modulation sensitivity function (MSF) were not significantly different between the trained and untrained eyes. Furthermore, training did not significantly change subjects' ability in detecting first-order gratings. Our results suggest that perceptual learning in second-order detection might occur at the postchannel level in binocular neurons, possibly through reducing the internal noise of the visual system.

The duration of uncertain times: audiovisual information about intervals is integrated in a statistically optimal fashion

Often multisensory information is integrated in a statistically optimal fashion where each sensory source is weighted according to its precision. This integration scheme isstatistically optimal because it theoretically results in unbiased perceptual estimates with the highest precisionpossible.There is a current lack of consensus about how the nervous system processes multiple sensory cues to elapsed time.In order to shed light upon this, we adopt a computational approach to pinpoint the integration strategy underlying duration estimationof audio/visual stimuli. One of the assumptions of our computational approach is that the multisensory signals redundantly specify the same stimulus property. Our results clearly show that despite claims to the contrary, perceived duration is the result of an optimal weighting process, similar to that adopted for estimates of space. That is, participants weight the audio and visual information to arrive at the most precise, single duration estimate possible. The work also disentangles how different integration strategies - i.e. consideringthe time of onset/offset ofsignals - might alter the final estimate. As such we provide the first concrete evidence of an optimal integration strategy in human duration estimates.

Four theorems on the psychometric function

In a 2-alternative forced-choice (2AFC) discrimination task, observers choose which of two stimuli has the higher value. The psychometric function for this task gives the probability of a correct response for a given stimulus difference, . This paper proves four theorems about the psychometric function. Assuming the observer applies a transducer and adds noise, Theorem 1 derives a convenient general expression for the psychometric function. Discrimination data are often fitted with a Weibull function. Theorem 2 proves that the Weibull “slope” parameter, , can be approximated by , where is the of the Weibull function that fits best to the cumulative noise distribution, and depends on the transducer. We derive general expressions for and , from which we derive expressions for specific cases. One case that follows naturally from our general analysis is Pelli's finding that, when , . We also consider two limiting cases. Theorem 3 proves that, as sensitivity improves, 2AFC performance will usually approach that for a linear transducer, whatever the actual transducer; we show that this does not apply at signal levels where the transducer gradient is zero, which explains why it does not apply to contrast detection. Theorem 4 proves that, when the exponent of a power-function transducer approaches zero, 2AFC performance approaches that of a logarithmic transducer. We show that the power-function exponents of 0.4–0.5 fitted to suprathreshold contrast discrimination data are close enough to zero for the fitted psychometric function to be practically indistinguishable from that of a log transducer. Finally, Weibull reflects the shape of the noise distribution, and we used our results to assess the recent claim that internal noise has higher kurtosis than a Gaussian. Our analysis of for contrast discrimination suggests that, if internal noise is stimulus-independent, it has lower kurtosis than a Gaussian.

PowerPoint(®) Presentation Flaws and Failures: A Psychological Analysis

Electronic slideshow presentations are often faulted anecdotally, but little empirical work has documented their faults. In Study 1 we found that eight psychological principles are often violated in PowerPoint^® slideshows, and are violated to similar extents across different fields – for example, academic research slideshows generally were no better or worse than business slideshows. In Study 2 we found that respondents reported having noticed, and having been annoyed by, specific problems in presentations arising from violations of particular psychological principles. Finally, in Study 3 we showed that observers are not highly accurate in recognizing when particular slides violated a specific psychological rule. Furthermore, even when they correctly identified the violation, they often could not explain the nature of the problem. In sum, the psychological foundations for effective slideshow presentation design are neither obvious nor necessarily intuitive, and presentation designers in all fields, from education to business to government, could benefit from explicit instruction in relevant aspects of psychology.

Wohlgemuth was right: distracting attention from the adapting stimulus does not decrease the motion after-effect

We determined whether distracting the observer's attention from an adapting stimulus could decrease the motion after-effect. Unlike previous studies we used a relatively bias-free 2AFC procedure to measure the strength of adaptation. The strength of motion adaptation was measured by the effects of a moving grating on the contrast discrimination (T vs. C) function for gratings moving in the same or opposite direction. As in previous reports, the effect of adaptation was to move the T vs. C function upwards and rightwards, consistent with an increase in the C50 (semi-saturation) response in the transduction function of the neural mechanism underlying the discrimination. On the other hand, manipulating the attentional load of a distracting task during adaptation had no consistent effect on contrast discrimination, including the absolute detection threshold. It is suggested that previous reported effects of attentional load on adaptation may have depended on response bias, rather than changes in sensitivity.

The absolute threshold of cone vision

We report measurements of the absolute threshold of cone vision, which has been previously underestimated due to suboptimal conditions or overly strict subjective response criteria. We avoided these limitations by using optimized stimuli and experimental conditions while having subjects respond within a rating scale framework. Small (1' fwhm), brief (34 ms), monochromatic (550 nm) stimuli were foveally presented at multiple intensities in dark-adapted retina for 5 subjects. For comparison, 4 subjects underwent similar testing with rod-optimized stimuli. Cone absolute threshold, that is, the minimum light energy for which subjects were just able to detect a visual stimulus with any response criterion, was 203 ± 38 photons at the cornea, ~0.47 log unit lower than previously reported. Two-alternative forced-choice measurements in a subset of subjects yielded consistent results. Cone thresholds were less responsive to criterion changes than rod thresholds, suggesting a limit to the stimulus information recoverable from the cone mosaic in addition to the limit imposed by Poisson noise. Results were consistent with expectations for detection in the face of stimulus uncertainty. We discuss implications of these findings for modeling the first stages of human cone vision and interpreting psychophysical data acquired with adaptive optics at the spatial scale of the receptor mosaic.

The effect of aging on the spatial frequency selectivity of the human visual system

Changes in the physiological properties of senescent V1 neurons suggest that the mechanisms encoding spatial frequency in primate cortex may become more broadly tuned in old age (Zhang et al., European Journal of Neuroscience, 2008, 28, 201-207). We examined this possibility in two psychophysical experiments that used masking to estimate the bandwidth of spatial frequency-selective mechanisms in younger (age approximately 22years) and older (age approximately 65years) human adults. Contrary to predictions from physiological studies, in both experiments, the spatial frequency selectivity of masking was essentially identical in younger and older subjects.

Stochastic model for detection of signals in noise

Fifty years ago Birdsall, Tanner, and colleagues made rapid progress in developing signal detection theory into a powerful psychophysical tool. One of their major insights was the utility of adding external noise to the signals of interest. These methods have been enhanced in recent years by the addition of multipass and classification-image methods for opening up the black box. There remain a number of as yet unresolved issues. In particular, Birdsall developed a theorem that large amounts of external input noise can linearize nonlinear systems, and Tanner conjectured, with mathematical backup, that what had been previously thought of as a nonlinear system could actually be a linear system with uncertainty. Recent findings, both experimental and theoretical, have validated Birdsall's theorem and Tanner's conjecture. However, there have also been experimental and theoretical findings with the opposite outcome. In this paper we present new data and simulations in an attempt to sort out these issues. Our simulations and experiments plus data from others show that Birdsall's theorem is quite robust. We argue that uncertainty can serve as an explanation for violations of Birdsall's linearization by noise and also for reports of stochastic resonance. In addition, we modify present models to better handle detection of signals with both noise and pedestal backgrounds.

A common contrast pooling rule for suppression within and between the eyes

Recent work has revealed multiple pathways for cross-orientation suppression in cat and human vision. In particular, ipsiocular and interocular pathways appear to assert their influence before binocular summation in human but have different (1) spatial tuning, (2) temporal dependencies, and (3) adaptation after-effects. Here we use mask components that fall outside the excitatory passband of the detecting mechanism to investigate the rules for pooling multiple mask components within these pathways. We measured psychophysical contrast masking functions for vertical 1 cycle/deg sine-wave gratings in the presence of left or right oblique (±45 deg) 3 cycles/deg mask gratings with contrast C%, or a plaid made from their sum, where each component (i) had contrast 0.5Ci%. Masks and targets were presented to two eyes (binocular), one eye (monoptic), or different eyes (dichoptic). Binocular-masking functions superimposed when plotted against C, but in the monoptic and dichoptic conditions, the grating produced slightly more suppression than the plaid when Ci ≥ 16%. We tested contrast gain control models involving two types of contrast combination on the denominator: (1) spatial pooling of the mask after a local nonlinearity (to calculate either root mean square contrast or energy) and (2) “linear suppression” (Holmes & Meese, 2004, Journal of Vision4, 1080–1089), involving the linear sum of the mask component contrasts. Monoptic and dichoptic masking were typically better fit by the spatial pooling models, but binocular masking was not: it demanded strict linear summation of the Michelson contrast across mask orientation. Another scheme, in which suppressive pooling followed compressive contrast responses to the mask components (e.g., oriented cortical cells), was ruled out by all of our data. We conclude that the different processes that underlie monoptic and dichoptic masking use the same type of contrast pooling within their respective suppressive fields, but the effects do not sum to predict the binocular case.

Crowding--an essential bottleneck for object recognition: a mini-review

Crowding, generally defined as the deleterious influence of nearby contours on visual discrimination, is ubiquitous in spatial vision. Crowding impairs the ability to recognize objects in clutter. It has been extensively studied over the last 80 years or so, and much of the renewed interest is the hope that studying crowding may lead to a better understanding of the processes involved in object recognition. Crowding also has important clinical implications for patients with macular degeneration, amblyopia and dyslexia. There is no shortage of theories for crowding-from low-level receptive field models to high-level attention. The current picture is that crowding represents an essential bottleneck for object perception, impairing object perception in peripheral, amblyopic and possibly developing vision. Crowding is neither masking nor surround suppression. We can localize crowding to the cortex, perhaps as early as V1; however, there is a growing consensus for a two-stage model of crowding in which the first stage involves the detection of simple features (perhaps in V1), and a second stage is required for the integration or interpretation of the features as an object beyond V1. There is evidence for top-down effects in crowding, but the role of attention in this process remains unclear. The strong effect of learning in shrinking the spatial extent of crowding places strong constraints on possible models for crowding and for object recognition. The goal of this review is to try to provide a broad, balanced and succinct review that organizes and summarizes the diverse and scattered studies of crowding, and also helps to explain it to the non-specialist. A full understanding of crowding may allow us to understand this bottleneck to object recognition and the rules that govern the integration of features into objects.

Perceptual learning in monocular pattern masking: experiments and explanations by the twin summation gain control model of contrast processing

We investigated practice effects on contrast thresholds for target patterns. Results showed that practice decreased contrast thresholds when targets were presented on maskers. Thresholds tended to decrease more at the higher end of the masker contrast range. At least partially, learning transferred to stimuli of the untrained phase. We simulated changes in threshold versus contrast functions using a contrast-processing model and then fit the model to pre- and posttraining data. The simulation results and model fit suggest that learning in pattern masking can be accounted for by changes in nonlinear transducer functions for divisive inhibitory signals.

The response of the amblyopic visual system to noise

Visual perception is limited by both the strength of the neural signals, and by the noise in the visual nervous system. Here we use one-dimensional white noise as input, to study the response of amblyopic visual system. We measured the thresholds for detection and discrimination of noise contrast. Using an N-pass reverse correlation technique, we derived classification images and estimated response consistency. Our results provide the first report of the sensitivity of the amblyopic visual system to white noise. We show that amblyopes have markedly reduced sensitivity for detecting noise, particularly at high spatial frequencies, and much less loss for discriminating suprathreshold noise contrast. Compensating for the detection loss almost (but not quite) equates performance of the amblyopic and normal visual system. The classification images suggest that the amblyopic visual system contains adjustable channels for noise, similar to those found in normal vision, but "tuned" to slightly lower spatial frequencies than in normal observers. Our N-pass results show that the predominant factor limiting performance in our task in both normal and amblyopic vision is internal random multiplicative noise. For the detection of white noise the raised thresholds of the amblyopic visual system can be attributed primarily to extra additive noise. However, for the discrimination of suprathreshold white noise contrast, there is surprisingly little additional deficit, after accounting for the visibility of the noise.

Remote facilitation in the Fourier domain

To explore spatial interactions between visual mechanisms in the Fourier domain we measured detection thresholds for vertical and horizontal sine-wave gratings (4.4 deg diameter) over a range of spatial frequencies (0.5-23 c/deg) in the presence of grating and plaid masks with component contrasts of 8%, orientations of +/-45 degrees and a spatial frequency of 1c/deg. The mask suppressed the target grating over a range of +/-1 octave, and the plaid produced more suppression than the grating, consistent with summation of mask components in a broadly tuned contrast gain pool. At greater differences in spatial frequency ( approximately 3 octaves), the plaid and grating masks both produced about 3 dB of facilitation (they reduced detection thresholds by a factor of about square root 2). At yet further distances ( approximately 4 octaves) the masks had no effect. The facilitation cannot be attributed to a reduction of uncertainty by the mask because (a) it occurs for mask components that have very different spatial frequencies and orientations from the test and (b) the large stimulus size and central fixation point mean there was no spatial uncertainty that could be reduced. We suggest the results are due to long-range sensory interactions (in the Fourier domain) between mask and test-channels. The effects could be due to either direct facilitation or disinhibition.

Stochastic re-calibration: contextual effects on perceived tilt

The human visual system exaggerates the difference between the tilts of adjacent lines or grating patches. In addition to this tilt illusion, we found that oblique flanks reduced acuity for small changes of tilt in the centre of the visual field. However, no flanks--regardless of their tilts--decreased sensitivity to contrast. Thus, the foveal tilt illusion should not be attributed to orientation-selective lateral inhibition. Nor is it similar to conventional crowding, which typically does not impair letter recognition in the fovea. Our observers behaved as though the reference orientation (horizontal) had a small tilt in the direction of the flanks. We suggest that the extent of this re-calibration varies randomly over trials, and we demonstrate that this stochastic re-calibration can explain flank-induced acuity loss in the fovea.

Predicting the motion after-effect from sensitivity loss

The widely accepted disinhibition theory of the motion after-effect (MAE) proposes that the balance point of an opponent mechanism is changed by directional adaptation. To see if the post-adaptation balance point could be predicted from contrast adaptation, we measured threshold-vs-contrast (i.e., T-vs-C or dipper) functions, before and after adaptation to moving gratings. For test stimuli moving in the same direction, adaptation shifted the point of maximum facilitation (i.e., the dip) upwards and rightwards. For tests moving in the opposite direction, adaptation produced a similar, but smaller, shift. These shifts are consistent with a change in divisive gain control. They are also consistent with subtractive inhibition followed by half-wave rectification. We attempted to use transducer functions derived from these data to predict the strength of the MAE. When combined, gratings moving in the adapted and opposite directions appeared perfectly balanced (i.e., counterphasing) when the latter was given approximately 2% more contrast than was predicted on the basis of the derived transducers. This small under-prediction may be indicative of sensory recalibration. Finally, we found that adaptation did not alter the fact that low-contrast stimuli could be detected and their direction identified with similar accuracy. We conclude that both static and dynamic forms of MAE are primarily caused by a decreased sensitivity in directionally tuned mechanisms, as proposed by the disinhibition theory.

Only two phase mechanisms, ±cosine, in human vision

We evaluated the proposal that there exist detectors of the following four cardinal phases in human vision: +cosine, -cosine, +sine, and -sine. First, we assessed whether there was evidence that these cardinal phases were processed by independent 'labeled lines,' using a discrimination at detection threshold paradigm. Second, we assessed whether suprathreshold phase discrimination was best at phases intermediate between these cardinal values. Third, we tried to replicate previous evidence showing that an absence of facilitation occurs only between cosine pedestals and sine tests (or vice-versa). In all three experimental approaches we found no compelling evidence for four cardinal phase groupings. We did however find evidence for independent detectors for pure increments and decrements (+/-cosine). We suggest that phase discrimination, whether at threshold or suprathreshold, is mediated by mechanisms that encode the relative positions and contrasts of local increments and decrements within the stimulus.

Uncertainty and invariance in the human visual cortex

The way in which input noise perturbs the behavior of a system depends on the internal processing structure of the system. In visual psychophysics, there is a long tradition of using external noise methods (i.e., adding noise to visual stimuli) as tools for system identification. Here, we demonstrate that external noise affects processing of visual scenes at different cortical areas along the human ventral visual pathway, from retinotopic regions to higher occipitotemporal areas implicated in visual shape processing. We found that when the contrast of the stimulus was held constant, the further away from the retinal input a cortical area was the more its activity, as measured with functional magnetic resonance imaging (fMRI), depended on the signal-to-noise ratio (SNR) of the visual stimulus. A similar pattern of results was observed when trials with correct and incorrect responses were analyzed separately. We interpret these findings by extending signal detection theory to fMRI data analysis. This approach reveals the sequential ordering of decision stages in the cortex by exploiting the relation between fMRI response and stimulus SNR. In particular, our findings provide novel evidence that occipitotemporal areas in the ventral visual pathway form a cascade of decision stages with increasing degree of signal uncertainty and feature invariance.

Classification images with uncertainty

Classification image and other similar noise-driven linear methods have found increasingly wider applications in revealing psychophysical receptive field structures or perceptual templates. These techniques are relatively easy to deploy, and the results are simple to interpret. However, being a linear technique, the utility of the classification-image method is believed to be limited. Uncertainty about the target stimuli on the part of an observer will result in a classification image that is the superposition of all possible templates for all the possible signals. In the context of a well-established uncertainty model, which pools the outputs of a large set of linear frontends with a max operator, we show analytically, in simulations, and with human experiments that the effect of intrinsic uncertainty can be limited or even eliminated by presenting a signal at a relatively high contrast in a classification-image experiment. We further argue that the subimages from different stimulus-response categories should not be combined, as is conventionally done. We show that when the signal contrast is high, the subimages from the error trials contain a clear high-contrast image that is negatively correlated with the perceptual template associated with the presented signal, relatively unaffected by uncertainty. The subimages also contain a "haze" that is of a much lower contrast and is positively correlated with the superposition of all the templates associated with the erroneous response. In the case of spatial uncertainty, we show that the spatial extent of the uncertainty can be estimated from the classification subimages. We link intrinsic uncertainty to invariance and suggest that this signal-clamped classification-image method will find general applications in uncovering the underlying representations of high-level neural and psychophysical mechanisms.

Accuracy of identification of grating contrast by human observers: Bayesian models of V1 contrast processing show correspondence between discrimination and identification performance

This paper presents the results of a contrast identification study, where accuracy in identification is quantified as mutual information between stimulus contrast and observer's response. The stimulus was a set of 2-8 gratings, spanning the range of visible contrasts. Gratings from the set were presented individually for 500 ms, and the observer had to respond by giving the number label corresponding to the contrast of the grating presented. Mutual information increased with set size up to a maximum of around 2.35 bits, i.e., only 5 clearly identifiable contrasts. Set sizes greater than 5 showed a plateau or decline in performance. These data were well fit by Bayesian models of V1 contrast coding, with the parameters obtained by fitting the contrast discrimination results of Chirimuuta and Tolhurst [Chirimuuta, M., & Tolhurst, D. J. (2005). Does a Bayesian model of V1 contrast coding offer a neurophysiological account of human contrast discrimination? Vision Research].

What is the signal in noise?

Visual perception is limited by both the strength of the neural signals, and by the noise in the visual nervous system; however, little is known about what aspects of the input noise the human visual system is sensitive to, i.e., what is the signal in noise? In order to investigate this question we asked observers to discriminate differences in the strength of one-dimensional white noise. We measured their response consistency and classification images and compared the results with an ideal energy detector. Our results and modelling show that discrimination of noise is limited by the observer's template (i.e., the weighted combination of energy in each stimulus component) plus higher order nonlinearities (systematic noise), and by sources of random internal noise. We found that systematic noise is present only near detection threshold. Surprisingly, we found that the human template is "adaptive"-- its shape depends on the spatial frequency band of the noise-suggesting that sensitivity to spatial noise is not simply determined via passive filtering.

Dual nonlinearities regulate contrast sensitivity in pattern discrimination tasks

Many current psychophysical models propose that visual processing in cortex is hierarchical, with nonlinearities sandwiched between linear stages of processing. In earlier publications, we proposed a model of this type to account for masking effects found with spatial frequency and orientation discriminations. Our model includes two nonlinear mechanisms that regulate contrast sensitivity in early cortical mechanisms. The first is a local within-pathway nonlinearity that accelerates at low contrasts but is compressive at high. The second is a pooled nonlinear gain control process that operates over a broad range of neurons with different tuning characteristics. Here, we test predictions of the model for spatial frequency discriminations. The model predicts that at low contrasts, adding a grating mask oriented parallel to test gratings will improve discrimination performance via operation of the within-pathway nonlinearity, analogous to the "dipper effect" found with contrast discriminations. Adding an orthogonally oriented mask is predicted to have no effect at low contrasts, where pooled gain control processes contribute little to performance. At high contrasts, the model predicts that performance will asymptote and become independent of contrast with either parallel or orthogonal masks. The results confirm model predictions.

Separating the effects of response nonlinearity and internal noise psychophysically

A psychophysical method is proposed to separate the contrast dependence of internal response and its noise. The resulting contrast relationships represent a signature of the visual processing stage that limits the human observer's performance. The method was applied to contrast discrimination for sustained and transient Gabor patches with a 3 cycle/deg spatial carrier. For both stimulus types the predominant noise was found to be multiplicative with a power exponent of 0.76-0.85 and the source of this noise preceded by an accelerating signal transducer with a power of 2-2.7. These exponents combine to account for the classic compressive power of about 0.4 for the signal-to-noise ratio in contrast discrimination. The estimated transducer acceleration suggests that there is a direct computation of contrast energy in the visual cortex.

Measuring, estimating, and understanding the psychometric function: a commentary

The psychometric function, relating the subject's response to the physical stimulus, is fundamental to psychophysics. This paper examines various psychometric function topics, many inspired by this special symposium issue of Perception & Psychophysics: What are the relative merits of objective yes/no versus forced choice tasks (including threshold variance)? What are the relative merits of adaptive versus constant stimuli methods? What are the relative merits of likelihood versus up-down staircase adaptive methods? Is 2AFC free of substantial bias? Is there no efficient adaptive method for objective yes/no tasks? Should adaptive methods aim for 90% correct? Can adding more responses to forced choice and objective yes/no tasks reduce the threshold variance? What is the best way to deal with lapses? How is the Weibull function intimately related to the d' function? What causes bias in the likelihood goodness-of-fit? What causes bias in slope estimates from adaptive methods? How good are nonparametric methods for estimating psychometric function parameters? Of what value is the psychometric function slope? How are various psychometric functions related to each other? The resolution of many of these issues is surprising.

Signal detection theory in the 2AFC paradigm: attention, channel uncertainty and probability summation

Neural implementation of classical High-Threshold Theory reveals fundamental flaws in its applicability to realistic neural systems and to the two-alternative forced-choice (2AFC) paradigm. For 2AFC, Signal Detection Theory provides a basis for accurate analysis of the observer's attentional strategy and effective degree of probability summation over attended neural channels. The resulting theory provides substantially different predictions from those of previous approximation analyses. In additive noise, attentional probability summation depends on the attentional model assumed. (1) For an ideal attentional strategy in additive noise, summation proceeds at a diminishing rate from an initial level of fourth-root summation for the first few channels. The maximum improvement asymptotes to about a factor of 4 by a million channels. (2) For a fixed attention field in additive noise, detection is highly inefficient at first and approximates fourth-root summation through the summation range. (3) In physiologically plausible root-multiplicative noise, on the other hand, attentional probability summation mimics a linear improvement in sensitivity up to about ten channels, approaching a factor of 1000 by a million channels. (4) Some noise sources, such as noise from eye movements, are fully multiplicative and would prevent threshold determination within their range of effectiveness. Such results may require reappraisal of previous interpretations of detection behavior in the 2AFC paradigm.

Revisiting spatial vision: toward a unifying model

We report contrast detection, contrast increment, contrast masking, orientation discrimination, and spatial frequency discrimination thresholds for spatially localized stimuli at 4 degrees of eccentricity. Our stimulus geometry emphasizes interactions among overlapping visual filters and differs from that used in previous threshold measurements, which also admits interactions among distant filters. We quantitatively account for all measurements by simulating a small population of overlapping visual filters interacting through divisive inhibition. We depart from previous models of this kind in the parameters of divisive inhibition and in using a statistically efficient decision stage based on Fisher information. The success of this unified account suggests that, contrary to Bowne [Vision Res. 30, 449 (1990)], spatial vision thresholds reflect a single level of processing, perhaps as early as primary visual cortex.

Facilitation from collinear flanks is cancelled by non-collinear flanks

Detection of a central Gabor pattern is facilitated by the presence of collinear flanking patterns. We find that this facilitation is greatly reduced when the collinear flanks are combined with non-collinear flanks to form a coherent surround. These results are unlikely to be explained by mechanisms that merely transduce local contrast in a nonlinear fashion. A model wherein the outputs of such mechanisms are combined anisotropically provides a better account for these results.

Pattern detection in the presence of maskers that differ in spatial phase and temporal offset: threshold measurements and a model

Four experiments are described in which brief Gabor patterns are detected in the presence of full-field gratings or Gabor patterns that are superimposed in space, but vary in spatial phase and temporal offset (SOA). E1: Threshold versus masker contrast (TvC) functions were determined for relative phases of 0, 90, 180 and 270 degrees at SOA = 0. For 0 degree relative phase, TvC functions decrease (facilitation) and then increase (masking) as contrast increases. For 90 degrees, there is little or no facilitation and thresholds increase with masker contrast. For 180 degrees, the form of the TvC function varies with observer and conditions. E2: Like E1, except that maskers are Gabor patterns. TvC functions are similar in form to those for full-field maskers, but there is less masking. E3: Forward masking. TvC functions were determined for relative phases of 0, 90, and 180 degrees at SOA = -33 ms. The forms of the TvC functions for 0 and 180 degrees are reversed relative to those at SOA = 0. E4: TvP (threshold versus phase) functions were determined for SOA's of -100, -67, -33, 0 and 33 ms at a constant masker contrast of 0.063. Masking occurs at all relative phases. For simultaneous and backward masking, the threshold is minimum for a relative phase of 0 and maximum at 180 degrees. For forward masking, the form of the function is inverted. A model of pattern masking and facilitation (Foley, J. M. (1994a) Journal of the Optical Society of America A, 11, 1710-1719) is extended to account for masker phase and SOA effects. The model assumes four mechanisms tuned to phases 90 degrees apart, and divisively inhibited by stimuli of all phases. Performance depends on the detection strategy of the observer.

Relationship between acuity for gratings and for tumbling-E letters in peripheral vision

Earlier studies have reported that grating resolution is sampling-limited in peripheral vision but that letter acuity is generally poorer than grating acuity. These results suggest that peripheral resolution of objects with rich Fourier spectra may be limited by some factor other than neural sampling. To examine this suggestion we formulated and tested the hypothesis that letter acuity in the periphery is sampling-limited, just as it is for extended and truncated gratings. We tested this hypothesis with improved methodology to avoid the confounding factors of target similarity, alphabet size, individual variation, peripheral refractive error, and stimulus size. Acuity was measured for an orientation-discrimination task (horizontal versus vertical) for a three-bar resolution target and for a block-E letter in which all strokes have the same length. We confirmed previous reports in the literature that acuity for these targets is worse than for extended sinusoidal gratings. To account for these results quantitatively, we used difference-spectrum analysis to identify those frequency components of the targets that might form a basis for performing the visual discrimination task. We find that discrimination performance for the three-bar targets and the block-E letters can be accounted for by a sampling-limited model, provided that the limited number of cycles that are present in the characteristic frequency of the stimulus is taken into account. Quantitative differences in acuity for discriminating other letter pairs (e.g., right versus left letters E or characters with short central strokes) could not be attributed to undersampling of either the characteristic frequency or the frequency of maximum energy in the difference spectrum. These results suggest additional tests of the sampling theory of visual resolution, which are the subject of a companion paper.

Bayesian adaptive estimation of psychometric slope and threshold

We introduce a new Bayesian adaptive method for acquisition of both threshold and slope of the psychometric function. The method updates posterior probabilities in the two-dimensional parameter space of psychometric functions and makes predictions based on the expected mean threshold and slope values. On each trial it sets the stimulus intensity that maximizes the expected information to be gained by completion of that trial. The method was evaluated in computer simulations and in a psychophysical experiment using the two-alternative forced-choice (2AFC) paradigm. Threshold estimation within 2 dB (23%) precision requires less than 30 trials for a typical 2AFC detection task. To get the slope estimate with the same precision takes about 300 trials.

Transducer model produces facilitation from opposite-sign flanks

Small spots, lines and Gabor patterns can be easier to detect when they are superimposed upon similar spots, lines and Gabor patterns. Traditionally, such facilitation has been understood to be a consequence of nonlinear contrast transduction. Facilitation has also been reported to arise from non-overlapping patterns with opposite sign. We point out that this result does not preclude the traditional explanation for superimposed targets. Moreover, we find that facilitation from opposite-sign flanks is weaker than facilitation from same-sign flanks. Simulations with a transducer model produce opposite-sign facilitation.

Characterizing human perceptual inefficiencies with equivalent internal noise

A widely used method for characterizing and comparing inefficiencies in perceptual processes is the method of equivalent internal noise--the amount of random internal noise necessary to produce the degree of inefficiency exhibited by the perceptual system in processing [J. Opt. Soc. Am. 46, 634 (1956)]. One normally estimates the amount of equivalent internal noise by systematically increasing the amount of external noise added to the signal stimulus and observing how threshold--signal stimulus energy required for an observer to maintain a given performance level--depends on the amount of external noise. In a variety of perceptual tasks, a simple noisy linear amplifier model [D. Pelli, Ph.D. dissertation (University of Cambridge, Cambridge, UK 1981)] has been utilized to estimate the equivalent internal noise Ninternal by fitting of the relation between threshold contrast c tau and external noise N(ext) at a single (d') performance level: c tau 2 =(d'/beta)2(N(ext)2 + Ninternal2). This model makes a strong prediction: Independent of observer and external noise contrast, the ratio between two thresholds at each external noise level is equal to the ratio of the two corresponding d' values. To our knowledge, this potential test for the internal consistency of the model had never been examined previously. In this study we estimated threshold ratios between multiple performance levels at various external noise contrasts in two different experiments: Gabor orientation identification, and Gabor detection. We found that, in both identification and detection, the observed threshold ratios between different performance levels departed substantially from the d' ratio predicted by the simple noisy linear amplifier model. An elaborated perceptual template model [Vision Res. 38, 1183 (1998)] with nonlinear transducer functions and multiplicative noise in addition to the additive noise in the simple linear amplifier model leads to a substantially better description of the data and suggests a reinterpretation of earlier results that relied on the simple noisy linear amplifier model. The relationship of our model and method to other recent parallel and independent developments [J. Opt. Soc. Am. A 14, 2406 (1997)] is discussed.

A wavelet visible difference predictor

In this paper, we describe a model of the human visual system (HVS) based on the wavelet transform. This model is largely based on a previously proposed model, but has a number of modifications that make it more amenable to potential integration into a wavelet based image compression scheme. These modifications include the use of a separable wavelet transform instead of the cortex transform, the application of a wavelet contrast sensitivity function (CSF), and a simplified definition of subband contrast that allows one to predict the noise visibility directly from the wavelet coefficients. Initially, we outline the luminance, frequency, and masking sensitivities of the HVS and discuss how these can be incorporated into the wavelet transform. We then outline a number of limitations of the wavelet transform as a model of the HVS, namely the lack of translational invariance and poor orientation sensitivity. In order to investigate the efficacy of this wavelet based model, a wavelet visible difference predictor (WVDP) is described. The WVDP is then used to predict visible differences between an original and compressed (or noisy) image. Results are presented to emphasize the limitations of commonly used measures of image quality and to demonstrate the performance of the WVDP. The paper concludes with suggestions on how the WVDP can be used to determine a visually optimal quantization strategy for wavelet coefficients and produce a quantitative measure of image quality.

Neuronal basis of contrast discrimination

Psychophysical contrast increment thresholds were compared with neuronal responses, inferred from functional magnetic resonance imaging (fMRI) to test the hypothesis that contrast discrimination judgements are limited by neuronal signals in early visual cortical areas. FMRI was used to measure human brain activity as a function of stimulus contrast, in each of several identifiable visual cortical areas. Contrast increment thresholds were measured for the same stimuli across a range of baseline contrasts using a temporal 2AFC paradigm. FMRI responses and psychophysical measurements were compared by assuming that: (1) fMRI responses are proportional to local average neuronal activity; (2) subjects choose the stimulus interval that evoked the greater average neuronal activity; and (3) variability in the observer's psychophysical judgements was due to additive (IID) noise. With these assumptions, FMRI responses in visual areas V1, V2d, V3d and V3A were found to be consistent with the psychophysical judgements, i.e. a contrast increment was detected when the fMRI responses in each of these brain areas increased by a criterion amount. Thus, the pooled activity of large numbers of neurons can reasonably well predict behavioral performance. The data also suggest that contrast gain in early visual cortex depends systematically on spatial frequency.