Spatial properties of envelope-responsive cells in area 17 and 18 neurons of the cat

Estimating receptive fields of simple and complex cells in early visual cortex: A convolutional neural network model with parameterized rectification

Neurons in the primary visual cortex respond selectively to simple features of visual stimuli, such as orientation and spatial frequency. Simple cells, which have phase-sensitive responses, can be modeled by a single receptive field filter in a linear-nonlinear model. However, it is challenging to analyze phase-invariant complex cells, which require more elaborate models having a combination of nonlinear subunits. Estimating parameters of these models is made additionally more difficult by cortical neurons' trial-to-trial response variability. We develop a simple convolutional neural network method to estimate receptive field models for both simple and complex visual cortex cells from their responses to natural images. The model consists of a spatiotemporal filter, a parameterized rectifier unit (PReLU), and a two-dimensional Gaussian "map" of the receptive field envelope. A single model parameter determines the simple vs. complex nature of the receptive field, capturing complex cell responses as a summation of homogeneous subunits, and collapsing to a linear-nonlinear model for simple type cells. The convolutional method predicts simple and complex cell responses to natural image stimuli as well as grating tuning curves. The fitted models yield a continuum of values for the PReLU parameter across the sampled neurons, showing that the simple/complex nature of cells can vary in a continuous manner. We demonstrate that complex-like cells respond less reliably than simple-like cells. However, compensation for this unreliability with noise ceiling analysis reveals predictive performance for complex cells proportionately closer to that for simple cells. Most spatial receptive field structures are well fit by Gabor functions, whose parameters confirm well-known properties of cat A17/18 receptive fields.

Second-order texture gratings produce overestimation of height in depictions of rectangles and steps

The horizontal-vertical illusion (HVI) has been proposed as a method to increase the perceived height of steps, increase toe clearance and prevent falls. High contrast vertical stripes are placed on the step riser abutting a horizontal edge-highlighter creating 'T' junctions which are thought to promote the illusion. Various configurations of the HVI were tested including luminance gratings (L) and second-order modulations of contrast (CM), spatial frequency (FM) and orientation (OM). Observers were asked to compare the apparent height of gratings with that of either filled, unmodulated rectangles or unfilled rectangles. Rectangles were presented alone or as part of a step with a highlighter. In some conditions highlighters matched the properties of the grating; in others or not. In one critical experiment, the HVI was compared for steps with highlighters that were separated from the riser by a thin line and those where the risers and highlighters were continuous. All gratings except FM appeared taller when presented in the step configuration with a continuous, matching highlighter. This effect was greatly reduced when a thin line separated the grating from the highlighter and abolished for mis-matched highlighters and risers. In the rectangle conditions, all cues appeared taller than blank rectangles and L and CM appeared taller than filled-unmodulated rectangles. In conclusion, second-order cues may be useful for inducing the HVI onto steps. However, the ability of vertical stripes and edge-highlighters to accentuate perceived step height may be due to aggregation of the highlighter into the grating rather than the normal horizontal-vertical illusion.

Perception and decision mechanisms involved in average estimation of spatiotemporal ensembles

A number of studies on texture and ensemble perception have shown that humans can immediately estimate the average of spatially distributed visual information. The present study characterized mechanisms involved in estimating averages for information distributed over both space and time. Observers viewed a rapid sequence of texture patterns in which elements' orientation were determined by dynamic Gaussian noise with variable spatial and temporal standard deviations (SDs). We found that discrimination thresholds increased beyond a certain spatial SD if temporal SD was small, but if temporal SD was large, thresholds remained nearly constant regardless of spatial SD. These data are at odds with predictions that threshold is uniquely determined by spatiotemporal SD. Moreover, a reverse correlation analysis revealed that observers judged the spatiotemporal average orientation largely depending on the spatial average orientation over the last few frames of the texture sequence - a recency effect widely observed in studies of perceptual decision making. Results are consistent with the notion that the visual system rapidly computes spatial ensembles and adaptively accumulates information over time to make a decision on spatiotemporal average. A simple computational model based on this notion successfully replicated observed data.

Seeing through transparent layers

The human visual system is remarkably good at decomposing local and global deformations in the flow of visual information into different perceptual layers, a critical ability for daily tasks such as driving through rain or fog or catching that evasive trout. In these scenarios, changes in the visual information might be due to a deforming object or deformations due to a transparent medium, such as structured glass or water, or a combination of these. How does the visual system use image deformations to make sense of layering due to transparent materials? We used eidolons to investigate equivalence classes for perceptually similar transparent layers. We created a stimulus space for perceptual equivalents of a fiducial scene by systematically varying the local disarray parameters reach and grain. This disarray in eidolon space leads to distinct impressions of transparency, specifically, high reach and grain values vividly resemble water whereas smaller grain values appear diffuse like structured glass. We asked observers to adjust image deformations so that the objects in the scene looked like they were seen (a) under water, (b) behind haze, or (c) behind structured glass. Observers adjusted image deformation parameters by moving the mouse horizontally (grain) and vertically (reach). For two conditions, water and glass, we observed high intraobserver consistency: responses were not random. Responses yielded a concentrated equivalence class for water and structured glass.

Object size determines the spatial spread of visual time

A key question for temporal processing research is how the nervous system extracts event duration, despite a notable lack of neural structures dedicated to duration encoding. This is in stark contrast with the orderly arrangement of neurons tasked with spatial processing. In this study, we examine the linkage between the spatial and temporal domains. We use sensory adaptation techniques to generate after-effects where perceived duration is either compressed or expanded in the opposite direction to the adapting stimulus' duration. Our results indicate that these after-effects are broadly tuned, extending over an area approximately five times the size of the stimulus. This region is directly related to the size of the adapting stimulus-the larger the adapting stimulus the greater the spatial spread of the after-effect. We construct a simple model to test predictions based on overlapping adapted versus non-adapted neuronal populations and show that our effects cannot be explained by any single, fixed-scale neural filtering. Rather, our effects are best explained by a self-scaled mechanism underpinned by duration selective neurons that also pool spatial information across earlier stages of visual processing.

Differences in spatial and temporal frequency interactions between central and peripheral parts of the feline area 18

The visual system demonstrates significant differences in information processing abilities between the central and peripheral parts of the visual field. Optical imaging based on intrinsic signals was used to investigate the difference in stimulus spatial and temporal frequency interactions related to receptive field eccentricity in the cat area 18. Changing either the spatial or the temporal frequency of grating stimuli had a significant impact on responses in the cortical areas corresponding to the centre of the visual field and more peripheral parts at 10 degrees eccentricity. The cortical region corresponding to the centre of the gaze was tuned to 0.4 cycles per degree (c/deg) for spatial frequency and 2 Hz for temporal frequency. In contrast, the cortical region corresponding to the periphery of the visual field was tuned to a lower spatial frequency of 0.15 c/deg and a higher temporal frequency of 4 Hz. Interestingly, when we simultaneously changed both the spatial frequency and the temporal frequency of the grating stimuli, the responses were significantly different from those estimated with an assumption of independence between the spatial and temporal frequency in the cortical region corresponding to the periphery of the visual field. However, in the cortical area corresponding to the centre of the gaze, spatial frequency showed significant independence from temporal frequency. These properties support the notion of relative specialization of visual information processing for peripheral representations in cortical areas.

Encoding and estimation of first- and second-order binocular disparity in natural images

•

First- and second-order responses to natural binocular images are correlated.

•

Second-order mechanisms can improve the accuracy of disparity estimation.

•

Second-order mechanisms can extend the depth range of binocular stereopsis.

The first stage of processing of binocular information in the visual cortex is performed by mechanisms that are bandpass-tuned for spatial frequency and orientation. Psychophysical and physiological evidence have also demonstrated the existence of second-order mechanisms in binocular processing, which can encode disparities that are not directly accessible to first-order mechanisms. We compared the responses of first- and second-order binocular filters to natural images. We found that the responses of the second-order mechanisms are to some extent correlated with the responses of the first-order mechanisms, and that they can contribute to increasing both the accuracy, and depth range, of binocular stereopsis.

Delayed signal transmission in area 17, area 18 and the posteromedial lateral suprasylvian area of aged cats

To investigate the effect of senescence on signal transmission, we have compared the visual response latency and spontaneous activity of cells in the lateral geniculate nucleus (LGN), area 17, area 18 and posteromedial lateral suprasylvian area (PMLS) of young and old cats. We found that LGN cells in old cats exhibit largely normal visual response latency. In contrast, all the other three areas exhibited significant aging-related delays in the visual response latency. On average, PMLS showed most pronounced delays among these three areas. Area 18 slowed more than area 17, but this was not significant. The degradation of signal timing in the visual cortex might provide insight into neuronal response mechanism underlying perception slowing during aging.

The efficiency of second order orientation coherence detection

Neurons in early visual cortex respond to both luminance- (1st order) and contrast-modulated (2nd order) local features in the visual field. In later extra-striate areas neurons with larger receptive fields integrate information across the visual field. For example, local luminance-defined features can be integrated into contours and shapes. Evidence for the global integration of features defined by contrast-modulation is less well established. While good performance in some shape tasks has been demonstrated with 2nd order stimuli, the integration of contours fails with 2nd order elements. Recently we developed a global orientation coherence task that is more basic than contour integration, bearing similarity to the well-established global motion coherence task. Similar to our previous 1st order result for this task, we find 2nd order coherence detection to be scale-invariant. There was a small but significant threshold elevation for 2nd order relative to 1st order. We used a noise masking approach to compare the efficiency of orientation integration for the 1st and 2nd order. We find a significant deficit for 2nd order detection at both the local and global level, however the small size of this effect stands in stark contrast against previous results from contour-integration experiments, which are almost impossible with 2nd order stimuli.

Possible functions of contextual modulations and receptive field nonlinearities: pop-out and texture segmentation

When analyzing a visual image, the brain has to achieve several goals quickly. One crucial goal is to rapidly detect parts of the visual scene that might be behaviorally relevant, while another one is to segment the image into objects, to enable an internal representation of the world. Both of these processes can be driven by local variations in any of several image attributes such as luminance, color, and texture. Here, focusing on texture defined by local orientation, we propose that the two processes are mediated by separate mechanisms that function in parallel. More specifically, differences in orientation can cause an object to "pop out" and attract visual attention, if its orientation differs from that of the surrounding objects. Differences in orientation can also signal a boundary between objects and therefore provide useful information for image segmentation. We propose that contextual response modulations in primary visual cortex (V1) are responsible for orientation pop-out, while a different kind of receptive field nonlinearity in secondary visual cortex (V2) is responsible for orientation-based texture segmentation. We review a recent experiment that led us to put forward this hypothesis along with other research literature relevant to this notion.

Perceptual learning of second order cues for layer decomposition

Luminance variations are ambiguous: they can signal changes in surface reflectance or changes in illumination. Layer decomposition-the process of distinguishing between reflectance and illumination changes-is supported by a range of secondary cues including colour and texture. For an illuminated corrugated, textured surface the shading pattern comprises modulations of luminance (first order, LM) and local luminance amplitude (second-order, AM). The phase relationship between these two signals enables layer decomposition, predicts the perception of reflectance and illumination changes, and has been modelled based on early, fast, feed-forward visual processing (Schofield et al., 2010). However, while inexperienced viewers appreciate this scission at long presentation times, they cannot do so for short presentation durations (250 ms). This might suggest the action of slower, higher-level mechanisms. Here we consider how training attenuates this delay, and whether the resultant learning occurs at a perceptual level. We trained observers to discriminate the components of plaid stimuli that mixed in-phase and anti-phase LM/AM signals over a period of 5 days. After training, the strength of the AM signal needed to differentiate the plaid components fell dramatically, indicating learning. We tested for transfer of learning using stimuli with different spatial frequencies, in-plane orientations, and acutely angled plaids. We report that learning transfers only partially when the stimuli are changed, suggesting that benefits accrue from tuning specific mechanisms, rather than general interpretative processes. We suggest that the mechanisms which support layer decomposition using second-order cues are relatively early, and not inherently slow.

Functional organization of envelope-responsive neurons in early visual cortex: organization of carrier tuning properties

It is well established that visual cortex neurons having similar selectivity for orientation, direction of motion, ocular dominance, and other properties of first-order (luminance-defined) stimuli are clustered into a columnar organization. However, the cortical architecture of neuronal responses to second-order (contrast/texture-defined) stimuli is poorly understood. A useful second-order stimulus is a contrast envelope, consisting of a finely detailed pattern (carrier) whose contrast varies on a coarse spatial scale (envelope). In this study, we analyzed the cortical organization of carrier tuning properties of neurons, which responded to contrast-modulated stimuli. We examined whether neurons tuned to similar carrier properties are clustered spatially and whether such spatial clusters are arranged in columns. To address these questions, we recorded single-unit activity, multiunit activity, and local field potentials simultaneously from area 18 of anesthetized cats, using single-channel microelectrodes and multielectrode arrays. Our data showed that neurons tuned to similar carrier spatial frequency are distributed in a highly clustered manner; neurons tuned to similar carrier orientation are also significantly clustered. Neurons along linear arrays perpendicular to the brain surface always exhibited similar optimal carrier spatial frequency, indicating a columnar organization. Multi-pronged tetrode recordings indicated that the diameter of these columns is ≥450 μm. Optimal carrier orientation was also significantly clustered but with finer-grain organization and greater scatter. These results indicate a fine anatomical structure of cortical organization of second-order information processing and suggest that there are probably more maps in cat area 18 than previously believed.

Stimulus-entrained oscillatory activity propagates as waves from area 18 to 17 in cat visual cortex

Previous studies in cat visual cortex reported that area 18 can actively drive neurons in area 17 through cortico-cortical projections. However, the dynamics of such cortico-cortical interaction remains unclear. Here we used multielectrode arrays to examine the spatiotemporal pattern of neuronal activity in cat visual cortex across the 17/18 border. We found that full-field contrast reversal gratings evoked oscillatory wave activity propagating from area 18 to 17. The wave direction was independent of the grating orientation, and could not be accounted for by the spatial distribution of receptive field latencies, suggesting that the waves are largely mediated by intrinsic connections in the cortex. Different from the evoked waves, spontaneous waves propagated along both directions across the 17/18 border. Together, our results suggest that visual stimulation may enhance the flow of information from area 18 to 17.

Motion-defined contour processing in the early visual cortex

From our daily experience, it is very clear that relative motion cues can contribute to correctly identifying object boundaries and perceiving depth. Motion-defined contours are not only generated by the motion of objects in a scene but also by the movement of an observer's head and body (motion parallax). However, the neural mechanism involved in detecting these contours is still unknown. To explore this mechanism, we extracellularly recorded visual responses of area 18 neurons in anesthetized and paralyzed cats. The goal of this study was to determine if motion-defined contours could be detected by neurons that have been previously shown to detect luminance-, texture-, and contrast-defined contours cue invariantly. Motion-defined contour stimuli were generated by modulating the velocity of high spatial frequency sinusoidal luminance gratings (carrier gratings) by a moving squarewave envelope. The carrier gratings were outside the luminance passband of a neuron, such that presence of the carrier alone within the receptive field did not elicit a response. Most neurons that responded to contrast-defined contours also responded to motion-defined contours. The orientation and direction selectivity of these neurons for motion-defined contours was similar to that of luminance gratings. A given neuron also exhibited similar selectivity for the spatial frequency of the carrier gratings of contrast- and motion-defined contours. These results suggest that different second-order contours are detected in a form-cue invariant manner, through a common neural mechanism in area 18.

Visual chimaeras obtained with the Riesz transform

Similar to an auditory chimaera (Smith et al, 2002 Nature 416 87-90), a visual chimaera can be defined as a synthetic image which has the fine spatial structure of one natural image and the envelope of another image in each spatial frequency band. Visual chimaeras constructed in this way could be useful to vision scientists interested in the study of interactions between first-order and second-order visual processing. Although it is almost trivial to generate 1-D chimaeras by means of the Hilbert transform and the analytic signal, problems arise in multidimensional signals like images given that the partial directional Hilbert transform and current 2-D demodulation algorithms are anisotropic or orientation-variant procedures. Here, we present a computational procedure to synthesise visual chimaeras by means of the Riesz transform--an isotropic generalisation of the Hilbert transform for multidimensional signals--and the associated monogenic signal--the vector-valued function counterpart of the analytic signal in which the Riesz transform replaces the Hilbert transform. Examples of visual chimaeras are shown for same/different category images.

A visual field dependent architecture for second order motion processing

The visual system exploits a cortical hierarchy to process complex inputs such as those defined by modulations of motion and/or texture. One class of visual stimuli, composed of alternate stripes of opposing motion requires at least 2 separate stages of computation within this cortical hierarchy, thought to involve cortical area V1 and extra-striate regions like global motion area MT respectively. Using a psychophysical task, we characterise sensitivity to such stimuli containing periodic spatial modulations of motion gradients as a function of the ratio of the spatial parameters at the two processing levels by manipulating the spatial properties of the carrier and modulator. We find band-passed functions for foveal stimulus presentations showing an optimum sensitivity at ratios in the range of r≤10, informative of the coupling relationship between frequency channels at the carrier and modulator levels. An annulus stimulus (excluding the fovea) with a radius of 15.5° exhibited optima of sensitivity at r>15. This difference in the optimal coupling between filtering stages reflects a processing architecture that changes with eccentricity, consistent with the previously observed smaller differences between mean receptive field sizes in striate and extra-striate filtering stages in the fovea compared to the periphery. This is also important for visual psychophysics when comparing sensitivity for first and second order stimuli across retinal eccentricity.

Visual responses to contrast-defined contours with equally spatial-scaled carrier in cat area 18

Contrast-defined contours are one type of second-order contours, across which there are no differences in luminance. Although they can be always perceived, their responses have been only investigated when the spatial frequency of carrier, the background texture whose contrast is modulated to form contours, is much higher than that of contrast-defined contours, due to the interference of responses to luminance contours in other cases. In the present study, we examined visual responses in cat area 18 to the contrast-defined contours with carrier at same spatial frequency equal to neuron's preferred value for luminance contours, by establishing a control stimulus including all the luminance components but lack of the contrast contour information. Using single unit recording and intrinsic optical imaging, we demonstrated that contrast gratings with equally spatial-scaled carrier induced responses in a proportion of cat area 18 neurons with the preferred orientation similar to that for luminance contours, and the responses generated orientation maps similar to those for luminance contours. Our finding suggests that early visual cortex can process second-order contours regardless of the spatial frequency of carriers, in a way similar to the processing of luminance contours. This uniform manner of early visual processing might underlie the visual detection of both luminance contours and non-luminance second-order contours.

Orientation gradient detection exhibits variable coupling between first- and second-stage filtering mechanisms

We investigated sensitivity to orientation modulation using visual stimuli with bandpass filtered noise carriers. We characterized the relationship between the spatial parameters of the modulator and the carrier using a 2-AFC detection task. The relationship between these two parameters is potentially informative of the underlying coupling between first- and second-stage filtering mechanisms, which, in turn, may bear on the interrelationship between striate and extrastriate cortical processing. Our previous experiments on analogous motion stimuli found an optimum sensitivity when the ratio of the carrier and modulator spatial frequency parameters (r) was approximately ten. The current results do not exhibit an optimum sensitivity at a given value of the ratio r. Previous experiments involving second-order modulation sensitivity show an inconsistent range of estimates of optimum sensitivity at values of r between 5 and 50. Our results, using a complementary approach, confirm these discrepancies, demonstrating that the coupling between carrier and modulator frequency parameters depends on a number of stimulus-specific factors, such as contrast sensitivity, stimulus eccentricity, and absolute values of the carrier and modulator spatial frequency parameters. We show that these observations are true for a stimulus limited in eccentricity and that this orientation-modulated stimulus does not exhibit scale invariance. Such processing can not be modeled by a generic filter-rectify-filter model.

Spatial summation of first-order and second-order motion in human vision

This study assessed spatial summation of first-order (luminance-defined) and second-order (contrast-defined) motion. Thresholds were measured for identifying the drift direction of 1c/deg., luminance-modulated and contrast-modulated dynamic noise drifting at temporal frequencies of 0.5, 2 and 8Hz. Image size varied from 0.125 degrees to 16 degrees . The effects of increasing image size on thresholds for luminance-modulated noise were also compared to those for luminance-defined gratings. In all cases, performance improved as image size increased. The rate at which performance improved with increasing image size was similar for all stimuli employed although the slopes corresponding to the initial improvement were steeper for first-order compared to second-order motion. The image sizes at which performance for first-order motion asymptote were larger than for second-order motion. In addition, findings showed that the minimum image size required to support reliable identification of the direction of moving stimuli is greater for second-order than first-order motion. Thus, although first-order and second-order motion processing have a number of properties in common, the visual system's sensitivity to each type of motion as a function of image size is quite different.

Mapping of contextual modulation in the population response of primary visual cortex

We review the evidence of long-range contextual modulation in V1. Populations of neurons in V1 are activated by a wide variety of stimuli outside of their classical receptive fields (RF), well beyond their surround region. These effects generally involve extra-RF features with an orientation component. The population mapping of orientation preferences to the upper layers of V1 is well understood, as far as the classical RF properties are concerned, and involves organization into pinwheel-like structures. We introduce a novel hypothesis regarding the organization of V1's contextual response. We show that RF and extra-RF orientation preferences are mapped in related ways. Orientation pinwheels are the foci of both types of features. The mapping of contextual features onto the orientation pinwheel has a form that recapitulates the organization of the visual field: an iso-orientation patch within the pinwheel also responds to extra-RF stimuli of the same orientation. We hypothesize that the same form of mapping applies to other stimulus properties that are mapped out in V1, such as colour and contrast selectivity. A specific consequence is that fovea-like properties will be mapped in a systematic way to orientation pinwheels. We review the evidence that cytochrome oxidase blobs comprise the foci of this contextual remapping for colour and low contrasts. Neurodynamics and motion in the visual field are argued to play an important role in the shaping and maintenance of this type of mapping in V1.

Models and Measurements of Functional Maps in V1

The organization of primary visual cortex has been heavily studied for nearly 50 years, and in the last 20 years functional imaging has provided high-resolution maps of its tangential organization. Recently, however, the usefulness of maps like those of orientation and spatial frequency (SF) preference has been called into question because they do not, by themselves, predict how moving images are represented in V1. In this review, we discuss a model for cortical responses (the spatiotemporal filtering model) that specifies the types of cortical maps needed to predict distributed activity within V1. We then review the structure and interrelationships of several of these maps, including those of orientation, SF, and temporal frequency preference. Finally, we discuss tests of the model and the sufficiency of the requisite maps in predicting distributed cortical responses. Although the spatiotemporal filtering model does not account for all responses within V1, it does, with reasonable accuracy, predict population responses to a variety of complex stimuli.

Neuronal response to texture- and contrast-defined boundaries in early visual cortex

Natural scenes contain a variety of visual cues that facilitate boundary perception (e.g., luminance, contrast, and texture). Here we explore whether single neurons in early visual cortex can process both contrast and texture cues. We recorded neural responses in cat A18 to both illusory contours formed by abutting gratings (ICs, texture-defined) and contrast-modulated gratings (CMs, contrast-defined). We found that if a neuron responded to one of the two stimuli, it also responded to the other. These neurons signaled similar contour orientation, spatial frequency, and movement direction of the two stimuli. A given neuron also exhibited similar selectivity for spatial frequency of the fine, stationary grating components (carriers) of the stimuli. These results suggest that the cue-invariance of early cortical neurons extends to different kinds of texture or contrast cues, and might arise from a common nonlinear mechanism.

Critical spatial frequencies for illusory contour processing in early visual cortex

Single neurons in primate V2 and cat A18 exhibit identical orientation tuning for sinewave grating and illusory contour stimuli. This cue invariance is also manifested in similar orientation maps to these stimuli, but in V1/A17 the illusory contour maps appear reversed. We hypothesized that this map reversal depends upon the spatial frequencies of the inducers in the illusory contours, relative to the spatial selectivities of these brain areas. We employed intrinsic signal optical imaging to measure orientation maps in cat A17/18 to illusory contours with inducers at spatial frequencies from 0.15 to 1.6 cpd. A17 illusory contour maps were indeed reversed compared with grating-driven maps for inducer spatial frequencies <1.3 cpd, whereas A18 maps were invariant. Simulations based on known neurophysiology demonstrated that map reversal can arise from linear filtering, and map invariance can be explained by a nonlinear (filter-rectify-filter) mechanism. The simulation also correctly predicted that A17 could show invariant maps when the inducer spatial frequency is sufficiently high (1.6 cpd), and that A18 maps could reverse at lower inducer frequencies (0.18 cpd). Thus, the map reversal or invariance to illusory contours depends critically on the relationship of the inducer spatial frequencies to the spatial filtering properties of neurons in each brain area.

Ecologically valid combinations of first- and second-order surface markings facilitate texture discrimination

Natural scenes contain localized variations in both first-order (luminance) and second-order (contrast and texture) information. There is much evidence that first- and second-order stimuli are detected by distinct mechanisms in the mammalian visual system. However, in natural scenes the two kinds of information tend to be spatially correlated. Do correlated and uncorrelated combinations of first- and second-order stimuli differentially affect perception? To address this question we employed orientation-modulated textures in which observers were required to discriminate the spatial frequency of the texture modulation. The textures consisted of micropatterns defined as either local variations in luminance (first-order) or luminance contrast (second-order). Performance was robust with textures composed of only first-order micropatterns, but impossible with only second-order micropatterns. However, when the second-order micropatterns were combined with the first-order micropatterns, they enhanced performance when the two were spatially correlated, but impaired performance when the two were spatially uncorrelated. We conclude that local second-order information may enhance texture modulation discrimination provided it is combined with first-order information in an ecologically valid manner.

Contour discontinuities subserve two types of form analysis that underlie motion processing

Form analysis subserves motion processing in at least two ways: first, in terms of figural segmentation dedicated to solving the problem of figure-to-figure matching over time, and second, in terms of defining trackable features whose unambiguous motion signals can be generalized to ambiguously moving portions of an object. The former is a primarily ventral process involving the lateral occipital complex and also retinotopic areas such as V2 and V4, and the latter is a dorsal process involving V3A. Contour discontinuities, such as corners, deep concavities, maxima of positive curvature, junctions, and terminators, play a central role in both types of form analysis. Transformational apparent motion will be discussed in the context of figural segmentation and matching, and rotational motion in the context of trackable features. In both cases the analysis of form must proceed in parallel with the analysis of motion, in order to constrain the ongoing analysis of motion.

Processing of first-order motion in marmoset visual cortex is influenced by second-order motion

We measured the responses of single neurons in marmoset visual cortex (V1, V2, and the third visual complex) to moving first-order stimuli and to combined first- and second-order stimuli in order to determine whether first-order motion processing was influenced by second-order motion. Beat stimuli were made by summing two gratings of similar spatial frequency, one of which was static and the other was moving. The beat is the product of a moving sinusoidal carrier (first-order motion) and a moving low-frequency contrast envelope (second-order motion). We compared responses to moving first-order gratings alone with responses to beat patterns with first-order and second-order motion in the same direction as each other, or in opposite directions to each other in order to distinguish first-order and second-order direction-selective responses. In the majority (72%, 67/93) of cells (V1 73%, 45/62; V2 70%, 16/23; third visual complex 75%, 6/8), responses to first-order motion were significantly influenced by the addition of a second-order signal. The second-order envelope was more influential when moving in the opposite direction to the first-order stimulus, reducing first-order direction sensitivity in V1, V2, and the third visual complex. We interpret these results as showing that first-order motion processing through early visual cortex is not separate from second-order motion processing; suggesting that both motion signals are processed by the same system.

The cellular basis for parallel neural transmission of a high-frequency stimulus and its low-frequency envelope

Sensory stimuli often have rich temporal and spatial structure. One class of stimuli that are common to visual and auditory systems and, as we show, the electrosensory system are signals that contain power in a narrow range of temporal (or spatial) frequencies. Characteristic of this class of signals is a slower variation in their amplitude, otherwise known as an envelope. There is evidence suggesting that, in the visual cortex, both narrowband stimuli and their envelopes are coded for in separate and parallel streams. The implementation of this parallel transmission is not well understood at the cellular level. We have identified the cellular basis for the parallel transmission of signal and envelope in the electrosensory system: a two-cell network consisting of an interneuron connected to a pyramidal cell by means of a slow synapse. This circuit could, in principle, be implemented in the auditory or visual cortex by the previously identified biophysics of cortical interneurons.

Neural mechanisms mediating responses to abutting gratings: luminance edges vs. illusory contours

The discontinuities of phase-shifted abutting line gratings give rise to perception of an "illusory contour" (IC) along the line terminations. Neuronal responses to such ICs have been interpreted as evidence for a specialized visual mechanism, since such responses cannot be predicted from conventional linear receptive fields. However, when the spatial scale of the component gratings (carriers) is large compared to the neuron's luminance passband, these IC responses might be evoked simply by the luminance edges at the line terminations. Thus by presenting abutting gratings at a series of carrier spatial scales to cat A18 neurons, we were able to distinguish genuine nonlinear responses from those due to luminance edges. Around half of the neurons (both simple and complex types) showed a bimodal response pattern to abutting gratings: one peak at a low carrier spatial frequency range that overlapped with the luminance passband, and a second distinct peak at much higher frequencies beyond the neuron's grating resolution. For those bimodally responding neurons, the low-frequency responses were sensitive to carrier phase, but the high-frequency responses were phase-invariant. Thus the responses at low carrier spatial frequencies could be understood via a linear model, while the higher frequency responses represented genuine nonlinear IC processing. IC responsive neurons also demonstrated somewhat lower spatial preference to the periodic contours (envelopes) compared to gratings, but the optimal orientation and motion direction for both were quite similar. The nonlinear responses to ICs could be explained by the same energy mechanism underlying responses to second-order stimuli such as contrast-modulated gratings. Similar neuronal preferences for ICs and for gratings may contribute to the form-cue invariant perception of moving contours.

Orientation-selective adaptation to first- and second-order patterns in human visual cortex

Second-order textures-patterns that cannot be detected by mechanisms sensitive only to luminance changes-are ubiquitous in visual scenes, but the neuronal mechanisms mediating perception of such stimuli are not well understood. We used an adaptation protocol to measure neural activity in the human brain selective for the orientation of second-order textures. Functional MRI (fMRI) responses were measured in three subjects to presentations of first- and second-order probe gratings after adapting to a high-contrast first- or second-order grating that was either parallel or orthogonal to the probe gratings. First-order (LM) stimuli were generated by modulating the stimulus luminance. Second-order stimuli were generated by modulating the contrast (CM) or orientation (OM) of a first-order carrier. We used four combinations of adapter and probe stimuli: LM:LM, CM:CM, OM:OM, and LM:OM. The fourth condition tested for cross-modal adaptation with first-order adapter and second-order probe stimuli. Attention was diverted from the stimulus by a demanding task at fixation. Both first- and second-order stimuli elicited orientation-selective adaptation in multiple cortical visual areas, including V1, V2, V3, V3A/B, a newly identified visual area anterior to dorsal V3 that we have termed LO1, hV4, and VO1. For first-order stimuli (condition LM:LM), the adaptation was no larger in extrastriate areas than in V1, implying that the orientation-selective first-order (luminance) adaptation originated in V1. For second-order stimuli (conditions CM:CM and OM:OM), the magnitude of adaptation, relative to the absolute response magnitude, was significantly larger in VO1 (and for condition CM:CM, also in V3A/B and LO1) than in V1, suggesting that second-order stimulus orientation was extracted by additional processing after V1. There was little difference in the amplitude of adaptation between the second-order conditions. No consistent effect of adaptation was found in the cross-modal condition LM:OM, in agreement with psychophysical evidence for weak interactions between first- and second-order stimuli and computational models of separate mechanisms for first- and second-order visual processing.

Spatiochromatic statistics of natural scenes: first- and second-order information and their correlational structure

Spatial filters that mimic receptive fields of visual cortex neurons provide an efficient representation of achromatic image structure, but the extension of this idea to chromatic information is at an early stage. Relatively few studies have looked at the statistical relationships between the modeled responses to natural scenes of the luminance (LUM), red-green (RG), and blue-yellow (BY) postreceptoral channels of the primate visual system. Here we consider the correlations among these channel responses in terms of pixel, first-order, and second-order information. First-order linear filtering was implemented by convolving the cosine-windowed images with oriented Gabor functions, whose gains were scaled to give equal amplitude response across spatial frequency to random fractal images. Second-order filtering was implemented via a filter-rectify-filter cascade, with Gabor functions for both first- and second-stage filters. Both signed and unsigned filter responses were obtained across a range of filter parameters (spatial frequency, 2-64 cycles/image; orientation, 0-135 degrees). The filter responses to the LUM channel images were larger than those for either RG or BY channel images. Cross correlations between the first-order channel responses and between the first- and second-order channel responses were measured. Results showed that the unsigned correlations between first-order channel responses were higher than expected on the basis of previous studies and that first-order channel responses were highly correlated with LUM, but not with RG or BY, second-order responses. These findings imply that course-scale color information correlates well with course-scale changes of fine-scale texture.

Boundary cue invariance in cortical orientation maps

We effortlessly perceive oriented boundaries defined by either luminance changes ('first-order' cues) or texture variations ('second-order' cues). Many neurons in mammalian visual cortex show orientation preference to both types of boundaries, but it is uncertain how they contribute to perceptual orientation cue-invariance at the neuronal population level. Using optical imaging in cat A 18, we observed highly similar orientation preference maps to first-order and a variety of second-order visual stimuli. Thus the neuronal representation of coarse-scale boundary orientation appears to be invariant to the characteristics (including local orientation) of the fine-scale textures by which those boundaries are defined. A common feature of second-order visual stimuli is that modulation shifts their Fourier energy for boundary orientation to the higher spatial frequencies of their constituent textures - our results suggest a common neural mechanism (demodulation) mediating visual processing of many kinds of texture boundary. The similarity between orientation maps to different stimuli implies that second-order responsive neurons are homogeneously distributed across the cortical surface. Such homogeneously cue-invariant orientation representation could provide a neural substrate for perceptual form-cue invariance, and reflect an optimal organization for encoding orientation information in natural scenes.

Is the direction of second-order, contrast-defined motion patterns visible to standard motion-energy detectors: a model answer?

Previous psychophysical studies (e.g., Smith & Ledgeway, 1997) have provided evidence that under some conditions, the detection of a particular class of stimuli (contrast-modulated static noise) widely employed to study second-order motion processing may be inadvertently based on encoding local imbalances in luminance motion energy. In particular when static noise composed of relatively large noise elements is used, direction-identification performance at threshold may actually be mediated by the same mechanisms that respond to first-order motion, due to the presence of persistent spatial clusters of noise elements of the same polarity. However, Benton and Johnson (1997) modeled the responses of conventional motion-energy detectors to contrast-modulated static noise patterns and found no evidence of any systematic directional biases in such stimuli when the mean opponent motion energy was used to quantify performance. In the present paper we sought to resolve this discrepancy and show that the precise manner in which computational models are implemented is crucial in determining their response to contrast-modulated, second-order motion patterns. In particular we demonstrate that by considering the information encapsulated by the peak (rather than the mean) opponent motion energy and the predominantly local nature of imbalances in motion energy that can arise in contrast-modulated static noise, it is possible to readily model the patterns of empirical results found.

Spatial frequency selective masking of first-order and second-order motion in the absence of off-frequency 'looking'

Converging evidence suggests that, at least initially, first-order (luminance defined) and second-order (e.g. contrast defined) motion are processed independently in human vision. However, adaptation studies suggest that second-order motion, like first-order motion, may be encoded by spatial frequency selective mechanisms each operating over a limited range of scales. Nonetheless, the precise properties of these mechanisms are indeterminate since the spatial frequency selectivity of adaptation aftereffects may not necessarily represent the frequency tuning of the underlying units [Vision Research 37 (1997) 2685]. To address this issue we used visual masking to investigate the spatial-frequency tuning of the mechanisms that encode motion. A dual-masking paradigm was employed to derive estimates of the spatial tuning of motion sensors, in the absence of off-frequency 'looking'. Modulation-depth thresholds for identifying the direction of a sinusoidal test pattern were measured over a 4-octave range (0.125-2 c/deg) in both the absence and presence of two counterphasing masks, simultaneously positioned above and below the test frequency. For second-order motion, the resulting masking functions were spatially bandpass in character and remained relatively invariant with changes in test spatial frequency, masking pattern modulation depth and the temporal properties of the noise carrier. As expected, bandpass spatial frequency tuning was also found for first-order motion. This provides compelling evidence that the mechanisms responsible for encoding each variety of motion exhibit spatial frequency selectivity. Thus, although first-order and second-order motion may be encoded independently, they must utilise similar computational principles.

First- and second-order information in natural images: a filter-based approach to image statistics

Previous analyses of natural image statistics have dealt mainly with their Fourier power spectra. Here we explore image statistics by examining responses to biologically motivated filters that are spatially localized and respond to first-order (luminance-defined) and second-order (contrast- or texture-defined) characteristics. We compare the distribution of natural image responses across filter parameters for first- and second-order information. We find that second-order information in natural scenes shows the same self-similarity previously described for first-order information but has substantially less orientational anisotropy. The magnitudes of the two kinds of information, as well as their mutual unsigned correlation, are much stronger for particular combinations of filter parameters in natural images but not in unstructured fractal images having the same power spectra.

Contour integration and cortical processing

Our understanding of visual processing in general, and contour integration in particular, has undergone great change over the last 10 years. There is now an accumulation of psychophysical and neurophysiological evidence that the outputs of cells with conjoint orientation preference and spatial position are integrated in the process of explication of rudimentary contours. Recent neuroanatomical and neurophysiological results suggest that this process takes place at the cortical level V1. The code for contour integration may be a temporal one in that it may only manifest itself in the latter part of the spike train as a result of feedback and lateral interactions. Here we review some of the properties of contour integration from a psychophysical perspective and we speculate on their underlying neurophysiological substrate.

Pulling the other one: 1st- and 2nd-order visual information interact to determine perceived location

We demonstrate that the 1st- and 2nd-order characteristics of a visual stimulus can have a profound influence on each other in terms of perceived position. We use the parameter of spatial separation to selectively manipulate the effect of one characteristic upon the other. 1st-order features have their largest effect upon the perceived position of 2nd-order structure when separation is small, whilst the reciprocal effect is maximal at large separations. Implications for models of 1st- and 2nd-order interaction are discussed.

Texture contrast attracts overt visual attention in natural scenes

In natural vision, the central nervous system actively selects information for detailed processing through mechanisms of visual attention. It is widely held that simple stimulus features such as color, orientation and intensity contribute to the determination of visual salience and thus can act to guide the selection process in a bottom-up fashion. Contrary to this view, Einhäuser, W. & König, P. [(2003) Eur. J. Neurosci., 17, 1089-1097] conclude from their study of human eye movements that luminance contrast does not contribute to the calculation of stimulus salience and that top-down, rather than bottom-up, factors therefore determine attentional allocation in natural scenes. In this article, we dispute their conclusion and argue that the Einhäuser and König study has a number of methodological problems, the most prominent of which is the unintentional introduction of changes in texture contrast. We hypothesize that texture contrast, like luminance contrast, can contribute to the guidance of attention in a bottom-up fashion, and that an appeal to top-down factors is not necessary. To test this hypothesis, we implement a purely bottom-up model of visual selective attention where salience is derived from both luminance and texture contrast. We find that the model can quantitatively account for Einhäuser and König's results and that texture contrast strongly influences attentional guidance in this particular paradigm. The significance of this result for attentional guidance in other paradigms is discussed.

Integration of first- and second-order orientation

The problem of how visual information such as orientation is combined across space bears on key visual abiities, such as texture perception. Orientation signals can be derived from both luminance and contrast, but it is not well understood how such information is pooled or how these different orientation signals interact in the integration process. We measured orientation discrimination thresholds for arrays of equivisible first-order and second-order Gabors. Thresholds were measured as the orientation variability in the arrays increased, and we estimated the number of samples (or efficiency) and internal noise of the mechanism being used. Observers were able to judge the mean orientation of arrays of either first- or second-order Gabors. For arrays of first-order and arrays of second-order Gabors, estimates of the number of samples used increased as the number of Gabors increased. When judging the orientation of arrays of either order, observers were able to ignore randomly oriented Gabors of the opposite order. If observers did not know which Gabor type carried the more useful orientation information, they tended to use the information from first-order Gabors (even when this was poorer information). Observers were unable to combine information from first- and second-order Gabors, though this would have improved their performance. The visual system appears to have separate integrators for combining local orientation across space for luminance- and contrast-defined features.

Longer VEP latencies and slower reaction times to the onset of second-order motion than to the onset of first-order motion

We compared visual evoked potentials and psychophysical reaction times to the onset of first- and second-order motion. The stimuli consisted of luminance-modulated (first-order) and contrast-modulated (second-order) 1 cpd vertical sine-wave gratings drifting rightward for 140 ms at a velocity of 6 degrees /s. For each condition, we analysed the latencies and peak-to-baseline amplitudes of the P1 and N2 peaks recorded at Oz. For first-order motion, both P1 and N2 peaks were present at low (3%) contrast (i.e., depth modulations) whereas for second-order motion they appeared only at higher (25%) contrasts. When the two types of motion were equated for visibility, responses were slower for second-order motion than for first-order motion: about 44 ms slower for P1 latencies, 53 ms slower for N2 latencies, and 76 ms slower for reaction times. The longer VEP latencies for second-order motion support models that postulate additional processing steps for the extraction of second-order motion. The slower reaction time to the onset of second-order motion suggests that the longer neurophysiological analysis translates into slower detection.

The importance of spatial scale in determining illusions of orientation

The twisted-cord illusion is a powerful demonstration of interaction between 1st-order (luminance-defined) and 2nd-order (contrast-defined) orientation processing. The perceived orientation of contrast-defined objects is pulled towards their 1st-order orientation content when the difference in orientation is small (Fraser effect), yet is pushed away from the 1st-order content at large orientation differences (Zöllner effect). Here we show that the relative spatial scale of carrier and envelope represents a decisive factor in determining the magnitude and direction of such interactions. We conclude that the perceived 2nd-order structure of a stimulus is biased by the properties of the 1st-order structure in a manner that depends on relative, rather than absolute spatial scale.

The duration of 3-d form analysis in transformational apparent motion

Transformational apparent motion (TAM) occurs when a figure changes discretely from one configuration to another overlapping configuration. Rather than an abrupt shape change, the initial shape is perceived to transform smoothly into the final shape as if animated by a series of intermediate shapes. We find that TAM follows an analysis of form that takes 80-140 msec. Form analysis can function both at and away from equiluminance and can occur over contours defined by uniform regions as well as outlines. Moreover, the forms analyzed can be 3-D, resulting in motion paths that appear to smoothly project out from or into the stimulus plane. The perceived transformation is generally the one that involves the least change in the shape or location of the initial figure in a 3-D sense. We conclude that perception of TAM follows an analysis of 3-D form that takes approximately 100 msec. This stage of form analysis may be common to both TAM and second-order motion.

Processing of second-order stimuli in the visual cortex

Naturally occurring visual stimuli are rich in examples of objects delineated from their backgrounds simply by differences in luminance, so-called first-order stimuli, as well as those defined by differences of contrast or texture, referred to as second-order stimuli. Here we provide a brief overview of visual cortical processing of second-order stimuli, as well as some comparative background on first-order processing, concentrating on single-unit neurophysiology, but also discussing relationships to human psychophysics and to neuroimaging. The selectivity of visual cortical neurons to orientation, spatial frequency, and direction of movement of first-order, luminance-defined stimuli is conventionally understood in terms of simple linear filter models, albeit with some minor nonlinearities such as thresholding and gain control. However, these kinds of models fail entirely to account for responses of neurons to second-order stimuli such as contrast envelopes, illusory contours, or texture borders. Second-order stimuli constructed from sinusoidal components have been used to analyze the neurophysiological mechanisms of such responses; these experiments demonstrate that the same neuron can exhibit three distinct kinds of tuning to spatial frequency, and also to orientation. These results can be understood in terms of a type of nonlinear 'filter-->rectify-->filter' model, which has been widely used in human psychophysics. Finally, several general issues will be discussed, including potential artifacts in experiments with second-order stimuli, and strategies for avoiding or controlling for them; caveats about definitions of first- vs. second-order mechanisms and stimuli; the concept of form-cue invariance; and the functional significance of second-order processing.