Cortical processing of second-order motion

Serotonin increases population coding of behaviorally relevant stimuli by enhancing responses of ON but not OFF-type sensory neurons

How neural populations encode sensory input to generate behavioral responses remains a central problem in systems neuroscience. Here we investigated how neuromodulation influences population coding of behaviorally relevant stimuli to give rise to behavior in the electrosensory system of the weakly electric fish Apteronotus leptorhynchus. We performed multi-unit recordings from ON and OFF sensory pyramidal cells in response to stimuli whose amplitude (i.e., envelope) varied in time, before and after electrical stimulation of the raphe nuclei. Overall, raphe stimulation increased population coding by ON- but not by OFF-type cells, despite both cell types showing similar sensitivities to the stimulus at the single neuron level. Surprisingly, only changes in population coding by ON-type cells were correlated with changes in behavioral responses. Taken together, our results show that neuromodulation differentially affects ON vs. OFF-type cells in order to enhance perception of behaviorally relevant sensory input.

Neuronal On- and Off-type heterogeneities improve population coding of envelope signals in the presence of stimulus-induced noise

Understanding the mechanisms by which neuronal population activity gives rise to perception and behavior remains a central question in systems neuroscience. Such understanding is complicated by the fact that natural stimuli often have complex structure. Here we investigated how heterogeneities within a sensory neuron population influence the coding of a noisy stimulus waveform (i.e., the noise) and its behaviorally relevant envelope signal (i.e., the signal). We found that On- and Off-type neurons displayed more heterogeneities in their responses to the noise than in their responses to the signal. These differences in heterogeneities had important consequences when quantifying response similarity between pairs of neurons. Indeed, the larger response heterogeneity displayed by On- and Off-type neurons made their pairwise responses to the noise on average more independent than when instead considering pairs of On-type or Off-type neurons. Such relative independence allowed for better averaging out of the noise response when pooling neural activities in a mixed-type (i.e., On- and Off-type) than for same-type (i.e., only On-type or only Off-type), thereby leading to greater information transmission about the signal. Our results thus reveal a function for the combined activities of On- and Off-type neurons towards improving information transmission of envelope stimuli at the population level. Our results will likely generalize because natural stimuli across modalities are characterized by a stimulus waveform whose envelope varies independently as well as because On- and Off-type neurons are observed across systems and species.

Object size determines the spatial spread of visual time

A key question for temporal processing research is how the nervous system extracts event duration, despite a notable lack of neural structures dedicated to duration encoding. This is in stark contrast with the orderly arrangement of neurons tasked with spatial processing. In this study, we examine the linkage between the spatial and temporal domains. We use sensory adaptation techniques to generate after-effects where perceived duration is either compressed or expanded in the opposite direction to the adapting stimulus' duration. Our results indicate that these after-effects are broadly tuned, extending over an area approximately five times the size of the stimulus. This region is directly related to the size of the adapting stimulus-the larger the adapting stimulus the greater the spatial spread of the after-effect. We construct a simple model to test predictions based on overlapping adapted versus non-adapted neuronal populations and show that our effects cannot be explained by any single, fixed-scale neural filtering. Rather, our effects are best explained by a self-scaled mechanism underpinned by duration selective neurons that also pool spatial information across earlier stages of visual processing.

More superimposition for contrast-modulated than luminance-modulated stimuli during binocular rivalry

Luminance-modulated noise (LM) and contrast-modulated noise (CM) gratings were presented with interocularly correlated, uncorrelated and anti-correlated binary noise to investigate their contributions to mixed percepts, specifically piecemeal and superimposition, during binocular rivalry. Stimuli were sine-wave gratings of 2 c/deg presented within 2 deg circular apertures. The LM stimulus contrast was 0.1 and the CM stimulus modulation depth was 1.0, equating to approximately 5 and 7 times detection threshold, respectively. Twelve 45 s trials, per noise configuration, were carried out. Fifteen participants with normal vision indicated via button presses whether an exclusive, piecemeal or superimposed percept was seen. For all noise conditions LM stimuli generated more exclusive visibility, and lower proportions of superimposition. CM stimuli led to greater proportions and longer periods of superimposition. For both stimulus types, correlated interocular noise generated more superimposition than did anti- or uncorrelated interocular noise. No significant effect of stimulus type (LM vs CM) or noise configuration (correlated, uncorrelated, anti-correlated) on piecemeal perception was found. Exclusive visibility was greater in proportion, and perceptual changes more numerous, during binocular rivalry for CM stimuli when interocular noise was not correlated. This suggests that mutual inhibition, initiated by non-correlated noise CM gratings, occurs between neurons processing luminance noise (first-order component), as well as those processing gratings (second-order component). Therefore, first- and second-order components can contribute to overall binocular rivalry responses. We suggest the addition of a new well to the current energy landscape model for binocular rivalry that takes superimposition into account.

Nonlinear Y-Like Receptive Fields in the Early Visual Cortex: An Intermediate Stage for Building Cue-Invariant Receptive Fields from Subcortical Y Cells

Many of the neurons in early visual cortex are selective for the orientation of boundaries defined by first-order cues (luminance) as well as second-order cues (contrast, texture). The neural circuit mechanism underlying this selectivity is still unclear, but some studies have proposed that it emerges from spatial nonlinearities of subcortical Y cells. To understand how inputs from the Y-cell pathway might be pooled to generate cue-invariant receptive fields, we recorded visual responses from single neurons in cat Area 18 using linear multielectrode arrays. We measured responses to drifting and contrast-reversing luminance gratings as well as contrast modulation gratings. We found that a large fraction of these neurons have nonoriented responses to gratings, similar to those of subcortical Y cells: they respond at the second harmonic (F2) to high-spatial frequency contrast-reversing gratings and at the first harmonic (F1) to low-spatial frequency drifting gratings ("Y-cell signature"). For a given neuron, spatial frequency tuning for linear (F1) and nonlinear (F2) responses is quite distinct, similar to orientation-selective cue-invariant neurons. Also, these neurons respond to contrast modulation gratings with selectivity for the carrier (texture) spatial frequency and, in some cases, orientation. Their receptive field properties suggest that they could serve as building blocks for orientation-selective cue-invariant neurons. We propose a circuit model that combines ON- and OFF-center cortical Y-like cells in an unbalanced push-pull manner to generate orientation-selective, cue-invariant receptive fields.

SIGNIFICANCE STATEMENT

A significant fraction of neurons in early visual cortex have specialized receptive fields that allow them to selectively respond to the orientation of boundaries that are invariant to the cue (luminance, contrast, texture, motion) that defines them. However, the neural mechanism to construct such versatile receptive fields remains unclear. Using multielectrode recording, we found a large fraction of neurons in early visual cortex with receptive fields not selective for orientation that have spatial nonlinearities like those of subcortical Y cells. These are strong candidates for building cue-invariant orientation-selective neurons; we present a neural circuit model that pools such neurons in an imbalanced "push-pull" manner, to generate orientation-selective cue-invariant receptive fields.

Phase-Dependent Interactions in Visual Cortex to Combinations of First- and Second-Order Stimuli

A fundamental task of the visual system is to extract figure-ground boundaries between objects, which are often defined, not only by differences in luminance, but also by "second-order" contrast or texture differences. Responses of cortical neurons to both first- and second-order patterns have been studied extensively, but only for responses to either type of stimulus in isolation. Here, we examined responses of visual cortex neurons to the spatial relationship between superimposed periodic luminance modulation (LM) and contrast modulation (CM) stimuli, the contrasts of which were adjusted to give equated responses when presented alone. Extracellular single-unit recordings were made in area 18 of the cat, the neurons of which show responses to CM and LM stimuli very similar to those in primate area V2 (Li et al., 2014). Most neurons showed a significant dependence on the relative phase of the combined LM and CM patterns, with a clear overall optimal response when they were approximately phase aligned. The degree of this phase preference, and the contributions of suppressive and/or facilitatory interactions, varied considerably from one neuron to another. Such phase-dependent and phase-invariant responses were evident in both simple- and complex-type cells. These results place important constraints on any future model of the underlying neural circuitry for second-order responses. The diversity in the degree of phase dependence between LM and CM stimuli that we observed could help to disambiguate different kinds of boundaries in natural scenes.

SIGNIFICANCE STATEMENT

Many visual cortex neurons exhibit orientation-selective responses to boundaries defined by differences either in luminance or in texture contrast. Previous studies have examined responses to either type of boundary in isolation, but here we measured systematically responses of cortical neurons to the spatial relationship between superimposed periodic luminance-modulated (LM) and contrast-modulated (CM) stimuli with contrasts adjusted to give equated responses. We demonstrate that neuronal responses to these compound stimuli are highly dependent on the relative phase between the LM and CM components. Diversity in the degree of such phase dependence could help to disambiguate different kinds of boundaries in natural scenes, for example, those arising from surface reflectance changes or from illumination gradients such as shading or shadows.

Neural Processing of Second-Order Motion in the Suprasylvian Cortex of the Cat

Neuronal responses to second-order motion, that is, to spatiotemporal variations of texture or contrast, have been reported in several cortical areas of mammals, including the middle-temporal (MT) area in primates. In this study, we investigated whether second-order responses are present in the cat posteromedial lateral suprasylvian (PMLS) cortex, a possible homolog of the primate area MT. The stimuli used were luminance-based sine-wave gratings (first-order) and contrast-modulated carrier stimuli (second-order), which consisted of a high-spatial-frequency static grating (carrier) whose contrast was modulated by a low-spatial-frequency drifting grating (envelope). Results indicate that most PMLS neurons responded to second-order motion and for the vast majority of cells, first- and second-order preferred directions were conserved. However, responses to second-order stimuli were significantly reduced when compared to those evoked by first-order gratings. Circular variance was increased for second-order stimuli, indicating that PMLS direction selectivity was weaker for this type of stimulus. Finally, carrier orientation selectivity was either absent or very broad and had no influence on the envelope's orientation selectivity. In conclusion, our data show that PMLS neurons exhibit similar first- and second-order response profiles and that, akin primate area MT cells, they perform a form-cue invariant analysis of motion signals.

Binocular functional architecture for detection of contrast-modulated gratings

Combination of signals from the two eyes is the gateway to stereo vision. To gain insight into binocular signal processing, we studied binocular summation for luminance-modulated gratings (L or LM) and contrast-modulated gratings (CM). We measured 2AFC detection thresholds for a signal grating (0.75c/deg, 216ms) shown to one eye, both eyes, or both eyes out-of-phase. For LM and CM, the carrier noise was in both eyes, even when the signal was monocular. Mean binocular thresholds for luminance gratings (L) were 5.4dB better than monocular thresholds - close to perfect linear summation (6dB). For LM and CM the binocular advantage was again 5-6dB, even when the carrier noise was uncorrelated, anti-correlated, or at orthogonal orientations in the two eyes. Binocular combination for CM probably arises from summation of envelope responses, and not from summation of these conflicting carrier patterns. Antiphase signals produced no binocular advantage, but thresholds were about 1-3dB higher than monocular ones. This is not consistent with simple linear summation, which should give complete cancellation and unmeasurably high thresholds. We propose a three-channel model in which noisy monocular responses to the envelope are binocularly combined in a contrast-weighted sum, but also remain separately available to perception via a max operator. Vision selects the largest of the three responses. With in-phase gratings the binocular channel dominates, but antiphase gratings cancel in the binocular channel and the monocular channels mediate detection. The small antiphase disadvantage might be explained by a subtle influence of background responses on binocular and monocular detection.

Encoding and estimation of first- and second-order binocular disparity in natural images

•

First- and second-order responses to natural binocular images are correlated.

•

Second-order mechanisms can improve the accuracy of disparity estimation.

•

Second-order mechanisms can extend the depth range of binocular stereopsis.

The first stage of processing of binocular information in the visual cortex is performed by mechanisms that are bandpass-tuned for spatial frequency and orientation. Psychophysical and physiological evidence have also demonstrated the existence of second-order mechanisms in binocular processing, which can encode disparities that are not directly accessible to first-order mechanisms. We compared the responses of first- and second-order binocular filters to natural images. We found that the responses of the second-order mechanisms are to some extent correlated with the responses of the first-order mechanisms, and that they can contribute to increasing both the accuracy, and depth range, of binocular stereopsis.

Contrast coding in the electrosensory system: parallels with visual computation

To identify and interact with moving objects, including other members of the same species, an animal's nervous system must correctly interpret patterns of contrast in the physical signals (such as light or sound) that it receives from the environment. In weakly electric fish, the motion of objects in the environment and social interactions with other fish create complex patterns of contrast in the electric fields that they produce and detect. These contrast patterns can extend widely over space and time and represent a multitude of relevant features, as is also true for other sensory systems. Mounting evidence suggests that the computational principles underlying contrast coding in electrosensory neural networks are conserved elements of spatiotemporal processing that show strong parallels with the vertebrate visual system.

Form-cue invariant second-order neuronal responses to contrast modulation in primate area V2

A fundamental task of the visual system is to extract figure-ground boundaries between images of objects, which in natural scenes are often defined not only by luminance differences but also by "second-order" contrast or texture differences. Responses to contrast modulation (CM) and other second-order stimuli have been extensively studied in human psychophysics, but the neuronal substrates of second-order responses in nonhuman primates remain poorly understood. In this study, we have recorded single neurons in area V2 of macaque monkeys, using both CM patterns as well as conventional luminance modulation (LM) gratings. CM stimuli were constructed from stationary sine wave grating carrier patterns, which were modulated by drifting envelope gratings of a lower spatial frequency. We found approximately one-third of visually responsive V2 neurons responded to CM stimuli with a pronounced selectivity to carrier spatial frequencies, and often orientations, that were clearly outside the neurons' passbands for LM gratings. These neurons were "form-cue invariant" in that their tuning to CM envelope spatial frequency and orientation was very similar to that for LM gratings. Neurons were tuned to carrier spatial frequencies that were typically 2-4 octaves higher than their optimal envelope spatial frequencies, similar to results from human psychophysics. These results are distinct from CM responses arising from surround suppression, but could be understood in terms of a filter-rectify-filter model. Such neurons could provide a functionally useful and explicit representation of segmentation boundaries as well as a plausible neural substrate for human perception of second-order boundaries.

Contextual modulation as de-texturizer

Contextual modulation refers to the effect of texture placed outside of a neuron's classical receptive field as well as the effect of surround texture on the perceptual properties of variegated regions within. In this minireview, we argue that one role of contextual modulation is to enhance the perception of contours at the expense of textures, in short to de-texturize the image. The evidence for this role comes mainly from three sources: psychophysical studies of shape after-effects, computational models of neurons that exhibit iso-orientation surround inhibition, and fMRI studies revealing specialized areas for contour as opposed to texture processing. The relationship between psychophysical studies that support the notion of contextual modulation as de-texturizer and those that investigate contour integration and crowding is discussed.

Depth perception from dynamic occlusion in motion parallax: roles of expansion-compression versus accretion-deletion

Motion parallax, or differential retinal image motion from observer movement, provides important information for depth perception. We previously measured the contribution of shear motion parallax to depth, which is only composed of relative motion information. Here, we examine the roles of relative motion and accretion-deletion information in dynamic occlusion motion parallax. Observers performed two-alternative forced choice depth-ordering tasks in response to low spatial frequency patterns of horizontal random dot motion that were synchronized to the observer's head movements. We examined conditions that isolated or combined expansion-compression and accretion-deletion across a range of simulated relative depths. At small depths, expansion-compression provided reliable depth perception while accretion-deletion had a minor contribution: When the two were in conflict, the perceived depth was dominated by expansion-compression. At larger depths in the cue-conflict experiment, accretion-deletion determined the depth-ordering performance. Accretion-deletion in isolation did not yield any percept of depth even though, in theory, it provided sufficient information for depth ordering. Thus, accretion-deletion can substantially enhance depth perception at larger depths but only in the presence of relative motion. The results indicate that expansion-compression contributes to depth from motion parallax across a broad range of depths whereas accretion-deletion contributes primarily at larger depths.

Functional organization of envelope-responsive neurons in early visual cortex: organization of carrier tuning properties

It is well established that visual cortex neurons having similar selectivity for orientation, direction of motion, ocular dominance, and other properties of first-order (luminance-defined) stimuli are clustered into a columnar organization. However, the cortical architecture of neuronal responses to second-order (contrast/texture-defined) stimuli is poorly understood. A useful second-order stimulus is a contrast envelope, consisting of a finely detailed pattern (carrier) whose contrast varies on a coarse spatial scale (envelope). In this study, we analyzed the cortical organization of carrier tuning properties of neurons, which responded to contrast-modulated stimuli. We examined whether neurons tuned to similar carrier properties are clustered spatially and whether such spatial clusters are arranged in columns. To address these questions, we recorded single-unit activity, multiunit activity, and local field potentials simultaneously from area 18 of anesthetized cats, using single-channel microelectrodes and multielectrode arrays. Our data showed that neurons tuned to similar carrier spatial frequency are distributed in a highly clustered manner; neurons tuned to similar carrier orientation are also significantly clustered. Neurons along linear arrays perpendicular to the brain surface always exhibited similar optimal carrier spatial frequency, indicating a columnar organization. Multi-pronged tetrode recordings indicated that the diameter of these columns is ≥450 μm. Optimal carrier orientation was also significantly clustered but with finer-grain organization and greater scatter. These results indicate a fine anatomical structure of cortical organization of second-order information processing and suggest that there are probably more maps in cat area 18 than previously believed.

Motion-defined contour processing in the early visual cortex

From our daily experience, it is very clear that relative motion cues can contribute to correctly identifying object boundaries and perceiving depth. Motion-defined contours are not only generated by the motion of objects in a scene but also by the movement of an observer's head and body (motion parallax). However, the neural mechanism involved in detecting these contours is still unknown. To explore this mechanism, we extracellularly recorded visual responses of area 18 neurons in anesthetized and paralyzed cats. The goal of this study was to determine if motion-defined contours could be detected by neurons that have been previously shown to detect luminance-, texture-, and contrast-defined contours cue invariantly. Motion-defined contour stimuli were generated by modulating the velocity of high spatial frequency sinusoidal luminance gratings (carrier gratings) by a moving squarewave envelope. The carrier gratings were outside the luminance passband of a neuron, such that presence of the carrier alone within the receptive field did not elicit a response. Most neurons that responded to contrast-defined contours also responded to motion-defined contours. The orientation and direction selectivity of these neurons for motion-defined contours was similar to that of luminance gratings. A given neuron also exhibited similar selectivity for the spatial frequency of the carrier gratings of contrast- and motion-defined contours. These results suggest that different second-order contours are detected in a form-cue invariant manner, through a common neural mechanism in area 18.

Visual chimaeras obtained with the Riesz transform

Similar to an auditory chimaera (Smith et al, 2002 Nature 416 87-90), a visual chimaera can be defined as a synthetic image which has the fine spatial structure of one natural image and the envelope of another image in each spatial frequency band. Visual chimaeras constructed in this way could be useful to vision scientists interested in the study of interactions between first-order and second-order visual processing. Although it is almost trivial to generate 1-D chimaeras by means of the Hilbert transform and the analytic signal, problems arise in multidimensional signals like images given that the partial directional Hilbert transform and current 2-D demodulation algorithms are anisotropic or orientation-variant procedures. Here, we present a computational procedure to synthesise visual chimaeras by means of the Riesz transform--an isotropic generalisation of the Hilbert transform for multidimensional signals--and the associated monogenic signal--the vector-valued function counterpart of the analytic signal in which the Riesz transform replaces the Hilbert transform. Examples of visual chimaeras are shown for same/different category images.

Measuring perceived differences in surface texture due to changes in higher order statistics

We investigate the ability of humans to perceive changes in the appearance of images of surface texture caused by the variation of their higher order statistics. We incrementally randomize their phase spectra while holding their first and second order statistics constant in order to ensure that the change in the appearance is due solely to changes in third and other higher order statistics. Stimuli comprise both natural and synthetically generated naturalistic images, with the latter being used to prevent observers from making pixel-wise comparisons. A difference scaling method is used to derive the perceptual scales for each observer, which show a sigmoidal relationship with the degree of randomization. Observers were maximally sensitive to changes within the 20%-60% randomization range. In order to account for this behavior we propose a biologically plausible model that computes the variance of local measurements of phase congruency.

Subcortical representation of non-Fourier image features

A fundamental goal of visual neuroscience is to identify the neural pathways representing different image features. It is widely argued that the early stages of these pathways represent linear features of the visual scene and that the nonlinearities necessary to represent complex visual patterns are introduced later in cortex. We tested this by comparing the responses of subcortical and cortical neurons to interference patterns constructed by summing sinusoidal gratings. Although a linear mechanism can detect the component gratings, a nonlinear mechanism is required to detect an interference pattern resulting from their sum. Consistent with in vitro retinal ganglion cell recordings, we found that interference patterns are represented subcortically by cat LGN Y-cells, but not X-cells. Linear and nonlinear tuning properties of LGN Y-cells were then characterized and compared quantitatively with those of cortical area 18 neurons responsive to interference patterns. This comparison revealed a high degree of similarity between the two neural populations, including the following: (1) the representation of similar spatial frequencies in both their linear and nonlinear responses, (2) comparable orientation selectivity for the high spatial frequency carrier of interference patterns, and (3) the same difference in their temporal frequency selectivity for drifting gratings versus the envelope of interference patterns. The present findings demonstrate that the nonlinear subcortical Y-cell pathway represents complex visual patterns and likely underlies cortical responses to interference patterns. We suggest that linear and nonlinear mechanisms important for encoding visual scenes emerge in parallel through distinct pathways originating at the retina.

Influence of parallel and orthogonal real lines on illusory contour perception

Real lines and illusory contours (ICs) have been reported to either interfere with or facilitate the perception of the other, depending on real line orientation and contrast. Here we investigate contextual effects of real lines on illusory contour perception. Curvature discrimination thresholds of Kanizsa-contours were measured for superimposed real lines of different sub- and suprathreshold contrasts. We find that parallel lines interfere with curvature discrimination at suprathreshold, whereas orthogonal lines interfere at subthreshold contrasts. We did not find stable facilitating effects of lines in any orientation or contrast. These results are discussed in relation to existing physiological and imaging data.

Surround suppression of V1 neurons mediates orientation-based representation of high-order visual features

Neurons with surround suppression have been implicated in processing high-order visual features such as contrast- or texture-defined boundaries and subjective contours. However, little is known regarding how these neurons encode high-order visual information in a systematic manner as a population. To address this issue, we have measured detailed spatial structures of classical center and suppressive surround regions of receptive fields of primary visual cortex (V1) neurons and examined how a population of such neurons allow encoding of various high-order features and shapes in visual scenes. Using a novel method to reconstruct structures, we found that the center and surround regions are often both elongated parallel to each other, reminiscent of on and off subregions of simple cells without surround suppression. These structures allow V1 neurons to extract high-order contours of various orientations and spatial frequencies, with a variety of optimal values across neurons. The results show that a wide range of orientations and widths of the high-order features are systematically represented by the population of V1 neurons with surround suppression.

Distinct perceptual grouping pathways revealed by temporal carriers and envelopes

S. E. Guttman, L. A. Gilroy, and R. Blake (2005) investigated whether observers could perform temporal grouping in multi-element displays where each local element was stochastically modulated over time along one of several potential dimensions--or "messenger types"--such as contrast, position, orientation, or spatial scale. Guttman et al.'s data revealed that grouping discards messenger type and therefore support a single-pathway model that groups elements with similar temporal waveforms. In the current study, we carried out three experiments in which temporal-grouping information resided either in the carrier, the envelope, or the combined carrier and envelope of each messenger's timecourse. Results revealed that grouping is highly specific for messenger type if carrier envelopes lack grouping information but largely messenger nonspecific if carrier envelopes contain grouping information. These imply that temporal grouping is mediated by several messenger-specific carrier pathways as well as by a messenger-nonspecific envelope pathways. Findings also challenge simple temporal-filtering accounts of perceptual grouping (E. H. Adelson & H. Farid, 1999).

Texture modulation detection by probability summation among orientation-selective and isotropic mechanisms

Substantial evidence has accumulated for the notion that modulations of second-order properties in the visual scene are processed by mechanisms which detect contrast variations within narrow orientation/spatial frequency channels. It has also been suggested that mechanisms exist which detect contrast modulations across all orientations. Many naturally occurring texture variations (e.g., modulations in orientation and/or spatial frequency) involve simultaneous contrast modulations in multiple channels. Contrasting conclusions have been drawn regarding the manner in which the information carried in multiple channels is combined. In a series of two experiments it is shown that simultaneous contrast modulations in two narrow orientation bands are detected by three mechanisms, two of which detect contrast modulations within the modulated bands only, the third of which integrates contrast across orientations in order to detect modulations of overall contrast. The three mechanisms combine their efforts by probability summation.

Neuronal response to texture- and contrast-defined boundaries in early visual cortex

Natural scenes contain a variety of visual cues that facilitate boundary perception (e.g., luminance, contrast, and texture). Here we explore whether single neurons in early visual cortex can process both contrast and texture cues. We recorded neural responses in cat A18 to both illusory contours formed by abutting gratings (ICs, texture-defined) and contrast-modulated gratings (CMs, contrast-defined). We found that if a neuron responded to one of the two stimuli, it also responded to the other. These neurons signaled similar contour orientation, spatial frequency, and movement direction of the two stimuli. A given neuron also exhibited similar selectivity for spatial frequency of the fine, stationary grating components (carriers) of the stimuli. These results suggest that the cue-invariance of early cortical neurons extends to different kinds of texture or contrast cues, and might arise from a common nonlinear mechanism.

Critical spatial frequencies for illusory contour processing in early visual cortex

Single neurons in primate V2 and cat A18 exhibit identical orientation tuning for sinewave grating and illusory contour stimuli. This cue invariance is also manifested in similar orientation maps to these stimuli, but in V1/A17 the illusory contour maps appear reversed. We hypothesized that this map reversal depends upon the spatial frequencies of the inducers in the illusory contours, relative to the spatial selectivities of these brain areas. We employed intrinsic signal optical imaging to measure orientation maps in cat A17/18 to illusory contours with inducers at spatial frequencies from 0.15 to 1.6 cpd. A17 illusory contour maps were indeed reversed compared with grating-driven maps for inducer spatial frequencies <1.3 cpd, whereas A18 maps were invariant. Simulations based on known neurophysiology demonstrated that map reversal can arise from linear filtering, and map invariance can be explained by a nonlinear (filter-rectify-filter) mechanism. The simulation also correctly predicted that A17 could show invariant maps when the inducer spatial frequency is sufficiently high (1.6 cpd), and that A18 maps could reverse at lower inducer frequencies (0.18 cpd). Thus, the map reversal or invariance to illusory contours depends critically on the relationship of the inducer spatial frequencies to the spatial filtering properties of neurons in each brain area.

Contour discontinuities subserve two types of form analysis that underlie motion processing

Form analysis subserves motion processing in at least two ways: first, in terms of figural segmentation dedicated to solving the problem of figure-to-figure matching over time, and second, in terms of defining trackable features whose unambiguous motion signals can be generalized to ambiguously moving portions of an object. The former is a primarily ventral process involving the lateral occipital complex and also retinotopic areas such as V2 and V4, and the latter is a dorsal process involving V3A. Contour discontinuities, such as corners, deep concavities, maxima of positive curvature, junctions, and terminators, play a central role in both types of form analysis. Transformational apparent motion will be discussed in the context of figural segmentation and matching, and rotational motion in the context of trackable features. In both cases the analysis of form must proceed in parallel with the analysis of motion, in order to constrain the ongoing analysis of motion.

FMRI adaptation reveals separate mechanisms for first-order and second-order motion

A key unresolved debate in human vision concerns whether we have two different low-level mechanisms for encoding image motion. Separate neural mechanisms have been suggested for first-order (luminance modulation) and second-order (e.g., contrast modulation) motion in the retinal image but a single mechanism could handle both. Human functional magnetic resonance imaging (fMRI) has not so far convincingly revealed separate anatomical substrates. To examine whether two separate but co-localized mechanisms might exist, we used the technique of fast fMRI adaptation. We found direction-selective adaptation independently for each type of motion in the motion area V5/MT+ of the human brain. However, there was a total absence of cross-adaptation between first-order and second-order motion stimuli. This was true in both of the two subcomponents of MT+ (MT and MST) and similar results were found in V3A. This pattern of adaptation was consistent with psychophysical measurements of detection thresholds in similar stimulus sequences. The results provide strong evidence for separate neural populations that are responsible for detecting first- and second-order motion.

Neural mechanisms mediating responses to abutting gratings: luminance edges vs. illusory contours

The discontinuities of phase-shifted abutting line gratings give rise to perception of an "illusory contour" (IC) along the line terminations. Neuronal responses to such ICs have been interpreted as evidence for a specialized visual mechanism, since such responses cannot be predicted from conventional linear receptive fields. However, when the spatial scale of the component gratings (carriers) is large compared to the neuron's luminance passband, these IC responses might be evoked simply by the luminance edges at the line terminations. Thus by presenting abutting gratings at a series of carrier spatial scales to cat A18 neurons, we were able to distinguish genuine nonlinear responses from those due to luminance edges. Around half of the neurons (both simple and complex types) showed a bimodal response pattern to abutting gratings: one peak at a low carrier spatial frequency range that overlapped with the luminance passband, and a second distinct peak at much higher frequencies beyond the neuron's grating resolution. For those bimodally responding neurons, the low-frequency responses were sensitive to carrier phase, but the high-frequency responses were phase-invariant. Thus the responses at low carrier spatial frequencies could be understood via a linear model, while the higher frequency responses represented genuine nonlinear IC processing. IC responsive neurons also demonstrated somewhat lower spatial preference to the periodic contours (envelopes) compared to gratings, but the optimal orientation and motion direction for both were quite similar. The nonlinear responses to ICs could be explained by the same energy mechanism underlying responses to second-order stimuli such as contrast-modulated gratings. Similar neuronal preferences for ICs and for gratings may contribute to the form-cue invariant perception of moving contours.

Second-order spatial frequency and orientation channels in human vision

We compared the number of spatial frequency and orientation mechanisms underlying first- versus second-order processing by measuring discrimination at detection threshold for first- and second-order Gabors to determine the smallest difference in spatial frequency and orientation that permits accurate discrimination at threshold. For second-order gratings, the number of channels is the same as for first-order gratings for spatial frequencies up to about 2 cpd; however, there are fewer second-order channels at higher spatial frequencies. In contrast, the number of labeled channels for orientation is the same for first- and second-order gratings. In conclusion, our findings provide evidence for distinct spatial frequency and orientation labeled detectors in second-order visual processing. We also show that, relative to first-order, there are fewer second-order channels processing higher spatial frequencies. This is consistent with a filter-rectify-filter scheme for second-order in which the second stage of filtering is at lower spatial frequencies.

Greater losses in sensitivity to second-order local motion than to first-order local motion after early visual deprivation in humans

We compared sensitivity to first-order versus second-order local motion in patients treated for dense central congenital cataracts in one or both eyes. Amplitude modulation thresholds were measured for discriminating the direction of motion of luminance-modulated (first-order) and contrast modulated (second-order) horizontal sine-wave gratings. Early visual deprivation, whether monocular or binocular, caused losses in sensitivity to both first- and second-order motion, with greater losses for second-order motion than for first-order motion. These findings are consistent with the hypothesis that the two types of motion are processed by different mechanisms and suggest that those mechanisms are differentially sensitive to early visual input.

Orientation-selective adaptation to first- and second-order patterns in human visual cortex

Second-order textures-patterns that cannot be detected by mechanisms sensitive only to luminance changes-are ubiquitous in visual scenes, but the neuronal mechanisms mediating perception of such stimuli are not well understood. We used an adaptation protocol to measure neural activity in the human brain selective for the orientation of second-order textures. Functional MRI (fMRI) responses were measured in three subjects to presentations of first- and second-order probe gratings after adapting to a high-contrast first- or second-order grating that was either parallel or orthogonal to the probe gratings. First-order (LM) stimuli were generated by modulating the stimulus luminance. Second-order stimuli were generated by modulating the contrast (CM) or orientation (OM) of a first-order carrier. We used four combinations of adapter and probe stimuli: LM:LM, CM:CM, OM:OM, and LM:OM. The fourth condition tested for cross-modal adaptation with first-order adapter and second-order probe stimuli. Attention was diverted from the stimulus by a demanding task at fixation. Both first- and second-order stimuli elicited orientation-selective adaptation in multiple cortical visual areas, including V1, V2, V3, V3A/B, a newly identified visual area anterior to dorsal V3 that we have termed LO1, hV4, and VO1. For first-order stimuli (condition LM:LM), the adaptation was no larger in extrastriate areas than in V1, implying that the orientation-selective first-order (luminance) adaptation originated in V1. For second-order stimuli (conditions CM:CM and OM:OM), the magnitude of adaptation, relative to the absolute response magnitude, was significantly larger in VO1 (and for condition CM:CM, also in V3A/B and LO1) than in V1, suggesting that second-order stimulus orientation was extracted by additional processing after V1. There was little difference in the amplitude of adaptation between the second-order conditions. No consistent effect of adaptation was found in the cross-modal condition LM:OM, in agreement with psychophysical evidence for weak interactions between first- and second-order stimuli and computational models of separate mechanisms for first- and second-order visual processing.

Second-order spatial summation in amblyopia

Amblyopes show bilateral loss of sensitivity for second-order (contrast defined) stimuli that can be further suppressed by flanking second-order stimuli (whereas flanks facilitate sensitivity in normal observers). The suppressive flank effect in amblyopes might be explained by abnormal pooling of second-order contrast across visual space. In this study, we investigate whether amblyopes show abnormal second-order spatial summation by measuring contrast detection thresholds for 1c/deg modulations of random noise (stimuli 1-12 cycles) in amblyopic observers, strabismic observers with no visual acuity loss, and normal (control) observers. Non-control observers showed substantial bilateral loss of sensitivity relative to the control observers, as expected. However, all observers showed essentially equal second-order spatial summation: contrast detection threshold decreased with approximately the square root of the number of cycles, and then became independent of size at 6-8 cycles (similar asymptotes). We conclude that the pooling of second-order contrast across visual space is unaffected by amblyopia.

Boundary cue invariance in cortical orientation maps

We effortlessly perceive oriented boundaries defined by either luminance changes ('first-order' cues) or texture variations ('second-order' cues). Many neurons in mammalian visual cortex show orientation preference to both types of boundaries, but it is uncertain how they contribute to perceptual orientation cue-invariance at the neuronal population level. Using optical imaging in cat A 18, we observed highly similar orientation preference maps to first-order and a variety of second-order visual stimuli. Thus the neuronal representation of coarse-scale boundary orientation appears to be invariant to the characteristics (including local orientation) of the fine-scale textures by which those boundaries are defined. A common feature of second-order visual stimuli is that modulation shifts their Fourier energy for boundary orientation to the higher spatial frequencies of their constituent textures - our results suggest a common neural mechanism (demodulation) mediating visual processing of many kinds of texture boundary. The similarity between orientation maps to different stimuli implies that second-order responsive neurons are homogeneously distributed across the cortical surface. Such homogeneously cue-invariant orientation representation could provide a neural substrate for perceptual form-cue invariance, and reflect an optimal organization for encoding orientation information in natural scenes.

The direction-selective contrast response of area 18 neurons is different for first- and second-order motion

Cortical neurons selective for the direction of motion often exhibit some limited response to motion in their nonpreferred directions. Here we examine the dependence of neuronal direction selectivity on stimulus contrast, both for first-order (luminance-modulated, sine-wave grating) and second-order (contrast-modulated envelope) stimuli. We measured responses from single neurons in area 18 of cat visual cortex to both kinds of moving stimuli over a wide range of contrasts (1.25-80%). Direction-selective contrast response functions (CRFs) were calculated as the preferred-minus-null difference in average firing frequency as a function of contrast. We also applied receiver operating characteristic analysis to our CRF data to obtain neurometric functions characterizing the potential ability of each neuron to discriminate motion direction at each contrast level tested. CRFs for sine-wave gratings were usually monotonic; however, a substantial minority of neurons (35%) exhibited nonmonotonic CRFs (such that the degree of direction selectivity decreased with increasing contrast). The underlying preferred and nonpreferred direction CRFs were diverse, often having different shapes in a given neuron. Neurometric functions for direction discrimination showed a similar degree of heterogeneity, including instances of nonmonotonicity. For contrast-modulated stimuli, however, CRFs for either carrier or envelope contrast were always monotonic. In a given neuron, neurometric thresholds were typically much higher for second- than for first-order stimuli. These results demonstrate that the degree of a cell's direction selectivity depends on the contrast at which it is measured, and therefore is not a characteristic parameter of a neuron. In general, contrast response functions for first-order stimuli were very heterogeneous in shape and sensitivity, while those for second-order stimuli showed less sensitivity and were quite stereotyped in shape.

“Crowding” in normal and amblyopic vision assessed with Gaussian and Gabor C’s

The purpose of this study was to investigate the extent and specificity of crowding in the normal fovea and periphery, and the central field of amblyopes, using "C"-like patterns. In the first experiment we measured the extent of crowding for C-patterns comprised of Gaussian patches, over a range of target sizes using a four-alternative forced-choice (up, down, left, right) method. We found that the extent of foveal crowding is proportional to target size. In contrast, in normal periphery and in the central field of amblyopes, crowding extends over large spatial distances and is not size dependent. Crowding for our stimuli occurred with both same-polarity and opposite polarity patches. To test whether the extended crowding in amblyopia resulted from a shift in the spatial scale of analysis, we measured crowding with band-limited C-patterns (comprised of Gabor patches) in a gap localization task (2-AFC). With band-limited stimuli, and a task that does not involve judging the orientation of the gap, the amblyopic eyes showed crowding over a longer distance than that of normal observers. We also tested the orientation specificity of crowding by varying the orientation of the flanks. In normal fovea, crowding is orientation specific: in amblyopia it is not. While crowding in normal fovea can be explained by simple pattern masking, crowding seen in normal periphery and amblyopes cannot. Instead we suggest that crowding in amblyopic and peripheral vision is a result of extended pooling at a stage following the stage of feature detection.

Deficits in the processing of local and global motion in very low birthweight children

This study evaluated the impact of premature birth on the development of local and global motion processing in a group of very low birthweight (<1500 g), 5- to 8-year-old children. Sensitivity to first- and second-order local motion stimuli and coherence thresholds for global motion in random dot kinematograms were measured. Relative to full-term controls, premature children showed deficits on all three aspects of motion processing. These problems could not be accounted for by stereo deficits, amblyopia, or attentional problems. A history of mild retinopathy of prematurity and/or intraventricular hemorrhage increased risk, but deficits were observed in some children with no apparent ocular or cerebral pathology. It is important to note that, despite the observed group differences, individual profiles of performance did vary; the results suggest that these three forms of motion processing may involve separate neural mechanisms. These findings serve to increase our understanding of the organization and functional development of motion-processing subsystems in humans, and of the impact of prematurity and associated complications on visual development.

Putting order into the development of sensitivity to global motion

We studied differences in the development of sensitivity to first-versus second-order global motion by comparing the motion coherence thresholds of 5-year-olds and adults tested at three speeds (1.5, 6, and 9 degrees s(-1)). We used Random Gabor Kinematograms (RGKs) formed with luminance-modulated (first-order) or contrast-modulated (second-order) concentric Gabor patterns with a sinusoidal spatial frequency of 3c deg(-1). To achieve equal visibility, modulation depth was set at 30% for first-order Gabors and at 100%, for second-order Gabors. Subjects were 24 adults and 24 5-year-olds. For both first- and second-order global motion, the motion coherence threshold of 5-year-olds was less mature for the slowest speed (1.5 degrees s(-1)) than for the two faster speeds (6 and 9 degrees s(-1)). In addition, at the slowest speed, the immaturity was greater for second-order than for first-order global motion. The findings suggest that the extrastriate mechanisms underlying the perception of global motion are different, at least in part, for first- versus second-order signals and for slower versus faster speeds. They also suggest that those separate mechanisms mature at different rates during middle childhood.

Investigating local network interactions underlying first- and second-order processing

We compared the spatial lateral interactions for first-order cues to those for second-order cues, and investigated spatial interactions between these two types of cues. We measured the apparent modulation depth of a target Gabor at fixation, in the presence and the absence of horizontally flanking Gabors. The Gabors' gratings were either added to (first-order) or multiplied with (second-order) binary 2-D noise. Apparent "contrast" or modulation depth (i.e., the perceived difference between the high and low luminance regions for the first-order stimulus, or between the high and low contrast regions for the second-order stimulus) was measured with a modulation depth-matching paradigm. For each observer, the first- and second-order Gabors were equated for apparent modulation depth without the flankers. Our results indicate that at the smallest inter-element spacing, the perceived reduction in modulation depth is significantly smaller for the second-order than for the first-order stimuli. Further, lateral interactions operate over shorter distances and the spatial frequency and orientation tuning of the suppression effect are broader for second- than first-order stimuli. Finally, first- and second-order information interact in an asymmetrical fashion; second-order flankers do not reduce the apparent modulation depth of the first-order target, whilst first-order flankers reduce the apparent modulation depth of the second-order target.

First- and second-order information in natural images: a filter-based approach to image statistics

Previous analyses of natural image statistics have dealt mainly with their Fourier power spectra. Here we explore image statistics by examining responses to biologically motivated filters that are spatially localized and respond to first-order (luminance-defined) and second-order (contrast- or texture-defined) characteristics. We compare the distribution of natural image responses across filter parameters for first- and second-order information. We find that second-order information in natural scenes shows the same self-similarity previously described for first-order information but has substantially less orientational anisotropy. The magnitudes of the two kinds of information, as well as their mutual unsigned correlation, are much stronger for particular combinations of filter parameters in natural images but not in unstructured fractal images having the same power spectra.

How is complex second-order motion processed?

Converging psychophysical and electrophysiological evidence suggests that first-order (luminance-defined) complex motion types i.e., radial and rotational motion, are processed by specialized extrastriate motion mechanisms. We ask whether radial and rotational second-order (texture-defined) motion patterns are processed in a similar manner. The motion sensitivity to translating, radiating and rotating motion patterns of both first-order (luminance-modulated noise) and second-order (contrast-modulated noise) were measured for patterns presented at four different exposure durations (106, 240, 500 and 750 ms). No significant difference in motion sensitivity was found across motion type for the first-order motion class across exposure duration (i.e., from 240 to 750 ms) whereas direction-identification thresholds for radiating and rotating second-order motion were significantly greater than that of the second-order translational stimuli. Furthermore, thresholds to all second-order motion stimuli increased at a significantly faster rate with decreasing exposure duration compared to those of first-order motion. Interestingly, simple and complex second-order thresholds increased at similar rates. Taken together, the results suggest that complex second-order motion is not analyzed in a sequential manner. Rather, it seems that the same 'hard-wired' mechanisms responsible for complex first-order motion processing also mediate complex second-order motion, but not before the pre-processing (i.e., rectification) of local second-order motion signals.

Longer VEP latencies and slower reaction times to the onset of second-order motion than to the onset of first-order motion

We compared visual evoked potentials and psychophysical reaction times to the onset of first- and second-order motion. The stimuli consisted of luminance-modulated (first-order) and contrast-modulated (second-order) 1 cpd vertical sine-wave gratings drifting rightward for 140 ms at a velocity of 6 degrees /s. For each condition, we analysed the latencies and peak-to-baseline amplitudes of the P1 and N2 peaks recorded at Oz. For first-order motion, both P1 and N2 peaks were present at low (3%) contrast (i.e., depth modulations) whereas for second-order motion they appeared only at higher (25%) contrasts. When the two types of motion were equated for visibility, responses were slower for second-order motion than for first-order motion: about 44 ms slower for P1 latencies, 53 ms slower for N2 latencies, and 76 ms slower for reaction times. The longer VEP latencies for second-order motion support models that postulate additional processing steps for the extraction of second-order motion. The slower reaction time to the onset of second-order motion suggests that the longer neurophysiological analysis translates into slower detection.

The duration of 3-d form analysis in transformational apparent motion

Transformational apparent motion (TAM) occurs when a figure changes discretely from one configuration to another overlapping configuration. Rather than an abrupt shape change, the initial shape is perceived to transform smoothly into the final shape as if animated by a series of intermediate shapes. We find that TAM follows an analysis of form that takes 80-140 msec. Form analysis can function both at and away from equiluminance and can occur over contours defined by uniform regions as well as outlines. Moreover, the forms analyzed can be 3-D, resulting in motion paths that appear to smoothly project out from or into the stimulus plane. The perceived transformation is generally the one that involves the least change in the shape or location of the initial figure in a 3-D sense. We conclude that perception of TAM follows an analysis of 3-D form that takes approximately 100 msec. This stage of form analysis may be common to both TAM and second-order motion.

Processing of second-order stimuli in the visual cortex

Naturally occurring visual stimuli are rich in examples of objects delineated from their backgrounds simply by differences in luminance, so-called first-order stimuli, as well as those defined by differences of contrast or texture, referred to as second-order stimuli. Here we provide a brief overview of visual cortical processing of second-order stimuli, as well as some comparative background on first-order processing, concentrating on single-unit neurophysiology, but also discussing relationships to human psychophysics and to neuroimaging. The selectivity of visual cortical neurons to orientation, spatial frequency, and direction of movement of first-order, luminance-defined stimuli is conventionally understood in terms of simple linear filter models, albeit with some minor nonlinearities such as thresholding and gain control. However, these kinds of models fail entirely to account for responses of neurons to second-order stimuli such as contrast envelopes, illusory contours, or texture borders. Second-order stimuli constructed from sinusoidal components have been used to analyze the neurophysiological mechanisms of such responses; these experiments demonstrate that the same neuron can exhibit three distinct kinds of tuning to spatial frequency, and also to orientation. These results can be understood in terms of a type of nonlinear 'filter-->rectify-->filter' model, which has been widely used in human psychophysics. Finally, several general issues will be discussed, including potential artifacts in experiments with second-order stimuli, and strategies for avoiding or controlling for them; caveats about definitions of first- vs. second-order mechanisms and stimuli; the concept of form-cue invariance; and the functional significance of second-order processing.

Cellular basis for the response to second-order motion cues in Y retinal ganglion cells

We perceive motion when presented with spatiotemporal changes in contrast (second-order cue). This requires linear signals to be rectified and then summed in temporal order to compute direction. Although both operations have been attributed to cortex, rectification might occur in retina, prior to the ganglion cell. Here we show that the Y ganglion cell does indeed respond to spatiotemporal contrast modulations of a second-order motion stimulus. Responses in an OFF ganglion cell are caused by an EPSP/IPSP sequence evoked from within the dendritic field; in ON cells inhibition is indirect. Inhibitory effects, which are blocked by tetrodotoxin, clamp the response near resting potential thus preventing saturation. Apparently the computation for second-order motion can be initiated by Y cells and completed by cortical cells that sum outputs of multiple Y cells in a directionally selective manner.