A cortical locus for the processing of contrast-defined contours

Influences of local and global context on local orientation perception

Visual context modulates perception of local orientation attributes. These spatially very localised effects are considered to correspond to specific excitatory-inhibitory connectivity patterns of early visual areas as V1, creating perceptual tilt repulsion and attraction effects. Here, orientation misperception of small Gabor stimuli was used as a probe of this computational structure by sampling a large spatio-orientation space to reveal expected asymmetries due to the underlying neuronal processing. Surprisingly, the results showed a regular iso-orientation pattern of nearby location effects whose reference point was globally modulated by the spatial structure, without any complex interactions between local positions and orientation. This pattern of results was confirmed by the two perceptual parameters of bias and discrimination ability. Furthermore, the response times to stimulus configuration displayed variations that further provided evidence of how multiple early visual stages affect perception of simple stimuli.

Segmenting surface boundaries using luminance cues

Segmenting scenes into distinct surfaces is a basic visual perception task, and luminance differences between adjacent surfaces often provide an important segmentation cue. However, mean luminance differences between two surfaces may exist without any sharp change in albedo at their boundary, but rather from differences in the proportion of small light and dark areas within each surface, e.g. texture elements, which we refer to as a luminance texture boundary. Here we investigate the performance of human observers segmenting luminance texture boundaries. We demonstrate that a simple model involving a single stage of filtering cannot explain observer performance, unless it incorporates contrast normalization. Performing additional experiments in which observers segment luminance texture boundaries while ignoring super-imposed luminance step boundaries, we demonstrate that the one-stage model, even with contrast normalization, cannot explain performance. We then present a Filter–Rectify–Filter model positing two cascaded stages of filtering, which fits our data well, and explains observers' ability to segment luminance texture boundary stimuli in the presence of interfering luminance step boundaries. We propose that such computations may be useful for boundary segmentation in natural scenes, where shadows often give rise to luminance step edges which do not correspond to surface boundaries.

Predicting how surface texture and shape combine in the human visual system to direct attention

Objects differ from one another along a multitude of visual features. The more distinct an object is from other objects in its surroundings, the easier it is to find it. However, it is still unknown how this distinctiveness advantage emerges in human vision. Here, we studied how visual distinctiveness signals along two feature dimensions—shape and surface texture—combine to determine the overall distinctiveness of an object in the scene. Distinctiveness scores between a target object and distractors were measured separately for shape and texture using a search task. These scores were then used to predict search times when a target differed from distractors along both shape and texture. Model comparison showed that the overall object distinctiveness was best predicted when shape and texture combined using a Euclidian metric, confirming the brain is computing independent distinctiveness scores for shape and texture and combining them to direct attention.

A target contrast signal theory of parallel processing in goal-directed search

Feature Integration Theory (FIT) set out the groundwork for much of the work in visual cognition since its publication. One of the most important legacies of this theory has been the emphasis on feature-specific processing. Nowadays, visual features are thought of as a sort of currency of visual attention (e.g., features can be attended, processing of attended features is enhanced), and attended features are thought to guide attention towards likely targets in a scene. Here we propose an alternative theory - the Target Contrast Signal Theory - based on the idea that when we search for a specific target, it is not the target-specific features that guide our attention towards the target; rather, what determines behavior is the result of an active comparison between the target template in mind and every element present in the scene. This comparison occurs in parallel and is aimed at rejecting from consideration items that peripheral vision can confidently reject as being non-targets. The speed at which each item is evaluated is determined by the overall contrast between that item and the target template. We present computational simulations to demonstrate the workings of the theory as well as eye-movement data that support core predictions of the theory. The theory is discussed in the context of FIT and other important theories of visual search.

Proto-Object Based Saliency Model With Texture Detection Channel

The amount of visual information projected from the retina to the brain exceeds the information processing capacity of the latter. Attention, therefore, functions as a filter to highlight important information at multiple stages of the visual pathway that requires further and more detailed analysis. Among other functions, this determines where to fixate since only the fovea allows for high resolution imaging. Visual saliency modeling, i.e. understanding how the brain selects important information to analyze further and to determine where to fixate next, is an important research topic in computational neuroscience and computer vision. Most existing bottom-up saliency models use low-level features such as intensity and color, while some models employ high-level features, like faces. However, little consideration has been given to mid-level features, such as texture, for visual saliency models. In this paper, we extend a biologically plausible proto-object based saliency model by adding simple texture channels which employ nonlinear operations that mimic the processing performed by primate visual cortex. The extended model shows statistically significant improved performance in predicting human fixations compared to the previous model. Comparing the performance of our model with others on publicly available benchmarking datasets, we find that our biologically plausible model matches the performance of other models, even though those were designed entirely for maximal performance with little regard to biological realism.

Integration of contours defined by second-order contrast-modulation of texture

Boundaries in the visual world can be defined by changes in luminance and texture in the input image. A "contour integration" process joins together local changes into percepts of lines or edges. A previous study tested the integration of contours defined by second-order contrast-modulation. Their contours were placed in a background of random wavelets. Participants performed near chance. We re-visited second-order contour integration with a different task. Participants distinguished contours with "good continuation" from distractors. We measured thresholds in different amounts of external orientation or position noise. This gave two noise-masking functions. We also measured thresholds for contours with a baseline curvature to assess performance with more curvy targets. Our participants were able to discriminate the good continuation of second-order contours. Thresholds were higher than for first-order contours. In our modelling, we found this was due to multiple factors. There was a doubling of equivalent internal noise between first- and second-order contour integration. There was also a reduction in efficiency. The efficiency difference was only significant in our orientation noise condition. For both first- and second-order stimuli, participants were also able to perform our task with more curved contours. We conclude that humans can integrate second-order contours, even when they are curved. There is however reduced performance compared to first-order contours. We find both an impaired input to the integrating mechanism, and reduced efficiency seem responsible. Second-order contour integration may be more affected by the noise background used in the previous study. Difficulty segregating that background may explain their result.

Predicting how color and shape combine in the human visual system to direct attention

Objects in a scene can be distinct from one another along a multitude of visual attributes, such as color and shape, and the more distinct an object is from its surroundings, the easier it is to find it. However, exactly how this distinctiveness advantage arises in vision is not well understood. Here we studied whether and how visual distinctiveness along different visual attributes (color and shape, assessed in four experiments) combine to determine an object’s overall distinctiveness in a scene. Unidimensional distinctiveness scores were used to predict performance in six separate experiments where a target object differed from distractor objects along both color and shape. Results showed that there is mathematical law determining overall distinctiveness as the simple sum of the distinctiveness scores along each visual attribute. Thus, the brain must compute distinctiveness scores independently for each visual attribute before summing them into the overall score that directs human attention.

Modeling second-order boundary perception: A machine learning approach

Visual pattern detection and discrimination are essential first steps for scene analysis. Numerous human psychophysical studies have modeled visual pattern detection and discrimination by estimating linear templates for classifying noisy stimuli defined by spatial variations in pixel intensities. However, such methods are poorly suited to understanding sensory processing mechanisms for complex visual stimuli such as second-order boundaries defined by spatial differences in contrast or texture. We introduce a novel machine learning framework for modeling human perception of second-order visual stimuli, using image-computable hierarchical neural network models fit directly to psychophysical trial data. This framework is applied to modeling visual processing of boundaries defined by differences in the contrast of a carrier texture pattern, in two different psychophysical tasks: (1) boundary orientation identification, and (2) fine orientation discrimination. Cross-validation analysis is employed to optimize model hyper-parameters, and demonstrate that these models are able to accurately predict human performance on novel stimulus sets not used for fitting model parameters. We find that, like the ideal observer, human observers take a region-based approach to the orientation identification task, while taking an edge-based approach to the fine orientation discrimination task. How observers integrate contrast modulation across orientation channels is investigated by fitting psychophysical data with two models representing competing hypotheses, revealing a preference for a model which combines multiple orientations at the earliest possible stage. Our results suggest that this machine learning approach has much potential to advance the study of second-order visual processing, and we outline future steps towards generalizing the method to modeling visual segmentation of natural texture boundaries. This study demonstrates how machine learning methodology can be fruitfully applied to psychophysical studies of second-order visual processing.

Many naturally occurring visual boundaries are defined by spatial differences in features other than luminance, for example by differences in texture or contrast. Quantitative models of such “second-order” boundary perception cannot be estimated using the standard regression techniques (known as “classification images”) commonly applied to “first-order”, luminance-defined stimuli. Here we present a novel machine learning approach to modeling second-order boundary perception using hierarchical neural networks. In contrast to previous quantitative studies of second-order boundary perception, we directly estimate network model parameters using psychophysical trial data. We demonstrate that our method can reveal different spatial summation strategies that human observers utilize for different kinds of second-order boundary perception tasks, and can be used to compare competing hypotheses of how contrast modulation is integrated across orientation channels. We outline extensions of the methodology to other kinds of second-order boundaries, including those in natural images.

Stimulus vignetting and orientation selectivity in human visual cortex

Neural selectivity to orientation is one of the simplest and most thoroughly-studied cortical sensory features. Here, we show that a large body of research that purported to measure orientation tuning may have in fact been inadvertently measuring sensitivity to second-order changes in luminance, a phenomenon we term 'vignetting'. Using a computational model of neural responses in primary visual cortex (V1), we demonstrate the impact of vignetting on simulated V1 responses. We then used the model to generate a set of predictions, which we confirmed with functional MRI experiments in human observers. Our results demonstrate that stimulus vignetting can wholly determine the orientation selectivity of responses in visual cortex measured at a macroscopic scale, and suggest a reinterpretation of a well-established literature on orientation processing in visual cortex.

Descending pathways generate perception of and neural responses to weak sensory input

Natural sensory stimuli frequently consist of a fast time-varying waveform whose amplitude or contrast varies more slowly. While changes in contrast carry behaviorally relevant information necessary for sensory perception, their processing by the brain remains poorly understood to this day. Here, we investigated the mechanisms that enable neural responses to and perception of low-contrast stimuli in the electrosensory system of the weakly electric fish Apteronotus leptorhynchus. We found that fish reliably detected such stimuli via robust behavioral responses. Recordings from peripheral electrosensory neurons revealed stimulus-induced changes in firing activity (i.e., phase locking) but not in their overall firing rate. However, central electrosensory neurons receiving input from the periphery responded robustly via both phase locking and increases in firing rate. Pharmacological inactivation of feedback input onto central electrosensory neurons eliminated increases in firing rate but did not affect phase locking for central electrosensory neurons in response to low-contrast stimuli. As feedback inactivation eliminated behavioral responses to these stimuli as well, our results show that it is changes in central electrosensory neuron firing rate that are relevant for behavior, rather than phase locking. Finally, recordings from neurons projecting directly via feedback to central electrosensory neurons revealed that they provide the necessary input to cause increases in firing rate. Our results thus provide the first experimental evidence that feedback generates both neural and behavioral responses to low-contrast stimuli that are commonly found in the natural environment.

Feedback input from more central to more peripheral brain areas is found ubiquitously in the central nervous system of vertebrates. In this study, we used a combination of electrophysiological, behavioral, and pharmacological approaches to reveal a novel function for feedback pathways in generating neural and behavioral responses to weak sensory input in the weakly electric fish. We first determined that weak sensory input gives rise to responses that are phase locked in both peripheral sensory neurons and in the central neurons that are their downstream targets. However, central neurons also responded to weak sensory inputs that were not relayed via a feedforward input from the periphery, because complete inactivation of the feedback pathway abolished increases in firing rate but not the phase locking in response to weak sensory input. Because such inactivation also abolished the behavioral responses, our results show that the increases in firing rate in central neurons, and not the phase locking, are decoded downstream to give rise to perception. Finally, we discovered that the neurons providing feedback input were also activated by weak sensory input, thereby offering further evidence that feedback is necessary to elicit increases in firing rate that are needed for perception.

Apparent shift in long-range motion trajectory by local pattern orientation

The present study shows that the apparent direction of a moving pattern is systematically affected by its orientation. We found that the perceived direction of motion of a single Gabor grating changing position in discrete steps interleaved by blank inter-stimulus interval (ISI) is biased toward the orientation of the grating. This orientation-induced motion shift peaks for grating orientations ~±15 deg away from the physical motion trajectory and was profound for relatively short distances. Orientation adaptation revealed that the directional shift is determined by the apparent –not the physical –orientation of the grating, and a subsequent experiment demonstrated that directional shift is also influenced by the orientation of the contrast-defined stimulus envelope. Results provide further evidence that the apparent trajectory of a motion stimulus is determined by interactions between motion and pattern information at relatively high levels of visual processing.

Nonlinear Y-Like Receptive Fields in the Early Visual Cortex: An Intermediate Stage for Building Cue-Invariant Receptive Fields from Subcortical Y Cells

Many of the neurons in early visual cortex are selective for the orientation of boundaries defined by first-order cues (luminance) as well as second-order cues (contrast, texture). The neural circuit mechanism underlying this selectivity is still unclear, but some studies have proposed that it emerges from spatial nonlinearities of subcortical Y cells. To understand how inputs from the Y-cell pathway might be pooled to generate cue-invariant receptive fields, we recorded visual responses from single neurons in cat Area 18 using linear multielectrode arrays. We measured responses to drifting and contrast-reversing luminance gratings as well as contrast modulation gratings. We found that a large fraction of these neurons have nonoriented responses to gratings, similar to those of subcortical Y cells: they respond at the second harmonic (F2) to high-spatial frequency contrast-reversing gratings and at the first harmonic (F1) to low-spatial frequency drifting gratings ("Y-cell signature"). For a given neuron, spatial frequency tuning for linear (F1) and nonlinear (F2) responses is quite distinct, similar to orientation-selective cue-invariant neurons. Also, these neurons respond to contrast modulation gratings with selectivity for the carrier (texture) spatial frequency and, in some cases, orientation. Their receptive field properties suggest that they could serve as building blocks for orientation-selective cue-invariant neurons. We propose a circuit model that combines ON- and OFF-center cortical Y-like cells in an unbalanced push-pull manner to generate orientation-selective, cue-invariant receptive fields.

SIGNIFICANCE STATEMENT

A significant fraction of neurons in early visual cortex have specialized receptive fields that allow them to selectively respond to the orientation of boundaries that are invariant to the cue (luminance, contrast, texture, motion) that defines them. However, the neural mechanism to construct such versatile receptive fields remains unclear. Using multielectrode recording, we found a large fraction of neurons in early visual cortex with receptive fields not selective for orientation that have spatial nonlinearities like those of subcortical Y cells. These are strong candidates for building cue-invariant orientation-selective neurons; we present a neural circuit model that pools such neurons in an imbalanced "push-pull" manner, to generate orientation-selective cue-invariant receptive fields.

Neural Processing of Second-Order Motion in the Suprasylvian Cortex of the Cat

Neuronal responses to second-order motion, that is, to spatiotemporal variations of texture or contrast, have been reported in several cortical areas of mammals, including the middle-temporal (MT) area in primates. In this study, we investigated whether second-order responses are present in the cat posteromedial lateral suprasylvian (PMLS) cortex, a possible homolog of the primate area MT. The stimuli used were luminance-based sine-wave gratings (first-order) and contrast-modulated carrier stimuli (second-order), which consisted of a high-spatial-frequency static grating (carrier) whose contrast was modulated by a low-spatial-frequency drifting grating (envelope). Results indicate that most PMLS neurons responded to second-order motion and for the vast majority of cells, first- and second-order preferred directions were conserved. However, responses to second-order stimuli were significantly reduced when compared to those evoked by first-order gratings. Circular variance was increased for second-order stimuli, indicating that PMLS direction selectivity was weaker for this type of stimulus. Finally, carrier orientation selectivity was either absent or very broad and had no influence on the envelope's orientation selectivity. In conclusion, our data show that PMLS neurons exhibit similar first- and second-order response profiles and that, akin primate area MT cells, they perform a form-cue invariant analysis of motion signals.

Mice Can Use Second-Order, Contrast-Modulated Stimuli to Guide Visual Perception

Visual processing along the primate ventral stream takes place in a hierarchy of areas, characterized by an increase in both complexity of neuronal preferences and invariance to changes of low-level stimulus attributes. A basic type of invariance is form-cue invariance, where neurons have similar preferences in response to first-order stimuli, defined by changes in luminance, and global features of second-order stimuli, defined by changes in texture or contrast. Whether in mice, a now popular model system for early visual processing, visual perception can be guided by second-order stimuli is currently unknown. Here, we probed mouse visual perception and neural responses in areas V1 and LM using various types of second-order, contrast-modulated gratings with static noise carriers. These gratings differ in their spatial frequency composition and thus in their ability to invoke first-order mechanisms exploiting local luminance features. We show that mice can transfer learning of a coarse orientation discrimination task involving first-order, luminance-modulated gratings to the contrast-modulated gratings, albeit with markedly reduced discrimination performance. Consistent with these behavioral results, we demonstrate that neurons in area V1 and LM are less responsive and less selective to contrast-modulated than to luminance-modulated gratings, but respond with broadly similar preferred orientations. We conclude that mice can, at least in a rudimentary form, use second-order stimuli to guide visual perception.

SIGNIFICANCE STATEMENT

To extract object boundaries in natural scenes, the primate visual system does not only rely on differences in local luminance but can also take into account differences in texture or contrast. Whether the mouse, which has a much simpler visual system, can use such second-order information to guide visual perception is unknown. Here we tested mouse perception of second-order, contrast-defined stimuli and measured their neural representations in two areas of visual cortex. We find that mice can use contrast-defined stimuli to guide visual perception, although behavioral performance and neural representations were less robust than for luminance-defined stimuli. These findings shed light on basic steps of feature extraction along the mouse visual cortical hierarchy, which may ultimately lead to object recognition.

Categorically distinct types of receptive fields in early visual cortex

In the visual cortex, distinct types of neurons have been identified based on cellular morphology, response to injected current, or expression of specific markers, but neurophysiological studies have revealed visual receptive field (RF) properties that appear to be on a continuum, with only two generally recognized classes: simple and complex. Most previous studies have characterized visual responses of neurons using stereotyped stimuli such as bars, gratings, or white noise and simple system identification approaches (e.g., reverse correlation). Here we estimate visual RF models of cortical neurons using visually rich natural image stimuli and regularized regression system identification methods and characterize their spatial tuning, temporal dynamics, spatiotemporal behavior, and spiking properties. We quantitatively demonstrate the existence of three functionally distinct categories of simple cells, distinguished by their degree of orientation selectivity (isotropic or oriented) and the nature of their output nonlinearity (expansive or compressive). In addition, these three types have differing average values of several other properties. Cells with nonoriented RFs tend to have smaller RFs, shorter response durations, no direction selectivity, and high reliability. Orientation-selective neurons with an expansive output nonlinearity have Gabor-like RFs, lower spontaneous activity and responsivity, and spiking responses with higher sparseness. Oriented RFs with a compressive nonlinearity are spatially nondescript and tend to show longer response latency. Our findings indicate multiple physiologically defined types of RFs beyond the simple/complex dichotomy, suggesting that cortical neurons may have more specialized functional roles rather than lying on a multidimensional continuum.

Encoding and estimation of first- and second-order binocular disparity in natural images

•

First- and second-order responses to natural binocular images are correlated.

•

Second-order mechanisms can improve the accuracy of disparity estimation.

•

Second-order mechanisms can extend the depth range of binocular stereopsis.

The first stage of processing of binocular information in the visual cortex is performed by mechanisms that are bandpass-tuned for spatial frequency and orientation. Psychophysical and physiological evidence have also demonstrated the existence of second-order mechanisms in binocular processing, which can encode disparities that are not directly accessible to first-order mechanisms. We compared the responses of first- and second-order binocular filters to natural images. We found that the responses of the second-order mechanisms are to some extent correlated with the responses of the first-order mechanisms, and that they can contribute to increasing both the accuracy, and depth range, of binocular stereopsis.

Mapping entrained brain oscillations during transcranial alternating current stimulation (tACS)

Transcranial alternating current stimulation (tACS), a non-invasive and well-tolerated form of electric brain stimulation, can influence perception, memory, as well as motor and cognitive function. While the exact underlying neurophysiological mechanisms are unknown, the effects of tACS are mainly attributed to frequency-specific entrainment of endogenous brain oscillations in brain areas close to the stimulation electrodes, and modulation of spike timing dependent plasticity reflected in gamma band oscillatory responses. tACS-related electromagnetic stimulator artifacts, however, impede investigation of these neurophysiological mechanisms. Here we introduce a novel approach combining amplitude-modulated tACS during whole-head magnetoencephalography (MEG) allowing for artifact-free source reconstruction and precise mapping of entrained brain oscillations underneath the stimulator electrodes. Using this approach, we show that reliable reconstruction of neuromagnetic low- and high-frequency oscillations including high gamma band activity in stimulated cortical areas is feasible opening a new window to unveil the mechanisms underlying the effects of stimulation protocols that entrain brain oscillatory activity.

A conceptual framework of computations in mid-level vision

If a picture is worth a thousand words, as an English idiom goes, what should those words-or, rather, descriptors-capture? What format of image representation would be sufficiently rich if we were to reconstruct the essence of images from their descriptors? In this paper, we set out to develop a conceptual framework that would be: (i) biologically plausible in order to provide a better mechanistic understanding of our visual system; (ii) sufficiently robust to apply in practice on realistic images; and (iii) able to tap into underlying structure of our visual world. We bring forward three key ideas. First, we argue that surface-based representations are constructed based on feature inference from the input in the intermediate processing layers of the visual system. Such representations are computed in a largely pre-semantic (prior to categorization) and pre-attentive manner using multiple cues (orientation, color, polarity, variation in orientation, and so on), and explicitly retain configural relations between features. The constructed surfaces may be partially overlapping to compensate for occlusions and are ordered in depth (figure-ground organization). Second, we propose that such intermediate representations could be formed by a hierarchical computation of similarity between features in local image patches and pooling of highly-similar units, and reestimated via recurrent loops according to the task demands. Finally, we suggest to use datasets composed of realistically rendered artificial objects and surfaces in order to better understand a model's behavior and its limitations.

Processing deficits of motion of contrast-modulated gratings in anisometropic amblyopia

Several studies have indicated substantial processing deficits for static second-order stimuli in amblyopia. However, less is known about the perception of second-order moving gratings. To investigate this issue, we measured the contrast sensitivity for second-order (contrast-modulated) moving gratings in seven anisometropic amblyopes and ten normal controls. The measurements were performed with non-equated carriers and a series of equated carriers. For comparison, the sensitivity for first-order motion and static second-order stimuli was also measured. Most of the amblyopic eyes (AEs) showed reduced sensitivity for second-order moving gratings relative to their non-amblyopic eyes (NAEs) and the dominant eyes (CEs) of normal control subjects, even when the detectability of the noise carriers was carefully controlled, suggesting substantial processing deficits of motion of contrast-modulated gratings in anisometropic amblyopia. In contrast, the non-amblyopic eyes of the anisometropic amblyopes were relatively spared. As a group, NAEs showed statistically comparable performance to CEs. We also found that contrast sensitivity for static second-order stimuli was strongly impaired in AEs and part of the NAEs of anisometropic amblyopes, consistent with previous studies. In addition, some amblyopes showed impaired performance in perception of static second-order stimuli but not in that of second-order moving gratings. These results may suggest a dissociation between the processing of static and moving second-order gratings in anisometropic amblyopia.

Form-cue invariant second-order neuronal responses to contrast modulation in primate area V2

A fundamental task of the visual system is to extract figure-ground boundaries between images of objects, which in natural scenes are often defined not only by luminance differences but also by "second-order" contrast or texture differences. Responses to contrast modulation (CM) and other second-order stimuli have been extensively studied in human psychophysics, but the neuronal substrates of second-order responses in nonhuman primates remain poorly understood. In this study, we have recorded single neurons in area V2 of macaque monkeys, using both CM patterns as well as conventional luminance modulation (LM) gratings. CM stimuli were constructed from stationary sine wave grating carrier patterns, which were modulated by drifting envelope gratings of a lower spatial frequency. We found approximately one-third of visually responsive V2 neurons responded to CM stimuli with a pronounced selectivity to carrier spatial frequencies, and often orientations, that were clearly outside the neurons' passbands for LM gratings. These neurons were "form-cue invariant" in that their tuning to CM envelope spatial frequency and orientation was very similar to that for LM gratings. Neurons were tuned to carrier spatial frequencies that were typically 2-4 octaves higher than their optimal envelope spatial frequencies, similar to results from human psychophysics. These results are distinct from CM responses arising from surround suppression, but could be understood in terms of a filter-rectify-filter model. Such neurons could provide a functionally useful and explicit representation of segmentation boundaries as well as a plausible neural substrate for human perception of second-order boundaries.

Possible functions of contextual modulations and receptive field nonlinearities: pop-out and texture segmentation

When analyzing a visual image, the brain has to achieve several goals quickly. One crucial goal is to rapidly detect parts of the visual scene that might be behaviorally relevant, while another one is to segment the image into objects, to enable an internal representation of the world. Both of these processes can be driven by local variations in any of several image attributes such as luminance, color, and texture. Here, focusing on texture defined by local orientation, we propose that the two processes are mediated by separate mechanisms that function in parallel. More specifically, differences in orientation can cause an object to "pop out" and attract visual attention, if its orientation differs from that of the surrounding objects. Differences in orientation can also signal a boundary between objects and therefore provide useful information for image segmentation. We propose that contextual response modulations in primary visual cortex (V1) are responsible for orientation pop-out, while a different kind of receptive field nonlinearity in secondary visual cortex (V2) is responsible for orientation-based texture segmentation. We review a recent experiment that led us to put forward this hypothesis along with other research literature relevant to this notion.

Surround suppression supports second-order feature encoding by macaque V1 and V2 neurons

Single neurons in areas V1 and V2 of macaque visual cortex respond selectively to luminance-modulated stimuli. These responses are often influenced by context, for example when stimuli extend outside the classical receptive field (CRF). These contextual phenomena, observed in many sensory areas, reflect a fundamental cortical computation and may inform perception by signaling second-order visual features which are defined by spatial relationships of contrast, orientation and spatial frequency. In the anesthetized, paralyzed macaque, we measured single-unit responses to a drifting preferred sinusoidal grating; low spatial frequency sinusoidal contrast modulations were applied to the grating, creating contrast-modulated, second-order forms. Most neurons responded selectively to the orientation of the contrast modulation of the preferred grating and were therefore second-order orientation-selective. Second-order selectivity was created by the asymmetric spatial organization of the excitatory CRF and suppressive extraclassical surround. We modeled these receptive field subregions using spatial Gaussians, sensitive to the modulation of contrast (not luminance) of the preferred carrier grating, that summed linearly and were capable of recovering asymmetrical receptive field organizations. Our modeling suggests that second-order selectivity arises both from elongated excitatory CRFs, asymmetrically organized extraclassical surround suppression, or both. We validated the model by successfully testing its predictions against conventional surround suppression measurements and spike-triggered analysis of second-order form responses. Psychophysical adaptation measurements on human observers revealed a pattern of second-order form selectivity consistent with neural response patterns. We therefore propose that cortical cells in primates do double duty, providing signals about both first- and second-order forms.

Orientation-cue invariant population responses to contrast-modulated and phase-reversed contour stimuli in macaque V1 and V2

Visual scenes can be readily decomposed into a variety of oriented components, the processing of which is vital for object segregation and recognition. In primate V1 and V2, most neurons have small spatio-temporal receptive fields responding selectively to oriented luminance contours (first order), while only a subgroup of neurons signal non-luminance defined contours (second order). So how is the orientation of second-order contours represented at the population level in macaque V1 and V2? Here we compared the population responses in macaque V1 and V2 to two types of second-order contour stimuli generated either by modulation of contrast or phase reversal with those to first-order contour stimuli. Using intrinsic signal optical imaging, we found that the orientation of second-order contour stimuli was represented invariantly in the orientation columns of both macaque V1 and V2. A physiologically constrained spatio-temporal energy model of V1 and V2 neuronal populations could reproduce all the recorded population responses. These findings suggest that, at the population level, the primate early visual system processes the orientation of second-order contours initially through a linear spatio-temporal filter mechanism. Our results of population responses to different second-order contour stimuli support the idea that the orientation maps in primate V1 and V2 can be described as a spatial-temporal energy map.

Responses to orientation discontinuities in V1 and V2: physiological dissociations and functional implications

Segmenting the visual image into objects is a crucial stage of visual processing. Object boundaries are typically associated with differences in luminance, but discontinuities in texture also play an important role. We showed previously that a subpopulation of neurons in V2 in anesthetized macaques responds to orientation discontinuities parallel to their receptive field orientation. Such single-cell responses could be a neurophysiological correlate of texture boundary detection. Neurons in V1, on the other hand, are known to have contextual response modulations such as iso-orientation surround suppression, which also produce responses to orientation discontinuities. Here, we use pseudorandom multiregion grating stimuli of two frame durations (20 and 40 ms) to probe and compare texture boundary responses in V1 and V2 in anesthetized macaque monkeys. In V1, responses to texture boundaries were observed for only the 40 ms frame duration and were independent of the orientation of the texture boundary. However, in transient V2 neurons, responses to such texture boundaries were robust for both frame durations and were stronger for boundaries parallel to the neuron's preferred orientation. The dependence of these processes on stimulus duration and orientation indicates that responses to texture boundaries in V2 arise independently of contextual modulations in V1. In addition, because the responses in transient V2 neurons are sensitive to the orientation of the texture boundary but those of V1 neurons are not, we suggest that V2 responses are the correlate of texture boundary detection, whereas contextual modulation in V1 serves other purposes, possibly related to orientation "pop-out."

Statistics of the electrosensory input in the freely swimming weakly electric fish Apteronotus leptorhynchus

The neural computations underlying sensory-guided behaviors can best be understood in view of the sensory stimuli to be processed under natural conditions. This input is often actively shaped by the movements of the animal and its sensory receptors. Little is known about natural sensory scene statistics taking into account the concomitant movement of sensory receptors in freely moving animals. South American weakly electric fish use a self-generated quasi-sinusoidal electric field for electrolocation and electrocommunication. Thousands of cutaneous electroreceptors detect changes in the transdermal potential (TDP) as the fish interact with conspecifics and the environment. Despite substantial knowledge about the circuitry and physiology of the electrosensory system, the statistical properties of the electrosensory input evoked by natural swimming movements have never been measured directly. Using underwater wireless telemetry, we recorded the TDP of Apteronotus leptorhynchus as they swam freely by themselves and during interaction with a conspecific. Swimming movements caused low-frequency TDP amplitude modulations (AMs). Interacting with a conspecific caused additional AMs around the difference frequency of their electric fields, with the amplitude of the AMs (envelope) varying at low frequencies due to mutual movements. Both AMs and envelopes showed a power-law relationship with frequency, indicating spectral scale invariance. Combining a computational model of the electric field with video tracking of movements, we show that specific swimming patterns cause characteristic spatiotemporal sensory input correlations that contain information that may be used by the brain to guide behavior.

Functional organization of envelope-responsive neurons in early visual cortex: organization of carrier tuning properties

It is well established that visual cortex neurons having similar selectivity for orientation, direction of motion, ocular dominance, and other properties of first-order (luminance-defined) stimuli are clustered into a columnar organization. However, the cortical architecture of neuronal responses to second-order (contrast/texture-defined) stimuli is poorly understood. A useful second-order stimulus is a contrast envelope, consisting of a finely detailed pattern (carrier) whose contrast varies on a coarse spatial scale (envelope). In this study, we analyzed the cortical organization of carrier tuning properties of neurons, which responded to contrast-modulated stimuli. We examined whether neurons tuned to similar carrier properties are clustered spatially and whether such spatial clusters are arranged in columns. To address these questions, we recorded single-unit activity, multiunit activity, and local field potentials simultaneously from area 18 of anesthetized cats, using single-channel microelectrodes and multielectrode arrays. Our data showed that neurons tuned to similar carrier spatial frequency are distributed in a highly clustered manner; neurons tuned to similar carrier orientation are also significantly clustered. Neurons along linear arrays perpendicular to the brain surface always exhibited similar optimal carrier spatial frequency, indicating a columnar organization. Multi-pronged tetrode recordings indicated that the diameter of these columns is ≥450 μm. Optimal carrier orientation was also significantly clustered but with finer-grain organization and greater scatter. These results indicate a fine anatomical structure of cortical organization of second-order information processing and suggest that there are probably more maps in cat area 18 than previously believed.

Responses to second-order texture modulations undergo surround suppression

First-order (contrast) surround suppression has been well characterized both psychophysically and physiologically,but relatively little is known as to whether the perception of second-order visual stimuli exhibits analogous center–surround interactions. Second-order surround suppression was characterized by requiring subjects to detect second-order modulation in stimuli presented alone or embedded in a surround.Both contrast- (CM) and orientation-modulated (OM) stimuli were used. For most subjects and both OM and CM stimuli, second-order surrounds caused thresholds to be higher, indicative of second-order suppression. For CM stimuli, suppression was orientation-specific, i.e., higher thresholds for parallel than for orthogonal surrounds. However, the evidence for orientation specificity of suppression for OM stimuli was weaker. These results suggest that normalization, leading to surround suppression, operates at multiple stages in cortical processing.

Motion-defined contour processing in the early visual cortex

From our daily experience, it is very clear that relative motion cues can contribute to correctly identifying object boundaries and perceiving depth. Motion-defined contours are not only generated by the motion of objects in a scene but also by the movement of an observer's head and body (motion parallax). However, the neural mechanism involved in detecting these contours is still unknown. To explore this mechanism, we extracellularly recorded visual responses of area 18 neurons in anesthetized and paralyzed cats. The goal of this study was to determine if motion-defined contours could be detected by neurons that have been previously shown to detect luminance-, texture-, and contrast-defined contours cue invariantly. Motion-defined contour stimuli were generated by modulating the velocity of high spatial frequency sinusoidal luminance gratings (carrier gratings) by a moving squarewave envelope. The carrier gratings were outside the luminance passband of a neuron, such that presence of the carrier alone within the receptive field did not elicit a response. Most neurons that responded to contrast-defined contours also responded to motion-defined contours. The orientation and direction selectivity of these neurons for motion-defined contours was similar to that of luminance gratings. A given neuron also exhibited similar selectivity for the spatial frequency of the carrier gratings of contrast- and motion-defined contours. These results suggest that different second-order contours are detected in a form-cue invariant manner, through a common neural mechanism in area 18.

Visual chimaeras obtained with the Riesz transform

Similar to an auditory chimaera (Smith et al, 2002 Nature 416 87-90), a visual chimaera can be defined as a synthetic image which has the fine spatial structure of one natural image and the envelope of another image in each spatial frequency band. Visual chimaeras constructed in this way could be useful to vision scientists interested in the study of interactions between first-order and second-order visual processing. Although it is almost trivial to generate 1-D chimaeras by means of the Hilbert transform and the analytic signal, problems arise in multidimensional signals like images given that the partial directional Hilbert transform and current 2-D demodulation algorithms are anisotropic or orientation-variant procedures. Here, we present a computational procedure to synthesise visual chimaeras by means of the Riesz transform--an isotropic generalisation of the Hilbert transform for multidimensional signals--and the associated monogenic signal--the vector-valued function counterpart of the analytic signal in which the Riesz transform replaces the Hilbert transform. Examples of visual chimaeras are shown for same/different category images.

Neuronal responses to texture-defined form in macaque visual area V2

Human and macaque observers can detect and discriminate visual forms defined by differences in texture. The neurophysiological correlates of visual texture perception are not well understood and have not been studied extensively at the single-neuron level in the primate brain. We used a novel family of texture patterns to measure the selectivity of neurons in extrastriate cortical area V2 of the macaque (Macaca nemestrina, Macaca fascicularis) for the orientation of texture-defined form, and to distinguish responses to luminance- and texture-defined form. Most V2 cells were selective for the orientation of luminance-defined form; they signaled the orientation of the component gratings that made up the texture patterns but not the overall pattern orientation. In some cells, these luminance responses were modulated by the direction or orientation of the texture envelope, suggesting an interaction of luminance and texture signals. We found little evidence for a "cue-invariant" representation in monkey V2. Few cells showed selectivity for the orientation of texture-defined form; they signaled the orientation of the texture patterns and not that of the component gratings. Small datasets recorded in monkey V1 and cat area 18 showed qualitatively similar patterns of results. Consistent with human functional imaging studies, our findings suggest that signals related to texture-defined form in primate cortex are most salient in areas downstream of V2. V2 may still provide the foundation for texture perception, through the interaction of luminance- and texture-based signals.

Visual responses to contrast-defined contours with equally spatial-scaled carrier in cat area 18

Contrast-defined contours are one type of second-order contours, across which there are no differences in luminance. Although they can be always perceived, their responses have been only investigated when the spatial frequency of carrier, the background texture whose contrast is modulated to form contours, is much higher than that of contrast-defined contours, due to the interference of responses to luminance contours in other cases. In the present study, we examined visual responses in cat area 18 to the contrast-defined contours with carrier at same spatial frequency equal to neuron's preferred value for luminance contours, by establishing a control stimulus including all the luminance components but lack of the contrast contour information. Using single unit recording and intrinsic optical imaging, we demonstrated that contrast gratings with equally spatial-scaled carrier induced responses in a proportion of cat area 18 neurons with the preferred orientation similar to that for luminance contours, and the responses generated orientation maps similar to those for luminance contours. Our finding suggests that early visual cortex can process second-order contours regardless of the spatial frequency of carriers, in a way similar to the processing of luminance contours. This uniform manner of early visual processing might underlie the visual detection of both luminance contours and non-luminance second-order contours.

Human primary visual cortex (V1) is selective for second-order spatial frequency

A variety of cues can differentiate objects from their surrounds. These include "first-order" cues such as luminance modulations and "second-order" cues involving modulations of orientation and contrast. Human sensitivity to first-order modulations is well described by a computational model involving spatially localized filters that are selective for orientation and spatial frequency (SF). It is widely held that first-order modulations are represented by the firing rates of simple and complex cells ("first-order" neurons) in primary visual cortex (V1) that, likewise, have spatially localized receptive fields that are selective for orientation- and SF. Human sensitivity to second-order modulations is well described by a filter-rectify-filter (FRF) model, with first- and second-order filters selective for orientation and SF. However, little is known about how neuronal activity in visual cortex represents second-order modulations. We tested the FRF model by using an functional (f)MRI-adaptation protocol to characterize the selectivity of activity in visual cortex to second-order, orientation-defined gratings of two different SFs. fMRI responses throughout early visual cortex exhibited selective adaptation to these stimuli. The low-SF grating was a more effective adapter than the high-SF grating, incompatible with the FRF model. To explain the results, we extended the FRF model by incorporating normalization, yielding a filter-rectify-normalize-filter model, in which normalization enhances selectivity for second-order SF but only for low spatial frequencies. We conclude that neurons in human visual cortex are selective for second-order SF, that normalization (surround suppression) contributes to this selectivity, and that the selectivity in higher visual areas is simply fed forward from V1.

Neural heterogeneities influence envelope and temporal coding at the sensory periphery

Peripheral sensory neurons respond to stimuli containing a wide range of spatio-temporal frequencies. We investigated electroreceptor neuron coding in the gymnotiform wave-type weakly electric fish Apteronotus leptorhynchus. Previous studies used low to mid temporal frequencies (<256 Hz) and showed that electroreceptor neuron responses to sensory stimuli could be almost exclusively accounted for by linear models, thereby implying a rate code. We instead used temporal frequencies up to 425 Hz, which is in the upper behaviorally relevant range for this species. We show that electroreceptors can: (A) respond up to the highest frequencies tested and (B) display strong nonlinearities in their responses to such stimuli. These nonlinearities were manifested by the fact that the responses to repeated presentations of the same stimulus were coherent at temporal frequencies outside of those contained in the stimulus waveform. Specifically, these consisted of low frequencies corresponding to the time varying contrast or envelope of the stimulus as well as higher harmonics of the frequencies contained in the stimulus. Heterogeneities in the afferent population influenced nonlinear coding as afferents with the lowest baseline firing rates tended to display the strongest nonlinear responses. To understand the link between afferent heterogeneity and nonlinear responsiveness, we used a phenomenological mathematical model of electrosensory afferents. Varying a single parameter in the model was sufficient to account for the variability seen in our experimental data and yielded a prediction: nonlinear responses to the envelope and at higher harmonics are both due to afferents with lower baseline firing rates displaying greater degrees of rectification in their responses. This prediction was verified experimentally as we found that the coherence between the half-wave rectified stimulus and the response resembled the coherence between the responses to repeated presentations of the stimulus in our dataset. This result shows that rectification cannot only give rise to responses to low frequency envelopes but also at frequencies that are higher than those contained in the stimulus. The latter result implies that information is contained in the fine temporal structure of electroreceptor afferent spike trains. Our results show that heterogeneities in peripheral neuronal populations can have dramatic consequences on the nature of the neural code.

Development of sensitivity to visual texture modulation in macaque monkeys

In human and non-human primates, higher form vision matures substantially later than spatial acuity and contrast sensitivity, as revealed by performance on such tasks as figure-ground segregation and contour integration. Our goal was to understand whether delayed maturation on these tasks was intrinsically form-dependent or, rather, related to the nature of spatial integration necessary for extracting task-relevant cues. We used an intermediate-level form task that did not call for extensive spatial integration. We trained monkeys (6-201 weeks) to discriminate the orientation of pattern modulation in a two-alternative forced choice paradigm. We presented two families of form patterns, defined by texture or contrast variations, and luminance-defined patterns for comparison. Infant monkeys could discriminate texture- and contrast-defined form as early as 6 weeks; sensitivity improved up to 40 weeks. Surprisingly, sensitivity for texture- and contrast-defined form matured earlier than for luminance-defined form. These results suggest that intermediate-level form vision develops in concert with basic spatial vision rather than following sequentially. Comparison with earlier results reveals that different aspects of form vision develop over different time courses, with processes that depend on comparing local image content maturing earlier than those requiring "global" linking of multiple visual elements across a larger spatial extent.

Measuring perceived differences in surface texture due to changes in higher order statistics

We investigate the ability of humans to perceive changes in the appearance of images of surface texture caused by the variation of their higher order statistics. We incrementally randomize their phase spectra while holding their first and second order statistics constant in order to ensure that the change in the appearance is due solely to changes in third and other higher order statistics. Stimuli comprise both natural and synthetically generated naturalistic images, with the latter being used to prevent observers from making pixel-wise comparisons. A difference scaling method is used to derive the perceptual scales for each observer, which show a sigmoidal relationship with the degree of randomization. Observers were maximally sensitive to changes within the 20%-60% randomization range. In order to account for this behavior we propose a biologically plausible model that computes the variance of local measurements of phase congruency.

Subcortical representation of non-Fourier image features

A fundamental goal of visual neuroscience is to identify the neural pathways representing different image features. It is widely argued that the early stages of these pathways represent linear features of the visual scene and that the nonlinearities necessary to represent complex visual patterns are introduced later in cortex. We tested this by comparing the responses of subcortical and cortical neurons to interference patterns constructed by summing sinusoidal gratings. Although a linear mechanism can detect the component gratings, a nonlinear mechanism is required to detect an interference pattern resulting from their sum. Consistent with in vitro retinal ganglion cell recordings, we found that interference patterns are represented subcortically by cat LGN Y-cells, but not X-cells. Linear and nonlinear tuning properties of LGN Y-cells were then characterized and compared quantitatively with those of cortical area 18 neurons responsive to interference patterns. This comparison revealed a high degree of similarity between the two neural populations, including the following: (1) the representation of similar spatial frequencies in both their linear and nonlinear responses, (2) comparable orientation selectivity for the high spatial frequency carrier of interference patterns, and (3) the same difference in their temporal frequency selectivity for drifting gratings versus the envelope of interference patterns. The present findings demonstrate that the nonlinear subcortical Y-cell pathway represents complex visual patterns and likely underlies cortical responses to interference patterns. We suggest that linear and nonlinear mechanisms important for encoding visual scenes emerge in parallel through distinct pathways originating at the retina.

Three-dimensional shape from second-order orientation flows

In images of textured surfaces, orientation flows formed by perspective convergence invariably convey 3D shape. We show that orientation flows formed by contrast-modulated (CM) and illusory contours (IC) convey 3D shape, and that both stimulus types induce 3D shape aftereffects on CM and IC test stimuli. Adaptation to luminance-modulated (LM) orientation flows induce robust 3D shape aftereffects on CM and IC tests, however, aftereffects using CM/IC adapting stimuli on LM tests were substantially weaker. These results can be explained by the adaptation of 3D shape-selective neurons that invariantly extract first- and second-order orientation flows from striate and extra-striate signals, which receive stronger input from neurons selective for first-order orientation flows.

Surround suppression of V1 neurons mediates orientation-based representation of high-order visual features

Neurons with surround suppression have been implicated in processing high-order visual features such as contrast- or texture-defined boundaries and subjective contours. However, little is known regarding how these neurons encode high-order visual information in a systematic manner as a population. To address this issue, we have measured detailed spatial structures of classical center and suppressive surround regions of receptive fields of primary visual cortex (V1) neurons and examined how a population of such neurons allow encoding of various high-order features and shapes in visual scenes. Using a novel method to reconstruct structures, we found that the center and surround regions are often both elongated parallel to each other, reminiscent of on and off subregions of simple cells without surround suppression. These structures allow V1 neurons to extract high-order contours of various orientations and spatial frequencies, with a variety of optimal values across neurons. The results show that a wide range of orientations and widths of the high-order features are systematically represented by the population of V1 neurons with surround suppression.

Learning to identify near-threshold luminance-defined and contrast-defined letters in observers with amblyopia

We assessed whether or not the sensitivity for identifying luminance-defined and contrast-defined letters improved with training in a group of amblyopic observers who have passed the critical period of development. In Experiment 1, we tracked the contrast threshold for identifying luminance-defined letters with training in a group of 11 amblyopic observers. Following training, six observers showed a reduction in thresholds, averaging 20%, for identifying luminance-defined letters. This improvement transferred extremely well to the untrained task of identifying contrast-defined letters (average improvement=38%) but did not transfer to an acuity measurement. Seven of the 11 observers were subsequently trained on identifying contrast-defined letters in Experiment 2. Following training, five of these seven observers demonstrated a further improvement, averaging 17%, for identifying contrast-defined letters. This improvement did not transfer to the untrained task of identifying luminance-defined letters. Our findings are consistent with predictions based on the locus of learning for first- and second-order stimuli according to the filter-rectifier-filter model for second-order visual processing.

Distinct perceptual grouping pathways revealed by temporal carriers and envelopes

S. E. Guttman, L. A. Gilroy, and R. Blake (2005) investigated whether observers could perform temporal grouping in multi-element displays where each local element was stochastically modulated over time along one of several potential dimensions--or "messenger types"--such as contrast, position, orientation, or spatial scale. Guttman et al.'s data revealed that grouping discards messenger type and therefore support a single-pathway model that groups elements with similar temporal waveforms. In the current study, we carried out three experiments in which temporal-grouping information resided either in the carrier, the envelope, or the combined carrier and envelope of each messenger's timecourse. Results revealed that grouping is highly specific for messenger type if carrier envelopes lack grouping information but largely messenger nonspecific if carrier envelopes contain grouping information. These imply that temporal grouping is mediated by several messenger-specific carrier pathways as well as by a messenger-nonspecific envelope pathways. Findings also challenge simple temporal-filtering accounts of perceptual grouping (E. H. Adelson & H. Farid, 1999).

Texture modulation detection by probability summation among orientation-selective and isotropic mechanisms

Substantial evidence has accumulated for the notion that modulations of second-order properties in the visual scene are processed by mechanisms which detect contrast variations within narrow orientation/spatial frequency channels. It has also been suggested that mechanisms exist which detect contrast modulations across all orientations. Many naturally occurring texture variations (e.g., modulations in orientation and/or spatial frequency) involve simultaneous contrast modulations in multiple channels. Contrasting conclusions have been drawn regarding the manner in which the information carried in multiple channels is combined. In a series of two experiments it is shown that simultaneous contrast modulations in two narrow orientation bands are detected by three mechanisms, two of which detect contrast modulations within the modulated bands only, the third of which integrates contrast across orientations in order to detect modulations of overall contrast. The three mechanisms combine their efforts by probability summation.

Curvature-based perceptual singularities and texture saliency with early vision mechanisms

Recent work has shown that salient perceptual singularities occur in visual textures even in the absence of feature gradients. In smoothly varying orientation-defined textures, these striking non-smooth percepts can be predicted from two texture curvatures, one tangential and one normal [Proc. Natl. Acad. Sci. USA103, 15704 (2006)]. We address the issue of detecting these perceptual singularities in a biologically plausible manner and present three different models to compute the tangential and normal curvatures using early cortical mechanisms. The first model relies on the response summation of similarly scaled even-symmetric simple cells at different positions by utilizing intercolumnar interactions in the primary visual cortex (V1). The second model is based on intracolumnar interactions in a two-layer mechanism of simple cells having the same orientation tuning but significantly different scales. Our third model uses a three-layer circuit in which both even-symmetric and odd-symmetric receptive fields (RFs) are used to compute all possible directional derivatives of the dominant orientation, from which the tangential and normal curvatures at each spatial position are selected using nonlinear shunting inhibition. We show experimental results of all three models, we outline an extension to oriented textures with multiple dominant orientations at each point, and we discuss how our results may be relevant to the processing of general textures.