Nonlinear Y-Like Receptive Fields in the Early Visual Cortex: An Intermediate Stage for Building Cue-Invariant Receptive Fields from Subcortical Y Cells

Feature selectivity and invariance in marsupial primary visual cortex

A fundamental question in sensory neuroscience revolves around how neurons represent complex visual stimuli. In mammalian primary visual cortex (V1), neurons decode intricate visual features to identify objects, with most being selective for edge orientation, but with half of those also developing invariance to edge position within their receptive fields. Position invariance allows cells to continue to code an edge even when it moves around. Combining feature selectivity and invariance is integral to successful object recognition. Considering the marsupial-eutherian divergence 160 million years ago, we explored whether feature selectivity and invariance was similar in marsupials and eutherians. We recovered the spatial filters and non-linear processing characteristics of the receptive fields of neurons in wallaby V1 and compared them with previous results from cat cortex. We stimulated the neurons in V1 with white Gaussian noise and analysed responses using the non-linear input model. Wallabies exhibit the same high percentage of orientation selective neurons as cats. However, in wallabies we observed a notably higher prevalence of neurons with three or more filters compared to cats. We show that having three or more filters substantially increases phase invariance in the V1s of both species, but that wallaby V1 accentuates this feature, suggesting that the species condenses more processing into the earliest cortical stage. These findings suggest that evolution has led to more than one solution to the problem of creating complex visual processing strategies. KEY POINTS: Previous studies have shown that the primary visual cortex (V1) in mammals is essential for processing complex visual stimuli, with neurons displaying selectivity for edge orientation and position. This research explores whether the visual processing mechanisms in marsupials, such as wallabies, are similar to those in eutherian mammals (e.g. cats). The study found that wallabies have a higher prevalence of neurons with multiple spatial filters in V1, indicating more complex visual processing. Using a non-linear input model, we demonstrated that neurons with three or more filters increase phase invariance. These findings suggest that marsupials and eutherian mammals have evolved similar strategies for visual processing, but marsupials have condensed more capacity to build phase invariance into the first step in the cortical pathway.

Characterization of extracellular spike waveforms recorded in wallaby primary visual cortex

Extracellular recordings were made from 642 units in the primary visual cortex (V1) of a highly visual marsupial, the Tammar wallaby. The receptive field (RF) characteristics of the cells were objectively estimated using the non-linear input model (NIM), and these were correlated with spike shapes. We found that wallaby cortical units had 68% regular spiking (RS), 12% fast spiking (FS), 4% triphasic spiking (TS), 5% compound spiking (CS) and 11% positive spiking (PS). RS waveforms are most often associated with recordings from pyramidal or spiny stellate cell bodies, suggesting that recordings from these cell types dominate in the wallaby cortex. In wallaby, 70-80% of FS and RS cells had orientation selective RFs and had evenly distributed linear and nonlinear RFs. We found that 47% of wallaby PS units were non-orientation selective and they were dominated by linear RFs. Previous studies suggest that the PS units represent recordings from the axon terminals of non-orientation selective cells originating in the lateral geniculate nucleus (LGN). If this is also true in wallaby, as strongly suggested by their low response latencies and bursty spiking properties, the results suggest that significantly more neurons in wallaby LGN are already orientation selective. In wallaby, less than 10% of recorded spikes had triphasic (TS) or sluggish compound spiking (CS) waveforms. These units had a mixture of orientation selective and non-oriented properties, and their cellular origins remain difficult to classify.

Beyond ℓ1 sparse coding in V1

Growing evidence indicates that only a sparse subset from a pool of sensory neurons is active for the encoding of visual stimuli at any instant in time. Traditionally, to replicate such biological sparsity, generative models have been using the ℓ1 norm as a penalty due to its convexity, which makes it amenable to fast and simple algorithmic solvers. In this work, we use biological vision as a test-bed and show that the soft thresholding operation associated to the use of the ℓ1 norm is highly suboptimal compared to other functions suited to approximating ℓp with 0 ≤ p < 1 (including recently proposed continuous exact relaxations), in terms of performance. We show that ℓ1 sparsity employs a pool with more neurons, i.e. has a higher degree of overcompleteness, in order to maintain the same reconstruction error as the other methods considered. More specifically, at the same sparsity level, the thresholding algorithm using the ℓ1 norm as a penalty requires a dictionary of ten times more units compared to the proposed approach, where a non-convex continuous relaxation of the ℓ0 pseudo-norm is used, to reconstruct the external stimulus equally well. At a fixed sparsity level, both ℓ0- and ℓ1-based regularization develop units with receptive field (RF) shapes similar to biological neurons in V1 (and a subset of neurons in V2), but ℓ0-based regularization shows approximately five times better reconstruction of the stimulus. Our results in conjunction with recent metabolic findings indicate that for V1 to operate efficiently it should follow a coding regime which uses a regularization that is closer to the ℓ0 pseudo-norm rather than the ℓ1 one, and suggests a similar mode of operation for the sensory cortex in general.

Behavioral signatures of Y-like neuronal responses in human vision

Retinal ganglion cells initiating the magnocellular/Y-cell visual pathways respond nonlinearly to high spatial frequencies (SFs) and temporal frequencies (TFs). This nonlinearity is implicated in the processing of contrast modulation (CM) stimuli in cats and monkeys, but its contribution to human visual perception is not well understood. Here, we evaluate human psychophysical performance for CM stimuli, consisting of a high SF grating carrier whose contrast is modulated by a low SF sinewave envelope. Subjects reported the direction of motion of CM envelopes or luminance modulation (LM) gratings at different eccentricities. The performance on SF (for LMs) or carrier SF (for CMs) was measured for different TFs (LMs) or carrier TFs (CMs). The best performance for LMs was at lower TFs and SFs, decreasing systematically with eccentricity. However, performance with CMs was bandpass with carrier SF, largely independent of carrier TF, and at the highest carrier TF (20 Hz) decreased minimally with eccentricity. Since the nonlinear subunits of Y-cells respond better at higher TFs compared to the linear response components and respond best at higher SFs that are relatively independent of eccentricity, these results suggest that behavioral tasks employing CM stimuli might reveal nonlinear contributions of retinal Y-like cells to human perception.

Orientation pinwheels in primary visual cortex of a highly visual marsupial

Primary visual cortices in many mammalian species exhibit modular and periodic orientation preference maps arranged in pinwheel-like layouts. The role of inherited traits as opposed to environmental influences in determining this organization remains unclear. Here, we characterize the cortical organization of an Australian marsupial, revealing pinwheel organization resembling that of eutherian carnivores and primates but distinctly different from the simpler salt-and-pepper arrangement of eutherian rodents and rabbits. The divergence of marsupials from eutherians 160 million years ago and the later emergence of rodents and rabbits suggest that the salt-and-pepper structure is not the primitive ancestral form. Rather, the genetic code that enables complex pinwheel formation is likely widespread, perhaps extending back to the common therian ancestors of modern mammals.

Model-based characterization of the selectivity of neurons in primary visual cortex

Statistical models are increasingly being used to understand the complexity of stimulus selectivity in primary visual cortex (V1) in the context of complex time-varying stimuli, replacing averaging responses to simple parametric stimuli. Although such models often can more accurately reflect the computations performed by V1 neurons in more natural visual environments, they do not by themselves provide insight into V1 neural selectivity to basic stimulus features such as receptive field size, spatial frequency tuning, and phase invariance. Here, we present a battery of analyses that can be directly applied to encoding models to link complex encoding models to more interpretable aspects of stimulus selectivity. We apply this battery to nonlinear models of V1 neurons recorded in awake macaque during random bar stimuli. In linking model properties to more classical measurements, we demonstrate several novel aspects of V1 selectivity not available to simpler experimental measurements. For example, this approach reveals that individual spatiotemporal elements of the V1 models often have a smaller spatial scale than the neuron as a whole, resulting in nontrivial tuning to spatial frequencies. In addition, we propose measures of nonlinear integration that suggest that classical classifications of V1 neurons into simple versus complex cells will be spatial-frequency dependent. In total, rather than obfuscate classical characterizations of V1 neurons, model-based characterizations offer a means to more fully understand their selectivity, and link their classical tuning properties to their roles in more complex, natural, visual processing.NEW & NOTEWORTHY Visual neurons are increasingly being studied with more complex, natural visual stimuli, and increasingly complex models are necessary to characterize their response properties. Here, we describe a battery of analyses that relate these more complex models to classical characterizations. Using such model-based characterizations of V1 neurons furthermore yields several new insights into V1 processing not possible to capture in more classical means to measure their visual selectivity.

Macaque V1 responses to 2nd-order contrast-modulated stimuli and the possible subcortical and cortical contributions

Natural images comprise contours and boundaries defined by 1st-order luminance-modulated (LM) cues that are readily encoded by V1 neurons, and 2nd-order contrast-modulated (CM) cues that carry local, but not over-the-space, luminance changes. The neurophysiological foundations for CM processing remain unsolved. Here we used two-photon calcium imaging to demonstrate that V1 superficial-layer neurons respond to both LM and CM gratings in awake, fixating, macaques, with overall LM responses stronger than CM responses. Furthermore, adaptation experiments revealed that LM responses were similarly suppressed by LM and CM adaptation, with moderately larger effects by iso-orientation adaptation than by orthogonal adaptation, suggesting that LM and CM orientation responses likely share a strong orientation-non-selective subcortical origin. In contrast, CM responses were substantially more suppressed by iso-orientation than by orthogonal LM and CM adaptation, likely suggesting stronger orientation-specific intracortical influences for CM responses than for LM responses, besides shared orientation-non-selective subcortical influences. These results thus may indicate a subcortical-to-V1 filter-rectify-filter mechanism for CM processing: Local luminance changes in CM stimuli are initially encoded by orientation-non-selective subcortical neurons, and the outputs are half-wave rectified, and then summed by V1 neurons to signal CM orientation, which may be further substantially refined by intracortical influences.

Segmenting surface boundaries using luminance cues

Segmenting scenes into distinct surfaces is a basic visual perception task, and luminance differences between adjacent surfaces often provide an important segmentation cue. However, mean luminance differences between two surfaces may exist without any sharp change in albedo at their boundary, but rather from differences in the proportion of small light and dark areas within each surface, e.g. texture elements, which we refer to as a luminance texture boundary. Here we investigate the performance of human observers segmenting luminance texture boundaries. We demonstrate that a simple model involving a single stage of filtering cannot explain observer performance, unless it incorporates contrast normalization. Performing additional experiments in which observers segment luminance texture boundaries while ignoring super-imposed luminance step boundaries, we demonstrate that the one-stage model, even with contrast normalization, cannot explain performance. We then present a Filter–Rectify–Filter model positing two cascaded stages of filtering, which fits our data well, and explains observers' ability to segment luminance texture boundary stimuli in the presence of interfering luminance step boundaries. We propose that such computations may be useful for boundary segmentation in natural scenes, where shadows often give rise to luminance step edges which do not correspond to surface boundaries.

Voltage distributions in extracellular brain recordings

Extracellular recordings of brain voltage signals have many uses, including the identification of spikes and the characterization of brain states via analysis of local field potential (LFP) or EEG recordings. Though the factors underlying the generation of these signals are time varying and complex, their analysis may be facilitated by an understanding of their statistical properties. To this end, we analyzed the voltage distributions of high-pass extracellular recordings from a variety of structures, including cortex, thalamus, and hippocampus, in monkeys, cats, and rodents. We additionally investigated LFP signals in these recordings as well as human EEG signals obtained during different sleep stages. In all cases, the distributions were accurately described by a Gaussian within ±1.5 standard deviations from zero. Outside these limits, voltages tended to be distributed exponentially, that is, they fell off linearly on log-linear frequency plots, with variable heights and slopes. A possible explanation for this is that sporadically and independently occurring events with individual Gaussian size distributions can sum to produce approximately exponential distributions. For the high-pass recordings, a second explanation results from a model of the noisy behavior of ion channels that produce action potentials via Hodgkin-Huxley kinetics. The distributions produced by this model, relative to the averaged potential, were also Gaussian with approximately exponential flanks. The model also predicted time-varying noise distributions during action potentials, which were observed in the extracellular spike signals. These findings suggest a principled method for detecting spikes in high-pass recordings and transient events in LFP and EEG signals.NEW & NOTEWORTHY We show that the voltage distributions in brain recordings, including high-pass extracellular recordings, the LFP, and human EEG, are accurately described by a Gaussian within ±1.5 standard deviations from zero, with heavy, exponential tails outside these limits. This offers a principled way of setting event detection thresholds in high-pass recordings. It also offers a means for identifying event-like, transient signals in LFP and EEG recordings which may correlate with other neural phenomena.

Analysis of extracellular spike waveforms and associated receptive fields of neurons in cat primary visual cortex

KEY POINTS

Extracellular spikes recorded in the visual cortex (Area 17/18, V1) are commonly classified into either regular-spiking (RS) or fast-spiking (FS). Using multi-electrode arrays positioned in cat V1 and a broadband stimulus, we show that there is also a distinct class with positive-spiking (PS) waveforms. PS units were associated mainly with non-oriented receptive fields while RS and FS units had orientation-selective receptive fields. We suggest that PS units are recordings of axons originating from the thalamus. This conclusion was reinforced by our finding that we could record PS units after cortical silencing, but not record RS and FS units. The importance of our findings is that we were able to correlate spike shapes with receptive field characteristics with high precision using multi-electrode extracellular recording techniques. This allows considerable increases in the amount of information that can be extracted from future cortical experiments.

ABSTRACT

Extracellular spike waveforms from recordings in the visual cortex have been classified into either regular-spiking (RS) or fast-spiking (FS) units. While both these types of spike waveforms are negative-dominant, we show that there are also distinct classes of spike waveforms in visual Area 17/18 (V1) of anaesthetised cats with positive-dominant waveforms, which are not regularly reported. The spatial receptive fields (RFs) of these different spike waveform types were estimated, which objectively revealed the existence of oriented and non-oriented RFs. We found that units with positive-dominant spikes, which have been associated with recordings from axons in the literature, had mostly non-oriented RFs (84%), which are similar to the centre-surround RFs observed in the dorsal lateral geniculate nucleus (dLGN). Thus, we hypothesise that these positive-dominant waveforms may be recordings from dLGN afferents. We recorded from V1 before and after the application of muscimol (a cortical silencer) and found that the positive-dominant spikes (PS) remained while the RS and FS cells did not. We also noted that the PS units had spiking characteristics normally associated with dLGN units (i.e. higher response spike rates, lower response latencies and higher proportion of burst spikes). Our findings show quantitatively that it is possible to correlate the RF properties of cortical neurons with particular spike waveforms. This has implications for how extracellular recordings should be interpreted and complex experiments can now be contemplated that would have been very challenging previously, such as assessing the feedforward connectivity between brain areas in the same location of cortical tissue.

Modeling second-order boundary perception: A machine learning approach

Visual pattern detection and discrimination are essential first steps for scene analysis. Numerous human psychophysical studies have modeled visual pattern detection and discrimination by estimating linear templates for classifying noisy stimuli defined by spatial variations in pixel intensities. However, such methods are poorly suited to understanding sensory processing mechanisms for complex visual stimuli such as second-order boundaries defined by spatial differences in contrast or texture. We introduce a novel machine learning framework for modeling human perception of second-order visual stimuli, using image-computable hierarchical neural network models fit directly to psychophysical trial data. This framework is applied to modeling visual processing of boundaries defined by differences in the contrast of a carrier texture pattern, in two different psychophysical tasks: (1) boundary orientation identification, and (2) fine orientation discrimination. Cross-validation analysis is employed to optimize model hyper-parameters, and demonstrate that these models are able to accurately predict human performance on novel stimulus sets not used for fitting model parameters. We find that, like the ideal observer, human observers take a region-based approach to the orientation identification task, while taking an edge-based approach to the fine orientation discrimination task. How observers integrate contrast modulation across orientation channels is investigated by fitting psychophysical data with two models representing competing hypotheses, revealing a preference for a model which combines multiple orientations at the earliest possible stage. Our results suggest that this machine learning approach has much potential to advance the study of second-order visual processing, and we outline future steps towards generalizing the method to modeling visual segmentation of natural texture boundaries. This study demonstrates how machine learning methodology can be fruitfully applied to psychophysical studies of second-order visual processing.

Many naturally occurring visual boundaries are defined by spatial differences in features other than luminance, for example by differences in texture or contrast. Quantitative models of such “second-order” boundary perception cannot be estimated using the standard regression techniques (known as “classification images”) commonly applied to “first-order”, luminance-defined stimuli. Here we present a novel machine learning approach to modeling second-order boundary perception using hierarchical neural networks. In contrast to previous quantitative studies of second-order boundary perception, we directly estimate network model parameters using psychophysical trial data. We demonstrate that our method can reveal different spatial summation strategies that human observers utilize for different kinds of second-order boundary perception tasks, and can be used to compare competing hypotheses of how contrast modulation is integrated across orientation channels. We outline extensions of the methodology to other kinds of second-order boundaries, including those in natural images.