Functional mechanisms shaping lateral geniculate responses to artificial and natural stimuli

TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks

Spiking neural networks (SNNs) are attracting widespread interest due to their biological plausibility, energy efficiency, and powerful spatiotemporal information representation ability. Given the critical role of attention mechanisms in enhancing neural network performance, the integration of SNNs and attention mechanisms exhibits tremendous potential to deliver energy-efficient and high-performance computing paradigms. In this article, we present a novel temporal-channel joint attention mechanism for SNNs, referred to as TCJA-SNN. The proposed TCJA-SNN framework can effectively assess the significance of spike sequence from both spatial and temporal dimensions. More specifically, our essential technical contribution lies on: 1) we employ the squeeze operation to compress the spike stream into an average matrix. Then, we leverage two local attention mechanisms based on efficient 1-D convolutions to facilitate comprehensive feature extraction at the temporal and channel levels independently and 2) we introduce the cross-convolutional fusion (CCF) layer as a novel approach to model the interdependencies between the temporal and channel scopes. This layer effectively breaks the independence of these two dimensions and enables the interaction between features. Experimental results demonstrate that the proposed TCJA-SNN outperforms the state-of-the-art (SOTA) on all standard static and neuromorphic datasets, including Fashion-MNIST, CIFAR10, CIFAR100, CIFAR10-DVS, N-Caltech 101, and DVS128 Gesture. Furthermore, we effectively apply the TCJA-SNN framework to image generation tasks by leveraging a variation autoencoder. To the best of our knowledge, this study is the first instance where the SNN-attention mechanism has been employed for high-level classification and low-level generation tasks. Our implementation codes are available at https://github.com/ridgerchu/TCJA.

Neural pathways and computations that achieve stable contrast processing tuned to natural scenes

Natural scenes are highly dynamic, challenging the reliability of visual processing. Yet, humans and many animals perform accurate visual behaviors, whereas computer vision devices struggle with rapidly changing background luminance. How does animal vision achieve this? Here, we reveal the algorithms and mechanisms of rapid luminance gain control in Drosophila, resulting in stable visual processing. We identify specific transmedullary neurons as the site of luminance gain control, which pass this property to direction-selective cells. The circuitry further involves wide-field neurons, matching computational predictions that local spatial pooling drive optimal contrast processing in natural scenes when light conditions change rapidly. Experiments and theory argue that a spatially pooled luminance signal achieves luminance gain control via divisive normalization. This process relies on shunting inhibition using the glutamate-gated chloride channel GluClα. Our work describes how the fly robustly processes visual information in dynamically changing natural scenes, a common challenge of all visual systems.

Blind Video Quality Prediction by Uncovering Human Video Perceptual Representation

Blind video quality assessment (VQA) has become an increasingly demanding problem in automatically assessing the quality of ever-growing in-the-wild videos. Although efforts have been made to measure temporal distortions, the core to distinguish between VQA and image quality assessment (IQA), the lack of modeling of how the human visual system (HVS) relates to the temporal quality of videos hinders the precise mapping of predicted temporal scores to the human perception. Inspired by the recent discovery of the temporal straightness law of natural videos in the HVS, this paper intends to model the complex temporal distortions of in-the-wild videos in a simple and uniform representation by describing the geometric properties of videos in the visual perceptual domain. A novel videolet, with perceptual representation embedding of a few consecutive frames, is designed as the basic quality measurement unit to quantify temporal distortions by measuring the angular and linear displacements from the straightness law. By combining the predicted score on each videolet, a perceptually temporal quality evaluator (PTQE) is formed to measure the temporal quality of the entire video. Experimental results demonstrate that the perceptual representation in the HVS is an efficient way of predicting subjective temporal quality. Moreover, when combined with spatial quality metrics, PTQE achieves top performance over popular in-the-wild video datasets. More importantly, PTQE requires no additional information beyond the video being assessed, making it applicable to any dataset without parameter tuning. Additionally, the generalizability of PTQE is evaluated on video frame interpolation tasks, demonstrating its potential to benefit temporal-related enhancement tasks.

Stimulus type shapes the topology of cellular functional networks in mouse visual cortex

On the timescale of sensory processing, neuronal networks have relatively fixed anatomical connectivity, while functional interactions between neurons can vary depending on the ongoing activity of the neurons within the network. We thus hypothesized that different types of stimuli could lead those networks to display stimulus-dependent functional connectivity patterns. To test this hypothesis, we analyzed single-cell resolution electrophysiological data from the Allen Institute, with simultaneous recordings of stimulus-evoked activity from neurons across 6 different regions of mouse visual cortex. Comparing the functional connectivity patterns during different stimulus types, we made several nontrivial observations: (1) while the frequencies of different functional motifs were preserved across stimuli, the identities of the neurons within those motifs changed; (2) the degree to which functional modules are contained within a single brain region increases with stimulus complexity. Altogether, our work reveals unexpected stimulus-dependence to the way groups of neurons interact to process incoming sensory information.

Disparate nonlinear neural dynamics measured with different techniques in macaque and human V1

Diverse neuro-imaging techniques measure different aspects of neural responses with distinct spatial and temporal resolutions. Relating measured neural responses across different methods has been challenging. Here, we take a step towards overcoming this challenge, by comparing the nonlinearity of neural dynamics measured across methods. We used widefield voltage-sensitive dye imaging (VSDI) to measure neural population responses in macaque V1 to visual stimuli with a wide range of temporal waveforms. We found that stimulus-evoked VSDI responses are surprisingly near-additive in time. These results are qualitatively different from the strong sub-additive dynamics previously measured using fMRI and electrocorticography (ECoG) in human visual cortex with a similar set of stimuli. To test whether this discrepancy is specific to VSDI-a signal dominated by subthreshold neural activity, we repeated our measurements using widefield imaging of a genetically encoded calcium indicator (GcaMP6f)-a signal dominated by spiking activity, and found that GCaMP signals in macaque V1 are also near-additive. Therefore, the discrepancies in the extent of sub-additivity between the macaque and the human measurements are unlikely due to differences between sub- and supra-threshold neural responses. Finally, we use a simple yet flexible delayed normalization model to capture these different dynamics across measurements (with different model parameters). The model can potentially generalize to a broader set of stimuli, which aligns with previous suggestion that dynamic gain-control is a canonical computation contributing to neural processing in the brain.

Sensory Drive Modifies Brain Dynamics and the Temporal Integration Window

Perception is suggested to occur in discrete temporal windows, clocked by cycles of neural oscillations. An important testable prediction of this theory is that individuals' peak frequencies of oscillations should correlate with their ability to segregate the appearance of two successive stimuli. An influential study tested this prediction and showed that individual peak frequency of spontaneously occurring alpha (8-12 Hz) correlated with the temporal segregation threshold between two successive flashes of light [Samaha, J., & Postle, B. R. The speed of alpha-band oscillations predicts the temporal resolution of visual perception. Current Biology, 25, 2985-2990, 2015]. However, these findings were recently challenged [Buergers, S., & Noppeney, U. The role of alpha oscillations in temporal binding within and across the senses. Nature Human Behaviour, 6, 732-742, 2022]. To advance our understanding of the link between oscillations and temporal segregation, we devised a novel experimental approach. Rather than relying entirely on spontaneous brain dynamics, we presented a visual grating before the flash stimuli that is known to induce continuous oscillations in the gamma band (45-65 Hz). By manipulating the contrast of the grating, we found that high contrast induces a stronger gamma response and a shorter temporal segregation threshold, compared to low-contrast trials. In addition, we used a novel tool to characterize sustained oscillations and found that, for half of the participants, both the low- and high-contrast gratings were accompanied by a sustained and phase-locked alpha oscillation. These participants tended to have longer temporal segregation thresholds. Our results suggest that visual stimulus drive, reflected by oscillations in specific bands, is related to the temporal resolution of visual perception.

Postsynaptic mitochondria are positioned to support functional diversity of dendritic spines

Postsynaptic mitochondria are critical for the development, plasticity, and maintenance of synaptic inputs. However, their relationship to synaptic structure and functional activity is unknown. We examined a correlative dataset from ferret visual cortex with in vivo two-photon calcium imaging of dendritic spines during visual stimulation and electron microscopy reconstructions of spine ultrastructure, investigating mitochondrial abundance near functionally and structurally characterized spines. Surprisingly, we found no correlation to structural measures of synaptic strength. Instead, we found that mitochondria are positioned near spines with orientation preferences that are dissimilar to the somatic preference. Additionally, we found that mitochondria are positioned near groups of spines with heterogeneous orientation preferences. For a subset of spines with a mitochondrion in the head or neck, synapses were larger and exhibited greater selectivity to visual stimuli than those without a mitochondrion. Our data suggest mitochondria are not necessarily positioned to support the energy needs of strong spines, but rather support the structurally and functionally diverse inputs innervating the basal dendrites of cortical neurons.

Postsynaptic mitochondria are positioned to support functional diversity of dendritic spines

Postsynaptic mitochondria are critical to the development, plasticity, and maintenance of synaptic inputs. However, their relationship to synaptic structure and functional activity is unknown. We examined a correlative dataset from ferret visual cortex with in vivo two-photon calcium imaging of dendritic spines during visual stimulation and electron microscopy (EM) reconstructions of spine ultrastructure, investigating mitochondrial abundance near functionally- and structurally-characterized spines. Surprisingly, we found no correlation to structural measures of synaptic strength. Instead, we found that mitochondria are positioned near spines with orientation preferences that are dissimilar to the somatic preference. Additionally, we found that mitochondria are positioned near groups of spines with heterogeneous orientation preferences. For a subset of spines with mitochondrion in the head or neck, synapses were larger and exhibited greater selectivity to visual stimuli than those without a mitochondrion. Our data suggest mitochondria are not necessarily positioned to support the energy needs of strong spines, but rather support the structurally and functionally diverse inputs innervating the basal dendrites of cortical neurons.

Stimulus-dependent functional network topology in mouse visual cortex

Information is processed by networks of neurons in the brain. On the timescale of sensory processing, those neuronal networks have relatively fixed anatomical connectivity, while functional connectivity, which defines the interactions between neurons, can vary depending on the ongoing activity of the neurons within the network. We thus hypothesized that different types of stimuli, which drive different neuronal activities in the network, could lead those networks to display stimulus-dependent functional connectivity patterns. To test this hypothesis, we analyzed electrophysiological data from the Allen Brain Observatory, which utilized Neuropixels probes to simultaneously record stimulus-evoked activity from hundreds of neurons across 6 different regions of mouse visual cortex. The recordings had single-cell resolution and high temporal fidelity, enabling us to determine fine-scale functional connectivity. Comparing the functional connectivity patterns observed when different stimuli were presented to the mice, we made several nontrivial observations. First, while the frequencies of different connectivity motifs (i.e., the patterns of connectivity between triplets of neurons) were preserved across stimuli, the identities of the neurons within those motifs changed. This means that functional connectivity dynamically changes along with the input stimulus, but does so in a way that preserves the motif frequencies. Secondly, we found that the degree to which functional modules are contained within a single brain region (as opposed to being distributed between regions) increases with increasing stimulus complexity. This suggests a mechanism for how the brain could dynamically alter its computations based on its inputs. Altogether, our work reveals unexpected stimulus-dependence to the way groups of neurons interact to process incoming sensory information.

A visual encoding model links magnetoencephalography signals to neural synchrony in human cortex

Synchronization of neuronal responses over large distances is hypothesized to be important for many cortical functions. However, no straightforward methods exist to estimate synchrony non-invasively in the living human brain. MEG and EEG measure the whole brain, but the sensors pool over large, overlapping cortical regions, obscuring the underlying neural synchrony. Here, we developed a model from stimulus to cortex to MEG sensors to disentangle neural synchrony from spatial pooling of the instrument. We find that synchrony across cortex has a surprisingly large and systematic effect on predicted MEG spatial topography. We then conducted visual MEG experiments and separated responses into stimulus-locked and broadband components. The stimulus-locked topography was similar to model predictions assuming synchronous neural sources, whereas the broadband topography was similar to model predictions assuming asynchronous sources. We infer that visual stimulation elicits two distinct types of neural responses, one highly synchronous and one largely asynchronous across cortex.

Primary visual cortex straightens natural video trajectories

Many sensory-driven behaviors rely on predictions about future states of the environment. Visual input typically evolves along complex temporal trajectories that are difficult to extrapolate. We test the hypothesis that spatial processing mechanisms in the early visual system facilitate prediction by constructing neural representations that follow straighter temporal trajectories. We recorded V1 population activity in anesthetized macaques while presenting static frames taken from brief video clips, and developed a procedure to measure the curvature of the associated neural population trajectory. We found that V1 populations straighten naturally occurring image sequences, but entangle artificial sequences that contain unnatural temporal transformations. We show that these effects arise in part from computational mechanisms that underlie the stimulus selectivity of V1 cells. Together, our findings reveal that the early visual system uses a set of specialized computations to build representations that can support prediction in the natural environment.

Many behaviours depend on predictions about the environment. Here the authors find neural populations in primary visual cortex to straighten the temporal trajectories of natural video clips, facilitating the extrapolation of past observations.

Nonlinear spatial integration in retinal bipolar cells shapes the encoding of artificial and natural stimuli

The retina dissects the visual scene into parallel information channels, which extract specific visual features through nonlinear processing. The first nonlinear stage is typically considered to occur at the output of bipolar cells, resulting from nonlinear transmitter release from synaptic terminals. In contrast, we show here that bipolar cells themselves can act as nonlinear processing elements at the level of their somatic membrane potential. Intracellular recordings from bipolar cells in the salamander retina revealed frequent nonlinear integration of visual signals within bipolar cell receptive field centers, affecting the encoding of artificial and natural stimuli. These nonlinearities provide sensitivity to spatial structure below the scale of bipolar cell receptive fields in both bipolar and downstream ganglion cells and appear to arise at the excitatory input into bipolar cells. Thus, our data suggest that nonlinear signal pooling starts earlier than previously thought: that is, at the input stage of bipolar cells.

•

Some retinal bipolar cells represent visual contrast in a nonlinear fashion

•

These bipolar cells also nonlinearly integrate visual signals over space

•

The spatial nonlinearity affects the encoding of natural stimuli by bipolar cells

•

The nonlinearity results from feedforward input, not from feedback inhibition

Action-induced changes in the perceived temporal features of visual events

Perceived duration can be subject to deviations around the time of a voluntary action. Whether the mechanisms underlying action-induced visual duration effects are effector-specific or require a more generalized action-linked multimodal calibration with the transient visual system, however, is a question yet to be answered. Here, we investigate this using dynamic visual stimuli presented as contingent upon the execution of an arbitrarily associated voluntary manual response. Our results demonstrate that the duration of intervals with arbitrarily associated keypress-visual event pair is perceived as shorter than the duration in a pure visual condition, where the same stimuli are rather passively observed without the execution of a concurrent action. Whereas the control experiments show that motor memory and attention cannot explain the action-induced changes in perceived temporal features, action-induced changes in perceived speed are dissociated from those in perceived duration, and that the duration compression disappears using isoluminant or static stimuli, which together provide evidence that these two effects can be modulated in the motion-processing units, although via separate neural mechanisms.

Separable gain control of ongoing and evoked activity in the visual cortex by serotonergic input

Controlling gain of cortical activity is essential to modulate weights between internal ongoing communication and external sensory drive. Here, we show that serotonergic input has separable suppressive effects on the gain of ongoing and evoked visual activity. We combined optogenetic stimulation of the dorsal raphe nucleus (DRN) with wide-field calcium imaging, extracellular recordings, and iontophoresis of serotonin (5-HT) receptor antagonists in the mouse visual cortex. 5-HT1A receptors promote divisive suppression of spontaneous activity, while 5-HT2A receptors act divisively on visual response gain and largely account for normalization of population responses over a range of visual contrasts in awake and anesthetized states. Thus, 5-HT input provides balanced but distinct suppressive effects on ongoing and evoked activity components across neuronal populations. Imbalanced 5-HT1A/2A activation, either through receptor-specific drug intake, genetically predisposed irregular 5-HT receptor density, or change in sensory bombardment may enhance internal broadcasts and reduce sensory drive and vice versa.

The statistics of how natural images drive the responses of neurons

To model the responses of neurons in the early visual system, at least three basic components are required: a receptive field, a normalization term, and a specification of encoding noise. Here, we examine how the receptive field, the normalization factor, and the encoding noise affect the drive to model-neuron responses when stimulated with natural images. We show that when these components are modeled appropriately, the response drives elicited by natural stimuli are Gaussian-distributed and scale invariant, and very nearly maximize the sensitivity (d') for natural-image discrimination. We discuss the statistical models of natural stimuli that can account for these response statistics, and we show how some commonly used modeling practices may distort these results. Finally, we show that normalization can equalize important properties of neural response across different stimulus types. Specifically, narrowband (stimulus- and feature-specific) normalization causes model neurons to yield Gaussian response-drive statistics when stimulated with natural stimuli, 1/f noise stimuli, and white-noise stimuli. The current work makes recommendations for best practices and lays a foundation, grounded in the response statistics to natural stimuli, upon which to build principled models of more complex visual tasks.

Bisphenol A exposure inhibits contrast sensitivity in cats involving increased response noise and inhibitory synaptic transmission

Contrast sensitivity (CS) is one of the primary fundamental factors determining how well we can see, and it directly influences object recognition. Whether bisphenol-A (BPA, an environmental xenoestrogen) can perturb contrast detection in the visual system has yet to be elucidated. In the present study, we analyzed CS of single neurons in the primary visual cortex (area 17, A17) of cats before and after BPA exposure using a multiple-channel recording technique. The results showed that CS of A17 neurons was markedly depressed with an increased contrast threshold after two hour of intravenous BPA administration, which had a positive correlation with decreased firing rates of A17 neurons. Additionally, responses of these neurons presented an overt increase in the trial-to-trail response variability (a kind of neuronal noise), which could disturb the information-filtering function of single neurons. We also found that neuronal CS in the visual relay station was not disturbed after BPA administration, which rules out the contribution of CS alteration in the optical pathway. Importantly, acute BPA treatment obviously increased the inhibitory innervation to the visual pyramidal neurons. This implies that alteration of intracortical inhibitory regulation contributes to the compromised contrast detection in the visual system after BPA treatment.

Dynamic Signal Compression for Robust Motion Vision in Flies

Sensory systems need to reliably extract information from highly variable natural signals. Flies, for instance, use optic flow to guide their course and are remarkably adept at estimating image velocity regardless of image statistics. Current circuit models, however, cannot account for this robustness. Here, we demonstrate that the Drosophila visual system reduces input variability by rapidly adjusting its sensitivity to local contrast conditions. We exhaustively map functional properties of neurons in the motion detection circuit and find that local responses are compressed by surround contrast. The compressive signal is fast, integrates spatially, and derives from neural feedback. Training convolutional neural networks on estimating the velocity of natural stimuli shows that this dynamic signal compression can close the performance gap between model and organism. Overall, our work represents a comprehensive mechanistic account of how neural systems attain the robustness to carry out survival-critical tasks in challenging real-world environments.

An image-computable model for the stimulus selectivity of gamma oscillations

Gamma oscillations in visual cortex have been hypothesized to be critical for perception, cognition, and information transfer. However, observations of these oscillations in visual cortex vary widely; some studies report little to no stimulus-induced narrowband gamma oscillations, others report oscillations for only some stimuli, and yet others report large oscillations for most stimuli. To better understand this signal, we developed a model that predicts gamma responses for arbitrary images and validated this model on electrocorticography (ECoG) data from human visual cortex. The model computes variance across the outputs of spatially pooled orientation channels, and accurately predicts gamma amplitude across 86 images. Gamma responses were large for a small subset of stimuli, differing dramatically from fMRI and ECoG broadband (non-oscillatory) responses. We propose that gamma oscillations in visual cortex serve as a biomarker of gain control rather than being a fundamental mechanism for communicating visual information.

Predicting neuronal dynamics with a delayed gain control model

Visual neurons respond to static images with specific dynamics: neuronal responses sum sub-additively over time, reduce in amplitude with repeated or sustained stimuli (neuronal adaptation), and are slower at low stimulus contrast. Here, we propose a simple model that predicts these seemingly disparate response patterns observed in a diverse set of measurements-intracranial electrodes in patients, fMRI, and macaque single unit spiking. The model takes a time-varying contrast time course of a stimulus as input, and produces predicted neuronal dynamics as output. Model computation consists of linear filtering, expansive exponentiation, and a divisive gain control. The gain control signal relates to but is slower than the linear signal, and this delay is critical in giving rise to predictions matched to the observed dynamics. Our model is simpler than previously proposed related models, and fitting the model to intracranial EEG data uncovers two regularities across human visual field maps: estimated linear filters (temporal receptive fields) systematically differ across and within visual field maps, and later areas exhibit more rapid and substantial gain control. The model is further generalizable to account for dynamics of contrast-dependent spike rates in macaque V1, and amplitudes of fMRI BOLD in human V1.

Data-Driven Approaches to Understanding Visual Neuron Activity

With modern neurophysiological methods able to record neural activity throughout the visual pathway in the context of arbitrarily complex visual stimulation, our understanding of visual system function is becoming limited by the available models of visual neurons that can be directly related to such data. Different forms of statistical models are now being used to probe the cellular and circuit mechanisms shaping neural activity, understand how neural selectivity to complex visual features is computed, and derive the ways in which neurons contribute to systems-level visual processing. However, models that are able to more accurately reproduce observed neural activity often defy simple interpretations. As a result, rather than being used solely to connect with existing theories of visual processing, statistical modeling will increasingly drive the evolution of more sophisticated theories.

Functional characterization of retinal ganglion cells using tailored nonlinear modeling

The mammalian retina encodes the visual world in action potentials generated by 20-50 functionally and anatomically-distinct types of retinal ganglion cell (RGC). Individual RGC types receive synaptic input from distinct presynaptic circuits; therefore, their responsiveness to specific features in the visual scene arises from the information encoded in synaptic input and shaped by postsynaptic signal integration and spike generation. Unfortunately, there is a dearth of tools for characterizing the computations reflected in RGC spike output. Therefore, we developed a statistical model, the separable Nonlinear Input Model, to characterize the excitatory and suppressive components of RGC receptive fields. We recorded RGC responses to a correlated noise ("cloud") stimulus in an in vitro preparation of mouse retina and found that our model accurately predicted RGC responses at high spatiotemporal resolution. It identified multiple receptive fields reflecting the main excitatory and suppressive components of the response of each neuron. Significantly, our model accurately identified ON-OFF cells and distinguished their distinct ON and OFF receptive fields, and it demonstrated a diversity of suppressive receptive fields in the RGC population. In total, our method offers a rich description of RGC computation and sets a foundation for relating it to retinal circuitry.

Short-term global motion adaptation induces a compression in the subjective duration of dynamic visual events

Apparent duration can be manipulated in a local region of visual field by long-term adaptation to motion or flicker (Johnston, Arnold, & Nishida, 2006). These effects show narrow spatial tuning (Ayhan, Bruno, Nishida, & Johnston, 2009), as well as retinotopic position dependency (Bruno, Ayhan, & Johnston, 2010), supporting early locus in the visual pathway. Here, we introduce a new effect using RDK as a short-term visual adaptor and demonstrate that a brief, subsecond range adaptation induces a significant subjective duration compression (∼10%) on a subsequently presented test stimulus (RDK pattern) only for global motion patterns drifting at 50% motion coherence but not for those drifting at 0% coherence, suggesting a higher level area as a source of origin. In another set of experiments using a plaid stimulus as the adaptor and gratings as the tests, we report again a significant duration compression following a brief motion adaptation, although the effect does not seem to be consistently selective for a particular direction of the standard test relative to that of the plaid adaptor (two-dimensional motion) or its components (one-dimensional motion). Finally, we conduct an experiment using shutter glasses and find that the effects of a short-term adaptor presented monocularly to one eye transfer to the nonadapted eye, providing evidence for the interocular transfer. In a series of control experiments, we also show that the duration effects cannot be explained by adaptation-induced changes in perceived speed, perceived onset-and-offset, and attentional resource allocation. Overall, the duration compression effect requiring motion coherence in RDK, persisting in plaid stimulus, and showing interocular transfer imply explicit genuine mechanisms mediating duration effects in the higher level motion areas.

Perceptual straightening of natural videos

Many behaviors rely on predictions derived from recent visual input, but the temporal evolution of those inputs is generally complex and difficult to extrapolate. We propose that the visual system transforms these inputs to follow straighter temporal trajectories. To test this 'temporal straightening' hypothesis, we develop a methodology for estimating the curvature of an internal trajectory from human perceptual judgments. We use this to test three distinct predictions: natural sequences that are highly curved in the space of pixel intensities should be substantially straighter perceptually; in contrast, artificial sequences that are straight in the intensity domain should be more curved perceptually; finally, naturalistic sequences that are straight in the intensity domain should be relatively less curved. Perceptual data validate all three predictions, as do population models of the early visual system, providing evidence that the visual system specifically straightens natural videos, offering a solution for tasks that rely on prediction.

Firing-rate based network modeling of the dLGN circuit: Effects of cortical feedback on spatiotemporal response properties of relay cells

Visually evoked signals in the retina pass through the dorsal geniculate nucleus (dLGN) on the way to the visual cortex. This is however not a simple feedforward flow of information: there is a significant feedback from cortical cells back to both relay cells and interneurons in the dLGN. Despite four decades of experimental and theoretical studies, the functional role of this feedback is still debated. Here we use a firing-rate model, the extended difference-of-Gaussians (eDOG) model, to explore cortical feedback effects on visual responses of dLGN relay cells. For this model the responses are found by direct evaluation of two- or three-dimensional integrals allowing for fast and comprehensive studies of putative effects of different candidate organizations of the cortical feedback. Our analysis identifies a special mixed configuration of excitatory and inhibitory cortical feedback which seems to best account for available experimental data. This configuration consists of (i) a slow (long-delay) and spatially widespread inhibitory feedback, combined with (ii) a fast (short-delayed) and spatially narrow excitatory feedback, where (iii) the excitatory/inhibitory ON-ON connections are accompanied respectively by inhibitory/excitatory OFF-ON connections, i.e. following a phase-reversed arrangement. The recent development of optogenetic and pharmacogenetic methods has provided new tools for more precise manipulation and investigation of the thalamocortical circuit, in particular for mice. Such data will expectedly allow the eDOG model to be better constrained by data from specific animal model systems than has been possible until now for cat. We have therefore made the Python tool pyLGN which allows for easy adaptation of the eDOG model to new situations.

On route from the retina to primary visual cortex, visually evoked signals have to pass through the dorsal lateral geniculate nucleus (dLGN). However, this is not an exclusive feedforward flow of information as feedback exists from neurons in the cortex back to both relay cells and interneurons in the dLGN. The functional role of this feedback remains mostly unresolved. Here, we use a firing-rate model, the extended difference-of-Gaussians (eDOG) model, to explore cortical feedback effects on visual responses of dLGN relay cells. Our analysis indicates that a particular mix of excitatory and inhibitory cortical feedback agrees best with available experimental observations. In this configuration ON-center relay cells receive both excitatory and (indirect) inhibitory feedback from ON-center cortical cells (ON-ON feedback) where the excitatory feedback is fast and spatially narrow while the inhibitory feedback is slow and spatially widespread. In addition to the ON-ON feedback, the connections are accompanied by OFF-ON connections following a so-called phase-reversed (push-pull) arrangement. To facilitate further applications of the model, we have made the Python tool pyLGN which allows for easy modification and evaluation of the a priori quite general eDOG model to new situations.

Contrast gain control and retinogeniculate communication

Visual information processed in the retina is transmitted to primary visual cortex via relay cells in the lateral geniculate nucleus (LGN) of the dorsal thalamus. Although retinal ganglion cells are the primary source of driving input to LGN neurons, not all retinal spikes are transmitted to the cortex. Here, we investigate the relationship between stimulus contrast and retinogeniculate communication and test the hypothesis that both the time course and strength of retinogeniculate interactions are dynamic and dependent on stimulus contrast. By simultaneously recording the spiking activity of synaptically connected retinal ganglion cells and LGN neurons in the cat, we show that the temporal window for retinogeniculate integration and the effectiveness of individual retinal spikes are inversely proportional to stimulus contrast. This finding provides a mechanistic understanding for the phenomenon of augmented contrast gain control in the LGN-a nonlinear receptive field property of LGN neurons whereby response gain during low-contrast stimulation is enhanced relative to response gain during high-contrast stimulation. In addition, these results support the view that network interactions beyond the retina play an essential role in transforming visual signals en route from retina to cortex.

Biophysical network modeling of the dLGN circuit: Effects of cortical feedback on spatial response properties of relay cells

Despite half-a-century of research since the seminal work of Hubel and Wiesel, the role of the dorsal lateral geniculate nucleus (dLGN) in shaping the visual signals is not properly understood. Placed on route from retina to primary visual cortex in the early visual pathway, a striking feature of the dLGN circuit is that both the relay cells (RCs) and interneurons (INs) not only receive feedforward input from retinal ganglion cells, but also a prominent feedback from cells in layer 6 of visual cortex. This feedback has been proposed to affect synchronicity and other temporal properties of the RC firing. It has also been seen to affect spatial properties such as the center-surround antagonism of thalamic receptive fields, i.e., the suppression of the response to very large stimuli compared to smaller, more optimal stimuli. Here we explore the spatial effects of cortical feedback on the RC response by means of a a comprehensive network model with biophysically detailed, single-compartment and multicompartment neuron models of RCs, INs and a population of orientation-selective layer 6 simple cells, consisting of pyramidal cells (PY). We have considered two different arrangements of synaptic feedback from the ON and OFF zones in the visual cortex to the dLGN: phase-reversed (‘push-pull’) and phase-matched (‘push-push’), as well as different spatial extents of the corticothalamic projection pattern. Our simulation results support that a phase-reversed arrangement provides a more effective way for cortical feedback to provide the increased center-surround antagonism seen in experiments both for flashing spots and, even more prominently, for patch gratings. This implies that ON-center RCs receive direct excitation from OFF-dominated cortical cells and indirect inhibitory feedback from ON-dominated cortical cells. The increased center-surround antagonism in the model is accompanied by spatial focusing, i.e., the maximum RC response occurs for smaller stimuli when feedback is present.

The functional role of the dorsal lateral geniculate nucleus (dLGN), placed on route from retina to primary visual cortex in the early visual pathway, is still poorly understood. A striking feature of the dLGN circuit is that dLGN cells not only receive feedforward input from the retina, but also a prominent feedback from cells in the visual cortex. It has been seen in experiments that cortical feedback modifies the spatial properties of dLGN cells in response to visual stimuli. In particular, it has been shown to increase the center-surround antagonism for flashing-spot and patch-grating visual stimuli, i.e., the suppression of responses to very large stimuli compared to smaller stimuli. Here we investigate the putative mechanisms behind this feature by means of a comprehensive network model of biophysically detailed neuron models for RCs and INs in the dLGN and orientation-selective cortical cells providing the feedback. Our results support that the experimentally observed feedback effects may be due to a phase-reversed (‘push-pull’) arrangement of the cortical feedback where ON-symmetry RCs receive (indirect) inhibitory feedback from ON-dominated cortical cell and excitation from OFF-dominated cortical cells, and vice versa for OFF-symmetry RCs.

Perceptually optimized image rendering

We develop a framework for rendering photographic images by directly optimizing their perceptual similarity to the original visual scene. Specifically, over the set of all images that can be rendered on a given display, we minimize the normalized Laplacian pyramid distance (NLPD), a measure of perceptual dissimilarity that is derived from a simple model of the early stages of the human visual system. When rendering images acquired with a higher dynamic range than that of the display, we find that the optimization boosts the contrast of low-contrast features without introducing significant artifacts, yielding results of comparable visual quality to current state-of-the-art methods, but without manual intervention or parameter adjustment. We also demonstrate the effectiveness of the framework for a variety of other display constraints, including limitations on minimum luminance (black point), mean luminance (as a proxy for energy consumption), and quantized luminance levels (halftoning). We show that the method may generally be used to enhance details and contrast, and, in particular, can be used on images degraded by optical scattering (e.g., fog). Finally, we demonstrate the necessity of each of the NLPD components-an initial power function, a multiscale transform, and local contrast gain control-in achieving these results and we show that NLPD is competitive with the current state-of-the-art image quality metrics.

Retinal and Nonretinal Contributions to Extraclassical Surround Suppression in the Lateral Geniculate Nucleus

Extraclassical surround suppression is a prominent receptive field property of neurons in the lateral geniculate nucleus (LGN) of the dorsal thalamus, influencing stimulus size tuning, response gain control, and temporal features of visual responses. Despite evidence for the involvement of both retinal and nonretinal circuits in the generation of extraclassical suppression, we lack an understanding of the relative roles played by these pathways and how they interact during visual stimulation. To determine the contribution of retinal and nonretinal mechanisms to extraclassical suppression in the feline, we made simultaneous single-unit recordings from synaptically connected retinal ganglion cells and LGN neurons and measured the influence of stimulus size on the spiking activity of presynaptic and postsynaptic neurons. Results show that extraclassical suppression is significantly stronger for LGN neurons than for their retinal inputs, indicating a role for extraretinal mechanisms. Further analysis revealed that the enhanced suppression can be accounted for by mechanisms that suppress the effectiveness of retinal inputs in evoking LGN spikes. Finally, an examination of the time course for the onset of extraclassical suppression in the LGN and the size-dependent modulation of retinal spike efficacy suggests the early phase of augmented suppression involves local thalamic circuits. Together, these results demonstrate that the LGN is much more than a simple relay for retinal signals to cortex; it also filters retinal spikes dynamically on the basis of stimulus statistics to adjust the gain of visual signals delivered to cortex.

SIGNIFICANCE STATEMENT

The lateral geniculate nucleus (LGN) is the gateway through which retinal information reaches the cerebral cortex. Within the LGN, neuronal responses are often suppressed by stimuli that extend beyond the classical receptive field. This form of suppression, called extraclassical suppression, serves to adjust the size tuning, response gain, and temporal response properties of neurons. Given the important influence of extraclassical suppression on visual signals delivered to cortex, we performed experiments to determine the circuit mechanisms that contribute to extraclassical suppression in the LGN. Results show that suppression is augmented beyond that provided by direct retinal inputs and delayed, consistent with polysynaptic inhibition. Importantly, these mechanisms influence the effectiveness of incoming retinal signals, thereby filtering the signals ultimately conveyed to cortex.

Divisive suppression explains high-precision firing and contrast adaptation in retinal ganglion cells

Visual processing depends on specific computations implemented by complex neural circuits. Here, we present a circuit-inspired model of retinal ganglion cell computation, targeted to explain their temporal dynamics and adaptation to contrast. To localize the sources of such processing, we used recordings at the levels of synaptic input and spiking output in the in vitro mouse retina. We found that an ON-Alpha ganglion cell's excitatory synaptic inputs were described by a divisive interaction between excitation and delayed suppression, which explained nonlinear processing that was already present in ganglion cell inputs. Ganglion cell output was further shaped by spike generation mechanisms. The full model accurately predicted spike responses with unprecedented millisecond precision, and accurately described contrast adaptation of the spike train. These results demonstrate how circuit and cell-intrinsic mechanisms interact for ganglion cell function and, more generally, illustrate the power of circuit-inspired modeling of sensory processing.

The multifunctional lateral geniculate nucleus

Providing the critical link between the retina and visual cortex, the well-studied lateral geniculate nucleus (LGN) has stood out as a structure in search of a function exceeding the mundane 'relay'. For many mammals, it is structurally impressive: Exquisite lamination, sophisticated microcircuits, and blending of multiple inputs suggest some fundamental transform. This impression is bolstered by the fact that numerically, the retina accounts for a small fraction of its input. Despite such promise, the extent to which an LGN neuron separates itself from its retinal brethren has proven difficult to appreciate. Here, I argue that whereas retinogeniculate coupling is strong, what occurs in the LGN is judicious pruning of a retinal drive by nonretinal inputs. These nonretinal inputs reshape a receptive field that under the right conditions departs significantly from its retinal drive, even if transiently. I first review design features of the LGN and follow with evidence for 10 putative functions. Only two of these tend to surface in textbooks: parsing retinal axons by eye and functional group and gating by state. Among the remaining putative functions, implementation of the principle of graceful degradation and temporal decorrelation are at least as interesting but much less promoted. The retina solves formidable problems imposed by physics to yield multiple efficient and sensitive representations of the world. The LGN applies context, increasing content, and gates several of these representations. Even if the basic concentric receptive field remains, information transmitted for each LGN spike relative to each retinal spike is measurably increased.

A Computational Framework for Realistic Retina Modeling

Computational simulations of the retina have led to valuable insights about the biophysics of its neuronal activity and processing principles. A great number of retina models have been proposed to reproduce the behavioral diversity of the different visual processing pathways. While many of these models share common computational stages, previous efforts have been more focused on fitting specific retina functions rather than generalizing them beyond a particular model. Here, we define a set of computational retinal microcircuits that can be used as basic building blocks for the modeling of different retina mechanisms. To validate the hypothesis that similar processing structures may be repeatedly found in different retina functions, we implemented a series of retina models simply by combining these computational retinal microcircuits. Accuracy of the retina models for capturing neural behavior was assessed by fitting published electrophysiological recordings that characterize some of the best-known phenomena observed in the retina: adaptation to the mean light intensity and temporal contrast, and differential motion sensitivity. The retinal microcircuits are part of a new software platform for efficient computational retina modeling from single-cell to large-scale levels. It includes an interface with spiking neural networks that allows simulation of the spiking response of ganglion cells and integration with models of higher visual areas.

Push-Pull Receptive Field Organization and Synaptic Depression: Mechanisms for Reliably Encoding Naturalistic Stimuli in V1

Neurons in the primary visual cortex are known for responding vigorously but with high variability to classical stimuli such as drifting bars or gratings. By contrast, natural scenes are encoded more efficiently by sparse and temporal precise spiking responses. We used a conductance-based model of the visual system in higher mammals to investigate how two specific features of the thalamo-cortical pathway, namely push-pull receptive field organization and fast synaptic depression, can contribute to this contextual reshaping of V1 responses. By comparing cortical dynamics evoked respectively by natural vs. artificial stimuli in a comprehensive parametric space analysis, we demonstrate that the reliability and sparseness of the spiking responses during natural vision is not a mere consequence of the increased bandwidth in the sensory input spectrum. Rather, it results from the combined impacts of fast synaptic depression and push-pull inhibition, the later acting for natural scenes as a form of “effective” feed-forward inhibition as demonstrated in other sensory systems. Thus, the combination of feedforward-like inhibition with fast thalamo-cortical synaptic depression by simple cells receiving a direct structured input from thalamus composes a generic computational mechanism for generating a sparse and reliable encoding of natural sensory events.

Prediction of primary somatosensory neuron activity during active tactile exploration

Primary sensory neurons form the interface between world and brain. Their function is well-understood during passive stimulation but, under natural behaving conditions, sense organs are under active, motor control. In an attempt to predict primary neuron firing under natural conditions of sensorimotor integration, we recorded from primary mechanosensory neurons of awake, head-fixed mice as they explored a pole with their whiskers, and simultaneously measured both whisker motion and forces with high-speed videography. Using Generalised Linear Models, we found that primary neuron responses were poorly predicted by whisker angle, but well-predicted by rotational forces acting on the whisker: both during touch and free-air whisker motion. These results are in apparent contrast to previous studies of passive stimulation, but could be reconciled by differences in the kinematics-force relationship between active and passive conditions. Thus, simple statistical models can predict rich neural activity elicited by natural, exploratory behaviour involving active movement of sense organs.

Cortical Correlates of Low-Level Perception: From Neural Circuits to Percepts

Low-level perception results from neural-based computations, which build a multimodal skeleton of unconscious or self-generated inferences on our environment. This review identifies bottleneck issues concerning the role of early primary sensory cortical areas, mostly in rodent and higher mammals (cats and non-human primates), where perception substrates can be searched at multiple scales of neural integration. We discuss the limitation of purely bottom-up approaches for providing realistic models of early sensory processing and the need for identification of fast adaptive processes, operating within the time of a percept. Future progresses will depend on the careful use of comparative neuroscience (guiding the choices of experimental models and species adapted to the questions under study), on the definition of agreed-upon benchmarks for sensory stimulation, on the simultaneous acquisition of neural data at multiple spatio-temporal scales, and on the in vivo identification of key generic integration and plasticity algorithms validated experimentally and in simulations.

The Essential Complexity of Auditory Receptive Fields

Encoding properties of sensory neurons are commonly modeled using linear finite impulse response (FIR) filters. For the auditory system, the FIR filter is instantiated in the spectro-temporal receptive field (STRF), often in the framework of the generalized linear model. Despite widespread use of the FIR STRF, numerous formulations for linear filters are possible that require many fewer parameters, potentially permitting more efficient and accurate model estimates. To explore these alternative STRF architectures, we recorded single-unit neural activity from auditory cortex of awake ferrets during presentation of natural sound stimuli. We compared performance of > 1000 linear STRF architectures, evaluating their ability to predict neural responses to a novel natural stimulus. Many were able to outperform the FIR filter. Two basic constraints on the architecture lead to the improved performance: (1) factorization of the STRF matrix into a small number of spectral and temporal filters and (2) low-dimensional parameterization of the factorized filters. The best parameterized model was able to outperform the full FIR filter in both primary and secondary auditory cortex, despite requiring fewer than 30 parameters, about 10% of the number required by the FIR filter. After accounting for noise from finite data sampling, these STRFs were able to explain an average of 40% of A1 response variance. The simpler models permitted more straightforward interpretation of sensory tuning properties. They also showed greater benefit from incorporating nonlinear terms, such as short term plasticity, that provide theoretical advances over the linear model. Architectures that minimize parameter count while maintaining maximum predictive power provide insight into the essential degrees of freedom governing auditory cortical function. They also maximize statistical power available for characterizing additional nonlinear properties that limit current auditory models.

Understanding how the brain solves sensory problems can provide useful insight for the development of automated systems such as speech recognizers and image classifiers. Recent developments in nonlinear regression and machine learning have produced powerful algorithms for characterizing the input-output relationship of complex systems. However, the complexity of sensory neural systems, combined with practical limitations on experimental data, make it difficult to apply arbitrarily complex analyses to neural data. In this study we pushed analysis in the opposite direction, toward simpler models. We asked how simple a model can be while still capturing the essential sensory properties of neurons in auditory cortex. We found that substantially simpler formulations of the widely-used spectro-temporal receptive field are able to perform as well as the best current models. These simpler formulations define new basis sets that can be incorporated into state-of-the-art machine learning algorithms for a more exhaustive exploration of sensory processing.

Visual Functions of the Thalamus

The thalamus is the heavily interconnected partner of the neocortex. All areas of the neocortex receive afferent input from and send efferent projections to specific thalamic nuclei. Through these connections, the thalamus serves to provide the cortex with sensory input, and to facilitate interareal cortical communication and motor and cognitive functions. In the visual system, the lateral geniculate nucleus (LGN) of the dorsal thalamus is the gateway through which visual information reaches the cerebral cortex. Visual processing in the LGN includes spatial and temporal influences on visual signals that serve to adjust response gain, transform the temporal structure of retinal activity patterns, and increase the signal-to-noise ratio of the retinal signal while preserving its basic content. This review examines recent advances in our understanding of LGN function and circuit organization and places these findings in a historical context.

Mapping nonlinear receptive field structure in primate retina at single cone resolution

The function of a neural circuit is shaped by the computations performed by its interneurons, which in many cases are not easily accessible to experimental investigation. Here, we elucidate the transformation of visual signals flowing from the input to the output of the primate retina, using a combination of large-scale multi-electrode recordings from an identified ganglion cell type, visual stimulation targeted at individual cone photoreceptors, and a hierarchical computational model. The results reveal nonlinear subunits in the circuity of OFF midget ganglion cells, which subserve high-resolution vision. The model explains light responses to a variety of stimuli more accurately than a linear model, including stimuli targeted to cones within and across subunits. The recovered model components are consistent with known anatomical organization of midget bipolar interneurons. These results reveal the spatial structure of linear and nonlinear encoding, at the resolution of single cells and at the scale of complete circuits.

DOI:http://dx.doi.org/10.7554/eLife.05241.001

Light that enters the eye begins the process of vision by activating two types of photoreceptors: rods, which support vision under low light levels, and cones, which are responsible for fine detail and color vision. Activation of either type of photoreceptor triggers responses in bipolar cells, which activate the ganglion cells that transmit visual signals to the brain.

Bipolar cells therefore belong to a class of cells called interneurons, which relay information from certain cell types to others. Interneurons play an important role in information processing throughout the brain, but directly accessing them or characterizing their role in neural computation is often difficult.

To address this problem, Freeman, Field et al. have developed a combined computational and experimental approach to describe the flow of sensory signals through the circuits within the retina of primates. Large arrays of electrodes were used to record the responses of many retinal ganglion cells in response to the activation or de-activation of pairs of cones. These experiments revealed that the responses of ganglion cells are not simply the sum of the inputs that they receive from cones; specifically, the activation of one cone is not cancelled by the deactivation of another. Instead, the data suggest that bipolar cells add cone inputs together and then pass on the total activation (but not deactivation) to ganglion cells.

By analyzing the responses of ganglion cells to numerous random patterns of cone activation, Freeman, Field et al. were able to estimate the locations and arrangements of bipolar cells that connect to them. These predicted patterns of connectivity agreed with observations from anatomical studies. This work provides detailed insights into how the primate retina works. It also suggests that similar approaches may be used to characterize how signals flow across other brain networks in which large-scale recordings are now possible.

DOI:http://dx.doi.org/10.7554/eLife.05241.002

Spike-Triggered Covariance Analysis Reveals Phenomenological Diversity of Contrast Adaptation in the Retina

When visual contrast changes, retinal ganglion cells adapt by adjusting their sensitivity as well as their temporal filtering characteristics. The latter has classically been described by contrast-induced gain changes that depend on temporal frequency. Here, we explored a new perspective on contrast-induced changes in temporal filtering by using spike-triggered covariance analysis to extract multiple parallel temporal filters for individual ganglion cells. Based on multielectrode-array recordings from ganglion cells in the isolated salamander retina, we found that contrast adaptation of temporal filtering can largely be captured by contrast-invariant sets of filters with contrast-dependent weights. Moreover, differences among the ganglion cells in the filter sets and their contrast-dependent contributions allowed us to phenomenologically distinguish three types of filter changes. The first type is characterized by newly emerging features at higher contrast, which can be reproduced by computational models that contain response-triggered gain-control mechanisms. The second type follows from stronger adaptation in the Off pathway as compared to the On pathway in On-Off-type ganglion cells. Finally, we found that, in a subset of neurons, contrast-induced filter changes are governed by particularly strong spike-timing dynamics, in particular by pronounced stimulus-dependent latency shifts that can be observed in these cells. Together, our results show that the contrast dependence of temporal filtering in retinal ganglion cells has a multifaceted phenomenology and that a multi-filter analysis can provide a useful basis for capturing the underlying signal-processing dynamics.

Our sensory systems have to process stimuli under a wide range of environmental conditions. To cope with this challenge, the involved neurons adapt by adjusting their signal processing to the recently encountered intensity range. In the visual system, one finds, for example, that higher visual contrast leads to changes in how visual signals are temporally filtered, making signal processing faster and more band-pass-like at higher contrast. By analyzing signals from neurons in the retina of salamanders, we here found that these adaptation effects can be described by a fixed set of filters, independent of contrast, whose relative contributions change with contrast. Also, we found that different phenomena contribute to this adaptation. In particular, some cells change their relative sensitivity to light increments and light decrements, whereas other cells are influenced by a strong contrast-dependence of the exact timing of their responses. Our results show that contrast adaptation in the retina is not an entirely homogeneous phenomenon, and that models with multiple filters can help in characterizing sensory adaptation.

Cortical state determines global variability and correlations in visual cortex

The response of neurons in sensory cortex to repeated stimulus presentations is highly variable. To investigate the nature of this variability, we compared the spike activity of neurons in the primary visual cortex (V1) of cats with that of their afferents from lateral geniculate nucleus (LGN), in response to similar stimuli. We found variability to be much higher in V1 than in LGN. To investigate the sources of the additional variability, we measured the spiking activity of large V1 populations and found that much of the variability was shared across neurons: the variable portion of the responses of one neuron could be well predicted from the summed activity of the rest of the neurons. Variability thus mostly reflected global fluctuations affecting all neurons. The size and prevalence of these fluctuations, both in responses to stimuli and in ongoing activity, depended on cortical state, being larger in synchronized states than in more desynchronized states. Contrary to previous reports, these fluctuations invested the overall population, regardless of preferred orientation. The global fluctuations substantially increased variability in single neurons and correlations among pairs of neurons. Once this effect was removed, pairwise correlations were reduced and were similar regardless of cortical state. These results highlight the importance of cortical state in controlling cortical operation and can help reconcile previous studies, which differed widely in their estimate of neuronal variability and pairwise correlations.

Surround suppression and temporal processing of visual signals

Extraclassical surround suppression strongly modulates responses of neurons in the retina, lateral geniculate nucleus (LGN), and primary visual cortex. Although a great deal is known about the spatial properties of extraclassical suppression and the role it serves in stimulus size tuning, relatively little is known about how extraclassical suppression shapes visual processing in the temporal domain. We recorded the spiking activity of retinal ganglion cells and LGN neurons in the cat to test the hypothesis that extraclassical suppression influences temporal features of visual responses in the early visual system. Our results demonstrate that extraclassical suppression not only shifts the distribution of interspike intervals in a manner that decreases the efficacy of neuronal communication, it also decreases the reliability of neuronal responses to visual stimuli and it decreases the duration of visual responses, an effect that underlies a rightward shift in the temporal frequency tuning of LGN neurons. Taken together, these results reveal a dynamic relationship between extraclassical suppression and the temporal features of neuronal responses.

Understanding mid-level representations in visual processing

It is clear that early visual processing provides an image-based representation of the visual scene: Neurons in Striate cortex (V1) encode nothing about the meaning of a scene, but they do provide a great deal of information about the image features within it. The mechanisms of these "low-level" visual processes are relatively well understood. We can construct plausible models for how neurons, up to and including those in V1, build their representations from preceding inputs down to the level of photoreceptors. It is also clear that at some point we have a semantic, "high-level" representation of the visual scene because we can describe verbally the objects that we are viewing and their meaning to us. A huge number of studies are examining these "high-level" visual processes each year. Less well studied are the processes of "mid-level" vision, which presumably provide the bridge between these "low-level" representations of edges, colors, and lights and the "high-level" semantic representations of objects, faces, and scenes. This article and the special issue of papers in which it is published consider the nature of "mid-level" visual processing and some of the reasons why we might not have made as much progress in this domain as we would like.

Frequency-band signatures of visual responses to naturalistic input in ferret primary visual cortex during free viewing

Neuronal firing responses in visual cortex reflect the statistics of visual input and emerge from the interaction with endogenous network dynamics. Artificial visual stimuli presented to animals in which the network dynamics were constrained by anesthetic agents or trained behavioral tasks have provided fundamental understanding of how individual neurons in primary visual cortex respond to input. In contrast, very little is known about the mesoscale network dynamics and their relationship to microscopic spiking activity in the awake animal during free viewing of naturalistic visual input. To address this gap in knowledge, we recorded local field potential (LFP) and multiunit activity (MUA) simultaneously in all layers of primary visual cortex (V1) of awake, freely viewing ferrets presented with naturalistic visual input (nature movie clips). We found that naturalistic visual stimuli modulated the entire oscillation spectrum; low frequency oscillations were mostly suppressed whereas higher frequency oscillations were enhanced. In average across all cortical layers, stimulus-induced change in delta and alpha power negatively correlated with the MUA responses, whereas sensory-evoked increases in gamma power positively correlated with MUA responses. The time-course of the band-limited power in these frequency bands provided evidence for a model in which naturalistic visual input switched V1 between two distinct, endogenously present activity states defined by the power of low (delta, alpha) and high (gamma) frequency oscillatory activity. Therefore, the two mesoscale activity states delineated in this study may define the degree of engagement of the circuit with the processing of sensory input.

From image processing to computational neuroscience: a neural model based on histogram equalization

There are many ways in which the human visual system works to reduce the inherent redundancy of the visual information in natural scenes, coding it in an efficient way. The non-linear response curves of photoreceptors and the spatial organization of the receptive fields of visual neurons both work toward this goal of efficient coding. A related, very important aspect is that of the existence of post-retinal mechanisms for contrast enhancement that compensate for the blurring produced in early stages of the visual process. And alongside mechanisms for coding and wiring efficiency, there is neural activity in the human visual cortex that correlates with the perceptual phenomenon of lightness induction. In this paper we propose a neural model that is derived from an image processing technique for histogram equalization, and that is able to deal with all the aspects just mentioned: this new model is able to predict lightness induction phenomena, and improves the efficiency of the representation by flattening both the histogram and the power spectrum of the image signal.

Modeling lateral geniculate nucleus response with contrast gain control. Part 2: analysis

Cope et al. [J. Opt. Soc. Am. A30, 2401 (2013)] proposed a class of models for lateral geniculate nucleus (LGN) ON-cell behavior consisting of a linear response with divisive normalization by local stimulus contrast. Here, we analyze a specific model with the linear response defined by a difference-of-Gaussians filter, and a circular Gaussian for the gain pool weighting function. For sinusoidal grating stimuli, the parameter region for bandpass behavior of the linear response is determined, and the gain control response is shown to act as a switch (changing from "off" to "on" with increasing spatial frequency). It is also shown that large gain pools stabilize the optimal spatial frequency of the total nonlinear response at a fixed value independent of contrast and stimulus magnitude. Under- and super-saturation, as well as contrast saturation, occur as typical effects of stimulus magnitude. For circular spot stimuli, it is shown that large gain pools stabilize the spot size that yields the maximum response.

Visual pattern discrimination by population retinal ganglion cells' activities during natural movie stimulation

In the visual system, neurons often fire in synchrony, and it is believed that synchronous activities of group neurons are more efficient than single cell response in transmitting neural signals to down-stream neurons. However, whether dynamic natural stimuli are encoded by dynamic spatiotemporal firing patterns of synchronous group neurons still needs to be investigated. In this paper we recorded the activities of population ganglion cells in bullfrog retina in response to time-varying natural images (natural scene movie) using multi-electrode arrays. In response to some different brief section pairs of the movie, synchronous groups of retinal ganglion cells (RGCs) fired with similar but different spike events. We attempted to discriminate the movie sections based on temporal firing patterns of single cells and spatiotemporal firing patterns of the synchronous groups of RGCs characterized by a measurement of subsequence distribution discrepancy. The discrimination performance was assessed by a classification method based on Support Vector Machines. Our results show that different movie sections of the natural movie elicited reliable dynamic spatiotemporal activity patterns of the synchronous RGCs, which are more efficient in discriminating different movie sections than the temporal patterns of the single cells' spike events. These results suggest that, during natural vision, the down-stream neurons may decode the visual information from the dynamic spatiotemporal patterns of the synchronous group of RGCs' activities.