Local luminance and contrast in natural images

Both a Gauge and a Filter: Cognitive Modulations of Pupil Size

Over 50 years of research have established that cognitive processes influence pupil size. This has led to the widespread use of pupil size as a peripheral measure of cortical processing in psychology and neuroscience. However, the function of cortical control over the pupil remains poorly understood. Why does visual attention change the pupil light reflex? Why do mental effort and surprise cause pupil dilation? Here, we consider these functional questions as we review and synthesize two literatures on cognitive effects on the pupil: how cognition affects pupil light response and how cognition affects pupil size under constant luminance. We propose that cognition may have co-opted control of the pupil in order to filter incoming visual information to optimize it for particular goals. This could complement other cortical mechanisms through which cognition shapes visual perception.

Visual evoked cortical potential elicited by pseudoisochromatic stimulus

PURPOSE

Visual evoked cortical potentials (VECPs) are useful for investigating the mechanisms and dysfunctions of color vision. Chromatic sinusoidal gratings are generally used to elicit VECPs, but they require long psychophysical measurements to match the perceptual luminance between their stripes. An alternative method is to use pseudoisochromatic stimuli, which makes use of luminance noise to mask luminance clues and force the target perception to be dependent on chromatic contrast. In this study, we compared VECPs generated by sinusoidal gratings and pseudoisochromatic gratings. Contrary to chromatic sinusoidal gratings, pseudoisochromatic stimuli do not require the use of previous methods to find the equiluminance of the stimulus.

METHODS

Normal trichromats were recruited to be tested with red-green chromatic sinusoidal gratings and pseudoisochromatic gratings presented by pattern onset-offset and pattern reversal modes in five spatial frequencies. In addition, we also tested four different chromatic contrast pairs in pattern onset-offset mode presentation in five trichromats and one colorblind subject (deuteranope).

RESULTS

Pattern onset-offset VECPs elicited by sinusoidal gratings had a larger amplitude than those obtained with pseudoisochromatic stimuli, whereas pattern reversal VECPs elicited by pseudoisochromatic gratings had similar amplitudes compared to those elicited by sinusoidal gratings. We found no difference between the VECP amplitudes elicited by sinusoidal and pseudoisochromatic gratings containing different chromatic contrast. Color-blind subjects displayed absent or small responses to the stimuli.

CONCLUSION

Pseudoisochromatic stimulus can be an alternative stimulus to generate VECPs dominated by the chromatic mechanism.

Stimulus- and goal-oriented frameworks for understanding natural vision

Our knowledge of sensory processing has advanced dramatically in the last few decades, but this understanding remains far from complete, especially for stimuli with the large dynamic range and strong temporal and spatial correlations characteristic of natural visual inputs. Here we describe some of the issues that make understanding the encoding of natural images a challenge. We highlight two broad strategies for approaching this problem: a stimulus-oriented framework and a goal-oriented one. Different contexts can call for one framework or the other. Looking forward, recent advances, particularly those based in machine learning, show promise in borrowing key strengths of both frameworks and by doing so illuminating a path to a more comprehensive understanding of the encoding of natural stimuli.

The Perpetual Diamond: Contrast Reversals Along Thin Edges Create the Appearance of Motion in Objects

The Perpetual Diamond produces motion continuously and unambiguously in one direction despite never physically changing location. The phenomenon consists of a steady, mid-luminance diamond bordered by four thin edge strips and a surrounding background field. The direction of motion is determined by the relative phases of the luminance modulation between the edge strips and the background. Because the motion is generated entirely by changing contrast signals between the edge strips and background, the stimulus is a valuable tool for tests of spatial contrast, temporal contrast, contrast gain, and color contrast. We demonstrate that observers see motion even when the edge strips subtend only seconds of arc on the retina (which is less than the frequently reported 10 minutes of arc) and that perceived motion is due entirely to changes in the difference in contrast phase modulation, independent from the luminance phase.

The dynamic receptive fields of retinal ganglion cells

Retinal ganglion cells (RGCs) were one of the first classes of sensory neurons to be described in terms of a receptive field (RF). Over the last six decades, our understanding of the diversity of RGC types and the nuances of their response properties has grown exponentially. We will review the current understanding of RGC RFs mostly from studies in mammals, but including work from other vertebrates as well. We will argue for a new paradigm that embraces the fluidity of RGC RFs with an eye toward the neuroethology of vision. Specifically, we will focus on (1) different methods for measuring RGC RFs, (2) RF models, (3) feature selectivity and the distinction between fluid and stable RF properties, and (4) ideas about the future of understanding RGC RFs.

Natural image and receptive field statistics predict saccade sizes

Humans and other primates sample the visual environment using saccadic eye movements that shift a high-resolution fovea toward regions of interest to create a clear perception of a scene across fixations. Many mammals, however, like mice, lack a fovea, which raises the question of why they make saccades. Here we describe and test the hypothesis that saccades are matched to natural scene statistics and to the receptive field sizes and adaptive properties of neural populations. Specifically, we determined the minimum amplitude of saccades in natural scenes necessary to provide uncorrelated inputs to model neural populations. This analysis predicts the distributions of observed saccade sizes during passive viewing for nonhuman primates, cats, and mice. Furthermore, disrupting the development of receptive field properties by monocular deprivation changed saccade sizes consistent with this hypothesis. Therefore, natural-scene statistics and the neural representation of natural images appear to be critical factors guiding saccadic eye movements.

No Reference Quality Assessment for Screen Content Images With Both Local and Global Feature Representation

In this paper, we propose a novel no reference quality assessment method by incorporating statistical luminance and texture features (NRLT) for screen content images (SCIs) with both local and global feature representation. The proposed method is designed inspired by the perceptual property of the human visual system (HVS) that the HVS is sensitive to luminance change and texture information for image perception. In the proposed method, we first calculate the luminance map through the local normalization, which is further used to extract the statistical luminance features in global scope. Second, inspired by existing studies from neuroscience that high-order derivatives can capture image texture, we adopt four filters with different directions to compute gradient maps from the luminance map. These gradient maps are then used to extract the second-order derivatives by local binary pattern. We further extract the texture feature by the histogram of high-order derivatives in global scope. Finally, support vector regression is applied to train the mapping function from quality-aware features to subjective ratings. Experimental results on the public large-scale SCI database show that the proposed NRLT can achieve better performance in predicting the visual quality of SCIs than relevant existing methods, even including some full reference visual quality assessment methods.

Neuronal mechanisms underlying differences in spatial resolution between darks and lights in human vision

Artists and astronomers noticed centuries ago that humans perceive dark features in an image differently from light ones; however, the neuronal mechanisms underlying these dark/light asymmetries remained unknown. Based on computational modeling of neuronal responses, we have previously proposed that such perceptual dark/light asymmetries originate from a luminance/response saturation within the ON retinal pathway. Consistent with this prediction, here we show that stimulus conditions that increase ON luminance/response saturation (e.g., dark backgrounds) or its effect on light stimuli (e.g., optical blur) impair the perceptual discrimination and salience of light targets more than dark targets in human vision. We also show that, in cat visual cortex, the magnitude of the ON luminance/response saturation remains relatively constant under a wide range of luminance conditions that are common indoors, and only shifts away from the lowest luminance contrasts under low mesopic light. Finally, we show that the ON luminance/response saturation affects visual salience mostly when the high spatial frequencies of the image are reduced by poor illumination or optical blur. Because both low luminance and optical blur are risk factors in myopia, our results suggest a possible neuronal mechanism linking myopia progression with the function of the ON visual pathway.

The divisive normalization model of V1 neurons: a comprehensive comparison of physiological data and model predictions

The physiological responses of simple and complex cells in the primary visual cortex (V1) have been studied extensively and modeled at different levels. At the functional level, the divisive normalization model (DNM; Heeger DJ. Vis Neurosci 9: 181-197, 1992) has accounted for a wide range of single-cell recordings in terms of a combination of linear filtering, nonlinear rectification, and divisive normalization. We propose standardizing the formulation of the DNM and implementing it in software that takes static grayscale images as inputs and produces firing rate responses as outputs. We also review a comprehensive suite of 30 empirical phenomena and report a series of simulation experiments that qualitatively replicate dozens of key experiments with a standard parameter set consistent with physiological measurements. This systematic approach identifies novel falsifiable predictions of the DNM. We show how the model simultaneously satisfies the conflicting desiderata of flexibility and falsifiability. Our key idea is that, while adjustable parameters are needed to accommodate the diversity across neurons, they must be fixed for a given individual neuron. This requirement introduces falsifiable constraints when this single neuron is probed with multiple stimuli. We also present mathematical analyses and simulation experiments that explicate some of these constraints.

Constrained sampling experiments reveal principles of detection in natural scenes

A fundamental everyday visual task is to detect target objects within a background scene. Using relatively simple stimuli, vision science has identified several major factors that affect detection thresholds, including the luminance of the background, the contrast of the background, the spatial similarity of the background to the target, and uncertainty due to random variations in the properties of the background and in the amplitude of the target. Here we use an experimental approach based on constrained sampling from multidimensional histograms of natural stimuli, together with a theoretical analysis based on signal detection theory, to discover how these factors affect detection in natural scenes. We sorted a large collection of natural image backgrounds into multidimensional histograms, where each bin corresponds to a particular luminance, contrast, and similarity. Detection thresholds were measured for a subset of bins spanning the space, where a natural background was randomly sampled from a bin on each trial. In low-uncertainty conditions, both the background bin and the amplitude of the target were fixed, and, in high-uncertainty conditions, they varied randomly on each trial. We found that thresholds increase approximately linearly along all three dimensions and that detection accuracy is unaffected by background bin and target amplitude uncertainty. The results are predicted from first principles by a normalized matched-template detector, where the dynamic normalizing gain factor follows directly from the statistical properties of the natural backgrounds. The results provide an explanation for classic laws of psychophysics and their underlying neural mechanisms.

Selective Modulation of the Pupil Light Reflex by Microstimulation of Prefrontal Cortex

The prefrontal cortex (PFC) is thought to flexibly regulate sensorimotor responses, perhaps through modulating activity in other circuits. However, the scope of that control remains unknown: it remains unclear whether the PFC can modulate basic reflexes. One canonical example of a central reflex is the pupil light reflex (PLR): the automatic constriction of the pupil in response to luminance increments. Unlike pupil size, which depends on the interaction of multiple physiological and neuromodulatory influences, the PLR reflects the action of a simple brainstem circuit. However, emerging behavioral evidence suggests that the PLR may be modulated by cognitive processes. Although the neural basis of these modulations remains unknown, one possible source is the PFC, particularly the frontal eye field (FEF), an area of the PFC implicated in the control of attention. We show that microstimulation of the rhesus macaque FEF alters the magnitude of the PLR in a spatially specific manner. FEF microstimulation enhanced the PLR to probes presented within the stimulated visual field, but suppressed the PLR to probes at nonoverlapping locations. The spatial specificity of this effect parallels the effect of FEF stimulation on attention and suggests that FEF is capable of modulating visuomotor transformations performed at a lower level than was previously known. These results provide evidence of the selective regulation of a basic brainstem reflex by the PFC.SIGNIFICANCE STATEMENT The pupil light reflex (PLR) is our brain's first and most fundamental mechanism for light adaptation. Although it is often described in textbooks as being an immutable reflex, converging evidence suggests that the magnitude of the PLR is modulated by cognitive factors. The neural bases of these modulations are unknown. Here, we report that microstimulation in the prefrontal cortex (PFC) modulates the gain of the PLR, changing how a simple reflex circuit responds to physically identical stimuli. These results suggest that control structures such as the PFC can add complexity and flexibility to even a basic brainstem circuit.

Objective Quality Assessment of Screen Content Images by Uncertainty Weighting

In this paper, we propose a novel full-reference objective quality assessment metric for screen content images (SCIs) by structure features and uncertainty weighting (SFUW). The input SCI is first divided into textual and pictorial regions. The visual quality of textual regions is estimated based on perceptual structural similarity, where the gradient information is adopted as the structural feature. To predict the visual quality of pictorial regions in SCIs, we extract the structural features and luminance features for similarity computation between the reference and distorted pictorial patches. To obtain the final visual quality of SCI, we design an uncertainty weighting method by perceptual theories to fuse the visual quality of textual and pictorial regions effectively. Experimental results show that the proposed SFUW can obtain better performance of visual quality prediction for SCIs than other existing ones.

Retooling Laser Speckle Contrast Analysis Algorithm to Enhance Non-Invasive High Resolution Laser Speckle Functional Imaging of Cutaneous Microcirculation

Cutaneous microvasculopathy complicates wound healing. Functional assessment of gated individual dermal microvessels is therefore of outstanding interest. Functional performance of laser speckle contrast imaging (LSCI) systems is compromised by motion artefacts. To address such weakness, post-processing of stacked images is reported. We report the first post-processing of binary raw data from a high-resolution LSCI camera. Sharp images of low-flowing microvessels were enabled by introducing inverse variance in conjunction with speckle contrast in Matlab-based program code. Extended moving window averaging enhanced signal-to-noise ratio. Functional quantitative study of blood flow kinetics was performed on single gated microvessels using a free hand tool. Based on detection of flow in low-flow microvessels, a new sharp contrast image was derived. Thus, this work presents the first distinct image with quantitative microperfusion data from gated human foot microvasculature. This versatile platform is applicable to study a wide range of tissue systems including fine vascular network in murine brain without craniotomy as well as that in the murine dorsal skin. Importantly, the algorithm reported herein is hardware agnostic and is capable of post-processing binary raw data from any camera source to improve the sensitivity of functional flow data above and beyond standard limits of the optical system.

Contrast response functions in the visual wulst of the alert burrowing owl: a single-unit study

The neuronal representation of luminance contrast has not been thoroughly studied in birds. Here we present a detailed quantitative analysis of the contrast response of 120 individual neurons recorded from the visual wulst of awake burrowing owls (Athene cunicularia). Stimuli were sine-wave gratings presented within the cell classical receptive field and optimized in terms of eye preference, direction of drift, and spatiotemporal frequency. As contrast intensity was increased from zero to near 100%, most cells exhibited a monotonic response profile with a compressive, at times saturating, nonlinearity at higher contrasts. However, contrast response functions were found to have a highly variable shape across cells. With the view to capture a systematic trend in the data, we assessed the performance of four plausible models (linear, power, logarithmic, and hyperbolic ratio) using classical goodness-of-fit measures and more rigorous statistical tools for multimodel inferences based on the Akaike information criterion. From this analysis, we conclude that a high degree of model uncertainty is present in our data, meaning that no single descriptor is able on its own to capture the heterogeneous nature of single-unit contrast responses in the wulst. We further show that the generalizability of the hyperbolic ratio model established, for example, in the primary visual cortex of cats and monkeys is not tenable in the owl wulst mainly because most neurons in this area have a much wider dynamic range that starts at low contrast. The challenge for future research will be to understand the functional implications of these findings.

Neurons in primary visual cortex represent distribution of luminance

To efficiently detect a wide range of light-intensity changes, visual neurons must adapt to ambient luminance. However, how neurons in the primary visual cortex (V1) code the distribution of luminance remains unknown. We designed stimuli that represent rapid changes in luminance under different luminance distributions and investigated V1 neuron responses to these novel stimuli. We demonstrate that V1 neurons represent luminance changes by dynamically adjusting their responses when the luminance distribution changes. Many cells (35%) detected luminance changes by responding to dark stimuli when the distribution was dominated by bright stimuli, bright stimuli when dominated by dark stimuli, and both dark and bright stimuli when dominated by intermediate luminance stimuli; 13% of cells signaled the mean luminance that was varied with different distributions; the remaining 52% of cells gradually shifted the responses that were most sensitive to luminance changes when the luminance distribution varied. The remarkable response changes of the former two cell groups suggest their crucial roles in detecting luminance changes. These response characteristics demonstrate that V1 neurons are not only sensitive to luminance change, but also luminance distribution change. They encode luminance changes according to the luminance distribution. Mean cells represent the prevailing luminance and reversal cells represent the salient stimuli in the environment.

Orientation selectivity in rat primary visual cortex emerges earlier with low-contrast and high-luminance stimuli

In natural vision, rapid and sustained variations in luminance and contrast change the reliability of information available about a visual scene, and markedly affect both neuronal and behavioural responses. The hallmark property of neurons in primary visual cortex (V1), orientation selectivity, is unaffected by changes in stimulus contrast, but it remains unclear how sustained differences in mean luminance and contrast affect the time-course of orientation selectivity, and the amount of information that neurons carry about orientation. We used reverse correlation with characterize the temporal dynamics of orientation selectivity in rat V1 neurons under four luminance-contrast conditions. We show that orientation selectivity and mutual information between neuronal responses and stimulus orientation are invariant to contrast or mean luminance. Critically, the time-course of the emergence of orientation selectivity was affected by both factors; response latencies were longer for low- than high-luminance gratings, and surprisingly, response latencies were also longer for high- than low-contrast gratings. Modelling suggests that luminance-modulated changes in feedforward gain, in combination with hyperpolarization caused by high contrasts can account for our physiological data. The hyperpolarization at high contrasts may increase signal-to-noise ratios, whereas a more depolarized membrane may lead to greater sensitivity to weak stimuli.

A Unified Framework for Salient Structure Detection by Contour-Guided Visual Search

We define the task of salient structure (SS) detection to unify the saliency-related tasks, such as fixation prediction, salient object detection, and detection of other structures of interest in cluttered environments. To solve such SS detection tasks, a unified framework inspired by the two-pathway-based search strategy of biological vision is proposed in this paper. First, a contour-based spatial prior (CBSP) is extracted based on the layout of edges in the given scene along a fast non-selective pathway, which provides a rough, task-irrelevant, and robust estimation of the locations where the potential SSs are present. Second, another flow of local feature extraction is executed in parallel along the selective pathway. Finally, Bayesian inference is used to auto-weight and integrate the local cues guided by CBSP and to predict the exact locations of SSs. This model is invariant to the size and features of objects. The experimental results on six large datasets (three fixation prediction datasets and three salient object datasets) demonstrate that our system achieves competitive performance for SS detection (i.e., both the tasks of fixation prediction and salient object detection) compared with the state-of-the-art methods. In addition, our system also performs well for salient object construction from saliency maps and can be easily extended for salient edge detection.

V1 neurons respond to luminance changes faster than contrast changes

Luminance and contrast are two major attributes of objects in the visual scene. Luminance and contrast information received by visual neurons are often updated simultaneously. We examined the temporal response properties of neurons in the primary visual cortex (V1) to stimuli whose luminance and contrast were simultaneously changed by 50 Hz. We found that response tuning to luminance changes precedes tuning to contrast changes in V1. For most V1 neurons, the onset time of response tuning to luminance changes was shorter than that to contrast changes. Most neurons carried luminance information in the early response stage, while all neurons carried both contrast and luminance information in the late response stage. The early luminance response suggests that cortical processing for luminance is not as slow as previously thought.

Differential encoding of spatial information among retinal on cone bipolar cells

The retina is the first stage of visual processing. It encodes elemental features of visual scenes. Distinct cone bipolar cells provide the substrate for this to occur. They encode visual information, such as color and luminance, a principle known as parallel processing. Few studies have directly examined whether different forms of spatial information are processed in parallel among cone bipolar cells. To address this issue, we examined the spatial information encoded by mouse ON cone bipolar cells, the subpopulation excited by increments in illumination. Two types of spatial processing were identified. We found that ON cone bipolar cells with axons ramifying in the central inner plexiform layer were tuned to preferentially encode small stimuli. By contrast, ON cone bipolar cells with axons ramifying in the proximal inner plexiform layer, nearest the ganglion cell layer, were tuned to encode both small and large stimuli. This dichotomy in spatial tuning is attributable to amacrine cells providing stronger inhibition to central ON cone bipolar cells compared with proximal ON cone bipolar cells. Furthermore, background illumination altered this difference in spatial tuning. It became less pronounced in bright light, as amacrine cell-driven inhibition became pervasive among all ON cone bipolar cells. These results suggest that differential amacrine cell input determined the distinct spatial encoding properties among ON cone bipolar cells. These findings enhance the known parallel processing capacity of the retina.

Sensitivity to gaze-contingent contrast increments in naturalistic movies: An exploratory report and model comparison

Sensitivity to luminance contrast is a prerequisite for all but the simplest visual systems. To examine contrast increment detection performance in a way that approximates the natural environmental input of the human visual system, we presented contrast increments gaze-contingently within naturalistic video freely viewed by observers. A band-limited contrast increment was applied to a local region of the video relative to the observer's current gaze point, and the observer made a forced-choice response to the location of the target (≈25,000 trials across five observers). We present exploratory analyses showing that performance improved as a function of the magnitude of the increment and depended on the direction of eye movements relative to the target location, the timing of eye movements relative to target presentation, and the spatiotemporal image structure at the target location. Contrast discrimination performance can be modeled by assuming that the underlying contrast response is an accelerating nonlinearity (arising from a nonlinear transducer or gain control). We implemented one such model and examined the posterior over model parameters, estimated using Markov-chain Monte Carlo methods. The parameters were poorly constrained by our data; parameters constrained using strong priors taken from previous research showed poor cross-validated prediction performance. Atheoretical logistic regression models were better constrained and provided similar prediction performance to the nonlinear transducer model. Finally, we explored the properties of an extended logistic regression that incorporates both eye movement and image content features. Models of contrast transduction may be better constrained by incorporating data from both artificial and natural contrast perception settings.

Predicting cortical dark/bright asymmetries from natural image statistics and early visual transforms

The nervous system has evolved in an environment with structure and predictability. One of the ubiquitous principles of sensory systems is the creation of circuits that capitalize on this predictability. Previous work has identified predictable non-uniformities in the distributions of basic visual features in natural images that are relevant to the encoding tasks of the visual system. Here, we report that the well-established statistical distributions of visual features -- such as visual contrast, spatial scale, and depth -- differ between bright and dark image components. Following this analysis, we go on to trace how these differences in natural images translate into different patterns of cortical input that arise from the separate bright (ON) and dark (OFF) pathways originating in the retina. We use models of these early visual pathways to transform natural images into statistical patterns of cortical input. The models include the receptive fields and non-linear response properties of the magnocellular (M) and parvocellular (P) pathways, with their ON and OFF pathway divisions. The results indicate that there are regularities in visual cortical input beyond those that have previously been appreciated from the direct analysis of natural images. In particular, several dark/bright asymmetries provide a potential account for recently discovered asymmetries in how the brain processes visual features, such as violations of classic energy-type models. On the basis of our analysis, we expect that the dark/bright dichotomy in natural images plays a key role in the generation of both cortical and perceptual asymmetries.

Sensory systems must contend with a tremendous amount of diversity in the natural world. Gaining a detailed description of the natural world’s statistical regularities is a critical part of understanding how the nervous system is adapted to its environment. Here, we report that the well-established statistical distributions of basic visual features—such as visual contrast and spatial scale—diverge when separated into bright and dark components. Operations such as dark/bright segregation are key features of early visual pathways. By modeling these pathways, we demonstrate that the dark and bright visual patterns driving cortical networks are asymmetric across a number of visual features, producing previously unappreciated second-order regularities. The results provide a parsimonious account for recently discovered asymmetries in cortical activity.

Attachment style impacts behavior and early oculomotor response to positive, but not negative, pictures

The present study investigated whether oculomotor behavior is influenced by attachment styles. The Relationship Scales Questionnaire was used to assess attachment styles of forty-eight voluntary university students and to classify them into attachment groups (secure, preoccupied, fearful, and dismissing). Eye-tracking was recorded while participants engaged in a 3-seconds free visual exploration of stimuli presenting either a positive or a negative picture together with a neutral picture, all depicting social interactions. The task consisted in identifying whether the two pictures depicted the same emotion. Results showed that the processing of negative pictures was impermeable to attachment style, while the processing of positive pictures was significantly influenced by individual differences in insecure attachment. The groups highly avoidant regarding to attachment (dismissing and fearful) showed reduced accuracy, suggesting a higher threshold for recognizing positive emotions compared to the secure group. The groups with higher attachment anxiety (preoccupied and fearful) showed differences in automatic capture of attention, in particular an increased delay preceding the first fixation to a picture of positive emotional valence. Despite lenient statistical thresholds induced by the limited sample size of some groups (p < 0.05 uncorrected for multiple comparisons), the current findings suggest that the processing of positive emotions is affected by attachment styles. These results are discussed within a broader evolutionary framework.

Critical and maximally informative encoding between neural populations in the retina

Computation in the brain involves multiple types of neurons, yet the organizing principles for how these neurons work together remain unclear. Information theory has offered explanations for how different types of neurons can maximize the transmitted information by encoding different stimulus features. However, recent experiments indicate that separate neuronal types exist that encode the same filtered version of the stimulus, but then the different cell types signal the presence of that stimulus feature with different thresholds. Here we show that the emergence of these neuronal types can be quantitatively described by the theory of transitions between different phases of matter. The two key parameters that control the separation of neurons into subclasses are the mean and standard deviation (SD) of noise affecting neural responses. The average noise across the neural population plays the role of temperature in the classic theory of phase transitions, whereas the SD is equivalent to pressure or magnetic field, in the case of liquid-gas and magnetic transitions, respectively. Our results account for properties of two recently discovered types of salamander Off retinal ganglion cells, as well as the absence of multiple types of On cells. We further show that, across visual stimulus contrasts, retinal circuits continued to operate near the critical point whose quantitative characteristics matched those expected near a liquid-gas critical point and described by the nearest-neighbor Ising model in three dimensions. By operating near a critical point, neural circuits can maximize information transmission in a given environment while retaining the ability to quickly adapt to a new environment.

Input-dependent frequency modulation of cortical gamma oscillations shapes spatial synchronization and enables phase coding

Fine-scale temporal organization of cortical activity in the gamma range (∼25-80Hz) may play a significant role in information processing, for example by neural grouping ('binding') and phase coding. Recent experimental studies have shown that the precise frequency of gamma oscillations varies with input drive (e.g. visual contrast) and that it can differ among nearby cortical locations. This has challenged theories assuming widespread gamma synchronization at a fixed common frequency. In the present study, we investigated which principles govern gamma synchronization in the presence of input-dependent frequency modulations and whether they are detrimental for meaningful input-dependent gamma-mediated temporal organization. To this aim, we constructed a biophysically realistic excitatory-inhibitory network able to express different oscillation frequencies at nearby spatial locations. Similarly to cortical networks, the model was topographically organized with spatially local connectivity and spatially-varying input drive. We analyzed gamma synchronization with respect to phase-locking, phase-relations and frequency differences, and quantified the stimulus-related information represented by gamma phase and frequency. By stepwise simplification of our models, we found that the gamma-mediated temporal organization could be reduced to basic synchronization principles of weakly coupled oscillators, where input drive determines the intrinsic (natural) frequency of oscillators. The gamma phase-locking, the precise phase relation and the emergent (measurable) frequencies were determined by two principal factors: the detuning (intrinsic frequency difference, i.e. local input difference) and the coupling strength. In addition to frequency coding, gamma phase contained complementary stimulus information. Crucially, the phase code reflected input differences, but not the absolute input level. This property of relative input-to-phase conversion, contrasting with latency codes or slower oscillation phase codes, may resolve conflicting experimental observations on gamma phase coding. Our modeling results offer clear testable experimental predictions. We conclude that input-dependency of gamma frequencies could be essential rather than detrimental for meaningful gamma-mediated temporal organization of cortical activity.

Multifeature-based surround inhibition improves contour detection in natural images

To effectively perform visual tasks like detecting contours, the visual system normally needs to integrate multiple visual features. Sufficient physiological studies have revealed that for a large number of neurons in the primary visual cortex (V1) of monkeys and cats, neuronal responses elicited by the stimuli placed within the classical receptive field (CRF) are substantially modulated, normally inhibited, when difference exists between the CRF and its surround, namely, non-CRF, for various local features. The exquisite sensitivity of V1 neurons to the center-surround stimulus configuration is thought to serve important perceptual functions, including contour detection. In this paper, we propose a biologically motivated model to improve the performance of perceptually salient contour detection. The main contribution is the multifeature-based center-surround framework, in which the surround inhibition weights of individual features, including orientation, luminance, and luminance contrast, are combined according to a scale-guided strategy, and the combined weights are then used to modulate the final surround inhibition of the neurons. The performance was compared with that of single-cue-based models and other existing methods (especially other biologically motivated ones). The results show that combining multiple cues can substantially improve the performance of contour detection compared with the models using single cue. In general, luminance and luminance contrast contribute much more than orientation to the specific task of contour extraction, at least in gray-scale natural images.

Standard deviation of luminance distribution affects lightness and pupillary response

We examined whether the standard deviation (SD) of luminance distribution serves as information of illumination. We measured the lightness of a patch presented in the center of a scrambled-dot pattern while manipulating the SD of the luminance distribution. Results showed that lightness decreased as the SD of the surround stimulus increased. We also measured pupil diameter while viewing a similar stimulus. The pupil diameter decreased as the SD of luminance distribution of the stimuli increased. We confirmed that these results were not obtained because of the increase of the highest luminance in the stimulus. Furthermore, results of field measurements revealed a correlation between the SD of luminance distribution and illuminance in natural scenes. These results indicated that the visual system refers to the SD of the luminance distribution in the visual stimulus to estimate the scene illumination.

Temporal statistics of natural image sequences generated by movements with insect flight characteristics

Many flying insects, such as flies, wasps and bees, pursue a saccadic flight and gaze strategy. This behavioral strategy is thought to separate the translational and rotational components of self-motion and, thereby, to reduce the computational efforts to extract information about the environment from the retinal image flow. Because of the distinguishing dynamic features of this active flight and gaze strategy of insects, the present study analyzes systematically the spatiotemporal statistics of image sequences generated during saccades and intersaccadic intervals in cluttered natural environments. We show that, in general, rotational movements with saccade-like dynamics elicit fluctuations and overall changes in brightness, contrast and spatial frequency of up to two orders of magnitude larger than translational movements at velocities that are characteristic of insects. Distinct changes in image parameters during translations are only caused by nearby objects. Image analysis based on larger patches in the visual field reveals smaller fluctuations in brightness and spatial frequency composition compared to small patches. The temporal structure and extent of these changes in image parameters define the temporal constraints imposed on signal processing performed by the insect visual system under behavioral conditions in natural environments.

Orientation-cue invariant population responses to contrast-modulated and phase-reversed contour stimuli in macaque V1 and V2

Visual scenes can be readily decomposed into a variety of oriented components, the processing of which is vital for object segregation and recognition. In primate V1 and V2, most neurons have small spatio-temporal receptive fields responding selectively to oriented luminance contours (first order), while only a subgroup of neurons signal non-luminance defined contours (second order). So how is the orientation of second-order contours represented at the population level in macaque V1 and V2? Here we compared the population responses in macaque V1 and V2 to two types of second-order contour stimuli generated either by modulation of contrast or phase reversal with those to first-order contour stimuli. Using intrinsic signal optical imaging, we found that the orientation of second-order contour stimuli was represented invariantly in the orientation columns of both macaque V1 and V2. A physiologically constrained spatio-temporal energy model of V1 and V2 neuronal populations could reproduce all the recorded population responses. These findings suggest that, at the population level, the primate early visual system processes the orientation of second-order contours initially through a linear spatio-temporal filter mechanism. Our results of population responses to different second-order contour stimuli support the idea that the orientation maps in primate V1 and V2 can be described as a spatial-temporal energy map.

Infant Preference for Natural Texture Statistics is Modulated by Contrast Polarity

Adult observers are sensitive to statistical regularities present in natural images. Developmentally, research has shown that children do not show sensitivity to these natural regularities until approximately 8-10 years of age. This finding is surprising given that even infants gradually encode a range of high-level statistical regularities of their visual environment in the first year of life, We suggest that infants may in fact exhibit sensitivity to natural image statistics under circumstances where images of complex, natural textures, such as a photograph of rocks, are used as experimental stimuli and natural appearance is substantially manipulated. We tested this hypothesis by examining how infants' visual preference for real versus computer-generated synthetic textures was modulated by contrast negation, which produces an image similar to a photographic negative. We observed that older infants' (9-months of age) preferential looking behavior in this task was affected by contrast polarity, suggesting that the infant visual system is sensitive to deviations from natural texture appearance, including (1) discrepancies in appearance that differentiate natural and synthetic textures from one another and (2) the disruption of contrast polarity following negation. We discuss our results in the context of adult texture processing and the "perceptual narrowing" of visual recognition during the first year of life.

Measuring streetscape complexity based on the statistics of local contrast and spatial frequency

Streetscapes are basic urban elements which play a major role in the livability of a city. The visual complexity of streetscapes is known to influence how people behave in such built spaces. However, how and which characteristics of a visual scene influence our perception of complexity have yet to be fully understood. This study proposes a method to evaluate the complexity perceived in streetscapes based on the statistics of local contrast and spatial frequency. Here, 74 streetscape images from four cities, including daytime and nighttime scenes, were ranked for complexity by 40 participants. Image processing was then used to locally segment contrast and spatial frequency in the streetscapes. The statistics of these characteristics were extracted and later combined to form a single objective measure. The direct use of statistics revealed structural or morphological patterns in streetscapes related to the perception of complexity. Furthermore, in comparison to conventional measures of visual complexity, the proposed objective measure exhibits a higher correlation with the opinion of the participants. Also, the performance of this method is more robust regarding different time scenarios.

The separation of monocular and binocular contrast

The contrast asynchrony is a stimulus configuration that illustrates the visual system's separable responses to luminance and luminance contrast information (Shapiro, 2008; Shapiro et al., 2004). When two disks, whose luminances modulate in phase with each other, are each surrounded by a disk, one light and one dark, observers can see both the in-phase brightness signals and the antiphase contrast signals and can separate the two. Here we present the results of experiments in which observers viewed a similar stimulus dichoptically. We report that no asynchrony is perceived when one eye is presented with modulating disks and the other eye is presented with the black and white surround rings, nor is an asynchrony perceived in gradient versions of the contrast asynchrony. We also explore the "window shade illusion" (Shapiro, Charles, & Shear-Heyman, 2005) dichoptically and find that when a modulating disk is presented to one eye and a horizontally split black/white annulus is presented to the other, observers perceive a "shading" motion up and down the disk. This shading can be seen in either direction in the binocular condition, but it is almost always seen as moving towards low contrast in the monocular condition. These findings indicate the presence of separable retinal and cortical networks for contrast processing at different temporal and spatial scales.

Analysis of microsaccades and pupil dilation reveals a common decisional origin during visual search

During free viewing visual search, observers often refixate the same locations several times before and after target detection is reported with a button press. We analyzed the rate of microsaccades in the sequence of refixations made during visual search and found two important components. One related to the visual content of the region being fixated; fixations on targets generate more microsaccades and more microsaccades are generated for those targets that are more difficult to disambiguate. The other empathizes non-visual decisional processes; fixations containing the button press generate more microsaccades than those made on the same target but without the button press. Pupil dilation during the same refixations reveals a similar modulation. We inferred that generic sympathetic arousal mechanisms are part of the articulated complex of perceptual processes governing fixational eye movements.

Modeling lateral geniculate nucleus response with contrast gain control. Part 1: formulation

A class of models for lateral geniculate nucleus (LGN) on-cell behavior is proposed. The models consist of a linear filter with divisive normalization by root mean square local contrast and include an intrinsic noise density parameter. The properties of these models are shown to match observed LGN behavior: (1) a linear response to low-magnitude stimuli; (2) a linear response without saturation (luxotonic behavior) for zero-contrast stimuli (homogeneous fields) with increasing magnitude; and (3) response saturation for nonzero contrast stimuli with increasing magnitude. The models possess an intrinsic scale for signal-to-noise ratio (SNR). The models show under and supersaturation, as well as saturation, for sinusoidal grating stimuli with increasing contrast and predict that different SNR regimes will cause a single neuron to show different contrast response curves. A companion paper [1] provides a detailed analysis of the full nonlinear response for sinusoidal grating stimuli and circular spot stimuli.

Spatial segregation of adaptation and predictive sensitization in retinal ganglion cells

Sensory systems change their sensitivity based on recent stimuli to adjust their response range to the range of inputs and to predict future sensory input. Here, we report the presence of retinal ganglion cells that have antagonistic plasticity, showing central adaptation and peripheral sensitization. Ganglion cell responses were captured by a spatiotemporal model with independently adapting excitatory and inhibitory subunits, and sensitization requires GABAergic inhibition. Using a simple theory of signal detection, we show that the sensitizing surround conforms to an optimal inference model that continually updates the prior signal probability. This indicates that small receptive field regions have dual functionality--to adapt to the local range of signals but sensitize based upon the probability of the presence of that signal. Within this framework, we show that sensitization predicts the location of a nearby object, revealing prediction as a functional role for adapting inhibition in the nervous system.

Controlling gain one photon at a time

Adaptation is a salient property of sensory processing. All adaptational or gain control mechanisms face the challenge of obtaining a reliable estimate of the property of the input to be adapted to and obtaining this estimate sufficiently rapidly to be useful. Here, we explore how the primate retina balances the need to change gain rapidly and reliably when photons arrive rarely at individual rod photoreceptors. We find that the weakest backgrounds that decrease the gain of the retinal output signals are similar to those that increase human behavioral threshold, and identify a novel site of gain control in the retinal circuitry. Thus, surprisingly, the gain of retinal signals begins to decrease essentially as soon as background lights are detectable; under these conditions, gain control does not rely on a highly averaged estimate of the photon count, but instead signals from individual photon absorptions trigger changes in gain.

DOI:http://dx.doi.org/10.7554/eLife.00467.001

To process the sights and sounds around us, our senses must be attuned to a huge range of signals: from barely audible whispers to deafening rock concerts, and from dim glimmers of light to bright spotlights. Sensory neurons face the challenge of encoding this huge range of inputs within their much more restricted response range. Thus, neurons in our eyes and ears must continually adjust their gain or sensitivity to match changes in the light and sound inputs. These gain control processes must operate rapidly to keep up with the ever-changing input signals, but must also operate accurately so as not to distort the inputs.

The trade-off between rapid and accurate gain control can be illustrated by considering how the retina processes information at low light levels. There are two main types of light-sensitive cells in the retina: rods and cones. Vision at night relies on the ability of the rods to detect single photons—the smallest unit of light. In starlight, an individual rod will register photons only rarely, and most of the time, the majority of the rods will not register any photons. Neurons in the retinal circuits that read out the rod signals receive input from hundreds or thousands of rods, and those rod inputs are highly amplified to allow detection of the responses produced when a tiny fraction of the rods absorbs a photon. But this amplification is dangerous, as it could easily saturate retinal signals when light levels increase. Gain control mechanisms are needed to avoid such saturation.

Schwartz and Rieke now add to our understanding of this process by examining how the retinas of non-human primates behave in low light. They reveal that levels of background light that can only just be detected behaviorally trigger retinal gain controls; these gain controls operate when less than 1% of rods absorb a photon. Under these conditions, the physics of light itself will cause considerable variability in the stream of photons arriving at the retina, leading to high variability in the gain of retinal responses. Nonetheless, changes in gain occurred rapidly following changes in background, indicating that the underlying mechanisms spend little time averaging incident photons. Taken together, these findings will require revisiting our ideas about how adaptational mechanisms balance the competing demands of speed and reliability to help us see the world around us.

DOI:http://dx.doi.org/10.7554/eLife.00467.002

Network connections that evolve to circumvent the inverse optics problem

A fundamental problem in vision science is how useful perceptions and behaviors arise in the absence of information about the physical sources of retinal stimuli (the inverse optics problem). Psychophysical studies show that human observers contend with this problem by using the frequency of occurrence of stimulus patterns in cumulative experience to generate percepts. To begin to understand the neural mechanisms underlying this strategy, we examined the connectivity of simple neural networks evolved to respond according to the cumulative rank of stimulus luminance values. Evolved similarities with the connectivity of early level visual neurons suggests that biological visual circuitry uses the same mechanisms as a means of creating useful perceptions and behaviors without information about the real world.

Population response to natural images in the primary visual cortex encodes local stimulus attributes and perceptual processing

The primary visual cortex (V1) is extensively studied with a large repertoire of stimuli, yet little is known about its encoding of natural images. Using voltage-sensitive dye imaging in behaving monkeys, we measured neural population response evoked in V1 by natural images presented during a face/scramble discrimination task. The population response showed two distinct phases of activity: an early phase that was spread over most of the imaged area, and a late phase that was spatially confined. To study the detailed relation between the stimulus and the population response, we used a simple encoding model to compute a continuous map of the expected neural response based on local attributes of the stimulus (luminance and contrast), followed by an analytical retinotopic transformation. Then, we computed the spatial correlation between the maps of the expected and observed response. We found that the early response was highly correlated with the local luminance of the stimulus and was sufficient to effectively discriminate between stimuli at the single trial level. The late response, on the other hand, showed a much lower correlation to the local luminance, was confined to central parts of the face images, and was highly correlated with the animal's perceptual report. Our study reveals a continuous spatial encoding of low- and high-level features of natural images in V1. The low level is directly linked to the stimulus basic local attributes and the high level is correlated with the perceptual outcome of the stimulus processing.

Dynamics of local input normalization result from balanced short- and long-range intracortical interactions in area V1

To efficiently drive many behaviors, sensory systems have to integrate the activity of large neuronal populations within a limited time window. These populations need to rapidly achieve a robust representation of the input image, probably through canonical computations such as divisive normalization. However, little is known about the dynamics of the corticocortical interactions implementing these rapid and robust computations. Here, we measured the real-time activity of a large neuronal population in V1 using voltage-sensitive dye imaging in behaving monkeys. We found that contrast gain of the population increases over time with a time constant of ~30 ms and propagates laterally over the cortical surface. This dynamic is well accounted for by a divisive normalization achieved through a recurrent network that transiently increases in size after response onset with a slow swelling speed of 0.007-0.014 m/s, suggesting a polysynaptic intracortical origin. In the presence of a surround, this normalization pool is gradually balanced by lateral inputs propagating from distant cortical locations. This results in a centripetal propagation of surround suppression at a speed of 0.1-0.3 m/s, congruent with horizontal intracortical axons speed. We propose that a simple generalized normalization scheme can account for both the dynamical contrast response function through recurrent polysynaptic intracortical loops and for the surround suppression through long-range monosynaptic horizontal spread. Our results demonstrate that V1 achieves a rapid and robust context-dependent input normalization through a timely push-pull between local and lateral networks. We suggest that divisive normalization, a fundamental canonical computation, should be considered as a dynamic process.

Temporal and spatial adaptation of transient responses to local features

Interpreting visual motion within the natural environment is a challenging task, particularly considering that natural scenes vary enormously in brightness, contrast and spatial structure. The performance of current models for the detection of self-generated optic flow depends critically on these very parameters, but despite this, animals manage to successfully navigate within a broad range of scenes. Within global scenes local areas with more salient features are common. Recent work has highlighted the influence that local, salient features have on the encoding of optic flow, but it has been difficult to quantify how local transient responses affect responses to subsequent features and thus contribute to the global neural response. To investigate this in more detail we used experimenter-designed stimuli and recorded intracellularly from motion-sensitive neurons. We limited the stimulus to a small vertically elongated strip, to investigate local and global neural responses to pairs of local “doublet” features that were designed to interact with each other in the temporal and spatial domain. We show that the passage of a high-contrast doublet feature produces a complex transient response from local motion detectors consistent with predictions of a simple computational model. In the neuron, the passage of a high-contrast feature induces a local reduction in responses to subsequent low-contrast features. However, this neural contrast gain reduction appears to be recruited only when features stretch vertically (i.e., orthogonal to the direction of motion) across at least several aligned neighboring ommatidia. Horizontal displacement of the components of elongated features abolishes the local adaptation effect. It is thus likely that features in natural scenes with vertically aligned edges, such as tree trunks, recruit the greatest amount of response suppression. This property could emphasize the local responses to such features vs. those in nearby texture within the scene.

Spatially pooled contrast responses predict neural and perceptual similarity of naturalistic image categories

The visual world is complex and continuously changing. Yet, our brain transforms patterns of light falling on our retina into a coherent percept within a few hundred milliseconds. Possibly, low-level neural responses already carry substantial information to facilitate rapid characterization of the visual input. Here, we computationally estimated low-level contrast responses to computer-generated naturalistic images, and tested whether spatial pooling of these responses could predict image similarity at the neural and behavioral level. Using EEG, we show that statistics derived from pooled responses explain a large amount of variance between single-image evoked potentials (ERPs) in individual subjects. Dissimilarity analysis on multi-electrode ERPs demonstrated that large differences between images in pooled response statistics are predictive of more dissimilar patterns of evoked activity, whereas images with little difference in statistics give rise to highly similar evoked activity patterns. In a separate behavioral experiment, images with large differences in statistics were judged as different categories, whereas images with little differences were confused. These findings suggest that statistics derived from low-level contrast responses can be extracted in early visual processing and can be relevant for rapid judgment of visual similarity. We compared our results with two other, well- known contrast statistics: Fourier power spectra and higher-order properties of contrast distributions (skewness and kurtosis). Interestingly, whereas these statistics allow for accurate image categorization, they do not predict ERP response patterns or behavioral categorization confusions. These converging computational, neural and behavioral results suggest that statistics of pooled contrast responses contain information that corresponds with perceived visual similarity in a rapid, low-level categorization task.

Humans excel in rapid and accurate processing of visual scenes. However, it is unclear which computations allow the visual system to convert light hitting the retina into a coherent representation of visual input in a rapid and efficient way. Here we used simple, computer-generated image categories with similar low-level structure as natural scenes to test whether a model of early integration of low-level information can predict perceived category similarity. Specifically, we show that summarized (spatially pooled) responses of model neurons covering the entire visual field (the population response) to low-level properties of visual input (contrasts) can already be informative about differences in early visual evoked activity as well as behavioral confusions of these categories. These results suggest that low-level population responses can carry relevant information to estimate similarity of controlled images, and put forward the exciting hypothesis that the visual system may exploit these responses to rapidly process real natural scenes. We propose that the spatial pooling that allows for the extraction of this information may be a plausible first step in extracting scene gist to form a rapid impression of the visual input.

Natural versus synthetic stimuli for estimating receptive field models: a comparison of predictive robustness

An ultimate goal of visual neuroscience is to understand the neural encoding of complex, everyday scenes. Yet most of our knowledge of neuronal receptive fields has come from studies using simple artificial stimuli (e.g., bars, gratings) that may fail to reveal the full nature of a neuron's actual response properties. Our goal was to compare the utility of artificial and natural stimuli for estimating receptive field (RF) models. Using extracellular recordings from simple type cells in cat A18, we acquired responses to three types of broadband stimulus ensembles: two widely used artificial patterns (white noise and short bars), and natural images. We used a primary dataset to estimate the spatiotemporal receptive field (STRF) with two hold-back datasets for regularization and validation. STRFs were estimated using an iterative regression algorithm with regularization and subsequently fit with a zero-memory nonlinearity. Each RF model (STRF and zero-memory nonlinearity) was then used in simulations to predict responses to the same stimulus type used to estimate it, as well as to other broadband stimuli and sinewave gratings. White noise stimuli often elicited poor responses leading to noisy RF estimates, while short bars and natural image stimuli were more successful in driving A18 neurons and producing clear RF estimates with strong predictive ability. Natural image-derived RF models were the most robust at predicting responses to other broadband stimulus ensembles that were not used in their estimation and also provided good predictions of tuning curves for sinewave gratings.

Rapid contrast gain reduction following motion adaptation

Neural and sensory systems adapt to prolonged stimulation to allow signaling across broader input ranges than otherwise possible with the limited bandwidth of single neurons and receptors. In the visual system, adaptation takes place at every stage of processing, from the photoreceptors that adapt to prevailing luminance conditions, to higher-order motion-sensitive neurons that adapt to prolonged exposure to motion. Recent experiments using dynamic, fluctuating visual stimuli indicate that adaptation operates on a time scale similar to that of the response itself. Further work from our own laboratory has highlighted the role for rapid motion adaptation in reliable encoding of natural image motion. Physiologically, motion adaptation can be broken down into four separate components. It is not clear from the previous studies which of these motion adaptation components are involved in the fast and dynamic response changes. To investigate the adapted response in more detail, we therefore analyzed the effect of motion adaptation using a test-adapt-test protocol with adapting durations ranging from 20 ms to 20 s. Our results underscore the very rapid rate of motion adaptation, suggesting that under free flight, visual motion-sensitive neurons continuously adapt to the changing scenery. This might help explain recent observations of strong invariance in the response to natural scenes with highly variable contrast and image structure.