Visual processing of informative multipoint correlations arises primarily in V2

Broadband visual stimuli improve neuronal representation and sensory perception

Natural scenes consist of complex feature distributions that shape neural responses and perception. However, in contrast to single features like stimulus orientations, the impact of broadband feature distributions remains unclear. We, therefore, presented visual stimuli with parametrically-controlled bandwidths of orientations and spatial frequencies to awake mice while recording neural activity in their primary visual cortex (V1). Increasing orientation but not spatial frequency bandwidth strongly increased the number and response amplitude of V1 neurons. This effect was not explained by single-cell orientation tuning but rather a broadband-specific relief from center-surround suppression. Moreover, neurons in deeper V1 and the superior colliculus responded much stronger to broadband stimuli, especially when mixing orientations and spatial frequencies. Lastly, broadband stimuli increased the separability of neural responses and improved the performance of mice in a visual discrimination task. Our results show that surround modulation increases neural responses to complex natural feature distributions to enhance sensory perception.

Macrophage regulation in vascularization upon regeneration and repair of tissue injury and engineered organ transplantation

Tissue engineering and regenerative medicine are effective strategies for the treatment of damaged tissues and end-stage organ failure. Damaged tissue regeneration and organ transplantation require blood vessel reconstruction to facilitate tissue remodeling, the bottleneck for application research in this field. Immune cells are heavily involved in coordinating neovascularization, in which macrophage aggregation is a key factor in angiogenesis and arteriogenesis. Previous studies have promoted tissue vascularization by regulating macrophages; however, the mechanisms underlying macrophage-mediated vascularization remain nebulous. Studies on material-based regulation have primarily been observational and lack systematic and targeted research. Macrophages from different sources exhibit different phenotypes or functions in tissues, such as peripheral blood monocytes and tissue-resident macrophages, with each exhibiting complicated mechanisms for promoting tissue injury and graft remodeling. Therefore, in this review, we discuss the role of different tissue-resident macrophages and circulating monocytes in vascularization during injured tissue regeneration and graft remodeling and summarize the current strategies for the use of biomaterials to regulate macrophages and promote the vascularization of injured tissues and during organ transplantation. A better understanding of these mechanisms will facilitate future tissue engineering research that promotes vascularization by regulating macrophage reactions.

Developmentally stable representations of naturalistic image structure in macaque visual cortex

To determine whether post-natal improvements in form vision result from changes in mid-level visual cortex, we studied neuronal and behavioral responses to texture stimuli that were matched in local spectral content but varied in "naturalistic" structure. We made longitudinal measurements of visual behavior from 16 to 95 weeks of age, and of neural responses from 20 to 56 weeks. We also measured behavioral and neural responses in near-adult animals more than 3 years old. Behavioral sensitivity reached half-maximum around 25 weeks of age, but neural sensitivities remained stable through all ages tested. Neural sensitivity to naturalistic structure was highest in V4, lower in V2 and inferotemporal cortex (IT), and barely discernible in V1. Our results show a dissociation between stable neural performance and improving behavioral performance, which may reflect improved processing capacity in circuits downstream of visual cortex.

Adaptation of the inferior temporal neurons and efficient visual processing

Numerous studies examining the responses of individual neurons in the inferior temporal (IT) cortex have revealed their characteristics such as two-dimensional or three-dimensional shape tuning, objects, or category selectivity. While these basic selectivities have been studied assuming that their response to stimuli is relatively stable, physiological experiments have revealed that the responsiveness of IT neurons also depends on visual experience. The activity changes of IT neurons occur over various time ranges; among these, repetition suppression (RS), in particular, is robustly observed in IT neurons without any behavioral or task constraints. I observed a similar phenomenon in the ventral visual neurons in macaque monkeys while they engaged in free viewing and actively fixated on one consistent object multiple times. This observation indicates that the phenomenon also occurs in natural situations during which the subject actively views stimuli without forced fixation, suggesting that this phenomenon is an everyday occurrence and widespread across regions of the visual system, making it a default process for visual neurons. Such short-term activity modulation may be a key to understanding the visual system; however, the circuit mechanism and the biological significance of RS remain unclear. Thus, in this review, I summarize the observed modulation types in IT neurons and the known properties of RS. Subsequently, I discuss adaptation in vision, including concepts such as efficient and predictive coding, as well as the relationship between adaptation and psychophysical aftereffects. Finally, I discuss some conceptual implications of this phenomenon as well as the circuit mechanisms and the models that may explain adaptation as a fundamental aspect of visual processing.

Unsupervised learning of mid-level visual representations

Recently, a confluence between trends in neuroscience and machine learning has brought a renewed focus on unsupervised learning, where sensory processing systems learn to exploit the statistical structure of their inputs in the absence of explicit training targets or rewards. Sophisticated experimental approaches have enabled the investigation of the influence of sensory experience on neural self-organization and its synaptic bases. Meanwhile, novel algorithms for unsupervised and self-supervised learning have become increasingly popular both as inspiration for theories of the brain, particularly for the function of intermediate visual cortical areas, and as building blocks of real-world learning machines. Here we review some of these recent developments, placing them in historical context and highlighting some research lines that promise exciting breakthroughs in the near future.

Local texture statistics augment the power spectrum in modeling radiographic judgments of breast density

Purpose

Anatomical "noise" is an important limitation of full-field digital mammography. Understanding its impact on clinical judgments is made difficult by the complexity of breast parenchyma, which results in image texture not fully captured by the power spectrum. While the number of possible parameters for characterizing anatomical noise is quite large, a specific set of local texture statistics has been shown to be visually salient, and human sensitivity to these statistics corresponds to their informativeness in natural scenes.

Approach

We evaluate these local texture statistics in addition to standard power-spectral measures to determine whether they have additional explanatory value for radiologists' breast density judgments. We analyzed an image database consisting of 111 disease-free mammographic screening exams (4 views each) acquired at the University of Pittsburgh Medical Center. Each exam had a breast density score assigned by the examining radiologist. Power-spectral descriptors and local image statistics were extracted from images of breast parenchyma. Model-selection criteria and accuracy were used to assess the explanatory and predictive value of local image statistics for breast density judgments.

Results

The model selection criteria show that adding local texture statistics to descriptors of the power spectra produce better explanatory and predictive models of radiologists' judgments of breast density. Thus, local texture statistics capture, in some form, non-Gaussian aspects of texture that radiologists are using.

Conclusions

Since these local texture statistics are expected to be impacted by imaging factors like modality, dose, and image processing, they suggest avenues for understanding and optimizing observer performance.

Discrimination of textures with spatial correlations and multiple gray levels

Analysis of visual texture is important for many key steps in early vision. We study visual sensitivity to image statistics in three families of textures that include multiple gray levels and correlations in two spatial dimensions. Sensitivities to positive and negative correlations are approximately independent of correlation sign, and signals from different kinds of correlations combine quadratically. We build a computational model, fully constrained by prior studies of sensitivity to uncorrelated textures and black-and-white textures with spatial correlations. The model accounts for many features of the new data, including sign-independence, quadratic combination, and the dependence on gray-level distribution.

Efficient processing of natural scenes in visual cortex

Neural circuits in the periphery of the visual, auditory, and olfactory systems are believed to use limited resources efficiently to represent sensory information by adapting to the statistical structure of the natural environment. This "efficient coding" principle has been used to explain many aspects of early visual circuits including the distribution of photoreceptors, the mosaic geometry and center-surround structure of retinal receptive fields, the excess OFF pathways relative to ON pathways, saccade statistics, and the structure of simple cell receptive fields in V1. We know less about the extent to which such adaptations may occur in deeper areas of cortex beyond V1. We thus review recent developments showing that the perception of visual textures, which depends on processing in V2 and beyond in mammals, is adapted in rats and humans to the multi-point statistics of luminance in natural scenes. These results suggest that central circuits in the visual brain are adapted for seeing key aspects of natural scenes. We conclude by discussing how adaptation to natural temporal statistics may aid in learning and representing visual objects, and propose two challenges for the future: (1) explaining the distribution of shape sensitivity in the ventral visual stream from the statistics of object shape in natural images, and (2) explaining cell types of the vertebrate retina in terms of feature detectors that are adapted to the spatio-temporal structures of natural stimuli. We also discuss how new methods based on machine learning may complement the normative, principles-based approach to theoretical neuroscience.

Geometry of spiking patterns in early visual cortex: a topological data analytic approach

In the brain, spiking patterns live in a high-dimensional space of neurons and time. Thus, determining the intrinsic structure of this space presents a theoretical and experimental challenge. To address this challenge, we introduce a new framework for applying topological data analysis (TDA) to spike train data and use it to determine the geometry of spiking patterns in the visual cortex. Key to our approach is a parametrized family of distances based on the timing of spikes that quantifies the dissimilarity between neuronal responses. We applied TDA to visually driven single-unit and multiple single-unit spiking activity in macaque V1 and V2. TDA across timescales reveals a common geometry for spiking patterns in V1 and V2 which, among simple models, is most similar to that of a low-dimensional space endowed with Euclidean or hyperbolic geometry with modest curvature. Remarkably, the inferred geometry depends on timescale and is clearest for the timescales that are important for encoding contrast, orientation and spatial correlations.

Functional recursion of orientation cues in figure-ground separation

Visual texture is an important cue to figure-ground organization. While processing of texture differences is a prerequisite for the use of this cue to extract figure-ground organization, these stages are distinct processes. One potential indicator of this distinction is the possibility that texture statistics play a different role in the figure vs. in the ground. To determine whether this is the case, we probed figure-ground processing with a family of local image statistics that specified textures that varied in the strength and spatial scale of structure, and the extent to which features are oriented. For image statistics that generated approximately isotropic textures, the threshold for identification of figure-ground structure was determined by the difference in correlation strength in figure vs. ground, independent of whether the correlations were present in figure, ground, or both. However, for image statistics with strong orientation content, thresholds were up to two times higher for correlations in the ground, vs. the figure. This held equally for texture-defined objects with convex or concave boundaries, indicating that these threshold differences are driven by border ownership, not boundary shape. Similar threshold differences were found for presentation times ranging from 125 to 500 ms. These findings identify a qualitative difference in how texture is used for figure-ground analysis, vs. texture discrimination. Additionally, it reveals a functional recursion: texture differences are needed to identify tentative boundaries and consequent scene organization into figure and ground, but then scene organization modifies sensitivity to texture differences according to the figure-ground assignment.

Rat sensitivity to multipoint statistics is predicted by efficient coding of natural scenes

Efficient processing of sensory data requires adapting the neuronal encoding strategy to the statistics of natural stimuli. Previously, in Hermundstad et al., 2014, we showed that local multipoint correlation patterns that are most variable in natural images are also the most perceptually salient for human observers, in a way that is compatible with the efficient coding principle. Understanding the neuronal mechanisms underlying such adaptation to image statistics will require performing invasive experiments that are impossible in humans. Therefore, it is important to understand whether a similar phenomenon can be detected in animal species that allow for powerful experimental manipulations, such as rodents. Here we selected four image statistics (from single- to four-point correlations) and trained four groups of rats to discriminate between white noise patterns and binary textures containing variable intensity levels of one of such statistics. We interpreted the resulting psychometric data with an ideal observer model, finding a sharp decrease in sensitivity from two- to four-point correlations and a further decrease from four- to three-point. This ranking fully reproduces the trend we previously observed in humans, thus extending a direct demonstration of efficient coding to a species where neuronal and developmental processes can be interrogated and causally manipulated.

Opposing effects of selectivity and invariance in peripheral vision

Sensory processing necessitates discarding some information in service of preserving and reformatting more behaviorally relevant information. Sensory neurons seem to achieve this by responding selectively to particular combinations of features in their inputs, while averaging over or ignoring irrelevant combinations. Here, we expose the perceptual implications of this tradeoff between selectivity and invariance, using stimuli and tasks that explicitly reveal their opposing effects on discrimination performance. We generate texture stimuli with statistics derived from natural photographs, and ask observers to perform two different tasks: Discrimination between images drawn from families with different statistics, and discrimination between image samples with identical statistics. For both tasks, the performance of an ideal observer improves with stimulus size. In contrast, humans become better at family discrimination but worse at sample discrimination. We demonstrate through simulations that these behaviors arise naturally in an observer model that relies on a common set of physiologically plausible local statistical measurements for both tasks.

Modulation of the Earliest Component of the Human VEP by Spatial Attention: An Investigation of Task Demands

Spatial attention modulations of initial afferent activity in area V1, indexed by the first component “C1” of the human visual evoked potential, are rarely found. It has thus been suggested that early modulation is induced only by special task conditions, but what these conditions are remains unknown. Recent failed replications—findings of no C1 modulation using a certain task that had previously produced robust modulations—present a strong basis for examining this question. We ran 3 experiments, the first to more exactly replicate the stimulus and behavioral conditions of the original task, and the second and third to manipulate 2 key factors that differed in the failed replication studies: the provision of informative performance feedback, and the degree to which the probed stimulus features matched those facilitating target perception. Although there was an overall significant C1 modulation of 11%, individually, only experiments 1 and 2 showed reliable effects, underlining that the modulations do occur but not consistently. Better feedback induced greater P1, but not C1, modulations. Target-probe feature matching had an inconsistent influence on modulation patterns, with behavioral performance differences and signal-overlap analyses suggesting interference from extrastriate modulations as a potential cause.

Redundancy between spectral and higher-order texture statistics for natural image segmentation

Visual texture, defined by local image statistics, provides important information to the human visual system for perceptual segmentation. Second-order or spectral statistics (equivalent to the Fourier power spectrum) are a well-studied segmentation cue. However, the role of higher-order statistics (HOS) in segmentation remains unclear, particularly for natural images. Recent experiments indicate that, in peripheral vision, the HOS of the widely adopted Portilla-Simoncelli texture model are a weak segmentation cue compared to spectral statistics, despite the fact that both are necessary to explain other perceptual phenomena and to support high-quality texture synthesis. Here we test whether this discrepancy reflects a property of natural image statistics. First, we observe that differences in spectral statistics across segments of natural images are redundant with differences in HOS. Second, using linear and nonlinear classifiers, we show that each set of statistics individually affords high performance in natural scenes and texture segmentation tasks, but combining spectral statistics and HOS produces relatively small improvements. Third, we find that HOS improve segmentation for a subset of images, although these images are difficult to identify. We also find that different subsets of HOS improve segmentation to a different extent, in agreement with previous physiological and perceptual work. These results show that the HOS add modestly to spectral statistics for natural image segmentation. We speculate that tuning to natural image statistics under resource constraints could explain the weak contribution of HOS to perceptual segmentation in human peripheral vision.

Learning complex texture discrimination

Higher-order spatial correlations contribute strongly to visual structure and salience, and are common in the natural environment. One method for studying this structure has been through the use of highly controlled texture patterns whose obvious structure is defined entirely by third- and higher-order correlations. Here we examine the effects that longer-term training has on discrimination of 17 such texture types. Training took place in 14 sessions over 42 days. Discrimination performance increased at different rates for different textures. The time required to complete a visit reduced by 25.4% (p=0.0004). Factor analysis was applied to data from the learning and experienced phases of the experiment. This indicated that the gain in speed was accompanied by an increase in the number of mechanisms contributing to discrimination. Learning was not affected by sleep quality but was affected by extreme tiredness (p<0.01). The improved discrimination and speed were retained for 2.5 months. Overall, the effects were consistent with perceptual learning. The observed learning is likely related to the adaptation of innate mechanisms that underlie our ability to identify nonredundant, visually salient structure in natural images. It may involve cortical V2 and appears to involve increased strength, speed, and breadth of connections within our internal representation of this complex perceptual space.

Flexible contextual modulation of naturalistic texture perception in peripheral vision

Peripheral vision comprises most of our visual field, and is essential in guiding visual behavior. Its characteristic capabilities and limitations, which distinguish it from foveal vision, have been explained by the most influential theory of peripheral vision as the product of representing the visual input using summary statistics. Despite its success, this account may provide a limited understanding of peripheral vision, because it neglects processes of perceptual grouping and segmentation. To test this hypothesis, we studied how contextual modulation, namely the modulation of the perception of a stimulus by its surrounds, interacts with segmentation in human peripheral vision. We used naturalistic textures, which are directly related to summary-statistics representations. We show that segmentation cues affect contextual modulation, and that this is not captured by our implementation of the summary-statistics model. We then characterize the effects of different texture statistics on contextual modulation, providing guidance for extending the model, as well as for probing neural mechanisms of peripheral vision.

Visual Functions of Primate Area V4

Area V4-the focus of this review-is a mid-level processing stage along the ventral visual pathway of the macaque monkey. V4 is extensively interconnected with other visual cortical areas along the ventral and dorsal visual streams, with frontal cortical areas, and with several subcortical structures. Thus, it is well poised to play a broad and integrative role in visual perception and recognition-the functional domain of the ventral pathway. Neurophysiological studies in monkeys engaged in passive fixation and behavioral tasks suggest that V4 responses are dictated by tuning in a high-dimensional stimulus space defined by form, texture, color, depth, and other attributes of visual stimuli. This high-dimensional tuning may underlie the development of object-based representations in the visual cortex that are critical for tracking, recognizing, and interacting with objects. Neurophysiological and lesion studies also suggest that V4 responses are important for guiding perceptual decisions and higher-order behavior.

Manipulating the structure of natural scenes using wavelets to study the functional architecture of perceptual hierarchies in the brain

Functional neuroimaging experiments that employ naturalistic stimuli (natural scenes, films, spoken narratives) provide insights into cognitive function "in the wild". Natural stimuli typically possess crowded, spectrally dense, dynamic, and multimodal properties within a rich multiscale structure. However, when using natural stimuli, various challenges exist for creating parametric manipulations with tight experimental control. Here, we revisit the typical spectral composition and statistical dependences of natural scenes, which distinguish them from abstract stimuli. We then demonstrate how to selectively degrade subtle statistical dependences within specific spatial scales using the wavelet transform. Such manipulations leave basic features of the stimuli, such as luminance and contrast, intact. Using functional neuroimaging of human participants viewing degraded natural images, we demonstrate that cortical responses at different levels of the visual hierarchy are differentially sensitive to subtle statistical dependences in natural images. This demonstration supports the notion that perceptual systems in the brain are optimally tuned to the complex statistical properties of the natural world. The code to undertake these stimulus manipulations, and their natural extension to dynamic natural scenes (films), is freely available.

Laminar Differences in Responses to Naturalistic Texture in Macaque V1 and V2

Most single units recorded from macaque secondary visual cortex (V2) respond with higher firing rates to synthetic texture images containing "naturalistic" higher-order statistics than to spectrally matched "noise" images lacking these statistics. In contrast, few single units in V1 show this property. We explored how the strength and dynamics of response vary across the different layers of visual cortex by recording multiunit (defined as high-frequency power in the local field potential) and gamma-band activity evoked by brief presentations of naturalistic and noise images in V1 and V2 of anesthetized macaque monkeys of both sexes. As previously reported, recordings in V2 showed consistently stronger responses to naturalistic texture than to spectrally matched noise. In contrast to single-unit recordings, V1 multiunit activity showed a preference for images with naturalistic statistics, and in gamma-band activity this preference was comparable across V1 and V2. Sensitivity to naturalistic image structure was strongest in the supragranular and infragranular layers of V1, but weak in granular layers, suggesting that it might reflect feedback from V2. Response timing was consistent with this idea. Visual responses appeared first in V1, followed by V2. Sensitivity to naturalistic texture emerged first in V2, followed by the supragranular and infragranular layers of V1, and finally in the granular layers of V1. Our results demonstrate laminar differences in the encoding of higher-order statistics of natural texture, and suggest that this sensitivity first arises in V2 and is fed back to modulate activity in V1.SIGNIFICANCE STATEMENT The circuit mechanisms responsible for visual representations of intermediate complexity are largely unknown. We used a well validated set of synthetic texture stimuli to probe the temporal and laminar profile of sensitivity to the higher-order statistical structure of natural images. We found that this sensitivity emerges first and most strongly in V2 but soon after in V1. However, sensitivity in V1 is higher in the laminae (extragranular) and recording modalities (local field potential) most likely affected by V2 connections, suggesting a feedback origin. Our results show how sensitivity to naturalistic image structure emerges across time and circuitry in the early visual cortex.

Object shape and surface properties are jointly encoded in mid-level ventral visual cortex

Recognizing a myriad visual objects rapidly is a hallmark of the primate visual system. Traditional theories of object recognition have focused on how crucial form features, for example, the orientation of edges, may be extracted in early visual cortex and utilized to recognize objects. An alternative view argues that much of early and mid-level visual processing focuses on encoding surface characteristics, for example, texture. Neurophysiological evidence from primate area V4 supports a third alternative - the joint, but independent, encoding of form and texture - that would be advantageous for segmenting objects from the background in natural scenes and for object recognition that is independent of surface texture. Future studies that leverage deep convolutional network models, especially focusing on network failures to match biology and behavior, can advance our insights into how such a joint representation of form and surface properties might emerge in visual cortex.

The features that control discrimination of an isodipole texture pair

Visual features such as edges and corners are carried by high-order statistics. Previous analysis of discrimination of isodipole textures, which isolate specific high-order statistics, demonstrates visual sensitivity to these statistics but stops short of analyzing the underlying computations. Here we use a new texture centroid paradigm to probe these computations. We focus on two canonical isodipole textures, the even and odd textures: any 2 × 2 block of even (odd) texture contains an even (odd) number of black (and white) checks. Each stimulus comprised a spatially random array of black-and-white texture-disks (background = mean gray) that varied in their fourth-order statistics. In the Even (Odd) condition, disks varied along the continuum between random coinflip texture and pure (highly structured) even (odd) target texture. The task was to mouse-click the centroid of the disk array, weighting each disk location by the target structure level of the disk-texture (ranging from 0 for coinflip to 1 for even or odd). For each of block-sizes S=2×2, 2 × 3, 2 × 4 and 3 × 3, a linear model was used to estimate the weight exerted on the subject's responses by the differently patterned blocks of size S. Only the results with 2 × 4 and 3 × 3 blocks were consistent with the data. In the Even condition, homogeneous blocks exerted the most weight; in the odd condition, block-pattern symmetry was important. These findings show that visual mechanisms sensitive to four-point correlations do not compute evenness or oddness per se, but rather are activated selectively by features whose frequency varies across isodipole textures.

Normalization and pooling in hierarchical models of natural images

Divisive normalization and subunit pooling are two canonical classes of computation that have become widely used in descriptive (what) models of visual cortical processing. Normative (why) models from natural image statistics can help constrain the form and parameters of such classes of models. We focus on recent advances in two particular directions, namely deriving richer forms of divisive normalization, and advances in learning pooling from image statistics. We discuss the incorporation of such components into hierarchical models. We consider both hierarchical unsupervised learning from image statistics, and discriminative supervised learning in deep convolutional neural networks (CNNs). We further discuss studies on the utility and extensions of the convolutional architecture, which has also been adopted by recent descriptive models. We review the recent literature and discuss the current promises and gaps of using such approaches to gain a better understanding of how cortical neurons represent and process complex visual stimuli.

Cortical Areas Interact through a Communication Subspace

Most brain functions involve interactions among multiple, distinct areas or nuclei. For instance, visual processing in primates requires the appropriate relaying of signals across many distinct cortical areas. Yet our understanding of how populations of neurons in interconnected brain areas communicate is in its infancy. Here we investigate how trial-to-trial fluctuations of population responses in primary visual cortex (V1) are related to simultaneously recorded population responses in area V2. Using dimensionality reduction methods, we find that V1-V2 interactions occur through a communication subspace: V2 fluctuations are related to a small subset of V1 population activity patterns, distinct from the largest fluctuations shared among neurons within V1. In contrast, interactions between subpopulations within V1 are less selective. We propose that the communication subspace may be a general, population-level mechanism by which activity can be selectively routed across brain areas.

Image segmentation driven by elements of form

While luminance, contrast, orientation, and terminators are well-established features that are extracted in early visual processing and support the parsing of an image into its component regions, the role of more complex features, such as closure and convexity, is less clear. A main barrier in understanding the roles of such features is that manipulating their occurrence typically entails changes in the occurrence of more elementary features as well. To address this problem, we developed a set of synthetic visual textures, constructed by replacing the binary coloring of standard maximum-entropy textures with tokens (tiles) containing curved or angled elements. The tokens were designed so that there were no discontinuities at their edges, and so that changing the correlation structure of the underlying binary texture changed the shapes that were produced. The resulting textures were then used in psychophysical studies, demonstrating that the resulting feature differences sufficed to drive segmentation. However, in contrast to previous findings for lower-level features, sensitivities to increases and decreases of feature occurrence were unequal. Moreover, the texture-segregation response depended on the kind of token (curved vs. angular, filled-in vs. outlined), and not just on the correlation structure. Analysis of this dependence indicated that simple closed contours and convex elements suffice to drive image segmentation, in the absence of changes in lower-level cues.

Adaptive coding for dynamic sensory inference

Behavior relies on the ability of sensory systems to infer properties of the environment from incoming stimuli. The accuracy of inference depends on the fidelity with which behaviorally relevant properties of stimuli are encoded in neural responses. High-fidelity encodings can be metabolically costly, but low-fidelity encodings can cause errors in inference. Here, we discuss general principles that underlie the tradeoff between encoding cost and inference error. We then derive adaptive encoding schemes that dynamically navigate this tradeoff. These optimal encodings tend to increase the fidelity of the neural representation following a change in the stimulus distribution, and reduce fidelity for stimuli that originate from a known distribution. We predict dynamical signatures of such encoding schemes and demonstrate how known phenomena, such as burst coding and firing rate adaptation, can be understood as hallmarks of optimal coding for accurate inference.

Gradual Development of Visual Texture-Selective Properties Between Macaque Areas V2 and V4

Complex shape and texture representations are known to be constructed from V1 along the ventral visual pathway through areas V2 and V4, but the underlying mechanism remains elusive. Recent study suggests that, for processing of textures, a collection of higher-order image statistics computed by combining V1-like filter responses serves as possible representations of textures both in V2 and V4. Here, to gain a clue for how these image statistics are processed in the extrastriate visual areas, we compared neuronal responses to textures in V2 and V4 of macaque monkeys. For individual neurons, we adaptively explored their preferred textures from among thousands of naturalistic textures and fitted the obtained responses using a combination of V1-like filter responses and higher-order statistics. We found that, while the selectivity for image statistics was largely comparable between V2 and V4, V4 showed slightly stronger sensitivity to the higher-order statistics than V2. Consistent with that finding, V4 responses were reduced to a greater extent than V2 responses when the monkeys were shown spectrally matched noise images that lacked higher-order statistics. We therefore suggest that there is a gradual development in representation of higher-order features along the ventral visual hierarchy.

Contextual modulation of sensitivity to naturalistic image structure in macaque V2

The stimulus selectivity of neurons in V1 is well known, as is the finding that their responses can be affected by visual input to areas outside of the classical receptive field. Less well understood are the ways selectivity is modified as signals propagate to visual areas beyond V1, such as V2. We recently proposed a role for V2 neurons in representing the higher order statistical dependencies found in images of naturally occurring visual texture. V2 neurons, but not V1 neurons, respond more vigorously to "naturalistic" images that contain these dependencies than to "noise" images that lack them. In this work, we examine the dependency of these effects on stimulus size. For most V2 neurons, the preference for naturalistic over noise stimuli was modest when presented in small patches and gradually strengthened with increasing size, suggesting that the mechanisms responsible for this enhanced sensitivity operate over regions of the visual field that are larger than the classical receptive field. Indeed, we found that surround suppression was stronger for noise than for naturalistic stimuli and that the preference for large naturalistic stimuli developed over a delayed time course consistent with lateral or feedback connections. These findings are compatible with a spatially broad facilitatory mechanism that is absent in V1 and suggest that a distinct role for the receptive field surround emerges in V2 along with sensitivity for more complex image structure. NEW & NOTEWORTHY The responses of neurons in visual cortex are often affected by visual input delivered to regions of the visual field outside of the conventionally defined receptive field, but the significance of such contextual modulations are not well understood outside of area V1. We studied the importance of regions beyond the receptive field in establishing a novel form of selectivity for the statistical dependencies contained in natural visual textures that first emerges in area V2.

Texture and art with deep neural networks

Although the study of biological vision and computer vision attempt to understand powerful visual information processing from different angles, they have a long history of informing each other. Recent advances in texture synthesis that were motivated by visual neuroscience have led to a substantial advance in image synthesis and manipulation in computer vision using convolutional neural networks (CNNs). Here, we review these recent advances and discuss how they can in turn inspire new research in visual perception and computational neuroscience.

Textures as Probes of Visual Processing

Visual textures are a class of stimuli with properties that make them well suited for addressing general questions about visual function at the levels of behavior and neural mechanism. They have structure across multiple spatial scales, they put the focus on the inferential nature of visual processing, and they help bridge the gap between stimuli that are analytically convenient and the complex, naturalistic stimuli that have the greatest biological relevance. Key questions that are well suited for analysis via visual textures include the nature and structure of perceptual spaces, modulation of early visual processing by task, and the transformation of sensory stimuli into patterns of population activity that are relevant to perception.

Cross-orientation suppression in visual area V2

Object recognition relies on a series of transformations among which only the first cortical stage is relatively well understood. Already at the second stage, the visual area V2, the complexity of the transformation precludes a clear understanding of what specifically this area computes. Previous work has found multiple types of V2 neurons, with neurons of each type selective for multi-edge features. Here we analyse responses of V2 neurons to natural stimuli and find three organizing principles. First, the relevant edges for V2 neurons can be grouped into quadrature pairs, indicating invariance to local translation. Second, the excitatory edges have nearby suppressive edges with orthogonal orientations. Third, the resulting multi-edge patterns are repeated in space to form textures or texture boundaries. The cross-orientation suppression increases the sparseness of responses to natural images based on these complex forms of feature selectivity while allowing for multiple scales of position invariance.

V2 neurons exhibit complex and diverse selectivity for visual features. Here the authors use a statistical analytical framework to model V2 responses to natural stimuli and find three organizing principles, chief among them is the cross-orientation suppression that increases response selectivity.

Two representations of a high-dimensional perceptual space

A perceptual space is a mental workspace of points in a sensory domain that supports similarity and difference judgments and enables further processing such as classification and naming. Perceptual spaces are present across sensory modalities; examples include colors, faces, auditory textures, and odors. Color is perhaps the best-studied perceptual space, but it is atypical in two respects. First, the dimensions of color space are directly linked to the three cone absorption spectra, but the dimensions of generic perceptual spaces are not as readily traceable to single-neuron properties. Second, generic perceptual spaces have more than three dimensions. This is important because representing each distinguishable point in a high-dimensional space by a separate neuron or population is unwieldy; combinatorial strategies may be needed to overcome this hurdle. To study the representation of a complex perceptual space, we focused on a well-characterized 10-dimensional domain of visual textures. Within this domain, we determine perceptual distances in a threshold task (segmentation) and a suprathreshold task (border salience comparison). In N=4 human observers, we find both quantitative and qualitative differences between these sets of measurements. Quantitatively, observers' segmentation thresholds were inconsistent with their uncertainty determined from border salience comparisons. Qualitatively, segmentation thresholds suggested that distances are determined by a coordinate representation with Euclidean geometry. Border salience comparisons, in contrast, indicated a global curvature of the space, and that distances are determined by activity patterns across broadly tuned elements. Thus, our results indicate two representations of this perceptual space, and suggest that they use differing combinatorial strategies.

SIGNIFICANCE STATEMENT

To move from sensory signals to decisions and actions, the brain carries out a sequence of transformations. An important stage in this process is the construction of a "perceptual space" - an internal workspace of sensory information that captures similarities and differences, and enables further processing, such as classification and naming. Perceptual spaces for color, faces, visual and haptic textures and shapes, sounds, and odors (among others) are known to exist. How such spaces are represented is at present unknown. Here, using visual textures as a model, we investigate this. Psychophysical measurements suggest roles for two combinatorial strategies: one based on projections onto coordinate-like axes, and one based on patterns of activity across broadly tuned elements scattered throughout the space.

Recurrent Network Dynamics; a Link between Form and Motion

To discriminate visual features such as corners and contours, the brain must be sensitive to spatial correlations between multiple points in an image. Consistent with this, macaque V2 neurons respond selectively to patterns with well-defined multipoint correlations. Here, we show that a standard feedforward model (a cascade of linear–non-linear filters) does not capture this multipoint selectivity. As an alternative, we developed an artificial neural network model with two hierarchical stages of processing and locally recurrent connectivity. This model faithfully reproduced neurons’ selectivity for multipoint correlations. By probing the model, we gained novel insights into early form processing. First, the diverse selectivity for multipoint correlations and complex response dynamics of the hidden units in the model were surprisingly similar to those observed in V1 and V2. This suggests that both transient and sustained response dynamics may be a vital part of form computations. Second, the model self-organized units with speed and direction selectivity that was correlated with selectivity for multipoint correlations. In other words, the model units that detected multipoint spatial correlations also detected space-time correlations. This leads to the novel hypothesis that higher-order spatial correlations could be computed by the rapid, sequential assessment and comparison of multiple low-order correlations within the receptive field. This computation links spatial and temporal processing and leads to the testable prediction that the analysis of complex form and motion are closely intertwined in early visual cortex.

Selectivity and tolerance for visual texture in macaque V2

As information propagates along the ventral visual hierarchy, neuronal responses become both more specific for particular image features and more tolerant of image transformations that preserve those features. Here, we present evidence that neurons in area V2 are selective for local statistics that occur in natural visual textures, and tolerant of manipulations that preserve these statistics. Texture stimuli were generated by sampling from a statistical model, with parameters chosen to match the parameters of a set of visually distinct natural texture images. Stimuli generated with the same statistics are perceptually similar to each other despite differences, arising from the sampling process, in the precise spatial location of features. We assessed the accuracy with which these textures could be classified based on the responses of V1 and V2 neurons recorded individually in anesthetized macaque monkeys. We also assessed the accuracy with which particular samples could be identified, relative to other statistically matched samples. For populations of up to 100 cells, V1 neurons supported better performance in the sample identification task, whereas V2 neurons exhibited better performance in texture classification. Relative to V1, the responses of V2 show greater selectivity and tolerance for the representation of texture statistics.

Asymmetry of Drosophila ON and OFF motion detectors enhances real-world velocity estimation

The reliable estimation of motion across varied surroundings represents a survival-critical task for sighted animals. How neural circuits have adapted to the particular demands of natural environments, however, is not well understood. We explored this question in the visual system of Drosophila melanogaster. Here, as in many mammalian retinas, motion is computed in parallel streams for brightness increments (ON) and decrements (OFF). When genetically isolated, ON and OFF pathways proved equally capable of accurately matching walking responses to realistic motion. To our surprise, detailed characterization of their functional tuning properties through in vivo calcium imaging and electrophysiology revealed stark differences in temporal tuning between ON and OFF channels. We trained an in silico motion estimation model on natural scenes and discovered that our optimized detector exhibited differences similar to those of the biological system. Thus, functional ON-OFF asymmetries in fly visual circuitry may reflect ON-OFF asymmetries in natural environments.

Nonlinear circuits for naturalistic visual motion estimation

Many animals use visual signals to estimate motion. Canonical models suppose that animals estimate motion by cross-correlating pairs of spatiotemporally separated visual signals, but recent experiments indicate that humans and flies perceive motion from higher-order correlations that signify motion in natural environments. Here we show how biologically plausible processing motifs in neural circuits could be tuned to extract this information. We emphasize how known aspects of Drosophila's visual circuitry could embody this tuning and predict fly behavior. We find that segregating motion signals into ON/OFF channels can enhance estimation accuracy by accounting for natural light/dark asymmetries. Furthermore, a diversity of inputs to motion detecting neurons can provide access to more complex higher-order correlations. Collectively, these results illustrate how non-canonical computations improve motion estimation with naturalistic inputs. This argues that the complexity of the fly's motion computations, implemented in its elaborate circuits, represents a valuable feature of its visual motion estimator.

DOI:http://dx.doi.org/10.7554/eLife.09123.001

Many animals have evolved the ability to estimate the speed and direction of visual motion. They use these estimates to judge their own motion, so that they can navigate through an environment, and to judge how other animals are moving, which allows them to avoid predators or detect prey.

In the 1950s, a physicist and a biologist used measurements of beetle behavior in response to visual stimuli to develop a model for how the brain estimates motion. The model became known as the Hassenstein-Reichardt correlator (HRC). The HRC and related models accurately predict the behavioral and neural responses of insects and mammals to many types of motion stimuli.

However, there are visual stimuli that generate motion percepts in fruit flies (and humans) that cannot be accounted for by the HRC. Are these differences between real brains and the HRC simply imperfections in visual circuits, whose neurons cannot perform idealized mathematical operations, or are these deviations intentional, somehow improving motion estimates? In other words: are the observed deviations a bug or a feature of visual circuits?

To address this question, Fitzgerald and Clark evaluated how different models of motion detection performed when presented with natural scenes. Natural scenes are fundamentally different from most stimuli used in lab, since they contain a rich set of regularities that are not present in simple stimuli. Fitzgerald and Clark compared the ability of the HRC, along with new, more general models, to estimate the speed and direction at which images moved across a screen. This revealed that many models could out-perform the HRC by taking advantage of regularities in natural scenes. Those models that were tuned to perform well with natural scenes could also predict the paradoxical motion percepts that were not predicted by the HRC. This suggests that visual circuits may have evolved to perform well with natural inputs, and the paradoxical motion percepts represent a feature of the real circuit, rather than a bug.

Models that performed well with natural inputs treated light and dark visual information differently. This different treatment of light and dark is a property of most visual systems, but not of the HRC or related models. In the future, these models of motion processing may help us understand how biological details of the fruit fly's visual circuits help it to estimate motion.

DOI:http://dx.doi.org/10.7554/eLife.09123.002