Processing of visual statistics of naturalistic videos in macaque visual areas V1 and V4

Mesoscale functional organization and connectivity of color, disparity, and naturalistic texture in human second visual area

Although parallel processing has been extensively studied in the low-level geniculostriate pathway and the high-level dorsal and ventral visual streams, less is known at the intermediate-level visual areas. In this study, we employed high-resolution fMRI at 7T to investigate the columnar and laminar organizations for color, disparity, and naturalistic texture in the human secondary visual cortex (V2), and their informational connectivity with lower- and higher-order visual areas. Although fMRI activations in V2 showed reproducible interdigitated color-selective thin and disparity-selective thick 'stripe' columns, we found no clear evidence of columnar organization for naturalistic textures. Cortical depth-dependent analyses revealed the strongest color-selectivity in the superficial layers of V2, along with both feedforward and feedback informational connectivity with V1 and V4. Disparity selectivity was similar across different cortical depths of V2, which showed significant feedforward and feedback connectivity with V1 and V3ab. Interestingly, the selectivity for naturalistic texture was strongest in the deep layers of V2, with significant feedback connectivity from V4. Thus, while local circuitry within cortical columns is crucial for processing color and disparity information, feedback signals from V4 are involved in generating the selectivity for naturalistic textures in area V2.

Orthogonal neural representations support perceptual judgments of natural stimuli

In natural visually guided behavior, observers must separate relevant information from a barrage of irrelevant information. Many studies have investigated the neural underpinnings of this ability using artificial stimuli presented on blank backgrounds. Natural images, however, contain task-irrelevant background elements that might interfere with the perception of object features. Recent studies suggest that visual feature estimation can be modeled through the linear decoding of task-relevant information from visual cortex. So, if the representations of task-relevant and irrelevant features are not orthogonal in the neural population, then variation in the task-irrelevant features would impair task performance. We tested this hypothesis using human psychophysics and monkey neurophysiology combined with parametrically variable naturalistic stimuli. We demonstrate that (1) the neural representation of one feature (the position of an object) in visual area V4 is orthogonal to those of several background features, (2) the ability of human observers to precisely judge object position was largely unaffected by those background features, and (3) many features of the object and the background (and of objects from a separate stimulus set) are orthogonally represented in V4 neural population responses. Our observations are consistent with the hypothesis that orthogonal neural representations can support stable perception of object features despite the richness of natural visual scenes.

Unsupervised learning as a computational principle works in visual learning of natural scenes, but not of artificial stimuli

The question of whether we learn exposed visual features remains a subject of controversy. A prevalent computational model suggests that visual features frequently exposed to observers in natural environments are likely to be learned. However, this unsupervised learning model appears to be contradicted by the significant body of experimental results with human participants that indicates visual perceptual learning (VPL) of visible task-irrelevant features does not occur with frequent exposure. Here, we demonstrate a resolution to this controversy with a new finding: Exposure to a dominant global orientation as task-irrelevant leads to VPL of the orientation, particularly when the orientation is derived from natural scene images, whereas VPL did not occur with artificial images even with matched distributions of local orientations and spatial frequencies to natural scene images. Further investigation revealed that this disparity arises from the presence of higher-order statistics derived from natural scene images-global structures such as correlations between different local orientation and spatial frequency channels. Moreover, behavioral and neuroimaging results indicate that the dominant orientation from these higher-order statistics undergoes less attentional suppression than that from artificial images, which may facilitate VPL. Our results contribute to resolving the controversy by affirming the validity of unsupervised learning models for natural scenes but not for artificial stimuli. They challenge the assumption that VPL occurring in everyday life can be predicted by laws governing VPL for conventionally used artificial stimuli.

Exploring the role of texture features in deep convolutional neural networks: Insights from Portilla-Simoncelli statistics

It is well-understood that the performance of Deep Convolutional Neural Networks (DCNNs) in image recognition tasks is influenced not only by shape but also by texture information. Despite this, understanding the internal representations of DCNNs remains a challenging task. This study employs a simplified version of the Portilla-Simoncelli Statistics, termed "minPS," to explore how texture information is represented in a pre-trained VGG network. Using minPS features extracted from texture images, we perform a sparse regression on the activations across various channels in VGG layers. Our findings reveal that channels in the early to middle layers of the VGG network can be effectively described by minPS features. Additionally, we observe that the explanatory power of minPS sub-groups evolves as one ascends the network hierarchy. Specifically, sub-groups termed Linear Cross Scale (LCS) and Energy Cross Scale (ECS) exhibit weak explanatory power for VGG channels. To investigate the relationship further, we compare the original texture images with their synthesized counterparts, generated using VGG, in terms of minPS features. Our results indicate that the absence of certain minPS features suggests their non-utilization in VGG's internal representations.

A Texture Statistics Encoding Model Reveals Hierarchical Feature Selectivity across Human Visual Cortex

Midlevel features, such as contour and texture, provide a computational link between low- and high-level visual representations. Although the nature of midlevel representations in the brain is not fully understood, past work has suggested a texture statistics model, called the P-S model (Portilla and Simoncelli, 2000), is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex to natural scene images. To examine this, we constructed single-voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (both sexes) from the Natural Scenes Dataset (Allen et al., 2022). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas and higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex suggests that the representation of texture statistics features is widespread throughout the brain. Furthermore, using variance partitioning analyses, we identify which features are most uniquely predictive of brain responses and show that the contributions of higher-order texture features increase from early areas to higher areas on the ventral and lateral surfaces. We also demonstrate that patterns of sensitivity to texture statistics can be used to recover broad organizational axes within visual cortex, including dimensions that capture semantic image content. These results provide a key step forward in characterizing how midlevel feature representations emerge hierarchically across the visual system.SIGNIFICANCE STATEMENT Intermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.