Representation of Color, Form, and their Conjunction across the Human Ventral Visual Pathway

Cerebral perfusion differences in the visual cortex and fusiform subregions across the psychosis spectrum

Background

Approximately 50% of individuals with psychosis spectrum disorders (PSD) experience visual hallucinations and deficits in visual processing. Cerebral blood flow (CBF) alterations have been identified in the occipital lobe (OL) and fusiform gyrus (FG) in PSD. However, prior studies neither report on cytoarchitectonic subregions of the OL or FG, nor their correlations with cognition. Moreover, perfusion differences across neurobiologically defined psychosis Biotypes in these regions are not investigated yet.

Methods

ExploreASL and FreeSurfer were used to extract perfusion measures from pseudo-continuous arterial spin labeling scans of visual (hOc1-hOc3v, middle temporal area (MT)) and fusiform (FG2-FG4) subregions in 122 bipolar disorder with psychosis (BP), 179 schizoaffective disorder (SAD), 203 schizophrenia (SZ), and 350 healthy controls (NC), as well as psychosis Biotypes (BT1-3). The data was adjusted for scanner effects using ComBat. Analyses were co-varied for total gray matter CBF. We used R to perform statistical comparisons across PSD and NC and across Biotypes. Partial Spearman correlation was performed between CBF and cognitive measures. Benjamini & Hochberg correction was used to correct for multiple comparisons.

Results

PSD exhibited greater perfusion in MT and FG2 compared to NC. Perfusion significantly differed across psychosis Biotypes in hOc1 but not across diagnostic groups. Higher MT and FG4 perfusion in PSD were associated with worse overall cognitive performance.

Conclusions

Visual and fusiform subregions demonstrate significant perfusion alterations which may indicate neurovascular deficits in PSD. Moreover, these perfusion alterations may contribute to cognitive impairments and visual abnormalities in psychosis.

Differential affection of the visual information sub-streams in a patient with visual agnosia

Introduction

Visual agnosia is a deficit of object recognition addressed to the damage of the ventral stream (VS). The dorsal stream (DS) is usually intact in these patients, and it can be derived from well-preserved reaching and grasping of visually presented objects. In this study, we presented a new case of a visual agnosic patient (AC) with an extensive lesion of the secondary visual cortex.

Methods

We examined the kinematics of his grasping behavior towards common day-to-day objects compared to a healthy control group. Both colored and color-masked objects were presented, and participants were instructed to grasp-then-name and name-then-grasp them.

Results

The agnosic deficit was particularly evident when no color information was available to the patient: Although AC was able to recognize most colored objects with marked delay, his recognition of color-masked objects was very poor. Furthermore, the color-masked condition determined larger impairments in kinematic performance relative to the control group.

Discussion

Results support the view that spared color processing in the VS allows for partial compensation of deficits. Color information is also processed along the DS, contributing to visuomotor transformations.

The human posterior parietal cortices orthogonalize the representation of different streams of information concurrently coded in visual working memory

The key to adaptive visual processing lies in the ability to maintain goal-directed visual representation in the face of distraction. In visual working memory (VWM), distraction may come from the coding of distractors or other concurrently retained targets. This fMRI study reveals a common representational geometry that our brain uses to combat both types of distractions in VWM. Specifically, using fMRI pattern decoding, the human posterior parietal cortex is shown to orthogonalize the representations of different streams of information concurrently coded in VWM, whether they are targets and distractors, or different targets concurrently held in VWM. The latter is also seen in the human occipitotemporal cortex. Such a representational geometry provides an elegant and simple solution to enable independent information readout, effectively combating distraction from the different streams of information, while accommodating their concurrent representations. This representational scheme differs from mechanisms that actively suppress or block the encoding of distractors to reduce interference. It is likely a general neural representational principle that supports our ability to represent information beyond VWM in other situations where multiple streams of visual information are tracked and processed simultaneously.

Orienting of attention and spatial cognition

Humans orient to their sensory world through foveation of target location or through covert shifts of attention. Orienting provides primacy to the selected location and in humans improves the precision of discrimination. Covert orienting appears to arise separately from the mechanisms involved in saccadic eye movements. Covert orienting can serve to prioritize processing the target even increasing its subjective intensity and its acuity. However, this network does not appear to be involved in the operations related to binding and segmentation. Cells exist in the early visual cortex that are activated by both color and form features without attention, however, color and form appear to remain independent even when oriented to the target that is required to be reported. An understanding of the pathways that connect attention networks to memory networks may allow us to understand more complex aspects of spatial cognition and enhance orienting and thus improve spatial cognition.

Illusory light drives pupil responses in primates

In humans, the eye pupils respond to both physical light sensed by the retina and mental representations of light produced by the brain. Notably, our pupils constrict when a visual stimulus is illusorily perceived brighter, even if retinal illumination is constant. However, it remains unclear whether such perceptual penetrability of pupil responses is an epiphenomenon unique to humans or whether it represents an adaptive mechanism shared with other animals to anticipate variations in retinal illumination between successive eye fixations. To address this issue, we measured the pupil responses of both humans and macaque monkeys exposed to three chromatic versions (cyan, magenta, and yellow) of the Asahi brightness illusion. We found that the stimuli illusorily perceived brighter or darker trigger differential pupil responses that are very similar in macaques and human participants. Additionally, we show that this phenomenon exhibits an analogous cyan bias in both primate species. Beyond evincing the macaque monkey as a relevant model to study the perceptual penetrability of pupil responses, our results suggest that this phenomenon is tuned to ecological conditions because the exposure to a "bright cyan-bluish sky" may be associated with increased risks of dazzle and retinal damages.

Visual perceptual learning of feature conjunctions leverages non-linear mixed selectivity

Visual objects are often defined by multiple features. Therefore, learning novel objects entails learning feature conjunctions. Visual cortex is organized into distinct anatomical compartments, each of which is devoted to processing a single feature. A prime example are neurons purely selective to color and orientation, respectively. However, neurons that jointly encode multiple features (mixed selectivity) also exist across the brain and play critical roles in a multitude of tasks. Here, we sought to uncover the optimal policy that our brain adapts to achieve conjunction learning using these available resources. 59 human subjects practiced orientation-color conjunction learning in four psychophysical experiments designed to nudge the visual system towards using one or the other resource. We find that conjunction learning is possible by linear mixing of pure color and orientation information, but that more and faster learning takes place when both pure and mixed selectivity representations are involved. We also find that learning with mixed selectivity confers advantages in performing an untrained "exclusive or" (XOR) task several months after learning the original conjunction task. This study sheds light on possible mechanisms underlying conjunction learning and highlights the importance of learning by mixed selectivity.

Mapping the dynamics of visual feature coding: Insights into perception and integration

The basic computations performed in the human early visual cortex are the foundation for visual perception. While we know a lot about these computations, a key missing piece is how the coding of visual features relates to our perception of the environment. To investigate visual feature coding, interactions, and their relationship to human perception, we investigated neural responses and perceptual similarity judgements to a large set of visual stimuli that varied parametrically along four feature dimensions. We measured neural responses using electroencephalography (N = 16) to 256 grating stimuli that varied in orientation, spatial frequency, contrast, and colour. We then mapped the response profiles of the neural coding of each visual feature and their interactions, and related these to independently obtained behavioural judgements of stimulus similarity. The results confirmed fundamental principles of feature coding in the visual system, such that all four features were processed simultaneously but differed in their dynamics, and there was distinctive conjunction coding for different combinations of features in the neural responses. Importantly, modelling of the behaviour revealed that every stimulus feature contributed to perceptual judgements, despite the untargeted nature of the behavioural task. Further, the relationship between neural coding and behaviour was evident from initial processing stages, signifying that the fundamental features, not just their interactions, contribute to perception. This study highlights the importance of understanding how feature coding progresses through the visual hierarchy and the relationship between different stages of processing and perception.

Combining contour and region for closed boundary extraction of a shape

This study explored human ability to extract closed boundary of a target shape in the presence of noise using spatially global operations. Specifically, we investigated the contributions of contour-based processing using line edges and region-based processing using color, as well as their interaction. Performance of the subjects was reliable when the fixation was inside the shape, and it was much less reliable when the fixation was outside. With fixation inside the shape, performance was higher when both contour and color information were present compared to when only one of them was present. We propose a biologically-inspired model to emulate human boundary extraction. The model solves the shortest (least-cost) path in the log-polar representation, a representation which is a good approximation to the mapping from the retina to the visual cortex. Boundary extraction was framed as a global optimization problem with the costs of connections calculated using four features: distance of interpolation, turning angle, color similarity and color contrast. This model was tested on some of the conditions that were used in the psychophysical experiment and its performance was similar to the performance of subjects.

Parietal-driven visual working memory representation in occipito-temporal cortex

Human fMRI studies have documented extensively that the content of visual working memory (VWM) can be reliably decoded from fMRI voxel response patterns during the delay period in both the occipito-temporal cortex (OTC), including early visual areas (EVC), and the posterior parietal cortex (PPC).1,2,3,4 Further work has revealed that VWM signal in OTC is largely sustained by feedback from associative areas such as prefrontal cortex (PFC) and PPC.4,5,6,7,8,9 It is unclear, however, if feedback during VWM simply restores sensory representations initially formed in OTC or if it can reshape the representational content of OTC during VWM delay. Taking advantage of a recent finding showing that object representational geometry differs between OTC and PPC in perception,10 here we find that, during VWM delay, the object representational geometry in OTC becomes more aligned with that of PPC during perception than with itself during perception. This finding supports the role of feedback in shaping the content of VWM in OTC, with the VWM content of OTC more determined by information retained in PPC than by the sensory information initially encoded in OTC.

The time course of category-based attentional template pre-activation depends on the category framework

When searching for a target defined by a set of objects, attention can be directed toward task-relevant objects by creating a category-based attentional template (CAT). Previous studies have found that CAT can be activated before the onset of the target. However, the time course of CAT pre-activation and whether the category framework (prototypical or semantic) can modulate it remain unclear. To explore the time course of CAT pre-activation, we employed a rapid serial probe presentation paradigm (RSPP) with event-related potentials (ERPs). To investigate the effect of the category framework on the time course of CAT pre-activation, the target category was defined as the prototypical category (Experiment 1) or the semantic category (Experiment 2). The results showed that the prototype-based CAT was pre-activated 300 ms prior to the target, whereas the semantics-based CAT was pre-activated 1500 ms before the onset of the target. The difference in the time course of pre-activation between the two CAT types indicates that the category framework can modulate the time course of CAT pre-activation. Additionally, during the attentional selection phase, an overall comparison of the target revealed that a larger N2pc was elicited by the prototype-based CAT than by the semantics-based CAT, suggesting that the prototype-based CAT could capture more attention than the semantics-based CAT. The findings on the difference between the two CAT frameworks in the preparatory and attentional selection phases provide more evidence for categorical information in visual search and extend our understanding of the mechanism of categorical attentional selection.

Comparing the Dominance of Color and Form Information across the Human Ventral Visual Pathway and Convolutional Neural Networks

Color and form information can be decoded in every region of the human ventral visual hierarchy, and at every layer of many convolutional neural networks (CNNs) trained to recognize objects, but how does the coding strength of these features vary over processing? Here, we characterize for these features both their absolute coding strength-how strongly each feature is represented independent of the other feature-and their relative coding strength-how strongly each feature is encoded relative to the other, which could constrain how well a feature can be read out by downstream regions across variation in the other feature. To quantify relative coding strength, we define a measure called the form dominance index that compares the relative influence of color and form on the representational geometry at each processing stage. We analyze brain and CNN responses to stimuli varying based on color and either a simple form feature, orientation, or a more complex form feature, curvature. We find that while the brain and CNNs largely differ in how the absolute coding strength of color and form vary over processing, comparing them in terms of their relative emphasis of these features reveals a striking similarity: For both the brain and for CNNs trained for object recognition (but not for untrained CNNs), orientation information is increasingly de-emphasized, and curvature information is increasingly emphasized, relative to color information over processing, with corresponding processing stages showing largely similar values of the form dominance index.

Category-specific and category-general neural codes of recognition memory in the ventral visual pathway

Researchers have identified category-specific brain regions, such as the fusiform face area (FFA) and parahippocampal place area (PPA) in the ventral visual pathway, which respond preferentially to one particular category of visual objects. In addition to their category-specific role in visual object identification and categorization, regions in the ventral visual pathway play critical roles in recognition memory. Nevertheless, it is not clear whether the contributions of those brain regions to recognition memory are category-specific or category-general. To address this question, the present study adopted a subsequent memory paradigm and multivariate pattern analysis (MVPA) to explore category-specific and category-general neural codes of recognition memory in the visual pathway. The results revealed that the right FFA and the bilateral PPA showed category-specific neural patterns supporting recognition memory of faces and scenes, respectively. In contrast, the lateral occipital cortex seemed to carry category-general neural codes of recognition memory. These results provide neuroimaging evidence for category-specific and category-general neural mechanisms of recognition memory in the ventral visual pathway.

Color-biased regions in the ventral visual pathway are food selective

Color-biased regions have been found between face- and place-selective areas in the ventral visual pathway. To investigate the function of the color-biased regions in a pathway responsible for object recognition, we analyzed the natural scenes dataset (NSD), a large 7T fMRI dataset from 8 participants who each viewed up to 30,000 trials of images of colored natural scenes over more than 30 scanning sessions. In a whole-brain analysis, we correlated the average color saturation of the images with voxel responses, revealing color-biased regions that diverge into two streams, beginning in V4 and extending medially and laterally relative to the fusiform face area in both hemispheres. We drew regions of interest (ROIs) for the two streams and found that the images for each ROI that evoked the largest responses had certain characteristics: they contained food, circular objects, warmer hues, and had higher color saturation. Further analyses showed that food images were the strongest predictor of activity in these regions, implying the existence of medial and lateral ventral food streams (VFSs). We found that color also contributed independently to voxel responses, suggesting that the medial and lateral VFSs use both color and form to represent food. Our findings illustrate how high-resolution datasets such as the NSD can be used to disentangle the multifaceted contributions of many visual features to the neural representations of natural scenes.

An "Instantaneous" Response of a Human Visual System to Hue: An EEG-Based Study

(1) The article presents a new technique to interpret biomedical data (EEG) to assess cortical responses to continuous color/hue variations. We propose an alternative approach to analyze EEG activity evoked by visual stimulation. This approach may augment the traditional VEP analysis. (2) Considering ensembles of EEG epochs as multidimensional spatial vectors evolving over time (rather than collections of time-domain signals) and evaluating the similarity between such vectors across different EEG epochs may result in a more accurate detection of colors that evoke greater responses of the visual system. To demonstrate its suitability, the developed analysis technique was applied to the EEG data that we previously collected from 19 participants with normal color vision, while exposing them to stimuli of continuously varying hue. (3) Orange/yellow and dark blue/violet colors generally aroused better-pronounced cortical responses. The selection of EEG channels allowed for assessing the activity that predominantly originates from specific cortical regions. With such channel selection, the strongest response to the hue was observed from Parieto-Temporal region of the right hemisphere. The statistical test-Kruskal-Wallis one-way analysis of variance-indicates that the distance evaluated for spatial EEG vectors at different post-stimulus latencies generally originate from different statistical distributions with a probability exceeding 99.9% (α = 0.001).

Understanding transformation tolerant visual object representations in the human brain and convolutional neural networks

Forming transformation-tolerant object representations is critical to high-level primate vision. Despite its significance, many details of tolerance in the human brain remain unknown. Likewise, despite the ability of convolutional neural networks (CNNs) to exhibit human-like object categorization performance, whether CNNs form tolerance similar to that of the human brain is unknown. Here we provide the first comprehensive documentation and comparison of three tolerance measures in the human brain and CNNs. We measured fMRI responses from human ventral visual areas to real-world objects across both Euclidean and non-Euclidean feature changes. In single fMRI voxels in higher visual areas, we observed robust object response rank-order preservation across feature changes. This is indicative of functional smoothness in tolerance at the fMRI meso-scale level that has never been reported before. At the voxel population level, we found highly consistent object representational structure across feature changes towards the end of ventral processing. Rank-order preservation, consistency, and a third tolerance measure, cross-decoding success (i.e., a linear classifier's ability to generalize performance across feature changes) showed an overall tight coupling. These tolerance measures were in general lower for Euclidean than non-Euclidean feature changes in lower visual areas, but increased over the course of ventral processing for all feature changes. These characteristics of tolerance, however, were absent in eight CNNs pretrained with ImageNet images with varying network architecture, depth, the presence/absence of recurrent processing, or whether a network was pretrained with the original or stylized ImageNet images that encouraged shape processing. CNNs do not appear to develop the same kind of tolerance as the human brain over the course of visual processing.