Cortical topographic motifs emerge in a self-organized map of object space

Individual variation in the functional lateralization of human ventral temporal cortex: Local competition and long-range coupling

The ventral temporal cortex (VTC) of the human cerebrum is critically engaged in high-level vision. One intriguing aspect of this region is its functional lateralization, with neural responses to words being stronger in the left hemisphere, and neural responses to faces being stronger in the right hemisphere; such patterns can be summarized with a signed laterality index (LI), positive for leftward laterality. Converging evidence has suggested that word laterality emerges to couple efficiently with left-lateralized frontotemporal language regions, but evidence is more mixed regarding the sources of the right lateralization for face perception. Here, we use individual differences as a tool to test three theories of VTC organization arising from (1) local competition between words and faces driven by long-range coupling between words and language processes, (2) local competition between faces and other categories, and (3) long-range coupling with VTC and temporal areas exhibiting local competition between language and social processing. First, in an in-house functional MRI experiment, we did not obtain a negative correlation in the LIs of word and face selectivity relative to object responses, but did find a positive correlation when using selectivity relative to a fixation baseline, challenging ideas of local competition between words and faces driving rightward face lateralization. We next examined broader local LI interactions with faces using the large-scale Human Connectome Project (HCP) dataset. Face and tool LIs were significantly anti-correlated, while face and body LIs were positively correlated, consistent with the idea that generic local representational competition and cooperation may shape face lateralization. Last, we assessed the role of long-range coupling in the development of VTC lateralization. Within our in-house experiment, substantial positive correlation was evident between VTC text LI and that of several other nodes of a distributed text-processing circuit. In the HCP data, VTC face LI was both negatively correlated with language LI and positively correlated with social processing in different subregions of the posterior temporal lobe (PSL and STSp, respectively). In summary, we find no evidence of local face-word competition in VTC; instead, more generic local interactions shape multiple lateralities within VTC, including face laterality. Moreover, face laterality is also influenced by long-range coupling with social processing in the posterior temporal lobe, where social processing may become right lateralized due to local competition with language.

Tool Representations in Human Visual Cortex

Tools such as pens, forks, and scissors play an important role in many daily-life activities, an importance underscored by the presence in visual cortex of a set of tool-selective brain regions. This review synthesizes decades of neuroimaging research that investigated the representational spaces in the visual ventral stream for objects, such as tools, that are specifically characterized by action-related properties. Overall, results reveal a dissociation between representational spaces in ventral and lateral occipito-temporal cortex (OTC). While lateral OTC encodes both visual (shape) and action-related properties of objects, distinguishing between objects acting as end-effectors (e.g., tools, hands) versus similar noneffector manipulable objects (e.g., a glass), ventral OTC primarily represents objects' visual features such as their surface properties (e.g., material and texture). These areas act in concert with regions outside of OTC to support object interaction and tool use. The parallel investigation of the dimensions underlying object representations in artificial neural networks reveals both the possibilities and the difficulties in capturing the action-related dimensions that distinguish tools from other objects. Although artificial neural networks offer promise as models of visual cortex computations, challenges persist in replicating the action-related dimensions that go beyond mere visual features. Taken together, we propose that regions in OTC support the representation of tools based on a behaviorally relevant action code and suggest future paths to generate a computational model of this object space.

Rarely categorical, always high-dimensional: how the neural code changes along the cortical hierarchy

A long-standing debate in neuroscience concerns whether individual neurons are organized into functionally distinct populations that encode information differently ("categorical" representations [1-3]) and the implications for neural computation. Here, we systematically analyzed how cortical neurons encode cognitive, sensory, and movement variables across 43 cortical regions during a complex task (14,000+ units from the International Brain Laboratory public Brain-wide Map data set [4]) and studied how these properties change across the sensory-cognitive cortical hierarchy [5]. We found that the structure of the neural code was scale-dependent: on a whole-cortex scale, neural selectivity was categorical and organized across regions in a way that reflected their anatomical connectivity. However, within individual regions, categorical representations were rare and limited to primary sensory areas. Remarkably, the degree of categorical clustering of neural selectivity was inversely correlated to the dimensionality of neural representations, suggesting a link between single-neuron selectivity and computational properties of population codes that we explained in a mathematical model. Finally, we found that the fraction of linearly separable combinations of experimental conditions ("Shattering Dimensionality" [6]) was near maximal across all areas, indicating a robust and uniform ability for flexible information encoding throughout the cortex. In conclusion, our results provide systematic evidence for a non-categorical, high-dimensional neural code in all but the lower levels of the cortical hierarchy.

Individual variation in the functional lateralization of human ventral temporal cortex: Local competition and long-range coupling

The ventral temporal cortex (VTC) of the human cerebrum is critically engaged in high-level vision. One intriguing aspect of this region is its functional lateralization, with neural responses to words being stronger in the left hemisphere, and neural responses to faces being stronger in the right hemisphere; such patterns can be summarized with a signed laterality index (LI), positive for leftward laterality. Converging evidence has suggested that word laterality emerges to couple efficiently with left-lateralized frontotemporal language regions, but evidence is more mixed regarding the sources of the right-lateralization for face perception. Here, we use individual differences as a tool to test three theories of VTC organization arising from: 1) local competition between words and faces driven by long-range coupling between words and language processes, 2) local competition between faces and other categories, 3) long-range coupling with VTC and temporal areas exhibiting local competition between language and social processing. First, in an in-house functional MRI experiment, we did not obtain a negative correlation in the LIs of word and face selectivity relative to object responses, but did find a positive correlation when using selectivity relative to a fixation baseline, challenging ideas of local competition between words and faces driving rightward face lateralization. We next examined broader local LI interactions with faces using the large-scale Human Connectome Project (HCP) dataset. Face and tool LIs were significantly anti-correlated, while face and body LIs were positively correlated, consistent with the idea that generic local representational competition and cooperation may shape face lateralization. Last, we assessed the role of long-range coupling in the development of VTC lateralization. Within our in-house experiment, substantial positive correlation was evident between VTC text LI and that of several other nodes of a distributed text-processing circuit. In the HCP data, VTC face LI was both negatively correlated with language LI and positively correlated with social processing in different subregions of the posterior temporal lobe (PSL and STSp, respectively). In summary, we find no evidence of local face-word competition in VTC; instead, more generic local interactions shape multiple lateralities within VTC, including face laterality. Moreover, face laterality is also influenced by long-range coupling with social processing in the posterior temporal lobe, where social processing may become right-lateralized due to local competition with language.

Object-based attention during scene perception elicits boundary contraction in memory

Boundary contraction and extension are two types of scene transformations that occur in memory. In extension, viewers extrapolate information beyond the edges of the image, whereas in contraction, viewers forget information near the edges. Recent work suggests that image composition influences the direction and magnitude of boundary transformation. We hypothesize that selective attention at encoding is an important driver of boundary transformation effects, selective attention to specific objects at encoding leading to boundary contraction. In this study, one group of participants (N = 36) memorized 15 scenes while searching for targets, while a separate group (N = 36) just memorized the scenes. Both groups then drew the scenes from memory with as much object and spatial detail as they could remember. We asked online workers to provide ratings of boundary transformations in the drawings, as well as how many objects they contained and the precision of remembered object size and location. We found that search condition drawings showed significantly greater boundary contraction than drawings of the same scenes in the memorize condition. Search drawings were significantly more likely to contain target objects, and the likelihood to recall other objects in the scene decreased as a function of their distance from the target. These findings suggest that selective attention to a specific object due to a search task at encoding will lead to significant boundary contraction.

Face cells encode object parts more than facial configuration of illusory faces

Humans perceive illusory faces in everyday objects with a face-like configuration, an illusion known as face pareidolia. Face-selective regions in humans and monkeys, believed to underlie face perception, have been shown to respond to face pareidolia images. Here, we investigated whether pareidolia selectivity in macaque inferotemporal cortex is explained by the face-like configuration that drives the human perception of illusory faces. We found that face cells responded selectively to pareidolia images. This selectivity did not correlate with human faceness ratings and did not require the face-like configuration. Instead, it was driven primarily by the "eye" parts of the illusory face, which are simply object parts when viewed in isolation. In contrast, human perceptual pareidolia relied primarily on the global configuration and could not be explained by "eye" parts. Our results indicate that face-cells encode local, generic features of illusory faces, in misalignment with human visual perception, which requires holistic information.

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity - a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations - suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems.

A biologically inspired computational model of human ventral temporal cortex

Our minds represent miscellaneous objects in the physical world metaphorically in an abstract and complex high-dimensional object space, which is implemented in a two-dimensional surface of the ventral temporal cortex (VTC) with topologically organized object selectivity. Here we investigated principles guiding the topographical organization of object selectivities in the VTC by constructing a hybrid Self-Organizing Map (SOM) model that harnesses a biologically inspired algorithm of wiring cost minimization and adheres to the constraints of the lateral wiring span of human VTC neurons. In a series of in silico experiments with functional brain neuroimaging and neurophysiological single-unit data from humans and non-human primates, the VTC-SOM predicted the topographical structure of fine-scale category-selective regions (face-, tool-, body-, and place-selective regions) and the boundary in large-scale abstract functional maps (animate vs. inanimate, real-word small-size vs. big-size, central vs. peripheral), with no significant loss in functionality (e.g., categorical selectivity and view-invariant representations). In addition, when the same principle was applied to V1 orientation preferences, a pinwheel-like topology emerged, suggesting the model's broad applicability. In summary, our study illustrates that the simple principle of wiring cost minimization, coupled with the appropriate biological constraint of lateral wiring span, is able to implement the high-dimensional object space in a two-dimensional cortical surface.

Contrastive learning explains the emergence and function of visual category-selective regions

Modular and distributed coding theories of category selectivity along the human ventral visual stream have long existed in tension. Here, we present a reconciling framework-contrastive coding-based on a series of analyses relating category selectivity within biological and artificial neural networks. We discover that, in models trained with contrastive self-supervised objectives over a rich natural image diet, category-selective tuning naturally emerges for faces, bodies, scenes, and words. Further, lesions of these model units lead to selective, dissociable recognition deficits, highlighting their distinct functional roles in information processing. Finally, these pre-identified units can predict neural responses in all corresponding face-, scene-, body-, and word-selective regions of human visual cortex, under a highly constrained sparse positive encoding procedure. The success of this single model indicates that brain-like functional specialization can emerge without category-specific learning pressures, as the system learns to untangle rich image content. Contrastive coding, therefore, provides a unifying account of object category emergence and representation in the human brain.

The Quest for an Integrated Set of Neural Mechanisms Underlying Object Recognition in Primates

Inferences made about objects via vision, such as rapid and accurate categorization, are core to primate cognition despite the algorithmic challenge posed by varying viewpoints and scenes. Until recently, the brain mechanisms that support these capabilities were deeply mysterious. However, over the past decade, this scientific mystery has been illuminated by the discovery and development of brain-inspired, image-computable, artificial neural network (ANN) systems that rival primates in these behavioral feats. Apart from fundamentally changing the landscape of artificial intelligence, modified versions of these ANN systems are the current leading scientific hypotheses of an integrated set of mechanisms in the primate ventral visual stream that support core object recognition. What separates brain-mapped versions of these systems from prior conceptual models is that they are sensory computable, mechanistic, anatomically referenced, and testable (SMART). In this article, we review and provide perspective on the brain mechanisms addressed by the current leading SMART models. We review their empirical brain and behavioral alignment successes and failures, discuss the next frontiers for an even more accurate mechanistic understanding, and outline the likely applications.

A unifying framework for functional organization in early and higher ventral visual cortex

A key feature of cortical systems is functional organization: the arrangement of functionally distinct neurons in characteristic spatial patterns. However, the principles underlying the emergence of functional organization in the cortex are poorly understood. Here, we develop the topographic deep artificial neural network (TDANN), the first model to predict several aspects of the functional organization of multiple cortical areas in the primate visual system. We analyze the factors driving the TDANN's success and find that it balances two objectives: learning a task-general sensory representation and maximizing the spatial smoothness of responses according to a metric that scales with cortical surface area. In turn, the representations learned by the TDANN are more brain-like than in spatially unconstrained models. Finally, we provide evidence that the TDANN's functional organization balances performance with between-area connection length. Our results offer a unified principle for understanding the functional organization of the primate ventral visual system.

When the whole is only the parts: non-holistic object parts predominate face-cell responses to illusory faces

Humans are inclined to perceive faces in everyday objects with a face-like configuration. This illusion, known as face pareidolia, is often attributed to a specialized network of 'face cells' in primates. We found that face cells in macaque inferotemporal cortex responded selectively to pareidolia images, but this selectivity did not require a holistic, face-like configuration, nor did it encode human faceness ratings. Instead, it was driven mostly by isolated object parts that are perceived as eyes only within a face-like context. These object parts lack usual characteristics of primate eyes, pointing to the role of lower-level features. Our results suggest that face-cell responses are dominated by local, generic features, unlike primate visual perception, which requires holistic information. These findings caution against interpreting neural activity through the lens of human perception. Doing so could impose human perceptual biases, like seeing faces where none exist, onto our understanding of neural activity.

The neural code for "face cells" is not face-specific

Face cells are neurons that respond more to faces than to non-face objects. They are found in clusters in the inferotemporal cortex, thought to process faces specifically, and, hence, studied using faces almost exclusively. Analyzing neural responses in and around macaque face patches to hundreds of objects, we found graded response profiles for non-face objects that predicted the degree of face selectivity and provided information on face-cell tuning beyond that from actual faces. This relationship between non-face and face responses was not predicted by color and simple shape properties but by information encoded in deep neural networks trained on general objects rather than face classification. These findings contradict the long-standing assumption that face versus non-face selectivity emerges from face-specific features and challenge the practice of focusing on only the most effective stimulus. They provide evidence instead that category-selective neurons are best understood by their tuning directions in a domain-general object space.