Matching categorical object representations in inferior temporal cortex of man and monkey

Time-dependent scale-free brain dynamics during naturalistic inputs

Environmental processes, such as auditory and visual inputs, often follow power-law distributions with a time-dependent and constantly changing spectral exponent, β(t). However, it remains unclear how the brain's scale-free dynamics continuously respond to naturalistic inputs, such as by potentially alternating instead of static levels of the spectral exponent. Our fMRI study investigates the brain's dynamic, time-dependent spectral exponent, β(t), during movie-watching, and uses time-varying inter-subject correlation, ISC(t), to assess the extent to which input dynamics are reflected as shared brain activity across subjects in early sensory regions. Notably, we investigate the level of ISC particularly based on the modulation by time-dependent scale-free dynamics or β(t). We obtained three key findings: First, the brain's β(t) showed a distinct temporal structure in visual and auditory regions during naturalistic inputs compared to the resting-state, investigated in the 7 Tesla Human Connectome Project dataset. Second, β(t) and ISC(t) were positively correlated during naturalistic inputs. Third, grouping subjects based on the Rest-to-Movie standard deviation change of the time-dependent spectral exponent β(t) revealed that the brain's relative shift from intrinsic to stimulus-driven scale-free dynamics modulates the level of shared brain activity, or ISC(t), and thus the imprinting of inputs on brain activity. This modulation was further supported by the observation that the two groups displayed significantly different β(t)-ISC(t) correlations, where the group with a higher mean of ISC(t) during inputs also exhibited a higher β(t)-ISC(t) correlation in visual and auditory regions. In summary, our fMRI study underscores a positive relationship between time-dependent scale-free dynamics and ISC, where higher spectral exponents correspond to higher degrees of shared brain activity during ongoing audiovisual inputs.

Unsupervised alignment in neuroscience: Introducing a toolbox for Gromov-Wasserstein optimal transport

BACKGROUND

Understanding how sensory stimuli are represented across different brains, species, and artificial neural networks is a critical topic in neuroscience. Traditional methods for comparing these representations typically rely on supervised alignment, which assumes direct correspondence between stimuli representations across brains or models. However, it has limitations when this assumption is not valid, or when validating the assumption itself is the goal of the research.

NEW METHOD

To address the limitations of supervised alignment, we propose an unsupervised alignment method based on Gromov-Wasserstein optimal transport (GWOT). GWOT optimally identifies correspondences between representations by leveraging internal relationships without external labels, revealing intricate structural correspondences such as one-to-one, group-to-group, and shifted mappings.

RESULTS

We provide a comprehensive methodological guide and introduce a toolbox called GWTune for using GWOT in neuroscience. Our results show that GWOT can reveal detailed structural distinctions that supervised methods may overlook. We also demonstrate successful unsupervised alignment in key data domains, including behavioral data, neural activity recordings, and artificial neural network models, demonstrating its flexibility and broad applicability.

COMPARISON WITH EXISTING METHODS

Unlike traditional supervised alignment methods such as Representational Similarity Analysis, which assume direct correspondence between stimuli, GWOT provides a nuanced approach that can handle different types of structural correspondence, including fine-grained and coarse correspondences. Our method would provide richer insights into the similarity or difference of representations by revealing finer structural differences.

CONCLUSION

We anticipate that our work will significantly broaden the accessibility and application of unsupervised alignment in neuroscience, offering novel perspectives on complex representational structures. By providing a user-friendly toolbox and a detailed tutorial, we aim to facilitate the adoption of unsupervised alignment techniques, enabling researchers to achieve a deeper understanding of cross-brain and cross-species representation analysis.

The impact of spatial frequency on hierarchical category representation in macaque temporal cortex

Objects are recognized in three hierarchical levels: superordinate, mid-level, and subordinate. Psychophysics shows that mid-level categories and low spatial frequency (LSF) information are rapidly recognized. However, the interaction between spatial frequency (SF) and abstraction is not well understood. To address this, we examine neural responses in the inferior temporal cortex and superior temporal sulcus of two male macaque monkeys. Our findings reveal that mid-level categories are well represented at both LSF and high SF (HSF), suggesting robust mid-level boundary maps in these areas, unaffected by SF changes. Conversely, superordinate category representation depends on HSF, indicating its crucial role in encoding global category information. The absence of subordinate representation in both LSF and HSF compared to intact stimuli further implies that full SF content is essential for fine-category processing. A supporting human psychophysics task confirms that superordinate categorization relies on HSF, while subordinate object recognition requires both LSF and HSF.

Unsupervised alignment reveals structural commonalities and differences in neural representations of natural scenes across individuals and brain areas

Neuroscience research aims to identify universal neural mechanisms underlying sensory information encoding by comparing neural representations across individuals, typically using Representational Similarity Analysis. However, traditional methods assume direct stimulus correspondence across individuals, limiting the exploration of other possibilities. To address this, we propose an unsupervised alignment framework based on Gromov-Wasserstein Optimal Transport, which identifies correspondences between neural representations solely from internal similarity structures, without relying on stimulus labels. Applying this method to Neuropixels recordings in mice and fMRI data in humans viewing natural scenes, we found that the neural representations in the same visual cortical areas can be well aligned across individuals in an unsupervised manner. Furthermore, alignment across different brain areas is influenced by factors beyond the visual hierarchy, with higher-order visual areas aligning well with each other, but not with lower-order areas. This unsupervised approach reveals more nuanced structural commonalities and differences in neural representations than conventional methods.

Fine-Grained Concreteness Effects on Word Processing and Representation Across Three Tasks: An ERP Study

People process concrete words more quickly and accurately than abstract ones-the so-called "concreteness effect." This advantage also reflects differences in how the brain processes and stores concrete versus abstract words. In this electrophysiological study, we treated word concreteness as a continuous variable and examined its effects on ERPs across three tasks with distinct processing demands (semantic, affective, grammatical). Behavioral results revealed task-dependent concreteness effects: in the semantic task, reaction times were faster for words at both concreteness extremes, and the classical linear advantage emerged for concrete words. Mass univariate ERP analyses revealed distinct spatiotemporal patterns of task-dependent concreteness effects. In the semantic task, we identified three significant clusters reflecting increased parietal N2/P3-like and sustained bilateral fronto-temporal negativity ERPs and decreased central N400-like ERP for abstract words. By contrast, the affective task elicited an increased parietal P600-like ERP for abstract words. Moreover, results from multivariate representational similarity analysis and an intersection analysis revealed that concreteness is encoded in ERP spatiotemporal patterns from 450 ms onwards, regardless of task, suggesting its role not only as an organizational principle in semantic representation, but also as a factor influencing downstream word processing and univariate ERP concreteness effects. Our findings challenge and extend existing theories like the dual coding and context availability ones, highlighting the importance of treating concreteness as a continuous variable and considering task context in word processing studies. This approach, enabled by advanced analytical techniques, provides a more nuanced understanding of how the brain processes and represents words.

Brain feature maps reveal progressive animal-feature representations in the ventral stream

What are the fundamental principles that inform representation in the primate visual brain? While objects have become an intuitive framework for studying neurons in many parts of cortex, it is possible that neurons follow a more expressive organizational principle, such as encoding generic features present across textures, places, and objects. In this study, we used multielectrode arrays to record from neurons in the early (V1/V2), middle (V4), and later [posterior inferotemporal (PIT) cortex] areas across the visual hierarchy, estimating each neuron's local operation across natural scene via "heatmaps." We found that, while populations of neurons with foveal receptive fields across V1/V2, V4, and PIT responded over the full scene, they focused on salient subregions within object outlines. Notably, neurons preferentially encoded animal features rather than general objects, with this trend strengthening along the visual hierarchy. These results show that the monkey ventral stream is partially organized to encode local animal features over objects, even as early as primary visual cortex.

Worry and rumination elicit similar neural representations: neuroimaging evidence for repetitive negative thinking

Repetitive negative thinking (RNT) captures shared cognitive and emotional features of content-specific cognition, including future-focused worry and past-focused rumination. The degree to which these distinct but related processes recruit overlapping neural structures is undetermined, because most neuroscientific studies only examine worry or rumination in isolation. To address this, we developed a paradigm to elicit idiographic worries and ruminations during an fMRI scan in 39 young adults with a range of trait RNT scores. We measured concurrent emotion ratings and heart rate as a physiological metric of arousal. Multivariate representational similarity analysis revealed that regions distributed across default mode, salience, and frontoparietal control networks encode worry and rumination similarly. Moreover, heart rate did not differ between worry and rumination. Capturing the shared neural features between worry and rumination throughout networks supporting self-referential processing, memory, salience detection, and cognitive control provides novel empirical evidence to bolster cognitive and clinical models of RNT.

Core dimensions of human material perception

Visually categorizing and comparing materials is crucial for everyday behavior, but what organizational principles underlie our mental representation of materials? Here, we used a large-scale data-driven approach to uncover core latent dimensions of material representations from behavior. First, we created an image dataset of 200 systematically sampled materials and 600 photographs (STUFF dataset, https://osf.io/myutc/). Using these images, we next collected 1.87 million triplet similarity judgments and used a computational model to derive a set of sparse, positive dimensions underlying these judgments. The resulting multidimensional embedding space predicted independent material similarity judgments and the similarity matrix of all images close to the human intersubject consistency. We found that representations of individual images were captured by a combination of 36 material dimensions that were highly reproducible and interpretable, comprising perceptual (e.g., grainy, blue) as well as conceptual (e.g., mineral, viscous) dimensions. These results provide the foundation for a comprehensive understanding of how humans make sense of materials.

Alexithymia shapes intersubject synchrony in brain activity during interoceptive sensation representations

Alexithymia is a subclinical condition that affects individuals' processing of emotions. Emerging evidence suggests that alexithymia results from a multidomain and multidimensional interoceptive failure. Although extensive research has examined the relationship between alexithymia and interoception, less is known about how alexithymia modulates the brain activity evoked by interoceptive sensations. In this study, we used task-based functional magnetic resonance imaging (fMRI) to assess intersubject correlations in response to interoceptive sensation words in individuals with high alexithymia and low alexithymia. Participants with high alexithymia (n = 29) and low alexithymia (n = 28) were instructed to view words during MRI scanning, each word corresponding to a specific emotional category related to interoceptive sensations. Intersubject correlation analysis identified several brain regions exhibiting increased synchronization in individuals with high alexithymia, including those involved in cognitive control. Follow-up analyses revealed that the left middle occipital gyrus and the right inferior frontal gyrus (orbital part) were more active during interoceptive sensation events in individuals with high alexithymia. Validation analyses revealed that the amygdala and insula are also crucial in representing interoceptive sensations. These findings shed light on the neural basis of interoceptive deficits in high alexithymia and have significant implications for the mechanisms regulating these differences.

Decoding the impact of negative physical self-perception on inhibitory control ability from theta and beta rhythms

Previous studies have found inhibitory control differences between obese individuals and those of normal weight. However, some normal-weight individuals with high negative physical self-perception on the fatness subscale show restrictive eating behaviors and attentional bias toward high-calorie food, potentially influencing these differences. We collected behavioral and electroencephalography data using a novel inhibitory control task. Results showed that individuals with high negative physical self-perception on the fatness subscale exhibited significantly greater restraint eating behavior compared to controls. Both theta and beta power differed between groups, with higher theta power in the high negative physical self-perception on the fatness subscale group than in the obese group and more negative beta power in the high negative physical self-perception on the fatness subscale group compared to both other groups. Theta power was greater in no-go than go conditions, while beta power was more negative in response to high-calorie versus low-calorie food stimuli. Importantly, theta power successfully decoded go/no-go conditions across all groups using multivariate pattern analysis, while beta power distinguished these conditions only in the negative physical self-perception on the fatness subscale and control groups. These findings suggest that theta and beta power, along with multivariate pattern analysis, can reliably distinguish inhibitory control ability among the three groups, highlighting the importance of considering negative physical self-perception on the fatness subscale when assessing inhibitory control differences between normal-weight and obese individuals.

Enhanced and idiosyncratic neural representations of personally typical scenes

Previous research shows that the typicality of visual scenes (i.e. if they are good examples of a category) determines how easily they can be perceived and represented in the brain. However, the unique visual diets individuals are exposed to across their lifetimes should sculpt very personal notions of typicality. Here, we thus investigated whether scenes that are more typical to individual observers are more accurately perceived and represented in the brain. We used drawings to enable participants to describe typical scenes (e.g. a kitchen) and converted these drawings into three-dimensional renders. These renders were used as stimuli in a scene categorization task, during which we recorded electroencephalography (EEG). In line with previous findings, categorization was most accurate for renders resembling the typical scene drawings of individual participants. Our EEG analyses reveal two critical insights on how these individual differences emerge on the neural level. First, personally typical scenes yielded enhanced neural representations from around 200 ms after onset. Second, personally typical scenes were represented in idiosyncratic ways, with reduced dependence on high-level visual features. We interpret these findings in a predictive processing framework, where individual differences in internal models of scene categories formed through experience shape visual analysis in idiosyncratic ways.

Dynamic representation of multidimensional object properties in the human brain

Our visual world consists of an immense number of unique objects and yet, we are easily able to identify, distinguish, interact, and reason about the things we see within a few hundred milliseconds. This requires that we integrate and focus on a wide array of object properties to support diverse behavioral goals. In the current study, we used a large-scale and comprehensively sampled stimulus set and developed an analysis approach to determine if we could capture how rich, multidimensional object representations unfold over time in the human brain. We modelled time-resolved MEG signals evoked by viewing single presentations of tens of thousands of object images based on millions of behavioral judgments. Extracting behavior-derived object dimensions from similarity judgments, we developed a data-driven approach to guide our understanding of the neural representation of the object space and found that every dimension is reflected in the neural signal. Studying the temporal profiles for different object dimensions we found that the time courses fell into two broad types, with either a distinct and early peak (~125 ms) or a slow rise to a late peak (~300 ms). Further, early effects were stable across participants, in contrast to later effects which showed more variability, suggesting that early peaks may carry stimulus-specific and later peaks more participant-specific information. Dimensions with early peaks appeared to be primarily visual dimensions and those with later peaks more conceptual, suggesting that conceptual representations are more variable across people. Together, these data provide a comprehensive account of how behavior-derived object properties unfold in the human brain and form the basis for the rich nature of object vision.

Perceptual and semantic maps in individual humans share structural features that predict creative abilities

Building perceptual and associative links between internal representations is a fundamental neural process, allowing individuals to structure their knowledge about the world and combine it to enable efficient and creative behavior. In this context, the representational similarity between pairs of represented entities is thought to reflect their associative linkage at different levels of sensory processing, ranging from lower-order perceptual levels up to higher-order semantic levels. While recently specific structural features of semantic representational maps were linked with creative abilities of individual humans, it remains unclear if these features are also shared on lower level, perceptual maps. Here, we address this question by presenting 148 human participants with psychophysical scaling tasks, using two sets of independent and qualitatively distinct stimuli, to probe representational map structures in the lower-order auditory and the higher-order semantic domain. We quantify individual representational features with graph-theoretical measures and demonstrate a robust correlation of representational structures in the perceptual auditory and semantic modality. We delineate these shared representational features to predict multiple verbal standard measures of creativity, observing that both, semantic and auditory features, reflect creative abilities. Our findings indicate that the general, modality-overarching representational geometry of an individual is a relevant underpinning of creative thought.

Visual homogeneity computations in the brain enable solving property-based visual tasks

Most visual tasks involve looking for specific object features. But we also often perform property-based tasks where we look for specific property in an image, such as finding an odd item, deciding if two items are same, or if an object has symmetry. How do we solve such tasks? These tasks do not fit into standard models of decision making because their underlying feature space and decision process is unclear. Using well-known principles governing multiple object representations, we show that displays with repeating elements can be distinguished from heterogeneous displays using a property we define as visual homogeneity. In behavior, visual homogeneity predicted response times on visual search, same-different and symmetry tasks. Brain imaging during visual search and symmetry tasks revealed that visual homogeneity was localized to a region in the object-selective cortex. Thus, property-based visual tasks are solved in a localized region in the brain by computing visual homogeneity.

Emotional state dynamics impacts temporal memory

Emotional fluctuations are ubiquitous in everyday life, but precisely how they sculpt the temporal organisation of memories remains unclear. Here, we designed a novel task - the Emotion Boundary Task - wherein participants viewed sequences of negative and neutral images surrounded by a colour border. We manipulated perceptual context (border colour), emotional-picture valence, as well as the direction of emotional-valence shifts (i.e., shifts from neutral-to-negative and negative-to-neutral events) to create events with a shared perceptual and/or emotional context. We measured memory for temporal order and temporal distances for images processed within and across events. Negative images processed within events were remembered as closer in time compared to neutral ones. In contrast, temporal distances were remembered as longer for images spanning neutral-to-negative shifts - suggesting temporal dilation in memory with the onset of a negative event following a previously-neutral state. The extent of negative-picture induced temporal dilation in memory correlated with dispositional negativity across individuals. Lastly, temporal order memory was enhanced for recently-presented negative (versus neutral) images. These findings suggest that emotional-state dynamics matters when considering emotion-temporal memory interactions: While persistent negative events may compress subjectively remembered time, dynamic shifts from neutral-to-negative events produce temporal dilation in memory, with implications for adaptive emotional functioning.

Noradrenergic and Dopaminergic modulation of meta-cognition and meta-control

Humans and animals use multiple control systems for decision-making. This involvement is subject to meta-cognitive regulation - as a form of control over control or meta-control. However, the nature of this meta-control is unclear. For instance, Model-based (MB) control may be boosted when decision-makers generally lack confidence as it is more statistically efficient; or it may be suppressed, since the MB controller can correctly assess its own unreliability. Since control and metacontrol are themselves subject to the influence of neuromodulators, we examined the effects of perturbing the noradrenergic (NE) and dopaminergic (DA) systems with propranolol and L-DOPA, respectively. We first administered a simple perceptual task to examine the effects of the manipulations on meta-cognitive ability. Using Bayesian analyses, we found that 81% of group M-ratio samples were lower under propranolol relative to placebo, suggesting a decrease of meta-cognitive ability; and 60% of group M-ratio samples were higher under L-DOPA relative to placebo, considered as no effect of L-DOPA on meta-cognitive ability . We then asked subjects to provide choices and confidence ratings in a two-outcome decision-making task that has been used to dissociate Model-free (MF) and MB control. MB behavior was enhanced by propranolol, while MF behavior was not significantly affected by either drug. The interaction between confidence and MF/MB behavior was highly variable under propranolol, but under L-DOPA, the interaction was significantly lower/higher relative to placebo. Our results suggest a decrease in metacognitive ability under the influence of propranolol and an enhancement of MB behavior and meta-control under the influence of propranolol and L-DOPA, respectively. These findings shed light on the role of NE and DA in different aspects of control and meta-control and suggest potential avenues for mitigating dysfunction.

Neural representation of sensorimotor features in language-motor areas during auditory and visual perception

Speech processing involves a complex interplay between sensory and motor systems in the brain, essential for early language development. Recent studies have extended this sensory-motor interaction to visual word processing, emphasizing the connection between reading and handwriting during literacy acquisition. Here we show how language-motor areas encode motoric and sensory features of language stimuli during auditory and visual perception, using functional magnetic resonance imaging (fMRI) combined with representational similarity analysis. Chinese-speaking adults completed tasks involving the perception of spoken syllables and written characters, alongside syllable articulation and finger writing tasks to localize speech-motor and writing-motor areas. We found that both language-motor and sensory areas generally encode production-related motoric features across modalities, indicating cooperative interactions between motor and sensory systems. Notably, sensory encoding within sensorimotor areas was observed during auditory speech perception, but not in visual character perception. These findings underscore the dual encoding capacities of language-motor areas, revealing both shared and distinct neural representation patterns across modalities, which may be linked to innate sensory-motor mechanisms and modality-specific processing demands. Our results shed light on the sensorimotor integration mechanisms underlying language perception, highlighting the importance of a cross-modality perspective.

The brain prioritizes the basic level of object category abstraction

The same object can be described at multiple levels of abstraction ("parka", "coat", "clothing"), yet human observers consistently name objects at a mid-level of specificity known as the basic level. Little is known about the temporal dynamics involved in retrieving neural representations that prioritize the basic level, nor how these dynamics change with evolving task demands. In this study, observers viewed 1080 objects arranged in a three-tier category taxonomy while 64-channel EEG was recorded. Observers performed a categorical one-back task in different recording sessions on the basic or subordinate levels. We used time-resolved multiple regression to assess the utility of superordinate-, basic-, and subordinate-level categories across the scalp. We found robust use of basic-level category information starting at about 50 ms after stimulus onset and moving from posterior electrodes (149 ms) through lateral (261 ms) to anterior sites (332 ms). Task differences were not evident in the first 200 ms of processing but were observed between 200-300 ms after stimulus presentation. Together, this work demonstrates that the object category representations prioritize the basic level and do so relatively early, congruent with results that show that basic-level categorization is an automatic and obligatory process.

Mapping and modeling the semantic space of math concepts

Mathematics is an underexplored domain of human cognition. While many studies have focused on subsets of math concepts such as numbers, fractions, or geometric shapes, few have ventured beyond these elementary domains. Here, we attempted to map out the full space of math concepts and to answer two specific questions: can distributed semantic models, such a GloVe, provide a satisfactory fit to human semantic judgements in mathematics? And how does this fit vary with education? We first analyzed all of the French and English Wikipedia pages with math contents, and used a semi-automatic procedure to extract the 1000 most frequent math terms in both languages. In a second step, we collected extensive behavioral judgements of familiarity and semantic similarity between them. About half of the variance in human similarity judgements was explained by vector embeddings that attempt to capture latent semantic structures based on cooccurence statistics. Participants' self-reported level of education modulated familiarity and similarity, allowing us to create a partial hierarchy among high-level math concepts. Our results converge onto the proposal of a map of math space, organized as a database of math terms with information about their frequency, familiarity, grade of acquisition, and entanglement with other concepts.

Identifying Behavioral Correlates to Visual Discomfort

Outside of self-report surveys, there are no proven, reliable methods to quantify visual discomfort or visually induced motion sickness symptoms when using head-mounted displays. While valuable tools, self-report surveys suffer from potential biases and low sensitivity due to variability in how respondents may assess and report their experience. Consequently, extreme visual-vestibular conflicts are generally used to induce discomfort symptoms large enough to measure reliably with surveys (e.g., stationary participants riding virtual roller coasters). An emerging area of research is the prediction of discomfort survey results from physiological and behavioral markers. However, the signals derived from experimental paradigms that are explicitly designed to be uncomfortable may not generalize to more naturalistic experiences where comfort is prioritized. In this work we introduce a custom VR headset designed to introduce significant near-eye optical distortion (i.e., pupil swim) to induce visual discomfort during more typical VR experiences. We evaluate visual comfort in our headset while users play the popular VR title Job Simulator and show that eye-tracked dynamic distortion correction improves visual comfort in a multi-session, within-subjects user study. We additionally use representational similarity analysis to highlight changes in head and gaze behavior that are potentially more sensitive to visual discomfort than surveys.

Animacy processing by distributed and interconnected networks in the temporal cortex of monkeys

Animacy perception, the ability to discern living from non-living entities, is crucial for survival and social interaction, as it includes recognizing abstract concepts such as movement, purpose, and intentions. This process involves interpreting cues that may suggest the intentions or actions of others. It engages the temporal cortex (TC), particularly the superior temporal sulcus (STS) and the adjacent region of the inferior temporal cortex (ITC), as well as the dorsomedial prefrontal cortex (dmPFC). However, it remains unclear how animacy is dynamically encoded over time in these brain areas and whether its processing is distributed or localized. In this study, we addressed these questions by employing a symbolic categorization task involving animate and inanimate objects using natural movie stimuli. Simultaneously, electrocorticography were conducted in both the TC and dmPFC. Time-frequency analysis revealed region-specific frequency representations throughout the observation of the movies. Spatial searchlight decoding analysis demonstrated that animacy processing is represented in a distributed manner. Regions encoding animacy information were found to be dispersed across the fundus and lip of the STS, as well as in the ITC. Next, we examined whether these dispersed regions form functional networks. Independent component analysis revealed that the spatial distribution of the component with the most significant animacy information corresponded with the dispersed regions identified by the spatial decoding analysis. Furthermore, Granger causality analysis indicated that these regions exhibit frequency-specific directional functional connectivity, with a general trend of causal influence from the ITC to STS across multiple frequency bands. Notably, a prominent feedback flow in the alpha band from the ITC to both the ventral bank and fundus of the STS was identified. These findings suggest a distributed and functionally interconnected neural substrate for animacy processing across the STS and ITC.

Enhanced control of a brain-computer interface by tetraplegic participants via neural-network-mediated feature extraction

To infer intent, brain-computer interfaces must extract features that accurately estimate neural activity. However, the degradation of signal quality over time hinders the use of feature-engineering techniques to recover functional information. By using neural data recorded from electrode arrays implanted in the cortices of three human participants, here we show that a convolutional neural network can be used to map electrical signals to neural features by jointly optimizing feature extraction and decoding under the constraint that all the electrodes must use the same neural-network parameters. In all three participants, the neural network led to offline and online performance improvements in a cursor-control task across all metrics, outperforming the rate of threshold crossings and wavelet decomposition of the broadband neural data (among other feature-extraction techniques). We also show that the trained neural network can be used without modification for new datasets, brain areas and participants.

Unsupervised method for representation transfer from one brain to another

Although the anatomical arrangement of brain regions and the functional structures within them are similar across individuals, the representation of neural information, such as recorded brain activity, varies among individuals owing to various factors. Therefore, appropriate conversion and translation of brain information is essential when decoding neural information using a model trained using another person's data or to achieving brain-to-brain communication. We propose a brain representation transfer method that involves transforming a data representation obtained from one person's brain into that obtained from another person's brain, without relying on corresponding label information between the transferred datasets. We defined the requirements to enable such brain representation transfer and developed an algorithm that distills the assumption of common similarity structure across the brain datasets into a rotational and reflectional transformation across low-dimensional hyperspheres using encoders for non-linear dimensional reduction. We first validated our proposed method using data from artificial neural networks as substitute neural activity and examining various experimental factors. We then evaluated the applicability of our method to real brain activity using functional magnetic resonance imaging response data acquired from human participants. The results of these validation experiments showed that our method successfully performed representation transfer and achieved transformations in some cases that were similar to those obtained when using corresponding label information. Additionally, we reconstructed images from individuals' data without training personalized decoders by performing brain representation transfer. The results suggest that our unsupervised transfer method is useful for the reapplication of existing models personalized to specific participants and datasets to decode brain information from other individuals. Our findings also serve as a proof of concept for the methodology, enabling the exchange of the latent properties of neural information representing individuals' sensations.

Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision

We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation.

A data-driven analysis of the perceptual and neural responses to natural objects reveals organising principles of human visual cognition

A key challenge in understanding the functional organisation of visual cortex stems from the fact that only a small proportion of the objects experienced during natural viewing can be presented in a typical experiment. This constraint often leads to experimental designs that compare responses to objects from experimenter-defined stimulus conditions, potentially limiting the interpretation of the data. To overcome this issue, we used images from the THINGS initiative, which provides a systematic sampling of natural objects. A data-driven analysis was then applied to reveal the functional organisation of the visual brain, incorporating both perceptual and neural responses to these objects. Perceptual properties of the objects were taken from an analysis of similarity judgements, and neural properties were taken from whole brain fMRI responses to the same objects. Partial least squares regression (PLSR) was then used to predict neural responses across the brain from the perceptual properties while simultaneously applying dimensionality reduction. The PLSR model accurately predicted neural responses across visual cortex using only a small number of components. These components revealed smooth, graded neural topographies, which were similar in both hemispheres, and captured a variety of object properties including animacy, real-world size, and object category. However, they did not accord in any simple way with previous theoretical perspectives on object perception. Instead, our findings suggest that visual cortex encodes information in a statistically efficient manner, reflecting natural variability among objects.Significance statement The ability to recognise objects is fundamental to how we interact with our environment, yet the organising principles underlying neural representations of visual objects remain contentious. In this study, we sought to address this question by analysing perceptual and neural responses to a large, unbiased sample of objects. Using a data-driven approach, we leveraged perceptual properties of objects to predict neural responses using a small number of components. This model predicted neural responses with a high degree of accuracy across visual cortex. The components did not directly align with previous explanations of object perception. Instead, our findings suggest the organisation of the visual brain is based on the statistical properties of objects in the natural world.

Distinct mechanisms and functions of episodic memory

The concept of episodic memory (EM) faces significant challenges by two claims: EM might not be a distinct memory system, and EM might be an epiphenomenon of a more general capacity for mental time travel (MTT). Nevertheless, the observations leading to these arguments do not preclude the existence of a mechanically and functionally distinct EM system. First, modular systems, like cognition, can have distinct subsystems that may not be distinguishable in the system's final output. EM could be such a subsystem, even though its effects may be difficult to distinguish from those of other subsystems. Second, EM could have a distinct and consistent low-level function, which is used in diverse high-level functions such as MTT. This article introduces the scenario construction framework, proposing that EM crucially rests on memory traces containing the gist of an episodic experience. During retrieval, EM traces trigger the reconstruction of semantic representations, which were active during the remembered episode, and are further enriched with semantic information, to generate a scenario of the past experience. This conceptualization of EM is consistent with studies on the neural basis of EM and resolves the two challenges while retaining the key properties associated with EM. This article is part of the theme issue 'Elements of episodic memory: lessons from 40 years of research'.

Spatiotemporal hierarchies of face representation in the human ventral temporal cortex

In this study, we examined the relatively unexplored realm of face perception, investigating the activities within human brain face-selective regions during the observation of faces at both subordinate and superordinate levels. We recorded intracranial EEG signals from the ventral temporal cortex in neurosurgical patients implanted with subdural electrodes during viewing of face subcategories (human, mammal, bird, and marine faces) as well as various non-face control stimuli. The results revealed a noteworthy correlation in response patterns across all face-selective areas in the ventral temporal cortex, not only within the same face category but also extending to different face categories. Intriguingly, we observed a systematic decrease in response correlation coupled with an increased response onset time from human face to mammalian face, bird face and marine faces. Our result aligns with the notion that distinctions at the basic level category (e.g., human face versus non-human face) emerges earlier than those at the superordinate level (e.g., animate versus inanimate). This indicates response gradient in the representation of facial images within human face-sensitive regions, transitioning progressively from human faces to non-face stimuli. Our findings provide insights into spatiotemporal dynamic of face representations which varies spatially and at different timescales depending on the face subcategory represented.

High-level visual prediction errors in early visual cortex

Perception is shaped by both incoming sensory input and expectations derived from our prior knowledge. Numerous studies have shown stronger neural activity for surprising inputs, suggestive of predictive processing. However, it is largely unclear what predictions are made across the cortical hierarchy, and therefore what kind of surprise drives this up-regulation of activity. Here, we leveraged fMRI in human volunteers and deep neural network (DNN) models to arbitrate between 2 hypotheses: prediction errors may signal a local mismatch between input and expectation at each level of the cortical hierarchy, or prediction errors may be computed at higher levels and the resulting surprise signal is broadcast to earlier areas in the cortical hierarchy. Our results align with the latter hypothesis. Prediction errors in both low- and high-level visual cortex responded to high-level, but not low-level, visual surprise. This scaling with high-level surprise in early visual cortex strongly diverged from feedforward tuning. Combined, our results suggest that high-level predictions constrain sensory processing in earlier areas, thereby aiding perceptual inference.

Distributed representations of behaviour-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioural goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgements directly onto the brain. Our results reveal broadly distributed representations of behaviourally relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behaviour-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.

Human Visual Pathways for Action Recognition versus Deep Convolutional Neural Networks: Representation Correspondence in Late but Not Early Layers

Deep convolutional neural networks (DCNNs) have attained human-level performance for object categorization and exhibited representation alignment between network layers and brain regions. Does such representation alignment naturally extend to other visual tasks beyond recognizing objects in static images? In this study, we expanded the exploration to the recognition of human actions from videos and assessed the representation capabilities and alignment of two-stream DCNNs in comparison with brain regions situated along ventral and dorsal pathways. Using decoding analysis and representational similarity analysis, we show that DCNN models do not show hierarchical representation alignment to human brain across visual regions when processing action videos. Instead, later layers of DCNN models demonstrate greater representation similarities to the human visual cortex. These findings were revealed for two display formats: photorealistic avatars with full-body information and simplified stimuli in the point-light display. The discrepancies in representation alignment suggest fundamental differences in how DCNNs and the human brain represent dynamic visual information related to actions.

Representational dissimilarity component analysis (ReDisCA)

The principle of Representational Similarity Analysis (RSA) posits that neural representations reflect the structure of encoded information, allowing exploration of spatial and temporal organization of brain information processing. Traditional RSA when applied to EEG or MEG data faces challenges in accessing activation time series at the brain source level due to modeling complexities and insufficient geometric/anatomical data. To overcome this, we introduce Representational Dissimilarity Component Analysis (ReDisCA), a method for estimating spatial-temporal components in EEG or MEG responses aligned with a target representational dissimilarity matrix (RDM). ReDisCA yields informative spatial filters and associated topographies, offering insights into the location of "representationally relevant" sources. Applied to evoked response time series, ReDisCA produces temporal source activation profiles with the desired RDM. Importantly, while ReDisCA does not require inverse modeling its output is consistent with EEG and MEG observation equation and can be used as an input to rigorous source localization procedures. Demonstrating ReDisCA's efficacy through simulations and comparison with conventional methods, we show superior source localization accuracy and apply the method to real EEG and MEG datasets, revealing physiologically plausible representational structures without inverse modeling. ReDisCA adds to the family of inverse modeling free methods such as independent component analysis (Makeig, 1995), Spatial spectral decomposition (Nikulin, 2011), and Source power comodulation (Dähne, 2014) designed for extraction sources with desired properties from EEG or MEG data. Extending its utility beyond EEG and MEG analysis, ReDisCA is likely to find application in fMRI data analysis and exploration of representational structures emerging in multilayered artificial neural networks.

A large-scale examination of inductive biases shaping high-level visual representation in brains and machines

The rapid release of high-performing computer vision models offers new potential to study the impact of different inductive biases on the emergent brain alignment of learned representations. Here, we perform controlled comparisons among a curated set of 224 diverse models to test the impact of specific model properties on visual brain predictivity - a process requiring over 1.8 billion regressions and 50.3 thousand representational similarity analyses. We find that models with qualitatively different architectures (e.g. CNNs versus Transformers) and task objectives (e.g. purely visual contrastive learning versus vision- language alignment) achieve near equivalent brain predictivity, when other factors are held constant. Instead, variation across visual training diets yields the largest, most consistent effect on brain predictivity. Many models achieve similarly high brain predictivity, despite clear variation in their underlying representations - suggesting that standard methods used to link models to brains may be too flexible. Broadly, these findings challenge common assumptions about the factors underlying emergent brain alignment, and outline how we can leverage controlled model comparison to probe the common computational principles underlying biological and artificial visual systems.

The topology and geometry of neural representations

A central question for neuroscience is how to characterize brain representations of perceptual and cognitive content. An ideal characterization should distinguish different functional regions with robustness to noise and idiosyncrasies of individual brains that do not correspond to computational differences. Previous studies have characterized brain representations by their representational geometry, which is defined by the representational dissimilarity matrix (RDM), a summary statistic that abstracts from the roles of individual neurons (or responses channels) and characterizes the discriminability of stimuli. Here, we explore a further step of abstraction: from the geometry to the topology of brain representations. We propose topological representational similarity analysis, an extension of representational similarity analysis that uses a family of geotopological summary statistics that generalizes the RDM to characterize the topology while de-emphasizing the geometry. We evaluate this family of statistics in terms of the sensitivity and specificity for model selection using both simulations and functional MRI (fMRI) data. In the simulations, the ground truth is a data-generating layer representation in a neural network model and the models are the same and other layers in different model instances (trained from different random seeds). In fMRI, the ground truth is a visual area and the models are the same and other areas measured in different subjects. Results show that topology-sensitive characterizations of population codes are robust to noise and interindividual variability and maintain excellent sensitivity to the unique representational signatures of different neural network layers and brain regions.

A gradual transition toward categorical representations along the visual hierarchy during working memory, but not perception

The ability to stably maintain visual information over brief delays is central to healthy cognitive functioning, as is the ability to differentiate such internal representations from external inputs. One possible way to achieve both is via multiple concurrent mnemonic representations along the visual hierarchy that differ systematically from the representations of perceptual inputs. To test this possibility, we examine orientation representations along the visual hierarchy during perception and working memory. Human participants directly viewed, or held in mind, oriented grating patterns, and the similarity between fMRI activation patterns for different orientations was calculated throughout retinotopic cortex. During direct viewing of grating stimuli, similarity was relatively evenly distributed amongst all orientations, while during working memory the similarity was higher around oblique orientations. We modeled these differences in representational geometry based on the known distribution of orientation information in the natural world: The "veridical" model uses an efficient coding framework to capture hypothesized representations during visual perception. The "categorical" model assumes that different "psychological distances" between orientations result in orientation categorization relative to cardinal axes. During direct perception, the veridical model explained the data well. During working memory, the categorical model gradually gained explanatory power over the veridical model for increasingly anterior retinotopic regions. Thus, directly viewed images are represented veridically, but once visual information is no longer tethered to the sensory world there is a gradual progression to more categorical mnemonic formats along the visual hierarchy.

Inference to the best neuroscientific explanation

Neuroscientists routinely use reverse inference (RI) to draw conclusions about cognitive processes from neural activation data. However, despite its widespread use, the methodological status of RI is a matter of ongoing controversy, with some critics arguing that it should be rejected wholesale on the grounds that it instantiates a deductively invalid argument form. In response to these critiques, some have proposed to conceive of RI as a form of abduction or inference to the best explanation (IBE). We side with this response but at the same time argue that a defense of RI requires more than identifying it as a form of IBE. In this paper, we give an analysis of what determines the quality of an RI conceived as an IBE and on that basis argue that whether an RI is warranted needs to be decided on a case-by-case basis. Support for our argument will come from a detailed methodological discussion of RI in cognitive neuroscience in light of what the recent literature on IBE has identified as the main quality indicators for IBEs.

Quality space computations for consciousness

The quality space hypothesis about conscious experience proposes that conscious sensory states are experienced in relation to other possible sensory states. For instance, the colour red is experienced as being more like orange, and less like green or blue. Recent empirical findings suggest that subjective similarity space can be explained in terms of similarities in neural activation patterns. Here, we consider how localist, workspace, and higher-order theories of consciousness can accommodate claims about the qualitative character of experience and functionally support a quality space. We review existing empirical evidence for each of these positions, and highlight novel experimental tools, such as altering local activation spaces via brain stimulation or behavioural training, that can distinguish these accounts.

Spiking generative adversarial network with attention scoring decoding

Generative models based on neural networks present a substantial challenge within deep learning. As it stands, such models are primarily limited to the domain of artificial neural networks. Spiking neural networks, as the third generation of neural networks, offer a closer approximation to brain-like processing due to their rich spatiotemporal dynamics. However, generative models based on spiking neural networks are not well studied. Particularly, previous works on generative adversarial networks based on spiking neural networks are conducted on simple datasets and do not perform well. In this work, we pioneer constructing a spiking generative adversarial network capable of handling complex images and having higher performance. Our first task is to identify the problems of out-of-domain inconsistency and temporal inconsistency inherent in spiking generative adversarial networks. We address these issues by incorporating the Earth-Mover distance and an attention-based weighted decoding method, significantly enhancing the performance of our algorithm across several datasets. Experimental results reveal that our approach outperforms existing methods on the MNIST, FashionMNIST, CIFAR10, and CelebA. In addition to our examination of static datasets, this study marks our inaugural investigation into event-based data, through which we achieved noteworthy results. Moreover, compared with hybrid spiking generative adversarial networks, where the discriminator is an artificial analog neural network, our methodology demonstrates closer alignment with the information processing patterns observed in the mouse. Our code can be found at https://github.com/Brain-Cog-Lab/sgad.

Contrastive learning explains the emergence and function of visual category-selective regions

Modular and distributed coding theories of category selectivity along the human ventral visual stream have long existed in tension. Here, we present a reconciling framework-contrastive coding-based on a series of analyses relating category selectivity within biological and artificial neural networks. We discover that, in models trained with contrastive self-supervised objectives over a rich natural image diet, category-selective tuning naturally emerges for faces, bodies, scenes, and words. Further, lesions of these model units lead to selective, dissociable recognition deficits, highlighting their distinct functional roles in information processing. Finally, these pre-identified units can predict neural responses in all corresponding face-, scene-, body-, and word-selective regions of human visual cortex, under a highly constrained sparse positive encoding procedure. The success of this single model indicates that brain-like functional specialization can emerge without category-specific learning pressures, as the system learns to untangle rich image content. Contrastive coding, therefore, provides a unifying account of object category emergence and representation in the human brain.

A Multidimensional Neural Representation of Face Impressions

From a glimpse of a face, people form trait impressions that operate as facial stereotypes, which are largely inaccurate yet nevertheless drive social behavior. Behavioral studies have long pointed to dimensions of trustworthiness and dominance that are thought to underlie face impressions due to their evolutionarily adaptive nature. Using human neuroimaging (N = 26, 19 female, 7 male), we identify a two-dimensional representation of faces' inferred traits in the middle temporal gyrus (MTG), a region involved in domain-general conceptual processing including the activation of social concepts. The similarity of neural-response patterns for any given pair of faces in the bilateral MTG was predicted by their proximity in trustworthiness-dominance space, an effect that could not be explained by mere visual similarity. This MTG trait-space representation occurred automatically, was relatively invariant across participants, and did not depend on the explicit endorsement of face impressions (i.e., beliefs that face impressions are valid and accurate). In contrast, regions involved in high-level social reasoning (the bilateral temporoparietal junction and posterior superior temporal sulcus; TPJ-pSTS) and entity-specific social knowledge (the left anterior temporal lobe; ATL) also exhibited this trait-space representation but only among participants who explicitly endorsed forming these impressions. Together, the findings identify a two-dimensional neural representation of face impressions and suggest that multiple implicit and explicit mechanisms give rise to biases based on facial appearance. While the MTG implicitly represents a multidimensional trait space for faces, the TPJ-pSTS and ATL are involved in the explicit application of this trait space for social evaluation and behavior.

How Shape Perception Works, in Two Dimensions and Three Dimensions

The ventral visual pathway transforms retinal images into neural representations that support object understanding, including exquisite appreciation of precise 2D pattern shape and 3D volumetric shape. We articulate a framework for understanding the goals of this transformation and how they are achieved by neural coding at successive ventral pathway stages. The critical goals are (a) radical compression to make shape information communicable across axonal bundles and storable in memory, (b) explicit coding to make shape information easily readable by the rest of the brain and thus accessible for cognition and behavioral control, and (c) representational stability to maintain consistent perception across highly variable viewing conditions. We describe how each transformational step in ventral pathway vision serves one or more of these goals. This three-goal framework unifies discoveries about ventral shape processing into a neural explanation for our remarkable experience of shape as a vivid, richly detailed aspect of the natural world.

Phase-Dependent Visual and Sensorimotor Integration of Features for Grasp Computations before and after Effector Specification

The simple act of viewing and grasping an object involves complex sensorimotor control mechanisms that have been shown to vary as a function of multiple object and other task features such as object size, shape, weight, and wrist orientation. However, these features have been mostly studied in isolation. In contrast, given the nonlinearity of motor control, its computations require multiple features to be incorporated concurrently. Therefore, the present study tested the hypothesis that grasp computations integrate multiple task features superadditively in particular when these features are relevant for the same action phase. We asked male and female human participants to reach-to-grasp objects of different shapes and sizes with different wrist orientations. Also, we delayed the movement onset using auditory signals to specify which effector to use. Using electroencephalography and representative dissimilarity analysis to map the time course of cortical activity, we found that grasp computations formed superadditive integrated representations of grasp features during different planning phases of grasping. Shape-by-size representations and size-by-orientation representations occurred before and after effector specification, respectively, and could not be explained by single-feature models. These observations are consistent with the brain performing different preparatory, phase-specific computations; visual object analysis to identify grasp points at abstract visual levels; and downstream sensorimotor preparatory computations for reach-to-grasp trajectories. Our results suggest the brain adheres to the needs of nonlinear motor control for integration. Furthermore, they show that examining the superadditive influence of integrated representations can serve as a novel lens to map the computations underlying sensorimotor control.

Neural Representations of Sensory Uncertainty and Confidence Are Associated with Perceptual Curiosity

Humans are immensely curious and motivated to reduce uncertainty, but little is known about the neural mechanisms that generate curiosity. Curiosity is inversely associated with confidence, suggesting that it is triggered by states of low confidence (subjective uncertainty), but the neural mechanisms of this link, have been little investigated. Inspired by studies of sensory uncertainty, we hypothesized that visual areas provide multivariate representations of uncertainty, which are read out by higher-order structures to generate signals of confidence and, ultimately, curiosity. We scanned participants (17 female, 15 male) using fMRI while they performed a new task in which they rated their confidence in identifying distorted images of animals and objects and their curiosity to see the clear image. We measured the activity evoked by each image in the occipitotemporal cortex (OTC) and devised a new metric of "OTC Certainty" indicating the strength of evidence this activity conveys about the animal versus object categories. We show that, perceptual curiosity peaked at low confidence and OTC Certainty negatively correlated with curiosity, establishing a link between curiosity and a multivariate representation of sensory uncertainty. Moreover, univariate (average) activity in two frontal areas-vmPFC and ACC-correlated positively with confidence and negatively with curiosity, and the vmPFC mediated the relationship between OTC Certainty and curiosity. The results reveal novel mechanisms through which uncertainty about an event generates curiosity about that event.

Auditory guidance of eye movements toward threat-related images in the absence of visual awareness

The human brain is sensitive to threat-related information even when we are not aware of this information. For example, fearful faces attract gaze in the absence of visual awareness. Moreover, information in different sensory modalities interacts in the absence of awareness, for example, the detection of suppressed visual stimuli is facilitated by simultaneously presented congruent sounds or tactile stimuli. Here, we combined these two lines of research and investigated whether threat-related sounds could facilitate visual processing of threat-related images suppressed from awareness such that they attract eye gaze. We suppressed threat-related images of cars and neutral images of human hands from visual awareness using continuous flash suppression and tracked observers' eye movements while presenting congruent or incongruent sounds (finger snapping and car engine sounds). Indeed, threat-related car sounds guided the eyes toward suppressed car images, participants looked longer at the hidden car images than at any other part of the display. In contrast, neither congruent nor incongruent sounds had a significant effect on eye responses to suppressed finger images. Overall, our results suggest that only in a danger-related context semantically congruent sounds modulate eye movements to images suppressed from awareness, highlighting the prioritisation of eye responses to threat-related stimuli in the absence of visual awareness.

Pre-frontal cortex guides dimension-reducing transformations in the occipito-ventral pathway for categorization behaviors

To interpret our surroundings, the brain uses a visual categorization process. Current theories and models suggest that this process comprises a hierarchy of different computations that transforms complex, high-dimensional inputs into lower-dimensional representations (i.e., manifolds) in support of multiple categorization behaviors. Here, we tested this hypothesis by analyzing these transformations reflected in dynamic MEG source activity while individual participants actively categorized the same stimuli according to different tasks: face expression, face gender, pedestrian gender, and vehicle type. Results reveal three transformation stages guided by the pre-frontal cortex. At stage 1 (high-dimensional, 50-120 ms), occipital sources represent both task-relevant and task-irrelevant stimulus features; task-relevant features advance into higher ventral/dorsal regions, whereas task-irrelevant features halt at the occipital-temporal junction. At stage 2 (121-150 ms), stimulus feature representations reduce to lower-dimensional manifolds, which then transform into the task-relevant features underlying categorization behavior over stage 3 (161-350 ms). Our findings shed light on how the brain's network mechanisms transform high-dimensional inputs into specific feature manifolds that support multiple categorization behaviors.

Sensitivity and specificity of the action observation network to kinematics, target object, and gesture meaning

Hierarchical models have been proposed to explain how the brain encodes actions, whereby different areas represent different features, such as gesture kinematics, target object, action goal, and meaning. The visual processing of action-related information is distributed over a well-known network of brain regions spanning separate anatomical areas, attuned to specific stimulus properties, and referred to as action observation network (AON). To determine the brain organization of these features, we measured representational geometries during the observation of a large set of transitive and intransitive gestures in two independent functional magnetic resonance imaging experiments. We provided evidence for a partial dissociation between kinematics, object characteristics, and action meaning in the occipito-parietal, ventro-temporal, and lateral occipito-temporal cortex, respectively. Importantly, most of the AON showed low specificity to all the explored features, and representational spaces sharing similar information content were spread across the cortex without being anatomically adjacent. Overall, our results support the notion that the AON relies on overlapping and distributed coding and may act as a unique representational space instead of mapping features in a modular and segregated manner.

Patterns of saliency and semantic features distinguish gaze of expert and novice viewers of surveillance footage

When viewing the actions of others, we not only see patterns of body movements, but we also "see" the intentions and social relations of people. Experienced forensic examiners - Closed Circuit Television (CCTV) operators - have been shown to convey superior performance in identifying and predicting hostile intentions from surveillance footage than novices. However, it remains largely unknown what visual content CCTV operators actively attend to, and whether CCTV operators develop different strategies for active information seeking from what novices do. Here, we conducted computational analysis for the gaze-centered stimuli captured by experienced CCTV operators and novices' eye movements when viewing the same surveillance footage. Low-level image features were extracted by a visual saliency model, whereas object-level semantic features were extracted by a deep convolutional neural network (DCNN), AlexNet, from gaze-centered regions. We found that the looking behavior of CCTV operators differs from novices by actively attending to visual contents with different patterns of saliency and semantic features. Expertise in selectively utilizing informative features at different levels of visual hierarchy may play an important role in facilitating the efficient detection of social relationships between agents and the prediction of harmful intentions.

Helpless infants are learning a foundation model

Humans have a protracted postnatal helplessness period, typically attributed to human-specific maternal constraints causing an early birth when the brain is highly immature. By aligning neurodevelopmental events across species, however, it has been found that humans are not born with especially immature brains compared with animal species with a shorter helpless period. Consistent with this, the rapidly growing field of infant neuroimaging has found that brain connectivity and functional activation at birth share many similarities with the mature brain. Inspired by machine learning, where deep neural networks also benefit from a 'helpless period' of pre-training, we propose that human infants are learning a foundation model: a set of fundamental representations that underpin later cognition with high performance and rapid generalisation.

Reconciling shared versus context-specific information in a neural network model of latent causes

It has been proposed that, when processing a stream of events, humans divide their experiences in terms of inferred latent causes (LCs) to support context-dependent learning. However, when shared structure is present across contexts, it is still unclear how the "splitting" of LCs and learning of shared structure can be simultaneously achieved. Here, we present the Latent Cause Network (LCNet), a neural network model of LC inference. Through learning, it naturally stores structure that is shared across tasks in the network weights. Additionally, it represents context-specific structure using a context module, controlled by a Bayesian nonparametric inference algorithm, which assigns a unique context vector for each inferred LC. Across three simulations, we found that LCNet could (1) extract shared structure across LCs in a function learning task while avoiding catastrophic interference, (2) capture human data on curriculum effects in schema learning, and (3) infer the underlying event structure when processing naturalistic videos of daily events. Overall, these results demonstrate a computationally feasible approach to reconciling shared structure and context-specific structure in a model of LCs that is scalable from laboratory experiment settings to naturalistic settings.

Image classification and reconstruction from low-density EEG

Recent advances in visual decoding have enabled the classification and reconstruction of perceived images from the brain. However, previous approaches have predominantly relied on stationary, costly equipment like fMRI or high-density EEG, limiting the real-world availability and applicability of such projects. Additionally, several EEG-based paradigms have utilized artifactual, rather than stimulus-related information yielding flawed classification and reconstruction results. Our goal was to reduce the cost of the decoding paradigm, while increasing its flexibility. Therefore, we investigated whether the classification of an image category and the reconstruction of the image itself is possible from the visually evoked brain activity measured by a portable, 8-channel EEG. To compensate for the low electrode count and to avoid flawed predictions, we designed a theory-guided EEG setup and created a new experiment to obtain a dataset from 9 subjects. We compared five contemporary classification models with our setup reaching an average accuracy of 34.4% for 20 image classes on hold-out test recordings. For the reconstruction, the top-performing model was used as an EEG-encoder which was combined with a pretrained latent diffusion model via double-conditioning. After fine-tuning, we reconstructed images from the test set with a 1000 trial 50-class top-1 accuracy of 35.3%. While not reaching the same performance as MRI-based paradigms on unseen stimuli, our approach greatly improved the affordability and mobility of the visual decoding technology.

Distributed representations of behavior-derived object dimensions in the human visual system

Object vision is commonly thought to involve a hierarchy of brain regions processing increasingly complex image features, with high-level visual cortex supporting object recognition and categorization. However, object vision supports diverse behavioral goals, suggesting basic limitations of this category-centric framework. To address these limitations, we mapped a series of dimensions derived from a large-scale analysis of human similarity judgments directly onto the brain. Our results reveal broadly distributed representations of behaviorally-relevant information, demonstrating selectivity to a wide variety of novel dimensions while capturing known selectivities for visual features and categories. Behavior-derived dimensions were superior to categories at predicting brain responses, yielding mixed selectivity in much of visual cortex and sparse selectivity in category-selective clusters. This framework reconciles seemingly disparate findings regarding regional specialization, explaining category selectivity as a special case of sparse response profiles among representational dimensions, suggesting a more expansive view on visual processing in the human brain.