Representational geometry: integrating cognition, computation, and the brain

Unsupervised alignment in neuroscience: Introducing a toolbox for Gromov-Wasserstein optimal transport

BACKGROUND

Understanding how sensory stimuli are represented across different brains, species, and artificial neural networks is a critical topic in neuroscience. Traditional methods for comparing these representations typically rely on supervised alignment, which assumes direct correspondence between stimuli representations across brains or models. However, it has limitations when this assumption is not valid, or when validating the assumption itself is the goal of the research.

NEW METHOD

To address the limitations of supervised alignment, we propose an unsupervised alignment method based on Gromov-Wasserstein optimal transport (GWOT). GWOT optimally identifies correspondences between representations by leveraging internal relationships without external labels, revealing intricate structural correspondences such as one-to-one, group-to-group, and shifted mappings.

RESULTS

We provide a comprehensive methodological guide and introduce a toolbox called GWTune for using GWOT in neuroscience. Our results show that GWOT can reveal detailed structural distinctions that supervised methods may overlook. We also demonstrate successful unsupervised alignment in key data domains, including behavioral data, neural activity recordings, and artificial neural network models, demonstrating its flexibility and broad applicability.

COMPARISON WITH EXISTING METHODS

Unlike traditional supervised alignment methods such as Representational Similarity Analysis, which assume direct correspondence between stimuli, GWOT provides a nuanced approach that can handle different types of structural correspondence, including fine-grained and coarse correspondences. Our method would provide richer insights into the similarity or difference of representations by revealing finer structural differences.

CONCLUSION

We anticipate that our work will significantly broaden the accessibility and application of unsupervised alignment in neuroscience, offering novel perspectives on complex representational structures. By providing a user-friendly toolbox and a detailed tutorial, we aim to facilitate the adoption of unsupervised alignment techniques, enabling researchers to achieve a deeper understanding of cross-brain and cross-species representation analysis.

Unsupervised alignment reveals structural commonalities and differences in neural representations of natural scenes across individuals and brain areas

Neuroscience research aims to identify universal neural mechanisms underlying sensory information encoding by comparing neural representations across individuals, typically using Representational Similarity Analysis. However, traditional methods assume direct stimulus correspondence across individuals, limiting the exploration of other possibilities. To address this, we propose an unsupervised alignment framework based on Gromov-Wasserstein Optimal Transport, which identifies correspondences between neural representations solely from internal similarity structures, without relying on stimulus labels. Applying this method to Neuropixels recordings in mice and fMRI data in humans viewing natural scenes, we found that the neural representations in the same visual cortical areas can be well aligned across individuals in an unsupervised manner. Furthermore, alignment across different brain areas is influenced by factors beyond the visual hierarchy, with higher-order visual areas aligning well with each other, but not with lower-order areas. This unsupervised approach reveals more nuanced structural commonalities and differences in neural representations than conventional methods.

Insight predicts subsequent memory via cortical representational change and hippocampal activity

The neural mechanisms driving creative problem-solving, including representational change and its relation to memory, still remain largely unknown. We focus on the creative process of insight, wherein rapid knowledge reorganization and integration-termed representational change-yield solutions that evoke suddenness, certainty, positive emotion, and enduring memory. We posit that this process is associated with stronger shifts in activation patterns within brain regions housing solution-relevant information, including the visual cortex for visual problems, alongside regions linked to feelings of emotion, suddenness and subsequent memory. To test this, we collect participants' brain activity while they solve visual insight problems in the MRI. Our findings substantiate these hypotheses, revealing stronger representational changes in visual cortex, coupled with activations in the amygdala and hippocampus-forming an interconnected network. Importantly, representational change and hippocampal effects are positively associated with subsequent memory. This study provides evidence of an integrated insight mechanism influencing memory.

Cortical representational geometry of diverse tasks reveals subject-specific and subject-invariant cognitive structures

The variability in brain function forms the basis for our uniqueness. Prior studies indicate smaller individual differences and larger inter-subject correlation (ISC) in sensorimotor areas than in the association cortex. These studies, deriving information from brain activity, leave individual differences in cognitive structures based on task similarity relations unexplored. This study quantitatively evaluates these differences by integrating ISC, representational similarity analysis, and vertex-wise encoding models using functional magnetic resonance imaging across 25 cognitive tasks. ISC based on cognitive structures enables subject identification with 100% accuracy using at least 14 tasks. ISC is larger in the fronto-parietal association and higher-order visual cortices, suggesting subject-invariant cognitive structures in these regions. Principal component analysis reveals different cognitive structure configurations within these regions. This study provides evidence of individual variability and similarity in abstract cognitive structures.

Hippocampal-entorhinal cognitive maps and cortical motor system represent action plans and their outcomes

Efficiently interacting with the environment requires weighing and selecting among multiple alternative actions based on their associated outcomes. However, the neural mechanisms underlying these processes are still debated. We show that forming relations between arbitrary action-outcome associations involve building a cognitive map. Using an immersive virtual reality paradigm, participants learned 2D abstract motor action-outcome associations and later compared action combinations while their brain activity was monitored with fMRI. We observe a hexadirectional modulation of the activity in entorhinal cortex while participants compared different action plans. Furthermore, hippocampal activity scales with the 2D similarity between outcomes of these action plans. Conversely, the supplementary motor area represents individual actions, showing a stronger response to overlapping action plans. Crucially, the connectivity between hippocampus and supplementary motor area is modulated by the similarity between the action plans, suggesting their complementary roles in action evaluation. These findings provide evidence for the role of cognitive maps in action selection, challenging classical models of memory taxonomy and its neural bases.

Temporal dynamics analysis reveals that concurrent working memory load eliminates the Stroop effect through disrupting stimulus-response mapping

Concurrent verbal working memory task can eliminate the color-word Stroop effect. Previous research, based on specific and limited resources, suggested that the disappearance of the conflict effect was due to the memory information preempting the resources for distractors. However, it remains unclear which particular stage of Stroop conflict processing is influenced by working memory loads. In this study, electroencephalography (EEG) recordings with event-related potential (ERP) analyses, time-frequency analyses, multivariate pattern analyses (MVPAs), and representational similarity analyses (RSAs) were applied to provide an in-depth investigation of the aforementioned issue. Subjects were required to complete the single task (the classical manual color-word Stroop task) and the dual task (the Sternberg working memory task combined with the Stroop task), respectively. Behaviorally, the results indicated that the Stroop effect was eliminated in the dual-task condition. The EEG results showed that the concurrent working memory task did not modulate the P1, N450, and alpha bands. However, it modulated the sustained potential (SP), late theta (740-820 ms), and beta (920-1040 ms) power, showing no difference between congruent and incongruent trials in the dual-task condition but significant difference in the single-task condition. Importantly, the RSA results revealed that the neural activation pattern of the late theta was similar to the response interaction pattern. Together, these findings implied that the concurrent working memory task eliminated the Stroop effect through disrupting stimulus-response mapping.

Altered face perception in amnestic mild cognitive impairment: Evidence from representational similarity analysis of event-related potential

BackgroundStructural changes in medial temporal lobes including the fusiform gyrus, a critical area in face recognition, precede the progression of amnestic mild cognitive impairment (aMCI) to Alzheimer's disease (AD). However, how the neural correlates of face processing altered in aMCI, as well as their association with cognitive impairments, remain unclear.ObjectiveUsing electroencephalogram (EEG), we explored the electrophysiological markers of face-specific visual processing alterations in aMCI and examined their relationship with cognitive deficits.MethodsWe recruited participants with aMCI (n = 32) and healthy controls (HC, n = 41) and used a passive viewing task to measure the event-related potential (ERP) in response to faces and non-face objects. To compare face processing in aMCI patients and HCs, we adopted mass univariate analysis and representational similarity analysis (RSA) to explore aMCI-related alterations in ERPs.ResultsWe found that face inversion effect (FIE) in P1 amplitudes was absent in aMCI patients. Also, compared to HCs, aMCI patients exhibited a lack of right hemisphere advantage in N170 in response to faces. Furthermore, representation similarity analysis of ERP in posterior-temporal regions revealed that aMCI patients represent face and non-face objects distinctively from HCs in the early processing stage. Additionally, the FIE in P1 amplitude positively correlated to aMCI patients' visuospatial functions.ConclusionsThese findings showed aMCI-related changes in the early perceptual processing of faces and highlights the potential of the FIE in P1 amplitude and ERP patterns over occipital-temporal regions as electrophysiological markers for aMCI and AD.

Graspability in disguise: The cognitive and neural differences in processing words representing small and big objects

Size is a fundamental visual-spatial characteristic of the physical world. Previous studies have revealed distinct brain responses to small and big objects represented by pictures, implying that object size is a key dimension in organizing concrete concepts. However, it remains unknown whether the brain responses reflecting size-based categorization extend to symbolic input like words. Furthermore, several behavioral studies have indicated faster lexical decisions for words representing big objects (big words) than those representing small objects (small words). However, how this behavioral finding relates to potential neural differences in processing small and big words, as well as the underlying cognitive processes, remains unclear. Therefore, the present study investigates the cognitive and neural differences in processing small and big words. We compared the behavioral and neural responses (EEG) to small and big words using a lexical decision task (LDT) and a semantic decision task (SDT). Our results showed that in the LDT, reaction times to big words were significantly shorter than those to small words in the by-participant but not by-item analysis, suggesting a potential rather than robust processing advantage for big words. By contrast, no behavioral differences were observed in the SDT. Our EEG decoding results revealed distinct brain responses to small and big words at 190-250 msec in both tasks, with additional distinct neural responses at 390-520 msec only in the SDT. Importantly, the regression representational similarity analysis (RSA) suggested that these distinct brain responses could be explained by object graspability represented by small and big words, rather than object size. These findings illustrate the cognitive and neural differences in processing small and big words, identify graspability as the key influencing dimension, and demonstrate flexible, two-stage processing of semantic concepts. Moreover, we propose a novel hypothesis to explain the potential processing advantage for big words over small words.

Relationships and representations of brain structures, connectivity, dynamics and functions

The review explores the complex interplay between brain structures and their associated functions, presenting a diversity of hierarchical models that enhances our understanding of these relationships. Central to this approach are structure-function flow diagrams, which offer a visual representation of how specific neuroanatomical structures are linked to their functional roles. These diagrams are instrumental in mapping the intricate connections between different brain regions, providing a clearer understanding of how functions emerge from the underlying neural architecture. The study details innovative attempts to develop new functional hierarchies that integrate structural and functional data. These efforts leverage recent advancements in neuroimaging techniques such as fMRI, EEG, MEG, and PET, as well as computational models that simulate neural dynamics. By combining these approaches, the study seeks to create a more refined and dynamic hierarchy that can accommodate the brain's complexity, including its capacity for plasticity and adaptation. A significant focus is placed on the overlap of structures and functions within the brain. The manuscript acknowledges that many brain regions are multifunctional, contributing to different cognitive and behavioral processes depending on the context. This overlap highlights the need for a flexible, non-linear hierarchy that can capture the brain's intricate functional landscape. Moreover, the study examines the interdependence of these functions, emphasizing how the loss or impairment of one function can impact others. Another crucial aspect discussed is the brain's ability to compensate for functional deficits following neurological diseases or injuries. The investigation explores how the brain reorganizes itself, often through the recruitment of alternative neural pathways or the enhancement of existing ones, to maintain functionality despite structural damage. This compensatory mechanism underscores the brain's remarkable plasticity, demonstrating its ability to adapt and reconfigure itself in response to injury, thereby ensuring the continuation of essential functions. In conclusion, the study presents a system of brain functions that integrates structural, functional, and dynamic perspectives. It offers a robust framework for understanding how the brain's complex network of structures supports a wide range of cognitive and behavioral functions, with significant implications for both basic neuroscience and clinical applications.

Trial-level Representational Similarity Analysis

Neural representation refers to the brain activity that stands in for one's cognitive experience, and in cognitive neuroscience, the principal method to studying neural representations is representational similarity analysis (RSA). The classic RSA (cRSA) approach examines the overall quality of representations across numerous items by assessing the correspondence between two representational similarity matrices (RSMs): one based on a theoretical model of stimulus similarity and the other based on similarity in measured neural data. However, because cRSA cannot model representation at the level of individual trials, it is fundamentally limited in its ability to assess subject-, stimulus-, and trial-level variances that all influence representation. Here, we formally introduce trial-level RSA (tRSA), an analytical framework that estimates the strength of neural representation for singular experimental trials and evaluates hypotheses using multi-level models. First, we verified the correspondence between tRSA and cRSA in quantifying the overall representation strength across all trials. Second, we compared the statistical inferences drawn from both approaches using simulated data that reflected a wide range of scenarios. Compared to cRSA, the multi-level framework of tRSA was both more theoretically appropriate and significantly sensitive to true effects. Third, using real fMRI datasets, we further demonstrated several issues with cRSA, to which tRSA was more robust. Finally, we presented some novel findings of neural representations that could only be assessed with tRSA and not cRSA. In summary, tRSA proves to be a robust and versatile analytical approach for cognitive neuroscience and beyond.

Worry and rumination elicit similar neural representations: neuroimaging evidence for repetitive negative thinking

Repetitive negative thinking (RNT) captures shared cognitive and emotional features of content-specific cognition, including future-focused worry and past-focused rumination. The degree to which these distinct but related processes recruit overlapping neural structures is undetermined, because most neuroscientific studies only examine worry or rumination in isolation. To address this, we developed a paradigm to elicit idiographic worries and ruminations during an fMRI scan in 39 young adults with a range of trait RNT scores. We measured concurrent emotion ratings and heart rate as a physiological metric of arousal. Multivariate representational similarity analysis revealed that regions distributed across default mode, salience, and frontoparietal control networks encode worry and rumination similarly. Moreover, heart rate did not differ between worry and rumination. Capturing the shared neural features between worry and rumination throughout networks supporting self-referential processing, memory, salience detection, and cognitive control provides novel empirical evidence to bolster cognitive and clinical models of RNT.

Multi-voxel pattern analysis for developmental cognitive neuroscientists

The current prevailing approaches to analyzing task fMRI data in developmental cognitive neuroscience are brain connectivity and mass univariate task-based analyses, used either in isolation or as part of a broader analytic framework (e.g., BWAS). While these are powerful tools, it is somewhat surprising that multi-voxel pattern analysis (MVPA) is not more common in developmental cognitive neuroscience given its enhanced ability to both probe neural population codes and greater sensitivity relative to the mass univariate approach. Omitting MVPA methods might represent a missed opportunity to leverage a suite of tools that are uniquely poised to reveal mechanisms underlying brain development. The goal of this review is to spur awareness and adoption of MVPA in developmental cognitive neuroscience by providing a practical introduction to foundational MVPA concepts. We begin by defining MVPA and explain why examining multi-voxel patterns of brain activity can aid in understanding the developing human brain. We then survey four different types of MVPA: Decoding, representational similarity analysis (RSA), pattern expression, and voxel-wise encoding models. Each variant of MVPA is presented with a conceptual overview of the method followed by practical considerations and subvariants thereof. We go on to highlight the types of developmental questions that can be answered by MPVA, discuss practical matters in MVPA implementation germane to developmental cognitive neuroscientists, and make recommendations for integrating MVPA with the existing analytic ecosystem in the field.

Individual differences do not mask effects of unconscious processing

A wave of criticisms and replication failures is currently challenging claims about the scope of unconscious perception and cognition. Such failures to find unconscious processing effects at the population level may reflect the absence of individual-level effects, or alternatively, the averaging out of individual-level effects with opposing signs. Importantly, only the first suggests that consciousness may be necessary for the tested process to take place. To arbitrate between these two possibilities, we tested previously collected data where unconscious processing effects were not found (26 effects from 470 participants), using five frequentist and Bayesian tests that are robust to individual differences in effect signs. By and large, we found no reliable evidence for unconscious effects being masked by individual differences. In contrast, when we examined 136 non-significant effects from other domains, two novel non-parametric tests did reveal effects that were hidden by opposing individual results, though as we show, some of them might be driven by design-related factors. Taken together, five analysis approaches provide strong evidence for the restricted nature of unconscious processing effects not only across participants, but also across different trials within individuals. We provide analysis code and best-practice recommendations for testing for non-directional effects.

Is my "red" your "red"?: Evaluating structural correspondences between color similarity judgments using unsupervised alignment

Whether one person's subjective experience of the "redness" of red is equivalent to another's is a fundamental question in consciousness studies. Intersubjective comparison of the relational structures of sensory experiences, termed "qualia structures", can constrain the question. We propose an unsupervised alignment method, based on optimal transport, to find the optimal mapping between the similarity structures of sensory experiences without presupposing correspondences (such as "red-to-red"). After collecting subjective similarity judgments for 93 colors, we showed that the similarity structures derived from color-neurotypical participants can be "correctly" aligned at the group level. In contrast, those of color-blind participants could not be aligned with color-neurotypical participants. Our results provide quantitative evidence for interindividual structural equivalence or difference of color qualia, implying that color-neurotypical people's "red" is relationally equivalent to other color-neurotypical's "red", but not to color-blind people's "red". This method is applicable across modalities, enabling general structural exploration of subjective experiences.

The conceptual structure of human relationships across modern and historical cultures

A defining characteristic of social complexity in Homo sapiens is the diversity of our relationships. We build connections of various types in our families, workplaces, neighbourhoods and online communities. How do we make sense of such complex systems of human relationships? The basic organization of relationships has long been studied in the social sciences, but no consensus has been reached. Here, by using online surveys, laboratory cognitive tasks and natural language processing in diverse modern cultures across the world (n = 20,427) and ancient cultures spanning 3,000 years of history, we examined universality and cultural variability in the ways that people conceptualize relationships. We discovered a universal representational space for relationship concepts, comprising five principal dimensions (formality, activeness, valence, exchange and equality) and three core categories (hostile, public and private relationships). Our work reveals the fundamental cognitive constructs and cultural principles of human relationship knowledge and advances our understanding of human sociality.

Human infant EEG recordings for 200 object images presented in rapid visual streams

Understanding the neural basis of human object recognition and semantic knowledge has been a significant area of exploration, with recent focus aiming to reveal the developmental trajectory of this core brain function. At present, however, there is limited access to high-quality neuroimaging data obtained from human infants. Addressing this gap, we present a dataset comprising electroencephalography responses from 42 human infants obtained in response to visual presentations of various objects. Leveraging a rapid serial visual presentation paradigm, 42 infants between 2 and 12 months of age viewed 200 images spanning 50 distinct objects, with as many repetitions as possible tailored to individual infants' comfort. Our technical validation demonstrates discernible neural responses, affirming the dataset's robustness and utility for exploring the neural underpinnings of visual object recognition in infancy. Building upon insights gained from adult studies, our findings suggest that fast presentation paradigms hold promise for efficiently capturing electrophysiological responses to a large array of visual stimuli in human infants. This dataset represents a valuable resource for advancing our understanding of the developmental trajectory of object recognition and semantic knowledge in the early stages of human life.

Decoding the rhythmic representation and communication of visual contents

Rhythmic neural activity is considered essential for adaptively modulating responses in the visual system. In this opinion article we posit that visual brain rhythms also serve a key function in the representation and communication of visual contents. Collating a set of recent studies that used multivariate decoding methods on rhythmic brain signals, we highlight such rhythmic content representations in visual perception, imagery, and prediction. We argue that characterizing representations across frequency bands allows researchers to elegantly disentangle content transfer in feedforward and feedback directions. We further propose that alpha dynamics are central to content-specific feedback propagation in the visual system. We conclude that considering rhythmic content codes is pivotal for understanding information coding in vision and beyond.

Neural Representation Precision of Distance Predicts Children's Arithmetic Performance

Focusing on the distance between magnitudes as the starting point to investigate the mechanism of relation detection and its contribution to mathematical thinking, this study explores the precision of neural representations of numerical distance and their impact on children's arithmetic performance. By employing neural decoding techniques and representational similarity analysis, the present study investigates how accurately the brain represents numerical distances and how this precision relates to arithmetic skills. Twenty-nine school-aged children participated, completing a dot number comparison task during fMRI scanning and an arithmetic fluency test. Results indicated that neural activation patterns in the intra-parietal sulcus decoded the distance between the presented pair of dots, and higher precision in neural distance representation correlates with better arithmetic performance. These findings suggest that the accuracy of neural decoding can serve as an index of neural representation precision and that the ability to precisely encode numerical distances in the brain is a key factor in mathematical abilities. This provides new insights into the neural basis of mathematical cognition and learning.

Cortical Encoding of Spatial Structure and Semantic Content in 3D Natural Scenes

Our visual system enables us to effortlessly navigate and recognize real-world visual environments. Functional magnetic resonance imaging (fMRI) studies suggest a network of scene-responsive cortical visual areas, but much less is known about the temporal order in which different scene properties are analyzed by the human visual system. In this study, we selected a set of 36 full-color natural scenes that varied in spatial structure and semantic content that our male and female human participants viewed both in 2D and 3D while we recorded magnetoencephalography (MEG) data. MEG enables tracking of cortical activity in humans at millisecond timescale. We compared the representational geometry in the MEG responses with predictions based on the scene stimuli using the representational similarity analysis framework. The representational structure first reflected the spatial structure in the scenes in time window 90-125 ms, followed by the semantic content in time window 140-175 ms after stimulus onset. The 3D stereoscopic viewing of the scenes affected the responses relatively late, from ∼140 ms from stimulus onset. Taken together, our results indicate that the human visual system rapidly encodes a scene's spatial structure and suggest that this information is based on monocular instead of binocular depth cues.

Ordinal Characterization of Similarity Judgments

Characterizing judgments of similarity within a perceptual or semantic domain, and making inferences about the underlying structure of this domain from these judgments, has an increasingly important role in cognitive and systems neuroscience. We present a new framework for this purpose that makes limited assumptions about how perceptual distances are converted into similarity judgments. The approach starts from a dataset of empirical judgments of relative similarities: the fraction of times that a subject chooses one of two comparison stimuli to be more similar to a reference stimulus. These empirical judgments provide Bayesian estimates of underling choice probabilities. From these estimates, we derive indices that characterize the set of judgments in three ways: compatibility with a symmetric dis-similarity, compatibility with an ultrametric space, and compatibility with an additive tree. Each of the indices is derived from rank-order relationships among the choice probabilities that, as we show, are necessary and sufficient for local consistency with the three respective characteristics. We illustrate this approach with simulations and example psychophysical datasets of dis-similarity judgments in several visual domains and provide code that implements the analyses at https://github.com/jvlab/simrank.

Identifying representational structure in CA1 to benchmark theoretical models of cognitive mapping

Decades of theoretical and empirical work have suggested the hippocampus instantiates some form of a cognitive map. Yet, tests of competing theories have been limited in scope and largely qualitative in nature. Here, we develop a novel framework to benchmark model predictions against observed neuronal population dynamics as animals navigate a series of geometrically distinct environments. In this task space, we show a representational structure in the dynamics of hippocampal remapping that generalizes across brains, discriminates between competing theoretical models, and effectively constrains biologically viable model parameters. With this approach, we find that accurate models capture the correspondence in spatial coding of a changing environment. The present dataset and framework thus serve to empirically evaluate and advance theories of cognitive mapping in the brain.

Measuring self-similarity in empirical signals to understand musical beat perception

Experiencing music often entails the perception of a periodic beat. Despite being a widespread phenomenon across cultures, the nature and neural underpinnings of beat perception remain largely unknown. In the last decade, there has been a growing interest in developing methods to probe these processes, particularly to measure the extent to which beat-related information is contained in behavioral and neural responses. Here, we propose a theoretical framework and practical implementation of an analytic approach to capture beat-related periodicity in empirical signals using frequency-tagging. We highlight its sensitivity in measuring the extent to which the periodicity of a perceived beat is represented in a range of continuous time-varying signals with minimal assumptions. We also discuss a limitation of this approach with respect to its specificity when restricted to measuring beat-related periodicity only from the magnitude spectrum of a signal and introduce a novel extension of the approach based on autocorrelation to overcome this issue. We test the new autocorrelation-based method using simulated signals and by re-analyzing previously published data and show how it can be used to process measurements of brain activity as captured with surface EEG in adults and infants in response to rhythmic inputs. Taken together, the theoretical framework and related methodological advances confirm and elaborate the frequency-tagging approach as a promising window into the processes underlying beat perception and, more generally, temporally coordinated behaviors.

Greater neural pattern similarity to the native language is associated with better novel word learning

Introduction

Previous neuroimaging studies on bilingualism revealed that individuals tend to apply their native-language (L1) neural strategies to second language (L2) learning and processing. Nevertheless, it is still unclear how the utilization of the L1 neural strategies affects visual word learning in a new language.

Methods

To address this question, the present study scanned native Chinese speakers while performing implicit reading tasks before 9-day form-meaning learning in Experiment 1 and before 12-day comprehensive word learning in Experiment 2. To quantify the application of the L1 neural strategies in novel word learning, representational similarity analysis (RSA) was used to compute the neural pattern similarity (PS) between the L1 and artificial language (i.e., cross-language PS) before training.

Results

Univariate analysis revealed that reading both Chinese words (CWs) and artificial language words (ALWs) elicited activations in a typical reading network. More importantly, RSA revealed that greater pre-training cross-language PS in the left fusiform gyrus was associated with higher learning rate.

Discussion

These findings directly reveal the facilitating role of the L1 neural strategies in novel word learning and further extend the assimilation hypothesis from the utilization of the L1 neural network in L2 learning to its learning outcomes.

Parallel development of object recognition in newborn chicks and deep neural networks

How do newborns learn to see? We propose that visual systems are space-time fitters, meaning visual development can be understood as a blind fitting process (akin to evolution) in which visual systems gradually adapt to the spatiotemporal data distributions in the newborn's environment. To test whether space-time fitting is a viable theory for learning how to see, we performed parallel controlled-rearing experiments on newborn chicks and deep neural networks (DNNs), including CNNs and transformers. First, we raised newborn chicks in impoverished environments containing a single object, then simulated those environments in a video game engine. Second, we recorded first-person images from agents moving through the virtual animal chambers and used those images to train DNNs. Third, we compared the viewpoint-invariant object recognition performance of the chicks and DNNs. When DNNs received the same visual diet (training data) as chicks, the models developed common object recognition skills as chicks. DNNs that used time as a teaching signal-space-time fitters-also showed common patterns of successes and failures across the test viewpoints as chicks. Thus, DNNs can learn object recognition in the same impoverished environments as newborn animals. We argue that space-time fitters can serve as formal scientific models of newborn visual systems, providing image-computable models for studying how newborns learn to see from raw visual experiences.

Optimizing magnetometers arrays and analysis pipelines for multivariate pattern analysis

BACKGROUND

Multivariate pattern analysis (MVPA) has proven an excellent tool in cognitive neuroscience. It also holds a strong promise when applied to optically-pumped magnetometer-based magnetoencephalography.

NEW METHOD

To optimize OPM-MEG systems for MVPA experiments this study examines data from a conventional MEG magnetometer array, focusing on appropriate noise reduction techniques for magnetometers. We determined the least required number of sensors needed for robust MVPA for image categorization experiments.

RESULTS

We found that the use of signal space separation (SSS) without a proper regularization significantly lowered the classification accuracy considering a sub-array of 102 magnetometers or a sub-array of 204 gradiometers. We also found that classification accuracy did not improve when going beyond 30 sensors irrespective of whether SSS has been applied.

COMPARISON WITH EXISTING METHODS

The power spectra of data filtered with SSS has a substantially higher noise floor that data cleaned with SSP or HFC. Consequently, MVPA decoding results obtained from the SSS-filtered data are significantly lower compared to all other methods employed.

CONCLUSIONS

When designing MEG system based on SQUID magnetometers optimized for multivariate analysis for image categorization experiments, about 30 magnetometers are sufficient. We advise against applying SSS filters without a proper regularization to data from MEG and OPM systems prior to performing MVPA as this method, albeit reducing low-frequency external noise contributions, also introduces an increase in broadband noise. We recommend employing noise reduction techniques that either decrease or maintain the noise floor of the data like signal-space projection, homogeneous field correction and gradient noise reduction.

Neural representation of phonological wordform in temporal cortex

While the neural bases of the earliest stages of speech categorization have been widely explored using neural decoding methods, there is still a lack of consensus on questions as basic as how wordforms are represented and in what way this word-level representation influences downstream processing in the brain. Isolating and localizing the neural representations of wordform is challenging because spoken words activate a variety of representations (e.g., segmental, semantic, articulatory) in addition to form-based representations. We addressed these challenges through a novel integrated neural decoding and effective connectivity design using region of interest (ROI)-based, source-reconstructed magnetoencephalography/electroencephalography (MEG/EEG) data collected during a lexical decision task. To identify wordform representations, we trained classifiers on words and nonwords from different phonological neighborhoods and then tested the classifiers' ability to discriminate between untrained target words that overlapped phonologically with the trained items. Training with word neighbors supported significantly better decoding than training with nonword neighbors in the period immediately following target presentation. Decoding regions included mostly right hemisphere regions in the posterior temporal lobe implicated in phonetic and lexical representation. Additionally, neighbors that aligned with target word beginnings (critical for word recognition) supported decoding, but equivalent phonological overlap with word codas did not, suggesting lexical mediation. Effective connectivity analyses showed a rich pattern of interaction between ROIs that support decoding based on training with lexical neighbors, especially driven by right posterior middle temporal gyrus. Collectively, these results evidence functional representation of wordforms in temporal lobes isolated from phonemic or semantic representations.

Invariant representations in abstract concept grounding - the physical world in grounded cognition

Grounded cognition states that mental representations of concepts consist of experiential aspects. For example, the concept "cup" consists of the sensorimotor experiences from interactions with cups. Typical modalities in which concepts are grounded are: The sensorimotor system (including interoception), emotion, action, language, and social aspects. Here, we argue that this list should be expanded to include physical invariants (unchanging features of physical motion; e.g., gravity, momentum, friction). Research on physical reasoning consistently demonstrates that physical invariants are represented as fundamentally as other grounding substrates, and therefore should qualify. We assess several theories of concept representation (simulation, conceptual metaphor, conceptual spaces, predictive processing) and their positions on physical invariants. We find that the classic grounded cognition theories, simulation and conceptual metaphor theory, have not considered physical invariants, while conceptual spaces and predictive processing have. We conclude that physical invariants should be included into grounded cognition theories, and that the core mechanisms of simulation and conceptual metaphor theory are well suited to do this. Furthermore, conceptual spaces and predictive processing are very promising and should also be integrated with grounded cognition in the future.

Behaviorally-relevant features of observed actions dominate cortical representational geometry in natural vision

We effortlessly extract behaviorally relevant information from dynamic visual input in order to understand the actions of others. In the current study, we develop and test a number of models to better understand the neural representational geometries supporting action understanding. Using fMRI, we measured brain activity as participants viewed a diverse set of 90 different video clips depicting social and nonsocial actions in real-world contexts. We developed five behavioral models using arrangement tasks: two models reflecting behavioral judgments of the purpose (transitivity) and the social content (sociality) of the actions depicted in the video stimuli; and three models reflecting behavioral judgments of the visual content (people, objects, and scene) depicted in still frames of the stimuli. We evaluated how well these models predict neural representational geometry and tested them against semantic models based on verb and nonverb embeddings and visual models based on gaze and motion energy. Our results revealed that behavioral judgments of similarity better reflect neural representational geometry than semantic or visual models throughout much of cortex. The sociality and transitivity models in particular captured a large portion of unique variance throughout the action observation network, extending into regions not typically associated with action perception, like ventral temporal cortex. Overall, our findings expand the action observation network and indicate that the social content and purpose of observed actions are predominant in cortical representation.

Proximity within adolescent peer networks predicts neural similarity during affective experience

Individuals befriend others who are similar to them. One important source of similarity in relationships is similarity in felt emotion. In the present study, we used novel methods to assess whether greater similarity in the multivoxel brain representation of affective stimuli was associated with adolescents' proximity within real-world school-based social networks. We examined dyad-level neural similarity within a set of brain regions associated with the representation of affect including the ventromedial prefrontal cortex (vmPFC), amygdala, insula, and temporal pole. Greater proximity was associated with greater vmPFC neural similarity during pleasant and neutral experiences. Moreover, we used unsupervised clustering on social networks to identify groups of friends and observed that individuals from the same (versus different) friend groups were more likely to have greater vmPFC neural similarity during pleasant and negative experiences. These findings suggest that similarity in the multivoxel brain representation of affect may play an important role in adolescent friendships.

Representational dissimilarity component analysis (ReDisCA)

The principle of Representational Similarity Analysis (RSA) posits that neural representations reflect the structure of encoded information, allowing exploration of spatial and temporal organization of brain information processing. Traditional RSA when applied to EEG or MEG data faces challenges in accessing activation time series at the brain source level due to modeling complexities and insufficient geometric/anatomical data. To overcome this, we introduce Representational Dissimilarity Component Analysis (ReDisCA), a method for estimating spatial-temporal components in EEG or MEG responses aligned with a target representational dissimilarity matrix (RDM). ReDisCA yields informative spatial filters and associated topographies, offering insights into the location of "representationally relevant" sources. Applied to evoked response time series, ReDisCA produces temporal source activation profiles with the desired RDM. Importantly, while ReDisCA does not require inverse modeling its output is consistent with EEG and MEG observation equation and can be used as an input to rigorous source localization procedures. Demonstrating ReDisCA's efficacy through simulations and comparison with conventional methods, we show superior source localization accuracy and apply the method to real EEG and MEG datasets, revealing physiologically plausible representational structures without inverse modeling. ReDisCA adds to the family of inverse modeling free methods such as independent component analysis (Makeig, 1995), Spatial spectral decomposition (Nikulin, 2011), and Source power comodulation (Dähne, 2014) designed for extraction sources with desired properties from EEG or MEG data. Extending its utility beyond EEG and MEG analysis, ReDisCA is likely to find application in fMRI data analysis and exploration of representational structures emerging in multilayered artificial neural networks.

The human posterior parietal cortices orthogonalize the representation of different streams of information concurrently coded in visual working memory

The key to adaptive visual processing lies in the ability to maintain goal-directed visual representation in the face of distraction. In visual working memory (VWM), distraction may come from the coding of distractors or other concurrently retained targets. This fMRI study reveals a common representational geometry that our brain uses to combat both types of distractions in VWM. Specifically, using fMRI pattern decoding, the human posterior parietal cortex is shown to orthogonalize the representations of different streams of information concurrently coded in VWM, whether they are targets and distractors, or different targets concurrently held in VWM. The latter is also seen in the human occipitotemporal cortex. Such a representational geometry provides an elegant and simple solution to enable independent information readout, effectively combating distraction from the different streams of information, while accommodating their concurrent representations. This representational scheme differs from mechanisms that actively suppress or block the encoding of distractors to reduce interference. It is likely a general neural representational principle that supports our ability to represent information beyond VWM in other situations where multiple streams of visual information are tracked and processed simultaneously.

Language proficiency is associated with neural representational dimensionality of semantic concepts

Previous studies suggest that semantic concepts are characterized by high-dimensional neural representations and that language proficiency affects semantic processing. However, it is not clear whether language proficiency modulates the dimensional representations of semantic concepts at the neural level. To address this question, the present study adopted principal component analysis (PCA) and representational similarity analysis (RSA) to examine the differences in representational dimensionalities (RDs) and in semantic representations between words in highly proficient (Chinese) and less proficient (English) language. PCA results revealed that language proficiency increased the dimensions of lexical representations in the left inferior frontal gyrus, temporal pole, inferior temporal gyrus, supramarginal gyrus, angular gyrus, and fusiform gyrus. RSA results further showed that these regions represented semantic information and that higher semantic representations were observed in highly proficient language relative to less proficient language. These results suggest that language proficiency is associated with the neural representational dimensionality of semantic concepts.

Touch to text: Spatiotemporal evolution of braille letter representations in blind readers

Visual deprivation does not silence the visual cortex, which is responsive to auditory, tactile, and other nonvisual tasks in blind persons. However, the underlying functional dynamics of the neural networks mediating such crossmodal responses remain unclear. Here, using braille reading as a model framework to investigate these networks, we presented sighted (N=13) and blind (N=12) readers with individual visual print and tactile braille alphabetic letters, respectively, during MEG recording. Using time-resolved multivariate pattern analysis and representational similarity analysis, we traced the alphabetic letter processing cascade in both groups of participants. We found that letter representations unfolded more slowly in blind than in sighted brains, with decoding peak latencies ~200 ms later in braille readers. Focusing on the blind group, we found that the format of neural letter representations transformed within the first 500 ms after stimulus onset from a low-level structure consistent with peripheral nerve afferent coding to high-level format reflecting pairwise letter embeddings in a text corpus. The spatiotemporal dynamics of the transformation suggest that the processing cascade proceeds from a starting point in somatosensory cortex to early visual cortex and then to inferotemporal cortex. Together our results give insight into the neural mechanisms underlying braille reading in blind persons and the dynamics of functional reorganization in sensory deprivation.

The representational geometry for naturalistic textures in macaque V1 and V2

Our understanding of visual cortical processing has relied primarily on studying the selectivity of individual neurons in different areas. A complementary approach is to study how the representational geometry of neuronal populations differs across areas. Though the geometry is derived from individual neuronal selectivity, it can reveal encoding strategies difficult to infer from single neuron responses. In addition, recent theoretical work has begun to relate distinct functional objectives to different representational geometries. To understand how the representational geometry changes across stages of processing, we measured neuronal population responses in primary visual cortex (V1) and area V2 of macaque monkeys to an ensemble of synthetic, naturalistic textures. Responses were lower dimensional in V2 than V1, and there was a better alignment of V2 population responses to different textures. The representational geometry in V2 afforded better discriminability between out-of-sample textures. We performed complementary analyses of standard convolutional network models, which did not replicate the representational geometry of cortex. We conclude that there is a shift in the representational geometry between V1 and V2, with the V2 representation exhibiting features of a low-dimensional, systematic encoding of different textures and of different instantiations of each texture. Our results suggest that comparisons of representational geometry can reveal important transformations that occur across successive stages of visual processing.

The topology and geometry of neural representations

A central question for neuroscience is how to characterize brain representations of perceptual and cognitive content. An ideal characterization should distinguish different functional regions with robustness to noise and idiosyncrasies of individual brains that do not correspond to computational differences. Previous studies have characterized brain representations by their representational geometry, which is defined by the representational dissimilarity matrix (RDM), a summary statistic that abstracts from the roles of individual neurons (or responses channels) and characterizes the discriminability of stimuli. Here, we explore a further step of abstraction: from the geometry to the topology of brain representations. We propose topological representational similarity analysis, an extension of representational similarity analysis that uses a family of geotopological summary statistics that generalizes the RDM to characterize the topology while de-emphasizing the geometry. We evaluate this family of statistics in terms of the sensitivity and specificity for model selection using both simulations and functional MRI (fMRI) data. In the simulations, the ground truth is a data-generating layer representation in a neural network model and the models are the same and other layers in different model instances (trained from different random seeds). In fMRI, the ground truth is a visual area and the models are the same and other areas measured in different subjects. Results show that topology-sensitive characterizations of population codes are robust to noise and interindividual variability and maintain excellent sensitivity to the unique representational signatures of different neural network layers and brain regions.

Inference to the best neuroscientific explanation

Neuroscientists routinely use reverse inference (RI) to draw conclusions about cognitive processes from neural activation data. However, despite its widespread use, the methodological status of RI is a matter of ongoing controversy, with some critics arguing that it should be rejected wholesale on the grounds that it instantiates a deductively invalid argument form. In response to these critiques, some have proposed to conceive of RI as a form of abduction or inference to the best explanation (IBE). We side with this response but at the same time argue that a defense of RI requires more than identifying it as a form of IBE. In this paper, we give an analysis of what determines the quality of an RI conceived as an IBE and on that basis argue that whether an RI is warranted needs to be decided on a case-by-case basis. Support for our argument will come from a detailed methodological discussion of RI in cognitive neuroscience in light of what the recent literature on IBE has identified as the main quality indicators for IBEs.

Quality space computations for consciousness

The quality space hypothesis about conscious experience proposes that conscious sensory states are experienced in relation to other possible sensory states. For instance, the colour red is experienced as being more like orange, and less like green or blue. Recent empirical findings suggest that subjective similarity space can be explained in terms of similarities in neural activation patterns. Here, we consider how localist, workspace, and higher-order theories of consciousness can accommodate claims about the qualitative character of experience and functionally support a quality space. We review existing empirical evidence for each of these positions, and highlight novel experimental tools, such as altering local activation spaces via brain stimulation or behavioural training, that can distinguish these accounts.

Differential Mnemonic Contributions of Cortical Representations during Encoding and Retrieval

Several recent fMRI studies of episodic and working memory representations converge on the finding that visual information is most strongly represented in occipito-temporal cortex during the encoding phase but in parietal regions during the retrieval phase. It has been suggested that this location shift reflects a change in the content of representations, from predominantly visual during encoding to primarily semantic during retrieval. Yet, direct evidence on the nature of encoding and retrieval representations is lacking. It is also unclear how the representations mediating the encoding-retrieval shift contribute to memory performance. To investigate these two issues, in the current fMRI study, participants encoded pictures (e.g., picture of a cardinal) and later performed a word recognition test (e.g., word "cardinal"). Representational similarity analyses examined how visual (e.g., red color) and semantic representations (e.g., what cardinals eat) support successful encoding and retrieval. These analyses revealed two novel findings. First, successful memory was associated with representational changes in cortical location (from occipito-temporal at encoding to parietal at retrieval) but not with changes in representational content (visual vs. semantic). Thus, the representational encoding-retrieval shift cannot be easily attributed to a change in the nature of representations. Second, in parietal regions, stronger representations predicted encoding failure but retrieval success. This encoding-retrieval "flip" in representations mimics the one previously reported in univariate activation studies. In summary, by answering important questions regarding the content and contributions to the performance of the representations mediating the encoding-retrieval shift, our findings clarify the neural mechanisms of this intriguing phenomenon.

Navigating Memorability Landscapes: Hyperbolic Geometry Reveals Hierarchical Structures in Object Concept Memory

Why are some object concepts (e.g., birds, cars, vegetables, etc.) more memorable than others? Prior studies have suggested that features (e.g., color, animacy, etc.) and typicality (e.g., robin vs. penguin) of object images influences the likelihood of being remembered. However, a complete understanding of object memorability remains elusive. In this study, we examine whether the geometric relationship between object concepts explains differences in their memorability. Specifically, we hypothesize that image concepts will be geometrically arranged in hierarchical structures and that memorability will be explained by a concept's depth in these hierarchical trees. To test this hypothesis, we construct a Hyperbolic representation space of object concepts (N=1,854) from the THINGS database (Hebart et al., 2019), which consists of naturalistic images of concrete objects, and a space of 49 feature dimensions derived from data-driven models. Using ALBATROSS (Stier, A. J., Giusti, C., & Berman, M. G., In prep), a stochastic topological data analysis technique that detects underlying structures of data, we demonstrate that Hyperbolic geometry efficiently captures the hierarchical organization of object concepts above and beyond a traditional Euclidean geometry and that hierarchical organization is related to memorability. We find that concepts closer to the center of the representational space are more prototypical and also more memorable. Importantly, Hyperbolic distances are more predictive of memorability and prototypicality than Euclidean distances, suggesting that concept memorability and typicality are organized hierarchically. Taken together, our work presents a novel hierarchical representational structure of object concepts that explains memorability and typicality.

A geometrical solution underlies general neural principle for serial ordering

A general mathematical description of how the brain sequentially encodes knowledge remains elusive. We propose a linear solution for serial learning tasks, based on the concept of mixed selectivity in high-dimensional neural state spaces. In our framework, neural representations of items in a sequence are projected along a "geometric" mental line learned through classical conditioning. The model successfully solves serial position tasks and explains behaviors observed in humans and animals during transitive inference tasks amidst noisy sensory input and stochastic neural activity. This approach extends to recurrent neural networks performing motor decision tasks, where the same geometric mental line correlates with motor plans and modulates network activity according to the symbolic distance between items. Serial ordering is thus predicted to emerge as a monotonic mapping between sensory input and behavioral output, highlighting a possible pivotal role for motor-related associative cortices in transitive inference tasks.

Practice Reshapes the Geometry and Dynamics of Task-tailored Representations

Extensive practice makes task performance more efficient and precise, leading to automaticity. However, theories of automaticity differ on which levels of task representations (e.g., low-level features, stimulus-response mappings, or high-level conjunctive memories of individual events) change with practice, despite predicting the same pattern of improvement (e.g., power law of practice). To resolve this controversy, we built on recent theoretical advances in understanding computations through neural population dynamics. Specifically, we hypothesized that practice optimizes the neural representational geometry of task representations to minimally separate the highest-level task contingencies needed for successful performance. This involves efficiently reaching conjunctive neural states that integrate task-critical features nonlinearly while abstracting over non-critical dimensions. To test this hypothesis, human participants (n = 40) engaged in extensive practice of a simple, context-dependent action selection task over 3 days while recording EEG. During initial rapid improvement in task performance, representations of the highest-level, context-specific conjunctions of task-features were enhanced as a function of the number of successful episodes. Crucially, only enhancement of these conjunctive representations, and not lower-order representations, predicted the power-law improvement in performance. Simultaneously, over sessions, these conjunctive neural states became more stable earlier in time and more aligned, abstracting over redundant task features, which correlated with offline performance gain in reducing switch costs. Thus, practice optimizes the dynamic representational geometry as task-tailored neural states that minimally tesselate the task space, taming their high-dimensionality.

Network state changes in sensory thalamus represent learned outcomes

Thalamic brain areas play an important role in adaptive behaviors. Nevertheless, the population dynamics of thalamic relays during learning across sensory modalities remain unknown. Using a cross-modal sensory reward-associative learning paradigm combined with deep brain two-photon calcium imaging of large populations of auditory thalamus (medial geniculate body, MGB) neurons in male mice, we identified that MGB neurons are biased towards reward predictors independent of modality. Additionally, functional classes of MGB neurons aligned with distinct task periods and behavioral outcomes, both dependent and independent of sensory modality. During non-sensory delay periods, MGB ensembles developed coherent neuronal representation as well as distinct co-activity network states reflecting predicted task outcome. These results demonstrate flexible cross-modal ensemble coding in auditory thalamus during adaptive learning and highlight its importance in brain-wide cross-modal computations during complex behavior.

Functional alterations of lateral temporal cortex for processing voice prosody in adults with autism spectrum disorder

The human auditory system includes discrete cortical patches and selective regions for processing voice information, including emotional prosody. Although behavioral evidence indicates individuals with autism spectrum disorder (ASD) have difficulties in recognizing emotional prosody, it remains understudied whether and how localized voice patches (VPs) and other voice-sensitive regions are functionally altered in processing prosody. This fMRI study investigated neural responses to prosodic voices in 25 adult males with ASD and 33 controls using voices of anger, sadness, and happiness with varying degrees of emotion. We used a functional region-of-interest analysis with an independent voice localizer to identify multiple VPs from combined ASD and control data. We observed a general response reduction to prosodic voices in specific VPs of left posterior temporal VP (TVP) and right middle TVP. Reduced cortical responses in right middle TVP were consistently correlated with the severity of autistic symptoms for all examined emotional prosodies. Moreover, representation similarity analysis revealed the reduced effect of emotional intensity in multivoxel activation patterns in left anterior superior temporal cortex only for sad prosody. These results indicate reduced response magnitudes to voice prosodies in specific TVPs and altered emotion intensity-dependent multivoxel activation patterns in adult ASDs, potentially underlying their socio-communicative difficulties.

A Randomization-Based, Model-Free Approach to Functional Neuroimaging: A Proof of Concept

Functional neuroimaging analysis takes noisy multidimensional measurements as input and produces statistical inferences regarding the functional properties of brain regions as output. Such inferences are most commonly model-based, in that they assume a model of how neural activity translates to the measured signal (blood oxygenation level-dependent signal in the case of functional MRI). The use of models increases statistical sensitivity and makes it possible to ask fine-grained theoretical questions. However, this comes at the cost of making theoretical assumptions about the underlying data-generating process. An advantage of model-free approaches is that they can be used in cases where model assumptions are known not to hold. To this end, we introduce a randomization-based, model-free approach to functional neuroimaging. TWISTER randomization makes it possible to infer functional selectivity from correlations between experimental runs. We provide a proof of concept in the form of a visuomotor mapping experiment and discuss the possible strengths and limitations of this new approach in light of our empirical results.

The multisensory nature of human action imagery

Imagination can appeal to all our senses and may, therefore, manifest in very different qualities (e.g., visual, tactile, proprioceptive, or kinesthetic). One line of research addresses action imagery that refers to a process by which people imagine the execution of an action without actual body movements. In action imagery, visual and kinesthetic aspects of the imagined action are particularly important. However, other sensory modalities may also play a role. The purpose of the paper will be to address issues that include: (i) the creation of an action image, (ii) how the brain generates images of movements and actions, (iii) the richness and vividness of action images. We will further address possible causes that determine the sensory impression of an action image, like task specificity, instruction and experience. In the end, we will outline open questions and future directions.

Reconstructing Spatiotemporal Trajectories of Visual Object Memories in the Human Brain

How the human brain reconstructs, step-by-step, the core elements of past experiences is still unclear. Here, we map the spatiotemporal trajectories along which visual object memories are reconstructed during associative recall. Specifically, we inquire whether retrieval reinstates feature representations in a copy-like but reversed direction with respect to the initial perceptual experience, or alternatively, this reconstruction involves format transformations and regions beyond initial perception. Participants from two cohorts studied new associations between verbs and randomly paired object images, and subsequently recalled the objects when presented with the corresponding verb cue. We first analyze multivariate fMRI patterns to map where in the brain high- and low-level object features can be decoded during perception and retrieval, showing that retrieval is dominated by conceptual features, represented in comparatively late visual and parietal areas. A separately acquired EEG dataset is then used to track the temporal evolution of the reactivated patterns using similarity-based EEG-fMRI fusion. This fusion suggests that memory reconstruction proceeds from anterior frontotemporal to posterior occipital and parietal regions, in line with a conceptual-to-perceptual gradient but only partly following the same trajectories as during perception. Specifically, a linear regression statistically confirms that the sequential activation of ventral visual stream regions is reversed between image perception and retrieval. The fusion analysis also suggests an information relay to frontoparietal areas late during retrieval. Together, the results shed light onto the temporal dynamics of memory recall and the transformations that the information undergoes between the initial experience and its later reconstruction from memory.

A cortical surface template for human neuroscience

Neuroimaging data analysis relies on normalization to standard anatomical templates to resolve macroanatomical differences across brains. Existing human cortical surface templates sample locations unevenly because of distortions introduced by inflation of the folded cortex into a standard shape. Here we present the onavg template, which affords uniform sampling of the cortex. We created the onavg template based on openly available high-quality structural scans of 1,031 brains-25 times more than existing cortical templates. We optimized the vertex locations based on cortical anatomy, achieving an even distribution. We observed consistently higher multivariate pattern classification accuracies and representational geometry inter-participant correlations based on onavg than on other templates, and onavg only needs three-quarters as much data to achieve the same performance compared with other templates. The optimized sampling also reduces CPU time across algorithms by 1.3-22.4% due to less variation in the number of vertices in each searchlight.

Symbolic and non-symbolic representations of numerical zero in the human brain

Representing the quantity zero as a symbolic concept is considered a unique achievement of abstract human thought.1,2 To conceptualize zero, one must abstract away from the (absence of) sensory evidence to construct a representation of numerical absence: creating "something" out of "nothing."2,3,4 Previous investigations of the neural representation of natural numbers reveal distinct numerosity-selective neural populations that overlap in their tuning curves with adjacent numerosities.5,6 Importantly, a component of this neural code is thought to be invariant across non-symbolic and symbolic numerical formats.7,8,9,10,11 Although behavioral evidence indicates that zero occupies a place at the beginning of this mental number line,12,13,14 in humans zero is also associated with unique behavioral and developmental profiles compared to natural numbers,4,15,16,17 suggestive of a distinct neural basis for zero. We characterized the neural representation of zero in the human brain by employing two qualitatively different numerical tasks18,19 in concert with magnetoencephalography (MEG) recordings. We assay both neural representations of non-symbolic numerosities (dot patterns), including zero (empty sets), and symbolic numerals, including symbolic zero. Our results reveal that neural representations of zero are situated along a graded neural number line shared with other natural numbers. Notably, symbolic representations of zero generalized to predict non-symbolic empty sets. We go on to localize abstract representations of numerical zero to posterior association cortex, extending the purview of parietal cortex in human numerical cognition to encompass representations of zero.10,20.

Happy people are always similar: The evidence from brain morphological and functional inter-subject correlations

A fundamental question in the study of happiness is whether there is neural evidence to support a well-known hypothesis that happy people are always similar while unfortunate people have their own misfortunes. To investigate this, we employed several happiness-related questionnaires to identify potential components of happiness, and further investigated and confirmed their associations with personality, mood, aggressive behaviors, and amygdala reactivity to fearful faces within a substantial sample size of college students (n = 570). Additionally, we examined the functional and morphological similarities and differences among happy individuals using the inter-subject representational similarity analysis (IS-RSA). IS-RSA emphasizes the geometric properties in a high-dimensional space constructed by brain or behavioral patterns and focuses on individual subjects. Our behavioral findings unveiled two factors of happiness: individual and social, both of which mediated the effect of personality traits on individual aggression. Subsequently, mood mediated the impact of happiness on aggressive behaviors across two subgroup splits. Functional imaging data revealed that individuals with higher levels of happiness exhibited reduced amygdala reactivity to fearful faces, as evidenced by a conventional face-matching task (n = 104). Moreover, IS-RSA demonstrated that these participants manifested similar neural activation patterns when processing fearful faces within the visual pathway, but not within the emotional network (e.g., amygdala). Morphological observations (n = 425) indicated that individuals with similar high happiness levels exhibited comparable gray matter volume patterns within several networks, including the default mode network, fronto-parietal network, visual network, and attention network. Collectively, these findings offer early neural evidence supporting the proposition that happy individuals may share common neural characteristics.

Deep learning based decoding of single local field potential events

How is information processed in the cerebral cortex? In most cases, recorded brain activity is averaged over many (stimulus) repetitions, which erases the fine-structure of the neural signal. However, the brain is obviously a single-trial processor. Thus, we here demonstrate that an unsupervised machine learning approach can be used to extract meaningful information from electro-physiological recordings on a single-trial basis. We use an auto-encoder network to reduce the dimensions of single local field potential (LFP) events to create interpretable clusters of different neural activity patterns. Strikingly, certain LFP shapes correspond to latency differences in different recording channels. Hence, LFP shapes can be used to determine the direction of information flux in the cerebral cortex. Furthermore, after clustering, we decoded the cluster centroids to reverse-engineer the underlying prototypical LFP event shapes. To evaluate our approach, we applied it to both extra-cellular neural recordings in rodents, and intra-cranial EEG recordings in humans. Finally, we find that single channel LFP event shapes during spontaneous activity sample from the realm of possible stimulus evoked event shapes. A finding which so far has only been demonstrated for multi-channel population coding.