Representational geometry: integrating cognition, computation, and the brain

Local neuronal relational structures underlying the contents of human conscious experience

While most theories of consciousness posit some kind of dependence on global network activities, I consider here an alternative, localist perspective-in which localized cortical regions each underlie the emergence of a unique category of conscious experience. Under this perspective, the large-scale activation often found in the cortex is a consequence of the complexity of typical conscious experiences rather than an obligatory condition for the emergence of conscious awareness-which can flexibly shift, depending on the richness of its contents, from local to more global activation patterns. This perspective fits a massive body of human imaging, recordings, lesions and stimulation data but opens a fundamental problem: how can the information, defining each content, be derived locally in each cortical region. Here, I will discuss a solution echoing pioneering structuralist ideas in which the content of a conscious experience is defined by its relationship to all other contents within an experiential category. In neuronal terms, this relationship structure between contents is embodied by the local geometry of similarity distances between cortical activation patterns generated during each conscious experience, likely mediated via networks of local neuronal connections. Thus, in order for any conscious experience to appear in an individual's mind, two central conditions must be met. First, a specific configural pattern ("bar-code") of neuronal activity must appear within a local relational geometry, i.e. a cortical area. Second, the individual neurons underlying the activated pattern must be bound into a unified functional ensemble through a burst of recurrent neuronal firing: local "ignitions".

Predicting Identity-Preserving Object Transformations across the Human Ventral Visual Stream

In everyday life, we have no trouble categorizing objects varying in position, size, and orientation. Previous fMRI research shows that higher-level object processing regions in the human lateral occipital cortex may link object responses from different affine states (i.e., size and viewpoint) through a general linear mapping function capable of predicting responses to novel objects. In this study, we extended this approach to examine the mapping for both Euclidean (e.g., position and size) and non-Euclidean (e.g., image statistics and spatial frequency) transformations across the human ventral visual processing hierarchy, including areas V1, V2, V3, V4, ventral occipitotemporal cortex, and lateral occipitotemporal cortex. The predicted pattern generated from a linear mapping function could capture a significant amount of the changes associated with the transformations throughout the ventral visual stream. The derived linear mapping functions were not category independent as performance was better for the categories included than those not included in training and better between two similar versus two dissimilar categories in both lower and higher visual regions. Consistent with object representations being stronger in higher than in lower visual regions, pattern selectivity and object category representational structure were somewhat better preserved in the predicted patterns in higher than in lower visual regions. There were no notable differences between Euclidean and non-Euclidean transformations. These findings demonstrate a near-orthogonal representation of object identity and these nonidentity features throughout the human ventral visual processing pathway with these nonidentity features largely untangled from the identity features early in visual processing.SIGNIFICANCE STATEMENT Presently we still do not fully understand how object identity and nonidentity (e.g., position, size) information are simultaneously represented in the primate ventral visual system to form invariant representations. Previous work suggests that the human lateral occipital cortex may be linking different affine states of object representations through general linear mapping functions. Here, we show that across the entire human ventral processing pathway, we could link object responses in different states of nonidentity transformations through linear mapping functions for both Euclidean and non-Euclidean transformations. These mapping functions are not identity independent, suggesting that object identity and nonidentity features are represented in a near rather than a completely orthogonal manner.

Which deep learning model can best explain object representations of within-category exemplars?

Deep neural network (DNN) models realize human-equivalent performance in tasks such as object recognition. Recent developments in the field have enabled testing the hierarchical similarity of object representation between the human brain and DNNs. However, the representational geometry of object exemplars within a single category using DNNs is unclear. In this study, we investigate which DNN model has the greatest ability to explain invariant within-category object representations by computing the similarity between representational geometries of visual features extracted at the high-level layers of different DNN models. We also test for the invariability of within-category object representations of these models by identifying object exemplars. Our results show that transfer learning models based on ResNet50 best explained both within-category object representation and object identification. These results suggest that the invariability of object representations in deep learning depends not on deepening the neural network but on building a better transfer learning model.

A Representational Similarity Analysis of Cognitive Control during Color-Word Stroop

Progress in understanding the neural bases of cognitive control has been supported by the paradigmatic color-word Stroop task, in which a target response (color name) must be selected over a more automatic, yet potentially incongruent, distractor response (word). For this paradigm, models have postulated complementary coding schemes: dorsomedial frontal cortex (DMFC) is proposed to evaluate the demand for control via incongruency-related coding, whereas dorsolateral PFC (DLPFC) is proposed to implement control via goal and target-related coding. Yet, mapping these theorized schemes to measured neural activity within this task has been challenging. Here, we tested for these coding schemes relatively directly, by decomposing an event-related color-word Stroop task via representational similarity analysis. Three neural coding models were fit to the similarity structure of multivoxel patterns of human fMRI activity, acquired from 65 healthy, young-adult males and females. Incongruency coding was predominant in DMFC, whereas both target and incongruency coding were present with indistinguishable strength in DLPFC. In contrast, distractor information was strongly encoded within early visual cortex. Further, these coding schemes were differentially related to behavior: individuals with stronger DLPFC (and lateral posterior parietal cortex) target coding, but weaker DMFC incongruency coding, exhibited less behavioral Stroop interference. These results highlight the utility of the representational similarity analysis framework for investigating neural mechanisms of cognitive control and point to several promising directions to extend the Stroop paradigm.SIGNIFICANCE STATEMENT How the human brain enables cognitive control - the ability to override behavioral habits to pursue internal goals - has been a major focus of neuroscience research. This ability has been frequently investigated by using the Stroop color-word naming task. With the Stroop as a test-bed, many theories have proposed specific neuroanatomical dissociations, in which medial and lateral frontal brain regions underlie cognitive control by encoding distinct types of information. Yet providing a direct confirmation of these claims has been challenging. Here, we demonstrate that representational similarity analysis, which estimates and models the similarity structure of brain activity patterns, can successfully establish the hypothesized functional dissociations within the Stroop task. Representational similarity analysis may provide a useful approach for investigating cognitive control mechanisms.

Untangling the Animacy Organization of Occipitotemporal Cortex

Some of the most impressive functional specializations in the human brain are found in the occipitotemporal cortex (OTC), where several areas exhibit selectivity for a small number of visual categories, such as faces and bodies, and spatially cluster based on stimulus animacy. Previous studies suggest this animacy organization reflects the representation of an intuitive taxonomic hierarchy, distinct from the presence of face- and body-selective areas in OTC. Using human functional magnetic resonance imaging, we investigated the independent contribution of these two factors-the face-body division and taxonomic hierarchy-in accounting for the animacy organization of OTC and whether they might also be reflected in the architecture of several deep neural networks that have not been explicitly trained to differentiate taxonomic relations. We found that graded visual selectivity, based on animal resemblance to human faces and bodies, masquerades as an apparent animacy continuum, which suggests that taxonomy is not a separate factor underlying the organization of the ventral visual pathway.SIGNIFICANCE STATEMENT Portions of the visual cortex are specialized to determine whether types of objects are animate in the sense of being capable of self-movement. Two factors have been proposed as accounting for this animacy organization: representations of faces and bodies and an intuitive taxonomic continuum of humans and animals. We performed an experiment to assess the independent contribution of both of these factors. We found that graded visual representations, based on animal resemblance to human faces and bodies, masquerade as an apparent animacy continuum, suggesting that taxonomy is not a separate factor underlying the organization of areas in the visual cortex.

The Dual Mechanisms of Cognitive Control Project

We describe an ambitious ongoing study that has been strongly influenced and inspired by Don Stuss's career-long efforts to identify key cognitive processes that characterize executive control, investigate potential unifying dimensions that define prefrontal function, and carefully attend to individual differences. The Dual Mechanisms of Cognitive Control project tests a theoretical framework positing two key control dimensions: proactive and reactive. The framework's central tenets are that proactive and reactive control modes reflect domain-general dimensions of individual variation, with distinctive neural signatures, involving the lateral pFC as a central node within associated brain networks (e.g., fronto-parietal, cingulo-opercular). In the Dual Mechanisms of Cognitive Control project, each participant is scanned while performing theoretically targeted variants of multiple well-established cognitive control tasks (Stroop, cued task-switching, AX-CPT, Sternberg working memory) in three separate imaging sessions, that each encourages utilization of different control modes plus also completes an extensive out-of-scanner individual differences battery. Additional key features of the project include a high spatio-temporal resolution (multiband) acquisition protocol and a sample that includes a substantial subset of monozygotic twin pairs and participants recruited from the Human Connectome Project. Although data collection is still continuing (target n = 200), we provide an overview of the study design and protocol, along with initial results (n = 80) revealing evidence of a domain-general neural signature of cognitive control and its modulation under reactive conditions. Aligned with Don Stuss's legacy of scientific community building, a partial data set has been publicly released, with the full data set released at project completion, so it can serve as a valuable resource.

Characterizing neural phase-space trajectories via Principal Louvain Clustering

BACKGROUND

With the growing size and richness of neuroscience datasets in terms of dimension, volume, and resolution, identifying spatiotemporal patterns in those datasets is increasingly important. Multivariate dimension-reduction methods are particularly adept at addressing these challenges.

NEW METHOD

In this paper, we propose a novel method, which we refer to as Principal Louvain Clustering (PLC), to identify clusters in a low-dimensional data subspace, based on time-varying trajectories of spectral dynamics across multisite local field potential (LFP) recordings in awake behaving mice. Data were recorded from prefrontal cortex, hippocampus, and parietal cortex in eleven mice while they explored novel and familiar environments.

RESULTS

PLC-identified subspaces and clusters showed high consistency across animals, and were modulated by the animals' ongoing behavior.

CONCLUSIONS

PLC adds to an important growing literature on methods for characterizing dynamics in high-dimensional datasets, using a smaller number of parameters. The method is also applicable to other kinds of datasets, such as EEG or MEG.

Orthogonal neural codes for speech in the infant brain

Creating invariant representations from an everchanging speech signal is a major challenge for the human brain. Such an ability is particularly crucial for preverbal infants who must discover the phonological, lexical, and syntactic regularities of an extremely inconsistent signal in order to acquire language. Within the visual domain, an efficient neural solution to overcome variability consists in factorizing the input into a reduced set of orthogonal components. Here, we asked whether a similar decomposition strategy is used in early speech perception. Using a 256-channel electroencephalographic system, we recorded the neural responses of 3-mo-old infants to 120 natural consonant-vowel syllables with varying acoustic and phonetic profiles. Using multivariate pattern analyses, we show that syllables are factorized into distinct and orthogonal neural codes for consonants and vowels. Concerning consonants, we further demonstrate the existence of two stages of processing. A first phase is characterized by orthogonal and context-invariant neural codes for the dimensions of manner and place of articulation. Within the second stage, manner and place codes are integrated to recover the identity of the phoneme. We conclude that, despite the paucity of articulatory motor plans and speech production skills, pre-babbling infants are already equipped with a structured combinatorial code for speech analysis, which might account for the rapid pace of language acquisition during the first year.

Self-face and emotional faces-are they alike?

The image of one’s own face is a particularly distinctive feature of the self. The self-face differs from other faces not only in respect of its familiarity but also in respect of its subjective emotional significance and saliency. The current study aimed at elucidating similarities/dissimilarities between processing of one’s own face and emotional faces: happy faces (based on the self-positive bias) and fearful faces (because of their high perceptual saliency, a feature shared with self-face). Electroencephalogram data were collected in the group of 30 participants who performed a simple detection task. Event-related potential analyses indicated significantly increased P3 and late positive potential amplitudes to the self-face in comparison to all other faces: fearful, happy and neutral. Permutation tests confirmed the differences between the self-face and all three types of other faces for numerous electrode sites and in broad time windows. Representational similarity analysis, in turn, revealed distinct processing of the self-face and did not provide any evidence in favour of similarities between the self-face and emotional (either negative or positive) faces. These findings strongly suggest that the self-face processing do not resemble those of emotional faces, thus implying that prioritized self-referential processing is driven by the subjective relevance of one’s own face.

Overfitting the Literature to One Set of Stimuli and Data

A large number of papers in Computational Cognitive Neuroscience are developing and testing novel analysis methods using one specific neuroimaging dataset and problematic experimental stimuli. Publication bias and confirmatory exploration will result in overfitting to the limited available data. We highlight the problems with this specific dataset and argue for the need to collect more good quality open neuroimaging data using a variety of experimental stimuli, in order to test the generalisability of current published results, and allow for more robust results in future work.

Being the Gatekeeper: How Thinking about Sharing Affects Neural Encoding of Information

Information transmission in a society depends on individuals' intention to share or not. Yet, little is known about whether being the gatekeeper shapes the brain's processing of incoming information. Here, we examine how thinking about sharing affects neural encoding of information, and whether this effect is moderated by the person's real-life social network position. In an functional magnetic resonance imaging study, participants rated abstracts of news articles on how much they wanted to read for themselves (read) or-as information gatekeepers-to share with a specific other (narrowcast) or to post on their social media feed (broadcast). In all conditions, consistent spatial blood oxygen level-dependent patterns associated with news articles were observed across participants in brain regions involved in perceptual and language processing as well as higher-order processes. However, when thinking about sharing, encoding consistency decreased in higher-order processing areas (e.g., default mode network), suggesting that the gatekeeper role involves more individualized processing in the brain, that is, person- and context-specific. Moreover, participants whose social networks had high ego-betweenness centrality (i.e., more likely to be information gatekeeper in real life) showed more individualized encoding when thinking about broadcasting. This study reveals how gatekeeping shapes our brain's processing of incoming information.

Neural Coding of Cognitive Control: The Representational Similarity Analysis Approach

Cognitive control relies on distributed and potentially high-dimensional frontoparietal task representations. Yet, the classical cognitive neuroscience approach in this domain has focused on aggregating and contrasting neural measures - either via univariate or multivariate methods - along highly abstracted, 1D factors (e.g., Stroop congruency). Here, we present representational similarity analysis (RSA) as a complementary approach that can powerfully inform representational components of cognitive control theories. We review several exemplary uses of RSA in this regard. We further show that most classical paradigms, given their factorial structure, can be optimized for RSA with minimal modification. Our aim is to illustrate how RSA can be incorporated into cognitive control investigations to shed new light on old questions.

Getting to Know You: Emerging Neural Representations during Face Familiarization

The successful recognition of familiar persons is critical for social interactions. Despite extensive research on the neural representations of familiar faces, we know little about how such representations unfold as someone becomes familiar. In three EEG experiments on human participants of both sexes, we elucidated how representations of face familiarity and identity emerge from different qualities of familiarization: brief perceptual exposure (Experiment 1), extensive media familiarization (Experiment 2), and real-life personal familiarization (Experiment 3). Time-resolved representational similarity analysis revealed that familiarization quality has a profound impact on representations of face familiarity: they were strongly visible after personal familiarization, weaker after media familiarization, and absent after perceptual familiarization. Across all experiments, we found no enhancement of face identity representation, suggesting that familiarity and identity representations emerge independently during face familiarization. Our results emphasize the importance of extensive, real-life familiarization for the emergence of robust face familiarity representations, constraining models of face perception and recognition memory.SIGNIFICANCE STATEMENT Despite extensive research on the neural representations of familiar faces, we know little about how such representations unfold as someone becomes familiar. To elucidate how face representations change as we get familiar with someone, we conducted three EEG experiments where we used brief perceptual exposure, extensive media familiarization, or real-life personal familiarization. Using multivariate representational similarity analysis, we demonstrate that the method of familiarization has a profound impact on face representations, and emphasize the importance of real-life familiarization. Additionally, familiarization shapes representations of face familiarity and identity differently: as we get to know someone, familiarity signals seem to appear before the formation of identity representations.

Joint representation of color and form in convolutional neural networks: A stimulus-rich network perspective

To interact with real-world objects, any effective visual system must jointly code the unique features defining each object. Despite decades of neuroscience research, we still lack a firm grasp on how the primate brain binds visual features. Here we apply a novel network-based stimulus-rich representational similarity approach to study color and form binding in five convolutional neural networks (CNNs) with varying architecture, depth, and presence/absence of recurrent processing. All CNNs showed near-orthogonal color and form processing in early layers, but increasingly interactive feature coding in higher layers, with this effect being much stronger for networks trained for object classification than untrained networks. These results characterize for the first time how multiple basic visual features are coded together in CNNs. The approach developed here can be easily implemented to characterize whether a similar coding scheme may serve as a viable solution to the binding problem in the primate brain.

Accounting for Biases in the Estimation of Neuronal Signal Correlation

Signal correlation (rs) is commonly defined as the correlation between the tuning curves of two neurons and is widely used as a metric of tuning similarity. It is fundamental to how populations of neurons represent stimuli and has been central to many studies of neural coding. Yet the classic estimate, Pearson's correlation coefficient, [Formula: see text], between the average responses of two neurons to a set of stimuli suffers from confounding biases. The estimate [Formula: see text] can be downwardly biased by trial-to-trial variability and also upwardly biased by trial-to-trial correlation between neurons, and these biases can hide important aspects of neural coding. Here we provide analytic results on the source of these biases and explore them for ranges of parameters that are relevant for electrophysiological experiments. We then provide corrections for these biases that we validate in simulation. Furthermore, we apply these corrected estimators to make the following novel experimental observation in cortical area MT: pairs of nearby neurons that are strongly tuned for motion direction tend to have high signal correlation, and pairs that are weakly tuned tend to have low signal correlation. We dismiss a trivial explanation for this and find that an analogous trend holds for orientation tuning in the primary visual cortex. We also consider the potential consequences for encoding whereby the association of signal correlation and tuning strength naturally regularizes the dimensionality of downstream computations.SIGNIFICANCE STATEMENT Fundamental to how cortical neurons encode information about the environment is their functional similarity, that is, the redundancy in what they encode and their shared noise. These properties have been extensively studied theoretically and experimentally throughout the nervous system, but here we show that a common estimator of functional similarity has confounding biases. We characterize these biases and provide estimators that do not suffer from them. Using our improved estimators, we demonstrate a novel result, that is, there is a positive relationship between tuning curve similarity and amplitude for nearby neurons in the visual cortical motion area MT. We provide a simple stochastic model explaining this relationship and discuss how it would naturally regularize the dimensionality of neural encoding.

Using distance on the Riemannian manifold to compare representations in brain and in models

Representational similarity analysis (RSA) summarizes activity patterns for a set of experimental conditions into a matrix composed of pairwise comparisons between activity patterns. Two examples of such matrices are the condition-by-condition inner product and correlation matrix. These representational matrices reside on the manifold of positive semidefinite matrices, called the Riemannian manifold. We hypothesize that representational similarities would be more accurately quantified by considering the underlying manifold of the representational matrices. Thus, we introduce the distance on the Riemannian manifold as a metric for comparing representations. Analyzing simulated and real fMRI data and considering a wide range of metrics, we show that the Riemannian distance is least susceptible to sampling bias, results in larger intra-subject reliability, and affords searchlight mapping with high sensitivity and specificity. Furthermore, we show that the Riemannian distance can be used for measuring multi-dimensional connectivity. This measure captures both univariate and multivariate connectivity and is also more sensitive to nonlinear regional interactions compared to the state-of-the-art measures. Applying our proposed metric to neural network representations of natural images, we demonstrate that it also possesses outstanding performance in quantifying similarity in models. Taken together, our results lend credence to the proposition that RSA should consider the manifold of the representational matrices to summarize response patterns in the brain and in models.

Age-related dedifferentiation and hyperdifferentiation of perceptual and mnemonic representations

Preliminary evidence indicates that occipito-temporal activation patterns for different visual stimuli are less distinct in older (OAs) than younger (YAs) adults, suggesting a dedifferentiation of visual representations with aging. Yet, it is unclear if this deficit (1) affects only sensory or also categorical aspects of representations during visual perception (perceptual representations), and (2) affects only perceptual or also mnemonic representations. To investigate these issues, we fMRI-scanned YAs and OAs viewing and then remembering visual scenes. First, using representational similarity analyses, we distinguished sensory vs. categorical features of perceptual representations. We found that, compared to YAs, sensory features in early visual cortex were less differentiated in OAs (i.e., age-related dedifferentiation), replicating previous research, whereas categorical features in anterior temporal lobe (ATL) were more differentiated in OAs. This is, to our knowledge, the first report of an age-related hyperdifferentiation. Second, we assessed the quality of mnemonic representations by measuring encoding-retrieval similarity (ERS) in activation patterns. We found that aging impaired mnemonic representations in early visual cortex and hippocampus but enhanced mnemonic representations in ATL. Thus, both perceptual and mnemonic representations in ATL were enhanced by aging. In sum, our findings suggest that aging impairs visual and mnemonic representations in posterior brain regions but enhances them in anterior regions.

Learning to recognize while learning to speak: Self-supervision and developing a speaking motor

Traditionally, learning speech synthesis and speech recognition were investigated as two separate tasks. This separation hinders incremental development for concurrent synthesis and recognition, where partially-learned synthesis and partially-learned recognition must help each other throughout lifelong learning. This work is a paradigm shift-we treat synthesis and recognition as two intertwined aspects of a lifelong learning agent. Furthermore, in contrast to existing recognition or synthesis systems, babies do not need their mothers to directly supervise their vocal tracts at every moment during the learning. We argue that self-generated non-symbolic states/actions at fine-grained time level help such a learner as necessary temporal contexts. Here, we approach a new and challenging problem-how to enable an autonomous learning system to develop an artificial speaking motor for generating temporally-dense (e.g., frame-wise) actions on the fly without human handcrafting a set of symbolic states. The self-generated states/actions are Muscles-like, High-dimensional, Temporally-dense and Globally-smooth (MHTG), so that these states/actions are directly attended for concurrent synthesis and recognition for each time frame. Human teachers are relieved from supervising learner's motor ends. The Candid Covariance-free Incremental (CCI) Principal Component Analysis (PCA) is applied to develop such an artificial speaking motor where PCA features drive the motor. Since each life must develop normally, each Developmental Network-2 (DN-2) reaches the same network (maximum likelihood, ML) regardless of randomly initialized weights, where ML is not just for a function approximator but rather an emergent Turing Machine. The machine-synthesized sounds are evaluated by both the neural network and humans with recognition experiments. Our experimental results showed learning-to-synthesize and learning-to-recognize-through-synthesis for phonemes. This work corresponds to a key step toward our goal to close a great gap toward fully autonomous machine learning directly from the physical world.

Large-Scale and Multiscale Networks in the Rodent Brain during Novelty Exploration

Neural activity is coordinated across multiple spatial and temporal scales, and these patterns of coordination are implicated in both healthy and impaired cognitive operations. However, empirical cross-scale investigations are relatively infrequent, because of limited data availability and to the difficulty of analyzing rich multivariate datasets. Here, we applied frequency-resolved multivariate source-separation analyses to characterize a large-scale dataset comprising spiking and local field potential (LFP) activity recorded simultaneously in three brain regions (prefrontal cortex, parietal cortex, hippocampus) in freely-moving mice. We identified a constellation of multidimensional, inter-regional networks across a range of frequencies (2-200 Hz). These networks were reproducible within animals across different recording sessions, but varied across different animals, suggesting individual variability in network architecture. The theta band (∼4-10 Hz) networks had several prominent features, including roughly equal contribution from all regions and strong inter-network synchronization. Overall, these findings demonstrate a multidimensional landscape of large-scale functional activations of cortical networks operating across multiple spatial, spectral, and temporal scales during open-field exploration.

Unsupervised learning predicts human perception and misperception of gloss

Reflectance, lighting and geometry combine in complex ways to create images. How do we disentangle these to perceive individual properties, such as surface glossiness? We suggest that brains disentangle properties by learning to model statistical structure in proximal images. To test this hypothesis, we trained unsupervised generative neural networks on renderings of glossy surfaces and compared their representations with human gloss judgements. The networks spontaneously cluster images according to distal properties such as reflectance and illumination, despite receiving no explicit information about these properties. Intriguingly, the resulting representations also predict the specific patterns of ‘successes’ and ‘errors’ in human perception. Linearly decoding specular reflectance from the model’s internal code predicts human gloss perception better than ground truth, supervised networks or control models, and it predicts, on an image-by-image basis, illusions of gloss perception caused by interactions between material, shape and lighting. Unsupervised learning may underlie many perceptual dimensions in vision and beyond.

Storrs et al. train unsupervised generative neural networks on glossy surfaces and show how gloss perception in humans may emerge in an unsupervised fashion from learning to model statistical structure.

The contributions of the left hippocampus and bilateral inferior parietal lobule to form-meaning associative learning

Existing studies have identified crucial roles for the hippocampus and a distributed set of cortical regions (e.g., the inferior parietal cortex) in learning novel words. Nevertheless, researchers have not clearly determined how the hippocampus and cortical regions dynamically interact during novel word learning, especially during form-meaning associative learning. As a method to address this question, we used an online learning paradigm and representational similarity analysis to explore the contributions of the hippocampus and neocortex to form-meaning associative learning. Twenty-nine native Chinese college students were recruited to learn 30 form-meaning pairs, which were repeated 7 times during fMRI scan. Form-meaning associative learning elicited activations in a wide neural network including regions required for word processing (i.e., the bilateral inferior frontal gyrus and the occipitotemporal cortex), regions required for encoding (i.e., the bilateral parahippocampus and hippocampus), and regions required for cognitive control (i.e., the anterior cingulate cortex and dorsolateral prefrontal cortex). More importantly, our study revealed the differential roles of the left hippocampus and bilateral inferior parietal lobule (IPL) in form-meaning associative learning. Specifically, higher pattern similarity in the bilateral IPL in the early learning phase (repetitions 1 to 3) was related to better learning performance, while higher pattern similarity in the left hippocampus in the late learning phase (repetitions 5 to 7) was associated with better learning performance. These findings indicate that the hippocampus and cortical regions (e.g., the IPL) contribute to form-meaning learning in different stages.

Cognition as a Mechanical Process

Cognition is often defined as a dual process of physical and non-physical mechanisms. This duality originated from past theory on the constituent parts of the natural world. Even though material causation is not an explanation for all natural processes, phenomena at the cellular level of life are modeled by physical causes. These phenomena include explanations for the function of organ systems, including the nervous system and information processing in the cerebrum. This review restricts the definition of cognition to a mechanistic process and enlists studies that support an abstract set of proximate mechanisms. Specifically, this process is approached from a large-scale perspective, the flow of information in a neural system. Study at this scale further constrains the possible explanations for cognition since the information flow is amenable to theory, unlike a lower-level approach where the problem becomes intractable. These possible hypotheses include stochastic processes for explaining the processes of cognition along with principles that support an abstract format for the encoded information.

Scene context shapes category representational geometry during processing of tools

Tools are ubiquitous in human environments and to think about them we use concepts. Increasingly, conceptual representation is thought to be dynamic and sensitive to the goals of the observer. Indeed, observer goals can reshape representational geometry within cortical networks supporting concepts. In the present study, we investigated the novel hypothesis that task-irrelevant scene context may implicitly alter the representational geometry of regions within the tool network. Participants performed conceptual judgments on images of tools embedded in scenes that either suggested their use (i.e., a kitchen timer sitting on a kitchen counter with vegetables in a frying pan) or that they would simply be moved (i.e., a kitchen timer sitting in an open drawer with other miscellaneous kitchen items around). We investigated whether representations in the tool network reflect category, grip, and shape information using a representational similarity analysis (RSA). We show that a) a number of regions of the tool network reflect category information about tools and b) category information predicts patterns in supramarginal gyrus more strongly in use contexts than in move contexts. Together, these results show that information about tool category is distributed across different regions of the tool network and that scene context helps shape the representational geometry of the tool network.

Synchronous caregiving from birth to adulthood tunes humans' social brain

A birth-to-adulthood study tested the effects of maternal–newborn contact and synchronous caregiving on the social processing brain in human adults. For two decades, we followed preterm and full-term neonates, who received or lacked initial maternal bodily contact, repeatedly observing mother–child social synchrony. We measured the brain basis of affect-specific empathy in young adulthood to pinpoint regions sensitive to others’ distinct emotions. Maternal–newborn contact enhanced social synchrony across development, which, in turn, predicted amygdalar and insular sensitivity to emotion-specific empathy. Findings demonstrate the long-term effects of maternal caregiving in humans, similar to their role in other mammals, particularly in tuning core regions implicated in salience detection, simulation, and interoception that sustain empathy and human attachment.

Mammalian young are born with immature brain and rely on the mother’s body and caregiving behavior for maturation of neurobiological systems that sustain adult sociality. While research in animal models indicated the long-term effects of maternal contact and caregiving on the adult brain, little is known about the effects of maternal–newborn contact and parenting behavior on social brain functioning in human adults. We followed human neonates, including premature infants who initially lacked or received maternal–newborn skin-to-skin contact and full-term controls, from birth to adulthood, repeatedly observing mother–child social synchrony at key developmental nodes. We tested the brain basis of affect-specific empathy in young adulthood and utilized multivariate techniques to distinguish brain regions sensitive to others’ distinct emotions from those globally activated by the empathy task. The amygdala, insula, temporal pole (TP), and ventromedial prefrontal cortex (VMPFC) showed high sensitivity to others’ distinct emotions. Provision of maternal–newborn contact enhanced social synchrony across development from infancy and up until adulthood. The experience of synchrony, in turn, predicted the brain’s sensitivity to emotion-specific empathy in the amygdala and insula, core structures of the social brain. Social synchrony linked with greater empathic understanding in adolescence, which was longitudinally associated with higher neural sensitivity to emotion-specific empathy in TP and VMPFC. Findings demonstrate the centrality of synchronous caregiving, by which infants practice the detection and sharing of others’ affective states, for tuning the human social brain, particularly in regions implicated in salience detection, interoception, and mentalization that underpin affect sharing and human attachment.

Limits to visual representational correspondence between convolutional neural networks and the human brain

Convolutional neural networks (CNNs) are increasingly used to model human vision due to their high object categorization capabilities and general correspondence with human brain responses. Here we evaluate the performance of 14 different CNNs compared with human fMRI responses to natural and artificial images using representational similarity analysis. Despite the presence of some CNN-brain correspondence and CNNs' impressive ability to fully capture lower level visual representation of real-world objects, we show that CNNs do not fully capture higher level visual representations of real-world objects, nor those of artificial objects, either at lower or higher levels of visual representations. The latter is particularly critical, as the processing of both real-world and artificial visual stimuli engages the same neural circuits. We report similar results regardless of differences in CNN architecture, training, or the presence of recurrent processing. This indicates some fundamental differences exist in how the brain and CNNs represent visual information.

The dimensionality of neural representations for control

Cognitive control allows us to think and behave flexibly based on our context and goals. At the heart of theories of cognitive control is a control representation that enables the same input to produce different outputs contingent on contextual factors. In this review, we focus on an important property of the control representation's neural code: its representational dimensionality. Dimensionality of a neural representation balances a basic separability/generalizability trade-off in neural computation. We will discuss the implications of this trade-off for cognitive control. We will then briefly review current neuroscience findings regarding the dimensionality of control representations in the brain, particularly the prefrontal cortex. We conclude by highlighting open questions and crucial directions for future research.

Examining the Coding Strength of Object Identity and Nonidentity Features in Human Occipito-Temporal Cortex and Convolutional Neural Networks

A visual object is characterized by multiple visual features, including its identity, position and size. Despite the usefulness of identity and nonidentity features in vision and their joint coding throughout the primate ventral visual processing pathway, they have so far been studied relatively independently. Here in both female and male human participants, the coding of identity and nonidentity features was examined together across the human ventral visual pathway. The nonidentity features tested included two Euclidean features (position and size) and two non-Euclidean features (image statistics and spatial frequency (SF) content of an image). Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with identity outweighing the non-Euclidean but not the Euclidean features at higher levels of visual processing. In 14 convolutional neural networks (CNNs) pretrained for object categorization with varying architecture, depth, and with/without recurrent processing, nonidentity feature representation showed an initial large increase from early to mid-stage of processing, followed by a decrease at later stages of processing, different from brain responses. Additionally, from lower to higher levels of visual processing, position became more underrepresented and image statistics and SF became more overrepresented compared with identity in CNNs than in the human brain. Similar results were obtained in a CNN trained with stylized images that emphasized shape representations. Overall, by measuring the coding strength of object identity and nonidentity features together, our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.SIGNIFICANCE STATEMENT This study examined the coding strength of object identity and four types of nonidentity features along the human ventral visual processing pathway and compared brain responses with those of 14 convolutional neural networks (CNNs) pretrained to perform object categorization. Overall, identity representation increased and nonidentity feature representation decreased along the ventral visual pathway, with some notable differences among the different nonidentity features. CNNs differed from the brain in a number of aspects in their representations of identity and nonidentity features over the course of visual processing. Our approach provides a new tool for characterizing feature coding in the human brain and the correspondence between the brain and CNNs.

Towards semantic fMRI neurofeedback: navigating among mental states using real-time representational similarity analysis

Objective. Real-time functional magnetic resonance imaging neurofeedback (rt-fMRI-NF) is a non-invasive MRI procedure allowing examined participants to learn to self-regulate brain activity by performing mental tasks. A novel two-step rt-fMRI-NF procedure is proposed whereby the feedback display is updated in real-time based on high-level representations of experimental stimuli (e.g. objects to imagine) via real-time representational similarity analysis of multi-voxel patterns of brain activity.Approach. In a localizer session, the stimuli become associated with anchored points on a two-dimensional representational space where distances approximate between-pattern (dis)similarities. In the NF session, participants modulate their brain response, displayed as a movable point, to engage in a specific neural representation. The developed method pipeline is verified in a proof-of-concept rt-fMRI-NF study at 7 T involving a single healthy participant imagining concrete objects. Based on this data and artificial data sets with similar (simulated) spatio-temporal structure and variable (injected) signal and noise, the dependence on noise is systematically assessed.Main results. The participant in the proof-of-concept study exhibited robust activation patterns in the localizer session and managed to control the neural representation of a stimulus towards the selected target in the NF session. The offline analyses validated the rt-fMRI-NF results, showing that the rapid convergence to the target representation is noise-dependent.Significance. Our proof-of-concept study introduces a new NF method allowing the participant to navigate among different mental states. Compared to traditional NF designs (e.g. using a thermometer display to set the level of the neural signal), the proposed approach provides content-specific feedback to the participant and extra degrees of freedom to the experimenter enabling real-time control of the neural activity towards a target brain state without suggesting a specific mental strategy to the subject.

Reconstructing feedback representations in the ventral visual pathway with a generative adversarial autoencoder

While vision evokes a dense network of feedforward and feedback neural processes in the brain, visual processes are primarily modeled with feedforward hierarchical neural networks, leaving the computational role of feedback processes poorly understood. Here, we developed a generative autoencoder neural network model and adversarially trained it on a categorically diverse data set of images. We hypothesized that the feedback processes in the ventral visual pathway can be represented by reconstruction of the visual information performed by the generative model. We compared representational similarity of the activity patterns in the proposed model with temporal (magnetoencephalography) and spatial (functional magnetic resonance imaging) visual brain responses. The proposed generative model identified two segregated neural dynamics in the visual brain. A temporal hierarchy of processes transforming low level visual information into high level semantics in the feedforward sweep, and a temporally later dynamics of inverse processes reconstructing low level visual information from a high level latent representation in the feedback sweep. Our results append to previous studies on neural feedback processes by presenting a new insight into the algorithmic function and the information carried by the feedback processes in the ventral visual pathway.

Effects of age differences in memory formation on neural mechanisms of consolidation and retrieval

Episodic memory decline is a hallmark of cognitive aging and a multifaceted phenomenon. We review studies that target age differences across different memory processing stages, i.e., from encoding to retrieval. The available evidence suggests that age differences during memory formation may affect the quality of memory representations in an age-graded manner with downstream consequences for later processing stages. We argue that low memory quality in combination with age-related neural decline of key regions of the episodic memory network puts older adults in a double jeopardy situation that finally results in broader memory impairments in older compared to younger adults.

From "Aha!" to "Haha!" Using Humor to Cope with Negative Stimuli

Humor has been considered an effective emotion regulation strategy, and some behavioral studies have examined its superior effects on negative emotion regulation. However, its neural mechanisms remain unknown. Our functional magnetic resonance imaging study directly compared the emotion regulation effects and neural bases of humorous coping (reappraisal) and ordinary reappraisal following exposure to negative pictures. The behavioral results suggested that humorous reappraisal was more effective in downregulating negative emotions and upregulating positive emotions both in the short and long term. We also found 2 cooperative neural pathways involved in coping with negative stimuli by means of humor: the "hippocampal-thalamic-frontal pathway" and the "amygdala-cerebellar pathway." The former is associated with the restructuring of mental representations of negative situations and accompanied by an insightful ("Aha!") experience, while the latter is associated with humorous emotional release and accompanied by an expression of laughter ("Haha!"). Furthermore, the degree of hippocampal functional connectivity with both the thalamus and frontal cortex was positively correlated with changes in positive emotion, and this result implied that the degree of emotion regulation could be strongly directly related to the depth of cognitive reconstruction. These findings highlight that regulating negative emotions with humor involves cognitive restructuring and the release of positive emotions.

Conditional Entropy: A Potential Digital Marker for Stress

Recent decades have witnessed a substantial progress in the utilization of brain activity for the identification of stress digital markers. In particular, the success of entropic measures for this purpose is very appealing, considering (1) their suitability for capturing both linear and non-linear characteristics of brain activity recordings and (2) their direct association with the brain signal variability. These findings rely on external stimuli to induce the brain stress response. On the other hand, research suggests that the use of different types of experimentally induced psychological and physical stressors could potentially yield differential impacts on the brain response to stress and therefore should be dissociated from more general patterns. The present study takes a step toward addressing this issue by introducing conditional entropy (CE) as a potential electroencephalography (EEG)-based resting-state digital marker of stress. For this purpose, we use the resting-state multi-channel EEG recordings of 20 individuals whose responses to stress-related questionnaires show significantly higher and lower level of stress. Through the application of representational similarity analysis (RSA) and K-nearest-neighbor (KNN) classification, we verify the potential that the use of CE can offer to the solution concept of finding an effective digital marker for stress.

Dopamine and relapse to drug seeking

The actions of dopamine are essential to relapse to drug seeking but we still lack a precise understanding of how dopamine achieves these effects. Here we review recent advances from animal models in understanding how dopamine controls relapse to drug seeking. These advances have been enabled by important developments in understanding the basic neurochemical, molecular, anatomical, physiological and functional properties of the major dopamine pathways in the mammalian brain. The literature shows that although different forms of relapse to seeking different drugs of abuse each depend on dopamine, there are distinct dopamine mechanisms for relapse. Different circuit-level mechanisms, different populations of dopamine neurons and different activity profiles within these dopamine neurons, are important for driving different forms of relapse. This diversity highlights the need to better understand when, where and how dopamine contributes to relapse behaviours.

Finding Distributed Needles in Neural Haystacks

The human cortex encodes information in complex networks that can be anatomically dispersed and variable in their microstructure across individuals. Using simulations with neural network models, we show that contemporary statistical methods for functional brain imaging-including univariate contrast, searchlight multivariate pattern classification, and whole-brain decoding with L1 or L2 regularization-each have critical and complementary blind spots under these conditions. We then introduce the sparse-overlapping-sets (SOS) LASSO-a whole-brain multivariate approach that exploits structured sparsity to find network-distributed information-and show in simulation that it captures the advantages of other approaches while avoiding their limitations. When applied to fMRI data to find neural responses that discriminate visually presented faces from other visual stimuli, each method yields a different result, but existing approaches all support the canonical view that face perception engages localized areas in posterior occipital and temporal regions. In contrast, SOS LASSO uncovers a network spanning all four lobes of the brain. The result cannot reflect spurious selection of out-of-system areas because decoding accuracy remains exceedingly high even when canonical face and place systems are removed from the dataset. When used to discriminate visual scenes from other stimuli, the same approach reveals a localized signal consistent with other methods-illustrating that SOS LASSO can detect both widely distributed and localized representational structure. Thus, structured sparsity can provide an unbiased method for testing claims of functional localization. For faces and possibly other domains, such decoding may reveal representations more widely distributed than previously suspected.SIGNIFICANCE STATEMENT Brain systems represent information as patterns of activation over neural populations connected in networks that can be widely distributed anatomically, variable across individuals, and intermingled with other networks. We show that four widespread statistical approaches to functional brain imaging have critical blind spots in this scenario and use simulations with neural network models to illustrate why. We then introduce a new approach designed specifically to find radically distributed representations in neural networks. In simulation and in fMRI data collected in the well studied domain of face perception, the new approach discovers extensive signal missed by the other methods-suggesting that prior functional imaging work may have significantly underestimated the degree to which neurocognitive representations are distributed and variable across individuals.

Lessons From Deep Neural Networks for Studying the Coding Principles of Biological Neural Networks

One of the central goals in systems neuroscience is to understand how information is encoded in the brain, and the standard approach is to identify the relation between a stimulus and a neural response. However, the feature of a stimulus is typically defined by the researcher's hypothesis, which may cause biases in the research conclusion. To demonstrate potential biases, we simulate four likely scenarios using deep neural networks trained on the image classification dataset CIFAR-10 and demonstrate the possibility of selecting suboptimal/irrelevant features or overestimating the network feature representation/noise correlation. Additionally, we present studies investigating neural coding principles in biological neural networks to which our points can be applied. This study aims to not only highlight the importance of careful assumptions and interpretations regarding the neural response to stimulus features but also suggest that the comparative study between deep and biological neural networks from the perspective of machine learning can be an effective strategy for understanding the coding principles of the brain.

Towards a common template for neural reinforcement of finger individuation

The inability to individuate finger movements is a common impairment following stroke. Conventional physical therapy ignores underlying neural changes with recovery, leaving it unclear why sensorimotor function often remains impaired. Functional MRI neurofeedback can monitor neural activity and reinforce it towards a healthy template to restore function. However, identifying an individualized training template may not be possible depending on the severity of impairment. In this study, we investigated the use of functional alignment of brain data across healthy participants to create an idealized neural template to be used as a training target for new participants. We employed multi-voxel pattern analyses to assess the prediction accuracy and robustness to missing data of pre-trained functional templates corresponding to individual finger presses. We found a significant improvement in classification accuracy (p < 0.001) of individual finger presses when group data was aligned based on function (88%) rather than anatomy (46%). Importantly, we found no significant drop in performance when aligning a new participant to a pre-established template as compared to including this new participant in the creation of a new template. These results indicate that functionally aligned templates could provide an effective surrogate training target for patients following neurological injury.

Visual and Semantic Representations Predict Subsequent Memory in Perceptual and Conceptual Memory Tests

It is generally assumed that the encoding of a single event generates multiple memory representations, which contribute differently to subsequent episodic memory. We used functional magnetic resonance imaging (fMRI) and representational similarity analysis to examine how visual and semantic representations predicted subsequent memory for single item encoding (e.g., seeing an orange). Three levels of visual representations corresponding to early, middle, and late visual processing stages were based on a deep neural network. Three levels of semantic representations were based on normative observed ("is round"), taxonomic ("is a fruit"), and encyclopedic features ("is sweet"). We identified brain regions where each representation type predicted later perceptual memory, conceptual memory, or both (general memory). Participants encoded objects during fMRI, and then completed both a word-based conceptual and picture-based perceptual memory test. Visual representations predicted subsequent perceptual memory in visual cortices, but also facilitated conceptual and general memory in more anterior regions. Semantic representations, in turn, predicted perceptual memory in visual cortex, conceptual memory in the perirhinal and inferior prefrontal cortex, and general memory in the angular gyrus. These results suggest that the contribution of visual and semantic representations to subsequent memory effects depends on a complex interaction between representation, test type, and storage location.

Emerging native-similar neural representations underlie non-native speech category learning success

Learning non-native phonetic categories in adulthood is an exceptionally challenging task, characterized by large inter-individual differences in learning speed and outcomes. The neurobiological mechanisms underlying the inter-individual differences in the learning efficacy are not fully understood. Here we examined the extent to which training-induced neural representations of non-native Mandarin tone categories in English listeners (n = 53) are increasingly similar to those of the native listeners (n = 33) who acquired these categories early in infancy. We particularly assessed whether the neural similarities in representational structure between non-native learners and native listeners are robust neuromarkers of inter-individual differences in learning success. Using inter-subject neural representational similarity (IS-NRS) analysis and predictive modeling on two functional magnetic resonance imaging (fMRI) datasets, we examined the neural representational mechanisms underlying speech category learning success. Learners' neural representations that were significantly similar to the native listeners emerged in brain regions mediating speech perception following training; the extent of the emerging neural similarities with native listeners significantly predicted the learning speed and outcome in learners. The predictive power of IS-NRS outperformed models with other neural representational measures. Furthermore, neural representations underlying successful learning are multidimensional but cost-efficient in nature. The degree of the emergent native-similar neural representations was closely related to the robust neural sensitivity to feedback in the frontostriatal network. These findings provide important insights on experience-dependent representational neuroplasticity underlying successful speech learning in adulthood and could be leveraged in designing individualized feedback-based training paradigms that maximize learning efficiency.

Individualizing Representational Similarity Analysis

Representational similarity analysis (RSA) is a popular multivariate analysis technique in cognitive neuroscience that uses functional neuroimaging to investigate the informational content encoded in brain activity. As RSA is increasingly being used to investigate more clinically-geared questions, the focus of such translational studies turns toward the importance of individual differences and their optimization within the experimental design. In this perspective, we focus on two design aspects: applying individual vs. averaged behavioral dissimilarity matrices to multiple participants' neuroimaging data and ensuring the congruency between tasks when measuring behavioral and neural representational spaces. Incorporating these methods permits the detection of individual differences in representational spaces and yields a better-defined transfer of information from representational spaces onto multivoxel patterns. Such design adaptations are prerequisites for optimal translation of RSA to the field of precision psychiatry.

A distributed dynamic brain network mediates linguistic tone representation and categorization

Successful categorization requires listeners to represent the incoming sensory information, resolve the "blooming, buzzing confusion" inherent to noisy sensory signals, and leverage the accumulated evidence towards making a decision. Despite decades of intense debate, the neural systems underlying speech categorization remain unresolved. Here we assessed the neural representation and categorization of lexical tones by native Mandarin speakers (N = 31) across a range of acoustic and contextual variabilities (talkers, perceptual saliences, and stimulus-contexts) using functional magnetic imaging (fMRI) and an evidence accumulation model of decision-making. Univariate activation and multivariate pattern analyses reveal that the acoustic-variability-tolerant representations of tone category are observed within the middle portion of the left superior temporal gyrus (STG). Activation patterns in the frontal and parietal regions also contained category-relevant information that was differentially sensitive to various forms of variability. The robustness of neural representations of tone category in a distributed fronto-temporoparietal network is associated with trial-by-trial decision-making parameters. These findings support a hybrid model involving a representational core within the STG that operates dynamically within an extensive frontoparietal network to support the representation and categorization of linguistic pitch patterns.

Themes of advanced information processing in the primate brain

Here is a review of several empirical examples of information processing that occur in the primate cerebral cortex. These include visual processing, object identification and perception, information encoding, and memory. Also, there is a discussion of the higher scale neural organization, mainly theoretical, which suggests hypotheses on how the brain internally represents objects. Altogether they support the general attributes of the mechanisms of brain computation, such as efficiency, resiliency, data compression, and a modularization of neural function and their pathways. Moreover, the specific neural encoding schemes are expectedly stochastic, abstract and not easily decoded by theoretical or empirical approaches.

Reaction times predict dynamic brain representations measured with MEG for only some object categorisation tasks

Behavioural categorisation reaction times (RTs) provide a useful way to link behaviour to brain representations measured with neuroimaging. In this framework, objects are assumed to be represented in a multidimensional activation space, with the distances between object representations indicating their degree of neural similarity. Faster RTs have been reported to correlate with greater distances from a classification decision boundary for animacy. Objects inherently belong to more than one category, yet it is not known whether the RT-distance relationship, and its evolution over the time-course of the neural response, is similar across different categories. Here we used magnetoencephalography (MEG) to address this question. Our stimuli included typically animate and inanimate objects, as well as more ambiguous examples (i.e., robots and toys). We conducted four semantic categorisation tasks on the same stimulus set assessing animacy, living, moving, and human-similarity concepts, and linked the categorisation RTs to MEG time-series decoding data. Our results show a sustained RT-distance relationship throughout the time course of object processing for not only animacy, but also categorisation according to human-similarity. Interestingly, this sustained RT-distance relationship was not observed for the living and moving category organisations, despite comparable classification accuracy of the MEG data across all four category organisations. Our findings show that behavioural RTs predict representational distance for an organisational principle other than animacy, however further research is needed to determine why this relationship is observed only for some category organisations and not others.

Joint embedding: A scalable alignment to compare individuals in a connectivity space

A common coordinate space enabling comparison across individuals is vital to understanding human brain organization and individual differences. By leveraging dimensionality reduction algorithms, high-dimensional fMRI data can be represented in a low-dimensional space to characterize individual features. Such a representative space encodes the functional architecture of individuals and enables the observation of functional changes across time. However, determining comparable functional features across individuals in resting-state fMRI in a way that simultaneously preserves individual-specific connectivity structure can be challenging. In this work we propose scalable joint embedding to simultaneously embed multiple individual brain connectomes within a common space that allows individual representations across datasets to be aligned. Using Human Connectome Project data, we evaluated the joint embedding approach by comparing it to the previously established orthonormal alignment model. Alignment using joint embedding substantially increased the similarity of functional representations across individuals while simultaneously capturing their distinct profiles, allowing individuals to be more discriminable from each other. Additionally, we demonstrated that the common space established using resting-state fMRI provides a better overlap of task-activation across participants. Finally, in a more challenging scenario - alignment across a lifespan cohort aged from 6 to 85 - joint embedding provided a better prediction of age (r2 = 0.65) than the prior alignment model. It facilitated the characterization of functional trajectories across lifespan. Overall, these analyses establish that joint embedding can simultaneously capture individual neural representations in a common connectivity space aligning functional data across participants and populations and preserve individual specificity.

Individual differences among deep neural network models

Deep neural networks (DNNs) excel at visual recognition tasks and are increasingly used as a modeling framework for neural computations in the primate brain. Just like individual brains, each DNN has a unique connectivity and representational profile. Here, we investigate individual differences among DNN instances that arise from varying only the random initialization of the network weights. Using tools typically employed in systems neuroscience, we show that this minimal change in initial conditions prior to training leads to substantial differences in intermediate and higher-level network representations despite similar network-level classification performance. We locate the origins of the effects in an under-constrained alignment of category exemplars, rather than misaligned category centroids. These results call into question the common practice of using single networks to derive insights into neural information processing and rather suggest that computational neuroscientists working with DNNs may need to base their inferences on groups of multiple network instances.

An fMRI dataset in response to "The Grand Budapest Hotel", a socially-rich, naturalistic movie

Naturalistic stimuli evoke strong, consistent, and information-rich patterns of brain activity, and engage large extents of the human brain. They allow researchers to compare highly similar brain responses across subjects, and to study how complex representations are encoded in brain activity. Here, we describe and share a dataset where 25 subjects watched part of the feature film "The Grand Budapest Hotel" by Wes Anderson. The movie has a large cast with many famous actors. Throughout the story, the camera shots highlight faces and expressions, which are fundamental to understand the complex narrative of the movie. This movie was chosen to sample brain activity specifically related to social interactions and face processing. This dataset provides researchers with fMRI data that can be used to explore social cognitive processes and face processing, adding to the existing neuroimaging datasets that sample brain activity with naturalistic movies.

Revealing the multidimensional mental representations of natural objects underlying human similarity judgements

Objects can be characterized according to a vast number of possible criteria (such as animacy, shape, colour and function), but some dimensions are more useful than others for making sense of the objects around us. To identify these core dimensions of object representations, we developed a data-driven computational model of similarity judgements for real-world images of 1,854 objects. The model captured most explainable variance in similarity judgements and produced 49 highly reproducible and meaningful object dimensions that reflect various conceptual and perceptual properties of those objects. These dimensions predicted external categorization behaviour and reflected typicality judgements of those categories. Furthermore, humans can accurately rate objects along these dimensions, highlighting their interpretability and opening up a way to generate similarity estimates from object dimensions alone. Collectively, these results demonstrate that human similarity judgements can be captured by a fairly low-dimensional, interpretable embedding that generalizes to external behaviour.

The Role of the Human Auditory Corticostriatal Network in Speech Learning

We establish a mechanistic account of how the mature human brain functionally reorganizes to acquire and represent new speech sounds. Native speakers of English learned to categorize Mandarin lexical tone categories produced by multiple talkers using trial-by-trial feedback. We hypothesized that the corticostriatal system is a key intermediary in mediating temporal lobe plasticity and the acquisition of new speech categories in adulthood. We conducted a functional magnetic resonance imaging experiment in which participants underwent a sound-to-category mapping task. Diffusion tensor imaging data were collected, and probabilistic fiber tracking analysis was employed to assay the auditory corticostriatal pathways. Multivariate pattern analysis showed that talker-invariant novel tone category representations emerged in the left superior temporal gyrus (LSTG) within a few hundred training trials. Univariate analysis showed that the putamen, a subregion of the striatum, was sensitive to positive feedback in correctly categorized trials. With learning, functional coupling between the putamen and LSTG increased during error processing. Furthermore, fiber tractography demonstrated robust structural connectivity between the feedback-sensitive striatal regions and the LSTG regions that represent the newly learned tone categories. Our convergent findings highlight a critical role for the auditory corticostriatal circuitry in mediating the acquisition of new speech categories.