Representational Similarity Analyses

Smell the Label: Odors Influence Label Perception and Their Neural Processing

Providing nutrition or health labels on product packaging can be an effective strategy to promote a conscious and healthier diet. However, such labels also have the potential to be counterproductive by creating obstructive expectations about the flavor of the food and influencing odor perception. Conversely, olfaction could significantly influence label perception, whereby negative expectations could be mitigated by pleasant odors. This study explored the neural processing of the interplay between odors and nutrition labels using fMRI in 63 participants of either sex, to whom we presented beverage labels with different nutrition-related statements either with or without a congruent odor. On a behavioral level, the products for which the label was presented together with the odor were in general perceived as more positive than the same labels without an odor. Neuroimaging results revealed that added odors significantly altered activity in brain regions associated with flavor and label processing as well as decision-making, with higher activations in the right amygdala/piriform cortex (Amy/pirC) and orbitofrontal cortex. The presentation of odors induced pattern-based encoding in the right dorsolateral prefrontal cortex, the left ventral striatum/nucleus accumbens, and the right Amy/pirC when accounting for behavioral differences. This suggests that odors influence the effects of labels both on a neural and behavioral level and may offer the possibility of compensating for obstructive associations. The detailed mechanisms of odor and statement interactions within the relevant brain areas should be further investigated, especially for labels that evoke negative expectations.

Trial-level Representational Similarity Analysis

Neural representation refers to the brain activity that stands in for one's cognitive experience, and in cognitive neuroscience, the principal method to studying neural representations is representational similarity analysis (RSA). The classic RSA (cRSA) approach examines the overall quality of representations across numerous items by assessing the correspondence between two representational similarity matrices (RSMs): one based on a theoretical model of stimulus similarity and the other based on similarity in measured neural data. However, because cRSA cannot model representation at the level of individual trials, it is fundamentally limited in its ability to assess subject-, stimulus-, and trial-level variances that all influence representation. Here, we formally introduce trial-level RSA (tRSA), an analytical framework that estimates the strength of neural representation for singular experimental trials and evaluates hypotheses using multi-level models. First, we verified the correspondence between tRSA and cRSA in quantifying the overall representation strength across all trials. Second, we compared the statistical inferences drawn from both approaches using simulated data that reflected a wide range of scenarios. Compared to cRSA, the multi-level framework of tRSA was both more theoretically appropriate and significantly sensitive to true effects. Third, using real fMRI datasets, we further demonstrated several issues with cRSA, to which tRSA was more robust. Finally, we presented some novel findings of neural representations that could only be assessed with tRSA and not cRSA. In summary, tRSA proves to be a robust and versatile analytical approach for cognitive neuroscience and beyond.

Distinct neural representations of different linguistic components following sign language learning

Learning a new language is a process everyone undergoes at least once. However, studying the neural mechanisms behind first-time language learning is a challenging task. Here we aim to explore the functional alterations following learning Israeli Sign Language, a visuo-spatial rather than an auditory-based language. Specifically, we investigate how phonological, lexical, and sentence-level components of the language system differ in their neural representations. In this within-participant design, hearing individuals naïve to sign languages (n = 79) performed an fMRI task requiring the processing of different linguistic components, before and after attending an Israeli Sign Language course. A learning-induced increase in activation was detected in various brain regions in task contrasts related to all sign language linguistic components. Activation patterns while processing different linguistic components post-learning were spatially distinct, suggesting a unique neural representation for each component. Moreover, post-learning activation maps successfully predicted learning retention six months later, associating neural and performance measures.

Neural representation of sensorimotor features in language-motor areas during auditory and visual perception

Speech processing involves a complex interplay between sensory and motor systems in the brain, essential for early language development. Recent studies have extended this sensory-motor interaction to visual word processing, emphasizing the connection between reading and handwriting during literacy acquisition. Here we show how language-motor areas encode motoric and sensory features of language stimuli during auditory and visual perception, using functional magnetic resonance imaging (fMRI) combined with representational similarity analysis. Chinese-speaking adults completed tasks involving the perception of spoken syllables and written characters, alongside syllable articulation and finger writing tasks to localize speech-motor and writing-motor areas. We found that both language-motor and sensory areas generally encode production-related motoric features across modalities, indicating cooperative interactions between motor and sensory systems. Notably, sensory encoding within sensorimotor areas was observed during auditory speech perception, but not in visual character perception. These findings underscore the dual encoding capacities of language-motor areas, revealing both shared and distinct neural representation patterns across modalities, which may be linked to innate sensory-motor mechanisms and modality-specific processing demands. Our results shed light on the sensorimotor integration mechanisms underlying language perception, highlighting the importance of a cross-modality perspective.

Implications of neural integration of math and spatial experiences for math ability and math anxiety

Mathematical and spatial abilities are positively related at both the behavioral and neural levels. Much of the evidence illuminating this relationship comes from classic laboratory-based experimental methods focused on cognitive performance despite most individuals also experiencing math and space in other contexts, such as in conversations or lectures. To broaden our understanding of math-space integration in these more commonplace situations, we used an auditory memory-encoding task with stimuli whose content evoked a range of educational and everyday settings related to math or spatial thinking. We used a multivariate approach to directly assess the extent of neural similarity between activity patterns elicited by these math and spatial stimuli. Results from whole-brain searchlight analysis revealed a highly specific positive relation between math and spatial activity patterns in bilateral anterior hippocampi. Examining individual variation in math-space similarity, we found that greater math-space similarity in bilateral anterior hippocampi was associated with poorer math skills and higher anxiety about math. Integration of neural responses to mathematical and spatial content may not always portend positive outcomes. We suggest that episodic simulation of quotidian contexts may link everyday experiences with math and spatial thinking-and the strength of this link is predictive of math in a manner that diverges from math-space associations derived from more lab-based tasks. On a methodological level, this work points to the value of considering a wider range of experimental paradigms, and of the value of combining multivariate fMRI analysis with behavioral data to better contextualize interpretations of brain data.

Distinct connectivity patterns between perception and attention-related brain networks characterize dyslexia: Machine learning applied to resting-state fMRI

Diagnosis of dyslexia often occurs in late schooling years, leading to academic and psychological challenges. Furthermore, diagnosis is time-consuming, costly, and reliant on arbitrary cutoffs. On the other hand, automated algorithms hold great potential in medical and psychological diagnostics. The aim of the present study was to develop a machine learning tool for the detection of dyslexia in children based on the intrinsic connectivity patterns of different brain networks underlying perception and attention. Here, 117 children (8-12 years old; 58 females; 52 typical readers; TR and 65 children with dyslexia) completed cognitive and reading assessments and underwent 10 min of resting-state fMRI. Functional connectivity coefficients between 264 brain regions were used as features for machine learning. Different supervised algorithms were employed for classification of children with and without dyslexia. A classifier trained on dorsal attention network features exhibited the highest performance (accuracy .79, sensitivity .92, specificity .64). Auditory, visual, and fronto-parietal network-based classification showed intermediate accuracy levels (70-75%). These results highlight significant neurobiological differences in brain networks associated with visual attention between TR and children with dyslexia. Distinct neural integration patterns can differentiate dyslexia from typical development, which may be utilized in the future as a biomarker for the presence and/or severity of dyslexia.

Reduced temporal and spatial stability of neural activity patterns predict cognitive control deficits in children with ADHD

This study explores the neural underpinnings of cognitive control deficits in ADHD, focusing on overlooked aspects of trial-level variability of neural coding. We employed a novel computational approach to neural decoding on a single-trial basis alongside a cued stop-signal task which allowed us to distinctly probe both proactive and reactive cognitive control. Typically developing (TD) children exhibited stable neural response patterns for efficient proactive and reactive dual control mechanisms. However, neural coding was compromised in children with ADHD. Children with ADHD showed increased temporal variability and diminished spatial stability in neural responses in salience and frontal-parietal network regions, indicating disrupted neural coding during both proactive and reactive control. Moreover, this variability correlated with fluctuating task performance and with more severe symptoms of ADHD. These findings underscore the significance of modeling single-trial variability and representational similarity in understanding distinct components of cognitive control in ADHD, highlighting new perspectives on neurocognitive dysfunction in psychiatric disorders.

Event Integration and Temporal Differentiation: How Hierarchical Knowledge Emerges in Hippocampal Subfields through Learning

Everyday life is composed of events organized by changes in contexts, with each event containing an unfolding sequence of occurrences. A major challenge facing our memory systems is how to integrate sequential occurrences within events while also maintaining their details and avoiding over-integration across different contexts. We asked if and how distinct hippocampal subfields come to hierarchically and, in parallel, represent both event context and subevent occurrences with learning. Female and male human participants viewed sequential events defined as sequences of objects superimposed on shared color frames while undergoing high-resolution fMRI. Importantly, these events were repeated to induce learning. Event segmentation, as indexed by increased reaction times at event boundaries, was observed in all repetitions. Temporal memory decisions were quicker for items from the same event compared to across different events, indicating that events shaped memory. With learning, hippocampal CA3 multivoxel activation patterns clustered to reflect the event context, with more clustering correlated with behavioral facilitation during event transitions. In contrast, in the dentate gyrus (DG), temporally proximal items that belonged to the same event became associated with more differentiated neural patterns. A computational model explained these results by dynamic inhibition in the DG. Additional similarity measures support the notion that CA3 clustered representations reflect shared voxel populations, while DG's distinct item representations reflect different voxel populations. These findings suggest an interplay between temporal differentiation in the DG and attractor dynamics in CA3. They advance our understanding of how knowledge is structured through integration and separation across time and context.

Early visual cortices reveal interrelated item and category representations in aging

Neural dedifferentiation, the finding that neural representations tend to be less distinct in older adults compared with younger adults, has been associated with age-related declines in memory performance. Most studies assessing the relation between memory and neural dedifferentiation have evaluated how age impacts the distinctiveness of neural representations for different visual categories (e.g., scenes and objects). However, how age impacts the quality of neural representations at the level of individual items is still an open question. Here, we present data from an age-comparative fMRI study that aimed to understand how the distinctiveness of neural representations for individual stimuli differs between younger and older adults and relates to memory outcomes. Pattern similarity searchlight analyses yielded indicators of neural dedifferentiation at the level of individual items as well as at the category level in posterior occipital cortices. We asked whether age differences in neural distinctiveness at each representational level were associated with inter- and/or intraindividual variability in memory performance. While age-related dedifferentiation at both the item and category level related to between-person differences in memory, neural distinctiveness at the category level also tracked within-person variability in memory performance. Concurrently, neural distinctiveness at the item level was strongly associated with neural distinctiveness at the category level both within and across participants, elucidating a potential representational mechanism linking item- and category-level distinctiveness. In sum, we provide evidence that age-related neural dedifferentiation co-exists across multiple representational levels and is related to memory performance.Significance Statement Age-related memory decline has been associated with neural dedifferentiation, the finding that older adults have less distinctive neural representations than younger adults. This has been mostly shown for category information, while evidence for age differences in the specificity of item representations is meager. We used pattern similarity searchlight analyses to find indicators of neural dedifferentiation at both levels of representation (category and item) and linked distinctiveness to memory performance. Both item- and category-level dedifferentiation in the calcarine cortex were related to interindividual differences in memory performance, while category-level distinctiveness further tracked intraindividual variability. Crucially, neural distinctiveness was strongly tied between the item and category levels, indicating that intersecting representational properties of posterior occipital cortices reflect both individual exemplars and categories.

Early excitatory-inhibitory cortical modifications following skill learning are associated with motor memory consolidation and plasticity overnight

Consolidation of motor memories is vital to offline enhancement of new motor skills and involves short and longer-term offline processes following learning. While emerging evidence link glutamate and GABA dynamics in the primary motor cortex (M1) to online motor skill practice, its relationship with offline consolidation processes in humans is unclear. Using two-day repeated measures of behavioral and multimodal neuroimaging data before and following motor sequence learning, we show that short-term glutamatergic and GABAergic responses in M1 within minutes after learning were associated with longer-term learning-induced functional, structural, and behavioral modifications overnight. Furthermore, Glutamatergic and GABAergic modifications were differentially associated with different facets of motor memory consolidation. Our results point to unique and distinct roles of Glutamate and GABA in motor memory consolidation processes in the human brain across timescales and mechanistic levels, tying short-term changes on the neurochemical level to overnight changes in macroscale structure, function, and behavior.

BackWards - Unveiling the brain's topographic organization of paraspinal sensory input

Cortical reorganization and its potential pathological significance are being increasingly studied in musculoskeletal disorders such as chronic low back pain (CLBP) patients. However, detailed sensory-topographic maps of the human back are lacking, and a baseline characterization of such representations, reflecting the somatosensory organization of the healthy back, is needed before exploring potential sensory map reorganization. To this end, a novel pneumatic vibrotactile stimulation method was used to stimulate paraspinal sensory afferents, while studying their cortical representations in unprecedented detail. In 41 young healthy participants, vibrotactile stimulations at 20 Hz and 80 Hz were applied bilaterally at nine locations along the thoracolumbar axis while functional magnetic resonance imaging (fMRI) was performed. Model-based whole-brain searchlight representational similarity analysis (RSA) was used to investigate the organizational structure of brain activity patterns evoked by thoracolumbar sensory inputs. A model based on segmental distances best explained the similarity structure of brain activity patterns that were located in different areas of sensorimotor cortices, including the primary somatosensory and motor cortices and parts of the superior parietal cortex, suggesting that these brain areas process sensory input from the back in a "dermatomal" manner. The current findings provide a sound basis for testing the "cortical map reorganization theory" and its pathological relevance in CLBP.

Neural patterns differentiate traumatic from sad autobiographical memories in PTSD

For people with post-traumatic stress disorder (PTSD), recall of traumatic memories often displays as intrusions that differ profoundly from processing of 'regular' negative memories. These mnemonic features fueled theories speculating a unique cognitive state linked with traumatic memories. Yet, to date, little empirical evidence supports this view. Here we examined neural activity of patients with PTSD who were listening to narratives depicting their own memories. An intersubject representational similarity analysis of cross-subject semantic content and neural patterns revealed a differentiation in hippocampal representation by narrative type: semantically similar, sad autobiographical memories elicited similar neural representations across participants. By contrast, within the same individuals, semantically similar trauma memories were not represented similarly. Furthermore, we were able to decode memory type from hippocampal multivoxel patterns. Finally, individual symptom severity modulated semantic representation of the traumatic narratives in the posterior cingulate cortex. Taken together, these findings suggest that traumatic memories are an alternative cognitive entity that deviates from memory per se.

Awake Hippocampal-Cortical Co-reactivation Is Associated with Forgetting

Systems consolidation theories posit that consolidation occurs primarily through a coordinated communication between hippocampus and neocortex [Moscovitch, M., & Gilboa, A. Systems consolidation, transformation and reorganization: Multiple trace theory, trace transformation theory and their competitors. PsyArXiv, 2021; Kumaran, D., Hassabis, D., & McClelland, J. L. What learning systems do intelligent agents need? Complementary learning systems theory updated. Trends in Cognitive Sciences, 20, 512-534, 2016; McClelland, J. L., & O'Reilly, R. C. Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory. Psychological Review, 102, 419-457, 1995]. Recent sleep studies in rodents have shown that hippocampus and visual cortex replay the same information at temporal proximity ("co-replay"; Lansink, C. S., Goltstein, P. M., Lankelma, J. V., McNaughton, B. L., & Pennartz, C. M. A. Hippocampus leads ventral striatum in replay of place-reward information. PLoS Biology, 7, e1000173, 2009; Peyrache, A., Khamassi, M., Benchenane, K., Wiener, S. I., & Battaglia, F. P. Replay of rule-learning related neural patterns in the prefrontal cortex during sleep. Nature Neuroscience, 12, 919-926, 2009; Wierzynski, C. M., Lubenov, E. V., Gu, M., & Siapas, A. G. State-dependent spike-timing relationships between hippocampal and prefrontal circuits during sleep. Neuron, 61, 587-596, 2009; Ji, D., & Wilson, M. A. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature Neuroscience, 10, 100-107, 2007). We developed a novel repetition time (TR)-based co-reactivation analysis method to study hippocampal-cortical co-replays in humans using fMRI. Thirty-six young adults completed an image (face or scene) and location paired associate encoding task in the scanner, which were preceded and followed by resting state scans. We identified post-encoding rest TRs (± 1) that showed neural reactivation of each image-location trials in both hippocampus (HPC) and category-selective cortex (fusiform face area [FFA]). This allowed us to characterize temporally proximal coordinated reactivations ("co-reactivations") between HPC and FFA. Moreover, we found that increased HPC-FFA co-reactivations were associated with incorrectly recognized trials after a 1-week delay (p = .004). Finally, we found that these HPC-FFA co-reactivations were also associated with trials that were initially correctly recognized immediately after encoding but were later forgotten in 1-day (p = .043) and 1-week delay period (p = .031). We discuss these results from a trace transformation perspective [Sekeres, M. J., Winocur, G., & Moscovitch, M. The hippocampus and related neocortical structures in memory transformation. Neuroscience Letters, 680, 39-53, 2018; Winocur, G., & Moscovitch, M. Memory transformation and systems consolidation. Journal of the International Neuropsychological Society, 17, 766-780, 2011] and speculate that HPC-FFA co-reactivations may be integrating related events, at the expense of disrupting event-specific details, hence leading to forgetting.

Neural representations of ambiguous affective stimuli and resilience to anxiety in emerging adults

The tendency to interpret ambiguous stimuli as threatening has been associated with a range of anxiety disorders. Responses to ambiguity may be particularly relevant to mental health during the transition from adolescence to adulthood ("emerging adulthood"), when individuals encounter unfamiliar challenges and navigate novel social situations. However, it remains unclear whether neural representations of ambiguity relate to risk for anxiety. The present study sought to examine whether multivariate representations of ambiguity - and their similarity to representations of threat - relate to appraisals of ambiguity or anxiety in a sample of emerging adults. Participants (N = 41) viewed threatening (angry), nonthreatening (happy), and ambiguous (surprised) facial stimuli while undergoing fMRI. Outside of the scanner, participants were presented with the same stimuli and categorized the ambiguous faces as positive or negative. Using representational similarity analyses (RSA), we investigated whether the degree of pattern similarity in responses to ambiguous, nonthreatening, and threatening faces within the amygdala related to appraisals of ambiguous stimuli and anxiety symptomatology. We found that individuals who evidenced greater similarity (i.e., less differentiation) in neural representations of ambiguous and nonthreatening faces within the left amygdala reported lower concurrent anxiety. Additionally, trial-level pattern similarity predicted subsequent appraisals of ambiguous stimuli. These findings provide insight into how neural representations of ambiguity relate to risk or resilience for the development of anxiety.

Family Cohesion Moderates the Relation between Parent-Child Neural Connectivity Pattern Similarity and Youth's Emotional Adjustment

Despite a recent surge in research examining parent-child neural similarity using fMRI, there remains a need for further investigation into how such similarity may play a role in children's emotional adjustment. Moreover, no prior studies explored the potential contextual factors that may moderate the link between parent-child neural similarity and children's developmental outcomes. In this study, 32 parent-youth dyads (parents: M age = 43.53 years, 72% female; children: M age = 11.69 years, 41% female) watched an emotion-evoking animated film while being scanned using fMRI. We first quantified how similarly emotion network interacts with other brain regions in responding to the emotion-evoking film between parents and their children. We then examined how such parent-child neural similarity is associated with children's emotional adjustment, with attention to the moderating role of family cohesion. Results revealed that higher parent-child similarity in functional connectivity pattern during movie viewing was associated with better emotional adjustment, including less negative affect, lower anxiety, and greater ego resilience in youth. Moreover, such associations were significant only among families with higher cohesion, but not among families with lower cohesion. The findings advance our understanding of the neural mechanisms underlying how children thrive by being in sync and attuned with their parents, and provide novel empirical evidence that the effects of parent-child concordance at the neural level on children's development are contextually dependent.SIGNIFICANCE STATEMENT What neural processes underlie the attunement between children and their parents that helps children thrive? Using a naturalistic movie-watching fMRI paradigm, we find that greater parent-child similarity in how emotion network interacts with other brain regions during movie viewing is associated with youth's better emotional adjustment including less negative affect, lower anxiety, and greater ego resilience. Interestingly, these associations are only significant among families with higher cohesion, but not among those with lower cohesion. Our findings provide novel evidence that parent-child shared neural processes to emotional situations can confer benefits to children, and underscore the importance of considering specific family contexts in which parent-child neural similarity may be beneficial or detrimental to children's development, highlighting a crucial direction for future research.

Neural representations of anxiety in adolescents with anorexia nervosa: a multivariate approach

Anorexia nervosa (AN) is characterized by low body weight, fear of gaining weight, and distorted body image. Anxiety may play a role in the formation and course of the illness, especially related to situations involving food, eating, weight, and body image. To understand distributed patterns and consistency of neural responses related to anxiety, we enrolled 25 female adolescents with AN and 22 non-clinical female adolescents with mild anxiety who underwent two fMRI sessions in which they saw personalized anxiety-provoking word stimuli and neutral words. Consistency in brain response patterns across trials was determined using a multivariate representational similarity analysis (RSA) approach within anxiety circuits and in a whole-brain voxel-wise searchlight analysis. In the AN group there was higher representational similarity for anxiety-provoking compared with neutral stimuli predominantly in prefrontal regions including the frontal pole, medial prefrontal cortex, dorsolateral prefrontal cortex, and medial orbitofrontal cortex, although no significant group differences. Severity of anxiety correlated with consistency of brain responses within anxiety circuits and in cortical and subcortical regions including the frontal pole, middle frontal gyrus, orbitofrontal cortex, thalamus, lateral occipital cortex, middle temporal gyrus, and cerebellum. Higher consistency of activation in those with more severe anxiety symptoms suggests the possibility of a greater degree of conditioned brain responses evoked by personally-relevant emotional stimuli. Anxiety elicited by disorder-related stimuli may activate stereotyped, previously-learned neural responses within- and outside of classical anxiety circuits. Results have implications for understanding consistent and automatic responding to environmental stimuli that may play a role in maintenance of AN.

Neural similarity between overlapping events at learning differentially affects reinstatement across the cortex

Episodic memory often involves high overlap between the actors, locations, and objects of everyday events. Under some circumstances, it may be beneficial to distinguish, or differentiate, neural representations of similar events to avoid interference at recall. Alternatively, forming overlapping representations of similar events, or integration, may aid recall by linking shared information between memories. It is currently unclear how the brain supports these seemingly conflicting functions of differentiation and integration. We used multivoxel pattern similarity analysis (MVPA) of fMRI data and neural-network analysis of visual similarity to examine how highly overlapping naturalistic events are encoded in patterns of cortical activity, and how the degree of differentiation versus integration at encoding affects later retrieval. Participants performed an episodic memory task in which they learned and recalled naturalistic video stimuli with high feature overlap. Visually similar videos were encoded in overlapping patterns of neural activity in temporal, parietal, and occipital regions, suggesting integration. We further found that encoding processes differentially predicted later reinstatement across the cortex. In visual processing regions in occipital cortex, greater differentiation at encoding predicted later reinstatement. Higher-level sensory processing regions in temporal and parietal lobes showed the opposite pattern, whereby highly integrated stimuli showed greater reinstatement. Moreover, integration in high-level sensory processing regions during encoding predicted greater accuracy and vividness at recall. These findings provide novel evidence that encoding-related differentiation and integration processes across the cortex have divergent effects on later recall of highly similar naturalistic events.

Functionally analogous body- and animacy-responsive areas are present in the dog (Canis familiaris) and human occipito-temporal lobe

Comparing the neural correlates of socio-cognitive skills across species provides insights into the evolution of the social brain and has revealed face- and body-sensitive regions in the primate temporal lobe. Although from a different lineage, dogs share convergent visuo-cognitive skills with humans and a temporal lobe which evolved independently in carnivorans. We investigated the neural correlates of face and body perception in dogs (N = 15) and humans (N = 40) using functional MRI. Combining univariate and multivariate analysis approaches, we found functionally analogous occipito-temporal regions involved in the perception of animate entities and bodies in both species and face-sensitive regions in humans. Though unpredicted, we also observed neural representations of faces compared to inanimate objects, and dog compared to human bodies in dog olfactory regions. These findings shed light on the evolutionary foundations of human and dog social cognition and the predominant role of the temporal lobe.

Cognitive tasks affect the relationship between representational pattern similarity and subsequent item memory in the hippocampus

Episodic memories are records of personally experienced events, coded neurally via the hippocampus and surrounding medial temporal lobe cortex. Information about the neural signal corresponding to a memory representation can be measured in fMRI data when the pattern across voxels is examined. Prior studies have found that similarity in the voxel patterns across repetition of a to-be-remembered stimulus predicts later memory retrieval, but the results are inconsistent across studies. The current study investigates the possibility that cognitive goals (defined here via the task instructions given to participants) during encoding affect the voxel pattern that will later support memory retrieval, and therefore that neural representations cannot be interpreted based on the stimulus alone. The behavioral results showed that exposure to variable cognitive tasks across repetition of events benefited subsequent memory retrieval. Voxel patterns in the hippocampus indicated a significant interaction between cognitive tasks (variable vs. consistent) and memory (remembered vs. forgotten) such that reduced voxel pattern similarity for repeated events with variable cognitive tasks, but not consistent cognitive tasks, supported later memory success. There was no significant interaction in neural pattern similarity between cognitive tasks and memory success in medial temporal cortices or lateral occipital cortex. Instead, higher similarity in voxel patterns in right medial temporal cortices was associated with later memory retrieval, regardless of cognitive task. In conclusion, we found that the relationship between pattern similarity across repeated encoding and memory success in the hippocampus (but not medial temporal lobe cortex) changes when the cognitive task during encoding does or does not vary across repetitions of the event.

Typicality in the brain during semantic and episodic memory decisions

Concept typicality is a key semantic dimension supporting the categorical organization of items based on their features, such that typical items share more features with other members of their category than atypical items, which are more distinctive. Typicality effects manifest in better accuracy and faster response times during categorization tasks, but higher performance for atypical items in episodic memory tasks, due to their distinctiveness. At a neural level, typicality has been linked to the anterior temporal lobe (ATL) and the inferior frontal gyrus (IFG) in semantic decision tasks, but patterns of brain activity during episodic memory tasks remain to be understood. We investigated the neural correlates of typicality in semantic and episodic memory to determine the brain regions associated with semantic typicality and uncover effects arising when items are reinstated during retrieval. In an fMRI study, 26 healthy young subjects first performed a category verification task on words representing typical and atypical concepts (encoding), and then completed a recognition memory task (retrieval). In line with previous literature, we observed higher accuracy and faster response times for typical items in the category verification task, while atypical items were better recognized in the episodic memory task. During category verification, univariate analyses revealed a greater involvement of the angular gyrus for typical items and the inferior frontal gyrus for atypical items. During the correct recognition of old items, regions belonging to the core recollection network were activated. We then compared the similarity of the representations from encoding to retrieval (ERS) using Representation Similarity Analyses. Results showed that typical items were reinstated more than atypical ones in several regions including the left precuneus and left anterior temporal lobe (ATL). This suggests that the correct retrieval of typical items requires finer-grained processing, evidenced by greater item-specific reinstatement, which is needed to resolve their confusability with other members of the category due to their higher feature similarity. Our findings confirm the centrality of the ATL in the processing of typicality while extending it to memory retrieval.

How our understanding of memory replay evolves

Memory reactivations and replay, widely reported in the hippocampus and cortex across species, have been implicated in memory consolidation, planning, and spatial and skill learning. Technological advances in electrophysiology, calcium imaging, and human neuroimaging techniques have enabled neuroscientists to measure large-scale neural activity with increasing spatiotemporal resolution and have provided opportunities for developing robust analytic methods to identify memory replay. In this article, we first review a large body of historically important and representative memory replay studies from the animal and human literature. We then discuss our current understanding of memory replay functions in learning, planning, and memory consolidation and further discuss the progress in computational modeling that has contributed to these improvements. Next, we review past and present analytic methods for replay analyses and discuss their limitations and challenges. Finally, looking ahead, we discuss some promising analytic methods for detecting nonstereotypical, behaviorally nondecodable structures from large-scale neural recordings. We argue that seamless integration of multisite recordings, real-time replay decoding, and closed-loop manipulation experiments will be essential for delineating the role of memory replay in a wide range of cognitive and motor functions.

Representations of Complex Contexts: A Role for Hippocampus

The hippocampus plays a critical role in supporting episodic memory, in large part by binding together experiences and items with surrounding contextual information. At present, however, little is known about the roles of different hippocampal subfields in supporting this item-context binding. To address this question, we constructed a task in which items were affiliated with differing types of context-cognitive associations that vary at the local, item level and membership in temporally organized lists that linked items together at a global level. Participants made item recognition judgments while undergoing high-resolution fMRI. We performed voxel pattern similarity analyses to answer the question of how human hippocampal subfields represent retrieved information about cognitive states and the time at which a past event took place. As participants recollected previously presented items, activity patterns in the CA23DG subregion carried information about prior cognitive states associated with these items. We found no evidence to suggest reinstatement of information about temporal context at the level of list membership, but exploratory analyses revealed representations of temporal context at a coarse level in conjunction with representations of cognitive contexts. Results are consistent with characterizations of CA23DG as a critical site for binding together items and contexts in the service of memory retrieval.

Distinct neural representations for prosocial and self-benefiting effort

Prosocial behaviors-actions that benefit others-are central to individual and societal well-being. Although the mechanisms underlying the financial and moral costs of prosocial behaviors are increasingly understood, this work has often ignored a key influence on behavior: effort. Many prosocial acts are effortful, and people are averse to the costs of exerting them. However, how the brain encodes effort costs when actions benefit others is unknown. During fMRI, participants completed a decision-making task where they chose in each trial whether to "work" and exert force (30%-70% of maximum grip strength) or "rest" (no effort) for rewards (2-10 credits). Crucially, on separate trials, they made these decisions either to benefit another person or themselves. We used a combination of multivariate representational similarity analysis and model-based univariate analysis to reveal how the costs of prosocial and self-benefiting efforts are processed. Strikingly, we identified a unique neural signature of effort in the anterior cingulate gyrus (ACCg) for prosocial acts, both when choosing to help others and when exerting force to benefit them. This pattern was absent for self-benefiting behaviors. Moreover, stronger, specific representations of prosocial effort in the ACCg were linked to higher levels of empathy and higher subsequent exerted force to benefit others. In contrast, the ventral tegmental area and ventral insula represented value preferentially when choosing for oneself and not for prosocial acts. These findings advance our understanding of the neural mechanisms of prosocial behavior, highlighting the critical role that effort has in the brain circuits that guide helping others.

Behavioral Studies Using Large-Scale Brain Networks – Methods and Validations

Mapping human behaviors to brain activity has become a key focus in modern cognitive neuroscience. As methods such as functional MRI (fMRI) advance cognitive scientists show an increasing interest in investigating neural activity in terms of functional connectivity and brain networks, rather than activation in a single brain region. Due to the noisy nature of neural activity, determining how behaviors are associated with specific neural signals is not well-established. Previous research has suggested graph theory techniques as a solution. Graph theory provides an opportunity to interpret human behaviors in terms of the topological organization of brain network architecture. Graph theory-based approaches, however, only scratch the surface of what neural connections relate to human behavior. Recently, the development of data-driven methods, e.g., machine learning and deep learning approaches, provide a new perspective to study the relationship between brain networks and human behaviors across the whole brain, expanding upon past literatures. In this review, we sought to revisit these data-driven approaches to facilitate our understanding of neural mechanisms and build models of human behaviors. We start with the popular graph theory approach and then discuss other data-driven approaches such as connectome-based predictive modeling, multivariate pattern analysis, network dynamic modeling, and deep learning techniques that quantify meaningful networks and connectivity related to cognition and behaviors. Importantly, for each topic, we discuss the pros and cons of the methods in addition to providing examples using our own data for each technique to describe how these methods can be applied to real-world neuroimaging data.

Shared vivid remembering: age-related differences in across-participants similarity of neural representations during encoding and retrieval

Recent advances in multivariate neuroimaging analyses have made possible the examination of the similarity of the neural patterns of activations measured across participants, but it has not been investigated yet whether such measure is age-sensitive. Here, in the scanner, young and older participants viewed scene pictures associated with labels. At test, participants were presented with the labels and were asked to recollect the associated picture. We used Pattern Similarity Analyses by which we compared patterns of neural activation during the encoding or the remembering of each picture of one participant with the averaged pattern of activation across the remaining participants. Results revealed that across-participants neural similarity was higher in young than in older adults in distributed occipital, temporal and parietal areas during encoding and retrieval. These findings demonstrate that an age-related reduction in specificity of neural activation is also evident when the similarity of neural representations is examined across participants.

Robustness of individualized inferences from longitudinal resting state EEG dynamics

Tracking how individual human brains change over extended timescales is crucial to clinical scenarios ranging from stroke recovery to healthy aging. The use of resting state (RS) activity for tracking is a promising possibility. However, it is unresolved how a person's RS activity over time can be decoded to distinguish neurophysiological changes from confounding cognitive variability. Here, we develop a method to screen RS activity changes for these confounding effects by formulating it as a problem of change classification. We demonstrate a novel solution to change classification by linking individual-specific change to inter-individual differences. Individual RS-EEG was acquired over five consecutive days including task states devised to simulate the effects of inter-day cognitive variation. As inter-individual differences are shaped by neurophysiological differences, the inter-individual differences in RS activity on one day were analyzed (using machine learning) to identify distinctive configurations in each individual's RS activity. Using this configuration as a decision-rule, an individual could be re-identified from 2-second samples of the instantaneous oscillatory power spectrum acquired on a different day both from RS and confounded-RS with a limited loss in accuracy. Importantly, the low loss in accuracy in cross-day vs same-day classification was achieved with classifiers that combined information from multiple frequency bands at channels across the scalp (with a concentration at characteristic fronto-central and occipital zones). Taken together, these findings support the technical feasibility of screening RS activity for confounding effects and the suitability of longitudinal RS for robust individualized inferences about neurophysiological change in health and disease.

Distinct medial-tempora lobe mechanisms of encoding and amygdala-mediated memory reinstatement for disgust and fear

Current models of episodic memory posit that retrieval involves the reenactment of encoding processes. Recent evidence has shown that this reinstatement process - indexed by subsequent encoding-retrieval similarity of brain activity patterns - is related to the activity in the hippocampus during encoding. However, we tend to re-experience emotional events in memory more richly than dull events. The role of amygdala - a critical hub of emotion processing - in reinstatement of emotional events was poorly understood. To investigate it, we leveraged a previously overlooked divergence in the role of amygdala in memory modulation by distinct emotions - disgust and fear. Here we used a novel paradigm in which participants encoded complex events (word pairs) and their memory was tested after 3 weeks, both phases during fMRI scanning. Using representational similarity analysis and univariate analyses, we show that the strength of amygdala activation during encoding was correlated with memory reinstatement of individual event representations in emotion-specific regions. Critically, amygdala modulated reinstatement more for disgust than fear. This was in line with other differences observed at the level of memory performance and neural mechanisms of encoding. Specifically, amygdala and perirhinal cortex were more involved during encoding of disgust-related events, whereas hippocampus and parahippocampal gyrus during encoding of fear-related events. Together, these findings shed a new light on the role of the amygdala and medial temporal lobe regions in encoding and reinstatement of specific emotional memories.

Finding positive meaning in memories of negative events adaptively updates memory

Finding positive meaning in past negative memories is associated with enhanced mental health. Yet it remains unclear whether it leads to updates in the memory representation itself. Since memory can be labile after retrieval, this leaves the potential for modification whenever its reactivated. Across four experiments, we show that positively reinterpreting negative memories adaptively updates them, leading to the re-emergence of positivity at future retrieval. Focusing on the positive aspects after negative recall leads to enhanced positive emotion and changes in memory content during recollection one week later, remaining even after two months. Consistent with a reactivation-induced reconsolidation account, memory updating occurs only after a reminder and twenty four hours, but not a one hour delay. Multi-session fMRI showed adaptive updates are reflected in greater hippocampal and ventral striatal pattern dissimilarity across retrievals. This research highlights the mechanisms by which updating of maladaptive memories occurs through a positive emotion-focused strategy.

The memory trace of a stressful episode

Stress influences episodic memory formation via noradrenaline and glucocorticoid effects on amygdala and hippocampus. A common finding is the improvement of memory for central aspects of a stressful episode. This is putatively related to changes in the neural representations of specific experiences, i.e., their memory traces. Here we show that the memory improvement for objects that were encountered in a stressful episode relates to differences in the neural representations of these objects in the amygdala. Using functional magnetic resonance imaging, we found that stress specifically altered the representations of central objects: compared to control objects, they became more similar to one another and more distinct from objects that were not part of this episode. Furthermore, higher similarity of central objects to the main stressor-the faces of the stress-inducing committee members-predicted better memory. This suggests that the central objects were closely integrated into a stressor-centered memory representation. Our findings provide mechanistic insights into how stress shapes the memory trace and have profound implications for neurocognitive models of stressful and emotional memory.

Developmental Differences in Affective Representation Between Prefrontal and Subcortical Structures

Developmental studies have identified differences in prefrontal and subcortical affective structures between children and adults, which correspond with observed cognitive and behavioral maturations from relatively simplistic emotional experiences and expressions to more nuanced, complex ones. However, developmental changes in the neural representation of emotions have not yet been well explored. It stands to reason that adults and children may demonstrate observable differences in the representation of affect within key neurological structures implicated in affective cognition. Forty-five participants (25 children; 20 adults) passively viewed positive, negative, and neutral clips from popular films while undergoing functional magnetic resonance imaging (fMRI). Using representational similarity analysis (RSA) to measure variability in neural pattern similarity, we found developmental differences between children and adults in the amygdala, nucleus accumbens (NAcc), and ventromedial prefrontal cortex (vmPFC), such that children generated less pattern similarity within subcortical structures relative to the vmPFC; a phenomenon not replicated among their older counterparts. Furthermore, children generated valence-specific differences in representational patterns across regions; these valence-specific patterns were not found in adults. These results may suggest that affective representations grow increasingly dissimilar over development as individuals mature from visceral affective responses to more evaluative analyses.

Enhanced reinstatement of naturalistic event memories due to hippocampal-network-targeted stimulation

Episodic memory involves the reinstatement of distributed patterns of brain activity present when events were initially experienced. The hippocampus is thought to coordinate reinstatement via its interactions with a network of brain regions, but this hypothesis has not been causally tested in humans. The current study directly tested the involvement of the hippocampal network in reinstatement using network-targeted noninvasive stimulation. We measured reinstatement of multi-voxel patterns of functional magnetic resonance imaging (fMRI) activity during encoding and retrieval of naturalistic video clips depicting everyday activities. Reinstatement of video-specific activity patterns was robust in posterior parietal and occipital areas previously implicated in event reinstatement. Theta-burst stimulation targeting the hippocampal network increased video-specific reinstatement of fMRI activity patterns in occipital cortex and improved memory accuracy relative to stimulation of a control out-of-network location. Furthermore, stimulation targeting the hippocampal network influenced the trial-by-trial relationship between hippocampal activity during encoding and later reinstatement in occipital cortex. These findings implicate the hippocampal network in the reinstatement of spatially distributed patterns of event-specific activity and identify a role for the hippocampus in encoding complex naturalistic events that later undergo cortical reinstatement.

Simultaneous spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder

Exploring the spatial patterns and temporal dynamics of human brain activity has been of great interest, in the quest to better understand connectome-scale brain networks. Though modeling spatial and temporal patterns of functional brain networks have been researched for a long time, the development of a unified and simultaneous spatial-temporal model has yet to be realized. For instance, although some deep learning methods have been proposed recently in order to model functional brain networks, most of them can only represent either spatial or temporal perspective of functional Magnetic Resonance Imaging (fMRI) data and rarely model both domains simultaneously. Due to the recent success in applying sequential auto-encoders for brain decoding, in this paper, we propose a deep sparse recurrent auto-encoder (DSRAE) to be applied unsupervised to learn spatial patterns and temporal fluctuations of brain networks at the same time. The proposed DSRAE was evaluated and validated based on three tasks of the publicly available Human Connectome Project (HCP) fMRI dataset, resulting with promising evidence. To the best of our knowledge, the proposed DSRAE is among the early efforts in developing unified models that can extract connectome-scale spatial-temporal networks from 4D fMRI data simultaneously.

Mnemonic prediction errors bias hippocampal states

When our experience violates our predictions, it is adaptive to upregulate encoding of novel information, while down-weighting retrieval of erroneous memory predictions to promote an updated representation of the world. We asked whether mnemonic prediction errors promote hippocampal encoding versus retrieval states, as marked by distinct network connectivity between hippocampal subfields. During fMRI scanning, participants were cued to internally retrieve well-learned complex room-images and were then presented with either an identical or a modified image (0-4 changes). In the left hemisphere, we find that CA1-entorhinal connectivity increases, and CA1-CA3 connectivity decreases, with the number of changes. Further, in the left CA1, the similarity between activity patterns during cued-retrieval of the learned room and during the image is lower when the image includes changes, consistent with a prediction error signal in CA1. Our findings provide a mechanism by which mnemonic prediction errors may drive memory updating—by biasing hippocampal states.

When our expectations are violated, it is adaptive to update our internal models to improve predictions in the future. Here, the authors show that during mnemonic violations, hippocampal networks are biased towards an encoding state and away from a retrieval state to potentially update these predictions.

Pattern Similarity Analyses of FrontoParietal Task Coding: Individual Variation and Genetic Influences

Pattern similarity analyses are increasingly used to characterize coding properties of brain regions, but relatively few have focused on cognitive control processes in FrontoParietal regions. Here, we use the Human Connectome Project (HCP) N-back task functional magnetic resonance imaging (fMRI) dataset to examine individual differences and genetic influences on the coding of working memory load (0-back, 2-back) and perceptual category (Face, Place). Participants were grouped into 105 monozygotic twin, 78 dizygotic twin, 99 nontwin sibling, and 100 unrelated pairs. Activation pattern similarity was used to test the hypothesis that FrontoParietal regions would have higher similarity for same load conditions, while Visual regions would have higher similarity in same perceptual category conditions. Results confirmed this highly robust regional double dissociation in neural coding, which also predicted individual differences in behavioral performance. In pair-based analyses, anatomically selective genetic relatedness effects were observed: relatedness predicted greater activation pattern similarity in FrontoParietal only for load coding and in Visual only for perceptual coding. Further, in related pairs, the similarity of load coding in FrontoParietal regions was uniquely associated with behavioral performance. Together, these results highlight the power of task fMRI pattern similarity analyses for detecting key coding and heritability features of brain regions.