Distributed representations in memory: insights from functional brain imaging

Goal-directed learning in adolescence: neurocognitive development and contextual influences

Adolescence is a time during which we transition to independence, explore new activities and begin pursuit of major life goals. Goal-directed learning, in which we learn to perform actions that enable us to obtain desired outcomes, is central to many of these processes. Currently, our understanding of goal-directed learning in adolescence is itself in a state of transition, with the scientific community grappling with inconsistent results. When we examine metrics of goal-directed learning through the second decade of life, we find that many studies agree there are steady gains in performance in the teenage years, but others report that adolescent goal-directed learning is already adult-like, and some find adolescents can outperform adults. To explain the current variability in results, sophisticated experimental designs are being applied to test learning in different contexts. There is also increasing recognition that individuals of different ages and in different states will draw on different neurocognitive systems to support goal-directed learning. Through adoption of more nuanced approaches, we can be better prepared to recognize and harness adolescent strengths and to decipher the purpose (or goals) of adolescence itself.

Decoding context memories for threat in large-scale neural networks

Humans are often tasked with determining the degree to which a given situation poses threat. Salient cues present during prior events help bring online memories for context, which plays an informative role in this process. However, it is relatively unknown whether and how individuals use features of the environment to retrieve context memories for threat, enabling accurate inferences about the current level of danger/threat (i.e. retrieve appropriate memory) when there is a degree of ambiguity surrounding the present context. We leveraged computational neuroscience approaches (i.e. independent component analysis and multivariate pattern analyses) to decode large-scale neural network activity patterns engaged during learning and inferring threat context during a novel functional magnetic resonance imaging task. Here, we report that individuals accurately infer threat contexts under ambiguous conditions through neural reinstatement of large-scale network activity patterns (specifically striatum, salience, and frontoparietal networks) that track the signal value of environmental cues, which, in turn, allows reinstatement of a mental representation, primarily within a ventral visual network, of the previously learned threat context. These results provide novel insight into distinct, but overlapping, neural mechanisms by which individuals may utilize prior learning to effectively make decisions about ambiguous threat-related contexts as they navigate the environment.

Methodological advice for the young at heart investigator: Triangulation to build better foundations

In medicine and science, one is typically taught the main theories in a discipline or field along with standard models before receiving more instructions on how to apply certain methods. The aim of this work is not to address one method, but rather methodology, the study and evaluation of methods, by taking a philosophy of science detour. In this, a critique of biomedicine will be used as a starting point to address some positions regarding reductionism, specifying notions such as systems and mechanisms, as well as regarding the mind-body problem discussing psychosomatic medicine and psychoneuroimmunology. Some recommendations to make science more pluralistic, robust and translationally-relevant will then be made as a way to foster constructive debates on reductionism and the mind-body problem and, in turn, favor more interdisciplinary research.

dcSBM: A federated constrained source-based morphometry approach for multivariate brain structure mapping

The examination of multivariate brain morphometry patterns has gained attention in recent years, especially for their powerful exploratory capabilities in the study of differences between patients and controls. Among the many existing methods and tools for the analysis of brain anatomy based on structural magnetic resonance imaging data, data-driven source-based morphometry (SBM) focuses on the exploratory detection of such patterns. Here, we implement a semi-blind extension of SBM, called constrained source-based morphometry (constrained SBM), which enables the extraction of maximally independent reference-alike sources using the constrained independent component analysis (ICA) approach. To do this, we combine SBM with a set of reference components covering the full brain, derived from a large independent data set (UKBiobank), to provide a fully automated SBM framework. This also allows us to implement a federated version of constrained SBM (cSBM) to allow analysis of data that is not locally accessible. In our proposed decentralized constrained source-based morphometry (dcSBM), the original data never leaves the local site. Each site operates constrained ICA on its private local data using a common distributed computation platform. Next, an aggregator/master node aggregates the results estimated from each local site and applies statistical analysis to estimate the significance of the sources. Finally, we utilize two additional multisite patient data sets to validate our model by comparing the resulting group difference estimates from both cSBM and dcSBM.

Testing cognitive theories with multivariate pattern analysis of neuroimaging data

Multivariate pattern analysis (MVPA) has emerged as a powerful method for the analysis of functional magnetic resonance imaging, electroencephalography and magnetoencephalography data. The new approaches to experimental design and hypothesis testing afforded by MVPA have made it possible to address theories that describe cognition at the functional level. Here we review a selection of studies that have used MVPA to test cognitive theories from a range of domains, including perception, attention, memory, navigation, emotion, social cognition and motor control. This broad view reveals properties of MVPA that make it suitable for understanding the 'how' of human cognition, such as the ability to test predictions expressed at the item or event level. It also reveals limitations and points to future directions.

Prediction errors drive dynamic changes in neural patterns that guide behavior

Learning describes the process by which our internal expectation models of the world are updated by surprising outcomes (prediction errors [PEs]) to improve predictions of future events. However, the mechanisms through which error signals dynamically influence existing neural representations are unknown. Here, we use functional magnetic resonance imaging (fMRI) in humans solving a two-step Markov decision task to investigate changes in neural activation patterns following PEs. Using a dynamic multivariate pattern analysis, we can show that PE-related fMRI responses in error-coding regions predict trial-by-trial changes in multivariate neural patterns in the orbitofrontal cortex, the precuneus, and the ventromedial prefrontal cortex (vmPFC). Importantly, the dynamics of these pattern changes in the vmPFC also predicted upcoming changes in choice strategies and thus highlight the importance of these pattern changes for behavior.

Neural similarity between overlapping events at learning differentially affects reinstatement across the cortex

Episodic memory often involves high overlap between the actors, locations, and objects of everyday events. Under some circumstances, it may be beneficial to distinguish, or differentiate, neural representations of similar events to avoid interference at recall. Alternatively, forming overlapping representations of similar events, or integration, may aid recall by linking shared information between memories. It is currently unclear how the brain supports these seemingly conflicting functions of differentiation and integration. We used multivoxel pattern similarity analysis (MVPA) of fMRI data and neural-network analysis of visual similarity to examine how highly overlapping naturalistic events are encoded in patterns of cortical activity, and how the degree of differentiation versus integration at encoding affects later retrieval. Participants performed an episodic memory task in which they learned and recalled naturalistic video stimuli with high feature overlap. Visually similar videos were encoded in overlapping patterns of neural activity in temporal, parietal, and occipital regions, suggesting integration. We further found that encoding processes differentially predicted later reinstatement across the cortex. In visual processing regions in occipital cortex, greater differentiation at encoding predicted later reinstatement. Higher-level sensory processing regions in temporal and parietal lobes showed the opposite pattern, whereby highly integrated stimuli showed greater reinstatement. Moreover, integration in high-level sensory processing regions during encoding predicted greater accuracy and vividness at recall. These findings provide novel evidence that encoding-related differentiation and integration processes across the cortex have divergent effects on later recall of highly similar naturalistic events.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Individual-specific memory reinstatement patterns within human face-selective cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions involved in the sensory processing of prior events are reactivated to support this perception of the past. Recently, several studies have emphasized potential transformations in the spatial organization of reinstated activity patterns. In particular, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with those during perception. However, it is not clear that such transformations occur universally, with evidence lacking for other important stimulus categories, particularly faces. Critical to addressing these questions, and to studies of reinstatement more broadly, is the growing importance of considering meaningful variations in the organization of sensory systems across individuals. Therefore, we conducted a multi-session neuroimaging study to first carefully map individual participants face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the correspondence of activity patterns during face memory encoding and retrieval. Our results showed distinct configurations of face-selective regions within the VTC across individuals. While a significant degree of overlap was observed between face perception and memory encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions. Importantly, these activity patterns were consistently tied to individual-specific neural substrates, but did not show any consistent direction of spatial transformation (e.g., anteriorization). To provide further insight to these findings, we also report on unique human intracranial recordings from VTC under the same experimental conditions. Our findings highlight the importance of considering individual variations in functional neuroanatomy in the context of assessing the nature of cortical reinstatement. Consideration of such factors will be important for establishing general principles shaping the neural transformations that occur from perception to memory.

Identifying causal subsequent memory effects

Over 40 y of accumulated research has detailed associations between neuroimaging signals measured during a memory encoding task and later memory performance, across a variety of brain regions, measurement tools, statistical approaches, and behavioral tasks. But the interpretation of these subsequent memory effects (SMEs) remains unclear: if the identified signals reflect cognitive and neural mechanisms of memory encoding, then the underlying neural activity must be causally related to future memory. However, almost all previous SME analyses do not control for potential confounders of this causal interpretation, such as serial position and item effects. We collect a large fMRI dataset and use an experimental design and analysis approach that allows us to statistically adjust for nearly all known exogenous confounding variables. We find that, using standard approaches without adjustment, we replicate several univariate and multivariate subsequent memory effects and are able to predict memory performance across people. However, we are unable to identify any signal that reliably predicts subsequent memory after adjusting for confounding variables, bringing into doubt the causal status of these effects. We apply the same approach to subjects' judgments of learning collected following an encoding period and show that these behavioral measures of mnemonic status do predict memory after adjustments, suggesting that it is possible to measure signals near the time of encoding that reflect causal mechanisms but that existing neuroimaging measures, at least in our data, may not have the precision and specificity to do so.

Altered engagement of the speech motor network is associated with reduced phonological working memory in autism

Nonword repetition, a common clinical measure of phonological working memory, involves component processes of speech perception, working memory, and speech production. Autistic children often show behavioral challenges in nonword repetition, as do many individuals with communication disorders. It is unknown which subprocesses of phonological working memory are vulnerable in autistic individuals, and whether the same brain processes underlie the transdiagnostic difficulty with nonword repetition. We used functional magnetic resonance imaging (fMRI) to investigate the brain bases for nonword repetition challenges in autism. We compared activation during nonword repetition in functional brain networks subserving speech perception, working memory, and speech production between neurotypical and autistic children. Autistic children performed worse than neurotypical children on nonword repetition and had reduced activation in response to increasing phonological working memory load in the supplementary motor area. Multivoxel pattern analysis within the speech production network classified shorter vs longer nonword-repetition trials less accurately for autistic than neurotypical children. These speech production motor-specific differences were not observed in a group of children with reading disability who had similarly reduced nonword repetition behavior. These findings suggest that atypical function in speech production brain regions may contribute to nonword repetition difficulties in autism.

Enhancing learning and retention with distinctive virtual reality environments and mental context reinstatement

Memory is inherently context-dependent: internal and environmental cues become bound to learnt information, and the later absence of these cues can impair recall. Here, we developed an approach to leverage context-dependence to optimise learning of challenging, interference-prone material. While navigating through desktop virtual reality (VR) contexts, participants learnt 80 foreign words in two phonetically similar languages. Those participants who learnt each language in its own unique context showed reduced interference and improved one-week retention (92%), relative to those who learnt the languages in the same context (76%)-however, this advantage was only apparent if participants subjectively experienced VR-based contexts as "real" environments. A follow-up fMRI experiment confirmed that reinstatement of brain activity patterns associated with the original encoding context during word retrieval was associated with improved recall performance. These findings establish that context-dependence can be harnessed with VR to optimise learning and showcase the important role of mental context reinstatement.

On the Application of Developmental Cognitive Neuroscience in Educational Environments

The paper overviews components of neurologic processing efficiencies to develop innovative methodologies and thinking to school-based applications and changes in educational leadership based on sound findings in the cognitive neurosciences applied to schools and learners. Systems science can allow us to better manage classroom-based learning and instruction on the basis of relatively easily evaluated efficiencies or inefficiencies and optimization instead of simply examining achievement. "Medicalizing" the learning process with concepts such as "learning disability" or employing grading methods such as pass-fail does little to aid in understanding the processes that learners employ to acquire, integrate, remember, and apply information learned. The paper endeavors to overview and provided reference to tools that can be employed that allow a better focus on nervous system-based strategic approaches to classroom learning.

Machine Learning as a Model for Cultural Learning: Teaching an Algorithm What it Means to be Fat

Public culture is a powerful source of cognitive socialization; for example, media language is full of meanings about body weight. Yet it remains unclear how individuals process meanings in public culture. We suggest that schema learning is a core mechanism by which public culture becomes personal culture. We propose that a burgeoning approach in computational text analysis - neural word embeddings - can be interpreted as a formal model for cultural learning. Embeddings allow us to empirically model schema learning and activation from natural language data. We illustrate our approach by extracting four lower-order schemas from news articles: the gender, moral, health, and class meanings of body weight. Using these lower-order schemas we quantify how words about body weight "fill in the blanks" about gender, morality, health, and class. Our findings reinforce ongoing concerns that machine-learning models (e.g., of natural language) can encode and reproduce harmful human biases.

Neural Reinstatement of Overlapping Memories in Young and Older Adults

When we update our episodic memories with new information, mnemonic competition between old and new memories may result because of the presence of shared features. Behavioral studies suggest that this competition can lead to proactive interference, resulting in unsuccessful memory updating, particularly for older adults. It is difficult with behavioral data alone to measure the reactivation of old, overlapping memories during retrieval and its impact on memory for new memories. Here, we applied encoding-retrieval representational similarity (ERS) analysis to EEG data to estimate event-specific encoding-related neural reinstatement of old associations during the retrieval of new ones and its impact on memory for new associations in young and older adults. Our results showed that older adults' new associative memory performance was more negatively impacted by proactive interference from old memories than that of young adults. In both age groups, ERS for old associative memories was greater for trials for which new associative memories were forgotten than remembered. In contrast, ERS for new associative memories was greater when they were remembered than forgotten. In addition, older adults showed relatively attenuated target (i.e., new associates) and lure (i.e., old associates) ERS effects compared to younger adults. Collectively, these results suggest that the neural reinstatement of interfering memories during retrieval contributes to proactive interference across age, whereas overall attenuated ERS effect in older adults might reflect their reduced memory fidelity.

Neural Evidence for Different Types of Position Coding Strategies in Spatial Working Memory

Sustained neural activity during the delay phase of spatial working memory tasks is compelling evidence for the neural correlate of active storage and maintenance of spatial information, however, it does not provide insight into specific mechanisms of spatial coding. This activity may reflect a range of processes, such as maintenance of a stimulus position or a prepared motor response plan. The aim of our study was to examine neural evidence for the use of different coding strategies, depending on the characteristics and demands of a spatial working memory task. Thirty-one (20 women, 23 ± 5 years) and 44 (23 women, 21 ± 2 years) participants performed a spatial working memory task while we measured their brain activity using fMRI in two separate experiments. Participants were asked to remember the position of a briefly presented target stimulus and, after a delay period, to use a joystick to indicate either the position of the remembered target or an indicated non-matching location. The task was designed so that the predictability of the response could be manipulated independently of task difficulty and memory retrieval process. We were particularly interested in contrasting conditions in which participants (i) could use prospective coding of the motor response or (ii) had to rely on retrospective sensory information. Prospective motor coding was associated with activity in somatomotor, premotor, and motor cortices and increased integration of brain activity with and within the somatomotor network. In contrast, retrospective sensory coding was associated with increased activity in parietal regions and increased functional connectivity with and within secondary visual and dorsal attentional networks. The observed differences in activation levels, dynamics of differences over trial duration, and integration of information within and between brain networks provide compelling evidence for the use of complementary spatial working memory strategies optimized to meet task demands.

Suppression weakens unwanted memories via a sustained reduction of neural reactivation

Aversive events sometimes turn into intrusive memories. However, prior evidence indicates that such memories can be controlled via a mechanism of retrieval suppression. Here, we test the hypothesis that suppression exerts a sustained influence on memories by deteriorating their neural representations. This deterioration, in turn, would hinder their subsequent reactivation and thus impoverish the vividness with which they can be recalled. In an fMRI study, participants repeatedly suppressed memories of aversive scenes. As predicted, this process rendered the memories less vivid. Using a pattern classifier, we observed that suppression diminished the neural reactivation of scene information both globally across the brain and locally in the parahippocampal cortices. Moreover, the decline in vividness was associated with reduced reinstatement of unique memory representations in right parahippocampal cortex. These results support the hypothesis that suppression weakens memories by causing a sustained reduction in the potential to reactivate their neural representations.

Spectral Pattern Similarity Analysis: Tutorial and Application in Developmental Cognitive Neuroscience

The human brain encodes information in neural activation patterns. While standard approaches to analyzing neural data focus on brain (de-)activation (e.g., regarding the location, timing, or magnitude of neural responses), multivariate neural pattern similarity analyses target the informational content represented by neural activity. In adults, a number of representational properties have been identified that are linked to cognitive performance, in particular the stability, distinctiveness, and specificity of neural patterns. However, although growing cognitive abilities across childhood suggest advancements in representational quality, developmental studies still rarely utilize information-based pattern similarity approaches, especially in electroencephalography (EEG) research. Here, we provide a comprehensive methodological introduction and step-by-step tutorial for pattern similarity analysis of spectral (frequency-resolved) EEG data including a publicly available pipeline and sample dataset with data from children and adults. We discuss computation of single-subject pattern similarities and their statistical comparison at the within-person to the between-group level as well as the illustration and interpretation of the results. This tutorial targets both novice and more experienced EEG researchers and aims to facilitate the usage of spectral pattern similarity analyses, making these methodologies more readily accessible for (developmental) cognitive neuroscientists.

Computational mechanisms of distributed value representations and mixed learning strategies

Learning appropriate representations of the reward environment is challenging in the real world where there are many options, each with multiple attributes or features. Despite existence of alternative solutions for this challenge, neural mechanisms underlying emergence and adoption of value representations and learning strategies remain unknown. To address this, we measure learning and choice during a multi-dimensional probabilistic learning task in humans and trained recurrent neural networks (RNNs) to capture our experimental observations. We find that human participants estimate stimulus-outcome associations by learning and combining estimates of reward probabilities associated with the informative feature followed by those of informative conjunctions. Through analyzing representations, connectivity, and lesioning of the RNNs, we demonstrate this mixed learning strategy relies on a distributed neural code and opponency between excitatory and inhibitory neurons through value-dependent disinhibition. Together, our results suggest computational and neural mechanisms underlying emergence of complex learning strategies in naturalistic settings.

Reinstatement of Event Details during Episodic Simulation in the Hippocampus

According to the constructive episodic simulation hypothesis, episodic simulation (i.e., imagining specific novel future episodes) draws on some of the same neurocognitive processes that support episodic memory (i.e., recalling specific past episodes). Episodic retrieval supports the ability to simulate future experiences by providing access to episodic details (e.g., the people and locations that comprise memories) that can be recombined in new ways. In the current functional neuroimaging study, we test this hypothesis by examining whether the hippocampus, a region implicated in the reinstatement of episodic information during memory, supports reinstatement of episodic information during simulation. Employing a multivoxel pattern similarity analysis, we interrogated the similarity between hippocampal neural patterns during memory and simulation at the level of individual event details. Our findings indicate that the hippocampus supports the reinstatement of detail-specific information from episodic memory during simulation, with the level of reinstatement contributing to the subjective experience of simulated details.

Separation of item and context in item-method directed forgetting

Contextual information plays a critical role in directed forgetting (DF) of lists of items, whereas DF of individual items has been primarily associated with item-level processing. This study was designed to investigate whether context processing also contributes to the forgetting of individual items. Participants first viewed a series of words, with task-irrelevant scene images (used as "context tags") interspersed between them. Later, these words reappeared without the scenes and were followed by an instruction to remember or forget that word. Multivariate pattern analyses of fMRI data revealed that the reactivation of context information associated with the studied words (i.e., scene-related activity) was greater whereas the item-related information diminished after a forget instruction compared to a remember instruction. Critically, we found the magnitude of the separation between item information and context information predicted successful forgetting. These results suggest that the unbinding of an item from its context may support the intention to forget, and more generally they establish that contextual processing indeed contributes to item-method DF.

The presence of semantic content in a visual recognition memory task reduces the severity of neglect

Patients with right hemisphere damage often show a lateral bias when asked to report the left side of mental images held in visual working memory (i.e. representational neglect). The neural basis of representational neglect is not well understood. One hypothesis suggests that it reflects a deficit in attentional-exploratory mechanisms, i.e. an inability to direct attention to the left side of the image. Another proposition states that intact visual working memory (VWM) is necessary for correctly creating a mental image. Here we examined two components of VWM in patients with unilateral spatial neglect (USN): memory for identity, and memory for spatial position. We manipulated the strength of memory representations by presenting two distinct categories of objects, in separate blocks. These were familiar namable objects (fruits, etc.), and unfamiliar abstract objects. The former category elicits stronger working-memory traces, thanks to preexisting visual and semantic representations in long-term memory. We hypothesized that if USN patients show a lateralized deficit in VWM, it should be more pronounced for abstract objects, due to their weaker working-memory traces. Importantly, to isolate a spatially lateralized deficit in memory from a failure to fully perceive the object-arrays, we ensured that all included patients perceived every item during the encoding phase. We used a working-memory task: participants viewed object arrays and had to memorize items' identities and spatial positions. Then, single objects were presented requiring 'old/new' recognition, and retrieval of 'old' items' original positions. Our results show a lateral bias in patients' recognition-memory performance. Remarkably, it was threefold milder for namable objects compared to abstract objects. We conclude that VWM lateralized deficit is substantial in USN patients and could play a role in representational neglect.

Enhanced reinstatement of naturalistic event memories due to hippocampal-network-targeted stimulation

Episodic memory involves the reinstatement of distributed patterns of brain activity present when events were initially experienced. The hippocampus is thought to coordinate reinstatement via its interactions with a network of brain regions, but this hypothesis has not been causally tested in humans. The current study directly tested the involvement of the hippocampal network in reinstatement using network-targeted noninvasive stimulation. We measured reinstatement of multi-voxel patterns of functional magnetic resonance imaging (fMRI) activity during encoding and retrieval of naturalistic video clips depicting everyday activities. Reinstatement of video-specific activity patterns was robust in posterior parietal and occipital areas previously implicated in event reinstatement. Theta-burst stimulation targeting the hippocampal network increased video-specific reinstatement of fMRI activity patterns in occipital cortex and improved memory accuracy relative to stimulation of a control out-of-network location. Furthermore, stimulation targeting the hippocampal network influenced the trial-by-trial relationship between hippocampal activity during encoding and later reinstatement in occipital cortex. These findings implicate the hippocampal network in the reinstatement of spatially distributed patterns of event-specific activity and identify a role for the hippocampus in encoding complex naturalistic events that later undergo cortical reinstatement.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.^1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.^7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).¹⁵¹⁶ Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.^17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

Effects of age on goal-dependent modulation of episodic memory retrieval

Retrieval gating refers to the ability to modulate the retrieval of features of a single memory episode according to behavioral goals. Recent findings demonstrate that younger adults engage retrieval gating by attenuating the representation of task-irrelevant features of an episode. Here, we examine whether retrieval gating varies with age. Younger and older adults incidentally encoded words superimposed over scenes or scrambled backgrounds that were displayed in one of three spatial locations. Participants subsequently underwent fMRI as they completed two memory tasks: the background task, which tested memory for the word's background, and the location task, testing memory for the word's location. Employing univariate and multivariate approaches, we demonstrated that younger, but not older adults, exhibited attenuated reinstatement of scene information when it was goal-irrelevant (during the location task). Additionally, in younger adults only, the strength of scene reinstatement in the parahippocampal place area during the background task was related to item and source memory performance. Together, these findings point to an age-related decline in the ability to engage retrieval gating.

Delayed Judgments of Learning Are Associated With Activation of Information From Past Experiences: A Neurobiological Examination

Research in metacognition suggests that the information people use to predict their memory performance can vary depending on the contexts in which they make their predictions. For example, if people judge their memories after a delay from initial encoding, they may be more likely to use retrieved information about the past encoding experience than if they judged memories immediately after encoding. Although this seems intuitive, past behavioral and neuroimaging work has not tested whether delayed memory judgments are more strongly coupled with information about past experiences than immediate memory judgments. We scanned participants using functional MRI while they encoded paired associates and made predictions about their future memory performance either immediately after encoding or after a delay. Consistent with the hypothesis that people use retrieved information about past experiences to inform delayed memory judgments, our results showed that activation patterns associated with past experience were more strongly coupled with delayed memory judgments than with immediate ones.

Stress Impairs Episodic Retrieval by Disrupting Hippocampal and Cortical Mechanisms of Remembering

Despite decades of science investigating the neural underpinnings of episodic memory retrieval, a critical question remains: how does stress influence remembering and the neural mechanisms of recollection in humans? Here, we used functional magnetic resonance imaging and multivariate pattern analyses to examine the effects of acute stress during retrieval. We report that stress reduced the probability of recollecting the details of past experience, and that this impairment was driven, in part, by a disruption of the relationship between hippocampal activation, cortical reinstatement, and memory performance. Moreover, even memories expressed with high confidence were less accurate under stress, and this stress-induced decline in accuracy was explained by reduced posterior hippocampal engagement despite similar levels of category-level cortical reinstatement. Finally, stress degraded the relationship between the engagement of frontoparietal control networks and retrieval decision uncertainty. Collectively, these findings demonstrate the widespread consequences of acute stress on the neural systems of remembering.

Age Differences In Retrieval-Related Reinstatement Reflect Age-Related Dedifferentiation At Encoding

Age-related reductions in neural selectivity have been linked to cognitive decline. We examined whether age differences in the strength of retrieval-related cortical reinstatement could be explained by analogous differences in neural selectivity at encoding, and whether reinstatement was associated with memory performance in an age-dependent or an age-independent manner. Young and older adults underwent fMRI as they encoded words paired with images of faces or scenes. During a subsequent scanned memory test participants judged whether test words were studied or unstudied and, for words judged studied, also made a source memory judgment about the associated image category. Using multi-voxel pattern similarity analyses, we identified robust evidence for reduced scene reinstatement in older relative to younger adults. This decline was however largely explained by age differences in neural differentiation at encoding; moreover, a similar relationship between neural selectivity at encoding and retrieval was evident in young participants. The results suggest that, regardless of age, the selectivity with which events are neurally processed at the time of encoding can determine the strength of retrieval-related cortical reinstatement.

Properties and temporal dynamics of choice- and action-predictive signals during item recognition decisions

Decision-making is in the service of action regardless of whether the decision concerns perceptual information, goods or memories. Compared to recent advances in the neurobiology of perceptual or value-based decisions, however, the neural bases supporting the sampling of evidence in long-term memory, and the transformation of memory-based decisions into appropriate actions, are still poorly understood. In the present fMRI study, we used multivariate pattern analysis to investigate the temporal dynamics of choice- and action-predictive signals during an item recognition task that manipulated the association between memory choices (old/new) and motor responses (eye/hand) across subjects. Choice-predictive activity was mainly observed in striatal, lateral prefrontal and lateral parietal regions, was sensitive to the amount of decision evidence and showed a rapid increase after stimulus onset, followed by a fast decay. Action-predictive signals were found in primary sensory motor, premotor and occipito-parietal regions, were generally observed at the end of the decision phase and were not modulated by decision evidence. These findings suggest that a memory decision variable, potentially represented in a fronto-striato-parietal network, is not directly transformed into an action plan as often observed in perceptual decisions. Regions exhibiting choice predictive activity, and especially the striatum, however, also showed a second peak of decision-related activity that, unlike pure choice- or action-predictive signals, depended on the particular choice-response association. This second peak of activity in the striatum might represent the neural signature of the transformation of a memory decision into an appropriate motor response based on the specific choice-response association.

Mental imagery in animals: Learning, memory, and decision-making in the face of missing information

When we open our eyes, we see a world filled with objects and events. Yet, due to occlusion of some objects by others, we only have partial perceptual access to the events that transpire around us. I discuss the body of research on mental imagery in animals. I first cover prior studies of mental rotation in pigeons and imagery using working memory procedures first developed for human studies. Next, I discuss the seminal work on a type of learning called mediated conditioning in rats. I then provide more in-depth coverage of work from my lab suggesting that rats can use imagery to fill in missing details of the world that are expected but hidden from perception. We have found that rats make use of an active expectation (i.e., an image) of a hidden visual event. I describe the behavioral and neurobiological studies investigating the use of a mental image, its theoretical basis, and its connections to current human cognitive neuroscience research on episodic memory, imagination, and mental simulations. Collectively, the reviewed literature provides insight into the mechanisms that mediate the flexible use of an image during ambiguous situations. I position this work in the broader scientific and philosophical context surrounding the concept of mental imagery in human and nonhuman animals.

Eye Movement Reinstatement and Neural Reactivation During Mental Imagery

Half a century ago, Donald Hebb posited that mental imagery is a constructive process that emulates perception. Specifically, Hebb claimed that visual imagery results from the reactivation of neural activity associated with viewing images. He also argued that neural reactivation and imagery benefit from the re-enactment of eye movement patterns that first occurred at viewing (fixation reinstatement). To investigate these claims, we applied multivariate pattern analyses to functional MRI (fMRI) and eye tracking data collected while healthy human participants repeatedly viewed and visualized complex images. We observed that the specificity of neural reactivation correlated positively with vivid imagery and with memory for stimulus image details. Moreover, neural reactivation correlated positively with fixation reinstatement, meaning that image-specific eye movements accompanied image-specific patterns of brain activity during visualization. These findings support the conception of mental imagery as a simulation of perception, and provide evidence consistent with the supportive role of eye movement in neural reactivation.

Feature-specific neural reactivation during episodic memory

We present a multi-voxel analytical approach, feature-specific informational connectivity (FSIC), that leverages hierarchical representations from a neural network to decode neural reactivation in fMRI data collected while participants performed an episodic visual recall task. We show that neural reactivation associated with low-level (e.g. edges), high-level (e.g. facial features), and semantic (e.g. “terrier”) features occur throughout the dorsal and ventral visual streams and extend into the frontal cortex. Moreover, we show that reactivation of both low- and high-level features correlate with the vividness of the memory, whereas only reactivation of low-level features correlates with recognition accuracy when the lure and target images are semantically similar. In addition to demonstrating the utility of FSIC for mapping feature-specific reactivation, these findings resolve the contributions of low- and high-level features to the vividness of visual memories and challenge a strict interpretation the posterior-to-anterior visual hierarchy.

Memory recollection involves reactivation of neural activity that occurred during the recalled experience. Here, the authors show that neural reactivation can be decomposed into visual-semantic features, is widely synchronized throughout the brain, and predicts memory vividness and accuracy.

Revealing the information contents of memory within the stimulus information representation framework

The information contents of memory are the cornerstone of the most influential models in cognition. To illustrate, consider that in predictive coding, a prediction implies that specific information is propagated down from memory through the visual hierarchy. Likewise, recognizing the input implies that sequentially accrued sensory evidence is successfully matched with memorized information (categorical knowledge). Although the existing models of prediction, memory, sensory representation and categorical decision are all implicitly cast within an information processing framework, it remains a challenge to precisely specify what this information is, and therefore where, when and how the architecture of the brain dynamically processes it to produce behaviour. Here, we review a framework that addresses these challenges for the studies of perception and categorization–stimulus information representation (SIR). We illustrate how SIR can reverse engineer the information contents of memory from behavioural and brain measures in the context of specific cognitive tasks that involve memory. We discuss two specific lessons from this approach that generally apply to memory studies: the importance of task, to constrain what the brain does, and of stimulus variations, to identify the specific information contents that are memorized, predicted, recalled and replayed.

This article is part of the Theo Murphy meeting issue ‘Memory reactivation: replaying events past, present and future’.

Model-based whole-brain effective connectivity to study distributed cognition in health and disease

Neuroimaging techniques are now widely used to study human cognition. The functional associations between brain areas have become a standard proxy to describe how cognitive processes are distributed across the brain network. Among the many analysis tools available, dynamic models of brain activity have been developed to overcome the limitations of original connectivity measures such as functional connectivity. This goes in line with the many efforts devoted to the assessment of directional interactions between brain areas from the observed neuroimaging activity. This opinion article provides an overview of our model-based whole-brain effective connectivity to analyze fMRI data, while discussing the pros and cons of our approach with respect to other established approaches. Our framework relies on the multivariate Ornstein-Uhlenbeck (MOU) process and is thus referred to as MOU-EC. Once tuned, the model provides a directed connectivity estimate that reflects the dynamical state of BOLD activity, which can be used to explore cognition. We illustrate this approach using two applications on task-evoked fMRI data. First, as a connectivity measure, MOU-EC can be used to extract biomarkers for task-specific brain coordination, understood as the patterns of areas exchanging information. The multivariate nature of connectivity measures raises several challenges for whole-brain analysis, for which machine-learning tools present some advantages over statistical testing. Second, we show how to interpret changes in MOU-EC connections in a collective and model-based manner, bridging with network analysis. Our framework provides a comprehensive set of tools that open exciting perspectives to study distributed cognition, as well as neuropathologies.

Representation of viewed and recalled film clips in patterns of brain activity in a person with developmental amnesia

As clear memories transport us back into the past, the brain also revives prior patterns of neural activity, a phenomenon known as neural reactivation. While growing evidence indicates a link between neural reactivation and typical variations in memory performance in healthy individuals, it is unclear how and to what extent reactivation is disrupted by a memory disorder. The current study characterizes neural reactivation in a case of amnesia using Multivoxel Pattern Analysis (MVPA). We tested NC, an individual with developmental amnesia linked to a diencephalic stroke, and 19 young adult controls on a functional magnetic resonance imaging (fMRI) task during which participants viewed and recalled short videos multiple times. An encoding classifier trained and tested to identify videos based on brain activity patterns elicited at perception revealed superior classification in NC. The enhanced consistency in stimulus representation we observed in NC at encoding was accompanied by an absence of multivariate repetition suppression, which occurred over repeated viewing in the controls. Another recall classifier trained and tested to identify videos during mental replay indicated normal levels of classification in NC, despite his poor memory for stimulus content. However, a cross-condition classifier trained on perception trials and tested on mental replay trials-a strict test of reactivation-revealed significantly poorer classification in NC. Thus, while NC's brain activity was consistent and stimulus-specific during mental replay, this specificity did not reflect the reactivation of patterns elicited at perception to the same extent as controls. Fittingly, we identified brain regions for which activity supported stimulus representation during mental replay to a greater extent in NC than in controls. This activity was not modeled on perception, suggesting that compensatory patterns of representation based on generic knowledge can support consistent mental constructs when memory is faulty. Our results reveal several ways in which amnesia impacts distributed patterns of stimulus representation during encoding and retrieval.

Congruency and reactivation aid memory integration through reinstatement of prior knowledge

Building knowledge schemas that organize information and guide future learning is of great importance in everyday life. Such knowledge building is suggested to occur through reinstatement of prior knowledge during new learning, yielding integration of new with old memories supported by the medial prefrontal cortex (mPFC) and medial temporal lobe (MTL). Congruency with prior knowledge is also known to enhance subsequent memory. Yet, how reactivation and congruency interact to optimize memory integration is unknown. To investigate this question, we used an adapted AB-AC inference paradigm in combination with functional Magnetic Resonance Imaging (fMRI). Participants first studied an AB-association followed by an AC-association, so B (a scene) and C (an object) were indirectly linked through A (a pseudoword). BC-associations were either congruent or incongruent with prior knowledge (e.g. bathduck or hammer in a bathroom), and participants reported subjective B-reactivation strength while learning AC. Behaviorally, both congruency and reactivation enhanced memory integration. In the brain, these behavioral effects related to univariate and multivariate parametric effects in the MTL, mPFC, and Parahippocampal Place Area (PPA). Moreover, mPFC exhibited larger PPA-connectivity for more congruent associations. These outcomes provide insights into the neural mechanisms underlying memory enhancement, which has value for educational learning.

Activation of stimulus-specific processing regions at retrieval tracks the strength of relational memory

Many theories of episodic memory posit that the subjective experience of recollection may be driven by the activation of stimulus-specific cortical regions during memory retrieval. This study examined cortical activation during associative memory retrieval to identify brain regions that support confidence judgments of source memory in stimulus-specific ways. Adjectives were encoded with either a picture of a face or a scene. During a source memory test, the word was presented alone and the participant was asked if the word had been previously paired with a face or a scene. We identified brain regions that were selectively active when viewing pictures of scenes or faces with a separate localizer scan. We then identified brain regions that were differentially activated to words during the source memory test that had been previously paired with faces or scenes, masked by the localizer activations, and examined how those regions were modulated by the strength of the source memory. Bilateral amygdala activation tracked source memory confidence for faces, while parahippocampal cortex tracked source memory confidence for scenes. The magnitude of the activation of these domain-specific perceptual-processing brain regions during memory retrieval may contribute to the subjective strength of episodic recollection.

Reduced Fidelity of Neural Representation Underlies Episodic Memory Decline in Normal Aging

Emerging studies have emphasized the importance of the fidelity of cortical representation in forming enduring episodic memory. No study, however, has examined whether there are age-related reductions in representation fidelity that can explain memory declines in normal aging. Using functional MRI and multivariate pattern analysis, we found that older adults showed reduced representation fidelity in the visual cortex, which accounted for their decreased memory performance even after controlling for the contribution of reduced activation level. This reduced fidelity was specifically due to older adults' poorer item-specific representation, not due to their lower activation level and variance, greater variability in neuro-vascular coupling, or decreased selectivity of categorical representation (i.e., dedifferentiation). Older adults also showed an enhanced subsequent memory effect in the prefrontal cortex based on activation level, and their prefrontal activation was associated with greater fidelity of representation in the visual cortex and better memory performance. The fidelity of cortical representation thus may serve as a promising neural index for better mechanistic understanding of the memory declines and its compensation in normal aging.

Decision-making in sensorimotor control

Skilled sensorimotor interactions with the world result from a series of decision-making processes that determine, on the basis of information extracted during the unfolding sequence of events, which movements to make and when and how to make them. Despite this inherent link between decision-making and sensorimotor control, research into each of these two areas has largely evolved in isolation, and it is only fairly recently that researchers have begun investigating how they interact and, together, influence behaviour. Here, we review recent behavioural, neurophysiological and computational research that highlights the role of decision-making processes in the selection, planning and control of goal-directed movements in humans and nonhuman primates.

The hippocampal sharp wave-ripple in memory retrieval for immediate use and consolidation

Various cognitive functions have long been known to require the hippocampus. Recently, progress has been made in identifying the hippocampal neural activity patterns that implement these functions. One such pattern is the sharp wave-ripple (SWR), an event associated with highly synchronous neural firing in the hippocampus and modulation of neural activity in distributed brain regions. Hippocampal spiking during SWRs can represent past or potential future experience, and SWR-related interventions can alter subsequent memory performance. These findings and others suggest that SWRs support both memory consolidation and memory retrieval for processes such as decision-making. In addition, studies have identified distinct types of SWR based on representational content, behavioural state and physiological features. These various findings regarding SWRs suggest that different SWR types correspond to different cognitive functions, such as retrieval and consolidation. Here, we introduce another possibility - that a single SWR may support more than one cognitive function. Taking into account classic psychological theories and recent molecular results that suggest that retrieval and consolidation share mechanisms, we propose that the SWR mediates the retrieval of stored representations that can be utilized immediately by downstream circuits in decision-making, planning, recollection and/or imagination while simultaneously initiating memory consolidation processes.

Altered Static and Temporal Dynamic Amplitude of Low-Frequency Fluctuations in the Background Network During Working Memory States in Mild Cognitive Impairment

Previous studies investigating working memory performance in patients with mild cognitive impairment (MCI) have mainly focused on the neural mechanisms of alterations in activation. To date, very few studies have investigated background network alterations in the working memory state. Therefore, the present study investigated the static and temporal dynamic changes in the background network in MCI patients during a working memory task. A hybrid delayed-match-to-sample task was used to examine working memory performance in MCI patients. Functional magnetic resonance imaging (fMRI) data were collected and the marker of amplitude of low-frequency fluctuations (ALFF) was used to investigate alterations in the background network. The present study demonstrated static and dynamic alterations of ALFF in MCI patients during working memory tasks, relative to the resting state. Traditional static analysis revealed that ALFF decreased in the right ventrolateral prefrontal cortex (VLPFC), right dorsolateral PFC (DLPFC), and left supplementary motor area for normal controls (NCs) in the working memory state. However, the same regions showed increased ALFF in MCI patients. Furthermore, relative to NCs, MCI patients demonstrated altered performance-related functional connectivity (FC) patterns, with the right VLPFC and right DLPFC as ROIs. In terms of temporal dynamic analysis, the present study found that in the working memory state dynamic ALFF of bilateral thalamus regions was increased in NCs but decreased in MCI patients. Additionally, MCI patients demonstrated altered performance-related coefficient of variation patterns; the regions in MCI patients were larger and more widely distributed in the parietal and temporal lobes, relative to NCs. This is the first study to examine static and temporal dynamic alterations of ALFF in the background network of MCI patients in working memory states. The results extend previous studies by providing a new perspective on the neural mechanisms of working memory deficits in MCI patients.

Exploring the neurocognitive basis of episodic recollection in autism

Increasing evidence indicates that the subjective experience of recollection is diminished in autism spectrum disorder (ASD) compared to neurotypical individuals. The neurocognitive basis of this difference in how past events are re-experienced has been debated and various theoretical accounts have been proposed to date. Although each existing theory may capture particular features of memory in ASD, recent research questions whether any of these explanations are alone sufficient or indeed fully supported. This review first briefly considers the cognitive neuroscience of how episodic recollection operates in the neurotypical population, informing predictions about the encoding and retrieval mechanisms that might function atypically in ASD. We then review existing research on recollection in ASD, which has often not distinguished between different theoretical explanations. Recent evidence suggests a distinct difficulty engaging recollective retrieval processes, specifically the ability to consciously reconstruct and monitor a past experience, which is likely underpinned by altered functional interactions between neurocognitive systems rather than brain region-specific or process-specific dysfunction. This integrative approach serves to highlight how memory research in ASD may enhance our understanding of memory processes and networks in the typical brain. We make suggestions for future research that are important for further specifying the neurocognitive basis of episodic recollection in ASD and linking such difficulties to social developmental and educational outcomes.

Coordinated representational reinstatement in the human hippocampus and lateral temporal cortex during episodic memory retrieval

Theoretical models of episodic memory have proposed that retrieval depends on interactions between the hippocampus and neocortex, where hippocampal reinstatement of item-context associations drives neocortical reinstatement of item information. Here, we simultaneously recorded intracranial EEG from hippocampus and lateral temporal cortex (LTC) of epilepsy patients who performed a virtual reality spatial navigation task. We extracted stimulus-specific representations of both item and item-context associations from the time-frequency patterns of activity in hippocampus and LTC. Our results revealed a double dissociation of representational reinstatement across time and space: an early reinstatement of item-context associations in hippocampus preceded a later reinstatement of item information in LTC. Importantly, reinstatement levels in hippocampus and LTC were correlated across trials, and the quality of LTC reinstatement was predicted by the magnitude of phase synchronization between hippocampus and LTC. These findings confirm that episodic memory retrieval in humans relies on coordinated representational interactions within a hippocampal-neocortical network.

Episodic memory retrieval is hypothesized to rely on hippocampal reinstatement of item-context associations which drives reinstatement of item information in cortex. Here, the authors confirm this sequence of events, using iEEG recordings from the human hippocampus and lateral temporal cortex.

Neural specificity of scene representations is related to memory performance in childhood

Successful memory encoding is supported by medial temporal, retrosplenial, and occipital regions, which show developmental differences in recruitment from childhood to adulthood. However, little is known about the extent to which neural specificity in these brain regions, or the distinctiveness with which sensory information is represented, continues to develop during middle childhood and how it contributes to memory performance. The present study used multivariate pattern analysis to examine the distinctiveness of different scene representations in 169 children and 31 adults, and its relation to memory performance. Most children provided data over up to three measurement occasions between 8 and 15 years (267 total scans), allowing us to examine changes in memory and neural specificity over time. Memory performance was lower in children than in adults, and increased in children over time. Different scenes presented during memory encoding could be reliably decoded from parahippocampal, lateral occipital, and retrosplenial regions in children and adults. Neural specificity in children was similar to adults, and did not change reliably over time. Among children, higher neural specificity in scene-processing regions was associated with better memory concurrently. These results suggest that the distinctiveness with which incoming information is represented is important for memory performance in childhood, but other processes operating on these representations support developmental improvements in memory performance over time.