Transforming the Concept of Memory Reactivation

Perception and Memory Reinstatement Engage Overlapping Face-Selective Regions within Human Ventral Temporal Cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions specialized for processing specific sensory stimuli are reactivated to support content-specific retrieval. Recently, several studies have emphasized transformations in the spatial organization of these reinstated activity patterns. Specifically, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with the activity observed during perception. However, it is not clear that such transformations occur universally, with inconsistent evidence for other important stimulus categories, particularly faces. One challenge in addressing this question is the careful delineation of face-selective cortices, which are interdigitated with other selective regions, in configurations that spatially differ across individuals. Therefore, we conducted a multisession neuroimaging study to first carefully map individual participants' (nine males and seven females) face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the activity patterns within these regions during face memory encoding and retrieval. While face-selective regions were expectedly engaged during face perception at encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions, but did not show any consistent direction of spatial transformation (e.g., anteriorization). We also report on unique human intracranial recordings from VTC under the same experimental conditions. These findings highlight the importance of considering the complex configuration of category-selective cortex in elucidating principles shaping the neural transformations that occur from perception to memory.

Images with harder-to-reconstruct visual representations leave stronger memory traces

Much of what we remember is not because of intentional selection, but simply a by-product of perceiving. This raises a foundational question about the architecture of the mind: how does perception interface with and influence memory? Here, inspired by a classic proposal relating perceptual processing to memory durability, the level-of-processing theory, we present a sparse coding model for compressing feature embeddings of images, and show that the reconstruction residuals from this model predict how well images are encoded into memory. In an open memorability dataset of scene images, we show that reconstruction error not only explains memory accuracy, but also response latencies during retrieval, subsuming, in the latter case, all of the variance explained by powerful vision-only models. We also confirm a prediction of this account with 'model-driven psychophysics'. This work establishes reconstruction error as an important signal interfacing perception and memory, possibly through adaptive modulation of perceptual processing.

A shared neural code for perceiving and remembering social interactions in the human superior temporal sulcus

Recognizing and remembering social information is a crucial cognitive skill. Neural patterns in the superior temporal sulcus (STS) support our ability to perceive others' social interactions. However, despite the prominence of social interactions in memory, the neural basis of remembering social interactions is still unknown. To fill this gap, we investigated the brain mechanisms underlying memory of others' social interactions during free spoken recall of a naturalistic movie. By applying machine learning-based fMRI encoding analyses to densely labeled movie and recall data we found that a subset of the STS activity evoked by viewing social interactions predicted neural responses in not only held-out movie data, but also during memory recall. These results provide the first evidence that activity in the STS is reinstated in response to specific social content and that its reactivation underlies our ability to remember others' interactions. These findings further suggest that the STS contains representations of social interactions that are not only perceptually driven, but also more abstract or conceptual in nature.

Influencing dreams through sensory stimulation: A systematic review

Sleep is typically considered a state of disconnection from the environment, yet instances of external sensory stimuli influencing dreams have been reported for centuries. Explaining this phenomenon could provide valuable insight into dreams' generative and functional mechanisms, the factors that promote sleep continuity, and the processes that underlie conscious awareness. Moreover, harnessing sensory stimuli for dream engineering could benefit individuals suffering from dream-related alterations. This PRISMA-compliant systematic review assessed the current evidence concerning the influence of sensory stimulation on sleep mentation. We included 51 publications, of which 21 focused on auditory stimulation, ten on somatosensory stimulation, eight on olfactory stimulation, four on visual stimulation, two on vestibular stimulation, and one on multimodal stimulation. Furthermore, nine references explored conditioned associative stimulation: six focused on targeted memory reactivation protocols and three on targeted lucid reactivation protocols. The reported frequency of stimulus-dependent dream changes across studies ranged from 0 to ∼80%, likely reflecting a considerable heterogeneity of definitions and methodological approaches. Our findings highlight a lack of comprehensive understanding of the mechanisms, functions, and neurophysiological correlates of stimulus-dependent dream changes. We suggest that a paradigm shift is required for meaningful progress in this field.

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Encoded and updated spatial working memories share a common representational format in alpha activity

Working memory (WM) flexibly updates information to adapt to the dynamic environment. Here, we used alpha-band activity in the EEG to reconstruct the content of dynamic WM updates and compared this representational format to static WM content. An inverted encoding model using alpha activity precisely tracked both the initially encoded position and the updated position following an auditory cue signaling mental updating. The timing of the update, as tracked in the EEG, correlated with reaction times and saccade latency. Finally, cross-training analyses revealed a robust generalization of alpha-band reconstruction of WM contents before and after updating. These findings demonstrate that alpha activity tracks the dynamic updates to spatial WM and that the format of this activity is preserved across the encoded and updated representations. Thus, our results highlight a new approach for measuring updates to WM and show common representational formats during dynamic mental updating and static storage.

A retinotopic code structures the interaction between perception and memory systems

Conventional views of brain organization suggest that regions at the top of the cortical hierarchy processes internally oriented information using an abstract amodal neural code. Despite this, recent reports have described the presence of retinotopic coding at the cortical apex, including the default mode network. What is the functional role of retinotopic coding atop the cortical hierarchy? Here we report that retinotopic coding structures interactions between internally oriented (mnemonic) and externally oriented (perceptual) brain areas. Using functional magnetic resonance imaging, we observed robust inverted (negative) retinotopic coding in category-selective memory areas at the cortical apex, which is functionally linked to the classic (positive) retinotopic coding in category-selective perceptual areas in high-level visual cortex. These functionally linked retinotopic populations in mnemonic and perceptual areas exhibit spatially specific opponent responses during both bottom-up perception and top-down recall, suggesting that these areas are interlocked in a mutually inhibitory dynamic. These results show that retinotopic coding structures interactions between perceptual and mnemonic neural systems, providing a scaffold for their dynamic interaction.

The reactivation of task rules triggers the reactivation of task-relevant items

Working memory (WM) describes the temporary storage of task-relevant items and procedural rules to guide action. Despite its central importance for goal-directed behavior, the interplay between WM and long-term memory (LTM) remains poorly understood. Recent studies have shown that repeated use of the same task-relevant item in WM results in a hand-off of the storage of that item to LTM, and switching to a new item reactivates WM. To further elucidate the rules governing WM-LTM interactions, we here planned to probe whether a change in task rules, independent of a switch in task-relevant items, would also lead to WM reactivation of maintained items. To this end, we used scalp-recorded electroencephalogram (EEG) data, specifically the contralateral delay activity (CDA), to track WM item storage while manipulating repetitions and changes in task rules and task-relevant items across trials in a visual WM task. We tested two rival hypotheses: If changes in task rules result in a reactivation of the target item representation, then the CDA should increase when a task change is cued even when the same target has been repeated across trials. However, if the reactivation of a task-relevant item only depends on the mnemonic availability of the item itself instead of the task it is used for, then only the changes in task-relevant items should reactivate the representations. Accordingly, the CDA amplitude should decrease for repeated task-relevant items independently of a task change. We found a larger CDA on task-switch compared to task-repeat trials, suggesting that the reactivation of task rules triggers the reactivation of task-relevant items in WM. By demonstrating that WM reactivation of LTM is interdependent for task rules and task-relevant items, this study informs our understanding of visual WM and its interplay with LTM. PREREGISTERED STAGE 1 PROTOCOL: https://osf.io/zp9e8 (date of in-principle acceptance: 19/12/2021).

Intrinsic functional connectivity in medial temporal lobe networks is associated with susceptibility to misinformation

Memory is notoriously fallible and susceptible to misinformation. Yet little is known about the underlying cognitive and neural mechanisms that render individuals vulnerable to this type of false memory. The current experiments take an individual differences approach to examine whether susceptibility to misinformation reflects stable underlying factors related to memory retrieval. In Study 1, we report for the first time the existence of substantial individual variability in susceptibility to misinformation in the context of repeated memory retrieval, when the misinformation effect is most pronounced. This variability was not related to an individual's tendency to adopt an episodic retrieval style during remembering (trait mnemonics). In Study 2, we next examined whether susceptibility to misinformation is related to intrinsic functional connectivity in medial temporal lobe (MTL) networks known to coordinate memory reactivation during event retrieval. Stronger resting-state functional connectivity between the MTL and cortical areas associated with visual memory reactivation (occipital cortex) was associated with better protection from misinformation. Together, these results reveal that while memory distortion is a universal property of our reconstructive memory system, susceptibility to misinformation varies at the individual level and may depend on one's ability to reactivate visual details during memory retrieval.

Bridging phenomenology and neural mechanisms of inner speech: ALE meta-analysis on egocentricity and spontaneity in a dual-mechanistic framework

The neural mechanisms of inner speech remain unclear despite its importance in a variety of cognitive processes and its implication in aberrant perceptions such as auditory verbal hallucinations. Previous research has proposed a corollary discharge model in which inner speech is a truncated form of overt speech, relying on speech production-related regions (e.g. left inferior frontal gyrus). This model does not fully capture the diverse phenomenology of inner speech and recent research suggesting alternative perception-related mechanisms of generation. Therefore, we present and test a framework in which inner speech can be generated by two separate mechanisms, depending on its phenomenological qualities: a corollary discharge mechanism relying on speech production regions and a perceptual simulation mechanism within speech perceptual regions. The results of the activation likelihood estimation meta-analysis examining inner speech studies support the idea that varieties of inner speech recruit different neural mechanisms.

A retinotopic code structures the interaction between perception and memory systems

Conventional views of brain organization suggest that the cortical apex processes internally-oriented information using an abstract, amodal neural code. Yet, recent reports have described the presence of retinotopic coding at the cortical apex, including the default mode network. What is the functional role of retinotopic coding atop the cortical hierarchy? Here, we report that retinotopic coding structures interactions between internally-oriented (mnemonic) and externally-oriented (perceptual) brain areas. Using fMRI, we observed robust, inverted (negative) retinotopic coding in category-selective memory areas at the cortical apex, which is functionally linked to the classic (positive) retinotopic coding in category-selective perceptual areas in high-level visual cortex. Specifically, these functionally-linked retinotopic populations in mnemonic and perceptual areas exhibit spatially-specific opponent responses during both bottom-up perception and top-down recall, suggesting that these areas are interlocked in a mutually-inhibitory dynamic. Together, these results show that retinotopic coding structures interactions between perceptual and mnemonic neural systems, thereby scaffolding their dynamic interaction.

Real-world implications of aphantasia: episodic recall of eyewitnesses with aphantasia is less complete but no less accurate than typical imagers

Individuals with aphantasia report an inability to voluntarily visually image and reduced episodic memory, yet episodic accounts provided by witnesses and victims are fundamental for criminal justice. Using the mock-witness paradigm, we investigated eyewitness memory of individuals with aphantasia versus typical imagers. Participants viewed a mock crime and 48 hours later were interviewed about the event, randomly allocated to one of three conditions. Two interview conditions included techniques designed to support episodic retrieval mode, namely (i) Mental Reinstatement of Context (MRC) and (ii) Sketch Reinstatement of Context (Sketch-RC). A third Control condition did not include retrieval support. Aphantasic mock-eyewitnesses recalled 30% less correct information and accounts were less complete, but they made no more errors and were as accurate as typical imagers. Interaction effects revealed reduced correct recall and less complete accounts for aphantasic participants in MRC interviews versus Sketch-RC and Control. Aphantaisic participants in the Control outperformed those in both the Sketch-RC and MRC, although Sketch-RC improved completeness by 15% versus MRC. Our pattern of results indicates reduced mental imagery ability might be compensated for by alternative self-initiated cognitive strategies. Findings offer novel insights into episodic recall performance in information gathering interviews when ability to voluntarily visualize is impoverished.

Intersubject similarity in neural representations underlies shared episodic memory content

Individuals generally form their unique memories from shared experiences, yet the neural representational mechanisms underlying this subjectiveness of memory are poorly understood. The current study addressed this important question from the cross-subject neural representational perspective, leveraging a large functional magnetic resonance imaging dataset (n = 415) of a face-name associative memory task. We found that individuals' memory abilities were predicted by their synchronization to the group-averaged, canonical trial-by-trial activation level and, to a lesser degree, by their similarity to the group-averaged representational patterns during encoding. More importantly, the memory content shared between pairs of participants could be predicted by their shared local neural activation pattern, particularly in the angular gyrus and ventromedial prefrontal cortex, even after controlling for differences in memory abilities. These results uncover neural representational mechanisms for individualized memory and underscore the constructive nature of episodic memory.

Mapping multidimensional content representations to neural and behavioral expressions of episodic memory

Human neuroimaging studies have shown that the contents of episodic memories are represented in distributed patterns of neural activity. However, these studies have mostly been limited to decoding simple, unidimensional properties of stimuli. Semantic encoding models, in contrast, offer a means for characterizing the rich, multidimensional information that comprises episodic memories. Here, we extensively sampled four human fMRI subjects to build semantic encoding models and then applied these models to reconstruct content from natural scene images as they were viewed and recalled from memory. First, we found that multidimensional semantic information was successfully reconstructed from activity patterns across visual and lateral parietal cortices, both when viewing scenes and when recalling them from memory. Second, whereas visual cortical reconstructions were much more accurate when images were viewed versus recalled from memory, lateral parietal reconstructions were comparably accurate across visual perception and memory. Third, by applying natural language processing methods to verbal recall data, we showed that fMRI-based reconstructions reliably matched subjects' verbal descriptions of their memories. In fact, reconstructions from ventral temporal cortex more closely matched subjects' own verbal recall than other subjects' verbal recall of the same images. Fourth, encoding models reliably transferred across subjects: memories were successfully reconstructed using encoding models trained on data from entirely independent subjects. Together, these findings provide evidence for successful reconstructions of multidimensional and idiosyncratic memory representations and highlight the differential sensitivity of visual cortical and lateral parietal regions to information derived from the external visual environment versus internally-generated memories.

Age differences in neural distinctiveness during memory encoding, retrieval, and reinstatement

Robust evidence points to mnemonic deficits in older adults related to dedifferentiated, i.e. less distinct, neural responses during memory encoding. However, less is known about retrieval-related dedifferentiation and its role in age-related memory decline. In this study, younger and older adults were scanned both while incidentally learning face and house stimuli and while completing a surprise recognition memory test. Using pattern similarity searchlight analyses, we looked for indicators of neural dedifferentiation during encoding, retrieval, and encoding-retrieval reinstatement. Our findings revealed age-related reductions in neural distinctiveness during all memory phases in visual processing regions. Interindividual differences in retrieval- and reinstatement-related distinctiveness were strongly associated with distinctiveness during memory encoding. Both item- and category-level distinctiveness predicted trial-wise mnemonic outcomes. We further demonstrated that the degree of neural distinctiveness during encoding tracked interindividual variability in memory performance better than both retrieval- and reinstatement-related distinctiveness. All in all, we contribute to meager existing evidence for age-related neural dedifferentiation during memory retrieval. We show that neural distinctiveness during retrieval is likely tied to recapitulation of encoding-related perceptual and mnemonic processes.

Individual-specific memory reinstatement patterns within human face-selective cortex

Humans have the remarkable ability to vividly retrieve sensory details of past events. According to the theory of sensory reinstatement, during remembering, brain regions involved in the sensory processing of prior events are reactivated to support this perception of the past. Recently, several studies have emphasized potential transformations in the spatial organization of reinstated activity patterns. In particular, studies of scene stimuli suggest a clear anterior shift in the location of retrieval activations compared with those during perception. However, it is not clear that such transformations occur universally, with evidence lacking for other important stimulus categories, particularly faces. Critical to addressing these questions, and to studies of reinstatement more broadly, is the growing importance of considering meaningful variations in the organization of sensory systems across individuals. Therefore, we conducted a multi-session neuroimaging study to first carefully map individual participants face-selective regions within ventral temporal cortex (VTC), followed by a second session to examine the correspondence of activity patterns during face memory encoding and retrieval. Our results showed distinct configurations of face-selective regions within the VTC across individuals. While a significant degree of overlap was observed between face perception and memory encoding, memory retrieval engagement exhibited a more selective and constricted reinstatement pattern within these regions. Importantly, these activity patterns were consistently tied to individual-specific neural substrates, but did not show any consistent direction of spatial transformation (e.g., anteriorization). To provide further insight to these findings, we also report on unique human intracranial recordings from VTC under the same experimental conditions. Our findings highlight the importance of considering individual variations in functional neuroanatomy in the context of assessing the nature of cortical reinstatement. Consideration of such factors will be important for establishing general principles shaping the neural transformations that occur from perception to memory.

Engram neurons: Encoding, consolidation, retrieval, and forgetting of memory

Tremendous strides have been made in our understanding of the neurobiological substrates of memory - the so-called memory "engram". Here, we integrate recent progress in the engram field to illustrate how engram neurons transform across the "lifespan" of a memory - from initial memory encoding, to consolidation and retrieval, and ultimately to forgetting. To do so, we first describe how cell-intrinsic properties shape the initial emergence of the engram at memory encoding. Second, we highlight how these encoding neurons preferentially participate in synaptic- and systems-level consolidation of memory. Third, we describe how these changes during encoding and consolidation guide neural reactivation during retrieval, and facilitate memory recall. Fourth, we describe neurobiological mechanisms of forgetting, and how these mechanisms can counteract engram properties established during memory encoding, consolidation, and retrieval. Motivated by recent experimental results across these four sections, we conclude by proposing some conceptual extensions to the traditional view of the engram, including broadening the view of cell-type participation within engrams and across memory stages. In collection, our review synthesizes general principles of the engram across memory stages, and describes future avenues to further understand the dynamic engram.

Perception and memory retrieval states are reflected in distributed patterns of background functional connectivity

The same visual input can serve as the target of perception or as a trigger for memory retrieval depending on whether cognitive processing is externally oriented (perception) or internally oriented (memory retrieval). While numerous human neuroimaging studies have characterized how visual stimuli are differentially processed during perception versus memory retrieval, perception and memory retrieval may also be associated with distinct neural states that are independent of stimulus-evoked neural activity. Here, we combined human fMRI with full correlation matrix analysis (FCMA) to reveal potential differences in "background" functional connectivity across perception and memory retrieval states. We found that perception and retrieval states could be discriminated with high accuracy based on patterns of connectivity across (1) the control network, (2) the default mode network (DMN), and (3) retrosplenial cortex (RSC). In particular, clusters in the control network increased connectivity with each other during the perception state, whereas clusters in the DMN were more strongly coupled during the retrieval state. Interestingly, RSC switched its coupling between networks as the cognitive state shifted from retrieval to perception. Finally, we show that background connectivity (1) was fully independent from stimulus-related variance in the signal and, further, (2) captured distinct aspects of cognitive states compared to traditional classification of stimulus-evoked responses. Together, our results reveal that perception and memory retrieval are associated with sustained cognitive states that manifest as distinct patterns of connectivity among large-scale brain networks.

Reduced memory precision in older age is associated with functional and structural differences in the angular gyrus

Decreased fidelity of mnemonic representations plays a critical role in age-related episodic memory deficits, yet the brain mechanisms underlying such reductions remain unclear. Using functional and structural neuroimaging, we examined how changes in two key nodes of the posterior-medial network, the hippocampus and the angular gyrus (AG), might underpin loss of memory precision in older age. Healthy young and older adults completed a memory task that involved reconstructing object features on a continuous scale. Investigation of blood-oxygen-level-dependent (BOLD) activity during retrieval revealed an age-related reduction in activity reflecting successful recovery of object features in the hippocampus, whereas trial-wise modulation of BOLD signal by graded memory precision was diminished in the AG. Gray matter volume of the AG further predicted individual differences in memory precision in older age, beyond likelihood of successful retrieval. These findings provide converging evidence for a role of functional and structural integrity of the AG in constraining the fidelity of episodic remembering in older age, yielding new insights into parietal contributions to age-related episodic memory decline.

Memory reactivations during sleep: a neural basis of dream experiences?

Newly encoded memory traces are spontaneously reactivated during sleep. Since their discovery in the 1990s, these memory reactivations have been discussed as a potential neural basis for dream experiences. New results from animal and human research, as well as from the rapidly growing field of sleep and dream engineering, provide essential insights into this question, and reveal both strong parallels and disparities between the two phenomena. We suggest that, although memory reactivations may contribute to subjective experiences across different states of consciousness, they are not likely to be the primary neural basis of dreaming. We identify important limitations in current research paradigms and suggest novel strategies to address this question empirically.

Dynamics of working memory process revealed by independent component analysis in an fMRI study

Human memory is prone to errors in many everyday activities but also when cultivating hobbies such as traveling and/or learning a new language. For instance, while visiting foreign countries, people erroneously recall foreign language words that are meaningless to them. Our research simulated such errors in a modified Deese-Roediger-McDermott paradigm for short-term memory with phonologically related stimuli aimed at uncovering behavioral and neuronal indices of false memory formation with regard to time-of-day, a variable known to influence memory. Fifty-eight participants were tested in a magnetic resonance (MR) scanner twice. The results of an Independent Component Analysis revealed encoding-related activity of the medial visual network preceding correct recognition of positive probes and correct rejection of lure probes. The engagement of this network preceding false alarms was not observed. We also explored if diurnal rhythmicity influences working memory processes. Diurnal differences were seen in the default mode network and the medial visual network with lower deactivation in the evening hours. The GLM results showed greater activation of the right lingual gyrus, part of the visual cortex and the left cerebellum in the evening. The study offers new insight into the mechanisms associated with false memories, suggesting that deficient engagement of the medial visual network during the memorization phase of a task results in short-term memory distortions. The results shed new light on the dynamics of working memory processes by taking into account the effect of time-of-day on memory performance.

Decoding of human identity by computer vision and neuronal vision

Extracting meaning from a dynamic and variable flow of incoming information is a major goal of both natural and artificial intelligence. Computer vision (CV) guided by deep learning (DL) has made significant strides in recognizing a specific identity despite highly variable attributes. This is the same challenge faced by the nervous system and partially addressed by the concept cells-neurons exhibiting selective firing in response to specific persons/places, described in the human medial temporal lobe (MTL) ⁠. Yet, access to neurons representing a particular concept is limited due to these neurons' sparse coding. It is conceivable, however, that the information required for such decoding is present in relatively small neuronal populations. To evaluate how well neuronal populations encode identity information in natural settings, we recorded neuronal activity from multiple brain regions of nine neurosurgical epilepsy patients implanted with depth electrodes, while the subjects watched an episode of the TV series "24". First, we devised a minimally supervised CV algorithm (with comparable performance against manually-labeled data) to detect the most prevalent characters (above 1% overall appearance) in each frame. Next, we implemented DL models that used the time-varying population neural data as inputs and decoded the visual presence of the four main characters throughout the episode. This methodology allowed us to compare "computer vision" with "neuronal vision"-footprints associated with each character present in the activity of a subset of neurons-and identify the brain regions that contributed to this decoding process. We then tested the DL models during a recognition memory task following movie viewing where subjects were asked to recognize clip segments from the presented episode. DL model activations were not only modulated by the presence of the corresponding characters but also by participants' subjective memory of whether they had seen the clip segment, and by the associative strengths of the characters in the narrative plot. The described approach can offer novel ways to probe the representation of concepts in time-evolving dynamic behavioral tasks. Further, the results suggest that the information required to robustly decode concepts is present in the population activity of only tens of neurons even in brain regions beyond MTL.

Perception and memory have distinct spatial tuning properties in human visual cortex

Reactivation of earlier perceptual activity is thought to underlie long-term memory recall. Despite evidence for this view, it is unclear whether mnemonic activity exhibits the same tuning properties as feedforward perceptual activity. Here, we leverage population receptive field models to parameterize fMRI activity in human visual cortex during spatial memory retrieval. Though retinotopic organization is present during both perception and memory, large systematic differences in tuning are also evident. Whereas there is a three-fold decline in spatial precision from early to late visual areas during perception, this pattern is not observed during memory retrieval. This difference cannot be explained by reduced signal-to-noise or poor performance on memory trials. Instead, by simulating top-down activity in a network model of cortex, we demonstrate that this property is well explained by the hierarchical structure of the visual system. Together, modeling and empirical results suggest that computational constraints imposed by visual system architecture limit the fidelity of memory reactivation in sensory cortex.

Neural activity from perception is thought to be reactivated during memory recall. Here, the authors show that memory reactivation in visual cortex exhibits different tuning properties from those observed during perception.

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Brief memory reactivations induce learning in the numeric domain

Learning of arithmetic facts such as the multiplication table requires time-consuming, repeated practice. In light of evidence indicating that reactivation of encoded memories can modulate learning and memory processes at the synaptic, system and behavioral levels, we asked whether brief memory reactivations can induce human learning in the numeric domain. Adult participants performed a number-fact retrieval task in which they learned arbitrary numeric facts. Following encoding and a baseline test, 3 passive, brief reactivation sessions of only 40 s each were conducted on separate days. Learning was evaluated in a retest session. Results showed reactivations induced learning, with improved performance at retest relative to baseline test. Furthermore, performance was superior compared to a control group performing test-retest sessions without reactivations, who showed significant memory deterioration. A standard practice group completed active-retrieval sessions on 3 separate days, and showed significant learning gains. Interestingly, while these gains were higher than those of the reactivations group, subjects showing reactivation-induced learning were characterized by superior efficiency relative to standard practice subjects, with higher rate of improvement per practice time. A follow-up long-term retention experiment showed that 30 days following initial practice, weekly brief reactivations reduced forgetting, with participants performing superior to controls undergoing the same initial practice without reactivations. Overall, the results demonstrate that brief passive reactivations induce efficient learning and reduce forgetting within a numerical context. Time-efficient practice in the numeric domain carries implications for enhancement of learning strategies in daily-life settings.

Contextual Expectations Shape Cortical Reinstatement of Sensory Representations

When making a turn at a familiar intersection, we know what items and landmarks will come into view. These perceptual expectations, or predictions, come from our knowledge of the context; however, it is unclear how memory and perceptual systems interact to support the prediction and reactivation of sensory details in cortex. To address this, human participants learned the spatial layout of animals positioned in a cross maze. During fMRI, participants of both sexes navigated between animals to reach a target, and in the process saw a predictable sequence of five animal images. Critically, to isolate activity patterns related to item predictions, rather than bottom-up inputs, one-fourth of trials ended early, with a blank screen presented instead. Using multivariate pattern similarity analysis, we reveal that activity patterns in early visual cortex, posterior medial regions, and the posterior hippocampus showed greater similarity when seeing the same item compared with different items. Further, item effects in posterior hippocampus were specific to the sequence context. Critically, activity patterns associated with seeing an item in visual cortex and posterior medial cortex, were also related to activity patterns when an item was expected, but omitted, suggesting sequence predictions were reinstated in these regions. Finally, multivariate connectivity showed that patterns in the posterior hippocampus at one position in the sequence were related to patterns in early visual cortex and posterior medial cortex at a later position. Together, our results support the idea that hippocampal representations facilitate sensory processing by modulating visual cortical activity in anticipation of expected items.SIGNIFICANCE STATEMENT Our visual world is a series of connected events, where we can predict what we might see next based on our recent past. Understanding the neural circuitry and mechanisms of the perceptual and memory systems that support these expectations is fundamental to revealing how we perceive and act in our world. Using brain imaging, we studied what happens when we expect to see specific visual items, and how such expectations relate to top-down memory signals. We find both visual and memory systems reflect item predictions, and moreover, we show that hippocampal activity supports predictions of future expected items. This demonstrates that the hippocampus acts to predict upcoming items, and reinstates such predictions in cortex.

Episodic representation: A mental models account

This paper offers a modeling account of episodic representation. I argue that the episodic system constructs mental models: representations that preserve the spatiotemporal structure of represented domains. In prototypical cases, these domains are events: occurrences taken by subjects to have characteristic structures, dynamics and relatively determinate beginnings and ends. Due to their simplicity and manipulability, mental event models can be used in a variety of cognitive contexts: in remembering the personal past, but also in future-oriented and counterfactual imagination. As structural representations, they allow surrogative reasoning, supporting inferences about their constituents which can be used in reasoning about the represented events.

Neural signatures associated with temporal compression in the verbal retelling of past events

When we retell our past experiences, we aim to reproduce some version of the original events; this reproduced version is often temporally compressed relative to the original. However, it is currently unclear how this compression manifests in brain activity. One possibility is that a compressed retrieved memory manifests as a neural pattern which is more dissimilar to the original, relative to a more detailed or vivid memory. However, we argue that measuring raw dissimilarity alone is insufficient, as it confuses a variety of interesting and uninteresting changes. To address this problem, we examine brain pattern changes that are consistent across people. We show that temporal compression in individuals’ retelling of past events predicts systematic encoding-to-recall transformations in several higher associative regions. These findings elucidate how neural representations are not simply reactivated, but can also be transformed due to temporal compression during a universal form of human memory expression: verbal retelling.

Brain patterns measured while participants first watched a movie in the fMRI scanner, then recalled the movie’s key narrative features, demonstrate that temporal compression in individuals’ retelling of past events predicts encoding-to-recall transformations.

Category-sensitive incidental reinstatement in medial temporal lobe subregions during word recognition

During associative retrieval, the brain reinstates neural representations that were present during encoding. The human medial temporal lobe (MTL), with its subregions hippocampus (HC), perirhinal cortex (PRC), and parahippocampal cortex (PHC), plays a central role in neural reinstatement. Previous studies have given compelling evidence for reinstatement in the MTL during explicitly instructed associative retrieval. High-confident recognition may be similarly accompanied by recollection of associated information from the encoding context. It is unclear, however, whether high-confident recognition memory elicits reinstatement in the MTL even in the absence of an explicit instruction to retrieve associated information. Here, we addressed this open question using high-resolution fMRI. Twenty-eight male and female human volunteers engaged in a recognition memory task for words that they had previously encoded together with faces and scenes. Using complementary univariate and multivariate approaches, we show that MTL subregions including the PRC, PHC, and HC differentially reinstate category-sensitive representations during high-confident word recognition, even though no explicit instruction to retrieve the associated category was given. This constitutes novel evidence that high-confident recognition memory is accompanied by incidental reinstatement of associated category information in MTL subregions, and supports a functional model of the MTL that emphasizes content-sensitive representations during both encoding and retrieval.

Contributions of representational distinctiveness and stability to memory performance and age differences

Long-standing theories of cognitive aging suggest that memory decline is associated with age-related differences in the way information is neurally represented. Multivariate pattern similarity analyses enabled researchers to take a representational perspective on brain and cognition, and allowed them to study the properties of neural representations that support successful episodic memory. Two representational properties have been identified as crucial for memory performance, namely the distinctiveness and the stability of neural representations. Here, we review studies that used multivariate analysis tools for different neuroimaging techniques to clarify how these representational properties relate to memory performance across adulthood. While most evidence on age differences in neural representations involved stimulus category information , recent studies demonstrated that particularly item-level stability and specificity of activity patterns are linked to memory success and decline during aging. Overall, multivariate methods offer a versatile tool for our understanding of age differences in the neural representations underlying memory.

Neural reactivation and judgements of vividness reveal separable contributions to mnemonic representation

Mnemonic representations vary in fidelity, sharpness, and strength-qualities that can be examined using both introspective judgements of mental states and objective measures of brain activity. Subjective and objective measures are both valid ways of "reading out" the content of someone's internal mnemonic states, each with different strengths and weaknesses. St-Laurent and colleagues (2015) compared the neural correlates of memory vividness ratings with patterns of neural reactivation evoked during memory recall and found considerable overlap between the two, suggesting a common neural basis underlying these different markers of representational quality. Here we extended this work with meta-analytic methods by pooling together four neuroimaging datasets in order to contrast the neural substrates of neural reactivation and those of vividness judgements. While reactivation and vividness judgements correlated positively with one another and were associated with common univariate activity in the dorsal attention network and anterior hippocampus, some notable differences were also observed. Vividness judgments were tied to stronger activation in the striatum and dorsal attention network, together with activity suppression in default mode network nodes. We also observed a trend for reactivation to be more closely associated with early visual cortex activity. A mediation analysis found support for the hypothesis that neural reactivation is necessary for memory vividness, with activity in the anterior hippocampus associated with greater reactivation. Our results suggest that neural reactivation and vividness judgements reflect common mnemonic processes but differ in the extent to which they engage effortful, attentional processes. Additionally, the similarity between reactivation and vividness appears to arise, partly, through hippocampal engagement during memory retrieval.

Neural representations of naturalistic events are updated as our understanding of the past changes

The brain actively reshapes our understanding of past events in light of new incoming information. In the current study, we ask how the brain supports this updating process during the encoding and recall of naturalistic stimuli. One group of participants watched a movie ('The Sixth Sense') with a cinematic 'twist' at the end that dramatically changed the interpretation of previous events. Next, participants were asked to verbally recall the movie events, taking into account the new 'twist' information. Most participants updated their recall to incorporate the twist. Two additional groups recalled the movie without having to update their memories during recall: one group never saw the twist; another group was exposed to the twist prior to the beginning of the movie, and thus the twist information was incorporated both during encoding and recall. We found that providing participants with information about the twist beforehand altered neural response patterns during movie-viewing in the default mode network (DMN). Moreover, presenting participants with the twist at the end of the movie changed the neural representation of the previously-encoded information during recall in a subset of DMN regions. Further evidence for this transformation was obtained by comparing the neural activation patterns during encoding and recall and correlating them with behavioral signatures of memory updating. Our results demonstrate that neural representations of past events encoded in the DMN are dynamically integrated with new information that reshapes our understanding in natural contexts.

Transformative neural representations support long-term episodic memory

Memory is often conceived as a dynamic process that involves substantial transformations of mental representations. However, the neural mechanisms underlying these transformations and their role in memory formation and retrieval have only started to be elucidated. Combining intracranial EEG recordings with deep neural network models, we provide a detailed picture of the representational transformations from encoding to short-term memory maintenance and long-term memory retrieval that underlie successful episodic memory. We observed substantial representational transformations during encoding. Critically, more pronounced semantic representational formats predicted better subsequent long-term memory, and this effect was mediated by more consistent item-specific representations across encoding events. The representations were further transformed right after stimulus offset, and the representations during long-term memory retrieval were more similar to those during short-term maintenance than during encoding. Our results suggest that memory representations pass through multiple stages of transformations to achieve successful long-term memory formation and recall.

Shared Representational Formats for Information Maintained in Working Memory and Information Retrieved from Long-Term Memory

Current theories propose that the short-term retention of information in working memory (WM) and the recall of information from long-term memory (LTM) are supported by overlapping neural mechanisms in occipital and parietal cortex. However, the extent of the shared representations between WM and LTM is unclear. We designed a spatial memory task that allowed us to directly compare the representations of remembered spatial information in WM and LTM with carefully matched behavioral response precision between tasks. Using multivariate pattern analyses on functional magnetic resonance imaging data, we show that visual memories were represented in a sensory-like code in both memory tasks across retinotopic regions in occipital and parietal cortex. Regions in lateral parietal cortex also encoded remembered locations in both tasks, but in a format that differed from sensory-evoked activity. These results suggest a striking correspondence in the format of representations maintained in WM and retrieved from LTM across occipital and parietal cortex. On the other hand, we also show that activity patterns in nearly all parietal regions, but not occipital regions, contained information that could discriminate between WM and LTM trials. Our data provide new evidence for theories of memory systems and the representation of mnemonic content.

Replay in minds and machines

Experience-related brain activity patterns reactivate during sleep, wakeful rest, and brief pauses from active behavior. In parallel, machine learning research has found that experience replay can lead to substantial performance improvements in artificial agents. Together, these lines of research suggest replay has a variety of computational benefits for decision-making and learning. Here, we provide an overview of putative computational functions of replay as suggested by machine learning and neuroscientific research. We show that replay can lead to faster learning, less forgetting, reorganization or augmentation of experiences, and support planning and generalization. In addition, we highlight the benefits of reactivating abstracted internal representations rather than veridical memories, and discuss how replay could provide a mechanism to build internal representations that improve learning and decision-making.

Adaptive Memory Distortions Are Predicted by Feature Representations in Parietal Cortex

Similarity between memories is a primary cause of interference and forgetting. Exaggerating subtle differences between memories is therefore a potential mechanism for reducing interference. Here, we report a human fMRI study (n = 29, 19 female) that tested whether behavioral and neural expressions of memories are adaptively distorted to reduce interference. Participants learned and repeatedly retrieved object images, some of which were identical except for subtle color differences. Behavioral measures of color memory revealed exaggeration of differences between similar objects. Importantly, greater memory exaggeration was associated with lower memory interference. fMRI pattern analyses revealed that color information in parietal cortex was stronger during memory recall when color information was critical for discriminating competing memories. Moreover, greater representational distance between competing memories in parietal cortex predicted greater color memory exaggeration and lower memory interference. Together, these findings reveal that competition between memories induces adaptive, feature-specific distortions in parietal representations and corresponding behavioral expressions.SIGNIFICANCE STATEMENT Similarity between memories is a primary cause of interference and forgetting. Here, we show that, when remembering highly similar objects, subtle differences in the features of these objects are exaggerated in memory to reduce interference. These memory distortions are reflected in, and predicted by, overlap of activity patterns in lateral parietal cortex. These findings provide unique insight into how memory interference is resolved and specifically implicate lateral parietal cortex in representing feature-specific memory distortions.

Cortical Representations of Visual Stimuli Shift Locations with Changes in Memory States

Episodic memory retrieval is thought to rely on reactivation of the same content-sensitive neural activity patterns initially expressed during memory encoding.^1-6 Yet there are emerging examples of content representations expressed in different brain regions during encoding versus retrieval.^7-14 Although these differences have been observed by comparing encoding and retrieval tasks that differ in terms of perceptual experience and cognitive demands, there are many real-world contexts-e.g., meeting a new colleague who reminds you of an old acquaintance-where the memory system might be intrinsically biased either toward encoding (the new colleague) or retrieval (the old acquaintance).¹⁵¹⁶ Here, we test whether intrinsic memory states, independent of task demands, determine the cortical location of content representations. In a human fMRI study, subjects (n = 33) viewed object images and were instructed to either encode the current object or retrieve a similar object from memory. Using pattern classifiers, we show that biases toward encoding versus retrieval were reflected in large-scale attentional networks.^17-19 Critically, memory states decoded from these networks-even when entirely independent from task instructions-predicted shifts of object representations from visual cortex (encoding) to ventral parietal cortex (retrieval). Finally, visual versus ventral parietal cortices exhibited differential connectivity with the hippocampus during memory encoding versus retrieval, consistent with the idea that the hippocampus mediates cortical shifts in content representations. Collectively, these findings demonstrate that intrinsic biases toward memory encoding versus retrieval determine the specific cortical locations that express content information.

Quantifying aphantasia through drawing: Those without visual imagery show deficits in object but not spatial memory

Congenital aphantasia is a recently characterized variation of experience defined by the inability to form voluntary visual imagery, in individuals who are otherwise high performing. Because of this specific deficit to visual imagery, individuals with aphantasia serve as an ideal group for probing the nature of representations in visual memory, particularly the interplay of object, spatial, and symbolic information. Here, we conducted a large-scale online study of aphantasia and revealed a dissociation in object and spatial content in their memory representations. Sixty-one individuals with aphantasia and matched controls with typical imagery studied real-world scene images, and were asked to draw them from memory, and then later copy them during a matched perceptual condition. Drawings were objectively quantified by 2,795 online scorers for object and spatial details. Aphantasic participants recalled significantly fewer objects than controls, with less color in their drawings, and an increased reliance on verbal scaffolding. However, aphantasic participants showed high spatial accuracy equivalent to controls, and made significantly fewer memory errors. These differences between groups only manifested during recall, with no differences between groups during the matched perceptual condition. This object-specific memory impairment in individuals with aphantasia provides evidence for separate systems in memory that support object versus spatial information. The study also provides an important experimental validation for the existence of aphantasia as a variation in human imagery experience.