Delineating the effect of semantic congruency on episodic memory: the role of integration and relatedness

Integrating and fragmenting memories under stress and alcohol

Stress can powerfully influence the way we form memories, particularly the extent to which they are integrated or situated within an underlying spatiotemporal and broader knowledge architecture. These different representations in turn have significant consequences for the way we use these memories to guide later behavior. Puzzlingly, although stress has historically been argued to promote fragmentation, leading to disjoint memory representations, more recent work suggests that stress can also facilitate memory binding and integration. Understanding the circumstances under which stress fosters integration will be key to resolving this discrepancy and unpacking the mechanisms by which stress can shape later behavior. Here, we examine memory integration at multiple levels: linking together the content of an individual experience, threading associations between related but distinct events, and binding an experience into a pre-existing schema or sense of causal structure. We discuss neural and cognitive mechanisms underlying each form of integration as well as findings regarding how stress, aversive learning, and negative affect can modulate each. In this analysis, we uncover that stress can indeed promote each level of integration. We also show how memory integration may apply to understanding effects of alcohol, highlighting extant clinical and preclinical findings and opportunities for further investigation. Finally, we consider the implications of integration and fragmentation for later memory-guided behavior, and the importance of understanding which type of memory representation is potentiated in order to design appropriate interventions.

From silos to synergy: Integrating approaches to investigate the role of prior knowledge and expectations on episodic memory

Significant progress in the investigation of how prior knowledge influences episodic memory has been made using three sometimes isolated (but not mutually exclusive) approaches: strictly adult behavioral investigations, computational models, and investigations into the development of the system. Here we point out that these approaches are complementary, each approach informs and is informed by the other. Thus, a natural next step for research is to combine all three approaches to further our understanding of the role of prior knowledge in episodic memory. Here we use studies of memory for expectation-congruent and incongruent information from each of these often disparate approaches to illustrate how combining approaches can be used to test and revise theories from the other. This domain is particularly advantageous because it highlights important features of more general memory processes, further differentiates models of memory, and can shed light on developmental change in the memory system. We then present a case study to illustrate the progress that can be made from integrating all three approaches and highlight the need for more endeavors in this vein. As a first step, we also propose a new computational model of memory that takes into account behavioral and developmental factors that can influence prior knowledge and episodic memory interactions. This integrated approach has great potential for offering novel insights into the relationship between prior knowledge and episodic memory, and cognition more broadly.

Motivation and prediction-driven processing of social memoranda

Social semantic memory guides many aspects of behavior. Individuals rely on acquired and inferred knowledge about personal characteristics and group membership to predict the behavior and character of social targets. These predictions then determine the expectations from, the behavior in, and the interpretations of social interactions. According to predictive processing accounts, mnemonic and attentional mechanisms should enhance the processing of prediction-violating events. However, empirical findings suggest that prediction-consistent social events are often better remembered. This mini-review integrates recent evidence from social and non-social memory research to highlight the role of motivation in explaining these discrepancies. A particular emphasis is given to the continuous nature of prediction-(in)consistency, the epistemic tendency of perceivers to maintain or update their knowledge, and the dynamic influences of motivation on multiple steps in prediction-driven social memory. The suggested framework provides a coherent outlook of existing work and offers promising future directions to better understand the ebb and flow of social memoranda.

Schema-congruency supports the formation of unitized representations: Evidence from event-related potentials

The main goal of the present study was to investigate whether schema-based encoding of novel word pairs (i.e., novel compound words) supports the formation of unitized representations and thus, associative familiarity-based recognition. We report two experiments that both comprise an incidental learning task, in which novel noun-noun compound words were presented in semantically congruent contexts, enabling schema-supported processing of both constituents, contrasted with a schema-neutral condition. In Experiment 1, the effects of schema congruency on memory performance were larger for associative memory performance than for item memory performance in a memory test in which intact, recombined, and new compound words had to be discriminated. This supports the view that schema congruency boosts associative memory by promoting unitization. When contrasting event-related potentials (ERPs) for hits with correct rejections or associative misses, an N400 attenuation effect (520-676 ms) indicating absolute familiarity was present in the congruent condition, but not in the neutral condition. In line with this, a direct comparison of ERPs on hits across conditions revealed more positive waveforms in the congruent than in the neutral condition. This suggests that absolute familiarity contributes to associative recognition memory when schema-supported processing is established. In Experiment 2, we tested whether schema congruency enables the formation of semantically overlapping representations. Therefore, we included semantically similar lure compound words in the test phase and compared false alarm rates to these lures across conditions. In line with our hypothesis, we found higher false alarm rates in the congruent as compared to the neutral condition. In conclusion, we provide converging evidence for the view that schema congruency enables the formation of unitized representations and supports familiarity-based memory retrieval.

Harmonic memory signals in the human cerebral cortex induced by semantic relatedness of words

When we memorize multiple words simultaneously, semantic relatedness among those words assists memory. For example, the information about "apple", "banana," and "orange" will be connected via a common concept of "fruits" and become easy to retain and recall. Neural mechanisms underlying this semantic integration in verbal working memory remain unclear. Here I used electroencephalography (EEG) and investigated neural signals when healthy human participants memorized five nouns semantically related (Sem trial) or not (NonSem trial). The regularity of oscillatory signals (8-30 Hz) during the retention period was found to be lower in NonSem than Sem trials, indicating that memorizing words unrelated to each other induced a non-harmonic (irregular) waveform in the temporal cortex. These results suggest that (i) semantic features of a word are retained as a set of neural oscillations at specific frequencies and (ii) memorizing words sharing a common semantic feature produces harmonic brain responses through a resonance or integration (sharing) of the oscillatory signals.

Behavioral representational similarity analysis reveals how episodic learning is influenced by and reshapes semantic memory

While semantic and episodic memory have been shown to influence each other, uncertainty remains as to how this interplay occurs. We introduce a behavioral representational similarity analysis approach to assess whether semantic space can be subtly re-sculpted by episodic learning. Eighty participants learned word pairs that varied in semantic relatedness, and learning was bolstered via either testing or restudying. Next-day recall is superior for semantically related pairs, but there is a larger benefit of testing for unrelated pairs. Analyses of representational change reveal that successful recall is accompanied by a pulling together of paired associates, with cue words in semantically related (but not unrelated) pairs changing more across learning than target words. Our findings show that episodic learning is associated with systematic and asymmetrical distortions of semantic space which improve later recall by making cues more predictive of targets, reducing interference from potential lures, and establishing novel connections within pairs.

Predictions transform memories: How expected versus unexpected events are integrated or separated in memory

Our brains constantly generate predictions about the environment based on prior knowledge. Many of the events we experience are consistent with these predictions, while others might be inconsistent with prior knowledge and thus violate our predictions. To guide future behavior, the memory system must be able to strengthen, transform, or add to existing knowledge based on the accuracy of our predictions. We synthesize recent evidence suggesting that when an event is consistent with our predictions, it leads to neural integration between related memories, which is associated with enhanced associative memory, as well as memory biases. Prediction errors, in turn, can promote both neural integration and separation, and lead to multiple mnemonic outcomes. We review these findings and how they interact with factors such as memory reactivation, prediction error strength, and task goals, to offer insight into what determines memory for events that violate our predictions. In doing so, this review brings together recent neural and behavioral research to advance our understanding of how predictions shape memory, and why.

Conceptual relatedness promotes memory generalization at the cost of detailed recollection

An adaptive memory system is one that allows us to both retrieve detailed memories as well as generalize knowledge about our past, the latter termed memory generalization and is useful for making inferences about new situations. Research has indicated that memory generalization relies on forming knowledge structures by integrating experiences with shared encountered elements. Whether memory generalization occurs more readily when  experiences also have elements that share established (conceptual) information is less clear. It is also unclear if engaging in memory generalization during learning comes at the cost of retrieving detailed memories, the other function of episodic memory. To address these two knowledge gaps, we paired a modified version of the acquired equivalence task with a recognition memory test. Across three experiments, participants first learned a series of overlapping object-scene pairs (A-X, B-X and A-Y) in which half of the overlapping pairs contained conceptually-related objects (e.g., A-pencil; B-scissors; conceptual condition) and the other half contained unrelated objects (neutral condition). Participants ability to generalize to new overlapping object-scene pairs (B-Y) as well as not-learned but semantically-related objects was measured. Finally, participants completed a recognition memory test that included the encoded objects, perceptually similar lures or new foil objects. Across all experiments, we found higher rates of generalization but reduced detailed memory (indexed by increased false alarms to lure objects) for information learned in the conceptual than neutral condition. These results suggest the presence of conceptual knowledge biases an individual towards a generalization function of memory, which comes at the expense of detailed recollection.

Pre-associative item encoding influences associative memory: Behavioral and ERP evidence

It is unknown whether the manner with which an item is encoded in isolation, immediately before it is encoded into an inter-inter association, influences associative memory. We therefore presented the items of to-be-encoded associative pairings sequentially and manipulated how each first item of a pair was encoded (before associative encoding could begin). Furthermore, we recorded ERPs during memory encoding to investigate the neurocognitive processes that might relate pre-associative item encoding to subsequent associative memory performance. Behaviorally, we found that pre-associative item elaboration (vs. no elaboration) led to a memory tradeoff-enhanced item memory relative to impaired associative memory. This tradeoff likely reflected that item elaboration reduced cognitive resources for ensuing associative encoding, indexed by a reduced P300 and frontal slow wave at the time of associative encoding. However, frontal slow wave subsequent memory effects measured during pre-associative item encoding revealed that, for a given item, greater semantic elaboration was related to better item and associative memory while greater visual elaboration was related to better item and worse associative memory. Thus, there are likely two opposing ways in which pre-associative item encoding can influence associative memory: (1) by depleting encoding resources to impair associative memory and (2) by scaffolding inter-item associations to enhance associative memory. When item encoding occurs immediately before associative encoding, it appears that the temporary depletion of encoding resources is more important in determining later memory performance. Future research should compare the independent effects of resource depletion and encoding strategy during pre-associative item encoding.

Lack of source memory as a potential marker of early assimilation of novel items into current knowledge

In daily life, humans process a plethora of new information that can be either consistent (familiar) or inconsistent (novel) with prior knowledge. Over time, both types of information can integrate into our accumulated knowledge base via distinct pathways. However, the mnemonic processes supporting the integration of information that is inconsistent with prior knowledge remain under-characterized. In the current study, we used functional magnetic resonance imaging (fMRI) to examine the initial assimilation of novel items into the semantic network. Participants saw three repetitions of adjective-noun word pairs that were either consistent or inconsistent with prior knowledge. Twenty-four hours later, they were presented with the same stimuli again while undergoing fMRI scans. Outside the scanner, participants completed a surprise recognition test. We found that when the episodic context associated with initially inconsistent items was irretrievable, the neural signature of these items was indistinguishable from that of consistent items. In contrast, initially inconsistent items with accessible episodic contexts showed neural signatures that differed from those associated with consistent items. We suggest that, at least one day post encoding, items inconsistent with prior knowledge can show early assimilation into the semantic network only when their episodic contexts become inaccessible during retrieval, thus evoking a sense of familiarity.

Shape of U: The Nonmonotonic Relationship Between Object-Location Memory and Expectedness

The schema-linked interactions between medial prefrontal and medial temporal lobe (SLIMM) model predicts that memory for object locations is a U-shaped function of the expectancy of those locations. Using immersive virtual reality, we presented participants with 20 objects in locations that varied in their congruency with a kitchen schema. Bayes factors across four experiments (137 adults in total) confirmed the (preregistered) prediction of better memory for highly expected and unexpected locations relative to neutral locations. This U shape was found in location recall and in forced-choice recognition in which the foil locations were matched for expectancy, controlling for the bias toward guessing expected locations. A second prediction was that the two ends of the U shape are associated with different expressions of memory: recollection of unexpected locations and familiarity for expected locations. BFs, propagated across experiments, provided evidence against this second prediction; recollection was associated with both ends of the U shape. These findings further constrain theories about the role of schema in episodic memory.

Reward-motivated memories influence new learning across development

Previously rewarding experiences can influence choices in new situations. Past work has demonstrated that existing reward associations can either help or hinder future behaviors and that there is substantial individual variability in the transfer of value across contexts. Developmental changes in reward sensitivity may also modulate the impact of prior reward associations on later goal-directed behavior. The current study aimed to characterize how reward associations formed in the past affected learning in the present from childhood to adulthood. Participants completed a reinforcement learning paradigm using high- and low-reward stimuli from a task completed 24 h earlier, as well as novel stimuli, as choice options. We found that prior high-reward associations impeded learning across all ages. We then assessed how individual differences in the prioritization of high- versus low-reward associations in memory impacted new learning. Greater high-reward memory prioritization was associated with worse learning performance for previously high-reward relative to low-reward stimuli across age. Adolescents also showed impeded early learning regardless of individual differences in high-reward memory prioritization. Detrimental effects of previous reward on choice behavior did not persist beyond learning. These findings indicate that prior reward associations proactively interfere with future learning from childhood to adulthood and that individual differences in reward-related memory prioritization influence new learning across age.

Object memory is multisensory: Task-irrelevant sounds improve recollection

Hearing a task-irrelevant sound during object encoding can improve visual recognition memory when the sound is object-congruent (e.g., a dog and a bark). However, previous studies have only used binary old/new memory tests, which do not distinguish between recognition based on the recollection of details about the studied event or stimulus familiarity. In the present research, we hypothesized that hearing a task-irrelevant but semantically congruent natural sound at encoding would facilitate the formation of richer memory representations, resulting in increased recollection of details of the encoded event. Experiment 1 replicates previous studies showing that participants were more confident about their memory for items that were initially encoded with a congruent sound compared to an incongruent sound. Experiment 2 suggests that congruent object-sound pairings specifically facilitate recollection and not familiarity-based recognition memory, and Experiment 3 demonstrates that this effect was coupled with more accurate memory for audiovisual congruency of the item and sound from encoding rather than another aspect of the episode. These results suggest that even when congruent sounds are task-irrelevant, they promote formation of multisensory memories and subsequent recollection-based retention. Given the ubiquity of encounters with multisensory objects in our everyday lives, considering their impact on episodic memory is integral to building models of memory that apply to naturalistic settings.

The electrophysiological correlates of word pre-activation during associative word learning

Human beings continuously make use of learned associations to generate predictions about future occurrences in the environment. Such memory-related predictive processes provide a scaffold for learning in that mental representations of foreseeable events can be adjusted or strengthened based on a specific outcome. Learning the meaning of novel words through picture-word associations constitutes a prime example of associative learning because pictures preceding words can trigger word prediction through the pre-activation of the related mnemonic representations. In the present electroencephalography (EEG) study, we used event-related potentials (ERPs) to compare neural indices of word pre-activation between a word learning condition with maximal prediction likelihood and a non-learning control condition with low prediction. Results revealed that prediction-related N400 amplitudes in response to pictures decreased over time at central electrodes as a function of word learning, whereas late positive component (LPC) amplitudes increased. Notably, N400 but not LPC changes were also predictive of word learning performance, suggesting that the N400 component constitutes a sensitive marker of word pre-activation during associative word learning.

EXPRESS: Where was the Toaster? A systematic investigation of semantic construction in a new virtual episodic memory paradigm

Retrieved memories of past events are often inaccurate. The scenario construction model (SCM) postulates that during encoding, only the gist of an episode is stored in the episodic memory trace and during retrieval, information missing from that trace is constructed from semantic information. The current study aimed to find behavioural evidence for semantic construction in a realistic, yet controlled setting by introducing a new paradigm and adjusted memory tests that measure semantic construction. Using a desktop virtual reality (VR) participants navigated through a flat in which some household objects appeared in unexpected rooms, creating conflicts between the experienced episode and semantic expectations. The manipulation of congruence enabled us to identify influences from semantic information in cases of episodic memory failure. Besides we controlled for objects to be task-relevant or -irrelevant to the sequence of action. In addition to an established old/new recognition task we introduced spatial and temporal recall measures as possible superior memory measures quantifying semantic construction. The recognition task and the spatial recall revealed, that both congruence and task-relevance predicted correct episodic memory retrieval. In cases of episodic memory failure semantic construction was more likely than guessing and occurred more frequently for task-irrelevant objects. In the temporal recall object-pairs belonging to the same semantic room-category were temporally clustered together compared to object-pairs from different semantic categories (at the second retrieval). Taken together, our findings support the predictions of the SCM. The new VR-paradigm, including the new memory measures appears to be a promising tool for investigating semantic construction.

Semantic relatedness retroactively boosts memory and promotes memory interdependence across episodes

Two fundamental issues in memory research concern when later experiences strengthen or weaken initial memories and when the two memories become linked or remain independent. A promising candidate for explaining these issues is semantic relatedness. Here, across five paired-associate learning experiments (N=1000), we systematically varied the semantic relatedness between initial and later cues, initial and later targets, or both. We found that learning retroactively benefited long-term memory performance for semantically related words (vs. unshown control words), and these benefits increased as a function of relatedness. Critically, memory dependence between initial and later pairs also increased with relatedness, suggesting that pre-existing semantic relationships promote interdependence for memories formed across episodes. We also found that modest retroactive benefits, but not interdependencies, emerged when subjects learned via studying rather than practice testing. These findings demonstrate that semantic relatedness during new learning retroactively strengthens old associations while scaffolding new ones into well-fortified memory traces.

Emotion schema effects on associative memory differ across emotion categories at the behavioural, physiological and neural level: Emotion schema effects on associative memory differs for disgust and fear

Previous behavioural and neuroimaging studies have consistently reported that memory is enhanced for associations congruent or incongruent with the structure of prior knowledge, termed as schemas. However, it remains unclear if similar effects arise with emotion-related associations, and whether they depend on the type of emotions. Here, we addressed this question using a novel face-word pair association paradigm combined with fMRI and eye-tracking techniques. In two independent studies, we demonstrated and replicated that both congruency with emotion schemas and emotion category interact to affect associative memory. Overall, memory retrieval was higher for faces from pairs congruent vs. incongruent with emotion schemas, paralleled by a greater recruitment of left inferior frontal gyrus (IFG) during successful encoding. However, emotion schema effects differed across two negative emotion categories. Disgust was remembered better than fear, and only disgust activated left IFG stronger during encoding of congruent vs. incongruent pairs, suggestive of deeper semantic processing for the associations. On the contrary, encoding of congruent fear vs. disgust-related pairs was accompanied with greater activity in right fusiform gyrus (FG), suggesting a stronger sensory processing of faces. In addition, successful memory formation for congruent disgust pairs was associated with a higher pupil dilation index related to sympathetic activation, longer gaze time on words compared to faces, and more gaze switches between paired words and faces. This was reversed for fear-related congruent pairs where the faces attracted longer gaze time (compared to words). Overall, our results provide converging evidence from behavioural, physiological, and neural measures to suggest that congruency with available emotion schemas influence memory associations in a similar manner to semantic schemas. However, these effects vary across distinct emotion categories, pointing to a differential role of semantic processing and visual attention processes in the modulation of memory by disgust and fear, respectively.

The more you know: Schema-congruency supports associative encoding of novel compound words. Evidence from event-related potentials

We aimed to investigate the neurocognitive mechanisms of event congruency with prior (schema) knowledge for the learning of novel compound words. Event-related potentials (ERPs) were recorded during an incidental learning task, in which novel noun-noun compounds were presented in a semantically congruent context, enabling schema-supported processing, or in a neutral context. As expected, associative memory performance was better for compounds preceded by a congruent context. Although the N400 was attenuated in the congruent condition, subsequent memory effects (SMEs) in the N400 time interval did not differ across conditions, suggesting that the processes reflected in the N400 cannot account for the memory advantage in the congruent condition. However, a parietal SME was obtained for compounds preceded by a congruent context, only, which we interpret as reflecting the schema-supported formation of a conceptual compound representation. A late frontal SME was obtained in both conditions, presumably reflecting the more general inter-item associative encoding of compound constituents.

Building memories on prior knowledge: behavioral and fMRI evidence of impairment in early Alzheimer's disease

Impaired memory is a hallmark of prodromal Alzheimer's disease (AD). Prior knowledge associated with the memoranda improves memory in healthy individuals, but we ignore whether the same occurs in early AD. We used functional MRI to investigate whether prior knowledge enhances memory encoding in early AD, and whether the nature of this prior knowledge matters. Patients with early AD and Controls underwent a task-based fMRI experiment where they learned face-scene associations. Famous faces carried pre-experimental knowledge (PEK), while unknown faces with which participants were familiarized prior to learning carried experimental knowledge (EK). Surprisingly, PEK strongly enhanced subsequent memory in healthy controls, but importantly not in patients. Partly nonoverlapping brain networks supported PEK vs. EK associative encoding in healthy controls. No such networks were identified in patients. In addition, patients displayed impaired activation in a right sub hippocampal region where activity predicted successful associative memory formation for PEK stimuli. Despite the limited sample sizes of this study, these findings suggest that the role prior knowledge in new learning might have been so far overlooked and underestimated in AD patients. Prior knowledge may drive critical differences in the way healthy elderly and early AD patients learn novel associations.

Bizarreness and Typicality Effects of Color on Object Recognition Memory

The bizarreness effect on object recognition is a strong phenomenon, but its influence has been inconsistent for bizarre object color. In this study, we manipulated three factors in separate experiments to determine whether a color bizarreness effect on object recognition memory would occur and, if not, why. Participants first saw (i.e., learned) object pictures that were either bizarrely or typically colored; they then completed a recognition memory test. In three experiments, we then manipulated (a) degree of color bizarreness (Experiment 1), (b) the orientation task (Experiment 2), and (c) additional demands for object identification (Experiment 3). In Experiment 1, we provided 49 undergraduate participants with object pictures whose colors were typical, moderately atypical, or bizarre and found no color bizarreness effect on recognition memory even for extremely bizarre colors. In Experiment 2, we manipulated the orientation task in that 28 young adult participants expressed their preferences for the pictures on a three-point scale while another 28 participants judged how natural the pictures were. Each orientation task group better recognized typically-colored rather than bizarrely-colored objects (typicality effect). In Experiment 3, we asked 27 young adults to identify the objects during the learning phase to ensure that they paid attention to the objects' bizarre colors; recognition memory was then unaffected by either color bizarreness or typicality. Thus, despite a general bizarreness effect in recognition memory, bizarre colors are less likely to influence object recognition memory.

Neural Correlates of Successful Memory Encoding in Kindergarten and Early Elementary School Children: Longitudinal Trends and Effects of Schooling

From age 5 to 7, there are remarkable improvements in children's cognitive abilities ("5-7 shift"). In many countries, including Germany, formal schooling begins in this age range. It is, thus, unclear to what extent exposure to formal schooling contributes to the "5-7 shift." In this longitudinal study, we investigated if schooling acts as a catalyst of maturation. We tested 5-year-old children who were born close to the official cutoff date for school entry and who were still attending a play-oriented kindergarten. One year later, the children were tested again. Some of the children had experienced their first year of schooling whereas the others had remained in kindergarten. Using 2 functional magnetic resonance imaging tasks that assessed episodic memory formation (i.e., subsequent memory effect), we found that children relied strongly on the medial temporal lobe (MTL) at both time points but not on the prefrontal cortex (PFC). In contrast, older children and adults typically show subsequent memory effects in both MTL and PFC. Both children groups improved in their memory performance, but there were no longitudinal changes nor group differences in neural activation. We conclude that successful memory formation in this age group relies more heavily on the MTL than in older age groups.

Encoding-linked pupil response is modulated by expected and unexpected novelty: Implications for memory formation and neurotransmission

Whether a novel stimulus is expected or unexpected may have implications for the kind of ensuing encoding and the type of subsequent memory. Pupil response was used in the present study to explore the way expected and unexpected stimuli are encoded and whether encoding-linked pupil response is modulated by expectation. Participants first established a contingency relationship between a series of symbols and the type of stimulus (man-made or natural) that followed each one. At encoding, some of the target stimuli violated the previously established relationship (i.e., unexpected), while the majority conformed to this relationship (i.e., expected). Expectation at encoding had opposite effects on familiarity and recollection, the two types of memory that support recognition, and modulated differently the way pupil response predicted subsequent memory. Encoding of unexpected novel stimuli was associated with increased pupil dilation as a predictor of subsequent memory type and strength. In contrast, encoding of expected novel stimuli was associated with decreased pupil response (constriction), which was predictive of subsequent memory type and strength. The findings support the close link between pupil response and memory formation, but critically indicate that this is modulated by the type of novelty as defined by expectation. These novel findings have important implications for the encoding mechanisms involved when different types of novelty are detected and is proposed to indicate the operation of different neurotransmitters during memory formation.

Unitization could facilitate item recognition but inhibit verbatim recognition for picture stimuli: behavioral and event-related potential study

It is widely accepted that unitization can promote familiarity-based associative recognition, but the effect of unitization on item recognition remains unclear. The goals of this study were to elaborate on how unitization affected item recognition and the neural correlates of familiarity and recollection for item recognition. In study, the participants were asked to learn related and unrelated picture pairs, and in test, they were required to distinguish single old pictures form new pictures. In experiment 1, we used R/K/N paradigm to estimate the contribution of familiarity and recollection to item recognition, the results showed that unitization could improve item recognition through increasing recollection selectively. In experiment 2, we used ERP old/new effects to estimate the neural correlates of familiarity and recollection, the results showed that unitization could improve item recognition through a selective reduction in LPC effect. Inspired by DRM paradigm, in experiment 3, we divided the new pictures into semantically related lure pictures and semantically unrelated new pictures to explore the effects of unitization on item recognition and verbatim recognition (the ability to distinguish old pictures from lure pictures). The behavioral results showed that unitization could improve item recognition, but it damaged verbatim recognition. The ERP results revealed that there were larger LPC effects in the unrelated condition than in the related condition, regardless of item or verbatim recognition. In summary, we believed that unitization could indeed improve item recognition, but it damaged verbatim recognition.

Schematic memories develop quickly, but are not expressed unless necessary

Episodic memory retrieval is increasingly influenced by schematic information as memories mature, but it is unclear whether this is due to the slow formation of schemas over time, or the slow forgetting of the episodes. To address this, we separately probed memory for newly learned schemas as well as their influence on episodic memory decisions. In this experiment, participants encoded images from two categories, with the location of images in each category drawn from a different spatial distribution. They could thus learn schemas of category locations by encoding specific episodes. We found that images that were more consistent with these distributions were more precisely retrieved, and this schematic influence increased over time. However, memory for the schema distribution, measured using generalization to novel images, also became less precise over time. This incongruity suggests that schemas form rapidly, but their influence on episodic retrieval is dictated by the need to bolster fading memory representations.

Experience Matters

Abstract. The present study tested if word frequency effects on judgments of learning (JOLs) are exclusively due to beliefs or if the direct experience with the items also plays a role. Across four experiments, participants read prompts about the frequency of the words (high/low), which could be congruent/incongruent with the words’ actual frequency. They made pre-study JOLs (except Experiment 1b), immediate JOLs, and completed a recall test. If experience drives the effect, JOLs should be based on actual word frequency rather than the prompts. Results showed higher pre-study JOLs for prompts of high frequency, but higher immediate JOLs for high-frequency words regardless of the prompt, suggesting an effect of direct experience with the words. In Experiments 2 and 3, we manipulated participants’ beliefs, finding a small effect of beliefs on JOLs. We conclude that, regarding word frequency, direct experience with the items seems more relevant than beliefs when making immediate JOLs.

Prior knowledge promotes hippocampal separation but cortical assimilation in the left inferior frontal gyrus

An adaptive memory system rarely learns information tabula rasa, but rather builds on prior knowledge to facilitate learning. How prior knowledge influences the neural representation of novel associations remains unknown. Here, participants associated pairs of faces in two conditions: a famous, highly familiar face with a novel face or two novel faces while undergoing fMRI. We examine multivoxel activity patterns corresponding to individual faces before and after learning. The activity patterns representing members of famous-novel pairs becomes separated in the hippocampus, that is, more distinct from one another through learning, in striking contrast to paired novel faces that become similar. In the left inferior frontal gyrus, however, prior knowledge leads to integration, and in a specific direction: the representation of the novel face becomes similar to that of the famous face after learning, suggesting assimilation of new into old memories. We propose that hippocampal separation might resolve interference between existing and newly learned information, allowing cortical assimilation. Thus, associative learning with versus without prior knowledge relies on radically different computations.

Prior knowledge strongly impacts new learning, but its influence on the neural representation of novel information is unknown. Here, the authors show multiple neural codes for learning: prior knowledge leads to integrated cortical representations, while promoting hippocampal separation.

Future thinking about social targets: The influence of prediction outcome on memory

Work on future thinking suggests that people use what they know about the world (e.g., contents of memory) to make predictions about events to come, which reflects an adaptive use of memory. Less work, however, has examined whether the outcomes of these predictions-whether the outcome is consistent or inconsistent with predictions-influences memory. In two experiments, participants learned trait information about social targets and used that information to predict which of two behaviors social targets would be most likely to engage in: one behavior consistent with previously learned trait information about the target and the other behavior inconsistent. Participants then learned which behavior the social target actually performed (outcome) and then judged whether or not they expected that outcome (expectancy). Across both studies, prediction-consistent outcomes were better remembered than inconsistent ones, suggesting that participants relied on their existing representations of social targets when making memory judgments rather than incorporating inconsistent information into memory. Further, there was a memory advantage for prediction-inconsistent outcomes, but only when participants subjectively rated these outcomes as unexpected. Overall, these findings extend understanding of future thinking and suggest a reliable memory advantage for outcomes that are consistent with predictions.

The congruent, the incongruent, and the unexpected: Event-related potentials unveil the processes involved in schematic encoding

Learning is most effective when new information can be related to a preexisting knowledge structure or schema. In the present study, event-related potentials (ERPs) were used to investigate the temporal dynamics of the processes by which activated schemata support the encoding of schema-congruent information. Participants learned category exemplar words that were either semantically congruent or incongruent with a preceding category cue phrase. Congruent words were composed of expected (high typicality, HT) and unexpected (low typicality, LT) category exemplars. On the next day, recognition memory for the exemplars and the category cues they were presented with was tested. Semantically related lures were used in order to ascertain that memory judgements were based on episodic memory for specific category exemplars. Generally, congruent (HT and LT) exemplars were remembered better than incongruent exemplars. ERPs recorded during the encoding of the exemplar words were compared for subsequently remembered and forgotten items. Subsequent memory effects (SME) emerged in the N400 time window at frontal electrodes and did not differ between congruent and incongruent exemplars. In the same epoch, an SME with a parietal distribution was specific for congruent exemplars, suggesting that activated schemata strengthened memory for congruent exemplars by supporting the encoding of item-specific details. Subsequently remembered LT exemplars were associated with a late frontal positivity that is assumed to reflect expectancy mismatch-related processing such as the contextual integration of an unexpected word by the suppression of strongly expected words. A correlation analysis revealed that the greater the involvement of the processes reflected by the frontal positivity, the lower the level of false positive memory responses in the test phase one day later. These results suggest that the contextual integration of schema-congruent but unexpected events involves a weakening of the representations of semantically related, but unstudied items in memory and by this benefits subsequent memory.

Elaboration Benefits Source Memory Encoding Through Centrality Change

Variations in levels of processing affect memory encoding and subsequent retrieval performance, but it is unknown how processing depth affects communication patterns within the network of interconnected brain regions involved in episodic memory encoding. In 113 healthy adults scanned with functional MRI, we used graph theory to calculate centrality indices representing the brain regions' relative importance in the memory network. We tested how communication patterns in 42 brain regions involved in episodic memory encoding changed as a function of processing depth, and how these changes were related to episodic memory ability. Centrality changes in right middle frontal gyrus, right inferior parietal lobule and left superior frontal gyrus were positively related to semantic elaboration during encoding. In the same regions, centrality during successful episodic memory encoding was related to performance on the episodic memory task, indicating that these centrality changes reflect processes that support memory encoding through deep elaborative processing. Similar analyses were performed for congruent trials, i.e. events that fit into existing knowledge structures, but no relationship between centrality changes and congruity were found. The results demonstrate that while elaboration and congruity have similar beneficial effects on source memory performance, the cortical signatures of these processes are probably not identical.

Knowledge is power: Prior knowledge aids memory for both congruent and incongruent events, but in different ways

Events that conform to our expectations, that is, are congruent with our world knowledge or schemas, are better remembered than unrelated events. Yet events that conflict with schemas can also be remembered better. We examined this apparent paradox in 4 experiments, in which schemas were established by training ordinal relationships between randomly paired objects, whereas event memory was tested for the number of objects on each trial. Better memory was found for both congruent and incongruent trials, relative to unrelated trials, producing memory performance that was a "U-shaped" function of congruency. The congruency advantage but not incongruency advantage was mediated by postencoding processes, whereas the incongruency advantage, but not congruency advantage, emerged even if the information probed by the memory test was irrelevant to the schema. Schemas therefore augment event memory in multiple ways, depending on the match between novel and existing information. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Rapid, experience-related changes in the organization of children's semantic knowledge

The organization of knowledge according to relations between concepts is crucially important for many cognitive processes, and its emergence during childhood is a key focus of cognitive development research. Prior evidence about the role of learning and experience in the development of knowledge organization primarily comes from studies investigating differences between preexisting, naturally occurring groups (e.g., children from rural vs. urban settings, children who own a pet vs. children who do not) and a handful of studies on the effects of researcher-developed educational interventions. However, we know little about whether knowledge organization can be relatively rapidly molded by shorter-term real-world learning experiences (e.g., on a timescale of days vs. years or months). The current study investigated whether naturalistic learning experiences can drive rapid measurable changes in knowledge organization in children by investigating the effects of a week-long zoo summer camp (compared with a control school-based camp) on the degree to which 4- to 9-year-old children's knowledge about animals was organized according to taxonomic relations. Although there were no differences in taxonomic organization between the zoo camp and the school-based camp at pretest, only children who participated in the zoo camp showed increases in taxonomic organization at posttest. Moreover, analyses of changes in taxonomic organization in zoo camp children suggested that these changes were primarily driven by improvements in the degree to which children differentiated between taxonomic categories. These findings provide novel evidence that naturalistic experiences can drive rapid changes in knowledge organization.

Expectation affects learning and modulates memory experience at retrieval

Our ability to make predictions and monitor regularities has a profound impact on the way we perceive the environment, but the effect this mechanism has on memory is not well understood. In four experiments, we explored the effects on memory of the expectation status of information at encoding or at retrieval. In a rule-learning task participants learned a contingency relationship between 6 different symbols and the type of stimulus that followed each one. Either at encoding (Experiments 1a and 1b) or at retrieval (Experiments 2a and 2b), the established relationship was violated for a subset of stimuli resulting in the presentation of both expected and unexpected stimuli. The expectation status of the stimuli was found to have opposite effects on familiarity and recollection performance, the two kinds of memory that support recognition memory. At encoding (Experiments 1a and 1b), the presentation of expected stimuli selectively enhanced subsequent familiarity performance, while unexpected stimuli selectively enhanced subsequent recollection. Similarly, at retrieval (Experiments 2a and 2b), expected stimuli were more likely to be deemed familiar than unexpected stimuli, whereas unexpected stimuli were more likely to be recollected than were expected stimuli. These findings suggest that two separate memory enhancement mechanisms exist; one sensitive and modulating the accuracy of memory for the contextually distinctive or unexpected, and the other sensitive to and modulating the accuracy of memory for the expected. Therefore, the degree to which information fits with expectation has critical implications for the type of computational mechanism that will be engaged to support memory.

Beneficial and detrimental effects of schema incongruence on memory for contextual events

Mental schemas provide a framework into which new information can easily be integrated. In a series of experiments, we examined how incongruence that stems from a prediction error modulates memory for multicomponent events that instantiated preexisting schemas as noted in a previous study. Each event consisted of four stimulus pairs with overlapping components, presented in four blocks (A-B, B-C, C-D, D-A). A-B pairs elicited contextual expectations (A: Farm, B: Tractor) that were either met by a congruent C component (C: Farmer) or violated by an incongruent one (C: Lawyer). The baseline condition included unrelated pairs, where the C component was neither congruent nor incongruent. In experiment 2, events were presented in successive trials instead of blocks, and eye movements were recorded to analyze allocation of attention. Memory was tested through old-new item recognition followed by cued recall. Across experiments, recognition and recall performance for incongruent components was reduced compared to congruent components. Incongruent items were in some cases more accurately retrieved compared to unrelated ones, depending on task demands. Additionally, better recall was observed in the incongruent D-A pairs, compared to congruent and unrelated ones, because of reduced interference from C components. Eye-tracking revealed an increased number of fixations on C components in the incongruent and unrelated conditions. These results suggest that the integration of incongruent items into an episode is impaired, compared to congruent items, despite the contextual surprise and increased attention they elicited at encoding. However, there was a beneficial effect of prediction error on memory performance, compared to a baseline, depending on the task used.

A multisensory perspective on object memory

Traditional studies of memory and object recognition involved objects presented within a single sensory modality (i.e., purely visual or purely auditory objects). However, in naturalistic settings, objects are often evaluated and processed in a multisensory manner. This begets the question of how object representations that combine information from the different senses are created and utilised by memory functions. Here we review research that has demonstrated that a single multisensory exposure can influence memory for both visual and auditory objects. In an old/new object discrimination task, objects that were presented initially with a task-irrelevant stimulus in another sense were better remembered compared to stimuli presented alone, most notably when the two stimuli were semantically congruent. The brain discriminates between these two types of object representations within the first 100ms post-stimulus onset, indicating early "tagging" of objects/events by the brain based on the nature of their initial presentation context. Interestingly, the specific brain networks supporting the improved object recognition vary based on a variety of factors, including the effectiveness of the initial multisensory presentation and the sense that is task-relevant. We specify the requisite conditions for multisensory contexts to improve object discrimination following single exposures, and the individual differences that exist with respect to these improvements. Our results shed light onto how memory operates on the multisensory nature of object representations as well as how the brain stores and retrieves memories of objects.

Semantic Congruence Accelerates the Onset of the Neural Signals of Successful Memory Encoding

As the stream of experience unfolds, our memory system rapidly transforms current inputs into long-lasting meaningful memories. A putative neural mechanism that strongly influences how input elements are transformed into meaningful memory codes relies on the ability to integrate them with existing structures of knowledge or schemas. However, it is not yet clear whether schema-related integration neural mechanisms occur during online encoding. In the current investigation, we examined the encoding-dependent nature of this phenomenon in humans. We showed that actively integrating words with congruent semantic information provided by a category cue enhances memory for words and increases false recall. The memory effect of such active integration with congruent information was robust, even with an interference task occurring right after each encoding word list. In addition, via electroencephalography, we show in 2 separate studies that the onset of the neural signals of successful encoding appeared early (∼400 ms) during the encoding of congruent words. That the neural signals of successful encoding of congruent and incongruent information followed similarly ∼200 ms later suggests that this earlier neural response contributed to memory formation. We propose that the encoding of events that are congruent with readily available contextual semantics can trigger an accelerated onset of the neural mechanisms, supporting the integration of semantic information with the event input. This faster onset would result in a long-lasting and meaningful memory trace for the event but, at the same time, make it difficult to distinguish it from plausible but never encoded events (i.e., related false memories).

SIGNIFICANCE STATEMENT

Conceptual or schema congruence has a strong influence on long-term memory. However, the question of whether schema-related integration neural mechanisms occur during online encoding has yet to be clarified. We investigated the neural mechanisms reflecting how the active integration of words with congruent semantic categories enhances memory for words and increases false recall of semantically related words. We analyzed event-related potentials during encoding and showed that the onset of the neural signals of successful encoding appeared early (∼400 ms) during the encoding of congruent words. Our findings indicate that congruent events can trigger an accelerated onset of neural encoding mechanisms supporting the integration of semantic information with the event input.

Distinctiveness Benefits Novelty (and Not Familiarity), but Only Up to a Limit: The Prior Knowledge Perspective

Novelty is a pivotal player in cognition, and its contribution to superior memory performance is a widely accepted convention. On the other hand, mnemonic advantages for familiar information are also well documented. Here, we examine the role of experimental distinctiveness as a potential explanation for these apparently conflicting findings. Across two experiments, we demonstrate that conceptual novelty, an unfamiliar combination of familiar constituents, is sensitive to its experimental proportions: Improved memory for novelty was observed when novel stimuli were relatively rare. Memory levels for familiar items, in contrast, were completely unaffected by experimental proportions, highlighting their insensitivity to list-based distinctiveness. Finally, no mnemonic advantage for conceptual novelty over familiarity was observed even when novel stimuli were extremely rare at study. Together, these results imply that novel and familiar items are processed via partially distinct mechanisms, with (at least some facets of) novelty not providing a mnemonic advantage over familiarity.

Semantic Congruency Improves Recognition Memory Performance for Both Audiovisual and Visual Stimuli

Audiovisual semantic congruency during memory encoding has been shown to facilitate later recognition memory performance. However, it is still unclear whether this improvement is due to multisensory semantic congruency or just semantic congruencyper se. We investigated whether dual visual encoding facilitates recognition memory in the same way as audiovisual encoding. The participants memorized auditory or visual stimuli paired with a semantically congruent, incongruent or non-semantic stimulus in the same modality or in the other modality during encoding. Subsequent recognition memory performance was better when the stimulus was initially paired with a semantically congruent stimulus than when it was paired with a non-semantic stimulus. This congruency effect was observed with both audiovisual and dual visual stimuli. The present results indicate that not only multisensory but also unisensory semantically congruent stimuli can improve memory performance. Thus, the semantic congruency effect is not solely a multisensory phenomenon, as has been suggested previously.

Distinct Neural Suppression and Encoding Effects for Conceptual Novelty and Familiarity

Like yin and yang, novelty and familiarity are typically described as separate-yet-complementary aspects of an experience, two ends of a single continuum. However, novelty and familiarity are also multifaceted. For instance, novelty can sometimes result in enhanced mnemonic performance, whereas at other times familiarity is better remembered. As previous investigations focused primarily on the experimental aspect of novelty, the mechanisms supporting conceptual novelty (the novel combination of two previously unrelated existing concepts) remain unclear. Importantly, conceptual novelty can be recognized as such only when compared with preexperimental familiar knowledge, regardless of experimental status. Here we applied a combined repetition suppression/subsequent memory fMRI paradigm, focusing on the conceptual aspect of novelty and familiarity as the subject matter. Conceptual novelty was characterized by sustained neural activity; familiarity, on the other hand, exhibited repetition effects in multiple cortical regions, a subset of which was modulated by successful encoding. Subsequent memory of novelty was associated only with activation differences in a distinct set of regions, including the hippocampus and medial cortical regions. These results suggest that conceptual novelty (a) does not (easily) trigger the repetition suppression phenomenon but requires sustained neural recruitment and (b) activates dedicated encoding mechanisms. Conceptual familiarity, in contrast, allows rapid neural processing that depends upon existing neural representations. Overall, these findings challenge the definition of novelty as a unitary concept. Furthermore, they bear important implications for research into the neural bases of knowledge representation and recognition memory.

Semantic congruence affects hippocampal response to repetition of visual associations

Recent research has shown complementary engagement of the hippocampus and medial prefrontal cortex (mPFC) in encoding and retrieving associations based on pre-existing or experimentally-induced schemas, such that the latter supports schema-congruent information whereas the former is more engaged for incongruent or novel associations. Here, we attempted to explore some of the boundary conditions in the relative involvement of those structures in short-term memory for visual associations. The current literature is based primarily on intentional evaluation of schema-target congruence and on study-test paradigms with relatively long delays between learning and retrieval. We used a continuous recognition paradigm to investigate hippocampal and mPFC activation to first and second presentations of scene-object pairs as a function of semantic congruence between the elements (e.g., beach-seashell versus schoolyard-lamp). All items were identical at first and second presentation and the context scene, which was presented 500ms prior to the appearance of the target object, was incidental to the task which required a recognition response to the central target only. Very short lags 2-8 intervening stimuli occurred between presentations. Encoding the targets with congruent contexts was associated with increased activation in visual cortical regions at initial presentation and faster response time at repetition, but we did not find enhanced activation in mPFC relative to incongruent stimuli at either presentation. We did observe enhanced activation in the right anterior hippocampus, as well as regions in visual and lateral temporal and frontal cortical regions, for the repetition of incongruent scene-object pairs. This pattern demonstrates rapid and incidental effects of schema processing in hippocampal, but not mPFC, engagement during continuous recognition.

Effortful semantic decision-making boosts memory performance in older adults

A major concern in age-related cognitive decline is episodic memory (EM). Previous studies indicate that both resource and binding deficits contribute to EM decline. Environmental support by task manipulations encouraging stronger cognitive effort and deeper levels of processing may facilitate compensation for these two deficits. To clarify factors that can counteract age-related EM decline, we assessed effects of cognitive effort (four levels) and level of processing (LoP, shallow/deep) during encoding on subsequent retrieval. Young (YAs, N = 23) and older (OAs, N = 23) adults performed two incidental encoding tasks, deep/semantic and shallow/perceptual. Cognitive effort was manipulated by varying decision-making demands. EM performance, indexed by d-prime, was later tested using a recognition task. Results showed that regardless of LoP, increased cognitive effort caused higher d-primes in both age groups. Compared to YAs, OAs showed a lower d-prime after shallow encoding across all cognitive effort levels, and after deep encoding with low cognitive effort. Deep encoding with higher levels of cognitive effort completely eliminated these age differences. Our findings support an environmental-compensatory account of cognitive ageing and can have important therapeutic implications.