Stable maintenance of multiple representational formats in human visual short-term memory

Electrophysiological signatures underlying variability in human memory consolidation

We experience countless pieces of new information each day, but remembering them later depends on firmly instilling memory storage in the brain. Numerous studies have implicated non-rapid eye movement (NREM) sleep in consolidating memories via interactions between hippocampus and cortex. However, the temporal dynamics of this hippocampal-cortical communication and the concomitant neural oscillations during memory reactivations remains unclear. To address this issue, the present study used the procedure of targeted memory reactivation (TMR) following learning of object-location associations to selectively reactivate memories during human NREM sleep. Cortical pattern reactivation and hippocampal-cortical coupling were measured with intracranial EEG recordings in patients with epilepsy. We found that TMR produced variable amounts of memory enhancement across a set of object-location associations. Successful TMR increased hippocampal ripples and cortical spindles, apparent during two discrete sweeps of reactivation. The first reactivation sweep was accompanied by increased hippocampal-cortical communication and hippocampal ripple events coupled to local cortical activity (cortical ripples and high-frequency broadband activity). In contrast, hippocampal-cortical coupling decreased during the second sweep, while increased cortical spindle activity indicated continued cortical processing to achieve long-term storage. Taken together, our findings show how dynamic patterns of item-level reactivation and hippocampal-cortical communication support memory enhancement during NREM sleep.

Reinstatement and transformation of memory traces for recognition

Episodic memory relies on the formation and retrieval of content-specific memory traces. In addition to their veridical reactivation, previous studies have indicated that traces may undergo substantial transformations. However, the exact time course and regional distribution of reinstatement and transformation during recognition memory have remained unclear. We applied representational similarity analysis to human intracranial electroencephalography to track the spatiotemporal dynamics underlying the reinstatement and transformation of memory traces. Specifically, we examined how reinstatement and transformation of item-specific representations across occipital, ventral visual, and lateral parietal cortices contribute to successful memory formation and recognition. Our findings suggest that reinstatement in temporal cortex and transformation in parietal cortex coexist and provide complementary strategies for recognition. Further, we find that generalization and differentiation of neural representations contribute to memory and probe memory-specific correspondence with deep neural network (DNN) model features. Our results suggest that memory formation is particularly supported by generalized and mnemonic representational formats beyond the visual features of a DNN.

Reactivating and reorganizing activity-silent working memory: two distinct mechanisms underlying pinging the brain

Recent studies have proposed that visual information in working memory (WM) can be maintained in an activity-silent state and reactivated by task-irrelevant high-contrast visual impulses ("ping"). Although pinging the brain has become a popular tool for exploring activity-silent WM, its underlying mechanisms remain unclear. In the current study, we directly compared the neural reactivation effects and behavioral consequences of spatial-nonmatching and spatial-matching pings to distinguish the noise-reduction and target-interaction hypotheses of pinging the brain. Initially, in an electroencephalogram study, our neural decoding results showed that spatial-nonmatching pings reactivated activity-silent WM transiently without changing the original WM representations or recall performance. Conversely, spatial-matching pings reactivated activity-silent WM more durably and further reorganized WM information by decreasing neural representations' dynamics. Notably, only the reactivation strength of spatial-matching pings correlated with recall performance and was modulated by the location of memorized items, with neural reactivation occurring only when both items and pings were presented horizontally. Consistently, in a follow-up behavioral study, we found that only spatial-matching, horizontal pings impaired recall performance compared to no ping. Together, our results demonstrated two distinct mechanisms underlying pinging the brain, highlighting the critical role of the ping's context (i.e. spatial information) in reactivating and reorganizing activity-silent WM.

Neural deterioration and compensation in visual short-term memory among individuals with amnestic mild cognitive impairment

INTRODUCTION

Visual short-term memory (VSTM) is a critical indicator of Alzheimer's disease (AD), but whether its neural substrates could adapt to early disease progression and contribute to cognitive resilience in amnestic mild cognitive impairment (aMCI) has been unclear.

METHODS

Fifty-five aMCI patients and 68 normal controls (NC) performed a change-detection task and underwent multimodal neuroimaging scanning.

RESULTS

Among the atrophic brain regions in aMCI, VSTM performance correlated with the volume of the right prefrontal cortex (PFC) but not the medial temporal lobe (MTL), and this correlation was mainly present in patients with greater MTL atrophy. Furthermore, VSTM was primarily correlated with frontal structural connectivity in aMCI but was correlated with more distributed frontal and MTL connectivity in NC.

DISCUSSION

This study provided evidence on neural adaptation in the precursor stages of AD, highlighting the compensatory role of PFC as MTL deteriorated and suggesting potential targets in early intervention for cognitive preservation.

HIGHLIGHTS

Atrophic left medial temporal lobe (MTL) no longer correlated with visual short-term memory (VSTM) in amnestic mild cognitive impairment (aMCI). Atrophic right middle frontal area continued to correlate with VSTM in aMCI. Frontal brain-behavior correlation was mainly present in the aMCI subgroup with greater medial temporal lobe (MTL) atrophy. Reliance of VSTM on frontal connectivity increased in compensation for MTL dysfunction.

Simultaneous intracranial recordings of interacting brains reveal neurocognitive dynamics of human cooperation

Cooperative interactions profoundly shape individual and collective behaviors of social animals. Successful cooperation requires coordinated efforts by cooperators toward collective goals. However, the underlying behavioral dynamics and neuronal mechanisms within and between cooperating brains remain largely unknown. We recorded intracranial electrophysiological signals from human pairs engaged in a cooperation game. We show that teammate coordination and goal pursuit make distinct contributions to the behavioral cooperation dynamics. Increases and decreases in high-gamma activity in the temporoparietal junction (TPJ) and amygdala distinguish between establishing and maintaining cooperation and forecast transitions between these two states. High-gamma activity from distinct neuronal populations encodes teammate coordination and goal pursuit motives, with populations of TPJ neurons preferentially tracking dominant motives of different cooperation states. Across cooperating brains, high-gamma activity in the TPJ and amygdala synchronizes in a state-dependent manner that predicts how well cooperators coordinate. These findings provide fine-grained understandings of human cooperation dynamics as a state-dependent process with distinctive neurocognitive profiles of each state.

Re-evaluating human MTL in working memory: insights from intracranial recordings

The study of human working memory (WM) holds significant importance in neuroscience; yet, exploring the role of the medial temporal lobe (MTL) in WM has been limited by the technological constraints of noninvasive methods. Recent advancements in human intracranial neural recordings have indicated the involvement of the MTL in WM processes. These recordings show that different regions of the MTL are involved in distinct aspects of WM processing and also dynamically interact with each other and the broader brain network. These findings support incorporating the MTL into models of the neural basis of WM. This integration can better reflect the complex neural mechanisms underlying WM and enhance our understanding of WM's flexibility, adaptability, and precision.

Convolutional neural networks uncover the dynamics of human visual memory representations over time

The ability to accurately retrieve visual details of past events is a fundamental cognitive function relevant for daily life. While a visual stimulus contains an abundance of information, only some of it is later encoded into long-term memory representations. However, an ongoing challenge has been to isolate memory representations that integrate various visual features and uncover their dynamics over time. To address this question, we leveraged a novel combination of empirical and computational frameworks based on the hierarchal structure of convolutional neural networks and their correspondence to human visual processing. This enabled to reveal the contribution of different levels of visual representations to memory strength and their dynamics over time. Visual memory strength was measured with distractors selected based on their shared similarity to the target memory along low or high layers of the convolutional neural network hierarchy. The results show that visual working memory relies similarly on low and high-level visual representations. However, already after a few minutes and on to the next day, visual memory relies more strongly on high-level visual representations. These findings suggest that visual representations transform from a distributed to a stronger high-level conceptual representation, providing novel insights into the dynamics of visual memory over time.

Maintenance and transformation of representational formats during working memory prioritization

Visual working memory depends on both material-specific brain areas in the ventral visual stream (VVS) that support the maintenance of stimulus representations and on regions in the prefrontal cortex (PFC) that control these representations. How executive control prioritizes working memory contents and whether this affects their representational formats remains an open question, however. Here, we analyzed intracranial EEG (iEEG) recordings in epilepsy patients with electrodes in VVS and PFC who performed a multi-item working memory task involving a retro-cue. We employed Representational Similarity Analysis (RSA) with various Deep Neural Network (DNN) architectures to investigate the representational format of prioritized VWM content. While recurrent DNN representations matched PFC representations in the beta band (15-29 Hz) following the retro-cue, they corresponded to VVS representations in a lower frequency range (3-14 Hz) towards the end of the maintenance period. Our findings highlight the distinct coding schemes and representational formats of prioritized content in VVS and PFC.

Reconstructing Spatiotemporal Trajectories of Visual Object Memories in the Human Brain

How the human brain reconstructs, step-by-step, the core elements of past experiences is still unclear. Here, we map the spatiotemporal trajectories along which visual object memories are reconstructed during associative recall. Specifically, we inquire whether retrieval reinstates feature representations in a copy-like but reversed direction with respect to the initial perceptual experience, or alternatively, this reconstruction involves format transformations and regions beyond initial perception. Participants from two cohorts studied new associations between verbs and randomly paired object images, and subsequently recalled the objects when presented with the corresponding verb cue. We first analyze multivariate fMRI patterns to map where in the brain high- and low-level object features can be decoded during perception and retrieval, showing that retrieval is dominated by conceptual features, represented in comparatively late visual and parietal areas. A separately acquired EEG dataset is then used to track the temporal evolution of the reactivated patterns using similarity-based EEG-fMRI fusion. This fusion suggests that memory reconstruction proceeds from anterior frontotemporal to posterior occipital and parietal regions, in line with a conceptual-to-perceptual gradient but only partly following the same trajectories as during perception. Specifically, a linear regression statistically confirms that the sequential activation of ventral visual stream regions is reversed between image perception and retrieval. The fusion analysis also suggests an information relay to frontoparietal areas late during retrieval. Together, the results shed light onto the temporal dynamics of memory recall and the transformations that the information undergoes between the initial experience and its later reconstruction from memory.

Inter-individual variability (but intra-individual stability) of overt versus covert rehearsal strategies in a digital Corsi task

The Corsi (block-tapping) paradigm is a classic and well-established visuospatial working memory task in humans involving internal computations (memorizing of item sequences, organizing and updating the memorandum, and recall processes), as well as both overt and covert shifts of attention to facilitate rehearsal, serving to maintain the Corsi sequences during the retention phase. Here, we introduce a novel digital version of a Corsi task in which i) the difficulty of the memorandum (using sequence lengths ranging from 3 to 8) was controlled, ii) the execution of overt and/or covert attention as well as the visuospatial working memory load during the retention phase was manipulated, and iii) shifts of attention were quantified in all experimental phases. With this, we present behavioral data that demonstrate, characterize, and classify the individual effects of overt and covert strategies used as a means of encoding and rehearsal. In a full within-subject design, we tested 28 participants who had to solve three different Corsi conditions. While in condition A neither of the two strategies were restricted, in condition B the overt and in condition C the overt as well as the covert strategies were suppressed. Analyzing Corsi span, (eye) exploration index, and pupil size (change), data clearly show a continuum between overt and covert strategies over all participants (indicating inter-individual variability). Further, all participants showed stable strategy choice (indicating intra-individual stability), meaning that the preferred strategy was maintained in all three conditions, phases, and sequence lengths of the experiment.

Perceptual encoding benefit of visual memorability on visual memory formation

Human observers often exhibit remarkable consistency in remembering specific visual details, such as certain face images. This phenomenon is commonly attributed to visual memorability, a collection of stimulus attributes that enhance the long-term retention of visual information. However, the exact contributions of visual memorability to visual memory formation remain elusive as these effects could emerge anywhere from early perceptual encoding to post-perceptual memory consolidation processes. To clarify this, we tested three key predictions from the hypothesis that visual memorability facilitates early perceptual encoding that supports the formation of visual short-term memory (VSTM) and the retention of visual long-term memory (VLTM). First, we examined whether memorability benefits in VSTM encoding manifest early, even within the constraints of a brief stimulus presentation (100-200 ms; Experiment 1). We achieved this by manipulating stimulus presentation duration in a VSTM change detection task using face images with high- or low-memorability while ensuring they were equally familiar to the participants. Second, we assessed whether this early memorability benefit increases the likelihood of VSTM retention, even with post-stimulus masking designed to interrupt post-perceptual VSTM consolidation processes (Experiment 2). Last, we investigated the durability of memorability benefits by manipulating memory retention intervals from seconds to 24 h (Experiment 3). Across experiments, our data suggest that visual memorability has an early impact on VSTM formation, persisting across variable retention intervals and predicting subsequent VLTM overnight. Combined, these findings highlight that visual memorability enhances visual memory within 100-200 ms following stimulus onset, resulting in robust memory traces resistant to post-perceptual interruption and long-term forgetting.

Goal-directed attention transforms both working and long-term memory representations in the human parietal cortex

The abundance of distractors in the world poses a major challenge to our brain's limited processing capacity, but little is known about how selective attention modulates stimulus representations in the brain to reduce interference and support durable target memory. Here, we collected functional magnetic resonance imaging (fMRI) data in a selective attention task in which target and distractor pictures of different visual categories were simultaneously presented. Participants were asked to selectively process the target according to the effective cue, either before the encoding period (i.e., perceptual attention) or the maintenance period (i.e., reflective attention). On the next day, participants were asked to perform a memory recognition task in the scanner in which the targets, distractors, and novel items were presented in a pseudorandom order. Behavioral results showed that perceptual attention was better at enhancing target memory and reducing distractor memory than reflective attention, although the overall memory capacity (memory for both target and distractor) was comparable. Using multiple-voxel pattern analysis of the neural data, we found more robust target representation and weaker distractor representation in working memory for perceptual attention than for reflective attention. Interestingly, perceptual attention partially shifted the regions involved in maintaining the target representation from the visual cortex to the parietal cortex. Furthermore, the targets and distractors simultaneously presented in the perceptual attention condition showed reduced pattern similarity in the parietal cortex during retrieval compared to items not presented together. This neural pattern repulsion positively correlated with individuals' recognition of both targets and distractors. These results emphasize the critical role of selective attention in transforming memory representations to reduce interference and improve long-term memory performance.

Contributions of repeated learning to memory in humans: insights from single-neuron recordings in the hippocampus and amygdala

Despite the well-established phenomenon of improved memory performance through repeated learning, studies investigating the associated neural mechanisms have yielded complex and sometimes contradictory findings, and direct evidence from human neuronal recordings has been lacking. This study employs single-neuron recordings with exceptional spatial-temporal resolution, combined with representational similarity analysis, to explore the neural dynamics within the hippocampus and amygdala during repeated learning. Our results demonstrate that in the hippocampus, repetition enhances both representational specificity and fidelity, with these features predicting learning times. Conversely, the amygdala exhibits heightened representational specificity and fidelity during initial learning but does not show improvement with repetition, suggesting functional specialization of the hippocampus and amygdala during different stages of the learning repetition. Specifically, the hippocampus appears to contribute to sustained engagement necessary for benefiting from repeated learning, while the amygdala may play a role in the representation of novel items. These findings contribute to a comprehensive understanding of the intricate interplay between these brain regions in memory processes. Significance statement  For over a century, understanding how repetition contributes to memory enhancement has captivated researchers, yet direct neuronal evidence has been lacking, with a primary focus on the hippocampus and a neglect of the neighboring amygdala. Employing advanced single-neuron recordings and analytical techniques, this study unveils a nuanced functional specialization within the amygdala-hippocampal circuit during various learning repetition. The results highlight the hippocampus's role in sustaining engagement for improved memory with repetition, contrasting with the amygdala's superior ability in representing novel items. This exploration not only deepens our comprehension of memory enhancement intricacies but also sheds light on potential interventions to optimize learning and memory processes.

Long-Term Bridge Training Induces Functional Plasticity Changes in the Brain of Early-Adult Individuals

The aim of this study was to investigate the impact of extended bridge expertise on rapid perceptual processing and brain functional plasticity in early adulthood, utilizing functional magnetic resonance imaging (fMRI). In this investigation, we compared 6 high-level college bridge players with 25 college students lacking bridge experience, assessing their intelligence and working memory. Additionally, we scrutinized behavioral performance and whole-brain activation patterns during an image perceptual judgment task. Findings indicated significant group and interaction effects at the behavioral level. Bridge players exhibited prolonged reaction times and enhanced accuracy on card tasks. At the neural level, the activation level of bridge players in the occipital lobe exceeded that of ordinary college students, with more pronounced group effects in the motor area and inferior parietal lobule during card tasks. This implies that bridge expertise in early adulthood induces functional plasticity changes in regions associated with visual processing and automated mathematical computation.

Divergent roles of early visual cortex and inferior frontal junction in visual working memory

BACKGROUND

Recent studies indicate that both prefrontal and visual regions play critical roles in visual working memory (VWM), with prefrontal regions mainly associated with executive functions, and visual cortices linked to representations of memory contents. VWM involves the selective filtering of irrelevant information, yet the specific contributions of the prefrontal regions and visual cortex in this process remain unclear.

OBJECTIVE

To understand the dynamic causal roles of prefrontal and visual regions in VWM.

METHODS

The differentiation of VWM components was achieved using a computational model that incorporated a swap rate for non-target stimuli. Single-pulse magnetic transcranial stimulation (spTMS) was delivered to the early visual cortex (EVC) and the inferior frontal junction (IFJ) across different phases of an orientation recall task that with or without distractors.

RESULTS

Our results indicate that spTMS over the EVC and IFJ influences VWM particularly when distractors are present. VWM precision can be impacted by spTMS applied to either region during the early retention, while spTMS effect is especially prominent when EVC is stimulated during the late retention phase and when directed at the ipsilateral EVC. Conversely, the probability of accurately recalling the target exhibited comparable patterns when spTMS was administered to either the EVC or IFJ.

CONCLUSIONS

We highlight the "sensory recruitment" of VWM characterized by critical involvement of EVC particularly in the information-filtering process within VWM. The maintenance of memory content representations necessitates ongoing communication between the EVC and IFJ throughout the entirety of the VWM process in a dynamic pattern.

Neural evidence for attentional capture by salient distractors

Salient objects often capture our attention, serving as distractors and hindering our current goals. It remains unclear when and how salient distractors interact with our goals, and our knowledge on the neural mechanisms responsible for attentional capture is limited to a few brain regions recorded from non-human primates. Here we conducted a multivariate analysis on human intracranial signals covering most brain regions and successfully dissociated distractor-specific representations from target-arousal signals in the high-frequency (60-100 Hz) activity. We found that salient distractors were processed rapidly around 220 ms, while target-tuning attention was attenuated simultaneously, supporting initial capture by distractors. Notably, neuronal activity specific to the distractor representation was strongest in the superior and middle temporal gyrus, amygdala and anterior cingulate cortex, while there were smaller contributions from the parietal and frontal cortices. These results provide neural evidence for attentional capture by salient distractors engaging a much larger network than previously appreciated.

Representational formats of human memory traces

Neural representations are internal brain states that constitute the brain's model of the external world or some of its features. In the presence of sensory input, a representation may reflect various properties of this input. When perceptual information is no longer available, the brain can still activate representations of previously experienced episodes due to the formation of memory traces. In this review, we aim at characterizing the nature of neural memory representations and how they can be assessed with cognitive neuroscience methods, mainly focusing on neuroimaging. We discuss how multivariate analysis techniques such as representational similarity analysis (RSA) and deep neural networks (DNNs) can be leveraged to gain insights into the structure of neural representations and their different representational formats. We provide several examples of recent studies which demonstrate that we are able to not only measure memory representations using RSA but are also able to investigate their multiple formats using DNNs. We demonstrate that in addition to slow generalization during consolidation, memory representations are subject to semantization already during short-term memory, by revealing a shift from visual to semantic format. In addition to perceptual and conceptual formats, we describe the impact of affective evaluations as an additional dimension of episodic memories. Overall, these studies illustrate how the analysis of neural representations may help us gain a deeper understanding of the nature of human memory.

Toward viewing behavior for aerial scene categorization

Previous work has demonstrated similarities and differences between aerial and terrestrial image viewing. Aerial scene categorization, a pivotal visual processing task for gathering geoinformation, heavily depends on rotation-invariant information. Aerial image-centered research has revealed effects of low-level features on performance of various aerial image interpretation tasks. However, there are fewer studies of viewing behavior for aerial scene categorization and of higher-level factors that might influence that categorization. In this paper, experienced subjects' eye movements were recorded while they were asked to categorize aerial scenes. A typical viewing center bias was observed. Eye movement patterns varied among categories. We explored the relationship of nine image statistics to observers' eye movements. Results showed that if the images were less homogeneous, and/or if they contained fewer or no salient diagnostic objects, viewing behavior became more exploratory. Higher- and object-level image statistics were predictive at both the image and scene category levels. Scanpaths were generally organized and small differences in scanpath randomness could be roughly captured by critical object saliency. Participants tended to fixate on critical objects. Image statistics included in this study showed rotational invariance. The results supported our hypothesis that the availability of diagnostic objects strongly influences eye movements in this task. In addition, this study provides supporting evidence for Loschky et al.'s (Journal of Vision, 15(6), 11, 2015) speculation that aerial scenes are categorized on the basis of image parts and individual objects. The findings were discussed in relation to theories of scene perception and their implications for automation development.

Harmonic memory signals in the human cerebral cortex induced by semantic relatedness of words

When we memorize multiple words simultaneously, semantic relatedness among those words assists memory. For example, the information about "apple", "banana," and "orange" will be connected via a common concept of "fruits" and become easy to retain and recall. Neural mechanisms underlying this semantic integration in verbal working memory remain unclear. Here I used electroencephalography (EEG) and investigated neural signals when healthy human participants memorized five nouns semantically related (Sem trial) or not (NonSem trial). The regularity of oscillatory signals (8-30 Hz) during the retention period was found to be lower in NonSem than Sem trials, indicating that memorizing words unrelated to each other induced a non-harmonic (irregular) waveform in the temporal cortex. These results suggest that (i) semantic features of a word are retained as a set of neural oscillations at specific frequencies and (ii) memorizing words sharing a common semantic feature produces harmonic brain responses through a resonance or integration (sharing) of the oscillatory signals.

What is it like to do a visuo-spatial working memory task: A qualitative phenomenological study of the visual span task

Working memory is typically measured with specifically designed psychological tasks. When evaluating the validity of working memory tasks, we commonly focus on the reliability of the outcome measurements. Only rarely do we focus on how participants experience these tasks. Accounting for lived experience of working memory task may help us better understand variability in working memory performance and conscious experience in general. We replicated recently established protocols for the phenomenological investigation of working memory using the visual span task. We collected subjective reports from eighteen healthy participants (10 women) aged 21 to 35 years. We observed that working memory can be phenomenologically characterized at three different time scales: background feelings, strategies, and tactics. On the level of tactics, we identified transmodality (i.e., how one modality of lived experience can be transformed into another one) as the central phenomenological dynamic at play during working memory task performance.

Functional specialization and interaction in the amygdala-hippocampus circuit during working memory processing

Both the hippocampus and amygdala are involved in working memory (WM) processing. However, their specific role in WM is still an open question. Here, we simultaneously recorded intracranial EEG from the amygdala and hippocampus of epilepsy patients while performing a WM task, and compared their representation patterns during the encoding and maintenance periods. By combining multivariate representational analysis and connectivity analyses with machine learning methods, our results revealed a functional specialization of the amygdala-hippocampal circuit: The mnemonic representations in the amygdala were highly distinct and decreased from encoding to maintenance. The hippocampal representations, however, were more similar across different items but remained stable in the absence of the stimulus. WM encoding and maintenance were associated with bidirectional information flow between the amygdala and the hippocampus in low-frequency bands (1-40 Hz). Furthermore, the decoding accuracy on WM load was higher by using representational features in the amygdala during encoding and in the hippocampus during maintenance, and by using information flow from the amygdala during encoding and that from the hippocampus during maintenance, respectively. Taken together, our study reveals that WM processing is associated with functional specialization and interaction within the amygdala-hippocampus circuit.

Distinct encoding and post-encoding representational formats contribute to episodic sequence memory formation

In episodic encoding, an unfolding experience is rapidly transformed into a memory representation that binds separate episodic elements into a memory form to be later recollected. However, it is unclear how brain activity changes over time to accommodate the encoding of incoming information. This study aimed to investigate the dynamics of the representational format that contributed to memory formation of sequential episodes. We combined representational similarity analysis and multivariate decoding approaches on EEG data to compare whether "category-level" or "item-level" representations supported memory formation during the online encoding of a picture triplet sequence and offline, in the period that immediately followed encoding. The findings revealed a gradual integration of category-level representation during the online encoding of the picture sequence and a rapid item-based neural reactivation of the encoded sequence at the episodic offset. However, we found that only memory reinstatement at episodic offset was associated with successful memory retrieval from long-term memory. These results suggest that post-encoding memory reinstatement is crucial for the rapid formation of unique memory for episodes that unfold over time. Overall, the study sheds light on the dynamics of representational format changes that take place during the formation of episodic memories.

Trigeminal stimulation is required for neural representations of bimodal odor localization: A time-resolved multivariate EEG and fNIRS study

Whereas neural representations of spatial information are commonly studied in vision, olfactory stimuli might also be able to create such representations via the trigeminal system. We explored in two independent multi-method electroencephalography-functional near-infrared spectroscopy (EEG+fNIRS) experiments (n1=18, n2=14) if monorhinal odor stimuli can evoke spatial representations in the brain. We tested whether this representation depends on trigeminal properties of the stimulus, and if the retention in short-term memory follows the "sensorimotor recruitment theory", using multivariate representational similarity analysis (RSA). We demonstrate that the delta frequency band up to 5 Hz across the scull entail spatial information of which nostril has been stimulated. Delta frequencies were localized in a network involving primary and secondary olfactory, motor-sensory and occipital regions. RSA on fNIRS data showed that monorhinal stimulations evoke neuronal representations in motor-sensory regions and that this representation is kept stable beyond the time of perception. These effects were no longer valid when the odor stimulus did not sufficiently stimulate the trigeminal nerve as well. Our results are first evidence that the trigeminal system can create spatial representations of bimodal odors in the brain and that these representations follow similar principles as the other sensory systems.

The medial temporal lobe supports the quality of visual short-term memory representation

The quality of short-term memory (STM) underlies our ability to recall the exact details of a recent event, yet how the human brain enables this core cognitive function remains poorly understood. Here we use multiple experimental approaches to test the hypothesis that the quality of STM, such as its precision or fidelity, relies on the medial temporal lobe (MTL), a region commonly associated with the ability to distinguish similar information remembered in long-term memory. First, with intracranial recordings, we find that delay-period MTL activity retains item-specific STM content that is predictive of subsequent recall precision. Second, STM recall precision is associated with an increase in the strength of intrinsic MTL-to-neocortical functional connections during a brief retention interval. Finally, perturbing the MTL through electrical stimulation or surgical removal can selectively reduce STM precision. Collectively, these findings provide converging evidence that the MTL is critically involved in the quality of STM representation.

The entorhinal-DG/CA3 pathway in the medial temporal lobe retains visual working memory of a simple surface feature

Classic models consider working memory (WM) and long-term memory as distinct mental faculties that are supported by different neural mechanisms. Yet, there are significant parallels in the computation that both types of memory require. For instance, the representation of precise item-specific memory requires the separation of overlapping neural representations of similar information. This computation has been referred to as pattern separation, which can be mediated by the entorhinal-DG/CA3 pathway of the medial temporal lobe (MTL) in service of long-term episodic memory. However, although recent evidence has suggested that the MTL is involved in WM, the extent to which the entorhinal-DG/CA3 pathway supports precise item-specific WM has remained elusive. Here, we combine an established orientation WM task with high-resolution fMRI to test the hypothesis that the entorhinal-DG/CA3 pathway retains visual WM of a simple surface feature. Participants were retrospectively cued to retain one of the two studied orientation gratings during a brief delay period and then tried to reproduce the cued orientation as precisely as possible. By modeling the delay-period activity to reconstruct the retained WM content, we found that the anterior-lateral entorhinal cortex (aLEC) and the hippocampal DG/CA3 subfield both contain item-specific WM information that is associated with subsequent recall fidelity. Together, these results highlight the contribution of MTL circuitry to item-specific WM representation.

Post-encoding Reactivation Is Related to Learning of Episodes in Humans

Prior animal and human studies have shown that post-encoding reinstatement plays an important role in organizing the temporal sequence of unfolding episodes in memory. Here, we investigated whether post-encoding reinstatement serves to promote the encoding of "one-shot" episodic learning beyond the temporal structure in humans. In Experiment 1, participants encoded sequences of pictures depicting unique and meaningful episodic-like events. We used representational similarity analysis on scalp EEG recordings during encoding and found evidence of rapid picture-elicited EEG pattern reinstatement at episodic offset (around 500 msec post-episode). Memory reinstatement was not observed between successive elements within an episode, and the degree of memory reinstatement at episodic offset predicted later recall for that episode. In Experiment 2, participants encoded a shuffled version of the picture sequences from Experiment 1, rendering each episode meaningless to the participant but temporally structured as in Experiment 1, and we found no evidence of memory reinstatement at episodic offset. These results suggest that post-encoding memory reinstatement is akin to the rapid formation of unique and meaningful episodes that unfold over time.

From remembering to reconstruction: The transformative neural representation of episodic memory

Although memory has long been recognized as a generative process, neural research of memory in recent decades has been predominantly influenced by Tulving's "mental time traveling" perspective and focused on the reactivation and consolidation of encoded memory representations. With the development of multiple powerful analytical approaches to characterize the contents and formats of neural representations, recent studies are able to provide detailed examinations of the representations at various processing stages and have provided exciting new insights into the transformative nature of episodic memory. These studies have revealed the rapid, substantial, and continuous transformation of memory representation during the encoding, maintenance, consolidation, and retrieval of both single and multiple events, as well as event sequences. These transformations are characterized by the abstraction, integration, differentiation, and reorganization of memory representations, enabling the long-term retention and generalization of memory. These studies mark a significant shift in perspective from remembering to reconstruction, which might better reveal the nature of memory and its roles in supporting more effective learning, adaptive decision-making, and creative problem solving.

Conceptual knowledge shapes visual working memory for complex visual information

Human visual working memory (VWM) is a memory store people use to maintain the visual features of objects and scenes. Although it is obvious that bottom-up information influences VWM, the extent to which top-down conceptual information influences VWM is largely unknown. We report an experiment in which groups of participants were trained in one of two different categories of geologic faults (left/right lateral, or normal/reverse faults), or received no category training. Following training, participants performed a visual change detection task in which category knowledge was irrelevant to the task. Participants were more likely to detect a change in geologic scenes when the changes crossed a trained categorical distinction (e.g., the left/right lateral fault boundary), compared to within-category changes. In addition, participants trained to distinguish left/right lateral faults were more likely to detect changes when the scenes were mirror images along the left/right dimension. Similarly, participants trained to distinguish normal/reverse faults were more likely to detect changes when scenes were mirror images along the normal/reverse dimension. Our results provide direct empirical evidence that conceptual knowledge influences VWM performance for complex visual information. An implication of our results is that cognitive scientists may need to reconceptualize VWM so that it is closer to "conceptual short-term memory".

Effects of item distinctiveness on the retrieval of objects and object-location bindings from visual working memory

Visual working memory (VWM) is prone to interference from stored items competing for its limited capacity. Distinctiveness or similarity of the items is acknowledged to affect this competition, such that poor item distinctiveness causes a failure to discriminate between items sharing common features. In three experiments, we studied how the distinctiveness of studied real-world objects (i.e., whether the objects belong to the same or different basic categories) affects the retrieval of objects themselves (simple recognition) and object-location conjunctions (information about which object was where in a display, cued recall). In Experiments 1 and 2, we found that distinctiveness did not affect memories for objects or for locations, but low-distinctive objects were more frequently reported at "swapped" locations that originally contained other objects, showing object-location memory swaps. In Experiments 3 we found that observers swapped the location of a tested object with another object from the same category more frequently than with any of the objects from another category. This suggests that more similar studied objects cause more retrieval competition in object-location judgments than in simple recognition. Additionally, we discuss a possible role of categorical labeling of locations that can support object-location retrieval when the studied objects are highly distinct.

Transformative neural representations support long-term episodic memory

Memory is often conceived as a dynamic process that involves substantial transformations of mental representations. However, the neural mechanisms underlying these transformations and their role in memory formation and retrieval have only started to be elucidated. Combining intracranial EEG recordings with deep neural network models, we provide a detailed picture of the representational transformations from encoding to short-term memory maintenance and long-term memory retrieval that underlie successful episodic memory. We observed substantial representational transformations during encoding. Critically, more pronounced semantic representational formats predicted better subsequent long-term memory, and this effect was mediated by more consistent item-specific representations across encoding events. The representations were further transformed right after stimulus offset, and the representations during long-term memory retrieval were more similar to those during short-term maintenance than during encoding. Our results suggest that memory representations pass through multiple stages of transformations to achieve successful long-term memory formation and recall.

Semantic influence on visual working memory of object identity and location

Does semantic information-in particular, regularities in category membership across objects-influence visual working memory (VWM) processing? We predict that the answer is "yes". Four experiments evaluating this prediction are reported. Experimental stimuli were images of real-world objects arranged in either one or two spatial clusters. On coherent trials, all objects belonging to a cluster also belonged to the same category. On incoherent trials, at least one cluster contained objects from different categories. Experiments using a change-detection paradigm (Experiments 1-3) and an experiment in which participants recalled the locations of objects in a scene (Experiment 4) yielded the same result: participants showed better memory performance on coherent trials than on incoherent trials. Taken as a whole, these experiments provide the best (perhaps only) data to date demonstrating that statistical regularities in semantic category membership improve VWM performance. Because a conventional perspective in cognitive science regards VWM as being sensitive solely to bottom-up visual properties of objects (e.g., shape, color, orientation), our results indicate that cognitive science may need to modify its conceptualization of VWM so that it is closer to "conceptual short-term memory", a short-term memory store representing current stimuli and their associated concepts (Potter, 1993, 2012).

Neural Coding of Cognitive Control: The Representational Similarity Analysis Approach

Cognitive control relies on distributed and potentially high-dimensional frontoparietal task representations. Yet, the classical cognitive neuroscience approach in this domain has focused on aggregating and contrasting neural measures - either via univariate or multivariate methods - along highly abstracted, 1D factors (e.g., Stroop congruency). Here, we present representational similarity analysis (RSA) as a complementary approach that can powerfully inform representational components of cognitive control theories. We review several exemplary uses of RSA in this regard. We further show that most classical paradigms, given their factorial structure, can be optimized for RSA with minimal modification. Our aim is to illustrate how RSA can be incorporated into cognitive control investigations to shed new light on old questions.

Visual and Semantic Contributions to Visual Short-Term Memory

Liu et al. recently demonstrated novel neural evidence for visual and semantic contributions to the encoding and maintenance of object information in a delayed match-to-sample task. Their data highlight the close interaction between sensory experience and prior semantic knowledge in human visual short-term memory for naturalistic stimuli.