Prediction of subsequent recognition performance using brain activity in the medial temporal lobe

Hippocampal integration and separation processes with different temporal and spatial dynamics during learning for associative memory

The hippocampus is known to be critically involved in associative memory formation. However, the role of the hippocampus during the learning of associative memory is still controversial; while the hippocampus is considered to play a critical role in the integration of related stimuli, numerous studies also suggest a role of the hippocampus in the separation of different memory traces for rapid learning. Here, we employed an associative learning paradigm consisting of repeated learning cycles. By tracking the changes in the hippocampal representations of associated stimuli on a cycle-by-cycle basis as learning progressed, we show that both integration and separation processes occur in the hippocampus with different temporal dynamics. We found that the degree of shared representations for associated stimuli decreased significantly during the early phase of learning, whereas it increased during the later phase of learning. Remarkably, these dynamic temporal changes were observed only for stimulus pairs remembered 1 day or 4 weeks after learning, but not for forgotten pairs. Further, the integration process during learning was prominent in the anterior hippocampus, while the separation process was obvious in the posterior hippocampus. These results demonstrate temporally and spatially dynamic hippocampal processing during learning that can lead to the maintenance of associative memory.

Decoding declarative memory process for predicting memory retrieval based on source localization

Many studies have focused on understanding memory processes due to their importance in daily life. Differences in timing and power spectra of brain signals during encoding task have been linked to later remembered items and were recently used to predict memory retrieval performance. However, accuracies remain low when using non-invasive methods for acquiring brain signals, mainly due to the low spatial resolution. This study investigates the prediction of successful retrieval using estimated source activity corresponding either to cortical or subcortical structures through source localization. Electroencephalogram (EEG) signals were recorded while participants performed a declarative memory task. Frequency-time analysis was performed using signals from encoding and retrieval tasks to confirm the importance of neural oscillations and their relationship with later remembered and forgotten items. Significant differences in the power spectra between later remembered and forgotten items were found before and during the presentation of the stimulus in the encoding task. Source activity estimation revealed differences in the beta band power over the medial parietal and medial prefrontal areas prior to the presentation of the stimulus, and over the cuneus and lingual areas during the presentation of the stimulus. Additionally, there were significant differences during the stimuli presentation during the retrieval task. Prediction of later remembered items was performed using surface potentials and estimated source activity. The results showed that source localization increases classification performance compared to the one using surface potentials. These findings support the importance of incorporating spatial features of neural activity to improve the prediction of memory retrieval.

Predicting memory from study-related brain activity

For both basic and applied reasons, an important goal is to identify brain activity present while people study materials that enable us to predict whether they will remember those materials. We show that this is possible with the conventional event-related potential “subsequent-memory-effect” signals as well as with machine learning classifiers, but only to a small degree. This is in line with behavioral research, which supports many determinants of memory apart from the cognitive processes during study.

EEG-Based Prediction of Successful Memory Formation During Vocabulary Learning

Previous Electroencephalography (EEG) and neuroimaging studies have found differences between brain signals for subsequently remembered and forgotten items during learning of items - it has even been shown that single trial prediction of memorization success is possible with a few target items. There has been little attempt, however, in validating the findings in an application-oriented context involving longer test spans with realistic learning materials encompassing more items. Hence, the present study investigates subsequent memory prediction within the application context of foreign-vocabulary learning. We employed an off-line, EEG-based paradigm in which Korean participants without prior German language experience learned 900 German words in paired-associate form. Our results using convolutional neural networks optimized for EEG-signal analysis show that above-chance classification is possible in this context allowing us to predict during learning which of the words would be successfully remembered later.

Differentiation of Human Medial Prefrontal Cortex Activity Underlies Long-Term Resistance to Forgetting in Memory

It is well known that distributing study events over time leads to better memory over long time scales, compared with massing study events together. One explanation for such long-term resistance to forgetting is that distributed study leads to neural differentiation in memory, which supports retrieval of past experiences by disambiguating highly similar memory representations. Neuroanatomical models of episodic memory retrieval propose that the hippocampus and medial prefrontal cortex (MPFC) work together to enable retrieval of behaviorally appropriate memories. However, it is not known how representations in these regions jointly support resistance to forgetting long after initial learning. Using fMRI, we measured differentiation in retrieved memory representations following an extended delay in male and female human participants. After 1 week, word-object associations were better remembered if studied across 2 d (overnight), allowing associations to be learned in distinct temporal contexts, compared with learning within a single day (same day). MPFC retrieval patterns showed differentiation for overnight relative to same day memories, whereas hippocampal patterns reflected associative retrieval success. Overnight memory differentiation in MPFC was higher for associative than item memories and higher than differentiation assessed over a brain-wide set of retrieval-active voxels. The memory-related difference in MPFC pattern differentiation correlated with memory success for overnight learning and with hippocampal-MPFC functional connectivity. These results show that learning information across days leads to differentiated MPFC memory representations, reducing forgetting after 1 week, and suggest this arises from persistent interactions between MPFC and hippocampus.SIGNIFICANCE STATEMENT Neural activity in both the hippocampus and medial prefrontal cortex (MPFC) has been linked to memory-related representations, but prior work has not examined how these representations support episodic memory retrieval over extended time scales that are characteristic of everyday retrieval. We show that differentiation in MPFC activity 1 week after encoding is higher for retrieved information learned across 2 d compared with within a single day. In hippocampus, differentiation was greater for detailed memory retrieval but was not influenced by whether information had been learned over 1 or 2 d. Differentiation in MPFC predicted behavioral robustness to forgetting and was correlated with hippocampal-MPFC connectivity. The results suggest that context-based differentiation supports robust long-term memory via persistent MPFC-hippocampal interactions.

Oxytocin receptor gene variations predict neural and behavioral response to oxytocin in autism

Oxytocin appears beneficial for autism spectrum disorder (ASD), and more than 20 single-nucleotide polymorphisms (SNPs) in oxytocin receptor (OXTR) are relevant to ASD. However, neither biological functions of OXTR SNPs in ASD nor critical OXTR SNPs that determine oxytocin’s effects on ASD remains known. Here, using a machine-learning algorithm that was designed to evaluate collective effects of multiple SNPs and automatically identify most informative SNPs, we examined relationships between 27 representative OXTR SNPs and six types of behavioral/neural response to oxytocin in ASD individuals. The oxytocin effects were extracted from our previous placebo-controlled within-participant clinical trial administering single-dose intranasal oxytocin to 38 high-functioning adult Japanese ASD males. Consequently, we identified six different SNP sets that could accurately predict the six different oxytocin efficacies, and confirmed the robustness of these SNP selections against variations of the datasets and analysis parameters. Moreover, major alleles of several prominent OXTR SNPs—including rs53576 and rs2254298—were found to have dissociable effects on the oxytocin efficacies. These findings suggest biological functions of the OXTR SNP variants on autistic oxytocin responses, and implied that clinical oxytocin efficacy may be genetically predicted before its actual administration, which would contribute to establishment of future precision medicines for ASD.

Response to 'Hippocampus as a wormhole: gateway to consciousness'

Reply to: Behrendt R-P. Hippocampus as a wormhole: gateway to consciousness. WIREs Cogn Sci 2017, e1446. doi: 10.1002/wcs.1446.

Remembered or Forgotten?-An EEG-Based Computational Prediction Approach

Prediction of memory performance (remembered or forgotten) has various potential applications not only for knowledge learning but also for disease diagnosis. Recently, subsequent memory effects (SMEs)—the statistical differences in electroencephalography (EEG) signals before or during learning between subsequently remembered and forgotten events—have been found. This finding indicates that EEG signals convey the information relevant to memory performance. In this paper, based on SMEs we propose a computational approach to predict memory performance of an event from EEG signals. We devise a convolutional neural network for EEG, called ConvEEGNN, to predict subsequently remembered and forgotten events from EEG recorded during memory process. With the ConvEEGNN, prediction of memory performance can be achieved by integrating two main stages: feature extraction and classification. To verify the proposed approach, we employ an auditory memory task to collect EEG signals from scalp electrodes. For ConvEEGNN, the average prediction accuracy was 72.07% by using EEG data from pre-stimulus and during-stimulus periods, outperforming other approaches. It was observed that signals from pre-stimulus period and those from during-stimulus period had comparable contributions to memory performance. Furthermore, the connection weights of ConvEEGNN network can reveal prominent channels, which are consistent with the distribution of SME studied previously.

Attention promotes episodic encoding by stabilizing hippocampal representations

Attention influences what is later remembered, but little is known about how this occurs in the brain. We hypothesized that behavioral goals modulate the attentional state of the hippocampus to prioritize goal-relevant aspects of experience for encoding. Participants viewed rooms with paintings, attending to room layouts or painting styles on different trials during high-resolution functional MRI. We identified template activity patterns in each hippocampal subfield that corresponded to the attentional state induced by each task. Participants then incidentally encoded new rooms with art while attending to the layout or painting style, and memory was subsequently tested. We found that when task-relevant information was better remembered, the hippocampus was more likely to have been in the correct attentional state during encoding. This effect was specific to the hippocampus, and not found in medial temporal lobe cortex, category-selective areas of the visual system, or elsewhere in the brain. These findings provide mechanistic insight into how attention transforms percepts into memories.

Basal ganglia and cortical networks for sequential ordering and rhythm of complex movements

Voluntary actions require the concurrent engagement and coordinated control of complex temporal (e.g., rhythm) and ordinal motor processes. Using high-resolution functional magnetic resonance imaging (fMRI) and multi-voxel pattern analysis (MVPA), we sought to determine the degree to which these complex motor processes are dissociable in basal ganglia and cortical networks. We employed three different finger-tapping tasks that differed in the demand on the sequential temporal rhythm or sequential ordering of submovements. Our results demonstrate that sequential rhythm and sequential order tasks were partially dissociable based on activation differences. The sequential rhythm task activated a widespread network centered around the supplementary motor area (SMA) and basal-ganglia regions including the dorsomedial putamen and caudate nucleus, while the sequential order task preferentially activated a fronto-parietal network. There was also extensive overlap between sequential rhythm and sequential order tasks, with both tasks commonly activating bilateral premotor, supplementary motor, and superior/inferior parietal cortical regions, as well as regions of the caudate/putamen of the basal ganglia and the ventro-lateral thalamus. Importantly, within the cortical regions that were active for both complex movements, MVPA could accurately classify different patterns of activation for the sequential rhythm and sequential order tasks. In the basal ganglia, however, overlapping activation for the sequential rhythm and sequential order tasks, which was found in classic motor circuits of the putamen and ventro-lateral thalamus, could not be accurately differentiated by MVPA. Overall, our results highlight the convergent architecture of the motor system, where complex motor information that is spatially distributed in the cortex converges into a more compact representation in the basal ganglia.

Decoding the Formation of New Semantics: MVPA Investigation of Rapid Neocortical Plasticity during Associative Encoding through Fast Mapping

Neocortical structures typically only support slow acquisition of declarative memory; however, learning through fast mapping may facilitate rapid learning-induced cortical plasticity and hippocampal-independent integration of novel associations into existing semantic networks. During fast mapping the meaning of new words and concepts is inferred, and durable novel associations are incidentally formed, a process thought to support early childhood's exuberant learning. The anterior temporal lobe, a cortical semantic memory hub, may critically support such learning. We investigated encoding of semantic associations through fast mapping using fMRI and multivoxel pattern analysis. Subsequent memory performance following fast mapping was more efficiently predicted using anterior temporal lobe than hippocampal voxels, while standard explicit encoding was best predicted by hippocampal activity. Searchlight algorithms revealed additional activity patterns that predicted successful fast mapping semantic learning located in lateral occipitotemporal and parietotemporal neocortex and ventrolateral prefrontal cortex. By contrast, successful explicit encoding could be classified by activity in medial and dorsolateral prefrontal and parahippocampal cortices. We propose that fast mapping promotes incidental rapid integration of new associations into existing neocortical semantic networks by activating related, nonoverlapping conceptual knowledge. In healthy adults, this is better captured by unique anterior and lateral temporal lobe activity patterns, while hippocampal involvement is less predictive of this kind of learning.

Mean signal and response time influences on multivoxel signals of contextual retrieval in the medial temporal lobe

INTRODUCTION

The medial temporal lobe supports integrating the "what," "where," and "when" of an experience into a unified memory. However, it remains unclear how representations of these contextual features are neurally encoded and distributed across medial temporal lobe subregions.

METHODS

This study conducted functional magnetic resonance imaging of the medial temporal lobe, while participants retrieved pair, spatial, and temporal source memories. Multivoxel classifiers were trained to distinguish between retrieval conditions before and after correction for mean signal and response times, to more thoroughly characterize the multivoxel signal associated with memory context.

RESULTS

Activity in perirhinal and parahippocampal cortex dissociated between memory for associated items and memory for their spatiotemporal context, and hippocampal activity was linked to memory for spatial context. However, perirhinal and hippocampal classifiers were, respectively, driven by effects of mean signal amplitude and task difficulty, whereas the parahippocampal classifier survived correction for these effects.

CONCLUSION

These findings demonstrate dissociable coding mechanisms for episodic memory context across the medial temporal lobe, and further highlight a critical distinction between multivoxel representations driven by spatially distributed activity patterns, and those driven by the regional signal.

Network structure underlying resolution of conflicting non-verbal and verbal social information

Social judgments often require resolution of incongruity in communication contents. Although previous studies revealed that such conflict resolution recruits brain regions including the medial prefrontal cortex (mPFC) and posterior inferior frontal gyrus (pIFG), functional relationships and networks among these regions remain unclear. In this functional magnetic resonance imaging study, we investigated the functional dissociation and networks by measuring human brain activity during resolving incongruity between verbal and non-verbal emotional contents. First, we found that the conflict resolutions biased by the non-verbal contents activated the posterior dorsal mPFC (post-dmPFC), bilateral anterior insula (AI) and right dorsal pIFG, whereas the resolutions biased by the verbal contents activated the bilateral ventral pIFG. In contrast, the anterior dmPFC (ant-dmPFC), bilateral superior temporal sulcus and fusiform gyrus were commonly involved in both of the resolutions. Second, we found that the post-dmPFC and right ventral pIFG were hub regions in networks underlying the non-verbal- and verbal-content-biased resolutions, respectively. Finally, we revealed that these resolution-type-specific networks were bridged by the ant-dmPFC, which was recruited for the conflict resolutions earlier than the two hub regions. These findings suggest that, in social conflict resolutions, the ant-dmPFC selectively recruits one of the resolution-type-specific networks through its interaction with resolution-type-specific hub regions.

Dissociable memory traces within the macaque medial temporal lobe predict subsequent recognition performance

Functional magnetic resonance imaging (fMRI) studies have revealed that activity in the medial temporal lobe (MTL) predicts subsequent memory performance in humans. Because of limited knowledge on cytoarchitecture and axonal projections of the human MTL, precise localization and characterization of the areas that can predict subsequent memory performance are benefited by the use of nonhuman primates in which integrated approach of the MRI- and cytoarchiture-based boundary delineation is available. However, neural correlates of this subsequent memory effect have not yet been identified in monkeys. Here, we used fMRI to examine activity in the MTL during memory encoding of events that monkeys later remembered or forgot. Application of both multivoxel pattern analysis and conventional univariate analysis to high-resolution fMRI data allowed us to identify memory traces within the caudal entorhinal cortex (cERC) and perirhinal cortex (PRC), as well as within the hippocampus proper. Furthermore, activity in the cERC and the hippocampus, which are directly connected, was responsible for encoding the initial items of sequentially presented pictures, which may reflect recollection-like recognition, whereas activity in the PRC was not. These results suggest that two qualitatively distinct encoding processes work in the monkey MTL and that recollection-based memory is formed by the interplay of the hippocampus with the cERC, a focal cortical area anatomically closer to the hippocampus and hierarchically higher than previously believed. These findings will advance the understanding of common memory system between humans and monkeys and accelerate fine electrophysiological characterization of these dissociable memory traces in the monkey MTL.

Using single-trial EEG to predict and analyze subsequent memory

We show that it is possible to successfully predict subsequent memory performance based on single-trial EEG activity before and during item presentation in the study phase. Two-class classification was conducted to predict subsequently remembered vs. forgotten trials based on subjects' responses in the recognition phase. The overall accuracy across 18 subjects was 59.6% by combining pre- and during-stimulus information. The single-trial classification analysis provides a dimensionality reduction method to project the high-dimensional EEG data onto a discriminative space. These projections revealed novel findings in the pre- and during-stimulus periods related to levels of encoding. It was observed that the pre-stimulus information (specifically oscillatory activity between 25 and 35Hz) -300 to 0ms before stimulus presentation and during-stimulus alpha (7-12Hz) information between 1000 and 1400ms after stimulus onset distinguished between recollection and familiarity while the during-stimulus alpha information and temporal information between 400 and 800ms after stimulus onset mapped these two states to similar values.

Cortical reinstatement mediates the relationship between content-specific encoding activity and subsequent recollection decisions

Episodic recollection entails the conscious remembrance of event details associated with previously encountered stimuli. Recollection depends on both the establishment of cortical representations of event features during stimulus encoding and the cortical reinstatement of these representations at retrieval. Here, we used multivoxel pattern analyses of functional magnetic resonance imaging data to examine how cortical and hippocampal activity at encoding and retrieval drive recollective memory decisions. During encoding, words were associated with face or scene source contexts. At retrieval, subjects were cued to recollect the source associate of each presented word. Neurally derived estimates of encoding strength and pattern reinstatement in occipitotemporal cortex were computed for each encoding and retrieval trial, respectively. Analyses demonstrated that (1) cortical encoding strength predicted subsequent memory accuracy and reaction time, (2) encoding strength predicted encoding-phase hippocampal activity, and (3) encoding strength and retrieval-phase hippocampal activity predicted the magnitude of cortical reinstatement. Path analyses further indicated that cortical reinstatement partially mediated both the effect of cortical encoding strength and the effect of retrieval-phase hippocampal activity on subsequent source memory performance. Taken together, these results indicate that memory-guided decisions are driven in part by a pathway leading from hippocampally linked cortical encoding of event attributes to hippocampally linked cortical reinstatement at retrieval.

Disentangling spatial perception and spatial memory in the hippocampus: a univariate and multivariate pattern analysis fMRI study

Although the role of the hippocampus in spatial cognition is well accepted, it is unclear whether its involvement is restricted to the mnemonic domain or also extends to perception. We used fMRI to scan neurologically healthy participants during a scene oddity judgment task that placed no explicit demand on long-term memory. Crucially, a surprise recognition test was administered after scanning so that each trial could be categorized not only according to oddity accuracy but also according to subsequent memory. Univariate analyses showed significant hippocampal activity in association with correct oddity judgment, whereas greater parahippocampal place area (PPA) activity was observed during incorrect oddity trials, both irrespective of subsequent recognition performance. Consistent with this, multivariate pattern analyses revealed that a linear support vector machine was able to distinguish correct from incorrect oddity trials on the basis of activity in voxels within the hippocampus or PPA. Although no significant regions of activity were identified by univariate analyses in association with memory performance, a classifier was able to predict subsequent memory using voxels in either the hippocampus or PPA. Our findings are consistent with the idea that the hippocampus is important for processes beyond long-term declarative memory and that this structure may also play a role in complex spatial perception.

What makes a pattern? Matching decoding methods to data in multivariate pattern analysis

Research in neuroscience faces the challenge of integrating information across different spatial scales of brain function. A promising technique for harnessing information at a range of spatial scales is multivariate pattern analysis (MVPA) of functional magnetic resonance imaging (fMRI) data. While the prevalence of MVPA has increased dramatically in recent years, its typical implementations for classification of mental states utilize only a subset of the information encoded in local fMRI signals. We review published studies employing multivariate pattern classification since the technique’s introduction, which reveal an extensive focus on the improved detection power that linear classifiers provide over traditional analysis techniques. We demonstrate using simulations and a searchlight approach, however, that non-linear classifiers are capable of extracting distinct information about interactions within a local region. We conclude that for spatially localized analyses, such as searchlight and region of interest, multiple classification approaches should be compared in order to match fMRI analyses to the properties of local circuits.

Functional dissociation between anterior and posterior temporal cortical regions during retrieval of remote memory

Retrieval of remote memory is considered to differentially involve the anterior and posterior temporal neocortices. Previous neuropsychological studies suggest that the different posterior temporal cortical regions are involved in the retrieval of remote memory of different categories of stimuli, whereas the anterior region is involved more generally in remote memory retrieval. In the present study, using functional magnetic resonance imaging of human brains, we tested this dissociation by examining the more precise characteristics of the anterior and posterior temporal cortical regions. Two categories of stimuli, faces and scenes, were used for paired stimuli to be retrieved, and the brain activity during retrieval of paired stimuli that were learned immediately before the scanning was compared with that during retrieval of paired stimuli that were learned ∼8 weeks earlier. We found that the different posterior temporal cortical regions were activated during retrieval of different categories of remote memory in a category-specific manner, whereas the anterior temporal cortical region was activated during retrieval of remote memory in a category-general manner. Furthermore, by applying a multivariate pattern analysis to psychophysiological interactions during retrieval of remote memory relative to recent memory, we revealed the significant interaction from the category-specific posterior temporal cortical regions to the category-general anterior temporal region. These results suggest that the posterior temporal cortical regions are involved in representation and retrieval of category-specific remote memory, whereas the anterior cortical temporal region is involved in category-general retrieval process of remote memory.

Analyses of regional-average activation and multivoxel pattern information tell complementary stories

Multivariate pattern analysis (MVPA) has recently received increasing attention in functional neuroimaging due to its ability to decode mental states from fMRI signals. However, questions remain regarding both the empirical and conceptual relationships between results from MVPA and standard univariate analyses. In the current study, whole-brain univariate and searchlight MVPAs of parametric manipulations of monetary gain and loss in a decision making task (Tom et al., 2007) were compared to identify the differences in the results across these methods and the implications for understanding the underlying mental processes. The MVPA and univariate results did identify some overlapping regions in whole brain analyses. However, an analysis of consistency revealed that in many regions the effect size estimates obtained from MVPA and univariate analysis were uncorrelated. Moreover, comparison of sensitivity showed a general trend towards greater sensitivity to task manipulations by MVPA compared to univariate analysis. These results demonstrate that MVPA methods may provide a different view of the functional organization of mental processing compared to univariate analysis, wherein MVPA is more sensitive to distributed coding of information whereas univariate analysis is more sensitive to global engagement in ongoing tasks. The results also highlight the need for better ways to integrate these methods.

Distributed representations in memory: insights from functional brain imaging

Forging new memories for facts and events, holding critical details in mind on a moment-to-moment basis, and retrieving knowledge in the service of current goals all depend on a complex interplay between neural ensembles throughout the brain. Over the past decade, researchers have increasingly utilized powerful analytical tools (e.g., multivoxel pattern analysis) to decode the information represented within distributed functional magnetic resonance imaging activity patterns. In this review, we discuss how these methods can sensitively index neural representations of perceptual and semantic content and how leverage on the engagement of distributed representations provides unique insights into distinct aspects of memory-guided behavior. We emphasize that, in addition to characterizing the contents of memories, analyses of distributed patterns shed light on the processes that influence how information is encoded, maintained, or retrieved, and thus inform memory theory. We conclude by highlighting open questions about memory that can be addressed through distributed pattern analyses.

Multi-voxel patterns of visual category representation during episodic encoding are predictive of subsequent memory

Successful encoding of episodic memories is thought to depend on contributions from prefrontal and temporal lobe structures. Neural processes that contribute to successful encoding have been extensively explored through univariate analyses of neuroimaging data that compare mean activity levels elicited during the encoding of events that are subsequently remembered vs. those subsequently forgotten. Here, we applied pattern classification to fMRI data to assess the degree to which distributed patterns of activity within prefrontal and temporal lobe structures elicited during the encoding of word-image pairs were diagnostic of the visual category (Face or Scene) of the encoded image. We then assessed whether representation of category information was predictive of subsequent memory. Classification analyses indicated that temporal lobe structures contained information robustly diagnostic of visual category. Information in prefrontal cortex was less diagnostic of visual category, but was nonetheless associated with highly reliable classifier-based evidence for category representation. Critically, trials associated with greater classifier-based estimates of category representation in temporal and prefrontal regions were associated with a higher probability of subsequent remembering. Finally, consideration of trial-by-trial variance in classifier-based measures of category representation revealed positive correlations between prefrontal and temporal lobe representations, with the strength of these correlations varying as a function of the category of image being encoded. Together, these results indicate that multi-voxel representations of encoded information can provide unique insights into how visual experiences are transformed into episodic memories.