Detecting individual memories through the neural decoding of memory states and past experience

Artificial intelligence in insanity evaluation. Potential opportunities and current challenges

The formulation of a scientific opinion on whether the individual who committed a crime should be held responsible for his/her actions or should be considered not responsible by reason of insanity is very difficult. Indeed, forensic psychopathological decision on insanity is highly prone to errors and is affected by human cognitive biases, resulting in low inter-rater reliability. In this context, artificial intelligence can be extremely useful to improve the inter-subjectivity of insanity evaluation. In this paper, we discuss the possible applications of artificial intelligence in this field as well as the challenges and pitfalls that hamper the effective implementation of AI in insanity evaluation. In particular, thus far, it is possible to apply only supervised algorithms without knowing which is the ground truth and which data should be used to train and test the algorithms. In addition, it is not known which percentage of accuracy of the algorithms is sufficient to support partial or total insanity, nor which are the boundaries between sanity and partial or total insanity. Finally, ethical aspects have not been sufficiently investigated. We conclude that these pitfalls should be resolved before AI can be safely and reliably applied in criminal trials.

Enhancing prediction of human traits and behaviors through ensemble learning of traditional and novel resting-state fMRI connectivity analyses

Recent advances in cognitive neuroscience have focused on using resting-state functional connectivity (RSFC) data from fMRI scans to more accurately predict human traits and behaviors. Traditional approaches generally analyze RSFC by correlating averaged time-series data across regions of interest (ROIs) or networks, which may overlook important spatial signal patterns. To address this limitation, we introduced a novel linear regression technique that estimates RSFC by predicting spatial brain activity patterns in a target ROI from those in a seed ROI. We applied both traditional and our novel RSFC estimation methods to a large-scale dataset from the Human Connectome Project and the Brain Genomics Superstruct Project, analyzing resting-state fMRI data to predict sex, age, personality traits, and psychological task performance. To enhance prediction accuracy, we developed an ensemble learner that combines these qualitatively different methods using a weighted average approach. Our findings revealed that hierarchical clustering of RSFC patterns using our novel method displays distinct whole-brain grouping patterns compared to the traditional approach. Importantly, the ensemble model, integrating these diverse weak learners, outperformed the traditional RSFC method in predicting human traits and behaviors. Notably, the predictions from the traditional and novel methods showed relatively low similarity, indicating that our novel approach captures unique and previously undetected information about human traits and behaviors through fine-grained local spatial patterns of neural activation. These results highlight the potential of combining traditional and innovative RSFC analysis techniques to enrich our understanding of the neural basis of human traits and behaviors.

Have you been there before? Decoding recognition of spatial scenes from fMRI signals in precuneus

One potential application of forensic "brain reading" is to test whether a suspect has previously experienced a crime scene. Here, we investigated whether it is possible to decode real life autobiographic exposure to spatial locations using fMRI. In the first session, participants visited four out of eight possible rooms on a university campus. During a subsequent scanning session, subjects passively viewed pictures and videos from these eight possible rooms (four old, four novel) without giving any responses. A multivariate searchlight analysis was employed that trained a classifier to distinguish between "seen" versus "unseen" stimuli from a subset of six rooms. We found that bilateral precuneus encoded information that can be used to distinguish between previously seen and unseen rooms and that also generalized to the two stimuli left out from training. We conclude that activity in bilateral precuneus is associated with the memory of previously visited rooms, irrespective of the identity of the room, thus supporting a parietal contribution to episodic memory for spatial locations. Importantly, we could decode whether a room was visited in real life without the need of explicit judgments about the rooms. This suggests that recognition is an automatic response that can be decoded from fMRI data, thus potentially supporting forensic applications of concealed information tests for crime scene recognition.

Neuronal population representation of human emotional memory

Understanding how emotional processing modulates learning and memory is crucial for the treatment of neuropsychiatric disorders characterized by emotional memory dysfunction. We investigate how human medial temporal lobe (MTL) neurons support emotional memory by recording spiking activity from the hippocampus, amygdala, and entorhinal cortex during encoding and recognition sessions of an emotional memory task in patients with pharmaco-resistant epilepsy. Our findings reveal distinct representations for both remembered compared to forgotten and emotional compared to neutral scenes in single units and MTL population spiking activity. Additionally, we demonstrate that a distributed network of human MTL neurons exhibiting mixed selectivity on a single-unit level collectively processes emotion and memory as a network, with a small percentage of neurons responding conjointly to emotion and memory. Analyzing spiking activity enables a detailed understanding of the neurophysiological mechanisms underlying emotional memory and could provide insights into how emotion alters memory during healthy and maladaptive learning.

Adding Meaning to Memories: How Parietal Cortex Combines Semantic Content with Episodic Experience

Neuroimaging studies of human memory have consistently found that univariate responses in parietal cortex track episodic experience with stimuli (whether stimuli are 'old' or 'new'). More recently, pattern-based fMRI studies have shown that parietal cortex also carries information about the semantic content of remembered experiences. However, it is not well understood how memory-based and content-based signals are integrated within parietal cortex. Here, in humans (males and females), we used voxel-wise encoding models and a recognition memory task to predict the fMRI activity patterns evoked by complex natural scene images based on (1) the episodic history and (2) the semantic content of each image. Models were generated and compared across distinct subregions of parietal cortex and for occipitotemporal cortex. We show that parietal and occipitotemporal regions each encode memory and content information, but they differ in how they combine this information. Among parietal subregions, angular gyrus was characterized by robust and overlapping effects of memory and content. Moreover, subject-specific semantic tuning functions revealed that successful recognition shifted the amplitude of tuning functions in angular gyrus but did not change the selectivity of tuning. In other words, effects of memory and content were additive in angular gyrus. This pattern of data contrasted with occipitotemporal cortex where memory and content effects were interactive: memory effects were preferentially expressed by voxels tuned to the content of a remembered image. Collectively, these findings provide unique insight into how parietal cortex combines information about episodic memory and semantic content.SIGNIFICANCE STATEMENT Neuroimaging studies of human memory have identified multiple brain regions that not only carry information about "whether" a visual stimulus is successfully recognized but also "what" the content of that stimulus includes. However, a fundamental and open question concerns how the brain integrates these two types of information (memory and content). Here, using a powerful combination of fMRI analysis methods, we show that parietal cortex, particularly the angular gyrus, robustly combines memory- and content-related information, but these two forms of information are represented via additive, independent signals. In contrast, memory effects in high-level visual cortex critically depend on (and interact with) content representations. Together, these findings reveal multiple and distinct ways in which the brain combines memory- and content-related information.

Identifying causal subsequent memory effects

Over 40 y of accumulated research has detailed associations between neuroimaging signals measured during a memory encoding task and later memory performance, across a variety of brain regions, measurement tools, statistical approaches, and behavioral tasks. But the interpretation of these subsequent memory effects (SMEs) remains unclear: if the identified signals reflect cognitive and neural mechanisms of memory encoding, then the underlying neural activity must be causally related to future memory. However, almost all previous SME analyses do not control for potential confounders of this causal interpretation, such as serial position and item effects. We collect a large fMRI dataset and use an experimental design and analysis approach that allows us to statistically adjust for nearly all known exogenous confounding variables. We find that, using standard approaches without adjustment, we replicate several univariate and multivariate subsequent memory effects and are able to predict memory performance across people. However, we are unable to identify any signal that reliably predicts subsequent memory after adjusting for confounding variables, bringing into doubt the causal status of these effects. We apply the same approach to subjects' judgments of learning collected following an encoding period and show that these behavioral measures of mnemonic status do predict memory after adjustments, suggesting that it is possible to measure signals near the time of encoding that reflect causal mechanisms but that existing neuroimaging measures, at least in our data, may not have the precision and specificity to do so.

A machine learning based approach towards high-dimensional mediation analysis

Mediation analysis is used to investigate the role of intermediate variables (mediators) that lie in the path between an exposure and an outcome variable. While significant research has focused on developing methods for assessing the influence of mediators on the exposure-outcome relationship, current approaches do not easily extend to settings where the mediator is high-dimensional. These situations are becoming increasingly common with the rapid increase of new applications measuring massive numbers of variables, including brain imaging, genomics, and metabolomics. In this work, we introduce a novel machine learning based method for identifying high dimensional mediators. The proposed algorithm iterates between using a machine learning model to map the high-dimensional mediators onto a lower-dimensional space, and using the predicted values as input in a standard three-variable mediation model. Hence, the machine learning model is trained to maximize the likelihood of the mediation model. Importantly, the proposed algorithm is agnostic to the machine learning model that is used, providing significant flexibility in the types of situations where it can be used. We illustrate the proposed methodology using data from two functional Magnetic Resonance Imaging (fMRI) studies. First, using data from a task-based fMRI study of thermal pain, we combine the proposed algorithm with a deep learning model to detect distributed, network-level brain patterns mediating the relationship between stimulus intensity (temperature) and reported pain at the single trial level. Second, using resting-state fMRI data from the Human Connectome Project, we combine the proposed algorithm with a connectome-based predictive modeling approach to determine brain functional connectivity measures that mediate the relationship between fluid intelligence and working memory accuracy. In both cases, our multivariate mediation model links exposure variables (thermal pain or fluid intelligence), high dimensional brain measures (single-trial brain activation maps or resting-state brain connectivity) and behavioral outcomes (pain report or working memory accuracy) into a single unified model. Using the proposed approach, we are able to identify brain-based measures that simultaneously encode the exposure variable and correlate with the behavioral outcome.

A neuromarker for drug and food craving distinguishes drug users from non-users

Craving is a core feature of substance use disorders. It is a strong predictor of substance use and relapse and is linked to overeating, gambling, and other maladaptive behaviors. Craving is measured via self-report, which is limited by introspective access and sociocultural contexts. Neurobiological markers of craving are both needed and lacking, and it remains unclear whether craving for drugs and food involve similar mechanisms. Across three functional magnetic resonance imaging studies (n = 99), we used machine learning to identify a cross-validated neuromarker that predicts self-reported intensity of cue-induced drug and food craving (P < 0.0002). This pattern, which we term the Neurobiological Craving Signature (NCS), includes ventromedial prefrontal and cingulate cortices, ventral striatum, temporal/parietal association areas, mediodorsal thalamus and cerebellum. Importantly, NCS responses to drug versus food cues discriminate drug users versus non-users with 82% accuracy. The NCS is also modulated by a self-regulation strategy. Transfer between separate neuromarkers for drug and food craving suggests shared neurobiological mechanisms. Future studies can assess the discriminant and convergent validity of the NCS and test whether it responds to clinical interventions and predicts long-term clinical outcomes.

Neurolaw: Challenges and limits

This chapter canvasses the current relevance of behavioral neuroscience to the law, especially to issues of criminal responsibility and competence. It begins with an explanation of the legal doctrines at stake. I then explore the source of the often-inflated claims for the legal relevance of neuroscience. The next section discusses the scientific status of behavioral neuroscience. Then, it addresses two radical challenges to current conceptions of criminal responsibility that neuroscience allegedly poses: determinism and the death of agency. The question of the specific relevance of neuroscience to criminal law doctrine, practice, and institutions is considered next. This is followed by a discussion of how neuroscience evidence is being used in criminal cases in five different countries, including the United States. The penultimate section points to some areas warranting modest optimism. A brief conclusion suggests that neuroscience is at present of limited legal relevance, and advances in the science might alter that judgment.

The present and future of neural interfaces

The 2020's decade will likely witness an unprecedented development and deployment of neurotechnologies for human rehabilitation, personalized use, and cognitive or other enhancement. New materials and algorithms are already enabling active brain monitoring and are allowing the development of biohybrid and neuromorphic systems that can adapt to the brain. Novel brain-computer interfaces (BCIs) have been proposed to tackle a variety of enhancement and therapeutic challenges, from improving decision-making to modulating mood disorders. While these BCIs have generally been developed in an open-loop modality to optimize their internal neural decoders, this decade will increasingly witness their validation in closed-loop systems that are able to continuously adapt to the user's mental states. Therefore, a proactive ethical approach is needed to ensure that these new technological developments go hand in hand with the development of a sound ethical framework. In this perspective article, we summarize recent developments in neural interfaces, ranging from neurohybrid synapses to closed-loop BCIs, and thereby identify the most promising macro-trends in BCI research, such as simulating vs. interfacing the brain, brain recording vs. brain stimulation, and hardware vs. software technology. Particular attention is devoted to central nervous system interfaces, especially those with application in healthcare and human enhancement. Finally, we critically assess the possible futures of neural interfacing and analyze the short- and long-term implications of such neurotechnologies.

Advances in problematic usage of the internet research - A narrative review by experts from the European network for problematic usage of the internet

Global concern about problematic usage of the internet (PUI), and its public health and societal costs, continues to grow, sharpened in focus under the privations of the COVID-19 pandemic. This narrative review reports the expert opinions of members of the largest international network of researchers on PUI in the framework of the European Cooperation in Science and Technology (COST) Action (CA 16207), on the scientific progress made and the critical knowledge gaps remaining to be filled as the term of the Action reaches its conclusion. A key advance has been achieving consensus on the clinical definition of various forms of PUI. Based on the overarching public health principles of protecting individuals and the public from harm and promoting the highest attainable standard of health, the World Health Organisation has introduced several new structured diagnoses into the ICD-11, including gambling disorder, gaming disorder, compulsive sexual behaviour disorder, and other unspecified or specified disorders due to addictive behaviours, alongside naming online activity as a diagnostic specifier. These definitions provide for the first time a sound platform for developing systematic networked research into various forms of PUI at global scale. Progress has also been made in areas such as refining and simplifying some of the available assessment instruments, clarifying the underpinning brain-based and social determinants, and building more empirically based etiological models, as a basis for therapeutic intervention, alongside public engagement initiatives. However, important gaps in our knowledge remain to be tackled. Principal among these include a better understanding of the course and evolution of the PUI-related problems, across different age groups, genders and other specific vulnerable groups, reliable methods for early identification of individuals at risk (before PUI becomes disordered), efficacious preventative and therapeutic interventions and ethical health and social policy changes that adequately safeguard human digital rights. The paper concludes with recommendations for achievable research goals, based on longitudinal analysis of a large multinational cohort co-designed with public stakeholders.

Contributions of representational distinctiveness and stability to memory performance and age differences

Long-standing theories of cognitive aging suggest that memory decline is associated with age-related differences in the way information is neurally represented. Multivariate pattern similarity analyses enabled researchers to take a representational perspective on brain and cognition, and allowed them to study the properties of neural representations that support successful episodic memory. Two representational properties have been identified as crucial for memory performance, namely the distinctiveness and the stability of neural representations. Here, we review studies that used multivariate analysis tools for different neuroimaging techniques to clarify how these representational properties relate to memory performance across adulthood. While most evidence on age differences in neural representations involved stimulus category information , recent studies demonstrated that particularly item-level stability and specificity of activity patterns are linked to memory success and decline during aging. Overall, multivariate methods offer a versatile tool for our understanding of age differences in the neural representations underlying memory.

Imaging human engrams using 7 Tesla magnetic resonance imaging

The investigation of the physical traces of memories (engrams) has made significant progress in the last decade due to optogenetics and fluorescent cell tagging applied in rodents. Engram cells were identified. The ablation of engram cells led to the loss of the associated memory, silent memories were reactivated, and artificial memories were implanted in the brain. Human engram research lags behind engram research in rodents due to methodological and ethical constraints. However, advances in multivariate analysis techniques of functional magnetic resonance imaging (fMRI) data and machine learning algorithms allowed the identification of stable engram patterns in humans. In addition, MRI scanners with an ultrahigh field strength of 7 Tesla (T) have left their prototype state and became more common around the world to assist human engram research. Although most engram research in humans is still being performed with a field strength of 3T, fMRI at 7T will push engram research. Here, we summarize the current state and findings of human engram research and discuss the advantages and disadvantages of applying 7 versus 3T fMRI to image human memory traces.

Psychobiology of Stress and Adolescent Depression (PSY SAD) Study: Protocol overview for an fMRI-based multi-method investigation

Depression is a common, often recurrent disorder that causes substantial disease burden worldwide, and this is especially true for women following the pubertal transition. According to the Social Signal Transduction Theory of Depression, stressors involving social stress and rejection, which frequently precipitate major depressive episodes, induce depressive symptoms in vulnerable individuals in part by altering the activity and connectivity of stress-related neural pathways, and by upregulating components of the immune system involved in inflammation. To test this theory, we recruited adolescent females at high and low risk for depression and assessed their psychological, neural, inflammatory, and genomic responses to a brief (10 minute) social stress task, in addition to trait psychological and microbial factors affecting these responses. We then followed these adolescents longitudinally to investigate how their multi-level stress responses at baseline were related to their biological aging at baseline, and psychosocial and clinical functioning over one year. In this protocol paper, we describe the theoretical motivations for conducting this study as well as the sample, study design, procedures, and measures. Ultimately, our aim is to elucidate how social adversity influences the brain and immune system to cause depression, one of the most common and costly of all disorders.

The influence of item familiarization on neural discriminability during associative memory encoding and retrieval

Associative memory requires one to encode and form memory representations not just for individual items, but for the association or link between those items. Past work has suggested that associative memory is facilitated when individual items are familiar rather than simultaneously learning the items and their associative link. The current study employed multivoxel pattern analyses (MVPA) to investigate whether item familiarization prior to associative encoding affects the distinctiveness of neural patterns, and whether that distinctiveness is also present during associative retrieval. Our results suggest that prior exposure to item stimuli impacts the representations of their shared association compared to stimuli that are novel at the time of associative encoding throughout most of the associative memory network. While this distinction was also present at retrieval, the overall extent of the difference was diminished. Overall the results suggest that stimulus familiarity influences the representation of associative pairings during memory encoding and retrieval, and the pair-specific representation is maintained across memory phases irrespective of this distinction.

Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience

How we process ongoing experiences is shaped by our personal history, current needs, and future goals. Consequently, ventromedial prefrontal cortex (vmPFC) activity involved in processing these subjective appraisals appears to be highly idiosyncratic across individuals. To elucidate the role of the vmPFC in processing our ongoing experiences, we developed a computational framework and analysis pipeline to characterize the spatiotemporal dynamics of individual vmPFC responses as participants viewed a 45-minute television drama. Through a combination of functional magnetic resonance imaging, facial expression tracking, and self-reported emotional experiences across four studies, our data suggest that the vmPFC slowly transitions through a series of discretized states that broadly map onto affective experiences. Although these transitions typically occur at idiosyncratic times across people, participants exhibited a marked increase in state alignment during high affectively valenced events in the show. Our work suggests that the vmPFC ascribes affective meaning to our ongoing experiences.

Getting to Know Someone: Familiarity, Person Recognition, and Identification in the Human Brain

In our everyday life, we continuously get to know people, dominantly through their faces. Several neuroscientific experiments showed that familiarization changes the behavioral processing and underlying neural representation of faces of others. Here, we propose a model of the process of how we actually get to know someone. First, the purely visual familiarization of unfamiliar faces occurs. Second, the accumulation of associated, nonsensory information refines person representation, and finally, one reaches a stage where the effortless identification of very well-known persons occurs. We offer here an overview of neuroimaging studies, first evaluating how and in what ways the processing of unfamiliar and familiar faces differs and, second, by analyzing the fMRI adaptation and multivariate pattern analysis results we estimate where identity-specific representation is found in the brain. The available neuroimaging data suggest that different aspects of the information emerge gradually as one gets more and more familiar with a person within the same network. We propose a novel model of familiarity and identity processing, where the differential activation of long-term memory and emotion processing areas is essential for correct identification.

Re-imagining fMRI for awake behaving infants

Thousands of functional magnetic resonance imaging (fMRI) studies have provided important insight into the human brain. However, only a handful of these studies tested infants while they were awake, because of the significant and unique methodological challenges involved. We report our efforts to address these challenges, with the goal of creating methods for awake infant fMRI that can reveal the inner workings of the developing, preverbal mind. We use these methods to collect and analyze two fMRI datasets obtained from infants during cognitive tasks, released publicly with this paper. In these datasets, we explore and evaluate data quantity and quality, task-evoked activity, and preprocessing decisions. We disseminate these methods by sharing two software packages that integrate infant-friendly cognitive tasks and eye-gaze monitoring with fMRI acquisition and analysis. These resources make fMRI a feasible and accessible technique for cognitive neuroscience in awake and behaving human infants.

Multiple Brain Networks Mediating Stimulus-Pain Relationships in Humans

The brain transforms nociceptive input into a complex pain experience comprised of sensory, affective, motivational, and cognitive components. However, it is still unclear how pain arises from nociceptive input and which brain networks coordinate to generate pain experiences. We introduce a new high-dimensional mediation analysis technique to estimate distributed, network-level patterns that formally mediate the relationship between stimulus intensity and pain. We applied the model to a large-scale analysis of functional magnetic resonance imaging data (N = 284), focusing on brain mediators of the relationship between noxious stimulus intensity and trial-to-trial variation in pain reports. We identify mediators in both traditional nociceptive pathways and in prefrontal, midbrain, striatal, and default-mode regions unrelated to nociception in standard analyses. The whole-brain mediators are specific for pain versus aversive sounds and are organized into five functional networks. Brain mediators predicted pain ratings better than previous brain measures, including the neurologic pain signature (Wager et al. 2013). Our results provide a broader view of the networks underlying pain experience, as well as novel brain targets for interventions.

Distinguishing pain from nociception, salience, and arousal: How autonomic nervous system activity can improve neuroimaging tests of specificity

Pain is a subjective, multidimensional experience that is distinct from nociception. A large body of work has focused on whether pain processing is supported by specific, dedicated brain circuits. Despite advances in human neuroscience and neuroimaging analysis, dissociating acute pain from other sensations has been challenging since both pain and non-pain stimuli evoke salience and arousal responses throughout the body and in overlapping brain circuits. In this review, we discuss these challenges and propose that brain-body interactions in pain can be leveraged in order to improve tests for pain specificity. We review brain and bodily responses to pain and nociception and extant efforts toward identifying pain-specific brain networks. We propose that autonomic nervous system activity should be used as a surrogate measure of salience and arousal to improve these efforts and enable researchers to parse out pain-specific responses in the brain, and demonstrate the feasibility of this approach using example fMRI data from a thermal pain paradigm. This new approach will improve the accuracy and specificity of functional neuroimaging analyses and help to overcome current difficulties in assessing pain specific responses in the human brain.

Tracking Your Mind's Eye during Recollection: Decoding the Long-Term Recall of Short Audiovisual Clips

Unlike familiarity, recollection involves the ability to reconstruct mentally previous events that results in a strong sense of reliving. According to the reinstatement hypothesis, this specific feature emerges from the reactivation of cortical patterns involved during information exposure. Over time, the retrieval of specific details becomes more difficult, and memories become increasingly supported by familiarity judgments. The multiple trace theory (MTT) explains the gradual loss of episodic details by a transformation in the memory representation, a view that is not shared by the standard consolidation model. In this study, we tested the MTT in light of the reinstatement hypothesis. The temporal dynamics of mental imagery from long-term memory were investigated and tracked over the passage of time. Participant EEG activity was recorded during the recall of short audiovisual clips that had been watched 3 weeks, 1 day, or a few hours beforehand. The recall of the audiovisual clips was assessed using a Remember/Know/New procedure, and snapshots of clips were used as recall cues. The decoding matrices obtained from the multivariate pattern analyses revealed sustained patterns that occurred at long latencies (>500 msec poststimulus onset) that faded away over the retention intervals and that emerged from the same neural processes. Overall, our data provide further evidence toward the MTT and give new insights into the exploration of our "mind's eye."

The use of intracranial recordings to decode human language: Challenges and opportunities

Decoding speech from intracranial recordings serves two main purposes: understanding the neural correlates of speech processing and decoding speech features for targeting speech neuroprosthetic devices. Intracranial recordings have high spatial and temporal resolution, and thus offer a unique opportunity to investigate and decode the electrophysiological dynamics underlying speech processing. In this review article, we describe current approaches to decoding different features of speech perception and production - such as spectrotemporal, phonetic, phonotactic, semantic, and articulatory components - using intracranial recordings. A specific section is devoted to the decoding of imagined speech, and potential applications to speech prosthetic devices. We outline the challenges in decoding human language, as well as the opportunities in scientific and neuroengineering applications.

Coordinated representational reinstatement in the human hippocampus and lateral temporal cortex during episodic memory retrieval

Theoretical models of episodic memory have proposed that retrieval depends on interactions between the hippocampus and neocortex, where hippocampal reinstatement of item-context associations drives neocortical reinstatement of item information. Here, we simultaneously recorded intracranial EEG from hippocampus and lateral temporal cortex (LTC) of epilepsy patients who performed a virtual reality spatial navigation task. We extracted stimulus-specific representations of both item and item-context associations from the time-frequency patterns of activity in hippocampus and LTC. Our results revealed a double dissociation of representational reinstatement across time and space: an early reinstatement of item-context associations in hippocampus preceded a later reinstatement of item information in LTC. Importantly, reinstatement levels in hippocampus and LTC were correlated across trials, and the quality of LTC reinstatement was predicted by the magnitude of phase synchronization between hippocampus and LTC. These findings confirm that episodic memory retrieval in humans relies on coordinated representational interactions within a hippocampal-neocortical network.

Episodic memory retrieval is hypothesized to rely on hippocampal reinstatement of item-context associations which drives reinstatement of item information in cortex. Here, the authors confirm this sequence of events, using iEEG recordings from the human hippocampus and lateral temporal cortex.

Memory and the developing brain: From description to explanation with innovation in methods

Recent advances in human cognitive neuroscience show great promise in extending our understanding of the neural basis of memory development. We briefly review the current state of knowledge, highlighting that most work has focused on describing the neural correlates of memory in cross-sectional studies. We then delineate three examples of the application of innovative methods in addressing questions that go beyond description, towards a mechanistic understanding of memory development. First, structural brain imaging and the harmonization of measurements across laboratories may uncover ways in which the maturation of the brain constrains the development of specific aspects of memory. Second, longitudinal designs and sophisticated modeling of the data may identify age-driven changes and the factors that determine individual developmental trajectories. Third, recording memory-related activity directly from the developing brain presents an unprecedented opportunity to examine how distinct brain structures support memory in real time. Finally, the growing prevalence of data sharing offers additional means to tackle questions that demand large-scale datasets, ambitious designs, and access to rare samples. We propose that the use of such innovative methods will move our understanding of memory development from a focus on describing trends to explaining the causal factors that shape behavior.

The effect of mental countermeasures on neuroimaging-based concealed information tests

During the last decade and a half, functional magnetic resonance imaging (fMRI) has been used to determine whether it is possible to detect concealed knowledge by examining brain activation patterns, with mixed results. Concealed information tests rely on the logic that a familiar item (probe) elicits a stronger response than unfamiliar, but otherwise comparable items (irrelevants). Previous work has shown that physical countermeasures can artificially modulate neural responses in concealed information tests, decreasing the accuracy of these methods. However, the question remains as to whether purely mental countermeasures, which are much more difficult to detect than physical ones, can also be effective. An fMRI study was conducted to address this question by assessing the effect of attentional countermeasures on the accuracy of the classification between knowledge and no-knowledge cases using both univariate and multivariate analyses. Results replicate previous work and show reliable group activation differences between the probe and the irrelevants in fronto-parietal networks. Critically, classification accuracy was generally reduced by the mental countermeasures, but only significantly so with region of interest analyses (both univariate and multivariate). For whole-brain analyses, classification accuracy was relatively low, but it was not significantly reduced by the countermeasures. These results indicate that mental countermeasure need to be addressed before these paradigms can be used in applied settings and that methods to defeat countermeasures, or at least to detect their use, need to be developed. HIGHLIGHTS: FMRI-based concealed information tests are vulnerable to mental countermeasures Measures based on regions of interest are affected by mental countermeasures Whole-brain analyses may be more robust than region of interest ones Methods to detect mental countermeasure use are needed for forensic applications.

Effects of age on across-participant variability of cortical reinstatement effects

Using functional magnetic resonance imaging data, we assessed whether across-participant variability of content-selective retrieval-related neural activity differs with age. We addressed this question by employing across-participant multi-voxel pattern analysis (MVPA), predicting that increasing age would be associated with reduced variability of retrieval-related cortical reinstatement across participants. During study, 24 young and 24 older participants viewed objects and concrete words. Test items comprised studied words, names of studied objects, and unstudied words. Participants judged whether the items were recollected, familiar, or new by making 'Remember', 'Know' and 'New' responses, respectively. MVPA was conducted on each region belonging to the 'core recollection network', dorsolateral prefrontal cortex, and a previously identified content-selective voxel set. A leave-one-participant-out classification approach was employed whereby a classifier was trained on a subset of participants and tested on the data from a yoked pair of held-out participants. Classifiers were trained on the study phase data to discriminate the study trials as a function of content (picture or word). The classifiers were then applied to the test phase data to discriminate studied test words according to their study condition. In all of the examined regions, classifier performance demonstrated little or no sensitivity to age and, for the test data, was robustly above chance. Thus, there was little evidence to support the hypothesis that across-participant variability of retrieval-related cortical reinstatement differs with age. The findings extend prior evidence by demonstrating that content-selective cortical reinstatement is sufficiently invariant to support across-participant multi-voxel classification across the healthy adult lifespan.

Briefly Flashed Scenes Can Be Stored in Long-Term Memory

The capacity of human memory is impressive. Previous reports have shown that when asked to memorize images, participants can recognize several thousands of visual objects in great details even with a single viewing of a few seconds per image. In this experiment, we tested recognition performance for natural scenes that participants saw for 20 ms only once (untrained group) or 22 times over many days (trained group) in an unrelated task. 400 images (200 previously viewed and 200 novel images) were flashed one at a time and participants were asked to lift their finger from a pad whenever they thought they had already seen the image (go/no-go paradigm). Compared to previous reports of excellent recognition performance with only single presentations of a few seconds, untrained participants were able to recognize only 64% of the 200 images they had seen few minutes before. On the other hand, trained participants, who had processed the flashed images (20 ms) several times, could correctly recognize 89% of them. EEG recordings confirmed these behavioral results. As early as 230 ms after stimulus onset, a significant event-related-potential (ERP) difference between familiar and new images was observed for the trained but not for the untrained group. These results show that briefly flashed unmasked scenes can be incidentally stored in long-term memory when repeated.

Representation, Pattern Information, and Brain Signatures: From Neurons to Neuroimaging

Human neuroimaging research has transitioned from mapping local effects to developing predictive models of mental events that integrate information distributed across multiple brain systems. Here we review work demonstrating how multivariate predictive models have been utilized to provide quantitative, falsifiable predictions; establish mappings between brain and mind with larger effects than traditional approaches; and help explain how the brain represents mental constructs and processes. Although there is increasing progress toward the first two of these goals, models are only beginning to address the latter objective. By explicitly identifying gaps in knowledge, research programs can move deliberately and programmatically toward the goal of identifying brain representations underlying mental states and processes.

Dissociating neural markers of stimulus memorability and subjective recognition during episodic retrieval

While much of memory research takes an observer-centric focus looking at participant performance, recent work has pinpointed important item-centric effects on memory, or how intrinsically memorable a given stimulus is. However, little is known about the neural correlates of memorability during memory retrieval, or how such correlates relate to subjective memory behavior. Here, stimuli and blood-oxygen-level dependent data from a prior functional magnetic resonance imaging (fMRI) study were reanalyzed using a memorability-based framework. In that study, sixteen participants studied 200 novel face images and were scanned while making recognition memory judgments on those faces, interspersed with 200 unstudied faces. In the current investigation, memorability scores for those stimuli were obtained through an online crowd-sourced (N = 740) continuous recognition test that measured each image's corrected recognition rate. Representational similarity analyses were conducted across the brain to identify regions wherein neural pattern similarity tracked item-specific effects (stimulus memorability) versus observer-specific effects (individual memory performance). We find two non-overlapping sets of regions, with memorability-related information predominantly represented within ventral and medial temporal regions and memory retrieval outcome-related information within fronto-parietal regions. These memorability-based effects persist regardless of image history, implying that coding of stimulus memorability may be a continuous and automatic perceptual process.

Differential Medial Temporal Lobe and Parietal Cortical Contributions to Real-world Autobiographical Episodic and Autobiographical Semantic Memory

Autobiographical remembering can depend on two forms of memory: episodic (event) memory and autobiographical semantic memory (remembering personally relevant semantic knowledge, independent of recalling a specific experience). There is debate about the degree to which the neural signals that support episodic recollection relate to or build upon autobiographical semantic remembering. Pooling data from two fMRI studies of memory for real-world personal events, we investigated whether medial temporal lobe (MTL) and parietal subregions contribute to autobiographical episodic and semantic remembering. During scanning, participants made memory judgments about photograph sequences depicting past events from their life or from others’ lives, and indicated whether memory was based on episodic or semantic knowledge. Results revealed several distinct functional patterns: activity in most MTL subregions was selectively associated with autobiographical episodic memory; the hippocampal tail, superior parietal lobule, and intraparietal sulcus were similarly engaged when memory was based on retrieval of an autobiographical episode or autobiographical semantic knowledge; and angular gyrus demonstrated a graded pattern, with activity declining from autobiographical recollection to autobiographical semantic remembering to correct rejections of novel events. Collectively, our data offer insights into MTL and parietal cortex functional organization, and elucidate circuitry that supports different forms of real-world autobiographical memory.

Knowledge supports memory retrieval through familiarity, not recollection

Semantic memory, or general knowledge of the world, guides learning and supports the formation and retrieval of new episodic memories. Behavioral evidence suggests that this knowledge effect is supported by recollection-a more controlled form of memory retrieval generally accompanied by contextual details-to a greater degree than familiarity-a more automatic form of memory retrieval generally absent of contextual details. In the current study, we used functional magnetic resonance imaging (fMRI) to investigate the role that regions associated with recollection and familiarity play in retrieving recent instances of known (e.g., The Summer Olympic Games are held four years apart) and unknown (e.g., A flaky deposit found in port bottles is beeswing) statements. Our results revealed a surprising pattern: Episodic retrieval of known statements recruited regions associated with familiarity, but not recollection. Instead, retrieval of unknown statements recruited regions associated with recollection. These data, in combination with quicker reaction times for the retrieval of known than unknown statements, suggest that known statements can be successfully retrieved on the basis of familiarity, whereas unknown statements were retrieved on the basis of recollection. Our results provide insight into how knowledge influences episodic retrieval and demonstrate the role of neuroimaging in providing insights into cognitive processes in the absence of explicit behavioral responses.

Neuroanatomical substrates involved in unrelated false facial recognition

Identifying faces is a process central for social interaction and a relevant factor in eyewitness theory. False recognition is a critical mistake during an eyewitness's identification scenario because it can lead to a wrongful conviction. Previous studies have described neural areas related to false facial recognition using the standard Deese/Roediger-McDermott (DRM) paradigm, triggering related false recognition. Nonetheless, misidentification of faces without trying to elicit false memories (unrelated false recognition) in a police lineup could involve different cognitive processes, and distinct neural areas. To delve into the neural circuitry of unrelated false recognition, we evaluated the memory and response confidence of participants while watching faces photographs in an fMRI task. Functional activations of unrelated false recognition were identified by contrasting the activation on this condition vs. the activations related to recognition (hits) and correct rejections. The results identified the right precentral and cingulate gyri as areas with distinctive activations during false recognition events suggesting a conflict resulting in a dysfunction during memory retrieval. High confidence suggested that about 50% of misidentifications may be related to an unconscious process. These findings add to our understanding of the construction of facial memories and its biological basis, and the fallibility of the eyewitness testimony.

Choosing Prediction Over Explanation in Psychology: Lessons From Machine Learning

Psychology has historically been concerned, first and foremost, with explaining the causal mechanisms that give rise to behavior. Randomized, tightly controlled experiments are enshrined as the gold standard of psychological research, and there are endless investigations of the various mediating and moderating variables that govern various behaviors. We argue that psychology's near-total focus on explaining the causes of behavior has led much of the field to be populated by research programs that provide intricate theories of psychological mechanism but that have little (or unknown) ability to predict future behaviors with any appreciable accuracy. We propose that principles and techniques from the field of machine learning can help psychology become a more predictive science. We review some of the fundamental concepts and tools of machine learning and point out examples where these concepts have been used to conduct interesting and important psychological research that focuses on predictive research questions. We suggest that an increased focus on prediction, rather than explanation, can ultimately lead us to greater understanding of behavior.

Neural correlates of deception: lying about past events and personal beliefs

Although a growing body of literature suggests that cognitive control processes are involved in deception, much about the neural correlates of lying remains unknown. In this study, we tested whether brain activation associated with deception, as measured by functional magnetic resonance imaging (fMRI), can be detected either in preparation for or during the execution of a lie, and whether they depend on the content of the lie. We scanned participants while they lied or told the truth about either their personal experiences (episodic memories) or personal beliefs. Regions in the frontal and parietal cortex showed higher activation when participants lied compared with when they were telling the truth, regardless of whether they were asked about their past experiences or opinions. In contrast, lie-related activation in the right temporal pole, precuneus and the right amygdala differed by the content of the lie. Preparing to lie activated parietal and frontal brain regions that were distinct from those activated while participants executed lies. Our findings concur with previous reports on the involvement of frontal and parietal regions in deception, but specify brain regions involved in the preparation vs execution of deception, and those involved in deceiving about experiences vs opinions.

Multiple-object Tracking as a Tool for Parametrically Modulating Memory Reactivation

Converging evidence supports the "nonmonotonic plasticity" hypothesis, which states that although complete retrieval may strengthen memories, partial retrieval weakens them. Yet, the classic experimental paradigms used to study effects of partial retrieval are not ideally suited to doing so, because they lack the parametric control needed to ensure that the memory is activated to the appropriate degree (i.e., that there is some retrieval but not enough to cause memory strengthening). Here, we present a novel procedure designed to accommodate this need. After participants learned a list of word-scene associates, they completed a cued mental visualization task that was combined with a multiple-object tracking (MOT) procedure, which we selected for its ability to interfere with mental visualization in a parametrically adjustable way (by varying the number of MOT targets). We also used fMRI data to successfully train an "associative recall" classifier for use in this task: This classifier revealed greater memory reactivation during trials in which associative memories were cued while participants tracked one, rather than five, MOT targets. However, the classifier was insensitive to task difficulty when recall was not taking place, suggesting that it had indeed tracked memory reactivation rather than task difficulty per se. Consistent with the classifier findings, participants' introspective ratings of visualization vividness were modulated by MOT task difficulty. In addition, we observed reduced classifier output and slowing of responses in a postreactivation memory test, consistent with the hypothesis that partial reactivation, induced by MOT, weakened memory. These results serve as a "proof of concept" that MOT can be used to parametrically modulate memory retrieval-a property that may prove useful in future investigation of partial retrieval effects, for example, in closed-loop experiments.

Contextually Mediated Spontaneous Retrieval Is Specific to the Hippocampus

Although it is now well established that the hippocampus supports memory encoding [1, 2], little is known about hippocampal activity during spontaneous memory retrieval. Recent intracranial electroencephalographic (iEEG) work has shown that hippocampal activity during encoding predicts subsequent temporal organization of memories [3], supporting a role in contextual binding. It is an open question, however, whether the hippocampus similarly supports contextually mediated processes during retrieval. Here, we analyzed iEEG recordings obtained from 215 epilepsy patients as they performed a free recall task. To identify neural activity specifically associated with contextual retrieval, we compared correct recalls, intrusions (incorrect recall of either items from prior lists or items not previously studied), and deliberations (matched periods during recall when no items came to mind). Neural signals that differentiate correct recalls from both other retrieval classes reflect contextual retrieval, as correct recalls alone arise from the correct context. We found that in the hippocampus, high-frequency activity (HFA, 44-100 Hz), a proxy for neural activation [4], was greater prior to correct recalls relative to the other retrieval classes, with no differentiation between intrusions and deliberations. This pattern was not observed in other memory-related cortical regions, including DLPFC, thus supporting a specific hippocampal contribution to contextually mediated memory retrieval.

Cocaine dependence and thalamic functional connectivity: a multivariate pattern analysis

Cocaine dependence is associated with deficits in cognitive control. Previous studies demonstrated that chronic cocaine use affects the activity and functional connectivity of the thalamus, a subcortical structure critical for cognitive functioning. However, the thalamus contains nuclei heterogeneous in functions, and it is not known how thalamic subregions contribute to cognitive dysfunctions in cocaine dependence. To address this issue, we used multivariate pattern analysis (MVPA) to examine how functional connectivity of the thalamus distinguishes 100 cocaine-dependent participants (CD) from 100 demographically matched healthy control individuals (HC). We characterized six task-related networks with independent component analysis of fMRI data of a stop signal task and employed MVPA to distinguish CD from HC on the basis of voxel-wise thalamic connectivity to the six independent components. In an unbiased model of distinct training and testing data, the analysis correctly classified 72% of subjects with leave-one-out cross-validation (p < 0.001), superior to comparison brain regions with similar voxel counts (p < 0.004, two-sample t test). Thalamic voxels that form the basis of classification aggregate in distinct subclusters, suggesting that connectivities of thalamic subnuclei distinguish CD from HC. Further, linear regressions provided suggestive evidence for a correlation of the thalamic connectivities with clinical variables and performance measures on the stop signal task. Together, these findings support thalamic circuit dysfunction in cognitive control as an important neural marker of cocaine dependence.

Distinct neural patterns of social cognition for cooperation versus competition

How do people consider other minds during cooperation versus competition? Some accounts predict that theory of mind (ToM) is recruited more for cooperation versus competition or competition versus cooperation, whereas other accounts predict similar recruitment across these two contexts. The present fMRI study examined activity in brain regions for ToM (bilateral temporoparietal junction, precuneus, dorsomedial prefrontal cortex) across cooperative and competitive interactions with the same individual within the same paradigm. Although univariate analyses revealed that ToM regions overall were recruited similarly across interaction contexts, multivariate pattern analyses revealed that these regions nevertheless encoded information separating cooperation from competition. Specifically, ToM regions encoded differences between cooperation and competition when people believed the outcome was determined by their and their partner's choices but not when the computer determined the outcome. We propose that, when people are motivated to consider others' mental states, ToM regions encode different aspects of mental states during cooperation versus competition. Given the role of these regions for ToM, these findings reveal distinct patterns of social cognition for distinct motivational contexts.

Decoding fMRI Signatures of Real-world Autobiographical Memory Retrieval

Extant neuroimaging data implicate frontoparietal and medial-temporal lobe regions in episodic retrieval, and the specific pattern of activity within and across these regions is diagnostic of an individual's subjective mnemonic experience. For example, in laboratory-based paradigms, memories for recently encoded faces can be accurately decoded from single-trial fMRI patterns [Uncapher, M. R., Boyd-Meredith, J. T., Chow, T. E., Rissman, J., & Wagner, A. D. Goal-directed modulation of neural memory patterns: Implications for fMRI-based memory detection. Journal of Neuroscience, 35, 8531–8545, 2015; Rissman, J., Greely, H. T., & Wagner, A. D. Detecting individual memories through the neural decoding of memory states and past experience. Proceedings of the National Academy of Sciences, U.S.A., 107, 9849–9854, 2010]. Here, we investigated the neural patterns underlying memory for real-world autobiographical events, probed at 1- to 3-week retention intervals as well as whether distinct patterns are associated with different subjective memory states. For 3 weeks, participants (n = 16) wore digital cameras that captured photographs of their daily activities. One week later, they were scanned while making memory judgments about sequences of photos depicting events from their own lives or events captured by the cameras of others. Whole-brain multivoxel pattern analysis achieved near-perfect accuracy at distinguishing correctly recognized events from correctly rejected novel events, and decoding performance did not significantly vary with retention interval. Multivoxel pattern classifiers also differentiated recollection from familiarity and reliably decoded the subjective strength of recollection, of familiarity, or of novelty. Classification-based brain maps revealed dissociable neural signatures of these mnemonic states, with activity patterns in hippocampus, medial PFC, and ventral parietal cortex being particularly diagnostic of recollection. Finally, a classifier trained on previously acquired laboratory-based memory data achieved reliable decoding of autobiographical memory states. We discuss the implications for neuroscientific accounts of episodic retrieval and comment on the potential forensic use of fMRI for probing experiential knowledge.

The Effects of Age on the Neural Correlates of Recollection Success, Recollection-Related Cortical Reinstatement, and Post-Retrieval Monitoring

Functional magnetic resonance imaging (fMRI) was used to investigate whether age-related differences in episodic memory performance are accompanied by a reduction in the specificity of recollected information. We addressed this question by comparing recollection-related cortical reinstatement in young and older adults. At study, subjects viewed objects and concrete words, making 1 of 2 different semantic judgments depending on the study material. Test items were words that corresponded to studied words or the names of studied objects. Subjects indicated whether each test item was recollected, familiar, or novel. Reinstatement of information differentiating the encoding tasks was quantified both with a univariate analysis of the fMRI signal and with a multivoxel pattern analysis, using a classifier that had been trained to discriminate between the 2 classes of study episode. The results of these analyses converged to suggest that reinstatement did not differ according to age. Thus, there was no evidence that specificity of recollected information was reduced in older individuals. Additionally, there were no age effects in the magnitude of recollection-related modulations in regional activity or in the neural correlates of post-retrieval monitoring. Taken together, the findings suggest that the neural mechanisms engaged during successful episodic retrieval can remain stable with advancing age.