Dissociating neural markers of stimulus memorability and subjective recognition during episodic retrieval

Visual perception of highly memorable images is mediated by a distributed network of ventral visual regions that enable a late memorability response

Behavioral and neuroscience studies in humans and primates have shown that memorability is an intrinsic property of an image that predicts its strength of encoding into and retrieval from memory. While previous work has independently probed when or where this memorability effect may occur in the human brain, a description of its spatiotemporal dynamics is missing. Here, we used representational similarity analysis (RSA) to combine functional magnetic resonance imaging (fMRI) with source-estimated magnetoencephalography (MEG) to simultaneously measure when and where the human cortex is sensitive to differences in image memorability. Results reveal that visual perception of High Memorable images, compared to Low Memorable images, recruits a set of regions of interest (ROIs) distributed throughout the ventral visual cortex: a late memorability response (from around 300 ms) in early visual cortex (EVC), inferior temporal cortex, lateral occipital cortex, fusiform gyrus, and banks of the superior temporal sulcus. Image memorability magnitude results are represented after high-level feature processing in visual regions and reflected in classical memory regions in the medial temporal lobe (MTL). Our results present, to our knowledge, the first unified spatiotemporal account of visual memorability effect across the human cortex, further supporting the levels-of-processing theory of perception and memory.

Emotional modulation of memorability in mnemonic discrimination

Although elements such as emotion may serve to enhance or impair memory for images, some images are consistently remembered or forgotten by most people, an intrinsic characteristic of images known as memorability. Memorability explains some of the variability in memory performance, however, the underlying mechanisms of memorability remain unclear. It is known that emotional valence can increase the memorability of an experience, but how these two elements interact is still unknown. Hippocampal pattern separation, a computation that orthogonalizes overlapping experiences as distinct from one another, may be a candidate mechanism underlying memorability. However, these two literatures have remained largely separate. To explore the interaction between image memorability and emotion on pattern separation, we examined performance on an emotional mnemonic discrimination task, a putative behavioral correlate of hippocampal pattern separation, by splitting stimuli into memorable and forgettable categories as determined by a convolutional neural network as well as by emotion, lure similarity, and time of testing (immediately and 24-hour delay). We measured target recognition, which is typically used to determine memorability scores, as well as lure discrimination, which taxes hippocampal pattern separation and has not yet been examined within a memorability framework. Here, we show that more memorable images were better remembered across both target recognition and lure discrimination measures. However, for target recognition, this was only true upon immediate testing, not after a 24-hour delay. For lure discrimination, we found that memorability interacts with lure similarity, but depends on the time of testing, where memorability primarily impacts high similarity lure discrimination when tested immediately but impacts low similarity lure discrimination after a 24-hour delay. Furthermore, only lure discrimination showed an interaction between emotion and memorability, in which forgettable neutral images showed better lure discrimination compared to more memorable images. These results suggest that careful consideration is required of what makes an image memorable and may depend on what aspects of the image are more memorable (e.g., gist vs. detail, emotional vs. neutral).

Symbol superiority: Why $ is better remembered than 'dollar'

Memory typically is better for information presented in picture format than in word format. Dual-coding theory (Paivio, 1969) proposes that this is because pictures are spontaneously labelled, leading to the creation of two representational codes-image and verbal-whereas words often lead to only a single (verbal) code. With this perspective as motivation, the present investigation asked whether common graphic symbols (e.g.,!@#$%&) are afforded primarily verbal coding, akin to words, or whether they also invoke visual imagery, as do pictures. Across four experiments, participants were presented at study with graphic symbols or words (e.g., $ or 'dollar'). In Experiment 1, memory was assessed using free recall; in Experiment 2, memory was assessed using old-new recognition. In Experiment 3, the word set was restricted to a single category. In Experiment 4, memory for graphic symbols, pictures, and words was directly compared. All four experiments demonstrated a memory benefit for symbols relative to words. In a fifth experiment, machine learning estimations of inherent stimulus memorability were found to predict memory performance in the earlier experiments. This study is the first to present evidence that, like pictures, graphic symbols are better remembered than words, in line with dual-coding theory and with a distinctiveness account. We reason that symbols offer a visual referent for abstract concepts that are otherwise unlikely to be spontaneously imaged.

The features underlying the memorability of objects

What makes certain images more memorable than others? While much of memory research has focused on participant effects, recent studies using a stimulus-centric perspective have sparked debate on the determinants of memory, including the roles of semantic and visual features and whether the most prototypical or atypical items are best remembered. Prior studies have typically relied on constrained stimulus sets, limiting a generalized view of the features underlying what we remember. Here, we collected more than 1 million memory ratings for a naturalistic dataset of 26,107 object images designed to comprehensively sample concrete objects. We establish a model of object features that is predictive of image memorability and examined whether memorability could be accounted for by the typicality of the objects. We find that semantic features exert a stronger influence than perceptual features on what we remember and that the relationship between memorability and typicality is more complex than a simple positive or negative association alone.

Dissociating the Impact of Memorability on Electrophysiological Correlates of Memory Encoding Success

Despite its unlimited capacity, not all visual information we encounter is encoded into visual long-term memory. Traditionally, variability in encoding success has been ascribed to variability in the types and efficacy of an individual's cognitive processes during encoding. Accordingly, past studies have identified several neural correlates of variability in encoding success, namely, frontal positivity, occipital alpha amplitude, and frontal theta amplitude, by contrasting the electrophysiological signals recorded during successful and failed encoding processes (i.e., subsequent memory). However, recent research demonstrated individuals remember and forget consistent sets of stimuli, thereby elucidating stimulus-intrinsic factors (i.e., memorability) that determine the ease of memory encoding independent of individual-specific variability in encoding processes. The existence of memorability raises the possibility that canonical EEG correlates of subsequent memory may reflect variability in stimulus-intrinsic factors rather than individual-specific encoding processes. To test this, we recorded the EEG correlates of subsequent memory while participants encoded 600 images of real-world objects and assessed the unique contribution of individual-specific and stimulus-intrinsic factors on each EEG correlate. Here, we found that frontal theta amplitude and occipital alpha amplitude were only influenced by individual-specific encoding success, whereas frontal positivity was influenced by stimulus-intrinsic and individual-specific encoding success. Overall, our results offer novel interpretations of canonical EEG correlates of subsequent memory by demonstrating a dissociable impact of stimulus-intrinsic and individual-specific factors of memory encoding success.

Identifying causal subsequent memory effects

Over 40 y of accumulated research has detailed associations between neuroimaging signals measured during a memory encoding task and later memory performance, across a variety of brain regions, measurement tools, statistical approaches, and behavioral tasks. But the interpretation of these subsequent memory effects (SMEs) remains unclear: if the identified signals reflect cognitive and neural mechanisms of memory encoding, then the underlying neural activity must be causally related to future memory. However, almost all previous SME analyses do not control for potential confounders of this causal interpretation, such as serial position and item effects. We collect a large fMRI dataset and use an experimental design and analysis approach that allows us to statistically adjust for nearly all known exogenous confounding variables. We find that, using standard approaches without adjustment, we replicate several univariate and multivariate subsequent memory effects and are able to predict memory performance across people. However, we are unable to identify any signal that reliably predicts subsequent memory after adjusting for confounding variables, bringing into doubt the causal status of these effects. We apply the same approach to subjects' judgments of learning collected following an encoding period and show that these behavioral measures of mnemonic status do predict memory after adjustments, suggesting that it is possible to measure signals near the time of encoding that reflect causal mechanisms but that existing neuroimaging measures, at least in our data, may not have the precision and specificity to do so.

Radiogenomics of C9orf72 Expansion Carriers Reveals Global Transposable Element Derepression and Enables Prediction of Thalamic Atrophy and Clinical Impairment

Hexanucleotide repeat expansion (HRE) within C9orf72 is the most common genetic cause of frontotemporal dementia (FTD). Thalamic atrophy occurs in both sporadic and familial FTD but is thought to distinctly affect HRE carriers. Separately, emerging evidence suggests widespread derepression of transposable elements (TEs) in the brain in several neurodegenerative diseases, including C9orf72 HRE-mediated FTD (C9-FTD). Whether TE activation can be measured in peripheral blood and how the reduction in peripheral C9orf72 expression observed in HRE carriers relates to atrophy and clinical impairment remain unknown. We used FreeSurfer software to assess the effects of C9orf72 HRE and clinical diagnosis (n = 78 individuals, male and female) on atrophy of thalamic nuclei. We also generated a novel, human, whole-blood RNA-sequencing dataset to determine the relationships among peripheral C9orf72 expression, TE activation, thalamic atrophy, and clinical severity (n = 114 individuals, male and female). We confirmed global thalamic atrophy and reduced C9orf72 expression in HRE carriers. Moreover, we identified disproportionate atrophy of the right mediodorsal lateral nucleus in HRE carriers and showed that C9orf72 expression associated with clinical severity, independent of thalamic atrophy. Strikingly, we found global peripheral activation of TEs, including the human endogenous LINE-1 element L1HS L1HS levels were associated with atrophy of multiple pulvinar nuclei, a thalamic region implicated in C9-FTD. Integration of peripheral transcriptomic and neuroimaging data from human HRE carriers revealed atrophy of specific thalamic nuclei, demonstrated that C9orf72 levels relate to clinical severity, and identified marked derepression of TEs, including L1HS, which predicted atrophy of FTD-relevant thalamic nuclei.SIGNIFICANCE STATEMENT Pathogenic repeat expansion in C9orf72 is the most frequent genetic cause of FTD and amyotrophic lateral sclerosis (ALS; C9-FTD/ALS). The clinical, neuroimaging, and pathologic features of C9-FTD/ALS are well characterized, whereas the intersections of transcriptomic dysregulation and brain structure remain largely unexplored. Herein, we used a novel radiogenomic approach to examine the relationship between peripheral blood transcriptomics and thalamic atrophy, a neuroimaging feature disproportionately impacted in C9-FTD/ALS. We confirmed reduction of C9orf72 in blood and found broad dysregulation of transposable elements-genetic elements typically repressed in the human genome-in symptomatic C9orf72 expansion carriers, which associated with atrophy of thalamic nuclei relevant to FTD. C9orf72 expression was also associated with clinical severity, suggesting that peripheral C9orf72 levels capture disease-relevant information.

Item memorability has no influence on value-based decisions

While making decisions, we often rely on past experiences to guide our choices. However, not all experiences are remembered equally well, and some elements of an experience are more memorable than others. Thus, the intrinsic memorability of past experiences may bias our decisions. Here, we hypothesized that individuals would tend to choose more memorable options than less memorable ones. We investigated the effect of item memorability on choice in two experiments. First, using food images, we found that the same items were consistently remembered, and others consistently forgotten, across participants. However, contrary to our hypothesis, we found that participants did not prefer or choose the more memorable over the less memorable items when choice options were matched for the individuals' valuation of the items. Second, we replicated these findings in an alternate stimulus domain, using words that described the same food items. These findings suggest that stimulus memorability does not play a significant role in determining choice based on subjective value.

Predicting visual memory across images and within individuals

We only remember a fraction of what we see-including images that are highly memorable and those that we encounter during highly attentive states. However, most models of human memory disregard both an image's memorability and an individual's fluctuating attentional states. Here, we build the first model of memory synthesizing these two disparate factors to predict subsequent image recognition. We combine memorability scores of 1100 images (Experiment 1, n = 706) and attentional state indexed by response time on a continuous performance task (Experiments 2 and 3, n = 57 total). Image memorability and sustained attentional state explained significant variance in image memory, and a joint model of memory including both factors outperformed models including either factor alone. Furthermore, models including both factors successfully predicted memory in an out-of-sample group. Thus, building models based on individual- and image-specific factors allows for directed forecasting of our memories. SIGNIFICANCE STATEMENT: Although memory is a fundamental cognitive process, much of the time memory failures cannot be predicted until it is too late. However, in this study, we show that much of memory is surprisingly pre-determined ahead of time, by factors shared across the population and highly specific to each individual. Specifically, we build a new multidimensional model that predicts memory based just on the images a person sees and when they see them. This research synthesizes findings from disparate domains ranging from computer vision, attention, and memory into a predictive model. These findings have resounding implications for domains such as education, business, and marketing, where it is a top priority to predict (and even manipulate) what information people will remember.

An Open-Source Cognitive Test Battery to Assess Human Attention and Memory

Cognitive test batteries are widely used in diverse research fields, such as cognitive training, cognitive disorder assessment, or brain mechanism understanding. Although they need flexibility according to their usage objectives, most test batteries are not available as open-source software and are not be tuned by researchers in detail. The present study introduces an open-source cognitive test battery to assess attention and memory, using a javascript library, p5.js. Because of the ubiquitous nature of dynamic attention in our daily lives, it is crucial to have tools for its assessment or training. For that purpose, our test battery includes seven cognitive tasks (multiple-objects tracking, enumeration, go/no-go, load-induced blindness, task-switching, working memory, and memorability), common in cognitive science literature. By using the test battery, we conducted an online experiment to collect the benchmark data. Results conducted on 2 separate days showed the high cross-day reliability. Specifically, the task performance did not largely change with the different days. Besides, our test battery captures diverse individual differences and can evaluate them based on the cognitive factors extracted from latent factor analysis. Since we share our source code as open-source software, users can expand and manipulate experimental conditions flexibly. Our test battery is also flexible in terms of the experimental environment, i.e., it is possible to experiment either online or in a laboratory environment.

False memories when viewing overlapping scenes

Humans can memorize and later recognize many objects and complex scenes. In this study, we prepared large photographs and presented participants with only partial views to test the fidelity of their memories. The unpresented parts of the photographs were used as a source of distractors with similar semantic and perceptual information. Additionally, we presented overlapping views to determine whether the second presentation provided a memory advantage for later recognition tests. Experiment 1 (N = 28) showed that while people were good at recognizing presented content and identifying new foils, they showed a remarkable level of uncertainty about foils selected from the unseen parts of presented photographs (false alarm, 59%). The recognition accuracy was higher for the parts that were shown twice, irrespective of whether the same identical photograph was viewed twice or whether two photographs with overlapping content were observed. In Experiment 2 (N = 28), the memorability of the large image was estimated by a pre-trained deep neural network. Neither the recognition accuracy for an image part nor the tendency for false alarms correlated with the memorability. Finally, in Experiment 3 (N = 21), we repeated the experiment while measuring eye movements. Fixations were biased toward the center of the original large photograph in the first presentation, and this bias was repeated during the second presentation in both identical and overlapping views. Altogether, our experiments show that people recognize parts of remembered photographs, but they find it difficult to reject foils from unseen parts, suggesting that their memory representation is not sufficiently detailed to rule them out as distractors.

Tracking induced forgetting across both strong and weak memory representations to test competing theories of forgetting

Here we employ a novel analysis to address the question: what causes induced forgetting of pictures? We use baseline memorability as a measure of initial memory strength to ask whether induced forgetting is due to (1) recognition practice damaging the association between the memory representation and the category cue used to activate the representation, (2) the updating of a memory trace by incorporating information about a memory probe presented during recognition practice to the stored trace, (3) inhibitory mechanisms used to resolve the conflict created when correctly selecting the practiced item activates competing exemplars, (4) a global matching model in which repeating some items will hurt memory for other items, or (5) falling into the zone of destruction, where a moderate amount of activation leads to the highest degree of forgetting. None of the accounts of forgetting tested here can comprehensively account for both the novel analyses reported here and previous data using the induced forgetting paradigm. We discuss aspects of forgetting theories that are consistent with the novel analyses and existing data, a potential solution for existing models, proposals for future directions, and considerations when incorporating memorability into models of memory.

Content-specific vulnerability of recent episodic memories in Alzheimer's disease

Endel Tulving's episodic memory framework emphasizes the multifaceted re-experiencing of personal events. Indeed, decades of research focused on the experiential nature of episodic memories, usually treating recent episodic memory as a coherent experiential quality. However, recent insights into the functional architecture of the medial temporal lobe show that different types of mnemonic information are segregated into distinct neural pathways in brain circuits empirically associated with episodic memory. Moreover, recent memories do not fade as a whole under conditions of progressive neurodegeneration in these brain circuits, notably in Alzheimer's disease. Instead, certain memory content seem particularly vulnerable from the moment of their encoding while other content can remain memorable consistently across individuals and contexts. We propose that these observations are related to the content-specific functional architecture of the medial temporal lobe and consequently to a content-specific impairment of memory at different stages of the neurodegeneration. To develop Endel Tulving's inspirational legacy further and to advance our understanding of how memory function is affected by neurodegenerative conditions such as Alzheimer's disease, we postulate that it is compelling to focus on the representational content of recent episodic memories.

•

The functional anatomy of episodic memory segregates different memory content.

•

Alzheimer's disease may cause content-specific loss of recent memories

•

Content-specific memorability across individuals changes with Alzheimer's disease.

•

Content-specific assessment could provide new insights into episodic memory in health and disease

The visual and semantic features that predict object memory: Concept property norms for 1,000 object images

Humans have a remarkable fidelity for visual long-term memory, and yet the composition of these memories is a longstanding debate in cognitive psychology. While much of the work on long-term memory has focused on processes associated with successful encoding and retrieval, more recent work on visual object recognition has developed a focus on the memorability of specific visual stimuli. Such work is engendering a view of object representation as a hierarchical movement from low-level visual representations to higher level categorical organization of conceptual representations. However, studies on object recognition often fail to account for how these high- and low-level features interact to promote distinct forms of memory. Here, we use both visual and semantic factors to investigate their relative contributions to two different forms of memory of everyday objects. We first collected normative visual and semantic feature information on 1,000 object images. We then conducted a memory study where we presented these same images during encoding (picture target) on Day 1, and then either a Lexical (lexical cue) or Visual (picture cue) memory test on Day 2. Our findings indicate that: (1) higher level visual factors (via DNNs) and semantic factors (via feature-based statistics) make independent contributions to object memory, (2) semantic information contributes to both true and false memory performance, and (3) factors that predict object memory depend on the type of memory being tested. These findings help to provide a more complete picture of what factors influence object memorability. These data are available online upon publication as a public resource.

Image memorability is predicted by discriminability and similarity in different stages of a convolutional neural network

The features of an image can be represented at multiple levels-from its low-level visual properties to high-level meaning. What drives some images to be memorable while others are forgettable? We address this question across two behavioral experiments. In the first, different layers of a convolutional neural network (CNN), which represent progressively higher levels of features, were used to select the images that would be shown to 100 participants through a form of prospective assignment. Here, the discriminability/similarity of an image with others, according to different CNN layers dictated the images presented to different groups, who made a simple indoor versus outdoor judgment for each scene. We found that participants remember more scene images that were selected based on their low-level discriminability or high-level similarity. A second experiment replicated these results in an independent sample of 50 participants, with a different order of postencoding tasks. Together, these experiments provide evidence that both discriminability and similarity, at different visual levels, predict image memorability.

The effect of intrinsic image memorability on recollection and familiarity

Many photographs of real-life scenes are very consistently remembered or forgotten by most people, making these images intrinsically memorable or forgettable. Although machine vision algorithms can predict a given image's memorability very well, nothing is known about the subjective quality of these memories: are memorable images recognized based on strong feelings of familiarity or on recollection of episodic details? We tested people's recognition memory for memorable and forgettable scenes selected from image memorability databases, which contain memorability scores for each image, based on large-scale recognition memory experiments. Specifically, we tested the effect of intrinsic memorability on recollection and familiarity using cognitive computational models based on receiver operating characteristics (ROCs; Experiment 1 and 2) and on remember/know (R/K) judgments (Experiment 2). The ROC data of Experiment 2 indicated that image memorability boosted memory strength, but did not find a specific effect on recollection or familiarity. By contrast, ROC data from Experiment 2, which was designed to facilitate encoding and, in turn, recollection, found evidence for a specific effect of image memorability on recollection. Moreover, R/K judgments showed that, on average, memorability boosts recollection rather than familiarity. However, we also found a large degree of variability in these judgments across individual images: some images actually achieved high recognition rates by exclusively boosting familiarity rather than recollection. Together, these results show that current machine vision algorithms that can predict an image's intrinsic memorability in terms of hit rates fall short of describing the subjective quality of human memories.

Memorability of words in arbitrary verbal associations modulates memory retrieval in the anterior temporal lobe

Despite large individual differences in memory performance, people remember certain stimuli with overwhelming consistency. This phenomenon is referred to as the memorability of an individual item. However, it remains unknown whether memorability also affects our ability to retrieve associations between items. Here, using a paired-associates verbal memory task, we combine behavioural data, computational modelling and direct recordings from the human brain to examine how memorability influences associative memory retrieval. We find that certain words are correctly retrieved across participants irrespective of the cues used to initiate memory retrieval. These words, which share greater semantic similarity with other words, are more readily available during retrieval and lead to more intrusions when retrieval fails. Successful retrieval of these memorable items, relative to less memorable ones, results in faster reinstatement of neural activity in the anterior temporal lobe. Collectively, our data reveal how the brain prioritizes certain information to facilitate memory retrieval.

Neural correlates of retrieval-based enhancement of autobiographical memory in older adults

Lifelog photo review is considered to enhance the recall of personal events. While a sizable body of research has explored the neural basis of autobiographical memory (AM), there is limited neural evidence on the retrieval-based enhancement effect on event memory among older adults in the real-world environment. This study examined the neural processes of AM as was modulated by retrieval practice through lifelog photo review in older adults. In the experiment, blood-oxygen-level dependent response during subjects’ recall of recent events was recorded, where events were cued by photos that may or may not have been exposed to a priori retrieval practice (training). Subjects remembered more episodic details under the trained relative to non-trained condition. Importantly, the neural correlates of AM was exhibited by (1) dissociable cortical areas related to recollection and familiarity, and (2) a positive correlation between the amount of recollected episodic details and cortical activation within several lateral temporal and parietal regions. Further analysis of the brain activation pattern at a few regions of interest within the core remember network showed a training_condition × event_detail interaction effect, suggesting that the boosting effect of retrieval practice depended on the level of recollected event details.

Memorability of photographs in subjective cognitive decline and mild cognitive impairment: Implications for cognitive assessment

INTRODUCTION

Impaired long-term memory is a defining feature of mild cognitive impairment (MCI). We tested whether this impairment is item specific, limited to some memoranda, whereas some remain consistently memorable.

METHODS

We conducted item-based analyses of long-term visual recognition memory. Three hundred ninety-four participants (healthy controls, subjective cognitive decline [SCD], and MCI) in the multicentric DZNE-Longitudinal Cognitive Impairment and Dementia Study (DELCODE) were tested with images from a pool of 835 photographs.

RESULTS

We observed consistent memorability for images in healthy controls, SCD, and MCI, predictable by a neural network trained on another healthy sample. Looking at memorability differences between groups, we identified images that could successfully categorize group membership with higher success and a substantial image reduction than the original image set.

DISCUSSION

Individuals with SCD and MCI show consistent memorability for specific items, while other items show significant diagnosticity. Certain stimulus features could optimize diagnostic assessment, while others could support memory.

Population response magnitude variation in inferotemporal cortex predicts image memorability

Most accounts of image and object encoding in inferotemporal cortex (IT) focus on the distinct patterns of spikes that different images evoke across the IT population. By analyzing data collected from IT as monkeys performed a visual memory task, we demonstrate that variation in a complementary coding scheme, the magnitude of the population response, can largely account for how well images will be remembered. To investigate the origin of IT image memorability modulation, we probed convolutional neural network models trained to categorize objects. We found that, like the brain, different natural images evoked different magnitude responses from these networks, and in higher layers, larger magnitude responses were correlated with the images that humans and monkeys find most memorable. Together, these results suggest that variation in IT population response magnitude is a natural consequence of the optimizations required for visual processing, and that this variation has consequences for visual memory.

The perceptual neural trace of memorable unseen scenes

Some scenes are more memorable than others: they cement in minds with consistencies across observers and time scales. While memory mechanisms are traditionally associated with the end stages of perception, recent behavioral studies suggest that the features driving these memorability effects are extracted early on, and in an automatic fashion. This raises the question: is the neural signal of memorability detectable during early perceptual encoding phases of visual processing? Using the high temporal resolution of magnetoencephalography (MEG), during a rapid serial visual presentation (RSVP) task, we traced the neural temporal signature of memorability across the brain. We found an early and prolonged memorability related signal under a challenging ultra-rapid viewing condition, across a network of regions in both dorsal and ventral streams. This enhanced encoding could be the key to successful storage and recognition.