Experience-dependent eye movements, awareness, and hippocampus-dependent memory

Gaze dynamics during natural scene memorization and recognition

Humans can rapidly memorize numerous images, which is surprising considering the limited visual sampling of each image. To enhance the probability of recognition, it is crucial to focus on previously sampled locations most likely to support memory. How does the visuomotor system achieve this? To study this, we analyzed the eye movements of a group of neurotypical observers while they performed a natural scene memorization task. Using comprehensive gaze analysis and computational modeling, we show that observers traded off visual exploration for exploiting information at the most memorable scene locations with repeated viewing. Furthermore, both the explore-exploit trade-off and gaze consistency predicted accurate recognition memory. Finally, false alarms were predicted by confusion of the incoming visual information at fixated locations with previously sampled information from other images. Together, our findings shed light on the symbiotic relationship between attention and memory in facilitating accurate natural scene memory.

Visual Exploration and the Primate Hippocampal Formation

During the 1990s and early 2000s, research in humans and in the nonhuman primate model of human amnesia revealed that tasks involving free viewing of images provided an exceptionally sensitive measure of recognition memory. Performance on these tasks was sensitive to damage restricted to the hippocampus as well as to damage that included medial temporal lobe cortices. Early work in my laboratory used free-viewing tasks to assess the neurophysiological correlates of recognition memory, and the use of naturalistic visual exploration opened rich avenues to assess other aspects of the impact of eye movements on neural activity in the hippocampus and entorhinal cortex. Here, I summarize two main lines of this work and some of the stories of the trainees who made essential contributions to these discoveries.

Peering into the future: Eye movements predict neural repetition effects during episodic simulation

Imagining future scenarios involves recombining different elements of past experiences into a coherent event, a process broadly supported by the brain's default network. Prior work suggests that distinct brain regions may contribute to the inclusion of different simulation features. Here we examine how activity in these brain regions relates to the vividness of future simulations. Thirty-four healthy young adults imagined future events with familiar people and locations in a two-part study involving a repetition suppression paradigm. First, participants imagined events while their eyes were tracked during a behavioral session. Immediately after, participants imagined events during MRI scanning. The events to be imagined were manipulated such that some were identical to those imagined in the behavioral session while others involved new locations, new people, or both. In this way, we could examine how self-report ratings and eye movements predict brain activity during simulation along with specific simulation features. Vividness ratings were negatively correlated with eye movements, in contrast to an often-observed positive relationship with past recollection. Moreover, fewer eye movements predicted greater involvement of the hippocampus during simulation, an effect specific to location features. Our findings suggest that eye movements may facilitate scene construction for future thinking, lending support to frameworks that spatial information forms the foundation of episodic simulation.

Dramatic changes to well-known places go unnoticed

How well do we know our city? It turns out, much more poorly than we might imagine. We used declarative memory and eye-tracking techniques to examine people's ability to detect modifications to real-world landmarks and scenes in Toronto locales with which they have had extensive experience. Participants were poor at identifying which scenes contained altered landmarks, whether the modification was to the landmarks' relative size, internal features, or relation to surrounding context. To determine whether an indirect measure would prove more sensitive, we tracked eye movements during viewing. Changes in overall visual exploration, but not to specific regions of change, were related to participants' explicit endorsement of scenes as modified. These results support the contention that very familiar landmarks are represented at a global or gist level, but not local or fine-grained, level. These findings offer a unified view of memory for gist across verbal and spatial domains, and across recent and remote memory, with implications for hippocampal-neocortical interactions.

Overlapping and distinct phenotypic profiles in Alzheimer's disease and late onset epilepsy: a biologically-based approach

Due to shared hippocampal dysfunction, patients with Alzheimer's dementia and late-onset epilepsy (LOE) report memory decline. Multiple studies have described the epidemiological, pathological, neurophysiological, and behavioral overlap between Alzheimer's Disease and LOE, implying a bi-directional relationship. We describe the neurobiological decline occurring at different spatial in AD and LOE patients, which may explain why their phenotypes overlap and differ. We provide suggestions for clinical recognition of dual presentation and novel approaches for behavioral testing that reflect an "inside-out," or biologically-based approach to testing memory. New memory and language assessments could detect-and treat-memory impairment in AD and LOE at an earlier, actionable stage.

Eye movements dissociate between perceiving, sensing, and unconscious change detection in scenes

Detecting visual changes can be based on perceiving, whereby one can identify a specific detail that has changed, on sensing, whereby one knows that there is a change but is unable to identify what changed, or on unconscious change detection, whereby one is unaware of any change even though the change influences one's behavior. Prior work has indicated that the processes underlying these different types of change detection are functionally and neurally distinct, but the attentional mechanisms that are related to these different types of change detection remain largely unknown. In the current experiment, we examined eye movements during a change detection task in globally manipulated scenes, and participants indicated their change detection confidence on a scale that allowed us to isolate perceiving, sensing, and unconscious change detection. For perceiving-based change detection, but not sensing-based or unconscious change detection, participants were more likely to preferentially revisit highly changed scene regions across the first and second presentation of the scene (i.e., resampling). This increase in resampling started within 250 ms of the test scene onset, suggesting that the effect began within the first two fixations. In addition, changed scenes were related to more clustered (i.e., less dispersed) eye movements than unchanged scenes, particularly when the subjects were highly confident that no change had occurred - providing evidence for change detection outside of conscious awareness. The results indicate that perceiving, sensing, and unconscious change detection responses are related to partially distinct patterns of eye movements.

The extent and specificity of visual exploration determines the formation of recollected memories in complex scenes

Our visual memories of complex scenes often appear as robust, detailed records of the past. Several studies have demonstrated that active exploration with eye movements improves recognition memory for scenes, but it is unclear whether this improvement is due to stronger feelings of familiarity or more detailed recollection. We related the extent and specificity of fixation patterns at encoding and retrieval to different recognition decisions in an incidental memory paradigm. After incidental encoding of 240 real-world scene photographs, participants (N = 44) answered a surprise memory test by reporting whether an image was new, remembered (indicating recollection), or just known to be old (indicating familiarity). To assess the specificity of their visual memories, we devised a novel report procedure in which participants selected the scene region that they specifically recollected, that appeared most familiar, or that was particularly new to them. At encoding, when considering the entire scene,subsequently recollected compared to familiar or forgotten scenes showed a larger number of fixations that were more broadly distributed, suggesting that more extensive visual exploration determines stronger and more detailed memories. However, when considering only the memory-relevant image areas, fixations were more dense and more clustered for subsequently recollected compared to subsequently familiar scenes. At retrieval, the extent of visual exploration was more restricted for recollected compared to new or forgotten scenes, with a smaller number of fixations. Importantly, fixation density and clustering was greater in memory-relevant areas for recollected versus familiar or falsely recognized images. Our findings suggest that more extensive visual exploration across the entire scene, with a subset of more focal and dense fixations in specific image areas, leads to increased potential for recollecting specific image aspects.

Active vision in sight recovery individuals with a history of long-lasting congenital blindness

What we see is intimately linked to how we actively and systematically explore the world through eye movements. However, it is unknown to what degree visual experience during early development is necessary for such systematic visual exploration to emerge. The present study investigated visual exploration behavior in ten human participants whose sight had been restored only in childhood or adulthood, after a period of congenital blindness due to dense bilateral congenital cataracts. Participants freely explored real-world images while their eye movements were recorded. Despite severe residual visual impairments and gaze instability (nystagmus), visual exploration patterns were preserved in individuals with reversed congenital cataract. Modelling analyses indicated that similar to healthy controls, visual exploration in individuals with reversed congenital cataract was based on the low-level (luminance contrast) and high-level (object components) visual content of the images. Moreover, participants used visual short-term memory representations for narrowing down the exploration space. More systematic visual exploration in individuals with reversed congenital cataract was associated with better object recognition, suggesting that active vision might be a driving force for visual system development and recovery. The present results argue against a sensitive period for the development of neural mechanisms associated with visual exploration.SIGNIFICANCE STATEMENTHumans explore the visual world with systematic patterns of eye movements, but it is unknown whether early visual experience is necessary for the acquisition of visual exploration. Here, we show that sight recovery individuals who had been born blind demonstrate highly systematic eye movements while exploring real-world images, despite visual impairments and pervasive gaze instability. In fact, their eye movement patterns were predicted by those of normally sighted controls and models calculating eye movements based on low- and high-level visual features, and they moreover took memory information into account. Since object recognition performance was associated with systematic visual exploration it was concluded that eye movements might be a driving factor for the development of the visual system.

Aging changes the interactions between the oculomotor and memory systems

The use of multi-modal approaches, particularly in conjunction with multivariate analytic techniques, can enrich models of cognition, brain function, and how they change with age. Recently, multivariate approaches have been applied to the study of eye movements in a manner akin to that of neural activity (i.e., pattern similarity). Here, we review the literature regarding multi-modal and/or multivariate approaches, with specific reference to the use of eyetracking to characterize age-related changes in memory. By applying multi-modal and multivariate approaches to the study of aging, research has shown that aging is characterized by moment-to-moment alterations in the amount and pattern of visual exploration, and by extension, alterations in the activity and function of the hippocampus and broader medial temporal lobe (MTL). These methodological advances suggest that age-related declines in the integrity of the memory system has consequences for oculomotor behavior in the moment, in a reciprocal fashion. Age-related changes in hippocampal and MTL structure and function may lead to an increase in, and change in the patterns of, visual exploration in an effort to upregulate the encoding of information. However, such visual exploration patterns may be non-optimal and actually reduce the amount and/or type of incoming information that is bound into a lasting memory representation. This research indicates that age-related cognitive impairments are considerably broader in scope than previously realized.

Looking for the neural basis of memory

Memory neuroscientists often measure neural activity during task trials designed to recruit specific memory processes. Behavior is championed as crucial for deciphering brain-memory linkages but is impoverished in typical experiments that rely on summary judgments. We criticize this approach as being blind to the multiple cognitive, neural, and behavioral processes that occur rapidly within a trial to support memory. Instead, time-resolved behaviors such as eye movements occur at the speed of cognition and neural activity. We highlight successes using eye-movement tracking with in vivo electrophysiology to link rapid hippocampal oscillations to encoding and retrieval processes that interact over hundreds of milliseconds. This approach will improve research on the neural basis of memory because it pinpoints discrete moments of brain-behavior-cognition correspondence.

Deep Convolutional Neural Networks and Transfer Learning for Measuring Cognitive Impairment Using Eye-Tracking in a Distributed Tablet-Based Environment

OBJECTIVE

Alzheimer's disease (AD) is a neurodegenerative disorder that initially presents with memory loss in the presence of underlying neurofibrillary tangle and amyloid plaque pathology. Mild cognitive impairment is the initial symptomatic stage, which is an early window for detecting cognitive impairment prior to progressive decline and dementia. We recently developed the Visuospatial Memory Eye-Tracking Test (VisMET), a passive task capable of classifying cognitive impairment in AD in under five minutes. Here we describe the development of a mobile version of VisMET to enable efficient and widespread administration of the task.

METHODS

We delivered VisMET on iPad devices and used a transfer learning approach to train a deep neural network to track eye gaze. Eye movements were used to extract memory features to assess cognitive status in a population of 250 individuals.

RESULTS

Mild to severe cognitive impairment was identifiable with a test accuracy of 70%. By enforcing a minimal eye tracking calibration error of 2 cm, we achieved an accuracy of 76% which is equivalent to the accuracy obtained using commercial hardware for eye-tracking.

CONCLUSION

This work demonstrates a mobile version of VisMET capable of estimating the presence of cognitive impairment.

SIGNIFICANCE

Given the ubiquity of tablet devices, our approach has the potential to scale globally.

Feedforward prediction error signals during episodic memory retrieval

Our memories enable us to form expectations for our future experiences, yet the precise neural mechanisms underlying how we compare any experience to our memory remain unknown. Here, using intracranial EEG recordings, we show that episodic memories formed after a single visual experience establish expectations for future experience within neocortical-medial temporal lobe circuits. When subsequent experiences violate these expectations, we find a 80-120 Hz prediction error signal that emerges in both visual association areas and the medial temporal lobe. Critically, this error signal emerges in visual association areas first and then propagates to the medial temporal lobe. This error signal is accompanied by alpha coherence between the two regions. Our data therefore suggest that internal models formed from episodic memories are generated throughout the visual hierarchy after just a single exposure, and that these internal models are then used for comparison with future experiences.

The spatial distribution of attention predicts familiarity strength during encoding and retrieval

The memories we form are determined by what we attend to, and conversely, what we attend to is influenced by our memory for past experiences. Although we know that shifts of attention via eye movements are related to memory during encoding and retrieval, the role of specific memory processes in this relationship is unclear. There is evidence that attention may be especially important for some forms of memory (i.e., conscious recollection), and less so for others (i.e., familiarity-based recognition and unconscious influences of memory), but results are conflicting with respect to both the memory processes and eye movement patterns involved. To address this, we used a confidence-based method of isolating eye movement indices of spatial attention that are related to different memory processes (i.e., recollection, familiarity strength, and unconscious memory) during encoding and retrieval of real-world scenes. We also developed a new method of measuring the dispersion of eye movements, which proved to be more sensitive to memory processing than previously used measures. Specifically, in 2 studies, we found that familiarity strength-that is, changes in subjective reports of memory confidence-increased with (a) more dispersed patterns of viewing during encoding, (b) less dispersed viewing during retrieval, and (c) greater overlap in regions viewed between encoding and retrieval (i.e., resampling). Recollection was also related to these eye movements in a similar manner, though the associations with recollection were less consistent across experiments. Furthermore, we found no evidence for effects related to unconscious influences of memory. These findings indicate that attentional processes during viewing may not preferentially relate to recollection, and that the spatial distribution of eye movements is directly related to familiarity-based memory during encoding and retrieval. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Temporal context guides visual exploration during scene recognition

Memories for episodes are temporally structured. Cognitive models derived from list-learning experiments attribute this structure to the retrieval of temporal context information that indicates when a memory occurred. These models predict key features of memory recall, such as the strong tendency to retrieve studied items in the order in which they were first encountered. Can such models explain ecological memory behaviors, such as eye movements during encoding and retrieval of complex visual stimuli? We tested predictions from retrieved-context models using three data sets involving recognition memory and free viewing of complex scenes. Subjects reinstated sequences of eye movements from one scene-viewing episode to the next. Moreover, sequence reinstatement decayed over time and was associated with successful memory. We observed memory-driven reinstatement even after accounting for intrinsic scene properties that produced consistent eye movements. These findings confirm predictions of retrieved-context models, suggesting retrieval of temporal context influences complex behaviors generated during naturalistic memory experiences. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

Long-term memory and hippocampal function support predictive gaze control during goal-directed search

Eye movements during visual search change with prior experience for search stimuli. Previous studies measured these gaze effects shortly after initial viewing, typically during free viewing; it remains open whether the effects are preserved across long delays and for goal-directed search, and which memory system guides gaze. In Experiment 1, we analyzed eye movements of healthy adults viewing novel and repeated scenes while searching for a scene-embedded target. The task was performed across different time points to examine the repetition effects in long-term memory, and memory types were grouped based on explicit recall of targets. In Experiment 2, an amnesic person with bilateral extended hippocampal damage and the age-matched control group performed the same task with shorter intervals to determine whether or not the repetition effects depend on hippocampal function. When healthy adults explicitly remembered repeated target-scene pairs, search time and fixation duration decreased, and gaze was directed closer to the target region, than when they forgot targets. These effects were seen even after a one-month delay from their initial viewing, suggesting the effects are associated with long-term, explicit memory. Saccadic amplitude was not strongly modulated by scene repetition or explicit recall of targets. The amnesic person did not show explicit recall or implicit repetition effects, whereas his control group showed similar patterns to those seen in Experiment 1. The results reveal several aspects of gaze control that are influenced by long-term memory. The dependence of gaze effects on medial temporal lobe integrity support a role for this region in predictive gaze control.

Hippocampal theta coordinates memory processing during visual exploration

The hippocampus supports memory encoding and retrieval, which may occur at distinct phases of the theta cycle. These processes dynamically interact over rapid timescales, especially when sensory information conflicts with memory. The ability to link hippocampal dynamics to memory-guided behaviors has been limited by experiments that lack the temporal resolution to segregate encoding and retrieval. Here, we simultaneously tracked eye movements and hippocampal field potentials while neurosurgical patients performed a spatial memory task. Phase-locking at the peak of theta preceded fixations to retrieved locations, indicating that the hippocampus coordinates memory-guided eye movements. In contrast, phase-locking at the trough of theta followed fixations to novel object-locations and predicted intact memory of the original location. Theta-gamma phase amplitude coupling increased during fixations to conflicting visual content, but predicted memory updating. Hippocampal theta thus supports learning through two interleaved processes: strengthening encoding of novel information and guiding exploration based on prior experience.

Modeling the influence of the hippocampal memory system on the oculomotor system

Visual exploration is related to activity in the hippocampus (HC) and/or extended medial temporal lobe system (MTL), is influenced by stored memories, and is altered in amnesic cases. An extensive set of polysynaptic connections exists both within and between the HC and oculomotor systems such that investigating how HC responses ultimately influence neural activity in the oculomotor system, and the timing by which such neural modulation could occur, is not trivial. We leveraged TheVirtualBrain, a software platform for large-scale network simulations, to model the functional dynamics that govern the interactions between the two systems in the macaque cortex. Evoked responses following the stimulation of the MTL and some, but not all, subfields of the HC resulted in observable responses in oculomotor regions, including the frontal eye fields, within the time of a gaze fixation. Modeled lesions to some MTL regions slowed the dissipation of HC signal to oculomotor regions, whereas HC lesions generally did not affect the rapid MTL activity propagation to oculomotor regions. These findings provide a framework for investigating how information represented by the HC/MTL may influence the oculomotor system during a fixation and predict how HC lesions may affect visual exploration.

Alzheimer's disease: A clinical perspective and future nonhuman primate research opportunities

Alzheimer's disease (AD) is a progressive neurodegenerative disorder that is the sixth leading cause of death and the most common cause of dementia worldwide. Over the last few decades, significant advancements have been made in our understanding of AD by investigating the molecular mechanisms underlying amyloid-β and tau pathology. Despite this progress, no disease-modifying treatments exist for AD, an issue that will exacerbated by the rising costs and prevalence of the disorder. Moreover, effective therapies to address the devastating cognitive and behavioral symptoms are also urgently needed. This perspective focuses on the value of nonhuman primate (NHP) models in bridging the molecular, circuit, and behavioral levels of analysis to better understand the complex genetic and environmental/lifestyle factors that contribute to AD pathogenesis. These investigations could provide an opportunity for translating our understanding of the pathogenesis and physiological mechanisms underlying AD and related disorders into new diagnostic approaches and disease-modifying therapies to prevent disease or restore brain function for symptomatic individuals.

Hippocampal damage attenuates habituation to videos in monkeys

Monkeys with selective damage to the hippocampus are often unimpaired in matching-to-sample tests but are reportedly impaired in visual paired comparison. While both tests assess recognition of previously seen images, delayed matching-to-sample may engage active memory maintenance whereas visual paired comparison may not. Passive memory tests that are not rewarded with food and that do not require extensive training may provide more sensitive measures of hippocampal function. To test this hypothesis, we assessed memory in monkeys with hippocampal damage and matched controls by providing them the opportunity to repeatedly view small sets of videos. Monkeys pressed a button to play each video. The same 10 videos were used for six consecutive days, after which 10 new videos were introduced in each of seven cycles of testing. Our measure of memory was the extent to which monkeys habituated with repeated presentations, watching fewer videos per session over time. Monkeys with hippocampal lesions habituated more slowly than did control monkeys, indicating poorer memory for previous viewings. Both groups dishabituated each time new videos were introduced. These results, like those from preferential viewing, suggest that the hippocampus may be especially important for memory of incidentally encoded events.

Eye Movement Performance on the Stroop Test in Adults With ADHD

Objective: To compare the eye movement patterns of adults with ADHD with those of controls as they perform the Stroop test. Method: Thirty individuals with ADHD (ages 18-31), and 30 controls participated in this study. The hypothesis was that under the incongruent condition, the group with ADHD would focus longer on the distracter than the controls. Results: Participants with ADHD showed a more pronounced Stroop effect than the controls. Eye movements indicated that more time was spent fixating on the target than on the distracter. The most significant differences between the groups were the overall time spent on the target and the number of fixations on the target, rather than on the distracter. Furthermore, the group with ADHD made more transitions between the target and distracter stimuli. Conclusion: These results were interpreted to indicate an inefficient strategy used by the group with ADHD in their attempt to ignore the distracter stimuli.

Object-in-place Memory Predicted by Anterolateral Entorhinal Cortex and Parahippocampal Cortex Volume in Older Adults

The lateral portion of the entorhinal cortex is one of the first brain regions affected by tau pathology, an important biomarker for Alzheimer disease. Improving our understanding of this region's cognitive role may help identify better cognitive tests for early detection of Alzheimer disease. Based on its functional connections, we tested the idea that the human anterolateral entorhinal cortex (alERC) may play a role in integrating spatial information into object representations. We recently demonstrated that the volume of the alERC was related to processing the spatial relationships of the features within an object [Yeung, L. K., Olsen, R. K., Bild-Enkin, H. E. P., D'Angelo, M. C., Kacollja, A., McQuiggan, D. A., et al. Anterolateral entorhinal cortex volume predicted by altered intra-item configural processing. Journal of Neuroscience, 37, 5527–5538, 2017]. In this study, we investigated whether the human alERC might also play a role in processing the spatial relationships between an object and its environment using an eye-tracking task that assessed visual fixations to a critical object within a scene. Guided by rodent work, we measured both object-in-place memory, the association of an object with a given context [Wilson, D. I., Langston, R. F., Schlesiger, M. I., Wagner, M., Watanabe, S., & Ainge, J. A. Lateral entorhinal cortex is critical for novel object-context recognition. Hippocampus, 23, 352–366, 2013], and object-trace memory, the memory for the former location of objects [Tsao, A., Moser, M. B., & Moser, E. I. Traces of experience in the lateral entorhinal cortex. Current Biology, 23, 399–405, 2013]. In a group of older adults with varying stages of brain atrophy and cognitive decline, we found that the volume of the alERC and the volume of the parahippocampal cortex selectively predicted object-in-place memory, but not object-trace memory. These results provide support for the notion that the alERC may integrate spatial information into object representations.

VisMET: a passive, efficient, and sensitive assessment of visuospatial memory in healthy aging, mild cognitive impairment, and Alzheimer's disease

The entorhinal-hippocampal circuit is one of the earliest sites of cortical pathology in Alzheimer's disease (AD). Visuospatial memory paradigms that are mediated by the entorhinal-hippocampal circuit may offer a means to detect memory impairment during the early stages of AD. In this study, we developed a 4-min visuospatial memory paradigm called VisMET (Visuospatial Memory Eye-Tracking Task) that passively assesses memory using eye movements rather than explicit memory judgements. We had 296 control or memory-impaired participants view a set of images followed by a modified version of the images with either an object removed, or a new object added. Healthy controls spent significantly more time viewing these manipulations compared to subjects with mild cognitive impairment and AD. Using a logistic regression model, the amount of time that individuals viewed these manipulations could predict cognitive impairment and disease status with an out of sample area under the receiver-operator characteristic curve of 0.85. Based on these results, VisMET offers a passive, sensitive, and efficient memory paradigm capable of detecting objective memory impairment and predicting cognitive and disease status.

Conscious and unconscious memory differentially impact attention: Eye movements, visual search, and recognition processes

A hotly debated question is whether memory influences attention through conscious or unconscious processes. To address this controversy, we measured eye movements while participants searched repeated real-world scenes for embedded targets, and we assessed memory for each scene using confidence-based methods to isolate different states of subjective memory awareness. We found that memory-informed eye movements during visual search were predicted both by conscious recollection, which led to a highly precise first eye movement toward the remembered location, and by unconscious memory, which increased search efficiency by gradually directing the eyes toward the target throughout the search trial. In contrast, these eye movement measures were not influenced by familiarity-based memory (i.e., changes in subjective reports of memory strength). The results indicate that conscious recollection and unconscious memory can each play distinct and complementary roles in guiding attention to facilitate efficient extraction of visual information.

Evidence of relational retrieval, even in the absence of the relational eye movement effect (REME)

Eye movements indicate relational memory retrieval by disproportional viewing of a matching stimulus compared to non-matching stimuli after a pair learning study phase (the REME). Experiment 1 showed that the magnitude of the eye movement effect is linked to the subjective confidence level of the responses and we could not find evidence of the REME in incorrect responses, where participants lack conscious recollection. In experiment 2 we used a no-choice task, where participants were instructed to learn the test displays after they have completed a pair learning study phase. The REME only appeared in those participants, who could spontaneously report their relational retrieval (aware group). Meanwhile both aware and unaware participants demonstrated evidence of relational retrieval in a new eye movement measure. Taken together, we demonstrated that the REME is not necessary for retrieval and in both of our experiments the emergence of the REME was associated with conscious access.

Awareness of what is learned as a characteristic of hippocampus-dependent memory

We explored the relationship between memory performance and conscious knowledge (or awareness) of what has been learned in memory-impaired patients with hippocampal lesions or larger medial temporal lesions. Participants viewed familiar scenes or familiar scenes where a change had been introduced. Patients identified many fewer of the changes than controls. Across all of the scenes, controls preferentially directed their gaze toward the regions that had been changed whenever they had what we term robust knowledge about the change: They could identify that a change occurred, report what had changed, and indicate where the change occurred. Preferential looking did not occur when they were unaware of the change or had only partial knowledge about it. The patients, overall, did not direct their gaze toward the regions that had been changed, but on the few occasions when they had robust knowledge about the change they (like controls) did exhibit this effect. Patients did not exhibit this effect when they were unaware of the change or had partial knowledge. The findings support the idea that awareness of what has been learned is a key feature of hippocampus-dependent memory.

Eye movements support the link between conscious memory and medial temporal lobe function

When individuals select the recently studied (and familiar) item in a multiple-choice memory test, they direct a greater proportion of viewing time toward the to-be-selected item when their choice is correct than when their choice is incorrect. Thus, for both correct and incorrect choices, individuals indicate that the chosen item is old, but viewing time nevertheless distinguishes between old and new items. What kind of memory supports this preferential viewing effect? We recorded eye movements while participants made three-alternative, forced-choice recognition memory judgments for scenes. In experiment 1 (n = 30), the magnitude of the preferential viewing effect was strongly correlated with measures of conscious, declarative memory: recognition accuracy as well as the difference in confidence ratings and in response times for correct and incorrect choices. In four analyses that minimized the contribution of declarative memory in order to detect a possible contribution from other processes, the preferential viewing effect was absent. In experiment 2, five memory-impaired patients with medial temporal lobe lesions exhibited a diminished preferential viewing effect. These patients also exhibited poor recognition accuracy and reduced differences in confidence ratings and response times for correct and incorrect choices. We propose that the preferential viewing effect is a phenomenon of conscious, declarative memory and is dependent on the medial temporal lobe. The findings support the link between medial temporal lobe function and declarative memory. When the effects of experience depend on the medial temporal lobe, the effects reflect conscious memory.

The impact of music and stretched time on pupillary responses and eye movements in slow-motion film scenes

This study investigated the effects of music and playback speed on arousal and visual perception in slow-motion scenes taken from commercial films. Slow-motion scenes are a ubiquitous film technique and highly popular. Yet the psychological effects of mediated time-stretching compared to real-time motion have not been empirically investigated. We hypothesised that music affects arousal and attentional processes. Furthermore, we as-sumed that playback speed influences viewers’ visual perception, resulting in a higher number of eye movements and larger gaze dispersion. Thirty-nine participants watched three film excerpts in a repeated-measures design in conditions with or without music and in slow motion vs. adapted real-time motion (both visual-only). Results show that music in slow-motion film scenes leads to higher arousal compared to no music as indicated by larger pupil diameters in the former. There was no systematic effect of music on visual perception in terms of eye movements. Playback speed influenced visual perception in eye movement parameters such that slow motion resulted in more and shorter fixations as well as more saccades compared to adapted real-time motion. Furthermore, in slow motion there was a higher gaze dispersion and a smaller centre bias, indicating that individuals attended to more detail in slow motion scenes.

Retrieval Demands Adaptively Change Striatal Old/New Signals and Boost Subsequent Long-Term Memory

The striatum is a central part of the dopaminergic mesolimbic system and contributes both to the encoding and retrieval of long-term memories. In this regard, the co-occurrence of striatal novelty and retrieval success effects in independent studies underlines the structure's double duty and suggests dynamic contextual adaptation. To test this hypothesis and further investigate the underlying mechanisms of encoding and retrieval dynamics, human subjects viewed pre-familiarized scene images intermixed with new scenes and classified them as indoor versus outdoor (encoding task) or old versus new (retrieval task), while fMRI and eye tracking data were recorded. Subsequently, subjects performed a final recognition task. As hypothesized, striatal activity and pupil size reflected task-conditional salience of old and new stimuli, but, unexpectedly, this effect was not reflected in the substantia nigra and ventral tegmental area (SN/VTA), medial temporal lobe, or subsequent memory performance. Instead, subsequent memory generally benefitted from retrieval, an effect possibly driven by task difficulty and activity in a network including different parts of the striatum and SN/VTA. Our findings extend memory models of encoding and retrieval dynamics by pinpointing a specific contextual factor that differentially modulates the functional properties of the mesolimbic system.SIGNIFICANCE STATEMENT The mesolimbic system is involved in the encoding and retrieval of information but it is unclear how these two processes are achieved within the same network of brain regions. In particular, memory retrieval and novelty encoding were considered in independent studies, implying that novelty (new > old) and retrieval success (old > new) effects may co-occur in the striatum. Here, we used a common framework implicating the striatum, but not other parts of the mesolimbic system, in tracking context-dependent salience of old and new information. The current study, therefore, paves the way for a more comprehensive understanding of the functional properties of the mesolimbic system during memory encoding and retrieval.

Exploration and Exploitation in Natural Viewing Behavior

Many eye-tracking studies investigate visual behavior with a focus on image features and the semantic content of a scene. A wealth of results on these aspects is available, and our understanding of the decision process where to look has reached a mature stage. However, the temporal aspect, whether to stay and further scrutinize a region (exploitation) or to move on and explore image regions that were yet not in the focus of attention (exploration) is less well understood. Here, we investigate the trade-off between these two processes across stimuli with varying properties and sizes. In a free viewing task, we examined gaze parameters in humans, involving the central tendency, entropy, saccadic amplitudes, number of fixations and duration of fixations. The results revealed that the central tendency and entropy scaled with stimulus size. The mean saccadic amplitudes showed a linear increase that originated from an interaction between the distribution of saccades and the spatial bias. Further, larger images led to spatially more extensive sampling as indicated by a higher number of fixations at the expense of reduced fixation durations. These results demonstrate a profound shift from exploitation to exploration as an adaptation of main gaze parameters with increasing image size.

When eye movements express memory for old and new scenes in the absence of awareness and independent of hippocampus

Eye movements can reflect memory. For example, participants make fewer fixations and sample fewer regions when viewing old versus new scenes (the repetition effect). It is unclear whether the repetition effect requires that participants have knowledge (awareness) of the old–new status of the scenes or if it can occur independent of knowledge about old–new status. It is also unclear whether the repetition effect is hippocampus-dependent or hippocampus-independent. A complication is that testing conscious memory for the scenes might interfere with the expression of unconscious (unaware), experience-dependent eye movements. In experiment 1, 75 volunteers freely viewed old and new scenes without knowledge that memory for the scenes would later be tested. Participants then made memory judgments and confidence judgments for each scene during a surprise recognition memory test. Participants exhibited the repetition effect regardless of the accuracy or confidence associated with their memory judgments (i.e., the repetition effect was independent of their awareness of the old–new status of each scene). In experiment 2, five memory-impaired patients with medial temporal lobe damage and six controls also viewed old and new scenes without expectation of memory testing. Both groups exhibited the repetition effect, even though the patients were impaired at recognizing which scenes were old and which were new. Thus, when participants viewed scenes without expectation of memory testing, eye movements associated with old and new scenes reflected unconscious, hippocampus-independent memory. These findings are consistent with the formulation that, when memory is expressed independent of awareness, memory is hippocampus-independent.

An Anatomical Interface between Memory and Oculomotor Systems

Visual behavior is guided by memories from prior experience and knowledge of the visual scene. The hippocampal system (HC), in particular, has been implicated in the guidance of saccades: Amnesic patients, following damage to the HC, exhibit selective deficits in their gaze patterns. However, the neural circuitry by which mnemonic representations influence the oculomotor system remains unknown. We used a data-driven, network-based approach on directed anatomical connectivity from the macaque brain to reveal an extensive set of polysnaptic pathways spanning the extrastriate, posterior parietal and prefrontal cortices that potentially mediate the exchange of information between the memory and visuo-oculomotor systems. We additionally show how the potential for directed information flow from the hippocampus to oculomotor control areas is exceptionally high. In particular, the dorsolateral pFC and FEF—regions known to be responsible for the cognitive control of saccades—are topologically well positioned to receive information from the hippocampus. Together with neuropsychological evidence of altered gaze patterns following damage to the hippocampus, our findings suggest that a reconsideration of hippocampal involvement in oculomotor guidance is needed.

The relationship between eye movements and subsequent recognition: Evidence from individual differences and amnesia

There is consistent agreement regarding the positive relationship between cumulative eye movement sampling and subsequent recognition, but the role of the hippocampus in this sampling behavior is currently unknown. It is also unclear whether the eye movement repetition effect, i.e., fewer fixations to repeated, compared to novel, stimuli, depends on explicit recognition and/or an intact hippocampal system. We investigated the relationship between cumulative sampling, the eye movement repetition effect, subsequent memory, and the hippocampal system. Eye movements were monitored in a developmental amnesic case (H.C.), whose hippocampal system is compromised, and in a group of typically developing participants while they studied single faces across multiple blocks. The faces were studied from the same viewpoint or different viewpoints and were subsequently tested with the same or different viewpoint. Our previous work suggested that hippocampal representations support explicit recognition for information that changes viewpoint across repetitions (Olsen et al., 2015). Here, examination of eye movements during encoding indicated that greater cumulative sampling was associated with better memory among controls. Increased sampling, however, was not associated with better explicit memory in H.C., suggesting that increased sampling only improves memory when the hippocampal system is intact. The magnitude of the repetition effect was not correlated with cumulative sampling, nor was it related reliably to subsequent recognition. These findings indicate that eye movements collect information that can be used to strengthen memory representations that are later available for conscious remembering, whereas eye movement repetition effects reflect a processing change due to experience that does not necessarily reflect a memory representation that is available for conscious appraisal. Lastly, H.C. demonstrated a repetition effect for fixed viewpoint faces but not for variable viewpoint faces, which suggests that repetition effects are differentially supported by neocortical and hippocampal systems, depending upon the representational nature of the underlying memory trace.

Getting directions from the hippocampus: The neural connection between looking and memory

Investigations into the neural basis of memory in human and non-human primates have focused on the hippocampus and associated medial temporal lobe (MTL) structures. However, how memory signals from the hippocampus affect motor actions is unknown. We propose that approaching this question through eye movement, especially by assessing the changes in looking behavior that occur with experience, is a promising method for exposing neural computations within the hippocampus. Here, we review how looking behavior is guided by memory in several ways, some of which have been shown to depend on the hippocampus, and how hippocampal neural signals are modulated by eye movements. Taken together, these findings highlight the need for future research on how MTL structures interact with the oculomotor system. Probing how the hippocampus reflects and impacts motor output during looking behavior renders a practical path to advance our understanding of the hippocampal memory system.

Modeling Visual Exploration in Rhesus Macaques with Bottom-Up Salience and Oculomotor Statistics

There is a growing interest in studying biological systems in natural settings, in which experimental stimuli are less artificial and behavior is less controlled. In primate vision research, free viewing of complex images has elucidated novel neural responses, and free viewing in humans has helped discover attentional and behavioral impairments in patients with neurological disorders. In order to fully interpret data collected from free viewing of complex scenes, it is critical to better understand what aspects of the stimuli guide viewing behavior. To this end, we have developed a novel viewing behavior model called a Biased Correlated Random Walk (BCRW) to describe free viewing behavior during the exploration of complex scenes in monkeys. The BCRW can predict fixation locations better than bottom-up salience. Additionally, we show that the BCRW can be used to test hypotheses regarding specific attentional mechanisms. For example, we used the BCRW to examine the source of the central bias in fixation locations. Our analyses suggest that the central bias may be caused by a natural tendency to reorient the eyes toward the center of the stimulus, rather than a photographer's bias to center salient items in a scene. Taken together these data suggest that the BCRW can be used to further our understanding of viewing behavior and attention, and could be useful in optimizing stimulus and task design.

Relational Memory Is Evident in Eye Movement Behavior despite the Use of Subliminal Testing Methods

While it is generally agreed that perception can occur without awareness, there continues to be debate about the type of representational content that is accessible when awareness is minimized or eliminated. Most investigations that have addressed this issue evaluate access to well-learned representations. Far fewer studies have evaluated whether or not associations encountered just once prior to testing might also be accessed and influence behavior. Here, eye movements were used to examine whether or not memory for studied relationships is evident following the presentation of subliminal cues. Participants assigned to experimental or control groups studied scene-face pairs and test trials evaluated implicit and explicit memory for these pairs. Each test trial began with a subliminal scene cue, followed by three visible studied faces. For experimental group participants, one face was the studied associate of the scene (implicit test); for controls none were a match. Subsequently, the display containing a match was presented to both groups, but now it was preceded by a visible scene cue (explicit test). Eye movements were recorded and recognition memory responses were made. Participants in the experimental group looked disproportionately at matching faces on implicit test trials and participants from both groups looked disproportionately at matching faces on explicit test trials, even when that face had not been successfully identified as the associate. Critically, implicit memory-based viewing effects seemed not to depend on residual awareness of subliminal scene cues, as subjective and objective measures indicated that scenes were successfully masked from view. The reported outcomes indicate that memory for studied relationships can be expressed in eye movement behavior without awareness.

Eye movements provide an index of veridical memory for temporal order

The present research examined whether eye movements during retrieval capture the relation between an event and its temporal attributes. In two experiments (N=76), we found converging evidence that eye movements reflected the veridicality of memory for temporal order seconds before overt memory judgments, suggesting that these movements captured indirect access to temporal information. These eye movements did not entirely depend on the amount of contextual cueing available (Experiment 1) and reflected the unique ordinal position of an event in a sequence (Experiment 2). Based on our results, we conclude that eye movements reflected the absolute temporal order of past events.

Hippocampal contribution to implicit configuration memory expressed via eye movements during scene exploration

Although hippocampus unequivocally supports explicit/declarative memory, fewer findings have demonstrated its role in implicit expressions of memory. We tested for hippocampal contributions to an implicit expression of configural/relational memory for complex scenes using eye-movement tracking during functional magnetic resonance imaging (fMRI) scanning. Participants studied scenes and were later tested using scenes that resembled study scenes in their overall feature configuration but comprised different elements. These configurally similar scenes were used to limit explicit memory, and were intermixed with new scenes that did not resemble studied scenes. Scene configuration memory was expressed through eye movements reflecting exploration overlap (EO), which is the viewing of the same scene locations at both study and test. EO reliably discriminated similar study-test scene pairs from study-new scene pairs, was reliably greater for similarity-based recognition hits than for misses, and correlated with hippocampal fMRI activity. In contrast, subjects could not reliably discriminate similar from new scenes by overt judgments, although ratings of familiarity were slightly higher for similar than new scenes. Hippocampal fMRI correlates of this weak explicit memory were distinct from EO-related activity. These findings collectively suggest that EO was an implicit expression of scene configuration memory associated with hippocampal activity. Visual exploration can therefore reflect implicit hippocampal-related memory processing that can be observed in eye-movement behavior during naturalistic scene viewing.

The eye movement measure of memory and its relationship with explicit measures

This study examined whether the eye movement can be used to measure memory of past events and its relationship with the explicit measures. In Experiment 1, after studying a list of Chinese characters, the participants received a recognition memory test. For each trial the participants had to indicate, among one studied character and two nonstudied homonyms, which character they had studied. Participants' eye movements were monitored while they viewed the three-character test display. Both the time-course and response-locked measures showed that participants viewed the studied character longer than the nonstudied character regardless of their explicit response. Experiment 2 used a wagering task to assess participants' conscious awareness and found that wagering points predicted viewing time for the target better than the recognition accuracy did. These findings suggest that the effect of memory on viewing time occurs automatically and is weakly associated with subsequent conscious awareness of the studied event.

Accessing forgotten memory traces from long-term memory via visual movements

Because memory retrieval often requires overt responses, it is difficult to determine to what extend forgetting occurs as a problem in explicit accessing of long-term memory traces. In this study, we used eye-tracking measures in combination with a behavioral task that favored high forgetting rates to investigate the existence of memory traces from long-term memory in spite of failure in accessing them consciously. In two experiments, participants were encouraged to encode a large set of sound-picture-location associations. In a later test, sounds were presented and participants were instructed to visually scan, before a verbal memory report, for the correct location of the associated pictures in an empty screen. We found the reactivation of associated memories by sound cues at test biased oculomotor behavior towards locations congruent with memory representations, even when participants failed to consciously provide a memory report of it. These findings reveal the emergence of a memory-guided behavior that can be used to map internal representations of forgotten memories from long-term memory.

Social relevance drives viewing behavior independent of low-level salience in rhesus macaques

Quantifying attention to social stimuli during the viewing of complex social scenes with eye tracking has proven to be a sensitive method in the diagnosis of autism spectrum disorders years before average clinical diagnosis. Rhesus macaques provide an ideal model for understanding the mechanisms underlying social viewing behavior, but to date no comparable behavioral task has been developed for use in monkeys. Using a novel scene-viewing task, we monitored the gaze of three rhesus macaques while they freely viewed well-controlled composed social scenes and analyzed the time spent viewing objects and monkeys. In each of six behavioral sessions, monkeys viewed a set of 90 images (540 unique scenes) with each image presented twice. In two-thirds of the repeated scenes, either a monkey or an object was replaced with a novel item (manipulated scenes). When viewing a repeated scene, monkeys made longer fixations and shorter saccades, shifting from a rapid orienting to global scene contents to a more local analysis of fewer items. In addition to this repetition effect, in manipulated scenes, monkeys demonstrated robust memory by spending more time viewing the replaced items. By analyzing attention to specific scene content, we found that monkeys strongly preferred to view conspecifics and that this was not related to their salience in terms of low-level image features. A model-free analysis of viewing statistics found that monkeys that were viewed earlier and longer had direct gaze and redder sex skin around their face and rump, two important visual social cues. These data provide a quantification of viewing strategy, memory and social preferences in rhesus macaques viewing complex social scenes, and they provide an important baseline with which to compare to the effects of therapeutics aimed at enhancing social cognition.

Students apply research methods to consumer decisions about cognitive enhancing drinks

The goal of this class project was to provide students with a hands-on research experience that allowed autonomy, but eliminated duplication of effort and could be completed within one semester. Our resources were limited to a small supply budget and an introductory psychology subject pool. Six students from a behavioral neuroscience class tested claims made by a drink company that their product improves cognitive function. The students each chose a cognitive task for their part of the project. The tasks included the Donders Reaction Time Task, the Stroop Task, the Raven's Progressive Matrices, a short-term memory span test, the Rey-Osterrieth Complex Figure Test and a simple measure of prefrontal EEG activity. Participants were randomly assigned to an experimental or control drink. The experimental group received the putative cognitive enhancing drink and the control group received a placebo drink that was very similar in color and taste. The two drinks shared no active ingredients. Results suggest that the putative cognitive enhancing drink did not improve performance on any of the tasks and decreased performance on the short-term memory task. These findings are discussed in regard to implications for consumers as well as further research into supplements and their ability to improve cognitive performance. Each student presented his/her results at a university-wide research conference. This project provided a rich experience in which students had the opportunity to carry out a research project from conception to presentation.

A nonparametric method for detecting fixations and saccades using cluster analysis: removing the need for arbitrary thresholds

BACKGROUND

Eye tracking is an important component of many human and non-human primate behavioral experiments. As behavioral paradigms have become more complex, including unconstrained viewing of natural images, eye movements measured in these paradigms have become more variable and complex as well. Accordingly, the common practice of using acceleration, dispersion, or velocity thresholds to segment viewing behavior into periods of fixations and saccades may be insufficient.

NEW METHOD

Here we propose a novel algorithm, called Cluster Fix, which uses k-means cluster analysis to take advantage of the qualitative differences between fixations and saccades. The algorithm finds natural divisions in 4 state space parameters-distance, velocity, acceleration, and angular velocity-to separate scan paths into periods of fixations and saccades. The number and size of clusters adjusts to the variability of individual scan paths.

RESULTS

Cluster Fix can detect small saccades that were often indistinguishable from noisy fixations. Local analysis of fixations helped determine the transition times between fixations and saccades.

COMPARISON WITH EXISTING METHODS

Because Cluster Fix detects natural divisions in the data, predefined thresholds are not needed.

CONCLUSIONS

A major advantage of Cluster Fix is the ability to precisely identify the beginning and end of saccades, which is essential for studying neural activity that is modulated by or time-locked to saccades. Our data suggest that Cluster Fix is more sensitive than threshold-based algorithms but comes at the cost of an increase in computational time.

Saccades during visual exploration align hippocampal 3-8 Hz rhythms in human and non-human primates

Visual exploration in primates depends on saccadic eye movements (SEMs) that cause alternations of neural suppression and enhancement. This modulation extends beyond retinotopic areas, and is thought to facilitate perception; yet saccades may also influence brain regions critical for forming memories of these exploratory episodes. The hippocampus, for example, shows oscillatory activity that is generally associated with encoding of information. Whether or how hippocampal oscillations are influenced by eye movements is unknown. We recorded the neural activity in the human and macaque hippocampus during visual scene search. Across species, SEMs were associated with a time-limited alignment of a low-frequency (3–8 Hz) rhythm. The phase alignment depended on the task and not only on eye movements per se, and the frequency band was not a direct consequence of saccade rate. Hippocampal theta-frequency oscillations are produced by other mammals during repetitive exploratory behaviors, including whisking, sniffing, echolocation, and locomotion. The present results may reflect a similar yet distinct primate homologue supporting active perception during exploration.