Eye movements during visual imagery and perception show spatial correspondence but have unique temporal signatures

Differential eye movements and greater pupil size during mental scene construction in autobiographical recall

There is growing evidence supporting a role for eye movements during autobiographical recall, but their potential functionality remains unclear. We hypothesise that the oculomotor system facilitates the process of mental scene construction, in which complex scenes associated with an autobiographical event are generated and maintained during recall. To explore this, we examined spontaneous eye movements during retrieval of cued autobiographical memories. Participants' verbal descriptions of each memory were recorded in synchronisation with their eye movements and pupil size during recall. For each memory participants described the place (details of the environment where the event took place) and the event (details of what happened). Narratives were analyzed using the Autobiographical Interview procedure, which separated internal spatial (place) and non-spatial (event, thoughts and emotion) details. Eye movements during recall of spatial details had significantly higher fixation duration and smaller saccade amplitude and peak velocity, and a higher number of consecutive unidirectional saccades, in comparison to recall of non-spatial details. Recurrence quantification analysis indicated longer sequences of refixations and more repetitions of the same fixation pattern when participants described spatial details. Recall of spatial details was also associated with significantly greater pupil area. Overall findings are consistent with the spontaneous production of more structured saccade patterns and greater cognitive load during the recall of internal spatial episodic scene details in comparison to episodic non-spatial details. These results are consistent with the oculomotor system facilitating the activation and correct positioning of elements of a complex scene relative to other imagined elements during autobiographical recall.

Differential Effects of Visual and Auditory Cognitive Tasks on Smooth Pursuit Eye Movements

Smooth pursuit eye movements (SPEM) are important to gather visual information that guides our interactions with moving objects (e.g., tracking a tennis ball, or following a car when driving). In many neurological conditions, from Parkinson's disease to stroke, the voluntary control of SPEM can be compromised. Therefore, SPEMs can serve as sensitive proxies for assessing cognitive and sensorimotor function. Prior research has shown that SPEMs are influenced by attention and working memory load, yet it remains unclear how the sensory modality of concurrent tasks interacts with these effects. Here, we conducted a 3 (working memory load: no load, easy [low load], and hard [high load]) × 2 (sensory modality: visual vs. auditory) experiment to examine how working memory load and secondary task modality interact to affect SPEM in healthy young adults. Participants tracked a moving circle while simultaneously performing an arithmetic task, where they added either constant (1) or variable (1-5) numbers which were presented visually or auditorily. Our results showed that a secondary auditory task increased tracking variability during high cognitive load. In contrast, we found that the visual task improved tracking, reducing variability irrespective of cognitive load. We interpret our results as evidence that auditory processing requires additional top-down control that is critical for the control of smooth pursuit, diverting resources required for smooth pursuit and, consequently, increasing SPEM variability. These findings emphasize the importance of sensory modality in understanding the interactions between working memory and oculomotor control. We suggest that auditory secondary cognitive tasks may provide a more sensitive test of sensorimotor control deficits in future research with clinical populations.

A systematic review of eye movements during autobiographical recall: Does the mind's eye look at pictures of personal memories?

Retrieving personal memories is usually accompanied by eye movements. Although the functional significance of eye movements during retrieval is relatively well established in the case of episodic memory, their role in autobiographical memory is not clearly delineated in the literature. This systematic review critically examines existing studies in the field to summarize the current understanding of eye movements during autobiographical recall, leading to three conclusions. First, eye movements can be taken to reflect the retrieval of mental visual images in autobiographical memory. Second, eye movements may serve a functional role and support recall by helping retrieve visual details of the memory. Third, eye movements appear to be modulated by various aspects of the retrieval process, suggesting that they could meaningfully reflect aspects of the cognitive processes at play. The discussion highlights the major limitations of current research and proposes suggestions for future studies that will allow developing a more robust theoretical framework.

Brain regions supporting retrieval of words drawn at encoding: fMRI evidence for multimodal reactivation

Memory for words that are drawn or sketched by the participant, rather than written, during encoding is typically superior. While this drawing benefit has been reliably demonstrated in recent years, there has yet to be an investigation of its neural basis. Here, we asked participants to either create drawings, repeatedly write, or list physical characteristics depicting each target word during encoding. Participants then completed a recognition memory test for target words while undergoing functional magnetic resonance imaging (fMRI). Behavioural results showed memory was significantly higher for words drawn than written, replicating the typical drawing effect. Memory for words whose physical characteristics were listed at encoding was also higher than for those written repeatedly, but lower than for those drawn. Voxel-wise analyses of fMRI data revealed two distributed sets of brain regions more active for items drawn relative to written, the left angular gyrus (BA 39) and bilateral frontal (BA 10) regions, suggesting integration and self-referential processing during retrieval of drawn words. Brain-behaviour correlation analyses showed that the size of one's memory benefit for words drawn relative to written at encoding was positively correlated with activation in brain regions linked to visual representation and imagery (BA 17 and cuneus) and motor planning (premotor and supplementary motor areas; BA 6). This study suggests that drawing benefits memory by coactivating multiple sensory traces. Target words drawn during encoding are subsequently remembered by re-engaging visual, motoric, and semantic representations.

Internal coupling: Eye behavior coupled to visual imagery

Our eyes do not only respond to visual perception but also to internal cognition involving visual imagery, which can be referred to as internal coupling. This review synthesizes evidence on internal coupling across diverse domains including episodic memory and simulation, visuospatial memory, numerical cognition, object movement, body movement, and brightness imagery. In each domain, eye movements consistently reflect distinct aspects of mental imagery typically akin to those seen in corresponding visual experiences. Several findings further suggest that internal coupling may not only coincide with but also supports internal cognition as evidenced by improved cognitive performance. Available theoretical accounts suggest that internal coupling may serve at least two functional roles in visual imagery: facilitating memory reconstruction and indicating shifts in internal attention. Moreover, recent insights into the neurobiology of internal coupling highlight substantially shared neural pathways in externally and internally directed cognition. The review concludes by identifying open questions and promising avenues for future research such as exploring moderating roles of context and individual differences in internal coupling.

How workload and availability of spatial reference shape eye movement coupling in visuospatial working memory

Eyes are active in memory recall and visual imagination, yet our grasp of the underlying qualities and factors of these internally coupled eye movements is limited. To explore this, we studied 50 participants, examining how workload, spatial reference availability, and imagined movement direction influence internal coupling of eye movements. We designed a visuospatial working memory task in which participants mentally moved a black patch along a path within a matrix and each trial involved one step along this path (presented via speakers: up, down, left, or right). We varied workload by adjusting matrix size (3 × 3 vs. 5 × 5), manipulated availability of a spatial frame of reference by presenting either a blank screen (requiring participants to rely solely on their mental representation of the matrix) or spatial reference in the form of an empty matrix, and contrasted active task performance to two control conditions involving only active or passive listening. Our findings show that eye movements consistently matched the imagined movement of the patch in the matrix, not driven solely by auditory or semantic cues. While workload influenced pupil diameter, perceived demand, and performance, it had no observable impact on internal coupling. The availability of spatial reference enhanced coupling of eye movements, leading more frequent, precise, and resilient saccades against noise and bias. The absence of workload effects on coupled saccades in our study, in combination with the relatively high degree of coupling observed even in the invisible matrix condition, indicates that eye movements align with shifts in attention across both visually and internally represented information. This suggests that coupled eye movements are not merely strategic efforts to reduce workload, but rather a natural response to where attention is directed.

Behavioral and Neural Correlates of Visual Working Memory Reveal Metacognitive Aspects of Mental Imagery

Mental imagery (MI) is the ability to generate visual phenomena in the absence of sensory input. MI is often likened to visual working memory (VWM): the ability to maintain and manipulate visual representations. How MI is recruited during VWM is yet to be established. In a modified orientation change-discrimination task, we examined how behavioral (proportion correct) and neural (contralateral delay activity [CDA]) correlates of precision and capacity map onto subjective ratings of vividness and number of items in MI within a VWM task. During the maintenance period, 17 participants estimated the vividness of their MI or the number of items held in MI while they were instructed to focus on either precision or capacity of their representation and to retain stimuli at varying set sizes (1, 2, and 4). Vividness and number ratings varied over set sizes; however, subjective ratings and behavioral performance correlated only for vividness rating at set size 1. Although CDA responded to set size as was expected, CDA did not reflect subjective reports on high and low vividness and on nondivergent (reported the probed number of items in mind) or divergent (reported number of items diverged from probed) rating trials. Participants were more accurate in low set sizes compared with higher set sizes and in coarse (45°) orientation changes compared with fine (15°) orientation changes. We failed to find evidence for a relationship between the subjective sensory experience of precision and capacity of MI and the precision and capacity of VWM.

Imagery-related eye movements in 3D space depend on individual differences in visual object imagery

During recall of visual information people tend to move their eyes even though there is nothing to see. Previous studies indicated that such eye movements are related to the spatial location of previously seen items on 2D screens, but they also showed that eye movement behavior varies significantly across individuals. The reason for these differences remains unclear. In the present study we used immersive virtual reality to investigate how individual tendencies to process and represent visual information contribute to eye fixation patterns in visual imagery of previously inspected objects in three-dimensional (3D) space. We show that participants also look back to relevant locations when they are free to move in 3D space. Furthermore, we found that looking back to relevant locations depends on individual differences in visual object imagery abilities. We suggest that object visualizers rely less on spatial information because they tend to process and represent the visual information in terms of color and shape rather than in terms of spatial layout. This finding indicates that eye movements during imagery are subject to individual strategies, and the immersive setting in 3D space made individual differences more likely to unfold.

Unrestricted eye movements strengthen effective connectivity from hippocampal to oculomotor regions during scene construction

Scene construction is a key component of memory recall, navigation, and future imagining, and relies on the medial temporal lobes (MTL). A parallel body of work suggests that eye movements may enable the imagination and construction of scenes, even in the absence of external visual input. There are vast structural and functional connections between regions of the MTL and those of the oculomotor system. However, the directionality of connections between the MTL and oculomotor control regions, and how it relates to scene construction, has not been studied directly in human neuroimaging. In the current study, we used dynamic causal modeling (DCM) to interrogate effective connectivity between the MTL and oculomotor regions using a scene construction task in which participants' eye movements were either restricted (fixed-viewing) or unrestricted (free-viewing). By omitting external visual input, and by contrasting free- versus fixed- viewing, the directionality of neural connectivity during scene construction could be determined. As opposed to when eye movements were restricted, allowing free-viewing during construction of scenes strengthened top-down connections from the MTL to the frontal eye fields, and to lower-level cortical visual processing regions, suppressed bottom-up connections along the visual stream, and enhanced vividness of the constructed scenes. Taken together, these findings provide novel, non-invasive evidence for the underlying, directional, connectivity between the MTL memory system and oculomotor system associated with constructing vivid mental representations of scenes.

Eye-movement replay supports episodic remembering

When we bring to mind something we have seen before, our eyes spontaneously unfold in a sequential pattern strikingly similar to that made during the original encounter, even in the absence of supporting visual input. Oculomotor movements of the eye may then serve the opposite purpose of acquiring new visual information; they may serve as self-generated cues, pointing to stored memories. Over 50 years ago Donald Hebb, the forefather of cognitive neuroscience, posited that such a sequential replay of eye movements supports our ability to mentally recreate visuospatial relations during episodic remembering. However, direct evidence for this influential claim is lacking. Here we isolate the sequential properties of spontaneous eye movements during encoding and retrieval in a pure recall memory task and capture their encoding-retrieval overlap. Critically, we show that the fidelity with which a series of consecutive eye movements from initial encoding is sequentially retained during subsequent retrieval predicts the quality of the recalled memory. Our findings provide direct evidence that such scanpaths are replayed to assemble and reconstruct spatio-temporal relations as we remember and further suggest that distinct scanpath properties differentially contribute depending on the nature of the goal-relevant memory.

Perceptual reality monitoring: Neural mechanisms dissociating imagination from reality

There is increasing evidence that imagination relies on similar neural mechanisms as externally triggered perception. This overlap presents a challenge for perceptual reality monitoring: deciding what is real and what is imagined. Here, we explore how perceptual reality monitoring might be implemented in the brain. We first describe sensory and cognitive factors that could dissociate imagery and perception and conclude that no single factor unambiguously signals whether an experience is internally or externally generated. We suggest that reality monitoring is implemented by higher-level cortical circuits that evaluate first-order sensory and cognitive factors to determine the source of sensory signals. According to this interpretation, perceptual reality monitoring shares core computations with metacognition. This multi-level architecture might explain several types of source confusion as well as dissociations between simply knowing whether something is real and actually experiencing it as real. We discuss avenues for future research to further our understanding of perceptual reality monitoring, an endeavour that has important implications for our understanding of clinical symptoms as well as general cognitive function.

Early is left and up: Saccadic responses reveal horizontal and vertical spatial associations of serial order in working memory

Maintaining serial order in working memory is crucial for cognition. Recent theories propose that serial information is achieved by positional coding of items on a spatial frame of reference. In line with this, an early-left and late-right spatial-positional association of response code (SPoARC) effect has been established. Various theoretical accounts have been put forward to explain the SPoARC effect (the mental whiteboard hypothesis, conceptual metaphor theory, polarity correspondence, or the indirect spatial-numerical association effect). Crucially, while all these accounts predict a left-to-right orientation of the SPoARC effect, they make different predictions regarding the direction of a possible vertical SPoARC effect. In this study, we therefore investigated SPoARC effects along the horizontal and vertical spatial dimension by means of saccadic responses. We replicated the left-to-right horizontal SPoARC effect and established for the first time an up-to-down vertical SPoARC effect. The direction of the vertical SPoARC effect was in contrast to that predicted by metaphor theory, polarity correspondence, or by the indirect spatial-numerical association effect. Rather, our results support the mental whiteboard-hypothesis, according to which positions can be flexibly coded on an internal space depending on the task demands. We also found that the strengths of the horizontal and vertical SPoARC effects were correlated, showing that some people are more prone than others to use spatial references for position coding. Our results therefore suggest that context templates used for position marking are not necessarily spatial in nature but depend on individual strategy preferences.