Perceived Distance Alters Memory for Scene Boundaries

Object persistence explains event completion

Our minds consistently distort memories of objects and events. Oftentimes, these distortions serve to transform incoherent memories into coherent ones, as when we misremember partial events as whole ("event completion"). What mechanisms drive these distortions? Whereas extant work shows that representations of causality, continuity, familiarity, physical coherence, or event coherence create memory distortions, we suggest that a simpler and more fundamental mechanism may be at play: object persistence. Merely seeing an object take part in an event can create a persisting memory of its presence throughout that event. In 8 pre-registered experiments (N = 317 adults), participants performed a simple task where they watched an animation, then chose whether or not a frame from the animation contained an object. Participants falsely remembered seeing an object when it was not there (E1). These effects persisted in the absence of causality (E2), continuity (E3), event familiarity (E4), object familiarity (E5), even when the events violated physical laws (E6), and when the events themselves were not coherent (E7). However, the effect disappeared when we abolished object persistence (E8). Thus, object persistence alone creates rich, enduring, and coherent representations of objects and events.

Setting the scene for boundary extension: Methods, findings, connections, and theories

A previously viewed scene is often remembered as containing a larger extent of the background than was actually present, and information that was likely present just outside the boundaries of that view is often incorporated into the representation of that scene. This has been referred to as boundary extension. Methodologies used in studies on boundary extension (terminology, stimulus presentation, response measures) are described. Empirical findings regarding effects of characteristics of the stimulus (whether the stimulus depicts a scene, semantics of the scene, view angle, object size, object cropping, object orientation, object color, number of objects, depth of field, object distance, viewpoint production, scene orientation, motion, faces, emotions, modality, whether the scene is multimodal), characteristics of the display (aperture shape, aperture size, target duration, retention interval), and characteristics of the observer (allocation of attention, imagination, age, expectations and strategies, eye fixation, eye movements, monocular or binocular view, vantage point, confinement, prior exposure, expertise, arousal, pathology) on boundary extension are reviewed. Connections of boundary extension to other cognitive phenomena and processes (evolutionary adaptation, Gestalt principles, illusions, psychophysics, invariant physical principles, aesthetics, temporal boundary extension, normalization) are noted, and theories and theoretical considerations regarding boundary extension (multisource model, boundary transformation, mental imagery, 4E cognition, cognitive modularity, neurological mechanisms of scene representation) are discussed.

Memory-based predictions prime perceptual judgments across head turns in immersive, real-world scenes

Each view of our environment captures only a subset of our immersive surroundings. Yet, our visual experience feels seamless. A puzzle for human neuroscience is to determine what cognitive mechanisms enable us to overcome our limited field of view and efficiently anticipate new views as we sample our visual surroundings. Here, we tested whether memory-based predictions of upcoming scene views facilitate efficient perceptual judgments across head turns. We tested this hypothesis using immersive, head-mounted virtual reality (VR). After learning a set of immersive real-world environments, participants (n = 101 across 4 experiments) were briefly primed with a single view from a studied environment and then turned left or right to make a perceptual judgment about an adjacent scene view. We found that participants' perceptual judgments were faster when they were primed with images from the same (vs. neutral or different) environments. Importantly, priming required memory: it only occurred in learned (vs. novel) environments, where the link between adjacent scene views was known. Further, consistent with a role in supporting active vision, priming only occurred in the direction of planned head turns and only benefited judgments for scene views presented in their learned spatiotopic positions. Taken together, we propose that memory-based predictions facilitate rapid perception across large-scale visual actions, such as head and body movements, and may be critical for efficient behavior in complex immersive environments.

Object-based attention during scene perception elicits boundary contraction in memory

Boundary contraction and extension are two types of scene transformations that occur in memory. In extension, viewers extrapolate information beyond the edges of the image, whereas in contraction, viewers forget information near the edges. Recent work suggests that image composition influences the direction and magnitude of boundary transformation. We hypothesize that selective attention at encoding is an important driver of boundary transformation effects, selective attention to specific objects at encoding leading to boundary contraction. In this study, one group of participants (N = 36) memorized 15 scenes while searching for targets, while a separate group (N = 36) just memorized the scenes. Both groups then drew the scenes from memory with as much object and spatial detail as they could remember. We asked online workers to provide ratings of boundary transformations in the drawings, as well as how many objects they contained and the precision of remembered object size and location. We found that search condition drawings showed significantly greater boundary contraction than drawings of the same scenes in the memorize condition. Search drawings were significantly more likely to contain target objects, and the likelihood to recall other objects in the scene decreased as a function of their distance from the target. These findings suggest that selective attention to a specific object due to a search task at encoding will lead to significant boundary contraction.

Complexity Matters: Normalization to Prototypical Viewpoint Induces Memory Distortion along the Vertical Axis of Scenes

Scene memory is prone to systematic distortions potentially arising from experience with the external world. Boundary transformation, a well-known memory distortion effect along the near-far axis of the three-dimensional space, represents the observer's erroneous recall of scenes' viewing distance. Researchers argued that normalization to the prototypical viewpoint with the high-probability viewing distance influenced this phenomenon. Herein, we hypothesized that the prototypical viewpoint also exists in the vertical angle of view (AOV) dimension and could cause memory distortion along scenes' vertical axis. Human subjects of both sexes were recruited to test this hypothesis, and two behavioral experiments were conducted, revealing a systematic memory distortion in the vertical AOV in both the forced choice (n = 79) and free adjustment (n = 30) tasks. Furthermore, the regression analysis implied that the complexity information asymmetry in scenes' vertical axis and the independent subjective AOV ratings from a large set of online participants (n = 1,208) could jointly predict AOV biases. Furthermore, in a functional magnetic resonance imaging experiment (n = 24), we demonstrated the involvement of areas in the ventral visual pathway (V3/V4, PPA, and OPA) in AOV bias judgment. Additionally, in a magnetoencephalography experiment (n = 20), we could significantly decode the subjects' AOV bias judgments ∼140 ms after scene onset and the low-level visual complexity information around the similar temporal interval. These findings suggest that AOV bias is driven by the normalization process and associated with the neural activities in the early stage of scene processing.

Systematic transition from boundary extension to contraction along an object-to-scene continuum

After viewing a picture of an environment, our memory of it typically extends beyond what was presented, a phenomenon referred to as boundary extension. But, sometimes memory errors show the opposite pattern-boundary contraction-and the relationship between these phenomena is controversial. We constructed virtual three-dimensional environments and created a series of views at different distances, from object close-ups to wide-angle indoor views, and tested for memory errors along this object-to-scene continuum. Boundary extension was evident for close-scale views and transitioned parametrically to boundary contraction for far-scale views. However, this transition point was not tied to a specific position in the environment (e.g., the point of reachability). Instead, it tracked with judgments of the best-looking view of the environment, in both rich-object and low-object environments. We offer a dynamic-tension account, where competition between object-based and scene-based affordances determines whether a view will extend or contract in memory. This study demonstrates that boundary extension and boundary contraction are not two separate phenomena but rather two parts of a continuum, suggesting a common underlying mechanism. The transition point between the two is not fixed but depends on the observer's judgment of the best-looking view of the environment. These findings provide new insights into how we perceive and remember a view of environment.

Scene Perception and Visuospatial Memory Converge at the Anterior Edge of Visually Responsive Cortex

To fluidly engage with the world, our brains must simultaneously represent both the scene in front of us and our memory of the immediate surrounding environment (i.e., local visuospatial context). How does the brain's functional architecture enable sensory and mnemonic representations to closely interface while also avoiding sensory-mnemonic interference? Here, we asked this question using first-person, head-mounted virtual reality and fMRI. Using virtual reality, human participants of both sexes learned a set of immersive, real-world visuospatial environments in which we systematically manipulated the extent of visuospatial context associated with a scene image in memory across three learning conditions, spanning from a single FOV to a city street. We used individualized, within-subject fMRI to determine which brain areas support memory of the visuospatial context associated with a scene during recall (Experiment 1) and recognition (Experiment 2). Across the whole brain, activity in three patches of cortex was modulated by the amount of known visuospatial context, each located immediately anterior to one of the three scene perception areas of high-level visual cortex. Individual subject analyses revealed that these anterior patches corresponded to three functionally defined place memory areas, which selectively respond when visually recalling personally familiar places. In addition to showing activity levels that were modulated by the amount of visuospatial context, multivariate analyses showed that these anterior areas represented the identity of the specific environment being recalled. Together, these results suggest a convergence zone for scene perception and memory of the local visuospatial context at the anterior edge of high-level visual cortex.SIGNIFICANCE STATEMENT As we move through the world, the visual scene around us is integrated with our memory of the wider visuospatial context. Here, we sought to understand how the functional architecture of the brain enables coexisting representations of the current visual scene and memory of the surrounding environment. Using a combination of immersive virtual reality and fMRI, we show that memory of visuospatial context outside the current FOV is represented in a distinct set of brain areas immediately anterior and adjacent to the perceptually oriented scene-selective areas of high-level visual cortex. This functional architecture would allow efficient interaction between immediately adjacent mnemonic and perceptual areas while also minimizing interference between mnemonic and perceptual representations.

Spatial Scene Memories Are Biased Towards a Fixed Amount of Semantic Information

Scene memory has known spatial biases. Boundary extension is a well-known bias whereby observers remember visual information beyond an image's boundaries. While recent studies demonstrate that boundary contraction also reliably occurs based on intrinsic image properties, the specific properties that drive the effect are unknown. This study assesses the extent to which scene memory might have a fixed capacity for information. We assessed both visual and semantic information in a scene database using techniques from image processing and natural language processing, respectively. We then assessed how both types of information predicted memory errors for scene boundaries using a standard rapid serial visual presentation (RSVP) forced error paradigm. A linear regression model indicated that memories for scene boundaries were significantly predicted by semantic, but not visual, information and that this effect persisted when scene depth was considered. Boundary extension was observed for images with low semantic information, and contraction was observed for images with high semantic information. This suggests a cognitive process that normalizes the amount of semantic information held in memory.

Predictive Processing of Scene Layout Depends on Naturalistic Depth of Field

Boundary extension is a classic memory illusion in which observers remember more of a scene than was presented. According to predictive-processing accounts, boundary extension reflects the integration of visual input and expectations of what is beyond a scene's boundaries. According to normalization accounts, boundary extension rather reflects one end of a normalization process toward a scene's typically experienced viewing distance, such that close-up views give boundary extension but distant views give boundary contraction. Here, across four experiments (N = 125 adults), we found that boundary extension strongly depends on depth of field, as determined by the aperture settings on a camera. Photographs with naturalistic depth of field led to larger boundary extension than photographs with unnaturalistic depth of field, even when distant views were shown. We propose that boundary extension reflects a predictive mechanism with adaptive value that is strongest for naturalistic views of scenes. The current findings indicate that depth of field is an important variable to consider in the study of scene perception and memory.

Blurring the boundary between models and reality: Visual perception of scale assessed by performance

One of the primary jobs of visual perception is to build a three-dimensional representation of the world around us from our flat retinal images. These are a rich source of depth cues but no single one of them can tell us about scale (i.e., absolute depth and size). For example, the pictorial depth cues in a (perfect) scale model are identical to those in the real scene that is being modelled. Here we investigate image blur gradients, which derive naturally from the limited depth of field available for any optical device and can be used to help estimate visual scale. By manipulating image blur artificially to produce what is sometimes called fake tilt shift miniaturization, we provide the first performance-based evidence that human vision uses this cue when making forced-choice judgements about scale (identifying which of an image pair was a photograph of a full-scale railway scene, and which was a 1:76 scale model). The orientation of the blur gradient (relative to the ground plane) proves to be crucial, though its rate of change is less important for our task, suggesting a fairly coarse visual analysis of this image parameter.